Improving Truenas Scale performance

[carlitos009]

Thank you to those who take the time to read. I used FreeNAS (or a variant) back in 2003 or around that time. I used it from time to time but I am looking to grow from synology to something a little more flexible (hardware flexibility).

To test I am using an Intel Core 2 Duo CPU E6750 @ 2.66GHz - which I must say runs a bit slow for my taste, but it is only for testing. I also have only 4 gigs of RAM, it was a spare machine to see how it runs. I know it does not have the minimum 8 gigs but it does run with services and there is free still free RAM.
I have to say that while a bit slow, the interface is fairly responsive for such an old cpu, and useful since setting it up doesn't happen too often.

I managed to create a test backup task from my Synology to the Truenas box but the performance I was getting was only about 20 megabytes per second. They are both connected on a unifi gigabit switch. I must point out that I am using the connection I had for my xc png box and it did transfer a full gigabit speed with the Synology since I had an iscis target on it. I like to know if the limitation is because of the RAM or the CPU or a combination. If that is the case, I will continue testing and then I will use a faster machine I have with more RAM which exceeds the minimum requirements.
If that is not the case, is there something I can do to make it improve the transfer speed? I have a quad port intel/hp nic which I could try if the NIC might be at fault, but I see it recognized as 1000 speed. Also, I did run windows 10 on that same truenas box to update bios and to explore the hardware and also POP os and then LXLE and it ran fast and gigabit transfer speeds, so any suggestions will be greatly appreciated.

By the way, I will actually leave this box as a remote backup with a bunch of HDDs on it in a remote location where the backups will take place after midnight and said location only has a down spedd of 25 megabirs per seconds, even with that in mind is useful but if I am to use another instance locally then I like to know how to make it faster or if I just need better CPU and more RAM.

Sincerely,

Carlos Alfaro

 

[Kris Moore]

Hard to say without more details from a ticket / debug file to investigate. I'd be worried about memory though, we do say 8G minimum for a reason, ZFS needs some room to breath on the ram side.

FWIW, Using my i5-9400 CPU @ 2.90GHz + 64Gb of ram, I can nearly max out 10G/s network connection here over SMB, without any performance tuning.

 

[carlitos009]

Thank you for your prompt reply. I failed to mention that I have two identical drives for testing, one for the OS and the other for data.
I might just load it on another machine I have with more ram and a faster drive. It does not have an intel NIC but faster in every respect plus it has 4 cores. I am still not using zfs on it because I am waiting to see if I will go all in given that I do not need much more than a box which can receive data for backup. If I can acomplish that then I can look into moving away from Synology but right now DSM runs great and does a lot

 

[carlitos009]

sorry, last one was a repeat

Hard to say without more details from a ticket / debug file to investigate. I'd be worried about memory though, we do say 8G minimum for a reason, ZFS needs some room to breath on the ram side.

FWIW, Using my i5-9400 CPU @ 2.90GHz + 64Gb of ram, I can nearly max out 10G/s network connection here over SMB, without any performance tuning.

By the way, pretty impressive you can saturate such a pipe over smb

 

[carlitos009]

Ok, I got TrueNAS Scale loaded on a capable machine. I have an AMD A8-5500 which is listed info on AMD A8-5500,
It has 4 cores up to 3.7 Gigahertz with a base clock of 3.2 Gigahertz - only 4 MB of L2 cache. I believe it is a capable processor for TrueNAS. I used it as both a win10 machine and as y POP OS box for a year or so. It also has 12 Gigs of RAM. It has a 120Gig drive for TrueNAS and a 1 TB HDD. When I transferred files to my Synology I would get burst of 90+ megabytes per second and would settle in 80+. I am however having the same issue I had with the older core duo PC I made with only 2 Gigs of RAM. This new machine is slightly better because I am getting almost 27 Megabytes per second transfer and I was getting between 20 and 25 with the older machine.
It is just as a note, I have an older pc and out of curiosity I loaded OMV and I am getting around the same, 20 to 25 Megabytes per second transfer.

Am I doing anything wrong? I am doing the backup from Synology using rsync and I am getting only that speed. For testing I started a backup of around 500 Gigs from my synology around 10 am this morning and it is only about 70% done and it is almost 4 pm. The top speed is around 27 but it stays mostly around 25 megabytes per second.

Any suggestions?

 

[yottabit]

rsync is never going to be as fast as a pure copy via smb or nfs, especially if you are tunneling over SSH...

smb can win on single-threaded performance... nfs can win on multi-threaded performance...

It's also possible that your source array on the Synology just can't deliver bits faster? What is the iowait and load on the Synology during these transfers? And on TrueNAS?

 

[carlitos009]

rsync is never going to be as fast as a pure copy via smb or nfs, especially if you are tunneling over SSH...

smb can win on single-threaded performance... nfs can win on multi-threaded performance...

It's also possible that your source array on the Synology just can't deliver bits faster? What is the iowait and load on the Synology during these transfers? And on TrueNAS?

Thank you for the prompt reply. I was under the impression rsync would be as fast as smb but I am wrong.

The iowait is around 40% and peaks at around 67% while the trasnfer is going on and CPU is at about 48% max, haven't seen it go above that in a while. While i am looking at the DS interface it says 24% user, 10% system and as before, iowait is between 40 and 67%.

I will try NFS as a means of getting my backup next.

 

[yottabit]

Right, couple of clarification points:

• rsync can be faster, but only if you are transferring a large amount of data and the destination already contains most of it (i.e., only transferring changes)
• That iowait is very high; you are being limited by disk performance
• That cpu time is also very high, if you are using a cpu with hyperthreading (essentially 50% load is maximum "real" core performance, and anything over 50% is a bonus due to predictable workload that hyperthreading can take advantage of)

Hope this helps!

 

[carlitos009]

Right, couple of clarification points:

• rsync can be faster, but only if you are transferring a large amount of data and the destination already contains most of it (i.e., only transferring changes)
• That iowait is very high; you are being limited by disk performance
• That cpu time is also very high, if you are using a cpu with hyperthreading (essentially 50% load is maximum "real" core performance, and anything over 50% is a bonus due to predictable workload that hyperthreading can take advantage of)

Hope this helps!

Thanks.

I do believe I am looking at a few reasons. I am running a few services on this things, including Surveillance station and the array is in fact getting written information from a network camera and when there is motion it is stored on the drive. I have a second Synology and I will attempt to generate the backup from that one to see if I get the same result or better - hopefully.

Thank you

 

[carlitos009]

Ok, so I tried running two simultaneous backups from both synology units and I am getting right around 50 megabytes per second at the TrueNAS box, in line with the output of both. Definitely the problem is at the Synology units.

I will load up the old machine with only 4 gigs or ram (core duo) and see if I get the same results.

Anyone can suggest how to backup to an NFS share? I want to try something other than rsync to see if it will be faster.

 

[SnoppyFloppy]

Anyone can suggest how to backup to an NFS share? I want to try something other than rsync to see if it will be faster.

Is the nfs share on a zfs filesystem? If so, I would prefer ZFS snapshot replication. Its in the UI > Data Protection > Replication tasks or something like that. I think you can even replicate to a file in case the destination is not a zfs file system but I've never tired that.

 

[carlitos009]

Is the nfs share on a zfs filesystem? If so, I would prefer ZFS snapshot replication. Its in the UI > Data Protection > Replication tasks or something like that. I think you can even replicate to a file in case the destination is not a zfs file system but I've never tired that.

Is not an NFS share at the moment. I experimenting with the ideal setup.

I have two NAS units locally with BTRFS. My intent is to eventually move everything to TrueNAS - but that will take time. My Synology units are pretty new and they represent an investment. I eventually want to be "cloud free".

I want to start testing by locating a TrueNAS (DIY) at a location where I can use it as a target to backup my Synology units. I have not determined if it is better to move one of my Synology units to that location and then it is way easier to do backup and replication. I want to learn more since I have not used TrueNAS / FreeNAS since around 2003-2005 I think.

I have most of my data backed up in my Synology units and an online provider. If my plan works I will start migrating everything to TrueNAS scale by the time is finalized. In the mean time I will continue experimenting on how to backup data to it. I already have my pFsense boxes talking via a peer-to-peer VPN connxection so I can just point to them and not worry about indidvidual VPN client/servers, my locations are already communicating. I will do a backup locally and then just link to the backup once I have the trueNAS at the new location and it will only sync up differences going forward.

I am finding rsync slow but that looks like it is due to all the services I have running on my Synology units and I can live with that since the big "sync" will be a one time deal.

What you propose of making it just a snapshot sounds like a more efficient way of doing it but I will have to build a more resilient system (more drives RAM and redundancy) to make it effective. Since I am just testing now I am willing to live with having just a single drive on my remote NAS. I was testing OVM as well but I think for future proof TrueNAS Scale is the way to go.