speeding up rsync task

[William Bravin]

Hello all

I have 2 instances of truenas core. I perform 4 rsync tasks every week to replicate my medias and documents (about 9TB). This takes anywhere between 11 to 14 hours every week. All works fine

in an effert to improve my architecture and performance, i tend to tinker ad Infinium

At the moment both boxes are connect trough a netgear gs724. All connected via gig lan and cat 6 cables

Is there a way to drastically increase the speed of the replications ie connect both boxes via fibre link using a different process.

Thank you in advance for your help

 

[QonoS]

10G could help.

Test for your fastest RSYNC/SSH hash/encryption settings. (Seems FreeBSD comes with chacha20-poly1305@openssh.com + umac-64-etm@openssh.com as default which is most times best in my experience.)

Rsync skips hashing (which is relative slow) if size and modification timestamp are identical on both ends. Check if your rsync tasks do this skipping. I assume only some gigs change each day. So files that need hashing should be very few and skipping which is fast be the norm.

rsync(1)

www.freebsd.org

ssh(1)

www.freebsd.org

ssh_config(5)

www.freebsd.org

 

[Constantin]

I’d drop rsync for this task and use zfs send instead. It’s a much more efficient protocol since it only sends that which has changed vs. rsync which will exhaustively go up and down every directory tree looking for changes.

Worse, while zfs send will transmit a snapshot frozen in time and hence can execute a faithful copy every time, a rsync task may result in corruption or incomplete sends when the underlying data is changing while rsync operates. This is a common issue with Time Machine backups for example, where the bands may be listed as busy / inaccessible because they were recently being written to.

Thus, I’d explore how to take advantage of ZFS send instead. It’s also ssh encrypted and the recipient machine can be set up as a passive recipient that cannot decrypt the data it’s receiving.

 

[danb35]

I’d drop rsync for this task and use zfs send instead.

Ditto. In addition to what Constantin said, rsync is optimized for bandwidth, not for performance. Updating the network connection could certainly help, but ZFS replication would be the first change to make.

 

[NugentS]

Snapshots are the way forward as both @Constantin and @danb35 have said.
I will add that using snapshots will drastically reduce your transmission time down to probably minutes as it will only send changed data (unless you are changing 9TB of data every week)

 

[William Bravin]

Hello all

Thank you both for your recommendation. As i understand it zfs send only creates a read only snapshot. Both of my server have r-w-a-d functions to all files.

@Constantin
I not only use the end server as a backup, I also need it to be ready in the event that the first server is down for what ever reason. (i do not like a house with no music). I have one instance of kodi mapped to one server and another kodi instance mapped to the second server. if one server is down i just start the other kodi and continue where i left off. so i do not think the snapshot option

@QonoS

sorry i am not technical enough to understand your message. for me inserting a 10gig card in each server and connect them is a much simple task for me to execute

i add and or delete about 500Gb per week. so not much changes

however i am very curious checking the rsync task for sipping how would i do this?

once again thank you both for your help

 

[NugentS]

You are correct the ZFS send | ZFS Recv will leave you readonly on the destination and you can't change that without buggering up the process

 

[QonoS]

however i am very curious checking the rsync task for sipping how would i do this?

Code:

[user@truenas /var]# echo 'test' > f1

[user@truenas /var]# rsync -avv f1 f2
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
f1
total: matches=0  hash_hits=0  false_alarms=0 data=5
sent 93 bytes  received 102 bytes  390.00 bytes/sec
total size is 5  speedup is 0.03

[user@truenas /var]# rsync -avv f1 f2
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
f2 is uptodate
total: matches=0  hash_hits=0  false_alarms=0 data=0
sent 48 bytes  received 105 bytes  306.00 bytes/sec
total size is 5  speedup is 0.03

So here you see (1) file1 is created, (2) file2 is an exact copy of file1 (modification time and size is identical), (3) rsync skips copying with message "file2 is uptodate".

Important is that modification timestamp remain identical. If you initially rsync'd data without "-a", meaning "archive mode". That mode contains "-t" which stand for "preserve modification times". Then data is identical but modification timestamp are not. Because without "-a" or "-t" a new timestamp is applied reflecting the time of copying, resulting in rsync processing the whole file in the next run.
So just use rsync with "-a" or "-t" and it should speed up from the second run on.

EDIT:
TrueNAS Rsync Tasks seems to preserve modification times by default.

 

[Constantin]

Is there a way to drastically increase the speed of the replications ie connect both boxes via fibre link using a different process.

If rsync is a must and much of the data is dormant, then my suggestion would be to look into upgrading the RAM to 64GB and adding a persistent, metadata only L2ARC on both ends because rsync will be doing its reads and compares on both ends. Rsync does a bazillion directory traversals as it checks every file for changes.

While 10G helps with latency, metadata-only L2ARC that makes a much bigger difference re the time needed. See my testing on metadata-only L2ARC and dormant pools and the follow on with sVDEVs if you want to go even more aggressively into speeding up metadata retrieval.

Sizing for metadata-only L2ARC depends in part on how many files you’re trying to keep a track of. I used a 1TB drive for a 50TB pool, but if you have a lot of files then you may need more metadata space. Any decent SSD will do, fast reads are a plus since a persistent L2ARC is not written to much in your use case.

Keep in mind that metadata only L2ARC has to be specified explicitly, as does persistency. Neither L2ARC property is set that way by default. The nice thing about L2ARC is that it can fail 100% and the pool will be fine. But you have to have enough RAM to prevent crowding the ARC.

sVDEVs are even faster but unlike a L2ARC failure, a dead sVDEV will take the whole pool with it. So sVDEVs have to be as safe as the rest of the pool (I use a 3-way Intel SSD mirror) and since one cannot set a metadata-only sVDEV explicitly, you also have to contend with what space small files will need. sVDEVs also currently do not report how full they are, etc. so I'd use L2ARC unless you want to be on the bleeding edge. L2ARC is simply more mature.

Regardless of what approach you settle on, simple browsing of your media file tree will also speed up significantly, these upgrades are not just for rsync.

 

[William Bravin]

Thank you all for your suggestions
I think that the best way forward would be to: 1) increase the ram on both servers to 64gb 2) install an ssd on my media rig (the other server already is equipped with this) to enable L2arc and 3) eventually add 10gb cards to both server and connect them to each other.. do you foresee any issues with mellanox or intel cards?

thank you for your help

 

[NugentS]

intel > mellanox

 

[William Bravin]

tks

@NugentS

I looked at your setup and it looks interesting. Is putting drives in mirrored pairs using less overhead?

I will need to read and learn about L2arc and slog. I think this could be my next project

once again thank you

 

[NugentS]

Mirrored pairs gets me higher IOPS at the cost of space.

I run VM's on the mirrored SSD's and HDD's (I use the HDD's for unimportant VM's - trying to reduce the wear and tear on the SSD's). In practise all the VM's go on the SSD's - so I could turn the HDD's into a RaidZ2/3 Array for more resiliency and a little more space (that I don't need at this point) at the cost of a much higher resilver time - so I leave it as is (not to mention I hate restoring that much data)

Because I am using iSCSI - I use a SLOG for maximum safety at best speed and I use the special vdev that is tuned to hold a whole bunch of small files on SSD storage as part of the main data pool to improve access latency to those files from my workstation. It mostly seems to work.

Also being mirrors has the added advantage that I can remove the svdev / slog from the pool without trashing the pool if I feel like reconfiguring something to try out a new config (after ensuring up to date backups). Of course I have made it much harder for myself by manually splitting the SLOG's up, then mirroring them on two different pools (there are 2 Optane's with multiple mirrored partitions on different pools)

I have tested restoring from both backup devices as well.

The QNAS holds snapshots of anything important whilst BackupNAS does file by files backups from the SMB shares and also backs up the VM's individually. Finally use B2 to hold the really important stuff and am waiting for a friend to rejig his network and we will use each others NAS's to store important snapshots as well. VPN is up but the rest isn't ready yet.

If I was doing this again, I would do things a little bit differently. Fewer CPU cores with higher GHz on the main NAS. The other 2 both don't matter and I can't change them anyway.

When Scale looks better I will probably shift to Scale for Docker over Jails.

 

[Constantin]

Not to be persnickety, but it's not just L2ARC, it should be metadata-only, persistent L2ARC. Otherwise, you end up with a general-purpose L2ARC, where files will compete for space with metadata, resulting in metadata getting flushed. Also keep in mind that it might take a couple of rsysncs before the cache gets "hot".

Lastly, review @jgreco's excellent 10G primer and other resources on how to set up and tune 10GbE connections.

 

[William Bravin]

Folks thank you so much for sharing your experience.

I like your idea of putting the snapshots on the qnas. I could do this with my buffalonas every week and to my second nas once a month to the second serve. This in itself could save an enormous amount of time. .

Here is a definite dumb question

if i take snapshots from server a and put them on the buffalo .... they will be read only ok
can i restore the snapshots from the buffalo to second server if so will these files be read only or will the permissions on the server override the read only flag?

thank you so much

 

[NugentS]

Does the buffalo do ZFS?
My QNAS does (repurposed QNAP running TrueNAS Core)

 

[William Bravin]

i do not think so i use samba on it
need to find out

 

[NugentS]

Given the hardware requirements for ZFS I doubt it will on a cheap NAS (assuming that what the Buffalo is) so you are not going to be able to do ZZFS snapshots on it.

Other options I don't know about

 

[William Bravin]

hello all

I'm baaack

i had to rebuild my sata card failed on my main server . In addition i was experiencing come problems the rebooting freenas (it would not start up again when rebooting) and other minor issues. During this time i am using my backup rig (called freenas2) with out any issues.

At this time i installed truenas core 13 on my main rig. (still not fully configured). now the server is called truenas
i gave my pools, my user, my shares my acl and my rsync modules. fine.

for whatever reason i cannot push data from my backup server (freenas2) to my main server (truenas) I will figure this out

I can however do a rsync pull in my truenas server from the freenas2 server. i find it extremely slow (i have a data set of 235 GB (my documents) and i am pulling this data from my freenas2 to my truenas server.

The connection between the 2 servers (by way of power line which is a a constant speed between 430 to 460Mb/sec) is now in it's 6th hour of processing.

What am i missing?

In addition i read that replication is much faster than using rsync. Is this correct?


thank you in advance your help

 

[sretalla]

In addition i read that replication is much faster than using rsync. Is this correct?

Replication uses zfs send|recv (which is behind the zettarepl command used as a front-end for it).

See the commentary in this thread about that.