RSync to Pull From Remote to TrueNAS

[modi]

Hi,
what is the proper way to pull data from another (non-TrueNAS) device over SSH+RSync to my TrueNAS machine?
I have established an SSH connection, but there seems to be no option to browse the remote files in TrueNAS's Web GUI, so I can choose which files to pull to my system.

Do I just have to SSH to the remote system and initiate Rsync over my TrueNAS CLI?

 

[sretalla]

You're requesting specifically to use Rsync, so the best way to do that would be to setup an Rsync Module on the remote system (which should identify the files you want copied).

Then set up an Rsync Task in TrueNAS to pull from that module.

 

[modi]

Is it not possible to do over SSH?

Also a side question, I presume it won't be an issue, but if I make a dataset specifically to dump all the Rsync files in, there would be no issues later moving those files to different datasets on TrueNAS correct (given ACL's don't interfere)

 

[sretalla]

Is it not possible to do over SSH?

It is, but as you say, you can't just browse the remote data (since you're not connected at the time you're doing that).

there would be no issues later moving those files to different datasets on TrueNAS

Be mindful of what the implications are of what you're saying... move between datasets isn't a move... it's a copy and then delete... but it will get the end result of a mv just the same. If you want to really have a "move" of the file, it needs to be in the same dataset.

if I make a dataset specifically to dump all the Rsync files in, there would be no issues later moving those files to different datasets on TrueNAS

Rsync isn't necessarily a "dump"... it can be used to maintain sync on an ongoing basis, so if that was your intent, it will be an issue... if you're just doing a one-time copy, fine.

 

[modi]

What would be the easiest way to initiate it over SSH? I'm fine just pulling the entire user's data over as that is my end goal, but I did want to test a few files first. Although not a big deal.
Can I just jump into CLI and Rsync it all over, or could that potentially break the connection?

Also yes, that makes sense about copy delete.
And yes I'm not trying to currently setup a tasked sync, just trying to get all my files from a remote system to this fresh one. So a one time copy.

Any suggested paramaters to add btw? Looks like Rsync automatically does checksum verification which is good. I just want to ensure if either ends lose connection, it doesn't mess up something or have to restart the entire sync on reconnect.

 

[Patrick M. Hausen]

What would be the easiest way to initiate it over SSH?

rsync -avz user@remote.system:/some/remote/path /mnt/pool/some/local/path?

Provided you have setup either password or public key authentication for user@remote.system.

 

[modi]

rsync -avz user@remote.system:/some/remote/path /mnt/pool/some/local/path?

Provided you have setup either password or public key authentication for user@remote.system.

That will continue to run if I do it over the shell menu on the web gui, and then logout of the webgui? or will it cancel itself?
How can I monitor where it's at after logging back in?

 

[Patrick M. Hausen]

You need to login to the TrueNAS via SSH (e.g. putty.exe or Terminal window on a Mac) and leave that window open until the operation is finished. Don't get used to the shell in the web UI - it's broken.

The -v flag in my command example will show you the progress as the files are copied. -a is "archive mode", which means "create a copy identical to the source at the destination" and -z means "compress and decompress while transferring".

 

[modi]

Ok will give it a go, thank you!

 

[modi]

Seems to be doing well.
I was getting a lot of disconnects between my server and the remote one, but I used screen and I haven't noticed it yet.

One thing is, is it normal for TrueNAS to saturate the transfer instead of attempting to balance workload? rsync is maxxing out 76MB/s and I notice if I try to play a game off the NAS over iSCSI it seems to have trouble during loadtimes compared to when rsync is not running. Maybe there's a setting for this I am missing?

 

[sretalla]

You could try the --bwlimit= switch in the command... specified in Kilobytes/second (1024 would be 1 Megabyte per second).

 

[modi]

Well my point really comes down to, am I going to have issues with workload balance if I have a movie going on one device, while gaming on another, and editing a video on another, while maybe TrueNAS is doing automatic backup to a remote server.
Like why would TrueNAS not be able to balance this workload?

 

[sretalla]

Like why would TrueNAS not be able to balance this workload?

It does... it's just not producing the result you want/need, so you'll need to intervene.

You either need to provide sufficient resources to handle maximum requirements for all your workloads at the same time, or you need to do something to say which workloads aren't as important (by throttling them as you can with rsync).

You could also look into setting up VLANs and setting the Codepoints for 802.1p to ensure that some traffic gets priority over others (but should mean you can get maximum for any app when there's spare)... that's a lot of work to set up for a result I would argue isn't important.