SOLVED RClone and --fast-list/Python Help

[Dreded]

Need to modify /usr/local/lib/python3.6/site-packages/middlewared/plugins/backup.py as referenced by this post... https://forums.freenas.org/index.php?threads/modifying-a-py-file-in-freenas.60017/ to add the option of --fast-list to B2 Backup but my very entry level python skills are not up to the task so have no idea what to add where.

Without the option we hit the class C transaction CAP rather quickly with a once a day backup of ~500GB and its quite costly at ~ $73/month (relatively costly as with the --fast-list option we never even come close to the cap and it costs $5/month).

 

[dlavigne]

Notes that this is coming in 11.3: https://redmine.ixsystems.com/issues/37358 (and is already in nightlies for the impatient on non-production systems).

 

[Dreded]

Notes that this is coming in 11.3: https://redmine.ixsystems.com/issues/37358 (and is already in nightlies for the impatient on non-production systems).

yes, this is a production system and how long until 11.3.. is there a timeline someplace?

I would really like a way of just getting it working now

 

[dlavigne]

The BETA series will start in early April. There will most likely be at least 2 BETAs and 1 RC before RELEASE. It's available in the nightlies now, but nightlies are considered experimental and not for production use.

 

[Dreded]

The BETA series will start in early April. There will most likely be at least 2 BETAs and 1 RC before RELEASE. It's available in the nightlies now, but nightlies are considered experimental and not for production use.

ok, was oping someone who understands python better than I would/could read the file in question and tell me where to add additional commandline argumants when using B2... guess I was to hopeful :)

 

[dlavigne]

I'm pretty sure you can still run this manually from the CLI (though have not tried myself to verify): https://rclone.org/b2/.

 

[Dreded]

I'm pretty sure you can still run this manually from the CLI (though have not tried myself to verify): https://rclone.org/b2/.

I fairly certain this would work... it would be nice to know the command that FreeNas is using so that I can resume my backup rather than create a new one(wrong root directory that sort of thing etc)

 

[Ixian]

Code:

/usr/local/bin/rclone --config /<config file> -v --stats 1s --fast-list --transfers 8 --exclude-from /<exclude file> sync <local-sync-directory> remote:<b2-bucket-name>

Example config file:

Code:

account = <B2 account code>
key = <B2 account key>
type = b2
bucket = <bucket name>
folder = <local folder>
fast_list = True

Example exclude file:

Code:

*.bif
*.iso

Should get you started. This is based on how FreeNAS calls Rclone if configured via the GUI, btw (though it's fairly standard).

11.3 does a much nicer job with this, I'm using it on a test server, but I would strongly advise against using that in production, overall, as it has a ways to go. My guess would be early summer for GA based on current roadmap progress but just a guess.

Edit: Obviously, replace the items in <> with your own info, i.e. sync /mnt/media/backups remote:Freenas-bucket1 , etc.

Edit2: There are some redundant entries in the above (I pulled the command, and temp files, from a test 11.3 b2 backup test I set up in the GUI) - for example, with regular RClone you don't need to specify the destination bucket in the command if it is already in the config, same with fast_list. I don't know if there's some Freenas-middleware requirement behind this or if (more likely) it's still in early testing and they haven't cleaned it up yet. I'll update the bug tracker for it to get feedback.

 

[Dreded]

Edit: Of course before all this I disabled the Freenas Cloud Sync Tasks as they would be rather redundant

really appreciate the help, gave me an idea of where to start... what I ended up doing was hitting the manual "Run Now" option in Freenas Cloudsyncs area then quickly doing a ps ax | grep rclone to read the command line that Freenas was using which btw is...

Code:

/usr/local/bin/rclone --config /tmp/tmpiq8gbaxl -v --stats 1s sync /mnt/Basin/Storage/Main remote:Helios/Main

which then prompted me to copy that file to where it wouldn't be instantly removed... looked at the file which contents where:

Code:

[remote]
account = <Account>
key = <API Key>
type = b2
bucket = <Remote bucket>
folder = <REMOTE folder>

so then I created a new config that would work for all 3 of my cloud backups...

Code:

[Helios_Backblaze]
account = <Account>
key = <API Key>
type = b2
bucket = Helios<--probably not necessary

placed that in /root/myRclone.conf (didnt want to place it in .config/rclone/rclone.conf just in case FreeNAS team decides to use that location in the future
which I then run via cron which i setup under Tasks>Cron to run twice daily with command

Code:

/usr/local/bin/rclone --config /root/myRclone.conf -v --stats 1s --fast-list --transfers 16 sync /mnt/Basin/Storage/Main remote:Helios/Main

 

[Ixian]

That's how I grabbed the files too.

I recommend you put scripts in a dedicated folder on a datastore outside of root, btw. I run all my init scripts that way.

 

[Dreded]

So after adding the --fast-list option for me it has made a HUGE difference, went from 93,670 Class C transaction for a single backup of 500GB to 165 Class C transactions for the same backup.