Is it really impossible to backup Scale apps to EXTERNAL storage?

[sammael]

Hi,
I recently sold my proxmox server where I ran a docker dedicated vm as I distributed the containers and other vms to 2 truenas scale's, and while it's working perfectly, I cannot help but wonder what would happen if both drives in the mirror the apps run on happen to fail at once. I've searched this forum and reddit, and couldn't find an answer or the answer was generally that it's not possible. I'm not asking about "heavyscript", which I use and successfully used before to restore, but it only works locally.

Sorry if I misunderstood anything during my "research" of how to achieve this, but seems to me that:
- I can't use hostpaths and snapshot/replicate those because of the host path validation doohickey and even if I could I think some apps have hardcoded pvc's
- I can't snapshot the ix-applications dataset, for reasons delving into technobabble beyond my comprehension

So then is my only option to do what I've been doing so far and which takes literal hours to complete for about 30 apps? Which is:
- manually shutting down apps and mounting PVC's and using tar to create an archive which I can then copy to external storage
- manually backing up any databases with pgadmin and mysqlworkbech

IF my above assumptions are correct, I cannot help, but ask:
- How are paying enterprise customers fine with this?
- Are apps not meant to be used in production? (surely something designed to be un-backupable can't be production ready?)
(I realize I can't expect answers about the enterprise side of truenas here, so how much would it cost me to pay for 1 month of ix support so I can ask directly?)

Thanks for any insights

 

[sretalla]

I can't use hostpaths and snapshot/replicate those because of the host path validation doohickey and even if I could I think some apps have hardcoded pvc's

I don't see how that's correct... nothing stopping the snapshot or replication of a host path used in an app as far as I can see.

I can't snapshot the ix-applications dataset, for reasons delving into technobabble beyond my comprehension

There are already snapshots of those... did you look under zfs list -t snap

 

[sammael]

There are several apps which won't start with host storage that is also shared shared due to the host path validation and for a good reason, I've seen threads about host path validation "breaking" apps, and to make them start you have to either use pvc or nfs mount for storage, in case of pvc, you can't replicate that as it's not dataset, and in case of nfs mount you can't use that for db storage.

Yes there are many snapshots of the ix-application dataset, however they are local, and afaik from reading other's experiences with trying to snapshot and restore the ix-applications dataset it doesn't work and the data doesn't get restored properly and the apps will not start. I cannot afford to try it myself, because I do not have external backup of the apps in case something breaks.

Last time I tried to set up manual snapshots of ix-applications was when I had spare system to break, it ended up creating about 12000 snapshots in short span of time, and the restore didn't work even just from a local manual snapshot of ix-applications dataset making trying to replicate it externally moot.

I understand all I'm saying is I've read here and there that it doesn't work, but nowhere in documentation is a procedure to backup apps to external storage documented as far as I was able to find, so the random threads about folks trying and finding it not working is all I got. I didn't want to name the thread "How to backup apps externally?" as googling that provides threads of that name with either suggestions of using heavyscript and following discussion how that is local only, or suggestions about replication the whole dataset with following posts saying it doesn't work.

@sretalla I don't know if you personally use any apps, but if you do, what is your backup strategy? Let's say the pool hosting your apps gets corrupted irreversibly, how do you get your data back? I'm not trying to be rude, I am genuinely interested as it is something I am worried about and cannot find valid strategy to backup externally. If you do not use apps, is it because you do not consider the apps system production-ready ?

 

[sretalla]

There are several apps which won't start with host storage that is also shared shared due to the host path validation and for a good reason, I've seen threads about host path validation "breaking" apps, and to make them start you have to either use pvc or nfs mount for storage, in case of pvc, you can't replicate that as it's not dataset, and in case of nfs mount you can't use that for db storage.

OK for the apps forcing PVC... those fall under the "already have snapshots" category.

You can use heavyscript to create backups (which then also appear as snapshots) which can then be put back into a pool and restored from. https://github.com/Heavybullets8/heavy_script

I think there's some confusion around the validation issue you're making with folks who insist on sharing out their data via SMB and using that same path as a host path in the app... this is where the problem of validation is occurring.

If you don't need to SMB share your app's data in addition to using that data through the app itself, no issue.

@sretalla I don't know if you personally use any apps, but if you do, what is your backup strategy? Let's say the pool hosting your apps gets corrupted irreversibly, how do you get your data back? I'm not trying to be rude, I am genuinely interested as it is something I am worried about and cannot find valid strategy to backup externally. If you do not use apps, is it because you do not consider the apps system production-ready ?

The apps I use don't require me to share anything separately via SMB, so I'm able to use host path and also replicate those paths to other systems.

I'll freely admit to those apps not being mission-critical for me and I haven't really tested the restore process thoroughly, but I don't see how there would be an issue to put things back as they were in a host path and have it continue to work.

 

[sammael]

You can use heavyscript to create backups (which then also appear as snapshots) which can then be put back into a pool and restored from. https://github.com/Heavybullets8/heavy_script

I already use heavyscript and have daily backups set up locally. This is what I am confused about, I cannot find a documented way to replicate & restore heavyscript snapshots, maybe I'm just too dense to understand it I don't know. When I have time I will set up a test vm with truenas, install few apps and play around with the replication & restoring of ix-applications - I tried this quite a few versions back previously and it didn't work, am I to understand it should now, and is an officially recommended strategy to backup apps?

If you don't need to SMB share your app's data in addition to using that data through the app itself, no issue.

Sadly, I do - I have folks in the household who absolutely refuse to consume media in any other way than browsing the network share manually via total commander and playing the file in the local player.

The apps I use don't require me to share anything separately via SMB, so I'm able to use host path and also replicate those paths to other systems.

If I understand this correctly then, you backup only data not the apps themselves? So in case of a failure, you'd have to manually recreate all the apps with all the settings and assign the restored data as host path? I could set it up like this, the folks who want the network shares be damned, but it still sounds to me like a lot of work in case of actually needing to restore, I mean lots of apps have multitude of custom settings/passwords/ports/proxies/env variables/etc - how do you go about backing this up?

 

[sretalla]

If I understand this correctly then, you backup only data not the apps themselves? So in case of a failure, you'd have to manually recreate all the apps with all the settings and assign the restored data as host path?

I replicate it all, but indeed it would probably just be simpler to handle the data for apps that just have one or 2 host paths (config/data type of thing).

I hear you on the more complex apps and don't have a great answer for that other than the heavyscript backup which I haven't extensively tested the process for, so can't say if that's great or terrible, but I understand that the app config itself should be covered by the heavyscript backup.

 

[sammael]

Thank you for replies. Perhaps with upcoming truenas versions the app system will continue to become more robust and flexible far as backing up go - i.e. not having to rely on external script. Speaking of which - I have used heavyscript to restore twice previously and it worked flawlessly, although it's a bit inconvenient it has to recreate whole cluster, i.e. you can't just roll back 1 app.

 

[sretalla]

you can't just roll back 1 app.

I would guess... but again, have not tested, that you could selectively either rename the snapshots to not be in the expected location when doing a restore and then put them back afterwards.

Have a look at zfs list -t snap | grep backup-HeavyScript and imagine using zfs rename to put the unwanted ones in a temp location, then moving back after the restore.

 

[sretalla]

Also, did you manage to find this?:

Backup, Migrations and Restore | TrueCharts

This guide has been written with the best efforts of the staff and tested as best possible. We are not responsible if it doesn't work for every scenario or user situation.

truecharts.org

 

[victort]

I’m assuming you know about the option to turn off host path validation in the advanced settings of the apps?

 

[sretalla]

If the plan is really to access both via the app and via SMB, that's not the recommended solution to that problem.

But, yes, I do know that. (I guess OP does too based on the conversation, I may be wrong though, so thanks for checking)

 

[sammael]

@victort I do indeed know about that, but I opted to follow the security recommendations from ix as I grew concerned about safety of my data as more and more apps access the same media directory as the stubborn folks requiring the smb share do. Although based on the discussions and my own experiences at https://www.truenas.com/community/t...enas-scale-hostpathvalidation-setting.109100/ I guess I shouldn't have bothered and go with host paths from the start as in Cobia it will just become general warning that will let you proceed, but it's a bit too late for that now

@sretalla Thanks for the link, I indeed discovered that one later and wanted to include that in the tests I wanted to setup in VMs, but my recent experiences with replication reproducibly causing kernel panic and or crash (https://ixsystems.atlassian.net/browse/NAS-122883) very well put me off any experimentation with replication at the moment and the time I wanted to spend testing it in VMs was spent converting the tasks to rsync and rsyncing ~26TB of data. Not to mention my ups battery just decided to go and I have to deal with replacing it (thank god for capitalism and prime next day delivery).

I can see now there are several (to me seemingly unconfirmed and/or with conflicting reports of success or the lack thereof) methods I could potentially use to backup / restore apps externally, but putting myself into my shoes as it were, as a hobbyist homelab user, some of these delve into the cli territories I am not altogether comfortable/knowledgeable enough with.

I guess it is just a case of overt expectations on my side. I always liked Truenas, been using it since Core 9, but let's be honest, for my use I could've just as well run anything else (zfs on linux, openmediavault, unraid etc). But I got used to Truenas, and used it along a docker dedicated VM running on intel nuc, where backing up apps was as simple as "docker compose pause && cd .. && sudo tar cxvf stack.tar.gz stack/ && cd stack && docker compose unpause". Then Scale came along and my excitement was immeasurable as I made 900£ from selling the docker VM host as I was able to move all the stacks to Scale and I wanted one click backup/restore to any point in time as well. But I cannot find as simple a way to backup / restore as I could before (again probably just because of my technical inexperience).

In ideal world I would want a "Snapshot" and a "Restore" button on each app, where snapshot would, well, snapshot, and restore would show me all snapshots and let me roll the app back wherever I want and/or let me clone a snapshot to new app to test something etc.

 

[sammael]

Some late (quite busy at work & home atm) update:

I've successfully setup the replication according to instructions from https://truecharts.org/manual/SCALE/guides/backup-restore as I'm running 22.12.3.3 where the replication crashing was fixed I believe. Next weekend I'm gonna try to escape the real life long enough to set up a test truenas vm to simulate system failure and actually try to restore the replicated apps.

The replication task sometimes give me warnings like this, presumably these can be ignored?:

Code:

[zettarepl.transport.zfscli.exception] Caught 'failed to create mountpoint: Read-only file system' but ('pool1/apps-backup-truenas/releases/tdarr/volumes/pvc-f98c7f8f-2933-42ca-a5b5-97e9b262d493', 'ix-applications-backup-HeavyScript_2023_08_09_11_09_05') is present on remote side. Assuming replication success.