Deleted an App by mistake

[NugentS]

I deleted a container today.
I meant to delete a Unifi Controller, but deleted instead a Storj Node container - Oops. Brainfart mode on
I tried to restore using truetool - which didn't work (can't say I am suprised)

I use the IX version of the node (there is a backup in the truetool backup), directly on Scale through K3S

The good news is that I have the ID and Storage/Config on custom host paths - so it was (at least in theory) just a matter of rebuilding the container and pointing it at them however the container config contains some important info without which the node would have been lost.

My issue here is not about truetool - they admit that the restore is untested (which makes the backups pretty useless but whatever - I guess things will improve)

The whole point of a container is that it should be recreatable. If I use portainer (and I do) then I would have a stack file, so could just recreate the container with a few key presses. If I use docker run, then I would hope I would have a script file that creates the container with the correct parameters. As far as I can tell there is no such thing in Scale and K3S. I had a notepad file with the container parameters in (well most of them - I had to go to the firewall for the port numbers) but there is no mechanism to backup the container configuration that I can find.

Snapshots you may think - nope. We aren't meant to snapshot the ix-application dataset, and anyway I do not want to roll back all the apps when I screw up like this. There ought to be a method of backing up the container configuration as text / json / whatever files so that I can recreate the container. I move the app config and other storage away from the container so I can back them up using normal mechanisms, but there appears to be no backup of the container configuration itself, and no sensible way of doing so.

Is there, and I just don't know where it is - or is this something missing?

 

[MisterE2002]

Please correct me if i say things that are incorrect. The default of TrueCharts is to use PVC. That create's a zvol for the "configuration". The idea to make a snapshot of the configuration if you update a container (image) to a new release. So you can revert both. Not really tested this.

The image itself is throw away anyway. It is immutable.

I do not like that i rather use a normal dataset so i can "see" the files. Of 1 image i noticed the inside path to be "/config". Not sure if that is the default. Their documentation is lacking. (I usually first start a container to figure out the inside paths and maps for the second run)

If you used PVC did it also removed the zvol on deletion?

 

[NugentS]

I don't use PVC deliberately. I always use a host path to move the /config (or whatever) to another dataset that I can see. When I first tried Docker it was how things were explained to me - always use a seperate path (dataset) for the config, so the container can be recreated easily. Data (if container specific) is also on its own path (dataset) and that practise I kept when I started using apps on Scale.

As a result I can snapshot and or backup the app configs whenever I feel like it. The container is sort of throwaway (as it should be).

I have migrated all but three apps away from K3S now anyway to a VM running portainer as I do not like the obscure nature of the app setup or its utterly crap routing. The three apps like to have direct access to datastores on the NAS (Unmanic, Duplicati, Storj) and where the routing issues don't matter much (still effected, but not that important). I am assuming the routing issue is still on Bluefin as IX declined to fix it saying it wasn't important. (I disagree, but its their NAS software)

I assume any zvols got deleted when I deleted the container, but I don't know (it would make sense) as I have no reference to / of them. Portainer for me is a much better solution as I can back it up either internally to Portainer, or by backing up the entire VM. Even in Portainer I use external volumes for config folders mostly unless the app really doesn't have any config - like a socks5 proxy, so loss of the portainer server is not 100% critical. Everything is done by stacks so its very recreatable.

 

[MisterE2002]

so, if i understand correctly. You use Host Path normally, but you deleted a container which was using PVC?

Did some reverse engineering experiments in a VM for my own shits and giggles :)
I first expected a truecharts install to make a zvol. But this does not seem the case.
I installed "sabnzbd"

Code:

root@truenas[/mnt]# zfs list -t volume
NAME                     USED  AVAIL     REFER  MOUNTPOINT
tank/zvolletje/zvol01    104M  7.91G       56K  -
tank/zvolletje2/zvol02   104M  7.91G       56K  -

It never created a new item.

But it created this:

Code:

root@truenas[/mnt/tank/ix-applications]# zfs list |grep pvc   
tank/ix-applications/releases/dokuwiki2/volumes/pvc-f3b8cb02-d1ff-4b5a-897c-8d5beddcbba1  10.1M  7.81G     10.1M  legacy
tank/ix-applications/releases/sabnzbd/volumes/pvc-3829de01-5ac7-47ce-a34b-d46c3083e95b     120K  7.81G      120K  legacy

Code:

root@truenas[...applications/releases/sabnzbd/volumes]# pwd; ls -R
/mnt/tank/ix-applications/releases/sabnzbd/volumes
.:
ix_volumes

./ix_volumes:

So, a PSV is not a ZVOL nor a "normal" blob file it seems.

Useful page to mount PVC.

That said, after deleting the container it also seems to delete this "pvc" thing. I would assume if we update a container it would persist. (not sure how to test). So i think your (config) data is gone.

Why not a normal dataset? I think they want to avoid ACL/ownership files crap. And i think they want to create a snapshot of the PVC when you are updating a container. Not sure if already implemented though.

 

[MisterE2002]

Snapshots you may think - nope. We aren't meant to snapshot the ix-application dataset, and anyway I do not want to roll back all the apps when I screw up like this. There ought to be a method of backing up the container configuration as text / json / whatever files so that I can recreate the container. I move the app config and other storage away from the container so I can back them up using normal mechanisms, but there appears to be no backup of the container configuration itself, and no sensible way of doing so.

So, i understand the wish to make custom backups/snapshots of the PVC to save yourself from pbkac errors.

 

[NugentS]

Yup. That and ID10T Errors


At this stage I would like to be able to backup the container config, and (just as importantly) be able to restore the container. The app config is dealt with - I can back that up and restore as much as I like (snapshots and replication, file by file is questionable).