Lost Kubernetes (latest TrueNas version)

[Louis2]

I have seen issues reported like this one before ..... but it is ^not ok^ .... to put it in a mild way.

After restarting my system:
- I noticed on the console lots of messages like: [1343.452098] IP{VS: rr: UDP 172.17.0.10.53 - no destination available
- and all my apps where gone

I noticed in previous threats that people suggested to unset and set the Kubernetes pool an option in apps which does not seem available in apps any more. So I tried to move Kubernetes to an other pool which did not work (seems to have destroyed the old pool as well).

Since I was just doing some testing not dramatic. Never the less .... very serious IMHO. I really doubt if I am going to use this, given the what seems to be lacking stability.

It could have been related to a change I made in the NAS its default gateway, however ...... that should not have had any influence because:
- Kubernetes was setup to use another vlan (and should not use the NAS-vlan)
- and even if .... there is still a gateway

EDIT: I did also restart the system, perhaps a more likely cause of the problems ..
But be aware I do not know if there is any relation between my action and this problem at all !!!!

 

[MisterE2002]

I think we should not be overly dramatic. If we loose our data we should panic. SCALE containers are still not really mature but i found no clues about the overall reliability and data integrity.
I either case a lot of possibilities are possible: hardware issues, your actions and indeed software bugs.

We are getting a "free product" (for a cost). You can help the community by trying to reproduce the issue and file a detailed bug. For example. Create a virtual machine and do the same steps until you get the same (wrong) output.

I assume you mean well but it feels like a flame right now.

 

[Louis2]

I understand your reaction. However I was/am really not amused. It has to do with my expectation, that this is a stable product, intended for professional use.

Even for my home setup, I would not be happy, in case I am had a couple of apps running and lost them including related data!! Like what happened today, just after a few first test setups.
The very least you need is such a situation, is a guaranteed working recovery procedure (I did delete the ix-container, and reloaded a yesterday backup)

As far as I know my hardware is rock-stable, the only real clue I can imagine, is the restart (from the gui)...
Of course I do not know if that was the reason. It could have been something completely different of course!
(The system should withstand restarts, reboots or power outages and I assume that is tested)

 

[MisterE2002]

I understand. First off, are you really sure the data is really gone? It is data in PVC's? it is stored in the ix folder, so maybe you can (manually) recover data still.

My feelings are mixed about deleting PVC's when deleting apps (without clear warnings). Personally i only use HostPath sharing. I want to "see" my data. In either case, test by rebuilding a app before submitting to valuable data.

SCALE is marked as stable, however the community is recommending CORE.

 

[NugentS]

It happens from time to time - and I believe that IX are trying to figure out why. I have found that a 2nd reboot almost always clears the problem and the containers come back.
On the subject of PVC's and Hostpaths for config files (at least). I personally only use PVC's for a config less container - netdata or file browser being good examples. They have no data in config that I care about. Everything else uses hostpath for a config file which I can then replicate as a backup elsewhere and (more importantly) when I fubar a container I can restore an individual container.

Truetool / Heavy_Script is useful - but I am not going to rely on it for a backup until you can restore an individual container properly.

 

[Louis2]

Perhaps I could have recoverd some data, however:
- I have little knowledge related to Kubernetes
- it was just data relates to some test I did. So at this moment not really relevant data
And if the data had been relevant, I would really prefer to roll back a snapshot. Which is probably not that easy, since data is probably stored at multiple locations.

Perhaps due to my lack of experience, it feels that recovering a VM is much easier. Certainly if not one but multiple / all apps are affected

 

[NugentS]

Well in any container there is:
1. A config folder - I dislike keeping these on a PVC as stated above. Use hostpath or I suppose NFS (haven't tried that yet)
2. Data folders - also kept away from PVC's and on NFS Shares or hostpath. I find NFS to be more reliable, less permissions issues
3. Images and other internal K3S stuff - about which you can do nothing - so don't worry about them

The one thing you can't really backup is the container configuration (as opposed to the app config inside the container) , ie what image is being used, configration of data paths, VPN and other stuff (when first installing a container). Everything else you need is 1 & 2 from above