SMB shares locked out and most apps will not deploy

[cosine5000]

Woke up this morning to my SMB shares offline and most apps offline. As far as I know nothing was changed. SMB permissions all look same as always, all/everyone.

SMB share show up but return permission denied.

Connecting to the shell of most apps that fail to deploy returns "error: unable to upgrade connection: container not found" and Kubernetes events show "Back-off restarting failed container" for most failed apps.

I've checked everything I know of and restarted the server but so far no luck.

Truenas ver: TrueNAS-23.10.1 Cobia

 

[Kris Moore]

Not enough info here to begin to troubleshoot. Is the pool online? Did anything happen software wise? Updates, 3rd party tools, etc etc?

 

[cosine5000]

Thanks for the quick reply. The pool says it's fine and healthy, it's just not accessible through SMB. The SMB shares also report all good. Nothing changed that I am aware of. A few apps had been unreliable/slow to deploy/upgrade for a week or so but nothing major until now.

 

[cosine5000]

Something def seems off with file system permissions, when I check file system ACLs and click "Save access control list" without changing anything I get "[EPERM] Filesystem permissions on path /mnt prevent access for group "builtin_users" to the path /mnt/DATA2/data2. This may be fixed by granting the aforementioned group execute permissions on the path: /mnt."

 

[Kris Moore]

Filesystem permissions don't exactly change themselves... Are you running 3rd party tools, scripts or app catalogs?

 

[cosine5000]

The server runs Plex as an app, I don't know what would count as a 3rd party tool or script but the server is pretty simple.

Is there any procedure to reset permissions or is the data just lost?

 

[Kris Moore]

You had originally said:

Connecting to the shell of most apps that fail to deploy returns "error: unable to upgrade connection: container not found" and Kubernetes events show "Back-off restarting failed container" for most failed apps.

This sounds like more than just a single Plex container... Knowing how or what destroyed your permissions is a very helpful first step in figuring out how to undo the damage done. Custom things could be custom containers, third party App repositories (like TrueCharts), bash scripts, etc etc. If its something we ship with TrueNAS we should be able to investigate and figure out where the issue came from, and how to undo it. That said, yes you will need to figure out what was changed permission-wise. You usually would need to have proper chown permissions setup or other ACLs so your SMB shares work...

 

[cosine5000]

Syncthing can still access the files fine so I can copy them off the box, thanks for the help.