Truenas Scale GUI Unreachable

[js4616]

I will go ahead and apologize and let you know I am a complete Truenas newbie until about 2 months ago. I have fumbled my way through setup several months ago and now have an issue I can't seem to narrow down. I have 4 apps running, plex, radarr, ombi, sab, I have previously run this on windows and Ubuntu and Unraid and really enjoy zfs performance on Scale and do not want to go back.

The issue seems to be after several hours the GUI becomes unreachable, but I can reach all of the docker containers just fine. I can't seem to find where to pull a log file on the system in searching but maybe i missed something in my search. All my search results show guides for core instead.

This issue did not exist on Anglefish and only started on Bluefin. I had to fix a bunch of issues with the containers around pvc storage but have since resolved and all containers are behaving properly that I can tell.

The only way to get back to the system is a hard shut down, I am assuming ssh is working since the containers are but haven't attempted as I don't know where to go once I am ssh'ed in. If someone can guide me a troubleshooting direction I will pull the necessary log files and upload.

 

[js4616]

actually i fumbled my way through command line and found log files. the last one written before hard reset was as follows.
loop0: detected capacity change from 0 to 2794432

 

[ApexProxy]

I'm wondering if this is a networking issue. I have been dealing with some alias and DHCP issues the last few days and I am wondering if you are having similar issues. My resolution so far has been to set an alias and forgo DHCP entirely and things have been much more consistent.

If you are using DHCP, any chance you have the ability to connect to the server with a monitor and keyboard so you can see the console and check what your network settings are when this connectivity issue happens?

 

[js4616]

I'm wondering if this is a networking issue. I have been dealing with some alias and DHCP issues the last few days and I am wondering if you are having similar issues. My resolution so far has been to set an alias and forgo DHCP entirely and things have been much more consistent.

If you are using DHCP, any chance you have the ability to connect to the server with a monitor and keyboard so you can see the console and check what your network settings are when this connectivity issue happens?

I am not able as this is in a server closet in my home and I don't really have a second monitor. My desktop one it mounted on an arm and I have to sit the treunas box on my desk to do anything I can't via the gui so I will have to unplug and move. I did go ahead and disable dhcp and set a static. Will see if I have this happen in the next 24 hrs, I have had it basically every day for a week or so.

 

[ApexProxy]

It looks to me like using DHCP is not all that reliable when the address being constant is essential. Though from what I now its pretty much always recommended to set static IPs on servers. I also accidently caused some issues by using two interfaces on the same subnet which caused traffic to take extremely unpredictable paths.

 

[js4616]

It looks to me like using DHCP is not all that reliable when the address being constant is essential. Though from what I now its pretty much always recommended to set static IPs on servers. I also accidently caused some issues by using two interfaces on the same subnet which caused traffic to take extremely unpredictable paths.

This isn't the issue I don't believe. I set it to static and this morning I check it again and its unreachable and my apps are. I am in the process of trying to work my way through ssh and view log files to see what I can find.

 

[js4616]

I am still dealing with this and not sure how to troubleshoot at this point. this is purely an issue reaching the GUI as all the Apps connect. I just cannot reach login page

 

[Whattteva]

You could probably figure out the IP of the server from the DHCP lease list on your router.

 

[js4616]

You could probably figure out the IP of the server from the DHCP lease list on your router.

That isn't the issue I know the IP, I can reach the dockers from the same IP with the correct port, but the GUI isnt reachable.

 

[Whattteva]

Oh in that case, that isn't DHCP problem then. Somehow the web UI isn't binding to port 80 or 443. Have you tried rebooting it?

 

[js4616]

Oh in that case, that isn't DHCP problem then. Somehow the web UI isn't binding to port 80 or 443. Have you tried rebooting it?

Yes, I covered this in the original post.

On a hard reboot it becomes reachable for a period of time, at some point it is no longer reachable. The dockers remain reachable its just the GUI that fails to load. I have tried multiple browsers to rule that out.

 

[browntiger]

I would as soon as gui is reachable enable ssh and look into /var/log/messages. If fails again ssh into the host see if you can pull the error out. I am also into some sort of networking /configuration error camp. If all fails reinstall truenas and reimport your zfs pools.

 

[FabrizioR8]

actually i fumbled my way through command line and found log files. the last one written before hard reset was as follows.
loop0: detected capacity change from 0 to 2794432

Why do you have a loop device set up? That's not part of the baseline install/config for Scale, at least not on my bluefin system, so please explain and maybe we can figure out what's going on. Whatever file your loop0 device is using suddenly changed from empty (0 bytes) to 2.66 MB.

while you can ssh in, maybe see what 'losetup -l -a -v' has to say?

 

[js4616]

Why do you have a loop device set up? That's not part of the baseline install/config for Scale, at least not on my bluefin system, so please explain and maybe we can figure out what's going on. Whatever file your loop0 device is using suddenly changed from empty (0 bytes) to 2.66 MB.

while you can ssh in, maybe see what 'losetup -l -a -v' has to say?

Honestly I don't even know what a loop device is... and now for some reason this morning I can't even ssh in but I can still reach the containers. I don't understand whats happening. Would it be helpful to download a specific log file and post here?

 

[FabrizioR8]

No idea... had to look-up what a loop device was myself. never needed one.

 

[Zain]

I'm seeing the same thing in my logs too? What is it?

Code:

Jan 12 12:57:06 truenas kernel: loop: module loaded
Jan 12 12:57:06 truenas kernel: loop0: detected capacity change from 0 to 2794432
Jan 12 12:57:06 truenas kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher
Jan 12 12:57:06 truenas kernel: loop0: detected capacity change from 0 to 2794432

 

[js4616]

I'm seeing the same thing in my logs too? What is it?

Code:

Jan 12 12:57:06 truenas kernel: loop: module loaded
Jan 12 12:57:06 truenas kernel: loop0: detected capacity change from 0 to 2794432
Jan 12 12:57:06 truenas kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher
Jan 12 12:57:06 truenas kernel: loop0: detected capacity change from 0 to 2794432

I dont know but my porblem is still unresolved. I don't know what else to do to troubleshoot.

 

[FabrizioR8]

anything noteworthy in the /var/log/* log files?
if k3s cluster and containers are okay, maybe something in the middlewared logs?

 

[FabrizioR8]

when you say “docker containers”… do you mean apps installed via the GUI to the kubernetes cluster, individual docker containers set up via the UI, or something done custom via the command line?

 

[js4616]

when you say “docker containers”… do you mean apps installed via the GUI to the kubernetes cluster, individual docker containers set up via the UI, or something done custom via the command line?

Well ssh is enabled and I can reach it when I first boot but I lose access to it as well once I lose access to the GUI. Yes I mean apps installed via GUI to the cluster. However the apps are still accessible