Excessive CPU usage after Cobia upgrade

[sixdas]

Hello everyone, I have updated my system to version 23.10.0 Cobia.
But I am experiencing excessive use of the CPU, I went from 10-20% to 30-70% with peaks even at 90%. Always in the same conditions of use. Can I do some tests to understand if it's a bug or does it depend on some of my configurations?

 

[Kris Moore]

Would be helpful to do a bit more investigation, see what that python3 process, if its TrueNAS middleware or some other process (app?).

 

[sixdas]

As I can verify, if you guide me I would be happy
Thank you

 

[sixdas]

I think I solved it, I did a test by unplugging all apps, I left only truenas on, I noticed that after 20 min the cpu went back to normal speed and the temperatures had stabilized as well. I then activated one app at a time checking the cpu from time to time and got to the app that was having problems it was home assistant , so I went into the logs and noticed it was giving this error:

2023-10-26 19:21:40.196622+02:00[33m2023-10-26 19:21:40.196 WARNING (SyncWorker_0) [custom_components.truenas.truenas_api] TrueNAS 192.168.78.210 unable to fetch data "reporting/get_data" (500)[0m
And it was repeating very fast. I think this was causing the process to get up, because by deleting it seems to me to be stabilized.

I will report the bug to the guy who created the component that displays truenas statistics in home assistant.

In this case it would have been useful to see directly from app summary which one consumed more resources. I still don't understand why there is a screen in the blog that also depicts ram cpu, in the new app scheramta while in the final version it was removed.

 

[FrostyCat]

I haven't upgraded yet so can't do this myself, can you check if k3s has metrics-server enabled? ("cat /lib/systemd/system/k3s.service").
If yes, there may be a way to see which app is using CPU by checking metrics.

 

[sixdas]

Here's what I get when I run the command

 

[FrostyCat]

Looks good.
Can you please in a shell run "k3s kubectl -n kube-system get pods" and look for "metrics-server"?
If you see metrics-server pods, run a "k3s kubectl -n kube-system top pods" and tell me if you see any CPU and memory usage information.
Edit: I upgraded mine and metrics-server is not there, looks like k3s removed it (see edit below).

If you want to install it, just run k3s kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml and then the top command for pods and nodes will work.

To check resource usage for ALL pods just run k3s kubectl top pods -A

To make it easy to use the command line, I generally have some aliases defined, like alias k='k3s kubectl'
---
CPU in Kubernetes is reported as milli-cpus, ie each core is split into 1000 little slices and reported as the number of these slices being used, e.g. 300m means 0.3 CPU cores used. If your machine has 8c/16t, your total CPU "capacity" will be 16000m.
---

Edit: metrics-server is now disabled in this file /etc/rancher/k3s/config.yaml and not in the k3s' command line. Still better to install it externally as it won't get reset on upgrades.

I do not understand why it's disabled by default but there must be reasons.

 

[sixdas]

Now I'll see if I can try your solution, I would like to understand why it was disabled, if installing it affects anything else. I don't know what metrics it shows, but I also found this:

k3s crictl statsp

It tells you the apps CPU and RAM usage.

 

[waqarahmed]

@sixdas can you please confirm if you are using home assistant application from official catalog ?

 

[FrostyCat]

Now I'll see if I can try your solution, I would like to understand why it was disabled, if installing it affects anything else. I don't know what metrics it shows, but I also found this:

k3s crictl statsp

It tells you the apps CPU and RAM usage.

That's interesting, cause CRI-O is not running. Probably k3s fakes it with containerd (crictl is the CLI interface to CRI-O, a different container runtime). There are also lower level tools you could use, from htop to systemd-cgtop for example.


I would assume metrics-server was just not in the scope of the iX implementation and testing. It won't affect anything if you run it.

 

[sixdas]

@sixdas puoi confermare se stai utilizzando l'applicazione dell'assistente domestico dal catalogo ufficiale?

No, I created a custom app using the official repo:ghcr.io/home-assistant/home-assistant

That's interesting, cause CRI-O is not running. Probably k3s fakes it with containerd (crictl is the CLI interface to CRI-O, a different container runtime). There are also lower level tools you could use, from htop to systemd-cgtop


I would assume metrics-server was just not in the scope of the iX implementation and testing. It won't affect anything if you run it.

ok I have to go into more detail thanks

 

[truecharts]

Looks good.
Can you please in a shell run "k3s kubectl -n kube-system get pods" and look for "metrics-server"?
If you see metrics-server pods, run a "k3s kubectl -n kube-system top pods" and tell me if you see any CPU and memory usage information.
Edit: I upgraded mine and metrics-server is not there, looks like k3s removed it (see edit below).

If you want to install it, just run k3s kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml and then the top command for pods and nodes will work.

To check resource usage for ALL pods just run k3s kubectl top pods -A

To make it easy to use the command line, I generally have some aliases defined, like alias k='k3s kubectl'
---
CPU in Kubernetes is reported as milli-cpus, ie each core is split into 1000 little slices and reported as the number of these slices being used, e.g. 300m means 0.3 CPU cores used. If your machine has 8c/16t, your total CPU "capacity" will be 16000m.
---

Edit: metrics-server is now disabled in this file /etc/rancher/k3s/config.yaml and not in the k3s' command line. Still better to install it externally as it won't get reset on upgrades.

I do not understand why it's disabled by default but there must be reasons.

This is a bad idea, it's already integrated in cobia and the correct command to enable it is:

Code:

midclt call -job kubernetes.update '{"metrics_server": true}'