Windows VM - CPU 100% by iowait

[PICcie]

Hello,

I'm using TrueNas Scale 21.02. I know that it's an ALPHA version but ZFS and KVM Virtualization are announced as RELEASE.
It run on the folowing configuration:
- i7 2700K 4cores 3.5Ghz
- 16GB RAM DDR3
- 3 disks 7200Rpm ~150MB/s configured as raidz
- 1Gbs ETH link

I have configured only one VM with Windows Server 2019 configured as this:
- 1vCPU 2core
- 4Gb RAM
- 1 zvol for system as AHCI
- 4 zvol for data as AHCI
- 1 nic as intel1000

Windows have the only task to share the 4 zvol with samba.
When I copy data to the sharing, it's copying at around 60MB/s and the CPU is used at around 20% but it's satured to 100% by iowait.
I have try all CPU mode but there is no change.

Is it due to the ALPHA version ? Have I made a mistake ? Did you have a solution ?

Best Regards,
Nicolas C

 

[sretalla]

Is it due to the ALPHA version ?

No, KVM itself is not Alpha

1 nic as intel1000

You might want to try getting the VirtIO driver for Windows and using the VirtIO NIC.

 

[PICcie]

Thank for your quick help,

Yes I have try VirtIO for NIC and also for DISK but with this configuration, Windows was unable to find disk and the ethernet card is not detected.
Maybe it need drivers but whitout ethernet, how to add and install it ?

 

[sretalla]

You can find the drivers here:

WindowsGuestDrivers/Download Drivers - KVM

www.linux-kvm.org

 

[PICcie]

Thank, I will try this.

 

[PICcie]

I have made some test.
I have put the NIC and data disks to VirtIO.
Now I have a copy speed up to 85MB/s and it's a good news.
However, I have again some very high CPU usage... It's less frenquent and under 100% but I think it's too much for only 1VM using 2 core of 8...
You cansee attached 3 pictures of htop displaying iowait in grey.

Is it normal ? caused by ZFS core ? or there is again a mistake ?

 

[ornias]

Did you install the windows guest drivers by now?

 

[PICcie]

Yes I think, I have install : virtio-win-0.1.185.iso from this link :

https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

Are there some other guest drivers needed ?
I'm a newbie in linux virtualization. I was using Hyper V but I need to change to TrueNAS for the ZFS performances.

 

[sretalla]

Are there some other guest drivers needed ?

The virtio drivers are one thing, the QEMU guest agent is another different thing, but the install docs are saying that you get both at the same time, so should be nothing more to do.

Container-native virtualization | Red Hat Product Documentation

Container-native virtualization | Red Hat Documentation

docs.openshift.com

Generally, a newly installed Windows machine will be doing a bunch of work that may or may not represent the performance of the OS in the long term. Things like indexing of the disks which could be stopped if that's not seen as useful for example.

 

[PICcie]

Generally, a newly installed Windows machine will be doing a bunch of work that may or may not represent the performance of the OS in the long term. Things like indexing of the disks which could be stopped if that's not seen as useful for example.

Yes, of course, but the iowait activity is only present when I read or I write data from the windows machine. In Idle state, the CPU usage is normal. The problem can be reproduce at any time, copying data cause this...

 

[sretalla]

Have you changed the driver in Windows for the drives to VirtIO? I think I've seen something about needing to reinstall the OS or at least the disks in order to get that to really change.

Also, there's a reasonable chance that if your 3 ZVOLs are all on the same RAIDZ1 pool of spinning disks, you're far too low on IOPS to run windows well. (you will be getting the IOPS of only one disk even when seeing the separate disks in the guest).

 

[PICcie]

Yes, after verification, the driver was changed an the devices was changed (not the same ID).
Of course, the speed of all disks is limite by the pool. I use multi-disk as virtual partition to manage the space allocated to different usage.
Also, in my test, I write only to one disk.

I don't understand the origin of this high iowait cpu usage. I will continue to evestigate.

 

[sretalla]

I think the best guess is IO bottleneck manifesting in CPU stuck waiting for blocks to arrive.

Do you have an SSD you can try with? that should eliminate the IOPS question.