FreeNAS disc setup on Vmware esxi

[Supergrobi]

Hi everybody! I am thinking about setting up a Freenas VM on a Esxi server since I am not satisfied with the virtualization features of Freenas. I understand that this is not a ideal setup regarding ZFS. I have read the article

https://www.ixsystems.com/blog/yes-you-can-virtualize-freenas/

I am not sure if I fully understood the recomendations. I have a i3 8100 cpu with 64gb memory, 1 512gb ssd, 2 6TB and 2 3TB disks. What do you think would be the safest storage setup? I do not want to use a separate HBA for PCI passthrough.

 

[jgreco]

Hi everybody! I am thinking about setting up a Freenas VM on a Esxi server since I am not satisfied with the virtualization features of Freenas. I understand that this is not a ideal setup regarding ZFS. I have read the article

https://www.ixsystems.com/blog/yes-you-can-virtualize-freenas/

I am not sure if I fully understood the recomendations. I have a i3 8100 cpu with 64gb memory, 1 512gb ssd, 2 6TB and 2 3TB disks. What do you think would be the safest storage setup? I do not want to use a separate HBA for PCI passthrough.

The issue here is that there's a significant risk of data loss if you don't do this correctly. We had many examples in the old days of people who did not want to do PCI passthrough, came up with a hacky way to get what they wanted done, and then ran into trouble when things went sideways, and then could not determine how to recover their system.

To that end, the iX post is misleadingly positive. It was allegedly written because iX felt my series of articles were too negative and made it seem "not possible."

So, with the caveat that IF YOU VALUE YOUR DATA, and IF YOU WANT SOMETHING THAT'LL STILL WORK NEXT YEAR, you will want to look at

https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

https://www.ixsystems.com/community...nas-in-production-as-a-virtual-machine.12484/

Under no circumstances should you try tricks with RDM on SATA disks, it was never supported by VMware and is now deprecated. People who are virtualization professionals, work with RDM on a daily basis, and who understand how to recover a failure are free to ignore this warning at their own peril.

So you do not need to use an HBA, but in that case, you should do PCI passthrough of the host's SATA storage controller. This gives FreeNAS full control over the drives and is absolutely fine as long as it works. Just as with passing through an HBA, you should do extensive testing to validate before trusting data to it. Usually if it works for a month it'll keep working indefinitely.

 

[Supergrobi]

Thanks for your fast reply! This is really helpful. I was not aware that you could passthrough the host's SATA controller. I assumed that this would not be possible since the recommendation was to get a separate HBA. I think I have to do some reading now ...

 

[jgreco]

The ideal situation for ESXi, at least in my opinion, is to use RAID1 for boot and VM datastores. This makes it attractive to buy a real RAID controller, we usually use the 9271CV-8i here, for the hypervisor itself. The mainboard chipset SATA is relatively useless for ESXi because it lacks redundancy, which means your whole platform fails if an ESXi boot device places on a mainboard SATA fails. So using that for FreeNAS instead is a great way to repurpose it to a more appropriate task.

 

[Supergrobi]

It seems that my system does not allow to passthrough the SATA Controller :-( ESXi only shows the passthrough option for my NVM Controller for my SSD (this is where i have installed ESXi). So maybe I have to get a real RAID Controller. Another question: If I manage to activate the passthrough option an use the direct access with a FreeNAS VM then it would be possible to put my zfs data disc in some other FreeNAS server and mount the ZFS pools there (in case my original hardware is damaged)?

 

[jgreco]

What kind of board is it? ESXi does not automatically allow passthrough in all cases. That doesn't make it impossible. The Wellsburg AHCI is particularly well-known for not being listed but works fine.

Code:

# Fix Wellsburg PCI passthru of SATA controller

% lspci | grep "Wellsburg AHCI"
0000:00:11.4 Mass storage controller: Intel Corporation Wellsburg AHCI Controller [vmhba0]
0000:00:1f.2 Mass storage controller: Intel Corporation Wellsburg AHCI Controller [vmhba1]

Now look up those prefixes in "lspci -n"

% lspci -n | egrep '0000:00:11.4|0000:00:1f.2'
0000:00:11.4 Class 0106: 8086:8d62 [vmhba0]
0000:00:1f.2 Class 0106: 8086:8d02 [vmhba1]

% vi /etc/vmware/passthru.map
add
# Intel Wellsburg AHCI/SATA
8086  8d62  d3d0     false
8086  8d02  d3d0     false

Using passthrough allows recoverability which is a key aspect of virtualizing without being stupid about it. I probably also meant to link to

https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

in my earlier reply because this explains some of that.

 

[Supergrobi]

ok, i made a big step forward. My board is a Fujitsu D3644-B with Intel C246 chipset:

Code:

lspci
0000:00:00.0 Bridge:  
0000:00:02.0 Display controller:  
0000:00:12.0 Signal processing controller:  
0000:00:14.0 Serial bus controller:  
0000:00:14.2 Memory controller:  
0000:00:16.0 Communication controller:  
0000:00:17.0 Mass storage controller: Intel Corporation Cannon Lake PCH-H AHCI Controller [vmhba0]
0000:00:1b.0 Bridge:   [PCIe RP[0000:00:1b.0]]
0000:00:1b.4 Bridge:   [PCIe RP[0000:00:1b.4]]
....

[root@vmserver:~] lspci -n|grep 0000:00:17.0
0000:00:17.0 Class 0106: 8086:a352 [vmhba0]
[root@vmserver:~]

so I added

8086 a352 d3d0 false

to my passthru.map. After activating passthrough and rebooting it seems to work:

Now I will have to find out how I can use this device in my FreeNAS VM.

Thanks again for your great help!