ZFS over ISCSI for Proxmox

[arrow]

is this possible at all? Somebody from proxmox says TrueNas does not support ZFS over ISCSI

 

[jgreco]

Well, "somebody from proxmox" is an idiot, then. TrueNAS doesn't give a crap what you're running over iSCSI. It's a block SAN protocol. As long as you can get the client to attach the iSCSI disks, you can put whatever you want on them, and it can certainly be ZFS if you want.

 

[arrow]

I see a couple of nice articles ...... but the links does not work of course step by step how to set that up from proxmox side and their documentation is also crap, better that it was but still ......

 

[jgreco]

No idea, sorry. Lots of people use FreeNAS and TrueNAS with iSCSI as a target from VMware ESXi hypervisors, Windows 10 iSCSI, etc., so iSCSI definitely works just fine. Proxmox is very new to the virtualization game though.

 

[ChrisRJ]

To rephrase: ZFS is the only file system that TrueNAS supports. iSCSI is one of several access methods, like SMB, AFS, and NFS. Whoever made the statement about "not supported" not only does not know anything about TrueNAS, but also has a very limited knowledge (put mildly) about storage in general.

 

[danb35]

Well, it's possible that, as our grinch says, "somebody from Proxmox" is an idiot. It's also possible that you posed your question there no better than you did here (and as a result they had no idea what you really wanted to do, and took a guess), or that they answered entirely correctly, and you just didn't understand their answer.

What exactly is it you're wanting to do? Five words don't constitute a question that can, normally, be helpfully addressed.

 

[arrow]

Regarding the topic:
1. TrueNas (latest stable version) ISCSI works fine in Proxmox 7.0
2. TrueNAs ZFS over ISCSI throws error:

command '/usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/172.21.100.100_id_rsa root@172.21.100.100 zfs list -o name,volsize,origin,type,refquota -t volume,filesystem -Hrp' failed: exit code 255 (500)

not sure if I choose wrong ISCSI provider: comstar

I want to believe I am doing something wrong here.

 

[Samuel Tai]

After you mounted the iSCSI volume, did you create a ZFS pool in ProxMox? At this point, I think you just have a bare volume that's not been formatted yet.

 

[rvassar]

I'll second @Samuel Tai here... iSCSI provides raw storage blocks. TrueNAS is storing those raw blocks on a ZFS pool, but the blocks provided are intended to appear as a raw block device. You need to create a ZFS pool or other filesystem on that iSCSI share for Proxmox to make use of it. And honestly, I don't see much point in implementing a ZFS pool on an iSCSI share hosted on an existing ZFS pool. You won't gain any additional compression, and you'd be implementing a bunch of overhead by layering snapshots, etc... Imagine trying to troubleshoot a disk full condition with ZFS snapshots on the TrueNAS pool, and on the client's pool. Let alone the wasted space required to keep both pools below the magic 80%.

 

[ebricca]

well this function seems to be a bit different and not so well understood in the thread - there is an article about it

ZFS over iSCSI for Proxmox and FreeNAS - GrandWazoo

FreeNAS ZFS over iSCSI for Proxmox Go to https://github.com/TheGrandWazoo/freenas-proxmox donate a little to GrandWazoo for going god's work.

marcelliot.net

what this function does
- it seems to be automated creation on truenas of a zvol & iscsi share from the proxmox side - that is afterwards used as image
to allow proxmox to create and control zvols&iscsi shares on truenas might be convenient but I would not allow this from a security perspective in production - unless i misunderstood the security implication

also visible by an ssh command - sent to the example nas provider (example from above)
"command '/usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/172.21.100.100_id_rsa root@172.21.100.100 zfs list -o name,volsize,origin,type,refquota -t volume,filesystem -Hrp' failed: exit code 255 (500)"
- this is a root ssh login on the nas via key - if someone gains this info with bad intentions goodbye nas
(or if the scripts just have an issue ex. after an update)

ps:
what i have been using until today is a pure truenas sparse iscsi mount with an ext4 filesystem that hosts the image files directly via ubuntu/kvm
(i am thinking about using proxmox - possibly with the gui config iscsi & lvm but am unsure - or a non gui config to again have the same ext4 mount over scsi and acess it as storage directory)

 

[jgreco]

well this function seems to be a bit different and not so well understood in the thread

Ah, so this isn't "ZFS over iSCSI" but rather "feature misnamed by idiot." We understand perfectly well what "ZFS over iSCSI" means. The purpose of words is to convey meaning between people, so if you say "ZFS over iSCSI", then we naturally assume you're running ZFS over iSCSI, not talking about some in-depth hypervisor integration that appears to be some sort of plug-in for Proxmox. This looks more like "Remote ZFS disk management for Proxmox"

TrueNAS doesn't give a crap what you're running over iSCSI.

So I take back what I said, I have no idea if this works, but if it doesn't, it's because nobody at Proxmox has put in the effort. Which is hardly shocking as they have their own on-host ZFS implementation, and probably very little incentive. I suspect that's why it looks like it is an add-on from a third party.

There's basically no reason that I see that this couldn't be made to work. The idea is similar to numerous other integrations in the VMware world. From a brief glance, it looks to me like @arrow should have used "freenas" as the iSCSI provider instead of "comstar".

 

[ebricca]

well yes it seems to be a plugin functionality for proxmox written in perl
one has to add a third-party version to proxmox as it is not included - which also is a bit so so

GitHub - TheGrandWazoo/freenas-proxmox: ZFS over iSCSI to FreeNAS API's from Proxmox VE

ZFS over iSCSI to FreeNAS API's from Proxmox VE. Contribute to TheGrandWazoo/freenas-proxmox development by creating an account on GitHub.

github.com

some hosting providers seem to use it ... so it might be more than demo level

ProxMox automatic periodic snapshots with help of cv4pve. Feat ZFS, FreeNAS/TrueNAS, ZFS-over-iSCSI.

Time codes:0:00 - Intro4:30 - CV4PVE configuration and showcaseGitHub link to the snapshoting software itself:https://github.com/Corsinvest/cv4pve-autosnap/r...

www.youtube.com

btw .. the third-party plugin does not yet exist for proxmox version 7 (just 5 and 6)
to use root to execute automated zfs commands via ssh wouldn't be my first choice
(rewritten with the truenas/truecommand api / proper right management i could see some benefit)