Zfs pool and recommendations and immutable.

[wesleyh]

What is the best recommended pool.
I have 2 things i want to do.
First is have a storage for ISCSI VM's
This system has 7 16TB drives and 4 1.9TB nvme
I want to know the recommended pool for that

Second is i want to create an immutable s3 storage config same hardware setup
Veeam will use S3 now so trying to find a guide for that.
Any recommendations?

 

[artlessknave]

you have none of your hardware listed, so it's very difficult to say.
mirrors will be the best performance.

s3 and veeam are not truenas; it sounds like that part of your post should be in forums for those applications?

 

[wesleyh]

The 16tb drives are on an 3008-it card for a zfs pool what I am asking is with 7 16 tb drives what is the best pool setup recommendation?

There is no good guides for it. System has bual 5620 CPU and 192gb ram

As far as the nvme there is 4 1.92tb.

For this setup I want to have iscsi luns and run vm's from VMware.


For the second box I have same hardware but need to create immutable storage and veeam recommends now an s3 storage but how do you setup truenas for immutable s3?

 

[artlessknave]

are you putting the VMs on the SSDs? then it wont really matter to much. raidz1 is probably fine, or just stripe them if they arent that important and you have backups. back them up to the spinners.

I would probably choose at least raidz2 for the 7 spinners, maybe raidz3, unless you need speed badly, then mirrors (would need another drive)
also note that zfs is nigh immutable as long as you protect the root account and have good snapshots.

note that raidz is NOT a backup. if you choose no backups, then 100% raidz3. those are chonky disks.

 

[jgreco]

There is no good guides for it. System has bual 5620 CPU and 192gb ram

Hogwash.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

raidz1

is not good for iSCSI, even with SSD's.

 

[artlessknave]

is not good for iSCSI, even with SSD's.

ah. that recommendation was when putting the VM's directly on the SSD. there was...relatively little to work with and I missed that they wanted to ISCSI the VM's, which I'm...not really sure why.

 

[samarium]

IIRC minio app from IX is S3 provider, I have installed it but have not as yet tried to use it. Possibly other apps too I guess.

 

[blanchet]

For on-premise veeam repository, the best setup is a Linux Hardened XFS Repository (it is better than ReFS) to benefit fast cloning and immutability.

In my case, I run a VM (2 vcpus, 8 GB RAM, 50TB zvol vdisk) with Linux Debian 11 directly on TrueNAS Core and I have followed the official Veeam guide to setup a hardened Veeam XFS repository.

It is very straightforward.

 

[wesleyh]

Hogwash.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

is not good for iSCSI, even with SSD's

Hogwash.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

is not good for iSCSI, even with SSD's.

What do you mean it is not good with iscsi? Than what should it use with VMware?

I am trying to setup a shared iscsi for VMware if there is something better what is best?

Also in the guide I don't really understand what we should select for the drives, 16tb drives SAS 12gb. One thing says don't use raidz pool than what should you use?

With the SSD drives I can either use as cache or a high speed SSD raid for databases

 

[wesleyh]

For on-premise veeam repository, the best setup is a Linux Hardened XFS Repository (it is better than ReFS) to benefit fast cloning and immutability.

In my case, I run a VM (2 vcpus, 8 GB RAM, 50TB zvol vdisk) with Linux Debian 11 directly on TrueNAS Core and I have followed the official Veeam guide to setup a hardened Veeam XFS repository.

It is very straightforward.

Did you follow another guide for this? I was hoping to use truenas directly for immutable and veeam now requires an s3 for immutable.

 

[blanchet]

I have not realized yet that you want to use TrueNAS Scale instead of TrueNAS Core for VMware and Veeam.

I have tried both, and I can say that actually TrueNAS Core still works better than TrueNAS Scale for storage-only application, especially for iSCSI.
It will probably change in the future, but in 2023, the best choice is still TrueNAS Core.

My personal opinion is : use TrueNAS Scale only if you really need GlusterFS or KVM, otherwise stay with Core.


Let's finish with other answers

S3 immutability
I have never tried myself, but few years ago, Veeam and S3 immutability on TrueNAS did not work

Veeam TrueNas and S3 bucket with immutability

I am hoping to setup a truenas box and use the built in Minio server to create S3 buckets with immutability to take advantage of Veeam's guaranteed ransomware recovery. I have been able to create an locked S3 bucket using SSH commands but veeam also requires versioning on the bucket. My...

www.truenas.com

For VMware

• If you want the maximum of IOPS: use a stripe of mirror with your 4 x NVMe disks (2x2), you will get about 3.84 TB of usable space
• If you lack of space you can also try a raidz1 with your 4 x NVMe disks (3+1),, you will get about 5.76 TB of usable space
• NFS datastores are really easier to manage than iSCSI datastores. So try NFSv3 first and switch to iSCSI only if you really need more performance.
• In my case, with a flash pool for VMware with 12 x Intel SSD SATA D3-S4610 configured with a stripe of mirrors, I have seen no manifest difference of performance between NFS and iSCSI datastores.

For Veeam

• Setup a raidz2 with your 7 x 16 TB disk (5+2)
  • you will get the maximum of usable space
  • Veeam do not need a lot of IOPS (except if you use Instant Recovery or SureBackup)

To create a VM with a serial console, just follow one of my guides

• https://www.truenas.com/community/t...an-10-in-bhyve-with-the-serial-console.85424/
• https://www.truenas.com/community/t...20-04-in-bhyve-with-the-serial-console.91636/

 

[jgreco]

What do you mean it is not good with iscsi? Than what should it use with VMware?

I am trying to setup a shared iscsi for VMware if there is something better what is best?

Did you read the article I pointed you at? Especially point two and its additional reading? RAIDZ is not good with iSCSI.

 

[jgreco]

I have tried both, and I can say that actually TrueNAS Core still works better than TrueNAS Scale for storage-only application, especially for iSCSI.
It will probably change in the future, but in 2023, the best choice is still TrueNAS Core.

What a curious thing to say. iXsystems had a custom FreeBSD iSCSI kernel subsystem written just for FreeNAS. I'm not aware of anything anywhere near as performant for serving iSCSI on Linux, and iXsystems has repeatedly recommended continuing to use CORE for iSCSI. What exactly causes you to believe that this is going to change in the future?

 

[blanchet]

Despite I really like FreeBSD, I have to admit that Linux is evolving faster and receive more innovation than FreeBSD.

I have also seen that

• In the past, basically all storage operating systems were based on FreeBSD: FreeNAS, but also Netapp ONTAP, Isilon OneFS
• But for few years, quite all the new storage operating systems are based on Linux: TrueNAS Scale, PureStorage, Dell PowerStore, Qumulo, Nutanix, VAST, etc. to benefit Linux innovation. They just put their storage stack in the userspace in docker containers to overcome the GPLv2 license

In 7 years, I can imagine that iSCSI will be replaced by NME-over-TCP and TrueNAS Enterprise will be based on TrueNAS Scale instead of TrueNAS Core.

 

[jgreco]

Despite I really like FreeBSD, I have to admit that Linux is evolving faster and receive more innovation than FreeBSD.

People have been saying that for twenty years. Yawn.

 

[wesleyh]

Did you read the article I pointed you at? Especially point two and its additional reading? RAIDZ is not good with iSCSI.

Yes but what is the recommended for iscsi than? Everything uses iscsi to vmware

 

[danb35]

what is the recommended for iscsi than?

What about that article is unclear? Because it really sounds like it directly answers your question.

s3 and veeam are not truenas;

IIRC minio app from IX is S3 provider,

There's also a built-in S3 service, which also uses Minio. I have no idea why iX built it in and also offers it as an add-on.

 

[NickF]

I don't know why S3 buckets (or object storage where it isn't really necessary) are so in vogue. Just because you can do something, doesn't mean you always should. You know what is also immutable?
A read-only pool in ZFS. You know how you can make sure it stays immutable? You back it up to an LTO tape and lock it up.

Dump your DR backups to a pool, change it to read only, and back it up to tape. Wanna send it to the "cloud" instead? https://www.truenas.com/docs/core/coretutorials/tasks/creatingcloudsynctasks/
Cloud Sync Tasks does the work for you. You don't need to worry about the fact that it is S3.

As a millennial myself, it will never cease to amaze me how much my generation tries to reinvent the wheel.