Split vdev, break pool, best performance

[aseiras]

Hi, sorry if this has answered in another way, I have searched but could not find the 'right' answer.

When I first created my NAS, I made 20 drives into one vdev / pool, with 4 hot spares.

Performance is really really bad, to the point that VMs from the virtualization system (not in TrueNas) are un-usable.

Talking to a person, pointed out that I would have better performance with several smaller vdev instead of one large one.

My current setup is 24 SSD 1TB, 20 + 4 hotspares.

It seems , ideally I should setup smaller vdevs, like 5+1 *4 , or any other configuration that would make smaller vdevs ..

Point is that I am totally lots now on hos to accomplish this, with out having to reformat the whole NAS, if that is even possible.

Please, any help / pointers would be much appreciated.

Alternatively, how can I test / improve performance on the vms?

 

[danb35]

Point is that I am totally lots now on hos to accomplish this, with out having to reformat the whole NAS, if that is even possible.

It is not possible.

 

[aseiras]

It is not possible.

Thanks , sad ..

So, I have to start from scratch?

if so, what geometry would give me the best performance (for VMs) I do not mind sacrificing space / hard drives in exchange for performance.

 

[Davvo]

Please read the following resources:

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.truenas.com

 

[NugentS]

Please read the forum rules about hardware and post your setup.
Also, whats your use case?

 

[NugentS]

The best geometry for VM's is mirrors

 

[Whattteva]

if so, what geometry would give me the best performance (for VMs) I do not mind sacrificing space / hard drives in exchange for performance.

Best is always striped mirrors. Here's what one of the core developers of ZFS has to say about it.

“For best performance on random IOPS, use a small number of disks in each RAID-Z group. E.g, 3-wide RAIDZ1, 6-wide RAIDZ2, or 9-wide RAIDZ3 (all of which use ⅓ of total storage for parity, in the ideal case of using large blocks). This is because RAID-Z spreads each logical block across all the devices (similar to RAID-3, in contrast with RAID-4/5/6). For even better performance, consider using mirroring.“

Pay attention to that last sentence.

 

[aseiras]

Please read the forum rules about hardware and post your setup.

Care to elaborate? what SETUP? isn't that exactly what I am asking and explaining in my first post?

 

[Davvo]

Care to elaborate? what SETUP? isn't that exactly what I am asking and explaining in my first post?

As per the forum's rules.

It's important to remember that hardware information is extremely important when diagnosing problems so please make sure to include this information if you need help troubleshooting a problem. This includes:

• Motherboard make and model
• CPU make and model
• RAM quantity
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
• Hard disk controllers
• Network cards

 

[aseiras]

As per the forum's rules.

I have not yet learned how to make that what you have "Show: truenas ... :
So, I made it part of my signature.

Here it is also.
Dell PowerEdge R730xd 24-Port Flex-Bay Chassis
2 x Intel Xeon E5-2660 v3
128GB Memory - PC4-1700R
Dell HBA330 12Gbps Non-RAID Controller
Dell Dual Internal SD Card Reader w/ two 16GB SD Cards - Boot drive
24 SSD 1tb, several brands models to minimize impact of failure by design. All dedicated to ONE Zpool, One vdev.. 20 active + 4 hotspares
(2) Dell 13G 14G 1100 Watt PSU
DELL BROADCOM 57810S DUAL PORT DA/SFP+ 10GB

As said, this is to store VMs drives.

Thanks

 

[Whattteva]

As per the forum's rules.

I'm assuming his secs are in his signature though it lacks details like RAID configuration etc.

 

[aseiras]

I'm assuming his secs are in his signature though it lacks details like RAID configuration etc.

No hardware raid, all ZFS, can not remember which one I chose, but I did choose the one with the most parity (loosing the most space in exchange for redundancy, I thought) .

 

[aseiras]

Not sure if this might help .. I have no idea how to get the RAID (zfs) info at this point..

 

[aseiras]

Found it, RAIDZ3

20 drives and 4 hot spares.

 

[jgreco]

For iscsi? That's going to have terrible performance and wasted space.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

 

[Davvo]

Thank you for giving your hardware specs.
Really, please read the resources we all linked you: most of your questions can be resolved by reading them, and it's way better to get a solid linear knowledge than some bits here and there from our answers.

 

[aseiras]

For iscsi? That's going to have terrible performance and wasted space.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

It is not on ISCSI, that was the name given to the pool, iscsi is not even enabled, it is all shared via NFS.

 

[aseiras]

Thank you for giving your hardware specs.
Really, please read the resources we all linked you: most of your questions can be resolved by reading them, and it's way better to get a solid linear knowledge than some bits here and there from our answers.

I would agree with you if this was something I was going to actually do over and over, in this case, I just want this ONE NAS to get to the perfomance it once had.

It seems I already got the answer I wanted.
1.- I can not fix it without reformatting (starting over) which will take a long time planning and preparing.
2.- Mirrors is the best option for performance, (this is the only thing I still do not know how to, but I am setting up a lab in proxmox to learn)
and the most important thing I learned in ths forum..
3.- People here do not want to help, only brag .

Thanks all.

 

[Davvo]

and the most important thing I learned in ths forum..
3.- People here do not want to help, only brag.

I honestly see no brag here, just people trying to point you in the right direction.
Also, I totally disagree with your statement: I have seen on my skin that this is a great community that really helps you to grow your understanding of TrueNAS and ZFS; I'm sorry you think otherwise.

 

[jgreco]

It is not on ISCSI, that was the name given to the pool, iscsi is not even enabled, it is all shared via NFS.

Block storage isn't that much different on NFS. RAIDZ3 is still a terrible fit. The linked article does a deep dive explaining why.