Creating Pools for Performance and Storage using 12Gbs SSDs

[fahadshery]

Calling all Gurus, experts and friendly people on this forum,

I just got 8 x 1.8T 12Gbs Enterprise SSDs for my TrueNas box.
I also have 5 x 800GB 12Gbs SSDs that I am planning to use in proxmox by mounting a 4 x str. mirror pool.

I have been scouring this forum to design my pools in the the best possible way that will give me performance and storage.
My potential applications are:

1. Postgres and MySQL databases
2. Media for Plex
3. Storage for Kubernetes (production and dev)
4. System Backups (both windows and Macs)
5. Hosting Website and blogs
6. Creating VM's and Containers using Proxmox
7. NFS mounts in Proxmox to host VMs or mount them inside a VM
8. iSCSI as a storage drive for Windows VM's to host Steam Library or other games

I already have a secondary TrueNas box with 4 x 2TB spindles in a raidz1 configuration. I was thinking to use it as a backup server.

So how should I lay them?

This is what I was thinking to do:

1. For Performance:
   1. Create a 4 x 800GB stripped mirror for VMs and containers
   2. 3 x 1.8TB SSDs in a Raidz1 configuration to host database and Kubernetes persistent volumes
2. For Storage:
   1. 8 x 1.8TB SSDs in a Raidz1 configuration for media
   2. Use it for NFS mounts in Proxmox
   3. Use it for SMB shares for Apple Time Machine and other Windows backups
   4. Use it for iSCSI at 32/64K block for Windows VM
3. 1 x 800GB spare

I have read somewhere about Fusion pools? What do you think if that is any good for my use cases (listed above?)

Lastly, I have 2 backplanes in a Dell R720 Server. 1 is connected to one HBA (that is passed to the TrueNas VM). The other is local ZFS pool (800GB SSDs + 3 1.8TB SSDs). Should I pass both HBA's to the TrueNas as is? Or should I pass HBA (with 8 disks attached) to TrueNas VM and other HBA to remain in Proxmox?

thank you for reading :)

 

[NugentS]

1. Postgres and MySQL databases
   1. Requires high IOPS
2. Media for Plex
   1. Low IOPS - High sequential
3. Storage for Kubernetes (production and dev)
   1. Mix really. Depends what they are doing and where the storage is.
4. System Backups (both windows and Macs)
   1. Low IOPS
5. Hosting Website and blogs
   1. Low IOPS
6. Creating VM's and Containers using Proxmox
   1. High IOPS
7. NFS mounts in Proxmox to host VMs or mount them inside a VM
   1. High IOPS
8. iSCSI as a storage drive for Windows VM's to host Steam Library or other games
   1. High IOPS

In general terms looks to me like you need two pools. One smaller high IOPS pool = mirrored pairs (at least 2 vdevs) and one bulk storage pool = RAIDZn
Fusion pool - not in your case. You don't have any HDD's to accelerate.
The actual size of the pools depends on how much data.

Always pass HBA's to TN if TN is managing the disks. Also do not attempt to run TN on virtual disks for anything other than testing.

 

[jgreco]

In general, you do not want to run block storage or databases on RAIDZ.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

 

[fahadshery]

In general, you do not want to run block storage or databases on RAIDZ.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Got it! That's why I was thinking to create 3 pools.

1. 2 x Stripped Mirrors (with 4 SSDs) (in Proxmox)
2. 1 x RAIDZ1 with 3 disks (in Proxmox)
3. 1 x RAIDZ1 with 8 disks (attached to HBA and passed on to TrueNas VM)

How do I find out about the ashift value of my SSDs?
These are the 800GB SSDs drives I am using:

Do you think creating a ZFS stripped mirror and RAIDZ1 pool with 3 disks is beneficial within the Proxmox or should I pass the second HBA to TrueNas as well to manage and then mount it using NFS/iSCSI back in the proxmox to stand up VMs and containers? Will this create additional network latency/performance overheard?

Cheers,

 

[fahadshery]

In general, you do not want to run block storage or databases on RAIDZ.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

The question is: How would TrueNas behave if I pass 1 HBA (with all the 8 drives) and passthrough 3 drives as a proxmox passthrough drives to the VM?

 

[fahadshery]

1. Postgres and MySQL databases
   1. Requires high IOPS
2. Media for Plex
   1. Low IOPS - High sequential
3. Storage for Kubernetes (production and dev)
   1. Mix really. Depends what they are doing and where the storage is.
4. System Backups (both windows and Macs)
   1. Low IOPS
5. Hosting Website and blogs
   1. Low IOPS
6. Creating VM's and Containers using Proxmox
   1. High IOPS
7. NFS mounts in Proxmox to host VMs or mount them inside a VM
   1. High IOPS
8. iSCSI as a storage drive for Windows VM's to host Steam Library or other games
   1. High IOPS

Based on this info, I have designed my pools as shown in the picture below.



Thanks for your help :)