Code_Is_Law
Cadet
- Joined
- Jun 1, 2022
- Messages
- 7
Hello everyone,
First and foremost, thanks to everyone who is taking their time to assist me with some questions I have, appreciated.
So, let's jump right into it.
I have been experimenting with TrueNas 12.8 for a while now with some large scale systems (250-640TiB), and I believe that there are multiple things that I could set-up a lot better. The workload that I experience on these devices is the following:
My main concern is not being able to do the check mechanism due to load elsewhere, the disks should be prioritised to do these heavy random-IO tasks always. Is there any way to set this up properly? My other question is working with another layer of SSD's inside the NAS, for cache or ZIL or anything else that would benefit this setup. Also, Raidz3 with so many disks might not be ideal, what is the preferred pool size for the configuration above? The last question I have is about scrubs and deep S.M.A.R.T. test, how often would you do these and how to make sure they do not run during the random-IO load?
I am upgrading to a new server with a 60 bay JBOD attached next month and it would be best to set this NAS up properly at once. Please let me know what you would change about the configuration, everything is acceptable. Main concern is setting up the VDEV/ZFS correctly to achieve a reasonable IO and not loosing most of my capacity. Server will be fully backed up eventually, so mirrors are not needed. Same load and usage as the server mentioned above.
Thanks for assisting so far, love to chat more with you about TrueNas 13 also!
First and foremost, thanks to everyone who is taking their time to assist me with some questions I have, appreciated.
So, let's jump right into it.
I have been experimenting with TrueNas 12.8 for a while now with some large scale systems (250-640TiB), and I believe that there are multiple things that I could set-up a lot better. The workload that I experience on these devices is the following:
- Writes up to 1GB/s consistently throughout the day, not constant but peaks every 10 minutes.
- Daily extremely high random-IO across all disks for 2 hours. Idling my writes during this period. (same hours every day - check mechanism)
- Rsync (mirror) of the whole server to an offsite location. Sometimes interferes with the random-IO load process.
- (2x) Xeon E5-2660 v3
- 128GB DDR4
- (36x) 8 or 18TB SAS (Seagate or WD)
- Raidz3 pool with 36 drives
- 4x 10G LAN (dedicated 10G towards the servers that do writes/IO/backup)
- 2x 480GB Sata (bootdrive)
- Storage is used mostly for archival and backup.
My main concern is not being able to do the check mechanism due to load elsewhere, the disks should be prioritised to do these heavy random-IO tasks always. Is there any way to set this up properly? My other question is working with another layer of SSD's inside the NAS, for cache or ZIL or anything else that would benefit this setup. Also, Raidz3 with so many disks might not be ideal, what is the preferred pool size for the configuration above? The last question I have is about scrubs and deep S.M.A.R.T. test, how often would you do these and how to make sure they do not run during the random-IO load?
I am upgrading to a new server with a 60 bay JBOD attached next month and it would be best to set this NAS up properly at once. Please let me know what you would change about the configuration, everything is acceptable. Main concern is setting up the VDEV/ZFS correctly to achieve a reasonable IO and not loosing most of my capacity. Server will be fully backed up eventually, so mirrors are not needed. Same load and usage as the server mentioned above.
- (2x) AMD EPYC 7413
- 512GB DDR4
- 480GB Micron 5300 PRO
- Mellanox 100GbE Adapter ConnectX-6 Dx (Support?)
- Broadcom HBA 9500-8e
- (60x) 18TB SAS (more JBOD added later)
Thanks for assisting so far, love to chat more with you about TrueNas 13 also!