Scrub of pool A cuase freenas to become unresponsive! Scrub on pool b has no problems.

Snow

Patron

Feb 28, 2020

#1

Its my main pool that is doing it. After 11.3 when I do a Scrub it causes FreeNas to become unresponsive.
The hardware is my main system posted below. I have never had this problem in the 8 or so years ruining FreeNas.
I am not sure where to start? It seems to me to point to a disk, but I have not received any smart test errors.
But I also have not ran a smart test from the cli. Any help would be welcome. I am stumped on this one.

This is the 2nd time it has done this to me. When I come home and freenas is locked up.
Only way I can fix it is to forces a reboot via IPMI.

https://www.youtube.com/c/TechNitWit/videos

SuperMicro CSE-846 24 SuperChassis with SAS3 BP (This is now used for TNW)
MB SuperMicro X9DRi-F
OS TrueNas 12.2
HBA 9300 LSI 8i
CPU 2x Intel(R) Xeon(R) CPU E5-2630L
RAM 160Gb DDR3 ECC Reg 1866
DISKS Pool 1 8x 6Tb Disks Raidz2
NIC Broadcom 10Gb Dual SFP+ Nic
PSUS Dual 1200W Platinum PSU's
SFP+ Cisco DAC TwinX SFP-H10GB-CU2M
UPS Liebert GXT4 3000VA BBU @ 48Vdc @ 64Amps With SNMP Card
UPS Liebert GXT4 BBU Expanson Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Silverstone 4U
Asus X399 Rog Strix,
TrueNas Sacle, BlueFish
TrueNas 12.0 RC3
FreeNas 11.3 U-7
Old Intel Xeon E5-2620 @ 2.00GHz 2x @ (24 cores)
New Intel Xeon E5-2630 v2 @ 2.60GHz 2x @ (24 cores)
AMD Ryzen Threadripper 2950X 16-Core Processor 32 Threads @ 4.2Ghz
GPU EVGA RTX 3090 24GB VRam.
128GB DDR4 3200Mhz ECC Reg Nimax
~~128Gb of Kingston KTH-PL310Q/16G 16Gb DDR3 PC3-8500 Reg4~~
LSI SAS 9302-8i-8e IT mode
LSI SAS 9300-81 IT Mode SSD HBA
~~LSI SAS 9211-8i Flashed to IR/IT Mode with newest Firmware P9~~
FSP Dual Reduadent PSU's 2x 930W
2x1100W 80 Plus Gold
10Gb/s Mellanox Conectx2 Nic1
Dual 10Gb/s Intel x520
40Gb/s Upgrade coming soon
Intel Pro NT 10/1000 2 Port Nic2
~~IPMI updated to newest Firmware 1st system with it and I love!!!~~
2x 45Bay Jbods 872 a dual and single BP
Pool 1 8x 4TB, 65x 3TB Seagate IronWolfs Mixed with WD Reds 123.11 TB
Pool 2 6x 1Tb Kingston and PNY SSD's for App pool.
1x 120Gb Kingston SSD Boot
3x 500Gb Kingston SSD Jail's Raidz1
2x500Gb Crucial MX500 for Jails, Junk drive's died
in under 50 day's smart errors on one drive
2x128GB AData SU800 SLOG added back helps with database loading.
Plugin's/Jails/Vm's Plex, ~~SickRage~~, ~~SickChill~~, Medusa, Sabnzbd, SNMP Server,
~~ZoneMinder~~, Home Ass, Transmission, CouchPotato, ~~PoS Datebase~~ Misc Vm's.
I am sorry sonarr sucks Just like UnRaid.
Cisco DAC TwinX SFP-H10GB-CU2M 10GB/s Copper SFP+ Cable
Liebert GXT3 2000VA UPS @ 48Vdc @ 64Amps With SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Server4 Win 2019 Server NVR Server WSUS, DNS.
Supermicro X11SCA-F-O
Chassis CSE-835TQ-R800B
CPU: Intel 8700K @ 4.3Ghz
Memory: None ECC 32Gb-DDR4
LSI Avago 9200 8x 1TB WD Blues Raid 6 +6.9Tb
Nvme Adata 500Gb System Disk.
Dual 120Gb Adat SSD for Back up.
Server5 (In the Works) Hp 380 Gen 8 Dual Zeon 8 cores, 96Gb DDR3-1600,
Server5 2x-10K-300Gb Disks, VM Host, Website, Domain Controller &

Switch 1 Unifi US-48-500W 1Gb's
Switch 2 Unifi US‑16‑XG 10Gb's
Server 3 Pfsense Xeon 3GHZ 8 core 64Gb/s ECC 6 10GB/s Nics 1x IPMI
Server 3 2x 500GB SSD in a Mirror
WIFI Ap's Unifi AC-AP 3x
Liebert GXT3 2000VA @ 48Vdc @ 64Amps UPS with SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
PDU/CDU Sentry Smart CDU Version 6.0g 208V 30Amp CS-16V2-L30M 2x
1 On mains the other on UPS. (16) x IEC 60320/C13 (4)x
20A Fused for two (2) branch with tool-less retraction.

Proper Power Supply Sizing Guidance
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/
Medusa Install Guide 11.2-U7
https://www.ixsystems.com/community/threads/guide-for-medusa.59425/page-2#post-557505
ZFS Primer Must Read if you plan on ruining a NAS System!
https://www.ixsystems.com/documentation/freenas/11.2/zfsprimer.html?highlight=primer
LSI Card Master List (LSI Raid Controller and Expanders, HBA Complete Listing Plus OEM Models)
https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/
The X9 Board Thread's
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/
https://forums.freenas.org/index.ph...ow-much-ram-for-vms-usable.63636/#post-470411
How To Migrate data from Pool A to a larger Pool B
https://forums.freenas.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
How to Move Jails Form A to B
https://www.ixsystems.com/community...n-volume-to-new-ssd-volume.42105/#post-271740]

https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

sretalla

Powered by Neutrality

Moderator

Feb 29, 2020

#2

You can do zpool scrub -s poolname to stop the scrub (just a general trick if you find yourself in that situation and need things to return to normal immediately)

Where is your system dataset? if it's in that pool that may be it.

Aiming to mostly replicate the build from @Stux (with some mods, hopefully around about as good as that link)

4 xSamsung 850 EVO Basic (500GB, 2.5") - - VMs/Jails
1 xASUS Z10PA-D8 (LGA 2011-v3, Intel C612 PCH, ATX) - - Dual socket MoBo
~~2 xWD Green 3D NAND (120GB, 2.5") - - Boot drives (maybe mess around trying out the thread to put swap here too link)~~
1 x Kingston UV400 120GB SSD - boot drive (hit the 3D NAND/TRIM bug with the original WD green selection, failing scrub and showing as corrupted OS files) Decided to go with no mirror and use the config backup script
2 xIntel Xeon E5-2620 v4 (LGA 2011-v3, 2.10GHz) - - 8 core/16 threads per Chip
2 xNoctua NH-U9S (12.50cm)
1 xCorsair HX1200 (1200W) - PSU to support 24 HDD + several SSD and PCI cards
4 xKingston Value RAM (32GB, DDR4-2400, ECC RDIMM 288)
2 xNoctua NF-A8 PWM Premium 80mm PC Computer Case Fan
~~3 xNoctua NF-F12 PWM Cooling Fan~~
3 xNoctua NF-F12 PPC 3000 PWM (120mm) * having noted later in Stux's thread that 1500 RPM is not sufficient to cool the HDDs. Corsair Commander Pro to control the fans (see script and code)
1 xNORCO 4U Rack Mount 24 x Hot-Swappable SATA/SAS 6G Drive Bays Server Rack mount RPC-4224
6 xCableCreation Internal Mini SAS HD Cable, Mini SAS SFF-8643 to Mini SAS 36Pin SFF-8087 Cable
1 xLSI Logic Controller Card 05-25699-00 9305-24i 24-Port SAS 12Gb/s PCI-Express 3.0 Host Bus Adapter
TrueNAS Core 13.0-U6.1
Use existing Drives 8 x10TB WD Red, 8 x4TB WD Purple, + a mix of WD Purple and shucked WD Elements 12TB x 8

ESXi-pfSense-TrueNAS-Docker host

CASE: Fractal Node 804
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Cooler block
RAM: CRUCIAL 64GB DDR4-2133 ECC RDIMMs
HDD: WD RED 3TBx8
SSD: 4 xSamsung 850 EVO Basic (500GB, 2.5") - - VMs/Jails
HBA: LSI 9300-16i
OS: 1 x Kingston UV400 120GB SSD - boot drive
PSU: Corsair RM1000
Version: TrueNAS CORE 13.0 -U6.1
FANS: 3xFractal R3 120mm - 3 Front, 1 Rear. Corsair Commander Pro to control the fans (see script and code)
CPU FAN: 1xCorsair H60 CPU Radiator - Front
NIC: Intel EXPI9402PTBLK Pro, Dual-Gigabit Adapter (plus the 2 onboard Intel NICs, 1x 210, 1x 218)

VM/Docker host, using ESXi and running pfSense alongside FreeNAS (separate Dual Intel NIC added, dedicated to the pfSense VM)

Other Systems

TrueNAS CORE test system:

CASE: Old Silverstone HTPC case
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Cooler block
RAM: CRUCIAL 32GB DDR4-2133 ECC RDIMMs
HDD: WD RED 8TBx3
OS: 1 x Kingston UV400 120GB SSD - boot drive
PSU: Corsair RM1000
Version: TrueNAS CORE 13.0-U6.1

SCALE Cluster:

2x Intel NUCs running TrueNAS SCALE 24.04-BETA1
64GB RAM
10th Generation Intel i7
Samsung NVME SSD 1TB, QVO SSD 1TB
Boot from Samsung Portable T7 SSD USBC

CASE: Fractal Define 7 running TrueNAS SCALE 24.04-BETA1
MB: ASUS P10S-I Series
RAM: 32 GB
CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz
HDD: 3 WD REDs and a few SSDs
PSU: Fractal ION 2+ 650W

Like my post (or find it helpful)?... Please Like my post over here... -------------------------------------------------------------------------------------->

Snow

Patron

Feb 29, 2020

#3

it is the dataset on the spindle disks, the larger of the 2 arrays.

https://www.youtube.com/c/TechNitWit/videos

SuperMicro CSE-846 24 SuperChassis with SAS3 BP (This is now used for TNW)
MB SuperMicro X9DRi-F
OS TrueNas 12.2
HBA 9300 LSI 8i
CPU 2x Intel(R) Xeon(R) CPU E5-2630L
RAM 160Gb DDR3 ECC Reg 1866
DISKS Pool 1 8x 6Tb Disks Raidz2
NIC Broadcom 10Gb Dual SFP+ Nic
PSUS Dual 1200W Platinum PSU's
SFP+ Cisco DAC TwinX SFP-H10GB-CU2M
UPS Liebert GXT4 3000VA BBU @ 48Vdc @ 64Amps With SNMP Card
UPS Liebert GXT4 BBU Expanson Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Silverstone 4U
Asus X399 Rog Strix,
TrueNas Sacle, BlueFish
TrueNas 12.0 RC3
FreeNas 11.3 U-7
Old Intel Xeon E5-2620 @ 2.00GHz 2x @ (24 cores)
New Intel Xeon E5-2630 v2 @ 2.60GHz 2x @ (24 cores)
AMD Ryzen Threadripper 2950X 16-Core Processor 32 Threads @ 4.2Ghz
GPU EVGA RTX 3090 24GB VRam.
128GB DDR4 3200Mhz ECC Reg Nimax
~~128Gb of Kingston KTH-PL310Q/16G 16Gb DDR3 PC3-8500 Reg4~~
LSI SAS 9302-8i-8e IT mode
LSI SAS 9300-81 IT Mode SSD HBA
~~LSI SAS 9211-8i Flashed to IR/IT Mode with newest Firmware P9~~
FSP Dual Reduadent PSU's 2x 930W
2x1100W 80 Plus Gold
10Gb/s Mellanox Conectx2 Nic1
Dual 10Gb/s Intel x520
40Gb/s Upgrade coming soon
Intel Pro NT 10/1000 2 Port Nic2
~~IPMI updated to newest Firmware 1st system with it and I love!!!~~
2x 45Bay Jbods 872 a dual and single BP
Pool 1 8x 4TB, 65x 3TB Seagate IronWolfs Mixed with WD Reds 123.11 TB
Pool 2 6x 1Tb Kingston and PNY SSD's for App pool.
1x 120Gb Kingston SSD Boot
3x 500Gb Kingston SSD Jail's Raidz1
2x500Gb Crucial MX500 for Jails, Junk drive's died
in under 50 day's smart errors on one drive
2x128GB AData SU800 SLOG added back helps with database loading.
Plugin's/Jails/Vm's Plex, ~~SickRage~~, ~~SickChill~~, Medusa, Sabnzbd, SNMP Server,
~~ZoneMinder~~, Home Ass, Transmission, CouchPotato, ~~PoS Datebase~~ Misc Vm's.
I am sorry sonarr sucks Just like UnRaid.
Cisco DAC TwinX SFP-H10GB-CU2M 10GB/s Copper SFP+ Cable
Liebert GXT3 2000VA UPS @ 48Vdc @ 64Amps With SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Server4 Win 2019 Server NVR Server WSUS, DNS.
Supermicro X11SCA-F-O
Chassis CSE-835TQ-R800B
CPU: Intel 8700K @ 4.3Ghz
Memory: None ECC 32Gb-DDR4
LSI Avago 9200 8x 1TB WD Blues Raid 6 +6.9Tb
Nvme Adata 500Gb System Disk.
Dual 120Gb Adat SSD for Back up.
Server5 (In the Works) Hp 380 Gen 8 Dual Zeon 8 cores, 96Gb DDR3-1600,
Server5 2x-10K-300Gb Disks, VM Host, Website, Domain Controller &

Switch 1 Unifi US-48-500W 1Gb's
Switch 2 Unifi US‑16‑XG 10Gb's
Server 3 Pfsense Xeon 3GHZ 8 core 64Gb/s ECC 6 10GB/s Nics 1x IPMI
Server 3 2x 500GB SSD in a Mirror
WIFI Ap's Unifi AC-AP 3x
Liebert GXT3 2000VA @ 48Vdc @ 64Amps UPS with SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
PDU/CDU Sentry Smart CDU Version 6.0g 208V 30Amp CS-16V2-L30M 2x
1 On mains the other on UPS. (16) x IEC 60320/C13 (4)x
20A Fused for two (2) branch with tool-less retraction.

Proper Power Supply Sizing Guidance
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/
Medusa Install Guide 11.2-U7
https://www.ixsystems.com/community/threads/guide-for-medusa.59425/page-2#post-557505
ZFS Primer Must Read if you plan on ruining a NAS System!
https://www.ixsystems.com/documentation/freenas/11.2/zfsprimer.html?highlight=primer
LSI Card Master List (LSI Raid Controller and Expanders, HBA Complete Listing Plus OEM Models)
https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/
The X9 Board Thread's
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/
https://forums.freenas.org/index.ph...ow-much-ram-for-vms-usable.63636/#post-470411
How To Migrate data from Pool A to a larger Pool B
https://forums.freenas.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
How to Move Jails Form A to B
https://www.ixsystems.com/community...n-volume-to-new-ssd-volume.42105/#post-271740]

https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

Snow

Patron

Mar 1, 2020

#4

Looks like In IPMI I see I am having a Error. Error That I see is OME" CPLD" CATERR". If any one Know this error better would be great All I can find is its a CPU or Ram Problem.

https://www.youtube.com/c/TechNitWit/videos

SuperMicro CSE-846 24 SuperChassis with SAS3 BP (This is now used for TNW)
MB SuperMicro X9DRi-F
OS TrueNas 12.2
HBA 9300 LSI 8i
CPU 2x Intel(R) Xeon(R) CPU E5-2630L
RAM 160Gb DDR3 ECC Reg 1866
DISKS Pool 1 8x 6Tb Disks Raidz2
NIC Broadcom 10Gb Dual SFP+ Nic
PSUS Dual 1200W Platinum PSU's
SFP+ Cisco DAC TwinX SFP-H10GB-CU2M
UPS Liebert GXT4 3000VA BBU @ 48Vdc @ 64Amps With SNMP Card
UPS Liebert GXT4 BBU Expanson Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Silverstone 4U
Asus X399 Rog Strix,
TrueNas Sacle, BlueFish
TrueNas 12.0 RC3
FreeNas 11.3 U-7
Old Intel Xeon E5-2620 @ 2.00GHz 2x @ (24 cores)
New Intel Xeon E5-2630 v2 @ 2.60GHz 2x @ (24 cores)
AMD Ryzen Threadripper 2950X 16-Core Processor 32 Threads @ 4.2Ghz
GPU EVGA RTX 3090 24GB VRam.
128GB DDR4 3200Mhz ECC Reg Nimax
~~128Gb of Kingston KTH-PL310Q/16G 16Gb DDR3 PC3-8500 Reg4~~
LSI SAS 9302-8i-8e IT mode
LSI SAS 9300-81 IT Mode SSD HBA
~~LSI SAS 9211-8i Flashed to IR/IT Mode with newest Firmware P9~~
FSP Dual Reduadent PSU's 2x 930W
2x1100W 80 Plus Gold
10Gb/s Mellanox Conectx2 Nic1
Dual 10Gb/s Intel x520
40Gb/s Upgrade coming soon
Intel Pro NT 10/1000 2 Port Nic2
~~IPMI updated to newest Firmware 1st system with it and I love!!!~~
2x 45Bay Jbods 872 a dual and single BP
Pool 1 8x 4TB, 65x 3TB Seagate IronWolfs Mixed with WD Reds 123.11 TB
Pool 2 6x 1Tb Kingston and PNY SSD's for App pool.
1x 120Gb Kingston SSD Boot
3x 500Gb Kingston SSD Jail's Raidz1
2x500Gb Crucial MX500 for Jails, Junk drive's died
in under 50 day's smart errors on one drive
2x128GB AData SU800 SLOG added back helps with database loading.
Plugin's/Jails/Vm's Plex, ~~SickRage~~, ~~SickChill~~, Medusa, Sabnzbd, SNMP Server,
~~ZoneMinder~~, Home Ass, Transmission, CouchPotato, ~~PoS Datebase~~ Misc Vm's.
I am sorry sonarr sucks Just like UnRaid.
Cisco DAC TwinX SFP-H10GB-CU2M 10GB/s Copper SFP+ Cable
Liebert GXT3 2000VA UPS @ 48Vdc @ 64Amps With SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Server4 Win 2019 Server NVR Server WSUS, DNS.
Supermicro X11SCA-F-O
Chassis CSE-835TQ-R800B
CPU: Intel 8700K @ 4.3Ghz
Memory: None ECC 32Gb-DDR4
LSI Avago 9200 8x 1TB WD Blues Raid 6 +6.9Tb
Nvme Adata 500Gb System Disk.
Dual 120Gb Adat SSD for Back up.
Server5 (In the Works) Hp 380 Gen 8 Dual Zeon 8 cores, 96Gb DDR3-1600,
Server5 2x-10K-300Gb Disks, VM Host, Website, Domain Controller &

Switch 1 Unifi US-48-500W 1Gb's
Switch 2 Unifi US‑16‑XG 10Gb's
Server 3 Pfsense Xeon 3GHZ 8 core 64Gb/s ECC 6 10GB/s Nics 1x IPMI
Server 3 2x 500GB SSD in a Mirror
WIFI Ap's Unifi AC-AP 3x
Liebert GXT3 2000VA @ 48Vdc @ 64Amps UPS with SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
PDU/CDU Sentry Smart CDU Version 6.0g 208V 30Amp CS-16V2-L30M 2x
1 On mains the other on UPS. (16) x IEC 60320/C13 (4)x
20A Fused for two (2) branch with tool-less retraction.

Proper Power Supply Sizing Guidance
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/
Medusa Install Guide 11.2-U7
https://www.ixsystems.com/community/threads/guide-for-medusa.59425/page-2#post-557505
ZFS Primer Must Read if you plan on ruining a NAS System!
https://www.ixsystems.com/documentation/freenas/11.2/zfsprimer.html?highlight=primer
LSI Card Master List (LSI Raid Controller and Expanders, HBA Complete Listing Plus OEM Models)
https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/
The X9 Board Thread's
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/
https://forums.freenas.org/index.ph...ow-much-ram-for-vms-usable.63636/#post-470411
How To Migrate data from Pool A to a larger Pool B
https://forums.freenas.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
How to Move Jails Form A to B
https://www.ixsystems.com/community...n-volume-to-new-ssd-volume.42105/#post-271740]

https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

Snow

Patron

Mar 7, 2020

#5

Well I did some more searching and found that the error is do to the restart. So It looks like the system is hanging well doing a scrub. Not sure why as it has more then enough Ram & CPU. What should I do next? I feel from what I was reading off of Supermicro support page. That I should be looking at the software. As the software is Freenas, It is what is causing the hanging?

I read other stuff off a different site that says it could be the CPU or Ram. But then again Supermicro support says that this errors is due to the hanging of the system. Then if you reboot it well it is hanging it cause that error. Should I just do a Memtest86? Its not like it is happening all the time, So I can not just pull dimms to see if it is fixed. I am not sure how to test the CPU's ? Is their a boot able stress tester for CPU's?

https://www.youtube.com/c/TechNitWit/videos

SuperMicro CSE-846 24 SuperChassis with SAS3 BP (This is now used for TNW)
MB SuperMicro X9DRi-F
OS TrueNas 12.2
HBA 9300 LSI 8i
CPU 2x Intel(R) Xeon(R) CPU E5-2630L
RAM 160Gb DDR3 ECC Reg 1866
DISKS Pool 1 8x 6Tb Disks Raidz2
NIC Broadcom 10Gb Dual SFP+ Nic
PSUS Dual 1200W Platinum PSU's
SFP+ Cisco DAC TwinX SFP-H10GB-CU2M
UPS Liebert GXT4 3000VA BBU @ 48Vdc @ 64Amps With SNMP Card
UPS Liebert GXT4 BBU Expanson Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Silverstone 4U
Asus X399 Rog Strix,
TrueNas Sacle, BlueFish
TrueNas 12.0 RC3
FreeNas 11.3 U-7
Old Intel Xeon E5-2620 @ 2.00GHz 2x @ (24 cores)
New Intel Xeon E5-2630 v2 @ 2.60GHz 2x @ (24 cores)
AMD Ryzen Threadripper 2950X 16-Core Processor 32 Threads @ 4.2Ghz
GPU EVGA RTX 3090 24GB VRam.
128GB DDR4 3200Mhz ECC Reg Nimax
~~128Gb of Kingston KTH-PL310Q/16G 16Gb DDR3 PC3-8500 Reg4~~
LSI SAS 9302-8i-8e IT mode
LSI SAS 9300-81 IT Mode SSD HBA
~~LSI SAS 9211-8i Flashed to IR/IT Mode with newest Firmware P9~~
FSP Dual Reduadent PSU's 2x 930W
2x1100W 80 Plus Gold
10Gb/s Mellanox Conectx2 Nic1
Dual 10Gb/s Intel x520
40Gb/s Upgrade coming soon
Intel Pro NT 10/1000 2 Port Nic2
~~IPMI updated to newest Firmware 1st system with it and I love!!!~~
2x 45Bay Jbods 872 a dual and single BP
Pool 1 8x 4TB, 65x 3TB Seagate IronWolfs Mixed with WD Reds 123.11 TB
Pool 2 6x 1Tb Kingston and PNY SSD's for App pool.
1x 120Gb Kingston SSD Boot
3x 500Gb Kingston SSD Jail's Raidz1
2x500Gb Crucial MX500 for Jails, Junk drive's died
in under 50 day's smart errors on one drive
2x128GB AData SU800 SLOG added back helps with database loading.
Plugin's/Jails/Vm's Plex, ~~SickRage~~, ~~SickChill~~, Medusa, Sabnzbd, SNMP Server,
~~ZoneMinder~~, Home Ass, Transmission, CouchPotato, ~~PoS Datebase~~ Misc Vm's.
I am sorry sonarr sucks Just like UnRaid.
Cisco DAC TwinX SFP-H10GB-CU2M 10GB/s Copper SFP+ Cable
Liebert GXT3 2000VA UPS @ 48Vdc @ 64Amps With SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Server4 Win 2019 Server NVR Server WSUS, DNS.
Supermicro X11SCA-F-O
Chassis CSE-835TQ-R800B
CPU: Intel 8700K @ 4.3Ghz
Memory: None ECC 32Gb-DDR4
LSI Avago 9200 8x 1TB WD Blues Raid 6 +6.9Tb
Nvme Adata 500Gb System Disk.
Dual 120Gb Adat SSD for Back up.
Server5 (In the Works) Hp 380 Gen 8 Dual Zeon 8 cores, 96Gb DDR3-1600,
Server5 2x-10K-300Gb Disks, VM Host, Website, Domain Controller &

Switch 1 Unifi US-48-500W 1Gb's
Switch 2 Unifi US‑16‑XG 10Gb's
Server 3 Pfsense Xeon 3GHZ 8 core 64Gb/s ECC 6 10GB/s Nics 1x IPMI
Server 3 2x 500GB SSD in a Mirror
WIFI Ap's Unifi AC-AP 3x
Liebert GXT3 2000VA @ 48Vdc @ 64Amps UPS with SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
PDU/CDU Sentry Smart CDU Version 6.0g 208V 30Amp CS-16V2-L30M 2x
1 On mains the other on UPS. (16) x IEC 60320/C13 (4)x
20A Fused for two (2) branch with tool-less retraction.

Proper Power Supply Sizing Guidance
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/
Medusa Install Guide 11.2-U7
https://www.ixsystems.com/community/threads/guide-for-medusa.59425/page-2#post-557505
ZFS Primer Must Read if you plan on ruining a NAS System!
https://www.ixsystems.com/documentation/freenas/11.2/zfsprimer.html?highlight=primer
LSI Card Master List (LSI Raid Controller and Expanders, HBA Complete Listing Plus OEM Models)
https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/
The X9 Board Thread's
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/
https://forums.freenas.org/index.ph...ow-much-ram-for-vms-usable.63636/#post-470411
How To Migrate data from Pool A to a larger Pool B
https://forums.freenas.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
How to Move Jails Form A to B
https://www.ixsystems.com/community...n-volume-to-new-ssd-volume.42105/#post-271740]

https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

sretalla

Powered by Neutrality

Moderator

Mar 7, 2020

#6

You might want to try moving the system dataset to the boot pool or another ssd pool if you have the option and see if the problem abates.

Aiming to mostly replicate the build from @Stux (with some mods, hopefully around about as good as that link)

4 xSamsung 850 EVO Basic (500GB, 2.5") - - VMs/Jails
1 xASUS Z10PA-D8 (LGA 2011-v3, Intel C612 PCH, ATX) - - Dual socket MoBo
~~2 xWD Green 3D NAND (120GB, 2.5") - - Boot drives (maybe mess around trying out the thread to put swap here too link)~~
1 x Kingston UV400 120GB SSD - boot drive (hit the 3D NAND/TRIM bug with the original WD green selection, failing scrub and showing as corrupted OS files) Decided to go with no mirror and use the config backup script
2 xIntel Xeon E5-2620 v4 (LGA 2011-v3, 2.10GHz) - - 8 core/16 threads per Chip
2 xNoctua NH-U9S (12.50cm)
1 xCorsair HX1200 (1200W) - PSU to support 24 HDD + several SSD and PCI cards
4 xKingston Value RAM (32GB, DDR4-2400, ECC RDIMM 288)
2 xNoctua NF-A8 PWM Premium 80mm PC Computer Case Fan
~~3 xNoctua NF-F12 PWM Cooling Fan~~
3 xNoctua NF-F12 PPC 3000 PWM (120mm) * having noted later in Stux's thread that 1500 RPM is not sufficient to cool the HDDs. Corsair Commander Pro to control the fans (see script and code)
1 xNORCO 4U Rack Mount 24 x Hot-Swappable SATA/SAS 6G Drive Bays Server Rack mount RPC-4224
6 xCableCreation Internal Mini SAS HD Cable, Mini SAS SFF-8643 to Mini SAS 36Pin SFF-8087 Cable
1 xLSI Logic Controller Card 05-25699-00 9305-24i 24-Port SAS 12Gb/s PCI-Express 3.0 Host Bus Adapter
TrueNAS Core 13.0-U6.1
Use existing Drives 8 x10TB WD Red, 8 x4TB WD Purple, + a mix of WD Purple and shucked WD Elements 12TB x 8

ESXi-pfSense-TrueNAS-Docker host

CASE: Fractal Node 804
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Cooler block
RAM: CRUCIAL 64GB DDR4-2133 ECC RDIMMs
HDD: WD RED 3TBx8
SSD: 4 xSamsung 850 EVO Basic (500GB, 2.5") - - VMs/Jails
HBA: LSI 9300-16i
OS: 1 x Kingston UV400 120GB SSD - boot drive
PSU: Corsair RM1000
Version: TrueNAS CORE 13.0 -U6.1
FANS: 3xFractal R3 120mm - 3 Front, 1 Rear. Corsair Commander Pro to control the fans (see script and code)
CPU FAN: 1xCorsair H60 CPU Radiator - Front
NIC: Intel EXPI9402PTBLK Pro, Dual-Gigabit Adapter (plus the 2 onboard Intel NICs, 1x 210, 1x 218)

VM/Docker host, using ESXi and running pfSense alongside FreeNAS (separate Dual Intel NIC added, dedicated to the pfSense VM)

Other Systems

TrueNAS CORE test system:

CASE: Old Silverstone HTPC case
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Cooler block
RAM: CRUCIAL 32GB DDR4-2133 ECC RDIMMs
HDD: WD RED 8TBx3
OS: 1 x Kingston UV400 120GB SSD - boot drive
PSU: Corsair RM1000
Version: TrueNAS CORE 13.0-U6.1

SCALE Cluster:

2x Intel NUCs running TrueNAS SCALE 24.04-BETA1
64GB RAM
10th Generation Intel i7
Samsung NVME SSD 1TB, QVO SSD 1TB
Boot from Samsung Portable T7 SSD USBC

CASE: Fractal Define 7 running TrueNAS SCALE 24.04-BETA1
MB: ASUS P10S-I Series
RAM: 32 GB
CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz
HDD: 3 WD REDs and a few SSDs
PSU: Fractal ION 2+ 650W

Like my post (or find it helpful)?... Please Like my post over here... -------------------------------------------------------------------------------------->

Snow

Patron

Mar 7, 2020

#7

boot is on dual ssd's

https://www.youtube.com/c/TechNitWit/videos

SuperMicro CSE-846 24 SuperChassis with SAS3 BP (This is now used for TNW)
MB SuperMicro X9DRi-F
OS TrueNas 12.2
HBA 9300 LSI 8i
CPU 2x Intel(R) Xeon(R) CPU E5-2630L
RAM 160Gb DDR3 ECC Reg 1866
DISKS Pool 1 8x 6Tb Disks Raidz2
NIC Broadcom 10Gb Dual SFP+ Nic
PSUS Dual 1200W Platinum PSU's
SFP+ Cisco DAC TwinX SFP-H10GB-CU2M
UPS Liebert GXT4 3000VA BBU @ 48Vdc @ 64Amps With SNMP Card
UPS Liebert GXT4 BBU Expanson Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Silverstone 4U
Asus X399 Rog Strix,
TrueNas Sacle, BlueFish
TrueNas 12.0 RC3
FreeNas 11.3 U-7
Old Intel Xeon E5-2620 @ 2.00GHz 2x @ (24 cores)
New Intel Xeon E5-2630 v2 @ 2.60GHz 2x @ (24 cores)
AMD Ryzen Threadripper 2950X 16-Core Processor 32 Threads @ 4.2Ghz
GPU EVGA RTX 3090 24GB VRam.
128GB DDR4 3200Mhz ECC Reg Nimax
~~128Gb of Kingston KTH-PL310Q/16G 16Gb DDR3 PC3-8500 Reg4~~
LSI SAS 9302-8i-8e IT mode
LSI SAS 9300-81 IT Mode SSD HBA
~~LSI SAS 9211-8i Flashed to IR/IT Mode with newest Firmware P9~~
FSP Dual Reduadent PSU's 2x 930W
2x1100W 80 Plus Gold
10Gb/s Mellanox Conectx2 Nic1
Dual 10Gb/s Intel x520
40Gb/s Upgrade coming soon
Intel Pro NT 10/1000 2 Port Nic2
~~IPMI updated to newest Firmware 1st system with it and I love!!!~~
2x 45Bay Jbods 872 a dual and single BP
Pool 1 8x 4TB, 65x 3TB Seagate IronWolfs Mixed with WD Reds 123.11 TB
Pool 2 6x 1Tb Kingston and PNY SSD's for App pool.
1x 120Gb Kingston SSD Boot
3x 500Gb Kingston SSD Jail's Raidz1
2x500Gb Crucial MX500 for Jails, Junk drive's died
in under 50 day's smart errors on one drive
2x128GB AData SU800 SLOG added back helps with database loading.
Plugin's/Jails/Vm's Plex, ~~SickRage~~, ~~SickChill~~, Medusa, Sabnzbd, SNMP Server,
~~ZoneMinder~~, Home Ass, Transmission, CouchPotato, ~~PoS Datebase~~ Misc Vm's.
I am sorry sonarr sucks Just like UnRaid.
Cisco DAC TwinX SFP-H10GB-CU2M 10GB/s Copper SFP+ Cable
Liebert GXT3 2000VA UPS @ 48Vdc @ 64Amps With SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
Using Cyberpower 100 SNMP drivers

Server4 Win 2019 Server NVR Server WSUS, DNS.
Supermicro X11SCA-F-O
Chassis CSE-835TQ-R800B
CPU: Intel 8700K @ 4.3Ghz
Memory: None ECC 32Gb-DDR4
LSI Avago 9200 8x 1TB WD Blues Raid 6 +6.9Tb
Nvme Adata 500Gb System Disk.
Dual 120Gb Adat SSD for Back up.
Server5 (In the Works) Hp 380 Gen 8 Dual Zeon 8 cores, 96Gb DDR3-1600,
Server5 2x-10K-300Gb Disks, VM Host, Website, Domain Controller &

Switch 1 Unifi US-48-500W 1Gb's
Switch 2 Unifi US‑16‑XG 10Gb's
Server 3 Pfsense Xeon 3GHZ 8 core 64Gb/s ECC 6 10GB/s Nics 1x IPMI
Server 3 2x 500GB SSD in a Mirror
WIFI Ap's Unifi AC-AP 3x
Liebert GXT3 2000VA @ 48Vdc @ 64Amps UPS with SNMP Card
Liebert GXT3 UPS Expansion Unit @ 48Vdc @ 128Amps
PDU/CDU Sentry Smart CDU Version 6.0g 208V 30Amp CS-16V2-L30M 2x
1 On mains the other on UPS. (16) x IEC 60320/C13 (4)x
20A Fused for two (2) branch with tool-less retraction.

Proper Power Supply Sizing Guidance
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/
Medusa Install Guide 11.2-U7
https://www.ixsystems.com/community/threads/guide-for-medusa.59425/page-2#post-557505
ZFS Primer Must Read if you plan on ruining a NAS System!
https://www.ixsystems.com/documentation/freenas/11.2/zfsprimer.html?highlight=primer
LSI Card Master List (LSI Raid Controller and Expanders, HBA Complete Listing Plus OEM Models)
https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/
The X9 Board Thread's
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/
https://forums.freenas.org/index.ph...ow-much-ram-for-vms-usable.63636/#post-470411
How To Migrate data from Pool A to a larger Pool B
https://forums.freenas.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
How to Move Jails Form A to B
https://www.ixsystems.com/community...n-volume-to-new-ssd-volume.42105/#post-271740]

https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

sretalla

Powered by Neutrality

Moderator

Mar 9, 2020

#8

Snow said:
boot is on dual ssd's

OK, so move the system dataset to it and see how things progress...

Aiming to mostly replicate the build from @Stux (with some mods, hopefully around about as good as that link)

4 xSamsung 850 EVO Basic (500GB, 2.5") - - VMs/Jails
1 xASUS Z10PA-D8 (LGA 2011-v3, Intel C612 PCH, ATX) - - Dual socket MoBo
~~2 xWD Green 3D NAND (120GB, 2.5") - - Boot drives (maybe mess around trying out the thread to put swap here too link)~~
1 x Kingston UV400 120GB SSD - boot drive (hit the 3D NAND/TRIM bug with the original WD green selection, failing scrub and showing as corrupted OS files) Decided to go with no mirror and use the config backup script
2 xIntel Xeon E5-2620 v4 (LGA 2011-v3, 2.10GHz) - - 8 core/16 threads per Chip
2 xNoctua NH-U9S (12.50cm)
1 xCorsair HX1200 (1200W) - PSU to support 24 HDD + several SSD and PCI cards
4 xKingston Value RAM (32GB, DDR4-2400, ECC RDIMM 288)
2 xNoctua NF-A8 PWM Premium 80mm PC Computer Case Fan
~~3 xNoctua NF-F12 PWM Cooling Fan~~
3 xNoctua NF-F12 PPC 3000 PWM (120mm) * having noted later in Stux's thread that 1500 RPM is not sufficient to cool the HDDs. Corsair Commander Pro to control the fans (see script and code)
1 xNORCO 4U Rack Mount 24 x Hot-Swappable SATA/SAS 6G Drive Bays Server Rack mount RPC-4224
6 xCableCreation Internal Mini SAS HD Cable, Mini SAS SFF-8643 to Mini SAS 36Pin SFF-8087 Cable
1 xLSI Logic Controller Card 05-25699-00 9305-24i 24-Port SAS 12Gb/s PCI-Express 3.0 Host Bus Adapter
TrueNAS Core 13.0-U6.1
Use existing Drives 8 x10TB WD Red, 8 x4TB WD Purple, + a mix of WD Purple and shucked WD Elements 12TB x 8

ESXi-pfSense-TrueNAS-Docker host

CASE: Fractal Node 804
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Cooler block
RAM: CRUCIAL 64GB DDR4-2133 ECC RDIMMs
HDD: WD RED 3TBx8
SSD: 4 xSamsung 850 EVO Basic (500GB, 2.5") - - VMs/Jails
HBA: LSI 9300-16i
OS: 1 x Kingston UV400 120GB SSD - boot drive
PSU: Corsair RM1000
Version: TrueNAS CORE 13.0 -U6.1
FANS: 3xFractal R3 120mm - 3 Front, 1 Rear. Corsair Commander Pro to control the fans (see script and code)
CPU FAN: 1xCorsair H60 CPU Radiator - Front
NIC: Intel EXPI9402PTBLK Pro, Dual-Gigabit Adapter (plus the 2 onboard Intel NICs, 1x 210, 1x 218)

VM/Docker host, using ESXi and running pfSense alongside FreeNAS (separate Dual Intel NIC added, dedicated to the pfSense VM)

Other Systems

TrueNAS CORE test system:

CASE: Old Silverstone HTPC case
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Cooler block
RAM: CRUCIAL 32GB DDR4-2133 ECC RDIMMs
HDD: WD RED 8TBx3
OS: 1 x Kingston UV400 120GB SSD - boot drive
PSU: Corsair RM1000
Version: TrueNAS CORE 13.0-U6.1

SCALE Cluster:

2x Intel NUCs running TrueNAS SCALE 24.04-BETA1
64GB RAM
10th Generation Intel i7
Samsung NVME SSD 1TB, QVO SSD 1TB
Boot from Samsung Portable T7 SSD USBC

CASE: Fractal Define 7 running TrueNAS SCALE 24.04-BETA1
MB: ASUS P10S-I Series
RAM: 32 GB
CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz
HDD: 3 WD REDs and a few SSDs
PSU: Fractal ION 2+ 650W

Like my post (or find it helpful)?... Please Like my post over here... -------------------------------------------------------------------------------------->

Important Announcement for the TrueNAS Community.

Scrub of pool A cuase freenas to become unresponsive! Scrub on pool b has no problems.

Snow

Patron

sretalla

Powered by Neutrality

Snow

Patron

Snow

Patron

Snow

Patron

sretalla

Powered by Neutrality

Snow

Patron

sretalla

Powered by Neutrality

Similar threads

Important Announcement for the TrueNAS Community.

Scrub of pool A cuase freenas to become unresponsive! Scrub on pool b has no problems.

Patron

Powered by Neutrality

Patron

Patron

Patron

Powered by Neutrality

Patron

Powered by Neutrality

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Scrub of pool A cuase freenas to become unresponsive! Scrub on pool b has no problems."

Similar threads