all my drives suddenly DEGRADED after months of working fine

[TheThomen]

CRITICAL​

Pool Shelf-Z2 state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.
The following devices are not healthy:

• Disk ST4000DM 000-1F2168 SM Z3019W0G is DEGRADED
• Disk ST4000DM 000-1F2168 SA Z30194ST is DEGRADED
• Disk ST4000DM 000-1F2168 SM Z304VQLY is DEGRADED
• Disk ST4000DM 000-1F2168 SM Z30195TE is DEGRADED
• Disk ST4000DM 000-1F2168 SM Z30125R5 is DEGRADED
• Disk ST4000DM 000-1F2168 SM Z3012WRX is DEGRADED
• Disk ST4000DM 000-1F2168 SM Z3019SPT is DEGRADED
• Disk ST4000DM 000-1F2168 SA Z30194L4 is DEGRADED

im useing a R320 with a LSI megaraid connected to a NETTAPP shelf with 8 4tb drives. not sure why but the pool works fine everythings running fine but the status is all the drives are degraded. tried reboot but no joy.

 

[Samuel Tai]

Is your MegaRAID flashed to IT mode? Or were you running in JBOD mode?

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[TheThomen]

Is your MegaRAID flashed to IT mode? Or were you running in JBOD mode?

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

Hi Samuel,

I flashed it mode on it about 6 months ago when it was in another system (that system never had this issue)
Please see the results in the image, the card is a LSI SAS 9200-8e

 

[TheThomen]

This is my setup, R320 with E5-2450L and the LSI connected to the shelf using 2 cables

 

[Redcoat]

Seagate's spec says these are SMR drives... See page 11 in https://www.seagate.com/www-content...a-fam/barracuda-new/en-us/docs/100805918n.pdf - since 2020 with revision "N". Presumably CMR pior to that???

 

[TheThomen]

Seagate's spec says these are SMR drives... See page 11 in https://www.seagate.com/www-content...a-fam/barracuda-new/en-us/docs/100805918n.pdf - since 2020 with revision "N". Presumably CMR pior to that???

how can I check? I just transferred a file and it saturated my upload

 

[TheThomen]

any ideas people?

 

[elvisimprsntr]

Desktop DM-SMR drives + ZFS = disaster.

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

 

[Samuel Tai]

any ideas people?

That spec sheet @Redcoat linked to indicates this model changed from CMR (which is OK for ZFS) to SMR (which is bad for ZFS) after 13 May 2020. If you purchased your drives after then, odds are your pool is constructed solely of the SMR drives.

 

[neofusion]

That spec sheet @Redcoat linked to indicates this model changed from CMR (which is OK for ZFS) to SMR (which is bad for ZFS) after 13 May 2020. If you purchased your drives after then, odds are your pool is constructed solely of the SMR drives.

The document is listed as having been revised on 13 May 2020.
It does not necessarily mean that they started rolling out the SMR-drives at that point in time; that may have happened before or after that date.

In other words, you would need to test the drives to be sure even if you bought them before that date.
I am not sure how to reliably do that.

 

[TheThomen]

gents after checking the drives were manufactured in 2015 with 16064 being the newest the rest are 14xxx

 

[TheThomen]

https://imgur.com/a/Jzloq0u all the drives

 

[Samuel Tai]

Yes, they're all SMR. They're also all desktop drives, which aren't suitable for NAS applications. Seagate's IronWolf line is their NAS line.

 

[neofusion]

Yes, they're all SMR. They're also all desktop drives, which aren't suitable for NAS applications. Seagate's IronWolf line is their NAS line.

Desktop drives, sure, but what is your source as to whether they are SMR or not?
ST4000DM004 is SMR but I haven't been able to find anything showing that ST4000DM000 is, quite the opposite actually.

 

[elvisimprsntr]

Desktop drives, sure, but what is your source as to whether they are SMR or not?
ST4000DM004 is SMR but I haven't been able to find anything showing that ST4000DM000 is, quite the opposite actually.

Seagate's own FAQ post.

CMR and SMR Hard Drives | Seagate US

See the list of recording technology utilized on our internal hard drives

www.seagate.com

Data sheet states TGMR, which from what I have read is a form of SMR

https://www.seagate.com/files/www-content/product-content/desktop-hdd-fam/en-us/docs/100782401d.pdf

You apparently chose to buy used drives that are out of warranty.

Bottom line, you put used desktop SMR drives in a server application. Accept your error and buy enterprise class NAS CMR drives.

 

[winnielinnie]

@TheThomen, are you using TrueNAS Core or SCALE?

If Core, do any of the drives report they support trim?

diskinfo -v ada0 | grep TRIM

Repeat the above for each drive, such as ada1, ada2, ada3, etc.

(You have to run the command as the root user or with sudo.)

 

[TheThomen]

@TheThomen, are you using TrueNAS Core or SCALE?

If Core, do any of the drives report they support trim?

diskinfo -v ada0 | grep TRIM

Repeat the above for each drive, such as ada1, ada2, ada3, etc.

(You have to run the command as the root user or with sudo.)

Hi @winnielinnie

same for all disks

im on core

 

[winnielinnie]

Some SMR drives will report they support "TRIM".

Based on the fact that none of your drives report they support "TRIM", and based on the purchase date of your drives, it's looking unlikely that they are SMR drives.

So perhaps the culprit for this sudden all drives degraded phenomenon is due to the LSI MegaRAID controller card?

 

[TheThomen]

Some SMR drives will report they support "TRIM".

Based on the fact that none of your drives report they support "TRIM", and based on the purchase date of your drives, it's looking unlikely that they are SMR drives.

So perhaps the culprit for this sudden all drives degraded phenomenon is due to the LSI MegaRAID controller card?

like corrupt firmware? should i reflash?

 

[NugentS]

Have you got decent cooling across the card?
Also post the result of zpool status please. Might need a -v in there as well