Pool Degraded after replacing disk with brand new one

[titust1]

Hi everyone,
I use the latest version: TrueNAS-SCALE-22.12.4.2. A few days ago I noticed that the pool got degraded due to errors on a WD disk. I ordered a brand new WD disk same model, and replaced the old one following web directions.
I don't know, maybe I did something wrong, but now the status of my pool is degraded, saying REPLACING see below....
I'm not expert in Shell commands, I thought I can operate the NAS using the GUI, but now I am totally confused.
I don't think this is something very difficult to solve... the pools is functional although degraded, datasets are OK. Last SCRUB didn't find any errors.
Could you guys help me with this.? Please.
What should I do now?
Thanks a lot
Titus

# zpool status -x
all polls are healthy

# zpool status
pool: Z-Pool
state: DEGRADED
scan: scrub repaired 0B in 03:37:25 with 0 errors on Sat Oct 21 14:37:27 2023
config:
NAME STATE READ WRITE CKSUM
Z-Pool DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
221e12ae-18cc-4ba9-b8bb-dc900a78a185 ONLINE 0 0 0
1613cb3c-b16b-400c-8522-cfd16ad3a679 ONLINE 0 0 0
replacing-2 UNAVAIL 0 0 62 insufficient replicas
7ec5cb4b-f45d-404d-b353-983ec843df05 OFFLINE 0 0 0
2dc3d0fb-2284-42ea-bdb0-618daec0d544 FAULTED 0 34 0 too many errors
logs
e308f94e-52dc-46bd-9f32-81aff23148e0 ONLINE 0 0 0

errors: No known data errors
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:02:06 with 0 errors on Thu Oct 19 03:47:08 2023
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sda3 ONLINE 0 0 0

errors: No known data errors

 

[Davvo]

and replaced the old one following web directions.

Define "web directions".

 

[titust1]

Define "web directions".

Disk Replacement

Describes how to replace a disk and restore the hot spare in TrueNAS CORE.

www.truenas.com

 

[Redcoat]

@titust1 , your 40EFAX drives are SMR drives and not suitable for TrueNAS. See the Hardware section in Resources for a list of affected drives and some discussion material on the background.
https://www.truenas.com/community/resources/list-of-known-smr-drives.141/
Don’t know if this is a contributor to your current issue or not.

 

[Davvo]

@titust1 , your 40EFAX drives are SMR drives and not suitable for TrueNAS. See the Hardware section in Resources for a list of affected drives and some discussion material on the background.
https://www.truenas.com/community/resources/list-of-known-smr-drives.141/
Don’t know if this is a contributor to your current issue or not.

It is, as written in the other thread.

Is a faulty disk with 34 errors really DEAD, should I replace it?

I use Truenas SCALE latest version. I noticed that one disk was labeled FAULTY because it had a few errors. I mean 34 errors it does not mean a completely dead drive. Actually this drive was new purchased from Amazon (WD Red 4GB) two weeks ago I can't believe that after two weeks it became...

www.truenas.com

 

[Redcoat]

It is

I don’t see anything clearly indicating that the OP’s problem is due to SMR. IIRC, earlier forum references indicate that some running service hours are required before write speed degrades and we have no time of service indication in these threads so far.
In fact we are “rather short” of info generally.

 

[Davvo]

I don’t see anything clearly indicating that the OP’s problem is due to SMR. IIRC, earlier forum references indicate that some running service hours are required before write speed degrades and we have no time of service indication in these threads so far.
In fact we are “rather short” of info generally.

He was resilvering, worst condition for a WD SMR: constant writes required.

 

[titust1]

@titust1 , your 40EFAX drives are SMR drives and not suitable for TrueNAS. See the Hardware section in Resources for a list of affected drives and some discussion material on the background.
https://www.truenas.com/community/resources/list-of-known-smr-drives.141/
Don’t know if this is a contributor to your current issue or not.

I had an IronWolf 4TB Hard Drive last year, which was more expensive the WD Red, and it was CMR. Despite that it still became faulty with write errors in Truenas. BTW this drive works just fine in Windows. That's when I decided that a more expensive drive does not make it more reliable in Truenas. I can't afford to replace all my Pool disks with CMR disks. Maybe I will find another alternative to Truenas and ZFS.
ZFS was supposed to be the most resilient..., best error correcting..., and reliable system.
I used to be a Solaris sysadmin some time ago, now I'm retired, and I loved ZFS. I was very reliable

 

[Davvo]

I had an IronWolf 4TB Hard Drive last year, which was more expensive the WD Red, and it was CMR. Despite that it still became faulty with write errors in Truenas. That's when I decided that a more expensive drive does not make it more reliable.

A CMR drive is 100% more reliable than a WD SMR one. WD SMR is not compatible with ZFS.

 

[sfatula]

That's when I decided that a more expensive drive does not make it more reliable in Truenas.

Truenas (IX) doesn't write zfs. It's simply openzfs. Same code.

 

[Davvo]

Help me out, please. Where do you see he was resilvering?

In the screenshot the pool is still stuck into "replacing" mode, plus he told us he just finished replacing the drive (and there is a scrub finished in the zpool status). I see this as a resilvering-related issue.

 

[Redcoat]

He was resilvering, worst condition for a WD SMR: constant writes required.

Yes, he must have resilvered, but from the "SMR issues" presented, would it matter on a fresh drive?

 

[Davvo]

Yes, he must have resilvered, but from the "SMR issues" presented, would it matter on a fresh drive?

Yes, because SMR suffers continuous writes such as those that happens during resilvering. The SMR drive was the new one.

 

[Redcoat]

Yes, because SMR suffers continuous writes such as those that happens during resilvering.

Thanks. Do you have a reference on that phenomenon for a fresh drive?

 

[Davvo]

Thanks. Do you have a reference on that phenomenon for a fresh drive?

No, but I have a WD official statement.

On WD Red NAS Drives - Western Digital Corporate Blog

On WD Red NAS Drives - Data - Western Digital Corporate Blog

blog.westerndigital.com

The increased amount of sustained random writes during ZFS resilvering (similar to a rebuild) causes a lack of idle time for DMSMR drives to execute internal data management tasks, resulting in significantly lower performance reported by users. While we work with iXsystems on DMSMR solutions for lower-workload ZFS customers, we currently recommend our CMR-based WD Red drives, including WD Red Pro and the forthcoming WD Red Plus.

 

[Redcoat]

Thaks you, yes, I've seen that of course, and I'm completely convinced the SMR isn't good for TrueNAS, and a vigorous supporter of the "information campaign" here to advise actual and potential SMR users of the issue.

However, this particular stuation on a new drive doesn't seem to "fit the tech" as I have seen it explained here.

e.g.: From @HoneyBadger https://www.truenas.com/community/threads/can-you-make-smr-drives-work.94552/#post-716529 "The SMR issue will only rear its ugly head if/when file deletions occur, and the drives start freeing up space that doesn't align with the SMR "zone" boundaries. If the pool is given sufficient "free time" then the drives can reshingle and be ready for the next set of writes."

And it's also my impression that we've had multiple reports of seemingly successful new TrueNAS installs on SMR HDD's that have, over time, started to display failures that were not immdiately apparent post-TrueNAS install.

My belief is that @titust1 has bigger problems today that just SMR drives.

 

[titust1]

Yes, he must have resilvered, but from the "SMR issues" presented, would it matter on a fresh drive?

Yes I did resilver it

 

[titust1]

No, but I have a WD official statement.

On WD Red NAS Drives - Western Digital Corporate Blog

On WD Red NAS Drives - Data - Western Digital Corporate Blog

blog.westerndigital.com

Thanks a lot Davvo this WD article is precious, I was not aware about this. It comes right from WD mouth, then shame on them.

.......The increased amount of sustained random writes during ZFS resilvering (similar to a rebuild) causes a lack of idle time for DMSMR drives to execute internal data management tasks, resulting in significantly lower performance reported by users. While we work with iXsystems on DMSMR solutions for lower-workload ZFS customers, we currently recommend our CMR-based WD Red drives, including WD Red Pro and the forthcoming WD Red Plus.

LOL. It's not possible to put on the market a drive labeled "for NAS" that is not capable of handlining resilvering... / rebuilding

 

[titust1]

ugly head if/when file deletions occur

My belief is that @titust1 has bigger problems today that just SMR drives.

Like what?

 

[sfatula]

Oh, they (WD) ended up finally acknowledging the issue? Didn't know they did. I remember when they were not officially acknowledging it! Took them quite some time as I recall, and a while to publish an actualy list of drives that were SMR or CMR. That was a bad time for them.