Certified Refurb drives: Horrible idea?

[Cicatrize]

I'm currently using 8x1TB Samsung/Seagate HD103SJ, and have been for many years without issues, but I'm looking to get some new drives to get a bigger pool and also set up a striped mirror. These drives are used on an LSI 9211-8i passed through ESXi to a Freenas 9.10 VM and also used as an iSCSI datastore for my other VMs.

I'm looking at these:
https://www.amazon.com/HGST-Ultrast...01LYVD7ME/ref=cm_cr_arp_d_product_top?ie=UTF8

I have a hard time believing a hard drive can be truly "refurbished". I don't see how it's possible...but I have been wrong once or twice in my life. (My wife would say more, but I disagree)

That being said, is this a generally terrible idea to use these for a home lab? If it's worth it, I have no problem spending the extra cash to get WD Reds (or whatever, within reason), but at only $70 a piece with great ratings, these refurbs are extremely enticing. With 6 drives, that's a savings of about $200.

 

[Jailer]

With the price of those drives not being much less than a new drive I would not go with the refurbs. There have been other members that have gone with white label drives that have suffered failures after a short period of time.

If this were a test system only I might be inclined to say go for it if they were cheaper. But at that price point save yourself the headache and buy new drives.

 

[Cicatrize]

With the price of those drives not being much less than a new drive I would not go with the refurbs. There have been other members that have gone with white label drives that have suffered failures after a short period of time.

If this were a test system only I might be inclined to say go for it if they were cheaper. But at that price point save yourself the headache and buy new drives.

Well, it is $40/drive more ($70 vs $110 for WD Reds), but I see what you're saying. Maybe I'll just go with some greens or something. There's a lot of stuff out there saying you should use NAS drives and blah, blah, but these 7200RPM Seagates have been plugging away nicely for almost 6 years. Well, the original Samsung ones, at least. The Seagate ones are a couple years newer.

 

[Cicatrize]

Interestingly enough, the green 3TBs are actually more expensive than the reds on Amazon, lol.

 

[Mr_N]

Dont use WD Greens lol, or Seagate 3TB's while your at it!

4TB or 6TB drives will probably be cheapest $/GB currently

 

[Cicatrize]

Dont use WD Greens lol, or Seagate 3TB's while your at it!

Why not?

Sent from my Nexus 6P using Tapatalk

 

[Mr_N]

Seagate 3TB drives worst in last 15 yrs for failures, and Green drives have some head parking thing which is bad for NAS but i'm sure you can find more info here or on google :)

 

[Jailer]

Seagate 3TB drives worst in last 15 yrs for failures, and Green drives have some head parking thing which is bad for NAS but i'm sure you can find more info here or on google :)

Agreed on the 3TB Seagates but the head parking issue can be mitigated on Greens with wdidle3.

 

[R.G.]

I just wrestled with this question myself. I have a main FreeNAS system ("Tank") that I use only for backups on the various machines in the house. I have a secondary FreeNAS system ("Pool") that I pressed into service as insurance against ransomware.

Pool's storage array is made from 9 refurb 750G drives that I bought cheaply when I was first setting up FreeNAS and getting past my personal "infant mortality" issues with a new software/server system. I originally bought 10 of these refurbs. One of the refurbs died about 500 hours into the first system's life, and I learned to find and replace HDDs on FreeNAS with that. The rest of the refurbs have worked without a squawk, until recently.

I always considered the refurbs to be marginal, so when I constructed Tank I built from only brand new parts, including seven new 3TB drives of scattered brands to avoid "defect clustering". All well and good.

But I've lost two of the original 3tB drives in Tank as compared to only one of the 10 original drives in Pool. And of course, my numbers are so small that none of this is statistically significant.

I think that what is significant is that I have a FreeNAS server running RaidZ3 on Tank, so I'd have to lose three drives before I could refill one drive to be at risk on the main back up system (Tank). And Pool is running in RaidZ3 as well, so I'd have to lose three drives in "Pool" before I would lose all of Pool, which backs up Tank.

I'm running out of space on Pool. six 0.75TB drives is clearly not equal to four 3TB drives on Tank. So I had to decide how to fix Pool's storage size problem.

What I wanted was eight new 3TB or 4TB drives. So I read a LOT on where the HDD industry is now. I also found that the HGST Ultrastar 3TB is coming off line in many storage venues and the inventory being flushed. A certified refurb 7K4000 with 3TB and a year warranty can be had for as little as $60. That makes a full " Pool" be $420, not $600-$700. After a long dithering period, I decided to go for refurbs, based on:
- the HGST Ultrastar 3TB shows failure rates of 1-2% in the Backblaze reports; it's a good overal drive.
- the failure rate on drives over the 1.5-3 year sweet spot runs to about 10%. Statistically, I could expect one or two failures in the first year on the refurb array. And if/when that happens, I have time to react and fix it before data loss.
I think that makes saving the money on refurbs acceptable, especially since I saved enough to buy a couple of new/upgrade drives to stick in when something fails.

This kind of reasoning is one of the beauties of FreeNAS. I would ..never.. trust my primary backup to refurbs if a single failure lost it all. Never. But I can pay the extra two or three drives up front and be pretty certain that I can save the data. That's the reasoning on the front-line system, Tank. For Pool, I can pay a minimal $180 over the cost of basic storage bits and get three-failures assurity on Pool, as well as Tank having to have previously failed. I think that means that I'd have to lose seven out of fourteen drives in Tank and Pool >AND< have a failure in the main system that T&P back up.

So I think I'm ok with certified refurbs, especially with 1Y warranty on them.

As an aside, the warranty and description on refurbs is a snakes' pit. It is very -very- difficult to get a straight answer on what warranty a refurb, certified refurb, or manufacturer-certified refurb drive carries. Frys is clear enough: they offered the HGST 7K4000s with a 90 day warranty. Amazon has a lot of vendors for similar drives, but mostly they don't say what the warranty is, other than Amazon makes their vendors supply at least 90 days. Newegg is much the same. On Amazon, there is some quirk of their vendor info system that lets you select the "available from other vendors from $xxxx new and used" and listed under "new" and have them still be the same old refurbs with unspecified warranty.

I found an outfit that clearly stated a 1 year warranty on these particular refurbs, and paid an extra $5 in asking price per drive to get it.

We'll see.

So - use refurbs or not? It depends on your storage strategy, and how you are willing to spend the money. I put a lot of worrying into how and what would have to fail before data loss, and reached my risk threshold. It's different for you, and your backup strategy, and the number of failures you're willing to risk.

 

[Cicatrize]

I forgot about this thread. Best Buy was running a sale last week on (retail) WD Red 4TB disks for $120 each, so I bought five of them, and they'll be here tomorrow. I decided to just play it safe and not risk the refurbs, especially with the price I got on the 4TB drives. I'm going to use my current JBOD array in a second FreeNAS system as a backup to my main drives. Still trying to figure out a second backup solution for the most important stuff...maybe cloud based.

 

[ewhac]

I found an outfit that clearly stated a 1 year warranty on these particular refurbs, and paid an extra $5 in asking price per drive to get it.

Care to name it?

 

[R.G.]

Sure. I got them through goharddrive on ebay. That seems too simple to be worth mentioning, but the actual story is complex.

Refurb hard drive sales are a snake pit, as I mentioned. There is a group of companies that bid on off-lease or replacement equipment. These companies either resell themselves, or farm the bits and pieces off to other sales companies. At the sales level, there are various degrees of "refurbishing' done, from as little to "does it power up" to "run all mfg tests and issue a warranty". Then there's how MUCH warranty, and whether the terms are like insurance viewed as a game: can the company make it so difficult for you to claim the prize that you give up?

I ran into GHD on Amazon, ebay, and their own site. Prices, quantity available, and shipping were different on each venue. That sounds shady, but GHD was much LESS variable than the other sellers. And they had thousands of reviews indicating good results and standing behind replacement. Other sellers got various degrees of "scam" rating. It is ...very... difficult to find a refurb seller that actually states a warranty as I mentioned above. GHD actually stated both the lowest price (on the same drive!) and the longest warranty (one year) on ebay. I have friends that sell on ebay, and ebay maintains some very strict rules for customer support on repeat sellers, so there is some reason to believe that GHD is in it for the longer haul, and may stand behind what they say.

Of course, my mileage may vary too. This could have been a total waste of money. In effect, I'm betting $60 a drive that I'll get more than one year of use out of each drive. We'll see. The drives are supposed to show up tomorrow.

 

[farmerpling2]

It's a crap shoot...

If these are data center drives (higher quality the consumer drives), that is a plus. In my experience data center drives last longer and are better built. They also have a premium price. You generally see fewer failures as compared to consumer / pro-consumer drives.

At a minimum, you want to know the power cycling count, run time, and error rate(s) (mainly bad blocks). Bad blocks are going to happen. Just the way the technology is. They also tend to happen in clusters, not just one bad block.

I am not a big believer in SMART technology. I have seen too many failures where SMART did not find the fault(s) in advance. SMART is just a tool to help provide information and is far from perfect. The Google disk drive doc on failures also says this, to some extent.

RISK/REWARD

On newegg.com, I have read the reviews of people who have bought some of these drives. It seems like the drives were running around 3 years, always on and very low count of power cycling - a sign of a commercial data center. These drives usually had a 5 year warranty and the data center were hedging their bets to reduce failure rate by replacing them. Not sure if the warranty was transferable, though. The sellers were offering any where from 90 days to 1 year warranty.

Why would a data center get rid of drives that were working well, with two-fifth of the warranty still available, is my question? Do they know something that I don't? A company wants to squeeze as much out of their dollar as they can.

Would you sell a disk drive at year 3 of a 5 year warranty and "lose" money in doing so? If the drives had a great service life for 5-7 years, you would let them go until the failure rate was more than the risk of keeping them.

Well, there is more to the story then just warranty...

DENSITY
It is also possible that the data center needed to increase the storage space and the cost of "retiring" 3TB drives and replacing them with 6TB or higher was cheaper than building on to their data center. The cost of TB per square feet is important in data center planning.

POWER SAVINGS
How much more power does it cost to migrate from a 3TB drive to 6TB drive? Not a lot and in some cases maybe less. A cabinet that has 96 - 3TB drives migrates to 96 - 6TB drives provides double the space at little to no additional cost in power, A/C, and building management. Providing the infrastructure did not require any upgrading, the cost is possibly just the new 6TB drives.

If they had to add a bunch of of new storage racks, there is the cost of floor space, providing power to new racks by having electricians run new bus bars, possibly having to upgrade A/C, which gets VERY expensive and uses more power, etc.

SO THE ANSWER IS...

Complicated

If you can find out a couple answers, you can make an intelligent decision.

1. What replaced these used drives? 3TB being replaced by 6TB?
2. Has there been an increases failure rate after certain run time for these drives?
3. What type of tests were run the drives to call them refurbished? SMART short? SMART LONG? Manufacturers testing tools?
4. How much of the manufacturers warranty is left on drive and does it transfer?

If question 2 is the reason why (and they will likely not tell you because they want your money), do not buy.

If question 1 is the reason, then question 3 and 4 are useful in helping solidify your decision making process.

Would I buy them?

• For a production, I would be VERY cautious - OK, not cautious, just plain NO.
• For a home usage using Z3 and good backups, maybe. Z2 or Z1 - NO.
• For a test bed, I would be willing to buy them if the price is right.

I hope this has been helpful.

 

[R.G.]

It most definitely is a crap shoot. But the odds on any given throw are not so easy to work out. I'm still mildly astonished that I've had more unit failures and more unit failures per power-on-hour from the all-new drive setup: two out of seven versus one out of nine. [discounting the one my gentle drive skills killed in the original setup of the refurbs.] And the refurbs were old when I got them. Perhaps all the infants were already dead.

I think there are some simple metrics: for production use, if you selected refurbs and lost something, you're fired. If you bought new, that's much harder to pin on you. It's an old variant of "nobody every got fired for buying IBM". If your bean counter made you buy cheaper new drives, you may be able to get -him- fired.

I'll certainly run short/long/conveyance/badblocks/thermal/etc tests on the refurbs when I get them on line. Since I last posted here, I've been in the hospital twice, so I'm not far along on the testing.But you're right - know your data and know what your likely failures will be and how many you can stand
I run tank (first line of defense, all new) in RAIDZ3. I run pool (tank's backup) in RAIDZ3 as well.

As it sits, pool is still humming. But it's running out of space. Gotta do something. Being too full of bits is 100% fatal for a drive. :) No soggy statistics needed.

 

[farmerpling2]

I have started a document on where TLR came from, why you want it in NAS drives, if you care about your data. It will help provide some thoughts on why you should not use "desktop" drives (i.e. blue, green, black, etc.) in a NAS configuration if you care about your data.

Saying ZFS will write it out to disk sooner or later is simplistic, once you understand what might occur and cause loss of your ZFS data.

I hope to have this done i the couple days. I tend to look a RAID/NAS from commercial aspect, FWIW.

A kind suggestion...

Download the manufacturers diagnostics and run a full disk erase and then a disk verification. This will take a LONG time, but it will help flush some of the bad blocks out on writes, so they get re-vectored to good reserved blocks.

 

[R.G.]

Thank you for your kind suggestion. It got to me a little late, as I've already kicked this off on the drives, but it may help folks here.

The specific drives I'm using here are HGST enterprise drives, so the manufacturer utilities are easier to find. That's not true for other WD drives (WD "ate" HGST in some mega-deal, and can be counted on to subsume the high quality of HGST drives down into the average of WD production over time. )

On TLER:
That would be a good intro for folks that are not familiar with the issues, or simply haven't run into the before. I encourage you to write this up and get it presented.

Yes, TLER is a good thing in NAS setups. Not having TLER leads to whole drives being marked "bad" because of an issue causing an undeterined-ly long error response to one write, as opposed to the disk actually hard-failing. Not having TLER promotes a delayed or hung sector write into a disk loss. This was one of the problems that made RAID5 be the data disaster it is/was; the other being the write-hole.

And even in a RAIDZ2 or Z3, you're losing a whole disk from your two- or three-disk loss margin, eating up much of the margin you paid for.

TLER was a big issue here on the FreeNAS forums a few years back. Some of the makers' disks could be set to TLER with firmware upgrade setters of one form or another. I vaguely remember WD trying to make enabling TLER impossible so as to force people to their higher-priced "NAS rated" drives.

1. Yes, TLER is an issue.
2. You may or may not be able to get/set TLER on on desktop drives.
3. Not having TLER is somewhere between an annoyance and an exposure in ZFS systems.

As with certified refurb disks, you pays your money and you'd darn well better have computed out the chances you're taking with your data and money. A bright spot on my personal refurb buy is that these drives are HGST enterprise drives and come with TLER already enabled, that being one of the hallmarks of NAS and enterprise class drives.

I can appreciate your looking at NAS from a commercial viewpoint. I tend to look at the whole mess from a systems-architecture viewpoint, which is my professional background.

 

[farmerpling2]

Thank you for your kind suggestion. It got to me a little late, as I've already kicked this off on the drives, but it may help folks here.

You did a great job summarizing what I was going to say.

I am just going to provide some history and perspective along with some technical thoughts.

Well done!

 

[R.G.]

Go for it! It needs said, and needs put where people who are trying hard to learn can find it. I hate to think of the hours I've spent digging out the whole TLER mess.

As a bit of perspective, I ran into this back when people really thought that Redundant Arrays of Inexpensive Disks [RAIDs] could be make up out of whatever disks you have lying about and have something useful. A friend was working for a Four-Letter-Computer-Company at the time, supporting their RAID products, and he was in the habit of commiserating with me over beer over the latest customers to have lost huge chunks of data. He was in on or around the findings of TLER and write holes (as well as users deciding to mix up the striping, etc. ) as it was discovered. I'd go read and try to discover things he could use before getting onto a plane to try to help the most recent furious customer. Got a lot of beer out of it too.

It needs written.

 

[farmerpling2]

Here is an example of a drive that says it PASSED, but all indications from the SMART Attributes Data section tell you this drive must have failed 3 or 4 times already.

If it fails tomorrow, I will not be surprised, but smart has been showing the seek errors/command times outs, etc. I surmise that there is a failure mapping the REVISION 10 smart data.

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.12
Device Model:     ST3500418AS
Serial Number:    9VMJLT4G
LU WWN Device Id: 5 000c50 026bc0d3d
Firmware Version: CC49
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Wed Mar 29 02:31:04 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  600) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (  86) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x103f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       228270229
  3 Spin_Up_Time            0x0003   098   097   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   093   093   020    Old_age   Always       -       7615
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   086   060   030    Pre-fail  Always       -       462786198
  9 Power_On_Hours          0x0032   070   070   000    Old_age   Always       -       26401
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   097   097   020    Old_age   Always       -       3428
183 Runtime_Bad_Block       0x0032   001   001   000    Old_age   Always       -       941
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   029   000    Old_age   Always       -       270587265263
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   073   051   045    Old_age   Always       -       27 (Min/Max 20/37)
194 Temperature_Celsius     0x0022   027   049   000    Old_age   Always       -       27 (0 13 0 0 0)
195 Hardware_ECC_Recovered  0x001a   047   021   000    Old_age   Always       -       228270229
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   197   000    Old_age   Always       -       34
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       34015 (10 74 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1511690101
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2292748319

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     26348         -