Drive array performance.

[Agent1]

I have been reading the words of wisdom here for a few years and I have been playing with freenas for a few more years than that and I am still a Noob! A Noob to ZFS for sure. I have a couple of real simple questions that need an experienced opinion.

First off I am about to build a FreeNas server for a company that is holding about 500GB of file data and is in need of a new file server. With the budget that they gave me to work with, I could easily build them another Windows File server, but what fun would that be right? My main concerns are reliability and then performance, so.......

1) If I buy 4 SAS 15K 600GB drives, What kind of performance can I expect with 2Xmirrors? I have the budget and the case capacity to use as many drives as I want. I am thinking that multiple VDevs mirrors will out perform raidZ. Thoughts?

2)I am also going to buy a bunch of 3TB SATA drives (4-6), mirror them and place them in a different ZPools. In the event this company starts to require more space. Also Going to store backups and snapshots. Will doing this make a substantial hit on RAM (ARC)? Incidentally I will be running 32GB of ECC RAM.

*the SAS drives will be connected to an Adaptec 6805 and the SATA drives will be attached to the MOBO.

I hope someone can help me clear up the confusion. Thanks.

Mike.

 

[cyberjock]

How many users are you servicing? Depending on their usage patters, 15KRPM drives may be overkill. I've seen several places with 15kRPM drives and they have rather high failure rates. I assume it is because they run HOT. ZFS has a very good caching system. If you are only serving office-type documents more than likely 7200RPM drives are good enough(but it depends on how many users you will have)

How many GB per year is the company creating? If it's not alot you could probably substitute some SSDs in mirrors. I've had great reliability from SSDs as long as I buy Intels. I avoid OCZ at all costs. For you, putting 5-6 Intel SSDs in a RAIDZ or RAIDZ2 may give phenomenal performance and provide the space you need.

I'm not a fan of mirroring. If you put 6 drives in mirrors(3 drive each) and you have a 2 drives fail at the same time in the same mirror you will lose everything. If you had used the same drives in a RAIDZ2 you'd need ANY 3 drives to lose your data.

I'm not sure if the 6805 is compatible with FreeNAS. I'd DEFINITELY find out before you buy it.

 

[Agent1]

Thanks for the response. This server will serve about 75 users. About 20 of them deal with files that are fairly large (CAD Drawings and 3D data Catia, NX Unigraphix and Key Creator). I was under the impression that the SAS drives tend to be more robust than both SATA and SSD's. Do you have any information that supports the failures of the SAS drives. I would love to read up on it.

This company's active file data has increased over the last 4 years from about 40GB to just under 1TB. They recently have done some house cleaning to create more space on a server that is about 8 years old, moving some older data to another server and backed up onto some tapes. They have about 500GB at the moment that they use regularly and want a solution that can be grown for the future.
As far as the difference between RaidZ and RaidZ2 and Mirroring, I was only considering the mirroring due to the fact that they run consistent daily, weekly and monthly backups of all their data and They want the absolute best performance. I have used a variety of SSD's over the last couple of years and they are less than impressive in the terms of durability. The data transfer speeds are out of this world, I agree, but I am not going to place this companies mission critical data on them.
I was certain that I read somewhere that someone was using the 6805 with good success. I will research this more before I order parts today. Are there any other hardware requirements/demands that you would recommend? I will be using a 4core AMD CPU and 32GB Ram for this server.

Thanks again.
Mike.

 

[cyberjock]

How big are these CAD drawings? I'll assume 100MB since I find it doubtful they are much bigger.

I don't have any evidence of SAS versus SSD. I know one company(which refused to be named) upgraded about 700 workstations to SSDs 18 months ago. They chose to buy smaller Intel SSDs vice the ever larger and larger desktop platter based drives. They said that they typically had 2-3 hard drive failures every week. They had a tech that spent alot of time resetting up these machines with new drives because of the typical failure rate. After upgrading all of the machines at the 18 month mark they had seen only 2 failures. The cost of the SSDs themselves paid off as they have downsized their tech support by 1 employee as a result.

I have several machines at home myself. I always hated hard drive failures for their inconvenience mostly. I always kept good backups. Typically I could expect 1-2 hard drive failures per year from my household. Well, after upgrading these same systems to all Intel SSDs I've had no failures since 2009.

In your case, I'm betting that buying something like 8 1TB drives in a RAIDZ2 at 7200RPM will probably meet your needs easily. Unless you have users that are streaming lots of stuff all the time your loading will likely be not much more than a blip on the radar. When opening files you'll load the whole file locally, then only saves will be your load. With 75 users that's just not much load. If you want more performance you could try a ZIL to increase write performance.

Note that I haven't made a FreeNAS server that runs with that many users in a production environment, but my guesstimate would be that even 8x1TB drives would be overkill performance-wise. This isn't bad since you'll have 6TB of storage space available, but it would be great to get that much space for so cheap. You could also go with 2 vdevs for double the performance. Maybe someone else with more experience in this situation can post their experience. RAIDZ/RAIDZ2 may not be the best option(you did mention mirroring after all) because you will have alot of small writes when office documents are saved.

 

[Agent1]

Rarely over 40MB but from time to time they can get over 100MB.
Good info on the SSD upgrades. I am surprised that a company would have 700 Workstations running SAS drives and not SATA drives. Especially if they were not in an array. By nature SAS drives are not designed to run on desktops and workstations performing normal office application tasks. The firmware on SAS drives is focused primarily on IOPs w/ MQD. When Multiple I/O requests are waiting in the queue, They will be sent in groups and processed by the SAS drive the best. SATA drives behave quite differently. But neither compare to SSD's in terms of IOPs. Or in $ per GB! I have been changing out failed SATA drives at an alarming rate in servers for the past year and a half. The people at Seagate and WD RMA dept. know me by name. LOL. I have drawn the assumption that SAS drives are longer lasting than SATA drives for two reasons. 1) I have been slowly adding them to new builds for the last year or so and have yet to have a failure (KNOCK on WOOD!). 2) I can't find a single thread of viable information that would leave me to believe that commercial system builders prefer SATA over SAS in their Server builds for enterprise systems. I could very well be mistaken on this.
I love the idea of having the drive/data redundancy with RaidZ and RaidZ2. If the consensus is that performance will not suffer then I will most certainly try it. I have built a couple of RaidZ boxes for small environments and they are working fine. But they are low load appliances.

Thanks.
Mike.

 

[cyberjock]

It was 700 SSDs, not SAS.

One of my FreeNAS servers has a circa 2008 i3 and it can saturate both of it's Gb LAN connections. Of course, 75 users has a different loading scheme. Your only potential limitation may be the RAIDZ2, but I doubt it unless you are doing lots and lots of disk activity simultaneously.

 

[Agent1]

It was 700 SSDs, not SAS.

Sorry, I misunderstood.

Honestly I don't think that this company has a tremendous amount of simultaneous writes, but they are keeping a lot of small files open and saving them constantly. office docs and pictures are the primary files that are being used here. I guess that I could start out with some SATA WD RED 3TB drives and see if there are any performance problems, then upgrade to the SAS drives and a LSI HBA if there is a need. I hate to start off low end when they give me a "sky's the limit" budget to begin with. I just have a sneaking suspicion that the SAS 15K drives would perform fine and that I wouldn't have to ever touch them. SATA have left a bad taste if you know what I mean.

 

[cyberjock]

What I've seen from working with small businesses is that if you are downright honest with the person that authorizes the money to be spent and say "this WILL for sure work 100%" but this is alot cheaper and should work they'll give you the benefit of the doubt. You can always buy 2 enterprise class Intel SSDs as a "just in case" situation.

I'd love to be in your shoes though.. I really liked working with/for small businesses that let you have an unlimited budget because they trust you to do what's right and not overspend.

 

[cyberjock]

Another option.. build the cheaper high capacity server. If it doesn't work repurpose it as a backup server for other servers and build the more expensive server you want with the 15k drives. I avoid 15k drives personally because they just seem to burn out so fast.

 

[Agent1]

One more quick question.....would you use a HBA such as the LSI 9201-16i or something else for the WD 3TB Red?
I have 8 open ports on the MOBO also......

 

[cyberjock]

Ports are ports as long as you aren't bottlenecking something like the PCI bus. I wouldn't buy controllers unless you need ports, and then I'd only buy enough to give you the ports you need.

 

[Agent1]

UPDATE!!!!! I went with 10 3TB WD Red HDD's and 4 of them are on their way back to NewEgg. Any recommendations on brands of SATA HDD's are they all pretty much junk these days. I am thinking I should have stuck with the SAS idea.

 

[cyberjock]

My friend just bought 10 WD Green WD30EZRX and zero failures after testing them all.

 

[Stephens]

UPDATE!!!!! I went with 10 3TB WD Red HDD's and 4 of them are on their way back to NewEgg. Any recommendations on brands of SATA HDD's are they all pretty much junk these days. I am thinking I should have stuck with the SAS idea.

Any brand/model can fail, and they can fail for many reasons, including how they're packaged, and how they're treated at every step from manufacture to your doorstep (and beyond). I think you get it. The best you can hope for is to test them thorougly when you get them before you put them into use. The only trend I've seen consistently so far is the error rates increase as the drive size increases. I don't feel all that gung-ho about Seagate these days, but they were the most reasonably priced when I got mine (3TB). I had one DOA. I replaced it and all are running fine now. Does that tell me Seagate is bad? Not really. Does it tell me Seagate is good? Again, not really. It tells me HDD margins have been pretty thin and consumers have told HDD manufacturers (with their dollars) they're more interested in inexpensive than reliability (to a point), so the manufacturers have complied. They do what they do and a certain percentage of manufactured drives are lemons. Find out if you have a lemon up front and replace it. I would say the same for ANY HDD. Green, Red, Blue, Black, WD, or Seagate. Admittedly, I did read a lot about how HDD's were manufactured at one point and decided I liked the Samsung HD204UI's. But even that model had a firmware issue which could result in loss of data, and of course Samsung sold their HDD business to Seagate. IMO, trying to predict HDD reliability by brand/model is a crap shot. Buy for the features and price you want, test your drives when you get them, and have parity and backups.

 

[BobCochran]

I agree with Stephens. Be sure you have a stock of spare drives available, too. You'll need to whip them out and replace a bad drive and hope that the replacement does not go bad during resilvering.

If you have 75 users then you should have a company like ixSystems provide the hardware and support for a while, until you have learned the ins and outs.

Bob

 

[Agent1]

If you have 75 users then you should have a company like ixSystems provide the hardware and support for a while, until you have learned the ins and outs.

Bob

Does IXSystems have access to hardware that other people do not? Trust me when I say, I have learned the ins and outs of replacing and supporting hardware.

If there is such a substantial learning curve with configuring Samba shares in freenas for a windows environment, I will just stick to Windows File Services and put the drives on a big hardware raid. Doesn't get much easier than that. Pardon the frustration that I reek of. I have had more HDD failures in the last 24 months then in the previous 15 years.

I don't like the idea of sitting back with one hand on the phone, waiting for some canned system to hiccup so I can spend all day talking on the phone with tech support. I have setup a few of these boxes and they are, for the most part, straight forward. The only failures that I have experienced have been drives. My biggest concern is with reliability. Second is performance. The point made about large drives being less consistent and producing the most errors is a good one. My curiosity about the ZFS file system lead me to this place. The answers I have received have been very helpful.

I am going to exchange the 4 failed drives that I RMA'ed with 4 new Seagate ES2 3TB drives. I am going to create 2X 5 drive Raid-Z2 Volumes. With a watchful eye on the remaining drives. If I start to see degraded zvols I will swap them over to SAS.

I hope you can understand my frustration. I feel like I let a good customer down by second guessing myself. They gave me an open ended budget to provide them with an enterprise file solution and they got a basement bargain weekend project from a garage sale. (not really, other then the hard drives it is quite stout.)

 

[cyberjock]

If there is such a substantial learning curve with configuring Samba shares in freenas for a windows environment, I will just stick to Windows File Services and put the drives on a big hardware raid. Doesn't get much easier than that. Pardon the frustration that I reek of. I have had more HDD failures in the last 24 months then in the previous 15 years.

Then you should consider Windows. FreeBSD has a steep learning curve. It is made very managable with FreeNAS, but on occasion it is necessary to "know your sh*t". If you plan to trust this thing with your data and not figure out the inner workings you may want to go back to Windows.

FreeBSD/FreeNAS isn't for everyone. There's no shame in deciding its not for you. I respect someone that decides to go back to Windows more than someone that tried to get FreeNAS to work and admit he has no intention of figuring it out and just wants it to work. The forum is littered with people that want to be spoonfed. You can only be spoonfed to a certain age. Then you have to learn to feed for yourself.

Honestly, I think it was just a fluke that you had that many failed drives at once. There's too many variables to say that "WD" sucks because of your order. I have 36 WD hard drives in my house right now. 26 of them are more than 2 years old, and 10 are more than 3 years old. I've had one failure in the last 36+ months. Never gotten a bad drive in the mail in the last 36 months. Before WD I had Seagates. I had 1 or 2 fail in about 5 years of using Seagates.

I'd just get more WD reds and give FreeNAS a try. It's not like FreeNAS is the reason for the failures(thank god). If I were building a system for a home user, I always recommend the WD greens. WD reds would be my recommendation for small business applications. You just got a box of drives that were probably dropped or something.

 

[Stephens]

So I handle my hard drive by discharging static, mounting them in cases that have rubber support (absorb vibrations), keep them cool, and try to keep airflow relatively clean.

But Delivery The Man Has Other Ideas.

I also have security cams and I've seen recordings of the same things occurring. Hopefully there is a lot of styrofoam or peanuts or whatever packing material, but stil. I can't even totally blame them because most of the times, the packages aren't marked fragile or anything. So as far as they're concerned, there could be a book inside. And as much as NewEgg has a good retailer reputation, it's only good compared to worse competitors. I've seen NewEgg ship individual drives with ZERO padding. Not to mention they wanted to send me a refurb in exchange for a DOA brand spanking new drive. Anyway, I finally decided to include the possibility of RMA in my life outlook. It was a good day. Previous to that, I wanted to record higher definition video of the delivery trucks, get the license plate, storm down to the office, find out who was driving that truck, wait for him to get off work, follow him home, and waterboard him in his bed. Granted, that might have been taking things a little too far. But only a little.

I really do hope you took/take my advice about doing the SMART long test on each and every drive. This will weed out a significant number of DOA/marginal drive issues.

 

[Agent1]

I appreciate the honesty.

Does the words "I don't like the idea of sitting back with one hand on the phone, waiting for some canned system to hiccup so I can spend all day talking on the phone with tech support." Sound like someone looking to be spoon fed?

I respect your opinion and I am very grateful for the insight you have provided me. Thank you.

I am responsible for hundreds of spindles. I see drives fail every week. I see drives fail more frequently now then in the past. That is a fact.

FreeBSD has a steep learning curve. It is made very managable with FreeNAS, but on occasion it is necessary to "know your sh*t". If you plan to trust this thing with your data and not figure out the inner workings you may want to go back to Windows.

FREENAS is a great GUI for managing Network Attached Storage using the FREEBSD Kernel rather than using CLI, but what pulled me in was ZFS. When it comes to FREEBSD I can hold my own, but it is the inter-workings of ZFS that I am still a little fuzzy on. That is why I am here.

Honestly, I think it was just a fluke that you had that many failed drives at once. There's too many variables to say that "WD" sucks because of your order........I'd just get more WD reds and give FreeNAS a try. It's not like FreeNAS is the reason for the failures(thank god)........ WD reds would be my recommendation for small business applications. You just got a box of drives that were probably dropped or something.

Do yourself a favor, and read the reviews on newegg for the WD 3TB reds. There seems to be about 2 failed drives indicated for every good one. I have not taken the time to look at the 1TB or the 2TB reviews so this might be just a 3TB problem at the moment. IDK. I never said that WD "sucks". I have had great luck with the Cav Black and RE enterprise drives. I am not so naive to believe that every drive that WD makes is going to be reviewed on newegg. I am actually starting to believe that my vendor might be a large part of the problem. Packaging for such a large quantity of drives seems shoddy at best.

Oh, and as for wanting a Freenas server to "Just work", That is exactly what I want and I have gotten just that on enough occasions that I am absolutely certain that I will obtain such results once again.

I didn't mean to turn this thread into a HDD debate so I will leave it at this. You are right, noobsauce80, I am going ahead with freenas and with SATA drives. I will just test them thoroughly.

Thanks to everyone for all your help.

 

[Agent1]

I really do hope you took/take my advice about doing the SMART long test on each and every drive. This will weed out a significant number of DOA/marginal drive issues.

I most certainly will and thanks for the advice.