Scrub times

[Wolfeman0101]

I have a scrub scheduled to run every 2 weeks. It takes almost 24 hours to finish and my box is so slow while it runs. I have 6 x 2TB in RAIDZ2.

Is this a normal time expected?

 

[ProtoSD]

It sounds reasonable, depends on how full your pool is also.

I have 5 x 2TB RaidZ2, it takes 12-15 hours.

 

[Wolfeman0101]

It sounds reasonable, depends on how full your pool is also.

I have 5 x 2TB RaidZ2, it takes 12-15 hours.

When it's running my data transfer rates are so awful and I even have issues getting onto the OS. Like SSH can take 1 min just to bring up the login. Is that normal?

 

[ProtoSD]

When it's running my data transfer rates are so awful and I even have issues getting onto the OS. Like SSH can take 1 min just to bring up the login. Is that normal?

Yes, I suppose if your CPU has a little more power it would be better, not sure how much, but it's normal.

 

[StephenFry]

Wolfeman0101 - I'd be interested in some other specs of your system (RAM, CPU). I have the same 6x2TB Z2 and while no cause for concern, would find 24 hours a bit on the long side unless the drives are absolutely stuffed.

 

[Stephens]

I was going to say it sounds long too, but then I thought about my own system. It has gone from 8 hours up to 12 hours, and I still have 3TB+ free (out of ~7.2TB usable) in the array (6x2TB RAIDZ2). And of course the more data you have, the longer it takes -- especially if it has to resilver anything. Also, drives work slower as you fill them up. So I can see 24 hours happening, depending. I'll have to make sure my scrubs occur on weekends as opposed to currently I just have to make sure they occur overnight.

 

[raidflex]

I have 8x2TB drives with using 7TB (about 60%) and it takes about 5 hours for a scrub.

My guess is the CPU and memory is holding it back.

Also I had an issue with very long scrub times, much worse then yours about 40hrs. Come to find out that my Areca RAID card that I was using in JBOD mode was the cause.

system:

Intel Motherboard
Intel G530 CPU
16GB DDR3
8x2TB drives
IBM m1015 HBA

 

[Wolfeman0101]

I'm at about 70% full. It took > 29 hours for the scrub to run. Here are my specs:

AMD FX(tm)-4100 Quad-Core Processor
16GB DDR3 RAM
Onboard controller

Code:

[root@nibbler] ~# zpool status -v
  pool: Vol1
 state: ONLINE
  scan: scrub repaired 0 in 29h21m with 0 errors on Mon Aug 27 05:22:19 2012
config:

        NAME        STATE     READ WRITE CKSUM
        Vol1        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada4p2  ONLINE       0     0     0
            ada5p2  ONLINE       0     0     0

errors: No known data errors
[root@nibbler] ~#

 

[cyberjock]

My zpool is 2 vdevs, 8x1.5TB and 8x1TB with both as RAIDZ2. I have tested a failure in each vdev simultaneously and resilvering took less than 9 hours with the zpool about 75% full.

 

[ProtoSD]

Yes, but you didn't tell us about your amped up octicore cpu and loads of RAM! ;) The rest of us aren't so lucky!

 

[cyberjock]

Yes, but you didn't tell us about your amped up octicore cpu and loads of RAM! ;) The rest of us aren't so lucky!

LOL. My FreeNAS server has 16GB of RAM and an i3-530. It was some spare parts I had lying around. LOL. Never underestimate the i3s! Even during a scrub the system is completely responsive. CPU usage never goes about 50% during scrubs :)

 

[Wolfeman0101]

I didn't want to start a new thread but how can I schedule a scrub from the GUI for the first Sunday of every month?

 

[Stephens]

I didn't want to start a new thread but how can I schedule a scrub from the GUI for the first Sunday of every month?

This has already been covered here at the forum several times.

 

[Wolfeman0101]

This has already been covered here at the forum several times.

Sorry I'll look.

 

[Wolfeman0101]

So I'm bumping my own thread because I don't see a need for a new one. While a scrub is running should I see really poor performance from my disks? Like even getting in on SSH takes a good time longer. Also how often should I scrub?

 

[Yell]

Run zpool scrub on a regular basis to identify data integrity problems. If you have consumer-quality drives, consider a weekly scrubbing schedule. If you have datacenter-quality drives, consider a monthly scrubbing schedule. You should also run a scrub prior to replacing devices or temporarily reducing a pool's redundancy to ensure that all devices are currently operational.

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

 

[cyberjock]

So I'm bumping my own thread because I don't see a need for a new one. While a scrub is running should I see really poor performance from my disks? Like even getting in on SSH takes a good time longer. Also how often should I scrub?

I'm not a fan of AMDs, so I can't vouch for how good or bad that processor is for FreeNAS. But if your CPU isn't very powerful you should absolutely expect a scrub to kill your performance. http://www.cpubenchmark.net/cpu.php?cpu=AMD+FX-4100+Quad-Core shows that an i7 2600, which is 2 years old has more than double the benchmarked performance.

You could have a hard drive that is starting to fail. Try running SMART tests on all of your hard disks. If they all pass I'd say a new CPU is in order.

 

[Wolfeman0101]

I'm not a fan of AMDs, so I can't vouch for how good or bad that processor is for FreeNAS. But if your CPU isn't very powerful you should absolutely expect a scrub to kill your performance. http://www.cpubenchmark.net/cpu.php?cpu=AMD+FX-4100+Quad-Core shows that an i7 2600, which is 2 years old has more than double the benchmarked performance.

You could have a hard drive that is starting to fail. Try running SMART tests on all of your hard disks. If they all pass I'd say a new CPU is in order.

I run a SMART test nightly on all 6 drives, no errors. Here is a top while my scrub is running right now.

 

[Yell]

As far as i know a scrub touches each block in "linear-time" order.
-> the first block that was written to disk is read/checked first, then the next block that was written

Do you have an really old (many snapshots created/deleted?) that over 70% full?


You can try to lower the ATA queue size:

FreeNas 3.8.1 defaults:

Code:

vfs.zfs.vdev.min_pending: 4                                                     
vfs.zfs.vdev.max_pending: 10

Change the max to the min value.

 

[titan_rw]

A scrub really hits the system hard. All the sata channels are going to be saturated, resources permitted. And it does take quite a bit of cpu.

I've got 10 drives in raidz3. The drives are 3tb. About 6tb used (33%). CPU is a i5-3570, true quad core, 3.4ghz. 32gig ram. The scrub only takes 2h or so, but it uses 50% of the cpu while it's running. It scrubs at around 900 mb/sec. System is still fully responsive though. No problems streaming stuff off of it, and no laggy ssh sessions.