RAIDZ expansion, best method when increasing disks and replacing with larger at same time

[bstev]

I currently have a 10x6TB in a single RAIDZ2 vdev. I will be replacing the disks with 10TB disks soon and may even add 1 or 2 more disks. I also currently have a spare 12 bay server that I can use to transfer the data if I want to change things. This pool is just a large media storage shared over a home network and a self hosted cloud for a few users. The pool is currently 99% full. Yes I know... didn't you hear the shame when I said 99% full

I understand if I add a drive or two to the RAIDZ then it will not fully utilize that space unless I rewrite all the data, because the old parity is still there too. So this makes me wonder if I add the two drives and then replace the other 10, will it be rewriting all the parity data?

I mean I can also just put all the new drives in the other 12 bay server and copy it all over that way at the cost of taking the pool offline and fixing the sharing configurations

I also see many posts saying not to have 12 drives on a single vdev, so maybe the 12 drives should be configured differently anyway?

 

[danb35]

and may even add 1 or 2 more disks

Not any time soon--this feature doesn't exist in TrueNAS.

 

[NugentS]

Doesn't even exist in ZFS yet. It might do at some point in the future, in the fullness of time, eventually, maybe, perhaps.

 

[Whattteva]

I currently have a 10x6TB in a single RAIDZ2 vdev. I will be replacing the disks with 10TB disks soon and may even add 1 or 2 more disks. I also currently have a spare 12 bay server that I can use to transfer the data if I want to change things. This pool is just a large media storage shared over a home network and a self hosted cloud for a few users. The pool is currently 99% full. Yes I know... didn't you hear the shame when I said 99% full

I understand if I add a drive or two to the RAIDZ then it will not fully utilize that space unless I rewrite all the data, because the old parity is still there too. So this makes me wonder if I add the two drives and then replace the other 10, will it be rewriting all the parity data?

I mean I can also just put all the new drives in the other 12 bay server and copy it all over that way at the cost of taking the pool offline and fixing the sharing configurations

I also see many posts saying not to have 12 drives on a single vdev, so maybe the 12 drives should be configured differently anyway?

This, right here, is why I chose to run striped mirrors. 10-wide vdev and it's 99% full. That'll probably take like a week (more than likely longer) for you to finally see capacity increase and that's assuming you're doing NOTHING with the pool. I don't even want to imagine the length of time it will take upgrading the 10 TB drives later on your next upgrade.

 

[bstev]

I guess I should have researched further. I figured the RAIDZ adding disks I read about earlier in the year had made it main stream into stable releases of truenas already. Sorry for the misunderstanding.

I guess that means I will be swapping the drives one by one and staying with 10 drives.

Thank you for all the quick responses.

 

[NugentS]

For the moment - yes.
I think its a feature that is highly anticipated

 

[Davvo]

If you have the space and a HBA card it could be cheaper to create another vdev with the same disk size.

 

[NugentS]

certainly quicker, if he has the space and HBA. Although he says he has a spare server (I assume with disks) that he can back up to and then trash and reset the pool with more disks. However at 10 wide the OP is approaching the limits of sense for a single vdev and should consider moving to two vdevs instead of one

 

[jgreco]

I guess that means I will be swapping the drives one by one and staying with 10 drives.

As noted previously, ZFS does not support expanding the RAIDZ vdevs.

Yes I know... didn't you hear the shame when I said 99% full

I'm sure there's a lot of people here who feel your pain. No shame in making use of a great technology to its max.

I figured the RAIDZ adding disks I read about earlier in the year had made it main stream into stable releases of truenas already. Sorry for the misunderstanding.

Major ZFS feature additions happen at a glacial pace. We're still waiting for stuff that's been talked about for a decade.

I don't even want to imagine the length of time it will take upgrading the 10 TB drives later on your next upgrade.

I believe it took approximately a day per disk when I bumped our 12TB's up to 14TB's on a 12 disk array that was about 50% full. You can laugh at me though because I think I managed to get through two or three drives before I realized that I should just load the chassis up, as it was a 24-bay chassis. I put the remaining disks in and the remainder finished in like two days or something like that. Which I mention as preface to ...

If you have the space and a HBA card it could be cheaper to create another vdev with the same disk size.

I think it is handy to have another chassis laying around that you can treat as a JBOD expander. I'm getting old and find it ever more difficult to wrestle 4U or 5U gear. If you can add an HBA, and you widen the vdev to 11 or 12, then this makes an easy way to do drive upgrades. Just put all your new disks in the JBOD, select them as replacements for each old disk in the pool, and let ZFS go to town. You cannot WIDEN the vdev with this strategy, but for bulk storage, 12 drives is about as wide as you want to go, so setting that as a parameter up front in the design phase is useful.

 

[Etorix]

I guess that means I will be swapping the drives one by one and staying with 10 drives.

Correct. If you're upgrading the existing you're stuck with a 10-wide raidz2 for ever (or until raidz expansion becomes release, whichever comes first). But, if this is in a 12-bay server ("may add 1 or 2 disks"), you can plug in (up to) two new 10 TB drives, replace existing drives while they are still in the array and then remove the replaced drives; repeat until all 6 TB drives are out. This is safer than taking drives out one by one, plugging a replacement in and letting resilver.

10-12 is about the practical upper limit for reasonably safe vdevs. You might consider a 12-wide raidz2 if you have a solid backup, otherwise two 6-wide vdevs would be preferred (same capacity as one 10-wide but higher performance and resiliency).
To change vdev layout the only procedure is "backup-destroy-restore". In your case, with a spare server, that would be: Set up a new NAS in the spare and replicate from old to new.

 

[jgreco]

You might consider a 12-wide raidz2 if you have a solid backup, otherwise two 6-wide vdevs would be preferred (same capacity as one 10-wide but higher performance and resiliency).

Or an 11-wide RAIDZ3 if you're paranoid about failures. This also leaves room for a warm spare (#12).

To change vdev layout the only procedure is "backup-destroy-restore". In your case, with a spare server, that would be: Set up a new NAS in the spare and replicate from old to new.

Not really the only procedure. A variation on your suggestion: It would be VERY much faster to use the spare server as a JBOD if you can, creating a temporary "new" pool with the new vdev layout, and then using local replication to copy the old pool to the new pool. Only one server involved and no network traversal.

 

[bstev]

Thank you all for your help and great info. A year ago I swapped from 4TB SATA to 6TB SAS and between the stress testing on the used 6TB drives and resilvering, it took a week or more. I used a spare bay to have one 6TB drive doing the stress tests while the resilvering was happening on another drive.

Luckly these 10TB drives I am getting come out of a working TrueNAS environment that regularly does full disk checks already. I started the first resilver last night and it completed today after about 22 hours. I am on my 2nd disk now.

For a few years now I have been happily using a R720xd running ESXi with 192GB RAM, 12x3.5"(10raidz2 and a mirror), 2x2.5"(slogs), and 2 NVMe(mirror for VM's). I am replacing it with a R720 8x3.5" bay plus MD1200 that was priced too nice to pass up. Gains me a few extra bays, plus the onboard sata ports (dell disables them on the xd model) to run datastore with TrueNAS on. Adds one more PCIe and a the piece of mind is when I power down the server to change up hardware then my spinners get to stay spinning on the MD1200. The slight down side is 50watts more energy between the MD1200 with no disks and the added H200e card to the R720.

The replacement R720 is in testing currently with TrueNAS 13 and giving me an issue after running for 24 hours or so. I am going to start a new thread on it here shortly.

 

[Davvo]

You wouldn't be the first with issues with those cards: just recently we had a thread about it, and iirc should still be an open question.

Here.

Dell R730xd slow IOPS/transfer rate with any drive

Hello all, I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon...

www.truenas.com

 

[bstev]

You wouldn't be the first with issues with those cards: just recently we had a thread about it, and iirc should still be an open question.

Here.

Dell R730xd slow IOPS/transfer rate with any drive

Hello all, I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon...

www.truenas.com

I was thinking the resilver speed was about as expected for running 7200rpm HDD's on this vdev. The 2nd drive is 50% complete and running at 785M/s. The HBA on this box handling this vdev is a LSI 9207-8i.