Which RAID configuration?

[WhiteTiger]

My motherboard supports 7 SATA drives and the case supports 5HDD and 2SDD.
Therefore I will mount an SSD 120GB for booting and mirroring it.

I already have 3 x 3GB WD Red HDDs.
I wonder if I should take a second WD Red drive and create a RAID 10 or take two and manage another RAID configuration.

I can go up to 16GB RAM, the NAS will be for a small office with 5 or 6 users maximum.

 

[sretalla]

RAID 10

There is no RAID 10 in ZFS.

You could add another VDEV and have 2 mirrored pairs in your pool which would be the equivalent... do you envisage the workload being heavy on IO or are the users working with just a few large files (maybe RAIDZ would be better for that.

More RAM is always better with FreeNAS

 

[WhiteTiger]

There is no RAID 10 in ZFS.
...
More RAM is always better with FreeNAS

More than 16GB RAM I cannot install
I also thought of a second configuration:
- 2 SSDs for booting
- 4 WD Red HDDs for work (with RAID5 or RAIDZ-2?)
- 1 HDD 3 / 4TB for an internal backup of the most important folders and for Sync with Dropbox.

 

[jenksdrummer]

There is no RAID 10 in ZFS.

You could add another VDEV and have 2 mirrored pairs in your pool which would be the equivalent... do you envisage the workload being heavy on IO or are the users working with just a few large files (maybe RAIDZ would be better for that.

More RAM is always better with FreeNAS

Literally stating what RAID 10 is, after saying there is no such thing...

 

[jenksdrummer]

More than 16GB RAM I cannot install
I also thought of a second configuration:
- 2 SSDs for booting
- 4 WD Red HDDs for work (with RAID5 or RAIDZ-2?)
- 1 HDD 3 / 4TB for an internal backup of the most important folders and for Sync with Dropbox.

4 disk Z2 is equal in space as a z10; but Z2 has much more overhead with writing data, and to some extent with reading data. Depends on your workload.

Those SSDs are being wasted for booting; when you can use them for caching (L2ARC) or special/dedup VDEVs, or as a SLOG/ZIL if you want to do sync writes. Same with that last HDD slot; one could use that with another SSD for some of those functions (L2ARC that one SSD, the other pair use for special VDEV), or use that as a boot drive

I'd do 2x mirror VDEV with the WD disks; and TBH, I would find a decent pair of USB sticks to boot from and just make sure the system dataset is assigned to the WD Pool, and during install opt to not create swap on the USB Sticks; you don't have a ton of ports/bays to use; I'm in the same boat with one of my builds and did similar. 4x data SATA, 2x SSD special and 1x NVMe M.2 L2ARC; with 2x32GB USB boot. Runs fine...

 

[sretalla]

Literally stating what RAID 10 is, after saying there is no such thing...

I don't appreciate the tone. My comments are intended to be helpful.

Although you can consider a pool of mirrors to be "equivalent" to RAID10, it's still not actually the same thing.

Until Top Level VDEV removal was introduced, that was a key difference in not being able to remove a mirror without destroying the pool. I agree that equivalency is closer now. It's still not there for other levels like RAID 50 or 60. https://openzfs.github.io/openzfs-docs/man/8/zpool-remove.8.html

 

[joeschmuck]

My motherboard supports 7 SATA drives and the case supports 5HDD and 2SDD.
Therefore I will mount an SSD 120GB for booting and mirroring it.

I already have 3 x 3GB WD Red HDDs.
I wonder if I should take a second WD Red drive and create a RAID 10 or take two and manage another RAID configuration.

I can go up to 16GB RAM, the NAS will be for a small office with 5 or 6 users maximum.

What is your hardware?

Why mirror a SSD boot device? It's a waste of a SSD in my opinion and of a SATA port.
What is the purpose of the NAS? Specifically how will it be used? Backups, Users working on live data, database, etc...? Or is it just a simple backup?
How much storage are you needing? Now double that size at a minimum because you will need more to maintain a healthy pool. How important is the data? Very important to critical means you need to make a resilient pool structure or HA servers.

Why do I ask? because depending on what you are doing will depend on the hardware and configuration required to make it work properly and fast enough. You just can't throw together a NAS and load it up with files and share those files, odds are you will run into problems. If it's just a backup server then you should be fine.

 

[WhiteTiger]

I'm confused.
On this topic I have read several pages on the Internet, here and in other forums, and the vast majority of opinions was that SSD is better than USB stick and that the boot disk mirror provided greater security.
I asked a question also here.

I have two NAS with the same characteristics: motherboard ASRock FM2A85X-ITX; CPU AMD FM2 A4 series with 2 cores; 16GB RAM, 7 SATA slot.
These are installed in two different offices, both small with 4/6 people working mainly with MS Office.
In one office it will be used mainly for backups of what is on PCs and in another it will be used for central file systems and backups.
In both cases FreeNAS will make a copy of the data on an external backup service and synchronize it on a Dropbox and GDrive account.

I have yet to decide which drives to mount.
I have 3 WD Red 3TB HDDs, I can't find a fourth drive compatible with these three.
I have also 2 WD Green 3TB HDDs that I would only use for internal backup.
The other HDDs I have yet to buy them.

The NAS are turned off in the evening and turned on in the morning.

I can also consider booting from a USB stick, however I don't work in these offices and if the USB stick fails, employees need to have a spare second USB stick to replace it quickly.
How can I keep them up to date without being present in those offices? remotely?

 

[Ericloewe]

USB flash drives are indeed a poor choice for most, but mirrored SSDs are also a bit on the side of overkill. It's all about how risk-averse you are when it comes to downtime and how much you're willing to pay for it. If the extra cost for the boot mirror is acceptable, go for it.

 

[HoneyBadger]

In the described scenario, where these units are going to be placed in ROBOs (remote office/branch office) and used to hold business data, I'd argue that mirrored boot devices are an excellent hedge against a potential downtime-causing scenario.

(With that said, I'd also advocate for ECC memory here and possibly Intel over AMD.)

The NAS are turned off in the evening and turned on in the morning.

I'd suggest leaving it on, and you can have your backups, scrubs, snapshots, or other maintenance operations run during the off-hours.

 

[joeschmuck]

I'm confused.
On this topic I have read several pages on the Internet, here and in other forums, and the vast majority of opinions was that SSD is better than USB stick and that the boot disk mirror provided greater security.
I asked a question also here.

I read that thread and what you asked and you were giving correct answers. What you didn't ask was should you mirror the boot drive if it were a SSD. So a SSD has a lot of resiliency and thus you do not need to mirror it unless you are going to use a true RAID card to mirror the drives in order to promote a true failover boot option. That does not exist with a software RAID (both FreeNAS and most motherboards). Odds are that if you have a failure of the primary SSD (configured in the BIOS as the primary boot drive) that your system would fail to boot until you reconfigured the BIOS to boot from the second SSD as now the primary boor drive. When a SSD fails, it's not like it is unplugged, the motherboard typically still sees it so it will still try to boot from it. If data is corrupt then the computer just stops working.

So a single SSD is good enough and if you do have a second SSD, you should just keep it ready as a spare and install it if you ever need to. Recovery is very easy if you keep a copy of the configuration file, or slightly more involved if you have to pull the configuration file from the pool. So my advice is to use a single SSD as the boot drive.

I can also consider booting from a USB stick, however I don't work in these offices and if the USB stick fails, employees need to have a spare second USB stick to replace it quickly.

Don't do it. Stick with the SSD, you will be much happier you did as USB Flash Drives fail more frequently and you never want to troubleshoot your office NAS when people are all over you to fix it now.

How can I keep them up to date without being present in those offices? remotely?

Remotely managing a NAS is not a real problem for any network computer guy/gal but since you are asking I suspect you are not sure how to configure company network switches and routers. I recommend you find out if that company would accept the security risk of opening up a hole in the network to enable remote management. Also, if you don't have the proper motherboards with IPMI then you won't be able to easily power off, power on, or configure the BIOS remotely. This is where true server motherboards shine.

Looking at your motherboard selection, it does not have remote features, it does not support ECC RAM (most businesses would desire for data integrity), it does support 32GB RAM but that is not very much for the NAS hosting MS Office files, you will likely see slow access times as people open multiple files or very large files.

CPU AMD FM2 A4 series with 2 cores

If you go cheap, you will pay the price in performance. This might be fine for the backup NAS, not the MS Office hosting files NAS.

I hope I've given you a lot to think about.

The NAS are turned off in the evening and turned on in the morning.

This is generally a no-no as you will generally induce premature hardware failures/death. The drives should never sleep as well and you must ensure you have SMART testing configured to test the drives frequently and then your email setup to get reports of good or bad results.

I agree but only if there is a true HBA card that is smart enough to automatically select the non-failed SSD, otherwise it saves very little time, especially if you can't remote into the BIOS to change the boot device.

There are a lot of factors here and as of right now @WhiteTiger , you need to select proper components for this task.

I hope that answers your questions.

 

[WhiteTiger]

I try to answer everything.
If I had a bigger budget I would have taken Supermicro MBs, but, as they say where I come from, "this is what the convent gives".

Currently the two offices both have a small QNAP with two HDD 1TB.
I think that even with an older CPU like mine, but with 16GB RAM, the performance is still superior to their current NAS.
So I focused more on data protection, making the most of the 7 SATA ports.

I am able to manage the network remotely.
When I was writing to manage the NAS remotely, I didn't mean the "boot phase" because this MB does not allow remote BIOS management. Instead, I was referring to configuration management when the NAS is up and running perfectly.
If the boot does not start correctly, as it seems to be from what you write, the only thing I can do is go to that office to get it started or find someone locally who can do it with instructions given over the phone.
The goal must be to keep downtime to a minimum.

As for the discs.
The budget is minimal and I can't afford to buy WD Red HDDs only. I have three. Either I use them at best, or I will go to use them in some other server and on these two NAS I will insert normal disks.
If I have to use only one SSD for booting, I can mount six 3.5" HDDs on one NAS and five 3.5" HDDs on the other.

Their Laptops use Dropbox to synchronize data, so I have to provide in FreeNAS the ability to create folders linked to Dropbox.
I also wanted to install NextCloud if possible.
I was thinking of using one or two 3TB WD Green drives for this and to manage a first backup from the NAS work disks.
FreeNAS must then manage a second backup to the current QNAP (which I continue to use only for this purpose) and a third backup to an Online service.

I thought and hoped that with two SDDs I had also solved the boot problem, but obviously I was wrong.
The rest of the configuration seemed OK to me, but I am happy to listen to your suggestions.

Hypothesis 1
3 Red 3TB HDD in RAIDZ1 plus 1 WD Green 3TB for backup and Dropbox+NextCloud

Hypothesis 2
4 or 5 1TB HDDs in RAIDZ2 plus 1 WD Green 3TB for backup and Dropbox+NextCloud

 

[jenksdrummer]

I don't appreciate the tone. My comments are intended to be helpful.

Although you can consider a pool of mirrors to be "equivalent" to RAID10, it's still not actually the same thing.

Until Top Level VDEV removal was introduced, that was a key difference in not being able to remove a mirror without destroying the pool. I agree that equivalency is closer now. It's still not there for other levels like RAID 50 or 60. https://openzfs.github.io/openzfs-docs/man/8/zpool-remove.8.html

technically speaking, ZFS is not RAID no matter how it's sliced up. It emulates RAID, but, as anyone who has multiple vdevs can attest, it imbalances itself where hardware RAID does not. Between mirrors, I'm sure it's all nice and tight....but if you have 3 vdevs of Z2 for an 'effectively R60 array', you will have some differences between allocations to each vdev; and I'm not speaking towards adding vdevs post creation...from the get-go. Same with 2 vdevs of mirrored pairs. IT's not the same. Effectively, it is. Basically we're both right, and we're both wrong. All good.

Link is interesting; wasn't aware that you could remove a data vdev from a pool. IE, using that 3x vdev example with each vdev being z2, one could effectively remove the 3rd vdev without data loss. A single vdev as I understand is equal to a raid function sans "0" on the end; save for stripe, which is RAID 0. IE, Mirror = R1, Z1 = R5, Z2 = R6. When you add a vdev, you're adding a stripe to the array, IE, tacking on a 0 to the level; Mirror becomes effectively R10, Z1 becomes R50, Z2 becomes R60. Effectively...or similar to...behaves like...functionally...

 

[joeschmuck]

@WhiteTiger
You can only do what your budget allows. If your company really wanted to safeguard the data then they would invest in this NAS project correctly, it's in reality a very small cost compared to any other NAS solution of a comparable speed/throughput. Maybe you can eventually get someone to open their checkbook to let you buy one proper system and make it the main file server.

Make sure you are not buying SMR drives, you will likely run into problems and they are slow. I would recommend that you run a SMART Short test nightly and a SMART Long/Extended test weekly on all drives at the same time. Make sure your email settings are working too.

As for RAID1 or RAIDZ2, always choose RAIDZ2. This is a huge time saver should you have two drives fail. With Z1 your data is lost with two drives down, with Z2 you can have two drives down without risk to your data. Lets be clear here too, if you have a single drive failure and you replace the drive, while that drive is resilvering should another drive fail and you have a RAIDZ1 setup, your pool is gone.

As for your data capacity, you should know exactly how much storage you need and then at a minimum double it and then you figure out how important the data is to have available and safe so you determine if you need RAIDZ1, Z2, Z3, or two to three drive mirrors. A properly configured system will make the business very happy with fast reliable data.

If you are selling a business this NAS and your services, I suggest you become very familiar with FreeNAS, permissions, etc... If this is for your own business you will still need to learn how to do all of this.

I wish you luck on your adventure.

 

[jenksdrummer]

@WhiteTiger
You can only do what your budget allows. If your company really wanted to safeguard the data then they would invest in this NAS project correctly, it's in reality a very small cost compared to any other NAS solution of a comparable speed/throughput. Maybe you can eventually get someone to open their checkbook to let you buy one proper system and make it the main file server.

Make sure you are not buying SMR drives, you will likely run into problems and they are slow. I would recommend that you run a SMART Short test nightly and a SMART Long/Extended test weekly on all drives at the same time. Make sure your email settings are working too.

As for RAID1 or RAIDZ2, always choose RAIDZ2. This is a huge time saver should you have two drives fail. With Z1 your data is lost with two drives down, with Z2 you can have two drives down without risk to your data. Lets be clear here too, if you have a single drive failure and you replace the drive, while that drive is resilvering should another drive fail and you have a RAIDZ1 setup, your pool is gone.

As for your data capacity, you should know exactly how much storage you need and then at a minimum double it and then you figure out how important the data is to have available and safe so you determine if you need RAIDZ1, Z2, Z3, or two to three drive mirrors. A properly configured system will make the business very happy with fast reliable data.

If you are selling a business this NAS and your services, I suggest you become very familiar with FreeNAS, permissions, etc... If this is for your own business you will still need to learn how to do all of this.

I wish you luck on your adventure.

I would add one more thing...your recommendation hints a bit, but I'd like to make it pretty straight forward...

ZFS performance starts to decline at around 50% consumed space. So, allowing for zero growth, double what you are currently using. I'd recommend multiply it by 4....

 

[WhiteTiger]

@jenksdrummer - @joeschmuck
I'm trying your suggestions, but maybe I'm getting lost somewhere

I summarize my TreeNAS which now has this configuration:

• 3 x 3TB WD Red HDD with RAIDZ1
• 1 3TB WD Green HDD that I would like to use for a backup
• 1 x 120GB SSD per boot.
• The motherboard has 7 SATA ports.
• The memory is 16GB

You advised me to use an SSD for cache and logs.
Even reading the documentation, I cannot understand if we are talking about configuring the boot SSD or if I need to configure a second SSD and how to configure it.

Also I still have a free SATA port, to other 120GB SSDs and 1TB HDDs that I can possibly use.

 

[joeschmuck]

You advised me to use an SSD for cache and logs.

I would advise against it because you have low RAM (16GB), I doubt a cache will help you and odds are it could hurt performance.

See if your system is working okay. If you feel it's not doing what you want, specify what it isn't doing and what you want it to do. Odds are you do not have enough computer to make it work but let's take it one step at a time.

 

[jgreco]

Link is interesting; wasn't aware that you could remove a data vdev from a pool. IE, using that 3x vdev example with each vdev being z2, one could effectively remove the 3rd vdev without data loss. A single vdev as I understand is equal to a raid function sans "0" on the end; save for stripe, which is RAID 0. IE, Mirror = R1, Z1 = R5, Z2 = R6. When you add a vdev, you're adding a stripe to the array, IE, tacking on a 0 to the level; Mirror becomes effectively R10, Z1 becomes R50, Z2 becomes R60. Effectively...or similar to...behaves like...functionally...

A single vdev is a virtual device that offers some sort of storage and hopefully redundancy characteristics.

Unfortunately, RAIDZ1, RAIDZ2, and RAIDZ3 are nothing like RAID5, RAID6, and RAIDdoesntexist, because while their goals are similar in that they would like to be able to survive the loss of a certain number of disks, the technology that implements this in ZFS is vastly different because of the way the parity is handled. Legacy RAID places parity at precomputed locations on the disks. ZFS does not, and, due to the way ZFS works, you can actually do some really nasty things that end up consuming a great deal of additional space if you wrongly assume that the ZFS guys simply wanted to call their RAID6 "RAIDZ2". A bicycle is not a motorcycle just because both happen to have two wheels. Likewise, while it might seem tempting to call mirrors "RAID1", what RAID level is a three-way mirror? The abstractions are different and so the language is different.

ZFS performance starts to decline at around 50% consumed space. So, allowing for zero growth, double what you are currently using. I'd recommend multiply it by 4....

ZFS performance may already have evaporated at 50% consumed space if you have a highly fragmented pool. The drop between 50% and 90% steady state performance is marginal compared to the drop between 10% and 50% steady state performance.

Idiot forumware still won't let me insert images.

https://extranet.www.sol.net/files/freenas/fragmentation/delphix-small.png

 

[WhiteTiger]

@jgreco - @jenksdrummer
Sorry, but do I have to deduce that with 50% of the space occupied I already have to start crying?
I'm happy
Today with a small full QNAP we work quite well.

 

[jgreco]

@jgreco - @jenksdrummer
Sorry, but do I have to deduce that with 50% of the space occupied I already have to start crying?
I'm happy
Today with a small full QNAP we work quite well.

At 50%, what you will find is that as time goes on, write performance slowly worsens until you reach a "steady state" which is what that graph is all about.

It is worth noting that even there, the performance is typically better than fragmented storage on a conventional filesystem like EXT3.