New to ZFS - help with budget NAS build

[k__]

Hello fellow storage enthusiasts!

After getting my laptop stolen this week and realizing that for as much as I tell people to have backups I really did not have a proper backup strategy, I thought I would set up a little NAS, and upon ~~rolling dice~~ some research I decided to go with TrueNAS.

This is mostly supposed to be a learning project, and I don't really mind if the performance is suboptimal, since the setup will be kinda janky anyway, but some degree of redundancy would be nice just in case. Mostly going to be keeping backups and non-critical files on this though.

My setup is probably fairly unconventional - I got a small intel atom based single board computer (with 4GB RAM, yes, I know the recommended mimum is higher, and I am considering getting some more RAM if this works okay) with a minipcie expansion card adding 4 SATA connectors, in addition to 1 onboard. I am currently running TrueNAS CORE off an m.2 (technically also SATA) SSD, but could move the installation over to an external drive and stuff a 2x SATA adapter board in there, giving me a total of 7 SATA ports. Another option is that I have several (quite a few actually) of the SBCs, and was thinking of maybe setting up TrueNAS SCALE instead and scaling horizontally, with a couple drives connected to each machine? This would in theory also make the computer less of a failure point? Cluster of 6 little computers with a disk attached to each of them with a dispersed Gluster? :D Again, performance is not my biggest priority (otherwise I would have just used something more powerful than tiny little SBCs with passive cooling, lol).

As for the storage, I got a bunch of drives laying around. Ideally I would like to get the most capacity out of the drive. One drive redundancy is fine for me. Also, I am fine with having some storage that would not be redundant if it meant getting more storage (for use with really non-critical data). The drives I have are:
2x 4TB
1x 6TB
1x 2TB
2x 1TB SSD
1x 1TB SSHD

One option I see could be to stripe the 2x4TB to make a 1x8TB, then mirror/replicate that to the 6TB+2TB?, Then maybe set up the 3 SSDs as a fast (not really because SATA) 2TB RAID-Z1, though that would probably need to be on another machine.

Another option could be if I could do a RAID-Z1 on the 2x4TB + 1x 6TB? even better if I could then only use 4TB on the 6TB and mirror the remaining 2TB onto the 1x2TB drive? But I'm not sure if that makes any sense or if it would be better to just give it the 6TB whole and later maybe replace them with something bigger, though in that case I think setting up a fresh pool probably would make a lot more sense.

 

[Ericloewe]

Another option is that I have several (quite a few actually) of the SBCs, and was thinking of maybe setting up TrueNAS SCALE instead and scaling horizontally, with a couple drives connected to each machine?

Gluster is a bit painful with big, capable servers and 10GbE networking... Your proposal sounds like hell on Earth.

Overall, I have one comment to your scenarios: Keep it simple. The further you stray from simple, the more things can go wrong, the more likely it is you'll lose data and the more likely it is that even if you can get your data back, it'll take much longer and more help. I'd also add that, a corollary of this is that having many small computers doing something that a single not-too-big machine could just as easily do creates a lot of entropy for very little gain. I realize that the "but I have it on hand" argument is compelling, but don't forget that throwing extra time at the problem that could've been avoided is a very real, very likely risk.

with 4GB RAM, yes, I know the recommended mimum is higher

It's actually a minimum requirement, which is to say that people are unlikely to invest their time helping to troubleshoot a system with just 4 GB of RAM when the minimum these days is 16 GB.

 

[ChrisRJ]

Please read up on ZFS (see resources linked in my signature). What you describe is very far from what I would personally consider a suitable system for ZFS. Also, the amount of complexity, as @Ericloewe already mentioned, is a considerable risk.

In a nutshell my view on ZFS is that it is a great system, if data protection is your goal and you want to pursue it without compromise. The latter puts a number of limits on the choice of hardware as well as the configuration of disks. ZFS was developed for high-end storage systems for the data center. Think hundreds of disk drives and a 6-digit-figure budget (or more). This means that design decisions were made based on the assumption that a certain kind of hardware is available. On the surface, i.e. under normal circumstances, things may work ok on less-than-optimal hardware. But there is increased risk of something going wrong.

If you are willing to live with that risk, that's ok. However, you loose the core advantage of ZFS. I don't know what exactly made you decide to go for TrueNAS (most YouTube content about it is crap, unfortunately). This would indeed be very helpful information to guide you further.

In the end we want you to be happy and not loose valuable data - whatever your choice for the NAS is.

Hope that helps.

 

[k__]

Gluster is a bit painful with big, capable servers and 10GbE networking... Your proposal sounds like hell on Earth.

Never said I was looking for the practical way of doing things, but thanks for the warning.

Overall, I have one comment to your scenarios: Keep it simple. The further you stray from simple, the more things can go wrong, the more likely it is you'll lose data and the more likely it is that even if you can get your data back, it'll take much longer and more help. I'd also add that, a corollary of this is that having many small computers doing something that a single not-too-big machine could just as easily do creates a lot of entropy for very little gain. I realize that the "but I have it on hand" argument is compelling, but don't forget that throwing extra time at the problem that could've been avoided is a very real, very likely risk.

Good points. In my case the extra time spent on the problem can be worth it for gaining experience with this system, but can see how keeping things simple might be worth it if not wasting too much time is a concern.

It's actually a minimum requirement, which is to say that people are unlikely to invest their time helping to troubleshoot a system with just 4 GB of RAM when the minimum these days is 16 GB.

Makes sense. The fact that the RAM is non-ECC is also not ideal. Still planning to go through with this to get a chance to try out TrueNAS though.

Please read up on ZFS (see resources linked in my signature). What you describe is very far from what I would personally consider a suitable system for ZFS. Also, the amount of complexity, as @Ericloewe already mentioned, is a considerable risk.

Thanks for recommending the resources! Definitely feel like I have a better grasp of how vdevs and pools work now and why my proposed setup isn't ideal. A 6TB+4TB+4TB RAIDZ1 would give me the most data with some redundancy but from what I understood adding new disks to a RAIDZ1 isn't supported yet? Thinking of just doing a 4TB+4TB mirror for now then, and another 1TB+1TB mirror. Maybe I'll put the second one on a separate NAS just to have some hardware redundancy and set up replication for critical data?

In a nutshell my view on ZFS is that it is a great system, if data protection is your goal and you want to pursue it without compromise. The latter puts a number of limits on the choice of hardware as well as the configuration of disks. ZFS was developed for high-end storage systems for the data center. Think hundreds of disk drives and a 6-digit-figure budget (or more). This means that design decisions were made based on the assumption that a certain kind of hardware is available. On the surface, i.e. under normal circumstances, things may work ok on less-than-optimal hardware. But there is increased risk of something going wrong.

Yup definitely get the same impression after reading your recommended resources, especially the ZFS layout whitepaper. I am not planning on getting quite up to those numbers of disks yet.

If you are willing to live with that risk, that's ok. However, you loose the core advantage of ZFS. I don't know what exactly made you decide to go for TrueNAS (most YouTube content about it is crap, unfortunately). This would indeed be very helpful information to guide you further.

Honestly had three (kind of stupid) reasons for choosing TrueNAS: I don't think I've ever had a machine running FreeBSD at home, it seemed more professional/enterprise(?) than something like openmediavault, and I was curious to try ZFS since I have only heard about it but never actually looked deeper into it. So basically the biggest reason was to learn something, as well as have something (hopefully) somewhat reliable, although I am planning to use it as a backup for my laptop and desktop. I'm planning to keep a separate backup for the most important data so it would only be an inconvenience and not a data loss problem if the NAS went down. Also, if this ends up working okay I am thinking of maybe setting up a more proper (ECC, more disks etc) system, but wanted to see if I like the general way TrueNAS works.

I'm just thinking to use the NAS as a NAS, and not running any special additional software on it. Well, was considering TrueNAS SCALE with Tailscale to be able to connect remotely, but that's a "nice to have", not a "must have". My first idea was to run Arch Linux with btrfs (if that says something about me tolerating a fragile system) but then decided to go for something a bit more stable that I wouldn't have to update as often :P

In the end we want you to be happy and not loose valuable data - whatever your choice for the NAS is.

Hope that helps.

Thanks for the help!

 

[Ericloewe]

Well, you can look at running vanilla FreeBSD, if you're mostly looking to stick to easy stuff like SFTP it's not much more trouble to setup and you avoid the overhead of the middleware and WebGUI. TrueNAS is valuable if you're looking for more complicated stuff, like SMB shares.

 

[k__]

Well, you can look at running vanilla FreeBSD, if you're mostly looking to stick to easy stuff like SFTP it's not much more trouble to setup and you avoid the overhead of the middleware and WebGUI. TrueNAS is valuable if you're looking for more complicated stuff, like SMB shares.

Have a Win10 desktop so SMB shares are useful. Think I'll go with TrueNAS a simple 4TB mirror setup for now just to get something working and get some more big drives and a better host machine in the future once I have more of a budget for those :D
Maybe set up SCALE on the future host and run some more services too, we'll see.

Again, thanks for the replies!