Hardware Compatibility and Advice

[Randymations]

Hi there,
First off, the goal of my NAS is to serve as archival storage, web storage, and a large file transfer system. Multiple web systems will be using it as their primary storage. My minimum requirements are 64 TB usable with either RAIDZ2 or RAID10. I've put a lot of time into finding suitable hardware and believe I have found a good solution, but am looking for confirmation that it will work as expected.

The system:

SUPERMICRO 6027TR-D70RF+ 2U 3.5" LFF 12-Bay 2-Node X9DRT-HF+ Server CTO | eBay

Find many great new & used options and get the best deals for SUPERMICRO 6027TR-D70RF+ 2U 3.5" LFF 12-Bay 2-Node X9DRT-HF+ Server CTO at the best online prices at eBay! Free shipping for many products!

www.ebay.com

The drives:

12TB SATA ST12000NM0007 Seagate 3.5" Enterprise Hard Drive Exos X12 7.2K 256MB 763649089729 | eBay

Find many great new & used options and get the best deals for 12TB SATA ST12000NM0007 Seagate 3.5" Enterprise Hard Drive Exos X12 7.2K 256MB at the best online prices at eBay! Free shipping for many products!

www.ebay.com

The setup:
I chose these drives because they achieved the minimum capacity I want (plus a nice bonus) for a fair price. I'm aware of the used drive gamble so I plan on dedicating 10 drives to RAIDZ2 (96 TB usable storage) plus two hotspares. I also plan on getting a 4 TB flash storage as cache for good transfer speed. Finally, I plan on keeping a compressed and encrypted copy on AWS Glacier Deep Archive (unless there is a better option).

The issue:
The system I picked out comes with a SAS backplane but the drives I chose are SATA. I'm having trouble finding information on this backplane and, in combination with me not being very knowledgeable in enterprise storage technology, am unable to determine what I need to do to remedy this. I found a manual on a backplane that appears the same, but begins with "BPN-". Is this the same product? Is it possible for me, within the restrictions of the protocol and physical space of the system, to purchase an adapter such as this or this? If so, what adapter? If this isn't possible, where would I find a suitable replacement backplane? Are there any other major issues that would make this entire system impractical or non-functional (particularly concerning the dual-motherboard, quad-CPU nature)?

Thanks for any and all information

 

[Ericloewe]

The system:

That's a two-node chassis. That's definitely not what you should be looking at, it's specialty hardware you can't trivially get to work as intended. Fortunately, there's a lot more appropriate systems out there.

The system I picked out comes with a SAS backplane but the drives I chose are SATA.

Not a problem generally (it would be with this chassis, in the sense that only one host would see the disks). You don't need any adapters, the drive bays take SATA and SAS disks. See also: https://pcper.com/2015/06/the-connector-formerly-known-as-sff-8639-now-called-u-2/

Are there any other major issues that would make this entire system impractical or non-functional (particularly concerning the dual-motherboard, quad-CPU nature)?

Yes, the dual-motherboard part is an instant problem.

 

[Davvo]

First, don't use hardware RAID terminilogy please.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

Second, before adding 4TB of L2ARC you should really consider how much RAM you are going to put into your system (not clear in the link).

You should consider creating a single 12 Wide VDEV in RAIDZ3 instead than a 10 wide one in RAIDZ2 + 2 hotspares.
However please note that RAIDZ# is not the best in terms of performance, especially regarding IOPS; can you expand more on the number of systems, users, and the performance you hope to get out of this?

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

SATA drives can be plugged into SAS interfaces, but not the way around.

 

[Randymations]

can you expand more on the number of systems, users, and the performance you hope to get out of this?

I don't expect or need any specific performance, I'm more maximizing space and cost and taking what performance I get out of it. The websites I'm hosting are low-traffic and it's all going to be low-access. Specifically, I have a bunch of old desktops I'm using to host a bunch of other services and I expect to get fewer than 100 visitors per year.

you should really consider how much RAM you are going to put into your system

Oops- I was planning on filling the system with as much memory as it can take as I read somewhere you should generally have 1 GB per TB.

You should consider creating a single 12 Wide VDEV in RAIDZ3 instead than a 10 wide one in RAIDZ2 + 2 hotspares.

This is an interesting suggestion; I was under the impression that hotspares are important especially when using used drives. What information is relevant in deciding which to go with?

Yes, the dual-motherboard part is an instant problem.

This was one of the cheapest options I found. Most of the other systems I've found don't come with motherboards so I was hoping to make this work. If it's really going to be an issue, what should I consider when finding a replacement?

 

[Ericloewe]

It's going to depend on local availability, but it's generally easy to find server barebones, if not outright full servers, when searching for Supermicro. The system you found is indeed cheap, but that's because it needs custom software and a lot of work to get running as intended.

 

[Randymations]

I suppose a better question for me to ask is: Do you know if it would be possible/worth it for me to remove the motherboard backplanes and run essentially two entirely separate systems within the same chassis? I'd dedicate one to drive operations and use the other for hosting. Looking closer at the listing, it actually doesn't appear to come with CPUs. A quick check of the Hardware Guide reveals there appear to be multiple processors under "Heavy Usage" that would fit the sockets on the board. Assuming it is possible for me to separate these boards, which of these processors would ensure that my RAIDZ2 or RAIDZ3 operations are not bottlenecked by the processors?

 

[Ericloewe]

I don't know, but some systems along these lines include very non-standard things like inter-host PCIe links, which are rather likely to be troublesome. It's really not worth the risk of ending up with a paperweight.
I guess you could probably use only one of the hosts and put the other one on a shelf as a spare, but you could probably find a decent single server of the same generation for like 250-300 bucks.

 

[Davvo]

This is an interesting suggestion; I was under the impression that hotspares are important especially when using used drives. What information is relevant in deciding which to go with?

• with 10 drives in RAIDZ2 + 2 hotspares you are effectively wasting two drives while in a 12 wide RAIDZ3 you are using all of them;
• with RAIDZ2 you can lose up to 2 drives before losing the pool while with RAIDZ3 you can lose up to 3 drives before losing the pool;
• with 2 hotspares you effectively reduce to 0 your time of reaction in case of a failure

Generally, hotspares are used with multiple VDEVs; imho higher parity is better than instant reaction speed in case of a drive failure if you are able to access the system. You also gain more space per money paid since you are using all of your drives.

It's going to depend on local availability, but it's generally easy to find server barebones, if not outright full servers, when searching for Supermicro. The system you found is indeed cheap, but that's because it needs custom software and a lot of work to get running as intended.

Also, it's a quite old system with Super X9DRT-HF+ motherboards (almost a decade old). Might be worth running a single node... worth noting that it doesn't seem to come with RAM.

 

[Randymations]

So following your suggestions, I'm dumping dual-board system. Searching for a replacement, I can't seem to find many complete systems minus drives. These are the two I found:

Supermicro SSG-6029P-E1CR12T X11DPH-T 2x Intel Xeon LGA-3647 2U Server CTO | eBay

Processor(s)Dual Socket P (LGA 3647). SUPERMICRO SSG-6029P-E1CR12T X11DPH-T 2U CTO Server with Rail Kit and 2x 2.5" Rear Bay HDD Kit installed. Form Factor2U 2U RACKMOUNT. Server Parts. Clean and tested by a qualified technician.

www.ebay.com

Supermicro Superserver Intel Xeon E5-2600 12 Bay Server - (6028U-TR4T+) for sale online | eBay

Find many great new & used options and get the best deals for Supermicro Superserver Intel Xeon E5-2600 12 Bay Server - (6028U-TR4T+) at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Mostly what I've found are half systems - No drives or mobo but everything else (chasis, PSUs, PDB, caddies, backplane, cables). Would it make more sense for me to get one of these "complete" systems or get a "half" system and find an enterprise board or consumer board to fill it?

Many examples of "half" systems:

Supermicro CSE-826BE1C4-R1K23LPB 12-Bay 3.5" SAS3 12Gbps 2U Server Chassis NVME | eBay

Expander support SAS3 12.0Gb/s. 2U chassis support for maximum motherboard size - 12" x 13", 13" x 13.68". Server Parts. Seven low-profile PCI expansion slots. Form Factor Support up to 12"x13" MB with a rear 2.5" HDD option installed.

www.ebay.com

Supermicro 3U 16 Bay 3.5 E-ATX ATX Storage Chassis Case 836TQ Full Height PCIe | eBay

Find many great new & used options and get the best deals for Supermicro 3U 16 Bay 3.5 E-ATX ATX Storage Chassis Case 836TQ Full Height PCIe at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro CSE-826BE16-R920LPB 2U Server Chassis 2x920W 12x 3.5" BPN-SAS2-826EL1 | eBay

Expander support SAS2 6.0Gb/s. SupermicroCSE-826BE16-R920LPB 12x 3.5" Drive Bays 2U rackmount chassis. Clean and tested by qualified technician. Sold as pictured, no other accessories are included. Rail kit is not included.

www.ebay.com

Supermicro 2U Server 8 or 12 HD Bay 3.5 LFF E ATX Storage Chassis Direct Connect | eBay

Find many great new & used options and get the best deals for Supermicro 2U Server 8 or 12 HD Bay 3.5 LFF E ATX Storage Chassis Direct Connect at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro 4U Server 36 Caddy Bay LFF E ATX Storage Chassis 6Gbs 847E16-R1400LPB | eBay

Supermicro 4U 36x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 36x 3.5" Supermicro caddy. Server Chassis/ Case: CSE-847E16-R1400LPB.

www.ebay.com

Supermicro 4U 36 Bay Storage Chassis SAS3 12GB/s ATX E-ATX PWS-1K28P-SQ w/ Rails | eBay

Backplane: SAS3 Expander Backplane. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Caddies Included: 36x Supermicro 3.5 " caddyCaddies. Build ID: 419201.

www.ebay.com

SuperMicro CSE 847 36 Bay 4U SAS 3 Barebone ATX E-ATX Chassis 2x PWS-1K28P-SQ | eBay

Find many great new & used options and get the best deals for SuperMicro CSE 847 36 Bay 4U SAS 3 Barebone ATX E-ATX Chassis 2x PWS-1K28P-SQ at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro 2U Server 12 Caddy Bay 3.5 LFF E ATX Storage Chassis SAS2 6GBPS Rail | eBay

Supermicro 2U 12x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 12x 3.5" Supermicro caddy. Rail Kit: Generic Sliding Rail kit.

www.ebay.com

Supermicro 3U 16 Bay 3.5 LFF E-ATX Storage Chassis PWS-920P-SQ Full Height PCI-E | eBay

Find many great new & used options and get the best deals for Supermicro 3U 16 Bay 3.5 LFF E-ATX Storage Chassis PWS-920P-SQ Full Height PCI-E at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro 2U Server 12 Caddy Bay 3.5 LFF E ATX Storage Chassis SAS3 12GBPS Rail | eBay

Supermicro 2U 12x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 12x 3.5" Supermicro caddy. Rail Kit: Generic Sliding Rail kit.

www.ebay.com

Should I be looking on sites other than eBay to find this stuff? If so, where?

For your additional consideration regarding RAID level and hotspares, I won't have physical access to this server for months at a time (maybe six max).

Thanks for all your help so far.

 

[Randymations]

@Davvo @Ericloewe I'm unsure if mentioning people is considered rude, but you've been very helpful so far and I don't know if you're watching this post. Would like to be clear, thankful for all info provided :)

 

[Davvo]

Does it cost more buying an used server or a used chassis + used motherboard and HBA?
What are the pros? What are the cons?

Depending on your needs, consider looking for SAS2 since it's cheaper.

Supermicro 2U Server 12 Caddy Bay 3.5 LFF E ATX Storage Chassis SAS2 6GBPS Rail | eBay

Supermicro 2U 12x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 12x 3.5" Supermicro caddy. Rail Kit: Generic Sliding Rail kit.

www.ebay.com

Regarding your pool config, I would still go RAIDZ3... maybe with a hotspare.

 

[Randymations]

Does it cost more buying an used server or a used chassis + used motherboard and HBA?
What are the pros? What are the cons?

Looking back at the chassis I listed, I agree that the one you posted is the best suited for my needs. I did a bunch of looking at motherboards and found an assortment that all look pretty similar. I restricted myself to DDR4 memory and ensured that it could take at least 128 GB. I've arranged them here in order of increasing price:

Supermicro X10DRH-C Socket R3 LGA 2011 C612 X99 TESTED CPU RAM Dual LAN I350-AM2 672042169764 | eBay

Find many great new & used options and get the best deals for Supermicro X10DRH-C Socket R3 LGA 2011 C612 X99 TESTED CPU RAM Dual LAN I350-AM2 at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro X10DRH-i Dual Intel XEON E5-2600v3/v4 LGA2011-v3 E-ATX Motherboard | eBay

Supermicro Model P/N: X10DRH-i. This SUPERMICRO X10DRH-i motherboard is in excellent condition and ready to be redeployed. The item you will be receiving will be labeled with part number X10DRH-i. Item sold as pictured.

www.ebay.com

Supermicro X10DRH-iT Dual Intel XEON E5-2600v3/v4 LGA2011-v3 E-ATX Motherboard | eBay

Supermicro Model P/N: X10DRH-iT. This SUPERMICRO X10DRH-iT motherboard is in excellent condition and ready to be redeployed. The item you will be receiving will be labeled with part number X10DRH-iT.

www.ebay.com

Supermicro H11SSL-i Motherboard(Configure parameters. View the description) | eBay

Find many great new & used options and get the best deals for Supermicro H11SSL-i Motherboard(Configure parameters. View the description) at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro X10DRI-T4+ Dual Xeon LGA2011-v3 4x10GbE LAN EE-ATX Motherboard for sale online | eBay

Find many great new & used options and get the best deals for Supermicro X10DRI-T4+ Dual Xeon LGA2011-v3 4x10GbE LAN EE-ATX Motherboard at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro X10DAi Dual Socket R3 Motherboard for sale online | eBay

Find many great new & used options and get the best deals for Supermicro X10DAi Dual Socket R3 Motherboard at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Supermicro X10SRH-CF ATX Intel C612 E5-16xx v3/4 LGA2011-3 DDR4 ATX Motherboard | eBay

Up to 1TB ECC 3DS LRDIMM, up to DDR4- 2400MHz; 8 DIMM slots. Single socket R3 (LGA 2011) supports. IO Shield is included. 4 USB 3.0 ports, 8 USB 2.0 ports. 1 VGA, 2 COM, 1 TPM. Printer Parts. 1 PCI-E 3.0 x4 (in x8), 1 PCI-E 3.0 x8 (in x16), 2 PCI-E 3.0 x8, 1 PCI-E 2.0 x2 (in x4), 1 PCI-E 2.0 x4...

www.ebay.com

Particularly of note: The cheapest board combined with the chassis is $350 - Only $2 cheaper than this complete system. In addition to board and chassis, I'd need an HBA card, heat syncs, RAM, and, for most of the boards, (a) CPU(s). Granted, I'll need to buy more RAM for the complete system, but it still comes out much cheaper than buying the parts. What am I missing here? How is it cheaper to buy the prebuilt system?

 

[Davvo]

What am I missing here? How is it cheaper to buy the prebuilt system?

A 25% discount is being applied.

 

[Randymations]

A 25% discount is being applied.

I tend to discount discounts as legitimate as I find they're often used for social engineering, but I suppose this one appears real. I wonder how they're turning a significant profit. Is there anything else I should be mindful of here? If
not, I'll stick with this and move on to memory.

On that topic, it doesn't seem to specify ECC? Unless I'm missing something, I'll message the seller.

 

[Davvo]

Also the HBA has (only) 8 lanes.

 

[Randymations]

Also the HBA has (only) 8 lanes.

I am a bit confused- The HBA (manual) seems to "match" the backplane in which they both have SFF-8643 connectors, supporting 4 lanes each. The backplane manual seems to confirm that there are only two ports to work with and uses a SAS expander to reach 12. What are you suggesting I do differently?

Memory:
Comparing the motherboard manual to the listing implies that, since the DIMMs are 4 x 16 GB, they must be RDIMMs. The hardware guide doesn't say much about the quantitative advantages of LRDIMMs over RDIMMS. Should I ask the seller to exclude RAM from my purchase and buy 128 GB of LRDIMMs or ask the seller to add an additional 64 GB of RDIMMs to reach 128 GB total?

 

[Ericloewe]

LRDIMMs are a necessary evil, rather than inherently beneficial. Or were, they've fallen a bit out of favor. Basically, by buffering every line on the DIMM, they allow for more DRAM ICs to be attached to the same line, in turn allowing for greater capacity. However, the latency is higher.
Don't expect to be able to mix and match in a single system, though, due to a mixture of hardware and firmware limitations.

 

[Randymations]

I see. Since I don't plan on ever needing 1.5 TB of RAM, I'll ask if the seller can add another 64 GB for less than I would pay buying it myself. Besides the HBA comment, is there anything else I should be mindful of?
Additional note: I had to change my drive selection and went through a more rigorous selection process including looking at more options and checking Backblaze records. These are better as they are higher capacity for a similar rate and 5400 RPM as recommended by the hardware guide.

Recap of build:
https://www.ebay.com/itm/204261717743 + 64 GB of the same RAM
https://www.ebay.com/itm/134678847479 x12

 

[Davvo]

I am not overly confident of my knowledge regarding backplanes and HBAs, but to me it looks that in order to use all the 12 slots you need to plug four cables?

 

[Ericloewe]

I am not overly confident of my knowledge regarding backplanes and HBAs, but to me it looks that in order to use all the 12 slots you need to plug four cables?

No.

• -TQ backplanes have individual SATA-style connectors for each disk
• -A backplanes consolidate this into SFF-8087 or SFF-8643 connectors for four disks per connector
• -EL1 and EL2 backplanes use expanders, so only a single cable is necessary, though two cables are also doable. Additional ports allow for cascading expanders.
  • -EL2 backplanes have two expanders for dual-port usage, which doesn't add much of anything