SuperMicro X10SL7 experience

[RyanJennings]

After reading a lot of the hardware forum I thought I should post my experience. Maybe it will help someone else out.

Background:
I started on zfs with nas4free about 2-3 years ago. Things worked alright. I ran 3 2TB drive in a radidz1 on an amd e-350. I did a rsync to a offsite running ext4 for the more important stuff. I ran my webserver, htpc, and a plex server on a different esxi box.

With the drives nearly full and always wishing for lower power consumption I started investigating a new build.

I wanted to decrease my power consumption along with upgrade my storage capacity.

I arrived at the following:
SuperMicro X10SL7 (edit:E3-1230v3)
16GB Crucial ECC + 16GB Samsung ECC
256GB M500 ssd
3x 4TB red
Radeon 5450 video card

I put EXSI 5.5 on a flash drive and have several vms running.

FreeNas 9.2.1.4.1 with 8GB and controller in IT mode passed through
Ubuntu 12.04 with 4GB for webserver and other various things
Win7 with 2GB for plex
Win7 with 16GB for HTPC (I wanted to do some sort of serious photo editing)

FreeNas is working well with around 100MB/sec on smb transfers.
The HTPC was a bit of a pain to get working, mostly because of the video card and usb passthrough issues. Won't go into a lot of detail since this is a FreeNas forum, but adding one thing at a time between reboots seemed to be the way to go.

So I have used a lot of suggestions from the forum, but have gone against some rules. Some would say FreeNas in ESXI is not a good idea. I can sure see that argument if your storage can't be lost. I am doing an offsite backup and a cold backup periodically so I felt that was enough to cover the risk. I also probably should have done a raidz2, but I didn't. I was please with the transfer speed up from my old 35MB/sec to 100MB/sec. Power consumption is around 80w with all vms running, but not really loaded.

Just wanted to post in case someone found the info useful. Feel free to ask questions if I left something out.

Ryan

 

[Ericloewe]

No stability issues due to having two different types of memory?
Why bother with plex on a seperate OS if FreeNAS can take care of that?
Why only 8GB for FreeNAS? 16GB is usually quoted as the sweet spot and I seriously doubt you need anywhere near 16GB for your photo editing. If you did, you'd be using a dedicated workstation.

 

[RyanJennings]

Good questions...

I had originally ordered the 16GB crucial. I had planned on getting some things running and then move on to 32GB shortly after. During that time I started reading of the issues people were having trying to run 32GB crucial so I went with the Samsung for the second 16GB. Not ideal, but I felt it was better than chancing things with 32 crucial.

I have plex on a seperate OS because I had it setup in a VM already so I just transferred it to the new hardware. I don't have any experience with the jails. Maybe someday I will switch things over.

I started with 8GB for FreeNAS based on the 1GB/TB ratio frequently mentioned. Then when I was able to get 100MB/sec I didn't see any room for improvement with more since that is all the network will handle. Maybe more ram would help my rates in cases where I am reading many small files?

I think the ram is probably helping me in the photo editing. It is able to cache more thumbnails and previews of my image catalog. I honestly haven't used it enough to tell yet though. One thing that really intrigues me about the photo editing sharing hardware with the NAS is that I should be able to get past the gigabit network limit by running virtual 10 gigabit cards. Anyone have thoughts or suggestions on that one?

 

[engmsf]

I wonder what has more risk:
a/ FreeNAS with no ECC
b/ FreeNAS with ECC under ESXi
c/ Shutting off the system every night .....

I have been running a FreeNAS system over a year now against a few rules and have not encountered any issues. I weighed the risk of loosing the NAS data versus having/setting up regularly offline backups, and the backups were more important. The system primarily functions as a dedicated Plex server so I can afford to loose the data.

1/ My system Gigabyte GA-H87N + G3220 has 16GB of regular memory. The G3220 is really only good enough to serve one client if you are going to transcode 100% of the time. Otherwise consider something more beefy.
2/ 3x 3TB Red in Z1. I have transferred the volume to 4 or 5 different hardware setups and it has never failed to import and restore. Amazing that it works so well!
3/ We shutoff the system every night, usually by just hitting the power button on the computer and then power it up again in the morning. The Plex server continues to run well.

 

[cyberjock]

No clue.. even finding information on how "dangerous" one is compared to the ideal system is hard to come by. We have plenty of reasons for our justification. But providing numbers is almost impossible because people that have no problems won't post. We only hear about the broken stuff.

I know that one of the FreeNAS developers said they get tons of kernel crash info every day, and 99.9% of them are... bad RAM errors. lol.

 

[Ericloewe]

I wonder what has more risk:
a/ FreeNAS with no ECC
b/ FreeNAS with ECC under ESXi
c/ Shutting off the system every night .....

I have been running a FreeNAS system over a year now against a few rules and have not encountered any issues. I weighed the risk of loosing the NAS data versus having/setting up regularly offline backups, and the backups were more important. The system primarily functions as a dedicated Plex server so I can afford to loose the data.

1/ My system Gigabyte GA-H87N + G3220 has 16GB of regular memory. The G3220 is really only good enough to serve one client if you are going to transcode 100% of the time. Otherwise consider something more beefy.
2/ 3x 3TB Red in Z1. I have transferred the volume to 4 or 5 different hardware setups and it has never failed to import and restore. Amazing that it works so well!
3/ We shutoff the system every night, usually by just hitting the power button on the computer and then power it up again in the morning. The Plex server continues to run well.

No ECC is definitely the worst offender, as it's the only one in the list guaranteed to make you lose data sooner or later (statistically).

Properly setup, virtualization is not a problem. The problem is that properly setting it up is not trivial.

Regular shutdowns may (or may not) lead to earlier hard drive failure. You'd need to examine wear at idle versus wear at spin up. In practice, neither is likely to be noticeably different in reliability.

 

[cyberjock]

Actually, all 3 of those have cost people their pools, and all 3 can happen with no warning whatsoever. Most people don't have backups, but...

1. Non-ECC is plenty bad because it can and will kill your primary pool and your backup before you knew what happened. Even the most amazing backup strategy can be killed with bad non-ECC RAM.
2. This will definitely kill the pool without warning. Backups, if they are not virtualized, are usually safe. But since most people here don't backup to begin with....
3. This won't kill a pool directly, but the fact that the regularly scheduled tasks have a much lower chance of actually occurring, things can get ugly. Too many people resilver a disk and shutdown the server forgetting that a resilver was in progress. They also shutdown the server before the scrub finishes, so you may be in a situation where a scrub never happens. Dealt with 4 people that hadn't done a scrub in more than 18 months. Not surprisingly I ended up being involved with their data recovery efforts because they hadn't done a scrub. Again, since most people here don't do backups to begin with...

When you say "how much risk" the question should also include every possible outcome for all situations. That's just untrackable because of the number of possibilities.