Recovering from a loose SATA cable

[izomiac]

For the second time I've lost a drive after physically touching my FreeNAS machine. The first time was when I popped open the case to upgrade the RAM, and now it's happened after I physically drove the server to my parent's house to let it replicate a few terabytes that got out of sync when they suspended their internet service for a few months. I'm now planning on replacing the SATA cable and case (a Node 804) due to this reoccurring problem (the cable in question has to make three very sharp turns and I suspect it's being worsened by the sliding action of the drive tray dock). I've already reattached it and am running a long smart test to verify the drive's ok.

As for what happens, I start getting checksum errors and FreeNAS starts crashing because it swaps on data disks (a behavior I plan to correct the next time I rebuild my pool*). When I fix the cable I can't wipe or re-partition the disk because FreeBSD happily finds and uses that swap partition since the drive works fine and FreeNAS seems to not let me reuse the disk if I just wipe the data partition, so I've had to physically put the drive in a USB enclosure and wipe it from my Windows laptop before replacing it and resilvering. Is there a better way to do this?

* I understand that disabling swap is "strongly discouraged". It's already caused a dozen crashes on my machine due to the prior SATA cable issue (the initial symptom was crashing during boot after installing new RAM, so it took me a bit to figure out). I've never been keen on using swap in any OS in any application, with the sole exception of running Samba and a dozen other services on a hacked router with 16 MB of RAM when I was a poor college student. Why use ECC RAM at all if I'm just going to swap to an unreliable failing disk? I consider myself lucky that my pool is still intact, and suspect it's only because the swap wasn't being heavily utilized as it wasn't necessary in the first place. That said, I recognize that most of my irrational hatred for swap is due to glitchy 1990's OSes, so I'm open to other opinions -- albeit I do remain unimpressed with the cargo cult logic of this technological carry-over that made much more sense when it was developed in the 1950's than today.

 

[Chris Moore]

What kind of case do you think you'll move your system into? if you have enough room for a couple of SSDs you can move your swap to SSD

 

[izomiac]

What kind of case do you think you'll move your system into? if you have enough room for a couple of SSDs you can move your swap to SSD

Something with hot swap bays, such as the U-NAS NSC-810A or a tower case (Ex: 1, 2, 3, 4) with 5.25" bay hard drive enclosures (Ex: 1, 2, 3). I could probably use nearly any other case and avoid this issue, I'm just being paranoid by picking something where the cables will remain undisturbed the next time I have to replace a drive. There's a small but non-trivial chance I'll superglue the cables in place to be extra sure.

As for swap on SSD, that's not a bad idea at all. I suspect I can run without it entirely, which I'll probably try initially, but if I encounter trouble that's what I'll do.

 

[SweetAndLow]

Swap is mirrored in the freenas 11.2 so having swap usage and losing a disk shouldn't cause problems anymore. Also you shouldn't be using swap at all, something is really broken with your system if your using swap.

 

[Chris Moore]

but if I encounter trouble that's what I'll do.

Every now and then, not very often, FreeNAS does need to use some swap. I have a system at work that has been running for over two months with over 311TB of data in a 550(ish)TB pool and fairly heavy use; but it has only used 800MB of the 16GB of swap space I configured. That system is much more hand-crafted and the swap is a mirrored on a pair of NVMe SSD drives with each drive having a 16GB swap partition. All the rest of the storage drives have no swap partition at all.

 

[izomiac]

It does appear mirroring swap is a pretty significant improvement in 11.2 versus when this happened last time about a year ago. I didn't get crashes this time, but I initially attributed it to a) not using swap especially with minimal VS maxed-out RAM, and b) immediately fixing the issue before giving it more than one chance to manifest. As for the SSDs, I'm out of SATA ports and lack an M.2 so I'd need to use an HBA like a SAS 9211, which afterwards would leave 6 additional SATA ports begging to be used, which resonates with my search for a new case... Yeah, my "simple & minimal" NAS is probably going to get a decent upgrade. Meh, it's a hobby.

As for the initial issue, with the new way that FreeNAS mirrors the swap, setting the disk to offline actually successfully deactivated the swap this time, which in turn allowed me to erase the partition table (gpart destroy -F /dev/ada5) and reinitialize the disk without removal. The GUI behaved a bit oddly (showed three identical cryptic entries for the disk to be replaced, so I replaced one at random), but it resilvered fine and seems to be chugging along swimmingly now. Murphy's law now dictates that I probably screwed up the GELI part and my happiness will only persist until my next reboot, but such is life.

 

[Chris Moore]

Yeah, my "simple & minimal" NAS is probably going to get a decent upgrade. Meh, it's a hobby.

Depending on your budget, you might want to consider a used rack chassis. I bought several over the years and I love them for the hot-swap bays. I did the math when I bought mine and it is the least expensive way to get hot-swap bays and it usually carries the additional benefit of redundant power supplies.
I can send you some links if you are interested?