Hardware question about USB devices - very strange thermal effects

[Patrick M. Hausen]

Hi all,

you know I would not recommend using USB for anything but boot, and then you have to pick your hardware carefully so it lasts.

Anyway, I have two NVMe to USB enclosures, really solid hardware, that I use as temporary "data tombs" for my Macbook Pro. I am not perfectly sure about the version/generation but they are similar to this current thingy:

IB-1817MCT-C31 | ICY BOX - Create Your Workspace

NVME & SATA: Das ICY BOX IB-1817MCT-C31 ist die richtige Wahl für Ihre M.2 SSD. Unser Gehäuse ist kompatibel mit NVMe und SATA SSDs gleichermaßen und bietet Platz für Speicher in den gängigen Größen bis zu 2280

icybox.de

Insert M.2 SSD, connect via USB-C, enjoy. Passive cooling really good.


Now being the ZFS fanboy that I am I installed ZFS on OS X on my Mac and both devices are "formatted" with ZFS. Today I wanted to do a complete copy of the zpools. My wired network is Fast Ethernet only, and the WiFi is faster, but still USB 3 should be able to beat that. So I connected them directly to my TN CORE system running 13.1 nightly (my production 13.0 would not import the pool due to missing feature flags).

Then I did a zfs scrub ...

The devices ran so hot during that operation one of them was "SUSPENDED" by ZFS. I could not even terminate the scrub or export the pool again. The whole scrub operation was running for short of half an hour at that time.

I had to pull the plug and reboot the TrueNAS system to get into a working state again.

Now, when I connected the same devices to my Mac, neither zfs scrub nor regular use posed any problem ever. After that unexpected behaviour I just did reconnect them to my Mac, did a zfs scrub - five minutes tops, devices hand warm - and now I am transferring the data over WiFi which will take a couple of hours but at least not fry my SSDs.


Does anyone who knows "USB" have any idea why the same device runs hot to the point of failing on one USB connection while running perfectly fine on another?


Thanks!
Patrick

 

[winnielinnie]

Does anyone who knows "USB" have any idea why the same device runs hot to the point of failing on one USB connection while running perfectly fine on another?

I can only guess that perhaps it has something to do with the USB abstraction between OSX vs Linux vs FreeBSD.

I might not be wording this correctly, but I'll drop an example, which is purely speculative.


For Linux users, there is a popular consumer model of Seagate portable USB drives, in which the usb-storage driver on Linux can either use UAS (default) or can be disabled with a module parameter, which will instead use the slower (and more compatible) BOT.

So to bring this back to the case of overheating on Core, but not on OSX, perhaps something similar is happening in the case of your NVMe + USB enclosure, in which FreeBSD is using a slower, less efficient method of I/O through the USB-to-NVMe chipset, which means longer runtime, and thus greater heat buildup without an opportunity to cool down?


It would be interesting to see if the overheating/performance issue doesn't exist on a Linux distro or even SCALE itself. (But I'm not one to tell you to do anything risky with whatever data your already have saved on these.)

 

[Patrick M. Hausen]

@winnielinnie For once I do not have the slightest idea what you are talking about The Linux part. Now with respect to FreeBSD vs. my Mac there's obviously something different. What I learned from your post is that there are different storage access methods at all. WTH of a standard is this? Yeah, I know ...

 

[winnielinnie]

@winnielinnie For once I do not have the slightest idea what you are talking about

@winnielinnie has told me the same thing about myself.

TL;DR:

USB enclosures can be used with BOT or UAS protocol, the latter being newer and faster. (This is the case on Linux and I believe OSX as well, which "should" default to UAS.)

FreeBSD ("Core") might not support UAS, and hence, a performance hit for USB enclosures.

Because of this performance hit, the device may in fact have to spend more time/power for the same amount of I/O, which might manifest as a net positive heat buildup in which it cannot cool down (or remain stable in temperature) if pummeled with nonstop heavy I/O. (M.2 form factor NVMe's are more susceptible to heat buildup as it is, just from their small form factor. So something like this might be enough to "push it over".)

That's why I'd be interested if you tested them on SCALE or a Linux distro, to see if they behave just as well as they do on OSX. (FreeBSD's lack of UAS may in fact be the culprit, but this is just a rushed speculation.)


EDIT: Even as recently as last year:

Reddit - The heart of the internet

www.reddit.com

Some testing has shown that a USB 3.0 adapter for a SATA/NVME drive seems to be the best bet, however the USB adapter needs to support UASP for the best performance. Interestingly, FreeBSD doesn't support UASP and AFAICT neither no any of the BSDs. I can live without the improved performance for my use case but it sure would be great if FreeBSD supported UASP as some tests have shown it to also use slightly less power.

Not sure how true that is today (with FreeBSD 14.0), but it is interesting.

 

[Arwen]

When USB was first invented, storage was not really a goal. So primitive and now old, (aka *rappy), method was used, BOT, (Bulk-Only Transfer);

USB mass storage device class - Wikipedia

en.wikipedia.org

Later, UAS / UASP was invented;

USB Attached SCSI - Wikipedia

en.wikipedia.org

which is noticeably faster. But, OS, OS driver and USB device need to support it.

 

[Davvo]

So I can't talk bad about Apple this time?

 

[Ericloewe]

With no data in hand, I would speculate that UASP would drive greater heating by allowing the SSD controller to work harder. I can't imagine the bridge overheating from being used without UASP. Then again, USB device vendors have an insane ability to limbo beneath the lowest bar of expectations you can set for them.

 

[winnielinnie]

I would speculate that UASP would drive greater heating by allowing the SSD controller to work harder.

To work harder might not manifest as a higher net positive heat buildup, as opposed to working longer.



The highlighted parts are what made me speculate this could be an issue of FreeBSD lacking UAS support, and thus a longer time is required for the same I/O operation (in this case, a ZFS scrub), and thus more heat buildup without the chance to cool down:

The devices ran so hot during that operation one of them was "SUSPENDED" by ZFS. I could not even terminate the scrub or export the pool again. The whole scrub operation was running for short of half an hour at that time.

( ... )

Now, when I connected the same devices to my Mac, neither zfs scrub nor regular use posed any problem ever. After that unexpected behaviour I just did reconnect them to my Mac, did a zfs scrub - five minutes tops, devices hand warm

The way I read it...

TrueNAS Core ("FreeBSD"): Almost 30 minutes have passed, and the scrub couldn't finish, as the devices overheated.

Mac: Scrub finished in only 5 minutes.

My guess is that running a ZFS scrub on the same devices in Linux (or TrueNAS SCALE) will also finish a scrub in about 5 minutes.

 

[Ericloewe]

That could be... Finishing quickly enough to not allow the heatsink to saturate would explain things.

 

[Patrick M. Hausen]

That could be... Finishing quickly enough to not allow the heatsink to saturate would explain things.

I doubt that. At least as a single cause. The zfs send/receive job ran for a couple of hours without the device overheating again. Connected to the Mac.

 

[Davvo]

Did you remove the plastic film/adhesive cover?

 

[Patrick M. Hausen]

Sure - I have been using these devices for years with my Mac. Connecting them directly to my TrueNAS CORE was the first time they got that hot ever.

 

[winnielinnie]

The zfs send/receive job ran for a couple of hours without the device overheating again. Connected to the Mac.

Then if not a "time" thing, perhaps a "power" thing?

Perhaps lack of UAS requires more power for I/O, which translates to more heat?

Still curious if this won't happen on Linux/SCALE.

* I could not sleep last night, as I stared at my ceiling with red eyes thinking about the mystery of the overheating-USB attached NVMEs.

 

[Ericloewe]

* I could not sleep last night, as I stared at my ceiling with red eyes thinking about the mystery of the overheating-USB attached NVMEs.

Some people buy nice mattresses to sleep better. You should buy an electronics lab's worth of gear to analyze SSD power consumption.