New Mini 3.0 X+ L2ARC SSD Constant Drive Activity

[moraga695]

Hello,
I purchased a new Mini 3.0 X+ system from iXSystems in December 2020. It came with 5 hard drives and two SSD's, one configured as a ZIL and the other as an L2ARC. The system arrived from iXsystems with Freenas 11.3 U5 and all was normal. I soon upgraded it to Truenas Core 12.1 and immediately the LED activity light for the L2ARC SSD started blinking as if there was drive activity 100% of the time. This did not happen with the original 11.3 U5 OS. I've since upgraded it to 12.1.1 and the drive light still blinks 100% of the time.

I'm concerned that if this is actually constant drive write activity, the SSD will have excessive, unnecessary wear. Has anybody else seen this happen? Any ideas?

Thanks very much.

Steve W.

 

[Samuel Tai]

12.x was a major upgrade to OpenZFS, and the caching is a bit more aggressive. You can tune this down via zfs set secondarycache=metadata <name of your pool>. The default is to cache everything, which duplicates ARC.

 

[moraga695]

Hello Samuel,

Where should this command go? If I open the shell in the GUI is says changes there are not persistent. But, it doesn't look like this can be entered in the Tunables section.

Thanks much.

 

[Samuel Tai]

In the shell. As this is a ZFS flag, this will persist in the ZFS properties of your pool's top-level dataset.

 

[moraga695]

Hello Again,

I found that the command you suggested is persistent when done via the shell. I ran the command "zfs get all tank" (my pool name) after a reboot and it says that secondary cache is set to metadata. However, the drive led is still flashing constantly. I even did a cold boot and that didn't change the constant flashing I'm guessing there must be some other difference between the Freenas 11 and Truenas 12 trains causing this behavior?

 

[Samuel Tai]

The LED could also flash for read activity. What does the disk graph show under Reporting?

 

[dktech]

Hello,
I purchased a new Mini 3.0 X+ system from iXSystems in December 2020. It came with 5 hard drives and two SSD's, one configured as a ZIL and the other as an L2ARC. The system arrived from iXsystems with Freenas 11.3 U5 and all was normal. I soon upgraded it to Truenas Core 12.1 and immediately the LED activity light for the L2ARC SSD started blinking as if there was drive activity 100% of the time. This did not happen with the original 11.3 U5 OS. I've since upgraded it to 12.1.1 and the drive light still blinks 100% of the time.

I'm concerned that if this is actually constant drive write activity, the SSD will have excessive, unnecessary wear. Has anybody else seen this happen? Any ideas?

Thanks very much.

Steve W.

Can I send out Mini 3.0 X+ system Tunables to take a look?

 

[moraga695]

Samuel,

Attached is a screen grab of disk ada6 report, which is the L2ARC SSD. This shows activity over the last hour when the device has been basically idle, yet there is constant write activity and no reads. I have no idea what's doing the writing. As I mentioned in my original post the disk activity did not happen in 11.3-U5, only after upgrade to 12.0-U1

 

[moraga695]

Dktech,

I think you were asking for a copy of my tunables. Attached are two pages with them.

 

[dktech]

Dktech,

I think you were asking for a copy of my tunables. Attached are two pages with them. View attachment 44782 View attachment 44783

Will the option of L2arc in the adjustable parameters make common data be written to L2arc after the system restarts?

 

[dktech]

Will the option of L2arc in the adjustable parameters make common data be written to L2arc after the system restarts?

Persistent L2ARC
Persistent L2ARC, or L2ARC rebuild, is a new feature of TrueNAS-12.0. Upon rebooting your system, the L2ARC gets emptied, but if you enable persistent L2ARC it will populate the cache device mapping after reboot via sysctl.

Persistent L2ARC is disabled by default due to performance impact when rebuilding with large amounts of data. Reactivating persistent L2ARC can result in degraded UI and Middleware performance in specific situations where large amounts of data is loaded into L2ARC and a rebuild is triggered.

 

[Samuel Tai]

Yes, @dktech has pointed out a tunable you have enabled that could be contributing to the issue. Please set vfs.zfs.l2arc_rebuild_enabled=0.

 

[moraga695]

Hi All,

I changed vfs.zfs.l2arc_rebuild_enabled=0 which, unfortunately, resulted in no change to the constant blinking of my L2ARC SSD drive activity light or the constant writing of something to the drive. I rebooted the Mini too just in case that was needed - but no change.

 

[Samuel Tai]

Check the SSD's partitioning. Maybe there was a swap partition added?

 

[moraga695]

Update to Truenas Core 12.0-U2 fixed the problem of constant writes to my L2ARC SSD. Since U2 it now behaves as it should.