Optimal Cache and log configuration

[Isma]

Hello, for some time I have been reading threads about whether it is better to have a read cache and a write cache.
Based on the fact that Truenas uses the RAM as its own cache, and the cache log is only useful for certain features or needs or if synchronization is always possible. I don't know if the following doubts about my "home" system compensate me.

My architecture is the following:
- 8 WD Red HDDs of 14 TB in RAIDZ2 for data storage
- 5 10TB WD White HDDs in RAIDZ1 for app data storage and other functions such as nextcloud user data for example
- 5 DC600M SSD of 960 GB in RAIDZ1 for VM
- 1 SAMSUNG PM883 1.92 TB SSD for apps
- 1 NVMe Kingston A2000 for system and boot only
- 2 NVMe Crucial P3 Plus 500GB purchased, 1 for read cache and another for write cache. (1 Cache/1 log)

The NAS has 128 GB of DDR4 RAM at 3200
The processor is a AMD Ryzen 7 5700G with Radeon Graphics
The board has two RJ45 connections of 1 gb/s and another of 10 gb/s (cat.8 cable)


One of the doubts I have is whether it is worth having those two NVMEs for cache or log or better to take advantage of them to set up another pool.


Thank you so much

 

[danb35]

I hope your "raid 5"/"raid 6" mentions are actually RAIDZ1/RAIDZ2, not some other form of RAID that you're running under ZFS. With that said...

The Crucial SSDs you mention are completely unsuitable for use as SLOG, but it's also unclear that you have any need for SLOG at all. As to whether you need L2ARC, check the reporting for your hit ratio.

 

[Patrick M. Hausen]

SLOG is not a write cache. Looking at your workloads it will hardly get used at all.
With 128 G of memory I would watch the ARC hit rate before thinking about adding an L2ARC.

Please try to use the correct terms when describing your intended setup. There is no write cache device in ZFS, it's always in memory, even with an SLOG. Also there is neither RAID5 nor RAID6 ...

 

[Isma]

SLOG is not a write cache. Looking at your workloads it will hardly get used at all.
With 128 G of memory I would watch the ARC hit rate before thinking about adding an L2ARC.

Please try to use the correct terms when describing your intended setup. There is no write cache device in ZFS, it's always in memory, even with an SLOG. Also there is neither RAID5 nor RAID6 ...

Basically, is this profitable to have?

My idea was to have a cache just so that if the HDDs were saturated, I could continue receiving information at a good speed and have a cache for quick access

I hope your "raid 5"/"raid 6" mentions are actually RAIDZ1/RAIDZ2, not some other form of RAID that you're running under ZFS. With that said...

The Crucial SSDs you mention are completely unsuitable for use as SLOG, but it's also unclear that you have any need for SLOG at all. As to whether you need L2ARC, check the reporting for your hit ratio.

Yes sorry RAIDZ1/RAIDZ2
fixed the bug

 

[Patrick M. Hausen]

Basically, is this profitable to have?

For writes via apps like Nextcloud an SLOG will not be used at all. Neither for writes via SMB. If the L2ARC is being used can be checked with arcstat. If you have a near 100% (> 99) ARC hit rate, L2ARC is also practically unused. You could even profit from removing it because the management of the L2ARC consumes memory that could better be used as cache.

 

[ChrisRJ]

My idea was to have a cache just so that if the HDDs were saturated, I could continue receiving information at a good speed and have a cache for quick access

There seems to be a misunderstanding about how a cache works in general. What it does is to receive a "copy" of the data while they are being read from the HDDs. If in a short enough time (before eviction kicks in) the same data are requested again, they will be served from the cache.

Conversely, something that has not been read before from the HDD cannot be served from the cache (be it RAM or L2ARC). So a cache is not a magical booster for read speed.

And the fastest cache (in this context) is RAM. With 128 GB I cannot think of a workload that benefits from an additional L2ARC in a private setting. As @Patrick M. Hausen already wrote, you need to watch your ARC stats to develop a better understanding.

Unless you are willing to throw ridiculous amounts of money at the problem, there is no "one size fits all" for storage performance optimization. It is the workload the determines what helps and what doesn't.

 

[Davvo]

My idea was to have a cache just so that if the HDDs were saturated, I could continue receiving information at a good speed and have a cache for quick access

What's you use case? If your drives are saturated the culprit is likely the IOPS, so you would want to switch to a mirror layout.

 

[Isma]

There seems to be a misunderstanding about how a cache works in general. What it does is to receive a "copy" of the data while they are being read from the HDDs. If in a short enough time (before eviction kicks in) the same data are requested again, they will be served from the cache.

Conversely, something that has not been read before from the HDD cannot be served from the cache (be it RAM or L2ARC). So a cache is not a magical booster for read speed.

And the fastest cache (in this context) is RAM. With 128 GB I cannot think of a workload that benefits from an additional L2ARC in a private setting. As @Patrick M. Hausen already wrote, you need to watch your ARC stats to develop a better understanding.

Unless you are willing to throw ridiculous amounts of money at the problem, there is no "one size fits all" for storage performance optimization. It is the workload the determines what helps and what doesn't.

I explained it wrong, I always thought that the cache is a "memory" that is speaking poorly and quickly, between the request for information and the HDD in this case, then that information is stored and when the same thing is consulted again, the information is already It will go from the cache and no request is made to the HDD, that is clear.
Now the question is the following: I want to upload 150gb of backup movies, videos, etc. to my raidz1 for example, 150gb to assume a figure, well, the information will begin to be uploaded with a 10gb/s connection and the disks will write for a few seconds until fill your cache at 1gb/s and then they will drop to the maximum speed of the hdd, let's say 250mb/s for "asynchronous synchronization", at this point is where I ask if the l2arc would be useful, if I use 128gb of ram, what free between system, operations, Vm etc, suppose that 60gb of ram is free, if I now want to put 500gb in a l2arc nvme and information continues to enter the hdd, I would like to know if asynchronous synchronization is still optimal, or the optimal thing would be to always synchronize it , and be that as it may, the data will be copied first to the l2arc, storing an amount of data until the nvme is full while it is emptied little by little to the HDD maintaining a writing speed of 1gb/s, in short, doing the same as the cache but first storing the data that is going to be written to the HDD.

 

[Patrick M. Hausen]

@Isma What you are describing is a hierarchical storage management system or HSM. ZFS does nothing of that sort, unfortunately. Specifically for writes it never copies data around. The SLOG is written to so the synchronous write operation can be acknowledge, then the data is still written from RAM to the HDD pool. The SLOG is never read, unless the system crashes. And it never holds more than five seconds (or was that twice five seconds) worth of data.

 

[Isma]

@Isma What you are describing is a hierarchical storage management system or HSM. ZFS does nothing of that sort, unfortunately. Specifically for writes it never copies data around. The SLOG is written to so the synchronous write operation can be acknowledge, then the data is still written from RAM to the HDD pool. The SLOG is never read, unless the system crashes. And it never holds more than five seconds (or was that twice five seconds) worth of data.

So it's only worth it as a pure read cache as always?

 

[Patrick M. Hausen]

No, SLOG persists synchronous writes so you won't lose those 5 seconds of data "in flight" when the power goes out. That is only useful in special applications that use synchronous writes like block storage for a hypervisor.

 

[Isma]

No, SLOG persists synchronous writes so you won't lose those 5 seconds of data "in flight" when the power goes out. That is only useful in special applications that use synchronous writes like block storage for a hypervisor.

Come on, what I call and use a UPS, it would be an interesting function if a "write cache" could be implemented, but thank you very much for helping me understand this topic

 

[Patrick M. Hausen]

System crash? Network outage? Must not be a power loss ...

 

[Davvo]

You can always contribute to the code.

 

[Patrick M. Hausen]

You can always contribute to the code.

An HSM is not a small feat I would easily trust to a single volunteer no matter how enthusiastic. Then again Pawel started the entire "let's port this ZFS thing from Solaris to FreeBSD" journey on his own, first.

 

[Davvo]

An HSM is not a small feat I would easily trust to a single volunteer no matter how enthusiastic.

That's an issue for those who decide whether to approve the merging or not
A step is better than not moving, it might inspire others or cause a reaction.