Some insights into SLOG/ZIL with ZFS on FreeNAS

[jgreco]

<div class="bbWrapper"><blockquote data-attributes="member: 50882" data-quote="wrath" data-source="post: 212550" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=212550" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-212550">wrath said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Running a vmware esxi on zfs without breaking the bank is not possible. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> It's all relative. If you go look at what the cost for running enterprise-grade storage for multiple ESXi hosts is, FreeNAS and ZFS come in looking very attractive indeed.</div>

 

[wrath]

<div class="bbWrapper">The math for hdd that does 150mb/s and having 4 of them makes it 600mb/s or 3gb/5secs. Considering 1/8 of ram is used for the transaction group. Having 16gb ram would give transaction group 2gb of ram. How does this scale up to lets say 6 disks at 150mb/s in raidZ2. Would it simply be 900mb/s or 4.5 gb/5s. So 1/8 of 32gb would be 4gb. Would 6 disk setup justify more ram without sacrificing performance? If you use 16gb ram for 6 disk setup, you essentially cut the performance in half. Am I right?</div>

 

[mjws00]

<div class="bbWrapper">You are mixing and matching concepts. You have pool speed limits correct. RAM is primarily used for ARC. Basically it boils down to the larger the ARC the higher the chance you will be able to read from it at full memory speed bypassing the pool. Transaction size is not terribly relevant. Except as a means to calculate the maximum volume written to the zil before we must flush to disk. Sync writes get limited by latency and iops. So we add a super fast low latency device so we can return confirmation of the block write to disk and get on to the next one.<br /> <br /> L2ARC is the device that affects low RAM systems as it can displace critical ARC info and end up hurting more than helping.<br /> <br /> Little simplified as I'm on phone.</div>

 

[sfcredfox]

<div class="bbWrapper">After reading this a few times, I believe I understand each pool has it's own ZIL. So in an example system having two pools on separate disk sets, if the first pool is built for a virtualization data store and has a heavy SYNC workload, this pool is expected to benefit having a SLOG. If the SAME system also has a second pool for everything else, and is built with the expectation it will NOT likely have a heavy SYNC load, it also has it's own ZIL and the default in pool ZIL could be fine (all things situationally workload dependent)?<br /> <br /> <u>Bottom line, every pool has it's own ZIL?</u><br /> <br /> I expect if you had two pools with heavy SYNC loads, you would want TWO distinct SLOGs so the two pools don't contend with each other for SLOG access?<br /> <br /> <u>Lastly, how does the calculation of the transaction groups account for having multiple pools?</u><br /> <br /> If having your transaction group be too large for your system's hardware to write it before the second transation group needs to be written, how is this effected by having two pools?<br /> <br /> Is there a transaction group for each pool? Or is there a single transaction group for all data intended for any pool? <br /> Example: If one pool of disks is slower and has lots of writes being committed, would that lock up IO intended for a second pool that is very fast and has no problems committing it's transaction groups to disk?<br /> <br /> How would you calculate the max transaction group size on a system with multiple pools?</div>

 

[Mirfster]

<div class="bbWrapper">Okay, so is there any correlation to SLOG and the Swap size on each hard drive? Meaning if one were to have a pair of SSDs (mirrored) allocated for a SLOG then would it be advisable to set the &quot;Swap size on each drive in GiB, affects new disks only&quot; to &quot;0&quot; (zero)? Or would FreeNas simply ignore that swap and use it in the event that the mirrored SLOG went down?<br /> <br /> Feel free to smack me if I am way off base here. ;)</div>

 

[depasseg]

<div class="bbWrapper"><blockquote data-attributes="member: 53160" data-quote="Mirfster" data-source="post: 284841" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=284841" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-284841">Mirfster said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> would it be advisable to set the &quot;Swap size on each drive in GiB, affects new disks only&quot; to &quot;0&quot; (zero)? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I don't think it ever makes sense to change swap to 0. If only for the fact that it allows you to replace a failed disk with one that isn't quite the same size.<br /> <br /> <blockquote data-attributes="member: 53160" data-quote="Mirfster" data-source="post: 284841" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=284841" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-284841">Mirfster said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> and use it in the event that the mirrored SLOG went down? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>If the SLOG fails, it will directly use the pool instead (not swap).</div>

 

[Mirfster]

<div class="bbWrapper"><blockquote data-attributes="member: 43399" data-quote="depasseg" data-source="post: 284854" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=284854" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-284854">depasseg said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I don't think it ever makes sense to change swap to 0. If only for the fact that it allows you to replace a failed disk with one that isn't quite the same size. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Thanks I will leave that alone, I was leery about wanting to muck with that.</div>

 

[jgreco]

<div class="bbWrapper"><blockquote data-attributes="member: 43399" data-quote="depasseg" data-source="post: 284854" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=284854" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-284854">depasseg said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I don't think it ever makes sense to change swap to 0. If only for the fact that it allows you to replace a failed disk with one that isn't quite the same size. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> That makes sense for the pool data drives, but it could possibly be argued that drives such as SLOG and L2ARC are of a somewhat different class of drive. The ZFS can-never-remove-a-vdev thing means that once added, you MUST have that drive in the pool. The others, however, are not, and have a less permanent, potentially ephemeral nature to them.<br /> <br /> For whatever it is worth, I think the FreeNAS swap strategy is vaguely flawed, because the swap isn't protected against drive failure, and perhaps isn't sized to scale.<br /> <br /> However, I cannot think of a significant value to be gained trying to play games with the swap size on something like a SLOG device, unless perhaps you've got something like a small battery backed RAM SSD where the size of the swap would be taking a considerable portion of the device.</div>

 

[depasseg]

<div class="bbWrapper">I didn't realize that slog and l2arc devices were used for swap. My comment was only referring to data drives.</div>

 

[jgreco]

<div class="bbWrapper"><blockquote data-attributes="member: 43399" data-quote="depasseg" data-source="post: 284928" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=284928" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-284928">depasseg said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I didn't realize that slog and l2arc devices were used for swap. My comment was only referring to data drives. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> Learn something new every day. <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite1" alt=":smile:" title="Smile :smile:" loading="lazy" data-shortname=":smile:" /></div>

 

[depasseg]

<div class="bbWrapper">Or I become more confused every day....<br /> ada0 is my SLOG and ada1 is my L2ARC. I don't see swap on them.<br /> <div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>[root@freenas1] ~# gpart show =&gt; 34 31249933 ada0 GPT (14G) 34 94 - free - (47k) 128 31249832 1 freebsd-zfs (14G) 31249960 7 - free - (3.5k) =&gt; 34 390721901 ada1 GPT (186G) 34 94 - free - (47k) 128 390721800 1 freebsd-zfs (186G) 390721928 7 - free - (3.5k) =&gt; 34 234441581 ada2 GPT (111G) 34 1024 1 bios-boot (512k) 1058 6 - free - (3.0k) 1064 234440544 2 freebsd-zfs (111G) 234441608 7 - free - (3.5k) =&gt; 34 234441581 ada3 GPT (111G) 34 1024 1 bios-boot (512k) 1058 6 - free - (3.0k) 1064 234440544 2 freebsd-zfs (111G) 234441608 7 - free - (3.5k) =&gt; 34 7814037101 da0 GPT (3.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 7809842696 2 freebsd-zfs (3.7T) 7814037128 7 - free - (3.5k) =&gt; 34 7814037101 da1 GPT (3.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 7809842696 2 freebsd-zfs (3.7T) 7814037128 7 - free - (3.5k) =&gt; 34 7814037101 da2 GPT (3.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 7809842696 2 freebsd-zfs (3.7T) 7814037128 7 - free - (3.5k) =&gt; 34 7814037101 da3 GPT (3.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 7809842696 2 freebsd-zfs (3.7T) 7814037128 7 - free - (3.5k)</pre></div></div>

 

[Mirfster]

<div class="bbWrapper"><blockquote data-attributes="member: 43399" data-quote="depasseg" data-source="post: 285073" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=285073" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-285073">depasseg said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Or I become more confused every day... </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Join the club. I am just experimenting at this point and learning. Pictures and a Flow Chart would help me. ;)</div>

 

[Mirfster]

<div class="bbWrapper"><blockquote data-attributes="member: 43399" data-quote="depasseg" data-source="post: 285073" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=285073" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-285073">depasseg said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> ada0 is my SLOG and ada1 is my L2ARC. I don't see swap on them. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Interestingly enough on a current test machine the two drives dedicated for SLOG do show a swap. These two drives (da2 &amp; da3) are mirrored so maybe that has something to do with it.<br /> <br /> Also, this is not a &quot;standard setup&quot;, it is a <u><b>test system</b></u> for FreeNas running on ESXi 6.0.<br /> The two SSDs (Intel 320 32GB) are actually being served to the FreeNas VM via virtual drives which are attached to a LSI 9260-8i (each is a RAID0) with &quot;write back w/bbu&quot;. ***Testing out a variant of having a SLOG with BBU... ;)<br /> <br /> Call it my variant of:<br /> <blockquote data-attributes="member: 115" data-quote="jgreco" data-source="post: 64328" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=64328" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-64328">jgreco said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> An interesting but unorthodox alternative for SLOG is to use a RAID controller with battery backed write cache, along with conventional hard disks. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> <div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>[root@ASC-FN01] ~# gpart show =&gt; 34 67108797 da0 GPT (32G) 34 1024 1 bios-boot (512K) 1058 6 - free - (3.0K) 1064 67107760 2 freebsd-zfs (32G) 67108824 7 - free - (3.5K) =&gt; 34 67108797 da1 GPT (32G) 34 1024 1 bios-boot (512K) 1058 6 - free - (3.0K) 1064 67107760 2 freebsd-zfs (32G) 67108824 7 - free - (3.5K) =&gt; 34 41942973 da2 GPT (20G) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 37748568 2 freebsd-zfs (18G) 41943000 7 - free - (3.5K) =&gt; 34 41942973 da3 GPT (20G) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 37748568 2 freebsd-zfs (18G) 41943000 7 - free - (3.5K) =&gt; 34 3907029101 da4 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da5 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da6 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da7 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da8 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da9 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da10 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da11 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da12 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da13 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da14 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) =&gt; 34 3907029101 da15 GPT (1.8T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5K) </pre></div></div>

 

[depasseg]

<div class="bbWrapper">Hmm, that's strange. I installed this system on 9.3 back around ~Jan 2015 (not 9.3.1) and have been upgrading it since. I'm still on 9.3.1. though, not 9.10. Maybe that has something to do with it?</div>

 

[Mirfster]

<div class="bbWrapper">Going to destroy and re-create the volume (doing some speed tests with Mirrors vs RaidZ2), so I will just add one SLOG and see if it creates a swap or not. Also, you are correct I am running 9.10.<br /> <br /> *** Update: Seems if you are not doing any mirroring for the SLOG then the SWAP is not created on the disk<br /> <div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre> =&gt; 34 41942973 da2 GPT (20G) 34 94 - free - (47K) 128 41942872 1 freebsd-zfs (20G) 41943000 7 - free - (3.5K) </pre></div></div>

 

[Mirfster]

<div class="bbWrapper">Out of curiosity, if all other components are equal and I was going to connect an Intel SSD DC S3500 (160 GB) as a SLOG which would be preferable?<br /> <br /> Directly to a SATA Port (3 Gb/s) or an HBA (6 Gb/s)?<br /> <br /> *** Note: Both the MB SATA Ports and HBA are being passed-through to a FreeNAS VM running on ESXi 6.0 U2<br /> <br /> I am leaning towards the SATA Port simply to circumvent all the extra traffic, but wanted to check my sanity...</div>

 

[Spearfoot]

<div class="bbWrapper"><blockquote data-attributes="member: 53160" data-quote="Mirfster" data-source="post: 305378" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=305378" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-305378">Mirfster said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Out of curiosity, if all other components are equal and I was going to connect an Intel SSD DC S3500 (160 GB) as a SLOG which would be preferable?<br /> <br /> Directly to a SATA Port (3 Gb/s) or an HBA (6 Gb/s)?<br /> <br /> *** Note: Both the MB SATA Ports and HBA are being passed-through to a FreeNAS VM running on ESXi 6.0 U2<br /> <br /> I am leaning towards the SATA Port simply to circumvent all the extra traffic, but wanted to check my sanity... </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I recommend attaching it to the HBA. You want a SLOG device to have really fast write speeds with low latency, so, all else being equal, it would probably be faster connected to a 6Gb/s port vs. a 3Gb/s port. Or so it seems to my little pea brain. <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite1" alt=":smile:" title="Smile :smile:" loading="lazy" data-shortname=":smile:" /><br /> <br /> Here is an interesting <a href="https://b3n.org/ssd-zfs-zil-slog-benchmarks-intel-dc-s3700-intel-dc-s3500-seagate-600-pro-crucial-mx100-comparison/" target="_blank" class="link link--external" rel="noopener">review of SSD SLOG devices</a> which shows that the S3500 is a pretty good contender. My understanding is that the S3700 SSD is a better choice because it's optimized for writes and has higher durability. I use S3700 SSDs as SLOG devices in my systems.</div>

 

[Mirfster]

<div class="bbWrapper">Yeah, I do have a couple S3710s (200 GB) that I could use as well. But, for now will stick with the S3500 for testing.<br /> <br /> While the HBA (Perc H200 Cross-Flashed) will provide 6Gb/s, it will take up one of my hard drive bays. Not a big deal, but I would prefer otherwise. Also, connecting it to a HBA would introduce a little latency (assumed)... <br /> <br /> Would I be incorrect in assuming that neither a S3500 or S3710 could really saturate a 3Gb/s connection?</div>

 

[depasseg]

<div class="bbWrapper">I think the difference is negligible, and if you don't want to lose a drive bay for the HBA, then go with SATA.<br /> I don't think the HBA will add any more latency than the SATA controller.<br /> I think the 6Gb/s will have less latency than the 3Gb/s, even if it doesn't saturate.<br /> <br /> Interesting reads:<br /> <a href="http://docs.avagotech.com/docs/12355775" target="_blank" class="link link--external" rel="nofollow ugc noopener">http://docs.avagotech.com/docs/12355775</a><br /> <a href="http://www.dell.com/downloads/global/products/pvaul/en/6gb-sas-transition.pdf" target="_blank" class="link link--external" rel="nofollow ugc noopener">http://www.dell.com/downloads/global/products/pvaul/en/6gb-sas-transition.pdf</a></div>

 

[Mirfster]

<div class="bbWrapper">Thanks for the input. I may have to test both and see if I notice anything.</div>