Total data written on boot-pool drives

[GrimmReaperNL]

A little more than a week ago I changed my boot-pool from one SSD to two new SSD's. Using @joeschmuck script I get daily smart reports via email. In this report the 'Total Data Written' gets reported.
On day 1, 4.5 GB was written. Day 2, 12.7 GB. Day 3, 28.60GB; 46 GB; 63.9 GB; 82.4 GB; 101 GB; 120GB; 139 GB; 156.8 GB; 173.8 GB; and finally today 190.7 GB.
That's what, about 20 GB a day written. What is TrueNAS writing, or does anyone know how I can find out what is being written?
Is these 20GB a lot? Is it normal?

Hope someone can help me get some insight.

 

[NugentS]

Honestly its boot disks - which should be cheap SSD's. Basically disposible. Hell I have used 16Gb Optane's cos they were (and still are, but getting a bit harder to find) so cheap. For a 300TB Endurance drive thats 4 years+ of cheap booting.

Where is the system dataset?

IX have muttered about moving a load of writes onto the boot-pool - but I didn't think they were doing it yet. With the system dataset off the boot-pool I thought that at the moment there should be very few writes

 

[WI_Hedgehog]

I use [expensive] high write volume SSDs in a triple-mirror because:
a.) one will fail at some point for some reason (like cabling)
b.) if there is a data glitch on the remaining drives ZFS will still have good data to use.
This allows me to not worry about sleeping spinners to save power and bearings (SSD don't spin), power consumption, other issues... and access times are instant.

20GB is a lot of writes daily, is there a tunable for logging/other?

 

[GrimmReaperNL]

Honestly its boot disks - which should be cheap SSD's. Basically disposible. Hell I have used 16Gb Optane's cos they were (and still are, but getting a bit harder to find) so cheap. For a 300TB Endurance drive thats 4 years+ of cheap booting.

Where is the system dataset?

IX have muttered about moving a load of writes onto the boot-pool - but I didn't think they were doing it yet. With the system dataset off the boot-pool I thought that at the moment there should be very few writes

I tried using two cheaper ssd's, but they kept giving me problems. Right now I'm using two crusial mx500's. It's hard to find cheap, low capacity ssd's.

What do you mean with 'system dataset'?
I've done a straight install of truenas core, without creating the 16gb swab.

I've not set anything else to be on those drives manually (that I know of).

 

[joeschmuck]

Hope someone can help me get some insight.

I added a special section for when the GrimmReaper uses it, I have it add about 20GB a day just to mess with you. Kidding of course.

Unfortunately I cannot verify this action since I run on a VM and the VM drive does not report that kind of data, only the real drives do.

Does the Used Space change any? Mine does not.

I can see that my fragmentation has gone up by 1% over several months. I'm sitting at 7%. Nothing to worry about however it does indicate that there is writing activity going on.

Checking the System Dataset is easy form the GUI, it will tell you what pool it resides on. Mine is using the default hard drive pool.

 

[Whattteva]

a I tried using two cheaper ssd's, but they kept giving me problems. Right now I'm using two crusial mx500's. It's hard to find cheap, low capacity ssd's.

Sure it is. You're not looking hard enough. I just did a quick search on ebay and found a used Intel DC S3500 120 GB for a mere $14.19 with free shipping. And since this is an enterprise SSD, it is basically designed for even way more writes than 20GB/day.

I got a pair for hosting a few VM's (which writes way more than TrueNAS). One was only at 2% wear, the other at 58%. I got it a year ago and it has only accumulated about 6% wear since. Keep in mind, this is running Proxmox VM's, which typically would kill consumer SSD's rather quickly (usually within 2-3 years). It may be a bit of a luck of the draw to get one with relatively low wear, but even at 60% wear, these enterprise SSD's will probably still outlast the life of your NAS.

 

[GrimmReaperNL]

I added a special section for when the GrimmReaper uses it, I have it add about 20GB a day just to mess with you. Kidding of course.

I knew it!

Unfortunately I cannot verify this action since I run on a VM and the VM drive does not report that kind of data, only the real drives do.

Does the Used Space change any? Mine does not.

I can see that my fragmentation has gone up by 1% over several months. I'm sitting at 7%. Nothing to worry about however it does indicate that there is writing activity going on.

Checking the System Dataset is easy form the GUI, it will tell you what pool it resides on. Mine is using the default hard drive pool.

Where can I check 'used space' and 'fragmentation'? And where in the GUI does it tell me where the system dataset resides on?

Sure it is. You're not looking hard enough. I just did a quick search on ebay and found a used Intel DC S3500 120 GB for a mere $14.19 with free shipping. And since this is an enterprise SSD, it is basically designed for even way more writes than 20GB/day.

Well they're not available on my side of the world. I just googled them and literally no store sell them over here.
Not the 80, 120, 160, 240, or 300 GB models.

 

[joeschmuck]

In the multi_report script email, look under the Zpool section for the fragmentation data and used space data.
In the GUI for CORE -> System -> System Dataset.

If I could email you a few SSD's, I would, but I can't get them to fit into the Outbox.

 

[Whattteva]

Well they're not available on my side of the world. I just googled them and literally no store sell them over here.
Not the 80, 120, 160, 240, or 300 GB models.

Not stores. You want the used markets. Check Facebook Marketplace, ebay, or other similar sites local to your region.

 

[GrimmReaperNL]

In the multi_report script email, look under the Zpool section for the fragmentation data and used space data.

Christ, that way to long to find.
I started with 2.36GB used with 0% fragmentation. Right now it's up to 4.05GB with 1% fragmentation.
So used space doesn't seem to increase a lot.

In the GUI for CORE -> System -> System Dataset.

Think I found it, it's a bit different under Scale. System dataset is set to 'boot-pool'. With a swap size of 2GiB.

If I could email you a few SSD's, I would, but I can't get them to fit into the Outbox.

Now this would be fancy

 

[joeschmuck]

So used space doesn't seem to increase a lot.

That is what I was thinking too and it makes sense if it's just changing a set amount of data and overwriting it often.

Think I found it, it's a bit different under Scale. System dataset is set to 'boot-pool'. With a swap size of 2GiB.

So that could be the issue, probably need to move the System Dataset to the data pool or live with it on the boot-pool.

Now this would be fancy

Star Trek had communicators, now we have smartphones, Star Trek had medical beds that could scan you, we have MRI, CAT SCAN, and others. And I know I saw a guy hop into a container and on the other side of the room he popped out as the largest ugliest damn fly I've ever seen. When they fix the fly thing, then we can fax you a few drives.

 

[GrimmReaperNL]

So that could be the issue, probably need to move the System Dataset to the data pool or live with it on the boot-pool.

Anyone done this before? Do I need to create a dataset first to move it to, or does TrueNAS deal with that itself?

 

[NugentS]

Anyone done this before? Do I need to create a dataset first to move it to, or does TrueNAS deal with that itself?

Just change it - no need to worry about things.
However I would leave it where it is and just not worry about things.
Enterprise drives are overkill for the boot-pool (or in some cases too small to be any use for anything else)

 

[joeschmuck]

I too would end up leaving it on the boot-pool since it's there already, however moving it is very easy as well if you really want to do that. The User Guide holds all the answers, or you can just go to the GUI and find out it's straight forward. I agree with @NugentS unless there is a reason you would prefer it off the boot-pool.

 

[WI_Hedgehog]

Off the boot pool is good when the destination drives are always spinning and have good data integrity.

On the boot pool is good if you want to sleep the data drives, increase integrity (though this is minor), decrease complexity (slightly), and/or the boot pool is SSD and data pool(s) are spinners.

In the end what's best depends on your individual goals.

 

[GrimmReaperNL]

Alright, thanks for all the insight guys. I'll leave it alone for now.
I was mainly just curious if 20GB daily written was something to be concerned about.