Your system has more snapshots than recommended (10000)

[CheeryFlame]

Hello,

I'm having this message again although I thought that I had setup my snapshots correctly this time.

Your system has more snapshots (10958) than recommended (10000). Performance or functionality might degrade.​

I've looked Under Left Menu > Datasets and clicked on each one of them and they were all showing between 0-30 maximum as you can see on this example.

I don't have 50x datasets but if I would it would make 1,500 snapshots for a period of 30 days which is far from the max 10,000 snapshots.

Here's a screenshot of my periodic snapshots tasks

I suspect there might be old snapshots that I don't want but it's too slow to navigate in Storage > Snapshots. Also I'm afraid I could delete something that I shouldn't be. Is there a command or a script that looks at your current snapshot tasks and will delete all unrelated snapshots part of older tasks?

Thanks for helping me!

 

[winnielinnie]

These are likely snapshots under the ix-applications dataset.

Probably a "feature" of SCALE.

 

[CheeryFlame]

I just checked and I have 10,634 snapshots for ix-applications.

1. Pollen/ix-applications have 5169 snapshots
2. Eden/backups/zappa/ix-applications have 5,077 snapshots.
3. Eden/backups/indiscipline/ix-applications have 388 snapshots

Which represent

1. ix-applications vdev
2. Local backup of ix-applications on another vdev
3. Backup of my 2nd Scale install ix-applications

I guess this alert isn't to be ignored. I'll try and reduce my ix-applications backups to 2 weeks.

The fact app backups relies on snapshots on Scale isn't practical and should be reworked from ground up.

 

[winnielinnie]

Pollen/ix-applications have 5169 snapshots

"It's a feature!"
-TrueNAS SCALE

 

[CheeryFlame]

I just checked and I have 10,634 snapshots for ix-applications.

1. Pollen/ix-applications have 5169 snapshots
2. Eden/backups/zappa/ix-applications have 5,077 snapshots.
3. Eden/backups/indiscipline/ix-applications have 388 snapshots

Which represent

1. ix-applications vdev
2. Local backup of ix-applications on another vdev
3. Backup of my 2nd Scale install ix-applications

Just setup 2 weeks instead of a month of backups and getting a total of 7,108 snapshots for ix-applications.

This is for approximately 40 apps. I still have to deploy many more apps.

Seems like Scale has a limit of apps.

 

[sfatula]

I use HeavyScript to worry about ix-applications snapshots. It keeps them under control.

 

[joeschmuck]

"It's a feature!"
-TrueNAS SCALE

And it was no extra cost.

 

[CheeryFlame]

I use HeavyScript to worry about ix-applications snapshots. It keeps them under control.

I use heavyscript and it has nothing to do with it. I fixed the issue reducing the number of backups to 14 days instead of a month. Although this may not work if I ever get more applications.

 

[winnielinnie]

I fixed the issue reducing the number of backups to 14 days instead of a month.

To reach 5,000+ snapshots for a dataset/children is not due to a backup schedule. SCALE is automatically creating a trove of snapshots for the ix-applications dataset, which is the underlying reason why you have too many snapshots.

I'm not sure what the rationale is, let alone if it's being resolved? (Others have noticed their SCALE systems filling up with thousands of snapshots.)

 

[CheeryFlame]

To reach 5,000+ snapshots for a dataset/children is not due to a backup schedule. SCALE is automatically creating a trove of snapshots for the ix-applications dataset, which is the underlying reason why you have too many snapshots.

I'm not sure what the rationale is, let alone if it's being resolved? (Others have noticed their SCALE systems filling up with thousands of snapshots.)

Is there an active ticket on Jira or a feature request somewhere so they could change how ix-applications is currently creating all this sub sets of snapshots. I really doubt there's a pro for it and it makes backing up applications a pain. 1 snapshot per application would be great and with the possibility to restore one app at a time instead of the whole ix-applications folder.

 

[winnielinnie]

Is there an active ticket on Jira or a feature request somewhere so they could change how ix-applications is currently creating all this sub sets of snapshots.

Ticket created in March 2022: https://ixsystems.atlassian.net/browse/NAS-115386

 

[sfatula]

I use heavyscript and it has nothing to do with it. I fixed the issue reducing the number of backups to 14 days instead of a month. Although this may not work if I ever get more applications.

It DOES have something to with it it as you shouldn't run any snapshots of ix-applications datasets yourself which is what I am assuming you are doing. If that assumption is wrong, then you're right. Heavyscript deletes it's backups of ix-applications based on the retention given in the command. As part of running heavyscript you should be doing a prune also. Been running scale for more than a year and I have < 1000 snapshots.

More discussion on how to clean those up can be found here:

SOLVED - [SCALE] Mass cleanup/removal of ix-applications snapshots solution required

As I've explained in another post/thread here, there is an issue with snapshots being created for ix-applications dataset and its child datasets. I mean in under a year with moderate to light use, my TrueNAS Scale server has over 47 thousand snapshots created. Yeah, you read that right - it's...

www.truenas.com

 

[winnielinnie]

@sfatula, the underlying problem is that the default behavior for SCALE creates way too many snapshots for ix-applications. So much so, that it even triggers its own alert of "Too many snapshots on this system!"

This needs to be addressed, rather than place the burden on the admin to use a third-party solution or manual intervention.

 

[CheeryFlame]

Thank you, I added a word over there but I'm no expert...

 

[sfatula]

@sfatula, the underlying problem is that the default behavior for SCALE creates way too many snapshots for ix-applications. So much so, that it even triggers its own alert of "Too many snapshots on this system!"

This needs to be addressed, rather than place the burden on the admin to use a third-party solution or manual intervention.

Thank you, however, I am fully aware of that. I do not have the issue due to the suggestions from the other thread, it's gone but I used to have the problem. So, my point in all this was it can be mitigated. If you read the thread, many people had success with removing the vast majority of those snapshots on an ongoing basis with very simple changes.

I agree in the end that ix should solve the problem. But in the meantime, I simply spent a minute making the needed changes.

 

[CheeryFlame]

Thank you, however, I am fully aware of that. I do not have the issue due to the suggestions from the other thread, it's gone but I used to have the problem. So, my point in all this was it can be mitigated. If you read the thread, many people had success with removing the vast majority of those snapshots on an ongoing basis with very simple changes.

I agree in the end that ix should solve the problem. But in the meantime, I simply spent a minute making the needed changes.

I'd rather not touch the ix-applications snapshots at all to be honest. I had to recover from backup after a re-silvering failure and I thank heavyscripts to exists otherwise it would have been a nightmare to figure it all by myself and I could have say goodbye to my CNPG apps. I feel like deleting snapshots from the ix-systems dataset is a very bad idea.

 

[sfatula]

I'd rather not touch the ix-applications snapshots at all to be honest. I had to recover from backup after a re-silvering failure and I thank heavyscripts to exists otherwise it would have been a nightmare to figure it all by myself and I could have say goodbye to my CNPG apps. I feel like deleting snapshots from the ix-systems dataset is a very bad idea.

Right, you shouldn't be creating any yourself via data protection, and, heavyscript can manage those snapshots. It's not a bad idea at all but up to you. IX has commented on this. Enough said.

 

[CheeryFlame]

Right, you shouldn't be creating any yourself via data protection, and, heavyscript can manage those snapshots. It's not a bad idea at all but up to you. IX has commented on this. Enough said.

Sorry I didn't meant to annoy you. I'm relatively new to this and from my understanding I have nothing else setup that would create snapshots of ix-applications apart of heavyscript itself. Again, from my understanding heavyscript setups the equivalent of a recursive zfs replication on the ix-applications dataset. All of these snapshots can't be avoided since they're created from the current dataset layout of ix-apps.

If there's anything I've missed I'd really like to know and in fact reduce my snapshots count.

 

[sfatula]

Well, curious how many snapshots HeavyScript has created for you, I don't see your command line to run it posted anywhere in this thread. So, from a Truenas shell, do:

zfs list -t snapshot | grep HeavyScript | wc -l

If that number is in the thousands, would be useful to see your HeavyScripts invocation, what options you are using. Mine totals to 434 with ~15 apps.

You can get total snapshots via:

zfs list -t snapshot | wc -l

Mine here totals to 856, and many of those are 3 different snapshots of my various datasets for backups. 2 for rotating offsite backups, and another for offsite replication.

 

[CheeryFlame]

Well, curious how many snapshots HeavyScript has created for you, I don't see your command line to run it posted anywhere in this thread. So, from a Truenas shell, do:

zfs list -t snapshot | grep HeavyScript | wc -l

If that number is in the thousands, would be useful to see your HeavyScripts invocation, what options you are using. Mine totals to 434 with ~15 apps.

You can get total snapshots via:

zfs list -t snapshot | wc -l

Mine here totals to 856, and many of those are 3 different snapshots of my various datasets for backups. 2 for rotating offsite backups, and another for offsite replication.

Here's my heavyscript cron job

Code:

bash /root/heavy_script/heavy_script.sh update --backup 14 --concurrent 10 --prune --rollback --sync --self-update

I don't think it varies much or at all from the original one shared on his Github.

Perhaps I have residual old snapshots I could delete?