Cannot delete snapshots - dataset is busy

[danwestness]

I have some of my apps mapped to host paths such as this:

I have a periodic snapshot task taking a daily snapshot with a 2 WEEK retention for this dataset
I noticed that the snapshots were accumulating far longer than 2 weeks as I have snapshots going back 9+ months at this point

I attempted to delete these old snapshots manually but get "dataset is busy" and it will not delete them. Assumably because the dataset is is use by kubernetes.
I tested this by first attempting to delete snapshots related to a dataset associated with an app's hotspath mapping and it failed. If i delete the app (or unmap the mapping to the path) then i can delete the snapshots.

Since there is no way to really just "Stop" the apps service that i have found without completely unsetting the pool and reconfiguring it, The only way I have been able to delete the snapshots is the reboot truenas and then there is a momentary windows of opportunity where the system will allow me to delete the snapshots (100 at a time maximum as the UI only allows this) before the application service starts up and locks them all again.

Since i have 1800 snapshots this is going to take me rebooting 18 times and quickly trying to delete snapshots during the initial boot up phases which of course is not sustainable and doesn't help me going forward not having these snapshots accumulate forever.

What is another apporach i can take here to have snapshots and a retention cycle for my application data?

 

[ABain]

I've tried to reproduce the above today but so far I am able to delete the snapshots created from a dataset which is assigned in the host path of an app. The only scenario where I could get the same error was where I had selected the Hold option on the snapshot, i assume this is not selected on your snapshots. Are there any other details you can share e.g. was the snapshot configured on a prior version before upgrade?