Persistent storage - how to approach systematically?

[Patrick M. Hausen]

Hi all,

I am running a hybrid ESXi/SCALE system out of curiosity. People who know me will be aware that I still advise to use jails over any other container technology. Adding mountpoints to a FreeBSD jail is easy, data is kept separate ... like it should be.

That being said after running only apps that don't keep any state - OnlyOffice is the most important one for me - I'm lured into the Linux container ecosystem by the easy availability of Home Assistant and Uptime Kuma. I see that the standard TrueCharts (I'm currently using TrueCharts for all applications) charts use something named Persistent Volume Claims (PVC) to persist data across updates. At least I hope that is what these things do

The UI warns not to use anything else or rollbacks may fail.

Can someone point me to some documentation about what is going on under the hood here? If I come to rely on e.g. the Home Assistant installation (really I live in a 4-room appartment and I have only two sensors: the power consumption of my home lab and the power production of my PV) would it be advisable to create a dataset and mount it inside the container? How do I go about finding out what to mount where?

With docker compose things were simple:

Code:

services:
  truecommand:
    volumes:
      - '/app/ixsystems/truecommand/data:/data'

And that was that. Much like a mount into a jail. How do I tackle this when I start to use more of the charts and come to rely on the data they create?

Kind regards and thanks,
Patrick

 

[samarium]

I've been watching this eco-system, but the entire TNS app system is still too flakey at the moment for me to want to trust it. The Truecharts discord has a heavyscript channel, for the heavyscript tool, which you can pull the source code, IIRC from github. If you examine the tool code there are some ideas of what is going on inside and what has to be done to make and restore backups, both the PVC and the TNS app configuration. I don't know if there is any other documentation, but you can at least interact with the author thru that discord channel.

 

[morganL]

Hi all,

I am running a hybrid ESXi/SCALE system out of curiosity. People who know me will be aware that I still advise to use jails over any other container technology. Adding mountpoints to a FreeBSD jail is easy, data is kept separate ... like it should be.

That being said after running only apps that don't keep any state - OnlyOffice is the most important one for me - I'm lured into the Linux container ecosystem by the easy availability of Home Assistant and Uptime Kuma. I see that the standard TrueCharts (I'm currently using TrueCharts for all applications) charts use something named Persistent Volume Claims (PVC) to persist data across updates. At least I hope that is what these things do

The UI warns not to use anything else or rollbacks may fail.

Can someone point me to some documentation about what is going on under the hood here? If I come to rely on e.g. the Home Assistant installation (really I live in a 4-room appartment and I have only two sensors: the power consumption of my home lab and the power production of my PV) would it be advisable to create a dataset and mount it inside the container? How do I go about finding out what to mount where?

With docker compose things were simple:

Code:

services:
  truecommand:
    volumes:
      - '/app/ixsystems/truecommand/data:/data'

And that was that. Much like a mount into a jail. How do I tackle this when I start to use more of the charts and come to rely on the data they create?

Kind regards and thanks,
Patrick

This extract from docs is worthwhile.... there are two choices for each App. Host path volumes is more like Jails storage.

Setting Up Persistent Volume Access​

Users developing applications should be mindful that if an application uses Persistent Volume Claims (PVC), those datasets are not mounted on the host and therefore are not accessible within a file browser. Upstream zfs-localpv uses this behavior to manage PVC(s).

If you want to consume or have file browser access to data that is present on the host, set up your custom application to use host path volumes.

 

[Patrick M. Hausen]

I just want the data to persist across app updates and be able to perform backups via snapshots and replication. So how do I find out where e,g. Home Assistant keeps its database? And wasn't setting up a snapshot task for the ix-something dataset a topic of debate lately?

 

[samarium]

I've seen the TC people saying that PVC is required, at least for some data, else certain data like database files become corrupted. I don't know if this is an artifact of not specifying an appropriate recordsize for a host path volume if used in this was, or it is just too difficult to configure to that level of detail via helm and TNS

Things like postgres and mysql have fairly well understood recordsize requirements. I've been looking for guidance for sqlite3 lately, and haven't found anything other that blocksize for sqllite3 was 1kB, and new versions might be 4kB, neither of then is a record size I want to try and use with zfs.

While I'm watching this unfold, I'm doing my own thing with normal x86-64/arm64 linux servers/vms running docker and portainer, and a zfs volume, or more, for each app, possibly with tuned recordsize if I think it warrants it.

I run my own automated, and manual if required, zfs snapshots of the docker datasets, and the app specific datasets, because I want to be able to backup/restore my app data too. Docker volume snapshots I'm ambivalent about, don't really care in general, but for a system rollback better to have the snapshots. Docker configuration is just the docker compose files in portainer, so easy to backup with the other app datasets. There is some other config for docker networks which I have scripted, and also living in the portainer dataset.

I'm not 100% sure this covers all the bases, but I'm 100% sure that there is a lot less frustration than watching the issues, including data loss, seemingly a lot of people have with the current TNS and/or TC app setups, and trying to rebuild/repair.

I've rebuild my config recently. I had to destroy all the docker configuration because docker seems to have a half deleted network than would not go away, so I know my rebuild plan works, rollback need testing.

I would like to see the TNS/TC stuff working well, but I would also like some confidence that my data was safe and recoverable too, and that state is not here yet.

 

[morganL]

I've seen the TC people saying that PVC is required, at least for some data, else certain data like database files become corrupted. I don't know if this is an artifact of not specifying an appropriate recordsize for a host path volume if used in this was, or it is just too difficult to configure to that level of detail via helm and TNS

Things like postgres and mysql have fairly well understood recordsize requirements. I've been looking for guidance for sqlite3 lately, and haven't found anything other that blocksize for sqllite3 was 1kB, and new versions might be 4kB, neither of then is a record size I want to try and use with zfs.

While I'm watching this unfold, I'm doing my own thing with normal x86-64/arm64 linux servers/vms running docker and portainer, and a zfs volume, or more, for each app, possibly with tuned recordsize if I think it warrants it.

I run my own automated, and manual if required, zfs snapshots of the docker datasets, and the app specific datasets, because I want to be able to backup/restore my app data too. Docker volume snapshots I'm ambivalent about, don't really care in general, but for a system rollback better to have the snapshots. Docker configuration is just the docker compose files in portainer, so easy to backup with the other app datasets. There is some other config for docker networks which I have scripted, and also living in the portainer dataset.

I'm not 100% sure this covers all the bases, but I'm 100% sure that there is a lot less frustration than watching the issues, including data loss, seemingly a lot of people have with the current TNS and/or TC app setups, and trying to rebuild/repair.

I've rebuild my config recently. I had to destroy all the docker configuration because docker seems to have a half deleted network than would not go away, so I know my rebuild plan works, rollback need testing.

I would like to see the TNS/TC stuff working well, but I would also like some confidence that my data was safe and recoverable too, and that state is not here yet.

The operating state (e.g small database) of an app is often put in a PVC.. it is data that is not shared via SMB.
The larger datastore (e.g Plex) is often shared via SMB and is put in a host path volume.

These databases are generally very small and mostly cached in RAM. Postgres will scale better than sqllite, but I doubt there will be many performance issues unless there's a surprisingly high transactional load. ZFS can cope with modest amounts of 4K writes, even if record size is larger.

I'd recommend trying 1 or 2 apps with current software (its better than it was 2 months ago) and check out the state. We'd appreciate inputs from real use-cases. Improvements can only be made with specific criticisms/issues.

 

[samarium]

The operating state (e.g small database) of an app is often put in a PVC.. it is data that is not shared via SMB.
The larger datastore (e.g Plex) is often shared via SMB and is put in a host path volume.

These databases are generally very small and mostly cached in RAM. Postgres will scale better than sqllite, but I doubt there will be many performance issues unless there's a surprisingly high transactional load. ZFS can cope with modest amounts of 4K writes, even if record size is larger.

I'd recommend trying 1 or 2 apps with current software (its better than it was 2 months ago) and check out the state. We'd appreciate inputs from real use-cases. Improvements can only be made with specific criticisms/issues.

Thanks, but for now I'll keep my apps outside TNS until I see significantly less people having problems, and I have a better idea about recoverability of apps. Trash and rebuild rather than just restore isn't workable.

Small writes and ARC caching I can see, but what I'm interested in is app data consistency vs app data captured and restored from snapshots, the classic issue when doing database backups, which for example, with oracle the databases used to be put into full block write mode while backups were happening, and then oracle would stitch the time timestamps together to arrive at a consistent database if we had to restore and rollback. This is probably more problematic with lots of small apps, and small databases like sqlite3, possibly with not a lot of thought out backup/restore processes, and not something I can see TNS ZFS directly affecting, but TN charts could better setup apps to better integrate with backups/restore via snapshot. Data consistency across multiple files cached in the apps them selves before it even gets to the ARC is maybe a problem, I don't know, and extrapolating large database techniques down to small apps probably has limited applicability.

So for TNS k3s I'll watch and observe. Meanwhile doing my own infrastructure management and testing and integration, and accepting my own risks, but with a much simpler environment.

 

[morganL]

Thanks, but for now I'll keep my apps outside TNS until I see significantly less people having problems, and I have a better idea about recoverability of apps. Trash and rebuild rather than just restore isn't workable.

Small writes and ARC caching I can see, but what I'm interested in is app data consistency vs app data captured and restored from snapshots, the classic issue when doing database backups, which for example, with oracle the databases used to be put into full block write mode while backups were happening, and then oracle would stitch the time timestamps together to arrive at a consistent database if we had to restore and rollback. This is probably more problematic with lots of small apps, and small databases like sqlite3, possibly with not a lot of thought out backup/restore processes, and not something I can see TNS ZFS directly affecting, but TN charts could better setup apps to better integrate with backups/restore via snapshot. Data consistency across multiple files cached in the apps them selves before it even gets to the ARC is maybe a problem, I don't know, and extrapolating large database techniques down to small apps probably has limited applicability.

So for TNS k3s I'll watch and observe. Meanwhile doing my own infrastructure management and testing and integration, and accepting my own risks, but with a much simpler environment.

Databases should use sync writes and ZFS provides guarantees.. unimpacted by the ARC/ZIL.
The snapshots are crash-consistent.
Postgres and MySQL and SQLite should be fine with these.

 

[samarium]

The Truecharts, TC, are saying that their apps have to use PVC for configuration, I think in relation to sqlite3, else corruption.
I haven't looked into pgsql / mysql as yet as I'm just using sqlite3 with netdata, and compromising on recordsize=16k.

But the question is how to do both bulk and selective backup/restore of k3s apps, both TNS and TC, their data and configuration, and the TNS app configuration.

That comes down into the ix-apps data set, non ix-app data, and interactions with midclt/cli to handle the internal apps configuration, from what I have seen looking at truetool/heavyscript.

I haven't see anything looking like a backup/restore interface into TNS app configurations, neither bulk nor selective. and both are required.

I tend to use single app datasets for my data, possibly nested, so individual apps can be rolled back or restored, and I have similar capability for my app configuration.

So there are questions about data integrity, maybe TC are correct about PVC being required, maybe not, I don't know. I'm not using their apps, just watching their discord. Helm charts seem to be rather opaque vs the type of configurability I want but that is up to what the chartmaker whats to expose vs what the app allows to be exposed.

There is a huge deficiency in manageability with the current TNS app setup, backup/restore vs how I am managing my apps. Hopefully this will be addressed in the future.

@Patrick M. Hausen I hope this is going in the directions you want to know about, as you have been fairly quiet, and I wasn't planning on hijacking your thread.

 

[marshalleq]

I came here hoping to understand how to persist my mariadb data for nextcloud. Weird that truenas doesn’t even have an official mariadb container especially because they do have official nextcloud and mariadb is the preferred database for nextcloud. But after reading this and other threads I am very concerned about data persistency. I simply cannot have our nextcloud data have issues. Documentation seems to always be missing context, be very sparse or missing entirely for these types of questions. I’m trying to use truenas as it’s intended but it’s getting more and more like I need to put unraid in a vm or perhaps a Linux distribution with portainer. After the years truenas has been out now and the extensive experience with the BSD flavour it’s embarrassing this stuff seems to be still so unpolished. Am I wrong?

 

[victort]

I came here hoping to understand how to persist my mariadb data for nextcloud. Weird that truenas doesn’t even have an official mariadb container especially because they do have official nextcloud and mariadb is the preferred database for nextcloud. But after reading this and other threads I am very concerned about data persistency. I simply cannot have our nextcloud data have issues. Documentation seems to always be missing context, be very sparse or missing entirely for these types of questions. I’m trying to use truenas as it’s intended but it’s getting more and more like I need to put unraid in a vm or perhaps a Linux distribution with portainer. After the years truenas has been out now and the extensive experience with the BSD flavour it’s embarrassing this stuff seems to be still so unpolished. Am I wrong?

Hence the reason I so enjoy CORE…

 

[sfatula]

I will add what I am doing here. I dislike the apps system catalogs as it is for some apps at least, quite inflexible. Things like Nextcloud where there are dozens of ways to set it up and run it is a simple example people may be familiar with. So, my solution is to simply use custom apps. I can then make the containers run the way I want to, in nextclouds case as an example, with redis and mariadb db (each with their own container) and I embedded crond within the nextcloud container along with image format updates, and pigz, vim for my use, etc. The answer in any case to the OPs question is to use hostpaths. While I get the arguments for PVC storage, I'd much rather use hostpaths. Backups are not an issue. You backup via however you would back up (again my example is Nextcloud) Nextcloud without using docker, kubernetes, etc.

I am using now 16 custom apps, some of which are really just the standard containers for a given app (think Kopia for example). I am not locked into the Truecharts ecosystem and limitations. I don't need all the extra stuff they want you to use, reverse proxies, ingress, etc. And even better, zero issues on Scale. I am serious about that. I simply have no app issues. Whoever maintains any given Scale app catalog can't mess me up.

I just finally updated from Bluefin to Cobia, had waited for certain bugs to be fixed. Guess what, all my (now called) custom apps run, zero changes. So, while I get many have app issues, I suspect it's the catalogs in most cases. Updates are so simple.

My mariadb container (standard from, docker hub) is backed up the way you should backup mariadb according to their doc. I have 5 different apps using it, 3 of which are actually not even on Scale as they run on different systems and just store data on Scale.

By using custom apps, in the case of apps like Emby or Nextcloud where you want to use the standard ports if possible, it's easy with custom apps as you can assign the app it's own IP, which means no port conflicts and easier client configurations. I only use custom IPs for those apps which benefit, most do not.

I find this method of running apps extremely reliable. That being said, I am familiar with containers so none of this is an issue for me. I used to use Truecharts and I spent far more time fixing things, breaking changes, etc. I hated PVC storage as that in turn creates thousands of replication snapshots, which then causes other issues and you don't share it either. The system is so much leaner and meaner now. Just my 2 cents.

 

[marshalleq]

Thankyou, @sfatula, this is actually close to the conclusion I made a few years ago around why I left TrueNAS. I pretty much saw all the same things you've eluded to except that back then I found that I couldn't make docker work as a custom app. To be fair, I'm not even sure a custom app here is what I think it is, but it's it's basically keeping in touch with upstream docker then yet that's what I want. This is what Unraid does. Their templates only point at docker hub and there's no bullxxxx it between. I really don't understand why we have to complicate things by adding an additional maintainer in the mix. Anyway, your post is extremely helpful as it seems to validate my thoughts. For now though, I've started with a few official apps and I will continue with them so I can learn how bad they are. As long as I have a persistent host data storage I should be right.

I did discover there is a link on most apps that takes to the docker hub page, this I've been using to find where to mount persistent storage. The only thing that's getting me through all this really odd and confusing non standard configuration that truenas has is my years of tech background. A basic user would not stand a chance. Not really good for what is essentially an appliance style app. But still I can deal with that, I came here for mature ZFS integration and file sharing and ability to make disk changes without have to shut down the OS - all stuff that unraid is terrible at. I think truenas will eventually win me over, I expect to be logging a lot of bugs and feature requests in Jira, probably writing a lot of documentation too - let's hope the ix systems team are responsive and agreeable - I think they were mostly last time - except for one person trying to deny how AMD CPU temps work for a while ;)

Marshalleq

 

[sfatula]

@marshalleq

Using Install Custom App

Provides information on using Install Custom App to configure custom or third-party applications in TrueNAS SCALE.

www.truenas.com

It ends up very little different than what docker people do and have done for a very long time, just a UI to configure it. And yes, docker hub or github is generally where they document what to mount, etc. I extend the containers where needed or desired.

This likely won't help too much those not familiar with running containers. I presume that's the intent of app catalogs. But those sure have caused so many issues, and people questioning Scale as a result. Thus far, they are not creating stability and those catalogs are doing the exact opposite. I hope all parties carefully consider how to go forward from here. So many noobs (not necessarily these forums, all over the internet) expecting to install Scale and Plex for example and run into so many issues. They don't know a thing about zfs, docker, kubernetes, host paths, linux, permissions, etc. and for some reason are assuming you just install and it runs.

 

[marshalleq]

Yeah, one would think ix-systems would read these forums and take a hint. Anyway, for nextcloud I decided to do exactly what you suggest, because the default container with it's persistent storage is so wonky I simply can't trust it - so I've installed a linuxserver docker container - same one I was using before. Hopefully it all works out - we shall see! The most annoying thing with doing this manually is having to input all the variables etc manually, I am wondering if we can create out own sort of 'truecharts' thing that doesn't need all this k3s stuff that adds no value, or if those additional catalogs are restricted to k3s. Something I will figure out in time I guess. If that could be done, surely someone would have done it already. I just want docker lol.

 

[marshalleq]

@sfatula out of interest why method do you use to see the dns name of a custom app given heavy script doesn't work for those? I can get the ip but it increments at each restart. Thanks.

 

[marshalleq]

I'll start a new thread.

 

[sfatula]

Yeah, I can't find an easy way to get internal kubernetes DNS name. There may be a way, but didn't spend much time on it so I am using a hard coded static IP for certain containers for now. HeavyScript actually does show custom app DNS names, what it doesn't show is custom app DNS names if you use a unique IP.

And the unique IP has to be hardcoded as the "MAC address" changes every time you start the app. Can't use DHCP yet.

As far as the environment variables, while it may seem pointless, I actually build my own containers, I extend the standard ones like the linuxserver nextcloud you mentioned. So, as part of that extension, my dockerfile has statements like:

ENV MYSQL_HOST=192.168.3.37 ENV REDIS_HOST=redis-ix-chart.ix-redis.svc.cluster.local

I then build my own container and have my own repo. So, Truenas uses my repo. This means when I add the custom app, I don't have to set any environment variables which for things like nextcloud is a lot of stuff. While all this sounds like a lot of work, the only work is initially creating the container how I want it which you mostly have to look into anyway. Updating is trivial and minutes of time. So, this in turn means if I did ever have to delete and recreate an app, something I have never done in over a year, then, less typing!

Hah, posting the env statement, looks like I need to change those IPs to my DNS names

 

[marshalleq]

I agree sound like it would be fun to do also, possibly some future opportunity to join heads on that.
So when you set a static IP for the custom app - how do you stop something else getting that IP, is it in a new range or something?

 

[marshalleq]

Also despite having a functioning nextcloud now, with reverse proxy using heavyscript does not list it. Weird that it works for you and not me.

root@Skywalker[...uenas_app_configs/nextcloud/log/nginx]# heavyscript dns ix-firefox firefox.ix-firefox.svc.cluster.local 10131/TCP ix-mariadb mariadb.ix-mariadb.svc.cluster.local 3306/TCP ix-nginx-proxy-manager nginx-proxy-manager.ix-nginx-proxy-manager.svc.cluster.local 10582/TCP nginx-proxy-manager-mariadb.ix-nginx-proxy-manager.svc.cluster.local 3306/TCP nginx-proxy-manager-web.ix-nginx-proxy-manager.svc.cluster.local 80/TCP nginx-proxy-manager-websecure.ix-nginx-proxy-manager.svc.cluster.local 443/TCP ix-plex plex.ix-plex.svc.cluster.local 32400/TCP plex-discovery.ix-plex.svc.cluster.local 32469/TCP,1900/UDP,32410/UDP,32412/UDP,32413/UDP,32414/UDP root@Skywalker[...uenas_app_configs/nextcloud/log/nginx]#