libtorrent 2.X (used by deluge, qbittorrent) introduced memory mapped files, implications with ZFS ARC?

[draetheus]

I have not yet done any upgrades yet but I happened to come across this issue with libtorrent (reported by various upstream users of deluge, qbittorrent, etc): https://github.com/arvidn/libtorrent/issues/6667

Does anyone who has a better grasp of ZFS know how memory mapped files might have an implication with the ZFS ARC? Is every torrented file going to be stored twice in RAM? This was the only relevant thing I could find but it could be very out of date: https://stackoverflow.com/questions/34668603/memory-usage-of-zfs-for-mapped-files

 

[winnielinnie]

Hold out with qBittorrent 4.3.9 as long as possible. I'm reading about some nasty behaviors and issues with 4.4.x, even on Windows and Linux.

I actually made a snapshot named "qbittorrent439" for my qbittorrent jail's root.

 

[asdpyro]

Do you know any way to revert to an older version, this memory issue it killing my system. qBittorrent will go from almost no memory usage to using every MB of available memory in the span of about 4 hours. I just had to reboot my system a bit ago because qBittorrent was using more than 25gigs of memory. Even after stopping downloads in qBittorrent, removing torrents, stopping qBittorrent as an application and even deleating it, the memory usage stayed in place. Only after a reboot did the memory clear.

After the reboot of my machine the system ran happy with only the ZFS system consuming about 50% of my memory, which is about normal for this system. Ran like this for mulitple days without issue. Then when I re-enabled qbittorent, also stayed low usage. As soon as I start a torrent on qBittorrent the usage starts rising. The file being downloaded is less than 5gigs and qBittorent will still consume the entire available memory in just a few hours.

 

[winnielinnie]

Do you know any way to revert to an older version, this memory issue it killing my system.

Unfortunately, I don't know how to downgrade a specific application with FreeBSD (or if it's even feasible.)

I'm just going to stick with qBittorrent 4.3.9 for as long as possible (on its own jail.)

 

[DeltaEntropy]

I was having issues with the "services" using an ever-increasing amount of memory until I restarted the server. Tracked it down to qbittorrent. Its behaviour seems to line up with what you are describing above with memory mapping.

Because I haven't had any luck figuring out how to downgrade to a version not in the "latest" or "quarterly" repos (if anyone knows, please help), I have upgraded to 4.4.3.1.

This appears to be slightly better, it will still "double-load" the file into services and zfs cache, but seems to then drop it instantly once done with it (i.e. finishes downloading) and the "services" ram slowly drops back down again. htop shows the reserved and physical memory drop instantly.

 

[winnielinnie]

This appears to be slightly better, it will still "double-load" the file into services and zfs cache, but seems to then drop it instantly once done with it (i.e. finishes downloading) and the "services" ram slowly drops back down again. htop shows the reserved and physical memory drop instantly.

That's promising news.

 

[DeltaEntropy]

So I have been tinkering and I've figured out how to downgrade to a version not in the "latest" or "quarterly" repos. There are some limitations in terms of what versions though i.e. I haven't tracked down all versions.

Fair warning, I am but a sentient meat sack smashing my fingers against the keyboard until something worked...

For my hardware, the default repo for pkg is http://pkg.FreeBSD.org/${ABI}/quarterly. Side note, "ABI" is the architecture of the hardware/CPU you are using (I think, correct me if I'm wrong), in my case 'amd64'.

But looking further into https://pkg.freebsd.org/, there are live repos other than "latest" and "quarterly".

The tricky part is that when drilling down into these, ".../All/" gives a "403 Forbidden" error for me. So what I found was within each of the above (lastest, quarterly, release_0, etc.) is "packagesite.txz", and within that is a file "packagesite.yaml". If you open that with a text editor and search for the pkg you are wanting and on that line it will say what version is in the repo.

E.g. for qbittorrent-nox in release_3, there is qbittorrent-nox-4.3.8.

You could just manually grab the pkg by appending the file name onto the end of the URL, e.g. https://pkg.freebsd.org/FreeBSD:12:amd64/release_3/All/qbittorrent-nox-4.3.8.pkg then using pkg add qbittorrent-nox-4.3.8.pkg but you'll have to then manually add the dependencies. I'd actually recommend pointing pkg at the repo you want, this way it will grab the dependencies and more importantly the correct version of the dependencies.

Next was figuring out how to point pkg at these repos.

In a manually created jail pkg uses "/mnt/<plugins/jail pool>/iocage/jails/<jail name>/root/etc/pkg/FreeBSD.conf". Change the following line to what ever repo you want:
url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
For "release_3":
url: "pkg+http://pkg.FreeBSD.org/${ABI}/release_3",

See options below for what to do if in a plugin jail.

Then it's a matter of updating pkg and installing qbittorrent-nox:
pkg update pkg install qbittorrent-nox-4.3.8

I usually confirm it grabbed the correct repo with the following and checking that it shows the version I want:
pkg search qbittorrent

Now to put it into use.

There are a couple of options:

Show : Downgrade existing plugin or existing manual jail

Once you have figured out which repo the particular version you want is in, update FreeBSD.conf to point pkg at it.
FreeBSD.conf is located at "/mnt/<plugins/jail pool>/iocage/jails/<jail name>/root/etc/pkg/FreeBSD.conf"
url: "pkg+http://pkg.FreeBSD.org/${ABI}/<REPO>",

Note that if you are doing this in a plugin jail then you'll need to update the following file for pkg to point at the new repo.
"/mnt/<plugins/jail pool>/iocage/jails/<jail name>/root/usr/local/etc/pkg/repos/FreeBSD.conf"
Update it from:
FreeBSD: { enabled: no }
To:
FreeBSD: { enabled: yes }

Then update pkg:
pkg update

Check it has the worked and has the right version package:
pkg search <package name>

Now the existing package (and dependencies) need to be removed:
pkg remove <package name and version> (use pkg info to find the exact name and version)
pkg autoremove - this removes the dependencies

Then install the package version you want:
pkg install <package name and version>

I would recommend deleting the qbittorrent.conf config file and starting fresh, as well as clearing the cache on the browser that you use to access the webui.

Show : Fresh manual jail

This is simpler in that you don't need to remove anything but there are a few extra steps.

Once you have figured out which repo the particular version you want is in, update FreeBSD.conf to point pkg at it.
FreeBSD.conf is located at "/mnt/<plugins/jail pool>/iocage/jails/<jail name>/root/etc/pkg/FreeBSD.conf"
url: "pkg+http://pkg.FreeBSD.org/${ABI}/<REPO>",

pkg will need to be installed and updated:
pkg update will do both

Check it has worked and has the right version package:
pkg search <package name>

Then install the package version you want:
pkg install <package name and version>

Now to get the package to run as a service:
sysrc -f /etc/rc.conf qbittorrent_user="qbittorrent" && sysrc -f /etc/rc.conf qbittorrent_enable="YES" service qbittorrent start

With this, I managed to first get 4.3.8 installed and running smoothly in the fresh manual jail. I then managed to downgrade my existing 4.4.3.1 plugin to 4.3.8 and it is running smoothly as well. I've had htop open for a while with virtual, reserved, and physical memory have gone up and down nicely, peaking at a few GB then dropping back down when seeding stops on a file. On 4.4.2 (and to a lesser extent 4.4.3.1) reserved and physical would routinely go up to ~10 GB or more, with virtual peaking at several hundred GB... saw it peak at 247 GB.

I would recommend, as others have mentioned in other threads, locking your packages with pkg lock <package name and version>. I locked qbittorrent-nox, libtorrent-rasterbar, and boost as those were different versions between 4.4.2 (and 4.4.3.1) and 4.3.8.

 

[asdpyro]

I was having issues with the "services" using an ever-increasing amount of memory until I restarted the server. Tracked it down to qbittorrent. Its behaviour seems to line up with what you are describing above with memory mapping.

Because I haven't had any luck figuring out how to downgrade to a version not in the "latest" or "quarterly" repos (if anyone knows, please help), I have upgraded to 4.4.3.1.

This appears to be slightly better, it will still "double-load" the file into services and zfs cache, but seems to then drop it instantly once done with it (i.e. finishes downloading) and the "services" ram slowly drops back down again. htop shows the reserved and physical memory drop instantly.

Coming back to this issue after a while and I am having similar positive finding with newer versions. qBittorrent still stacks up tones of services RAM while a file share is in progress but seems to relinquish cache when it's done. Still seems VERY stupid for it to cache thousands of copies of file and eat up all the RAM in this way. I recently shared a file with a friend that's measly 112MB and qBittorrent still used 40 Gigs of RAM.