bcachefs

[jcizzo]

I know that truenas hedges it's bets on zfs and for obvious reasons.

Bcachefs will be merged into kernel 6.7 and will probably wind up being the default linux filesystem at some point in the near future.

since truenas seems to be focusing all it's efforts on migrating it's product to the linux kernel, what'll happen if this new file system proves as successful as intended by the devs (Kent Overstreet et al). would truenas make bcachefs the default filesystem of scale?

 

[danb35]

truenas hedges it's bets on zfs

iXSystems has been all-in on ZFS for the last ten years or more. If by "hedges its bets" you're suggesting to the contrary (as that phrase would ordinarily mean), I'd be really interested in seeing your basis for that claim.

probably wind up being the default linux filesystem at some point in the near future.

This sounds extremely optimistic (especially since, per their own FAQ, it still isn't release-quality code), though if you aren't too picky about the definition of "near future" I guess it could be possible.

would truenas make bcachefs the default filesystem of scale?

I'd think it exceedingly unlikely, at least within the next decade or so.

I think the big question would be, why? Why change from a solid filesystem in which iX is already quite invested, to a brand-new one? Tiered storage looks interesting, at least in theory. OTOH, it's missing some major things, like scrub and replication. It's presumably GPL-licensed, but iX doesn't seem bothered by the license of ZFS. There's no cross-platform capability; it's Linux-only. So what compelling feature(s) or capability(ies) does it bring to the table? Edit: at least there's apparently no write hole; it has that over btrfs...

 

[jcizzo]

nothing more than an inquiring mind wanting to know

 

[Ericloewe]

if this new file system proves as successful as intended by the devs

Btrfs, despite a lot of development effort, finally seems to be at a point where it can mostly sorta be used as a traditional filesystem. The volume management side of things still seems to be a hot mess, by the wiki's own admission.
Bcachefs doesn't even seem to try beyond mirrors, with RAID5/6 functionality being nothing more than a vague description in the wiki (which still manages to raise important usability questions!).

It is wildly premature to discuss bcachefs as anything other than a curiosity. I wish them luck and hope they're more rigorous than the btrfs devs.

 

[Ericloewe]

Edit: at least there's apparently no write hole; it has that over btrfs...

Well, that's easy if you don't offer parity RAID. At least it's honest, unlike btrfs.

 

[danb35]

I hadn't heard of this filesystem until I saw this post, so my knowledge is (to say the least) quite limited. But there are some features ZFS lacks, and it seems that many of these are near-impossible now due to design decisions made early on. A few examples:

• As a broad category, "vdev manipulation." This would include RAIDZ expansion and contraction (i.e., disk removal from a vdev), level changes, etc.
• Rebalancing data after changes--be those adding a vdev, adding a disk to a RAIDZ vdev, or changing a dataset option like compression or encryption.
• I don't know if tiered storage is in quite the same category, but it is something that's requested here with some frequency, and ZFS just doesn't do it.

bcachefs claims to support the latter two, at least to some degree. If it can do so in a robust way, that'd be a win.

But what I'm missing at this point is what Overstreet thinks he's bringing to the table. Surely he knows that ZFS is out there. So, even leaving maturity aside, what compelling need does he see this filling? I don't see the website answering this question, but I can't imagine he'd have put the work into this without at least thinking he had a reason.

 

[Ericloewe]

Maybe bcachefs is to btrfs what Unix is to Multics.

 

[Saoshen]

The greatest thing that BTRFS brings is very easy and flexible (if slow) volume management that allows, disk addition/removal/replacement, along with raid level changes, all while online and working.

Prior to moving to truenas, I have been using btrfs 5+ years, moved countless disks around, changed from single, to raid1 to raid 5 and raid 6, and back again, on multiple volumes, with no data loss (other than my own fat fingering of commands).

I still have 1 btrfs raid 1 volume on my old server with left over disks as a backup for various data.

All that said, (from my limited understanding) I think the neatest addition bcachefs brings is flexible tiering/mixing of both ssd/hdd and differing sizes. Whether it is stable enough and feature-complete enough, I suppose is still up for discussion.

Phoronix has done some benchmarking of bcachefs vs other file systems recently that may be of interest to some.

 

[Ericloewe]

The greatest thing that BTRFS brings is very easy and flexible (if slow) volume management that allows, disk addition/removal/replacement, along with raid level changes, all while online and working.

Except it does not work, as documented by the btrfs wiki. This is not a bizarre vendetta someone started, it comes from the horse's mouth! Just two or so years ago, half the features on their wiki were marked in yellow or red for "not implemented yet, you will lose data" or "yeah, this is actually broken and you may lose data". I'm happy you didn't have a catastrophe on your hands, but the expression "playing with fire" comes to mind (and this is aimed at the btrfs devs, not at users who come along, effectively misled because they didn't ask first if the advertised features were real).

Prior to moving to truenas, I have been using btrfs 5+ years, moved countless disks around, changed from single, to raid1 to raid 5 and raid 6, and back again, on multiple volumes, with no data loss (other than my own fat fingering of commands).

Have you checked? Even today RAID5/6 come with a huge asterisk that says "DATA LOSS".

 

[Saoshen]

I know it worked and works, as I shuffled 30 some disks around in btrfs to move to truenas.

As always, YMMV, but I never had a btrfs specific data loss coming from netgear btrfs over mdraid, then to my own home build debian btrfs and btrfs native raid levels.

 

[Arwen]

I am glad all those who used BTRFS that had no data loss, (myself included). (I no longer use BTRFS... and have not since about 2014.)

But, their are other data loss issues that can occur. BTRFS is NOT 100% COW, Copy On Write. Here is the specific example of both not-COW and potential data loss. To be extremely fair, it would be so rare as to be 1 in a trillion or so... Yet, it is a design characteristic of BTRFS that no one has marked to be "fixed".

FAQ - btrfs Wiki

archive.kernel.org

Now again to be fair, the new BTRFS documents don't mention this problem. Or I could not find it. However, it is possible the problem still exists and the "new" BTRFS documents gloss over it.

ZFS on the other hand, guarantees that any rename of a file will either contain the old file data or new file data depending on when the crash happened. Not a chance of a zero byte file.

 

[Arwen]

On the subject of Bcachefs, based on prior work on EXT4 and BTRFS, I suspect Bcachefs will take at least 5 years more before it will be in regular use. Then, even more years to be feature similar to ZFS.

I say this because of all the features added to ZFS over decades. Even the simple Async Destroy Dataset / zVol feature took years, and bit Sun Microsystems customers for years until Sun in essence copied the OpenZFS feature too.

OpenZFS has had so much data stored, moved and scrubbed that the amount stored on BTRFS is tiny fraction, With Bcachefs being able to learn from prior file systems, it should take less time. But, ZFS has been in development over 22 years, and in production 17 years;

ZFS - Wikipedia

en.wikipedia.org

 

[morganL]

YALFS - Yet another linux File system.
bcachefs is mostly interesting to linux OS people that don't have the option of OpenZFS,
TrueNAS doesn't have the problem.

I would add the non-technical challenge for any file system is maintaining a smart and dedicated community that can improve things and fix bugs for a long period of time. OpenZFS just fixed a solaris-originated bug from 15 years ago.

 

[winnielinnie]

YALFS - Yet another linux File system.

@morganL coming in hot with the jokes! This marks a new era. Exciting times ahead of us.

 

[Ericloewe]

If the jokes write themselves... FreeBSD has two relevant filesystems, one for puny systems and the other is ZFS. Mac OS has one, plus ZFS. Windows has 1.5 filesystems, depending on how you count, plus ZFS for the adventurous. Other BSDs typically have one filesystem, plus ZFS in some cases.
Meanwhile Linux has:

• ext2/3/4, answering the question "What if Microsoft had inflicted FAT36, FAT48 and FAT64 upon the world instead of developing NTFS?"
• XFS, because the situation was bad enough that IRIX had to be raided for parts
• ReiserFS, written by a literal murderer and narcissist
• btrfs, because Not Invented Here can be expressed through filesystems
• bcachefs, as discussed here
• Plus ZFS

FAT/exFAT are freebies, since nobody seriously uses them outside of their niches.

 

[danb35]

Mac OS has one, plus ZFS.

Two: APFS and Mac OS Extended, as the Disk Utility currently informs me. But the point is nonetheless taken.

 

[Ericloewe]

Isn't everything new basically APFS, with the other one relegated to support for legacy disks?

 

[danb35]

Could be, really, I'm honestly not sure. I seem to recall hearing that APFS was developed for flash (SSDs), but not sure what would be preferred for spinners--though I'm not sure any Macs now come with spinners.

 

[Arwen]

Oh, another dig against other file systems.

OpenZFS has a test suite that actually tests the features. Any bug fix is required to run the test suite, SUCCESSFULLY, before the bug fix can be accepted. This is intended to help prevent, (note the wording, "help prevent", not "guarantee prevent"), breaking of something else. Further, any new feature is required to add test suite module(s), AND have them approved by people on GitHub, just like the actual feature needs to be approved.

I don't know if the OpenZFS test suite is from Sun ZFS... but it seems to me that all file systems, even OSes, should have such test suites.


As to why the file clone / hole file bug was not detected, race conditions are harder to reproduce until you know what you are testing for.

 

[Etorix]

Could be, really, I'm honestly not sure. I seem to recall hearing that APFS was developed for flash (SSDs), but not sure what would be preferred for spinners--though I'm not sure any Macs now come with spinners.

For spinners (as USB-attached drives), you're better with HFS+: APFS scatters its metadata all over the place instead of neatly grouping in blocks, as all file systems designed in the HDD era do. With SSDs, it is irrelevant; with HDDs, the many associated random read/writes are a serious bottleneck.