can't mount "unlocked by ancestor" dataset

[valhalla]

Code:

root@fremen[/mnt]# zfs mount -v Backup/valhalla
cannot mount 'Backup/valhalla': Permission denied
root@fremen[/mnt]#

Unless I misunderstood where to place the flag.

 

[winnielinnie]

I'll be honest. I'm lost.

I don't know if this is a SCALE thing, or if it's something underlying with ZFS itself.

You're the root user, so obviously it can't be a permission issue?

Did you create a non-root admin user in SCALE? Maybe try with them?

I'd page someone who uses SCALE, since they might be more savvy about what's going on here.

 

[valhalla]

no, it is the build in one. as shown here:

I am glad it is not just me that thinks that this is bizar. when I did the install of SCALE, i opped for the recommended admin route. So The root account here is just that I enabled password for the time being so I could SSH in directly as root just to be 10000% sure I am not ghosting here.

So as the non-root admin account I get this :

So I am not sure if I am running into a bug or something on the pool level that got chewed up.

 

[winnielinnie]

I'm not suggesting that you do this, but if it were me (to pique my own curiosity), I would boot into the latest version of an Ubuntu live USB, then try to unlock and mount the datasets. If it works in the new environment, then it might be an issue with SCALE / middleware / who-knows-what.

It might even still be possible to do this in Core. I'm not sure if you "upgraded" your pool.

 

[valhalla]

yeah, I was thinking of that too. The pool was created on Corbia Cobia, so I would assume the pool would already been on ZFS 2.2, so (AFAIK) core would be a no-go. The ubuntu route I can try. I will use 23.10 just to ensure it has the latest ZFS version on it. That I guess cannot harm. So appreciate that tip (and all the other help so far, it is truly well appreciated !). I will prep that and see how it goes.

EDIT: Typo

 

[valhalla]

quick sanity check here:

Code:

zpool import Backup (-f)
zfs load-key Backup
zfs get mountpoint Backup/valhalla
    if /mnt/Backup/valhalla
zfs mount Backup/valhalla
toss a back cat over the shoulders and hope it can mount without issues.

EDIT: And export pool first before going into Ubuntu.

 

[valhalla]

Well, unfortunately, get the same permission denied. So got to be the pool if you ask me.

 

[winnielinnie]

The passphrase successfully unlocks ("load-key") the encryptionroot.

Backup and Backup/valhalla are both decrypted and available to mount.

Yet under no circumstance, even on an Ubuntu system with ZFS 2.2.0, can you mount Backup/valhalla?

I'm really, really lost.

 

[winnielinnie]

Perhaps you can check the recent messages immediately after (failing) to mount the dataset?

Immediately after a failed mount attempt:

Code:

dmesg | tail -n 50

 

[chuck32]

I'm not a fraction as experienced as @winnielinnie but what I may try in your place:

Since for whatever reason the dataset got moved to an encrypted pool (see the symbol in #6), which it should not, I would again see the culprit here. Also moving to boot pool does not seem to work.

0) Backup your current configuration
1) Restore an older configuration from backup *
2) Use any spare HDD / SSD you have lying around, create a pool with it
3) move the systemdataset there
4) reboot

* I do not use keys, probably you should be sure that you exported your config with password seeds ('m not sure about this option)

I am not sure, but if I followed this thread correctly you destroyed .system when it is not clear to me whether the dataset actually resided on boot pool during that time. Because #6 showed it does reside on the backup pool and I interpret #26 as wrongly showing it resides on boot pool. Or did you actually move the set before #26?

 

[valhalla]

Perhaps you can check the recent messages immediately after (failing) to mount the dataset?

Immediately after a failed mount attempt:

Code:

dmesg | tail -n 50

dmesg did not spit anything out. I did some digging into /var/log/middlewared and got this:

Code:

[2023/11/30 19:18:21] (ERROR) ZFSDatasetService.mount():64 - Failed to mount dataset
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 57, in mount
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 60, in mount
    dataset.mount_recursive()
  File "libzfs.pyx", line 4095, in libzfs.ZFSDataset.mount_recursive
  File "libzfs.pyx", line 4116, in libzfs.ZFSDataset._mount_recursive
  File "libzfs.pyx", line 4113, in libzfs.ZFSDataset._mount_recursive
  File "libzfs.pyx", line 4110, in libzfs.ZFSDataset._mount_recursive
  File "libzfs.pyx", line 4089, in libzfs.ZFSDataset.mount
libzfs.ZFSException: cannot mount 'Backup/valhalla': Permission denied

There last 50 entries of dmesg are related to my NICs, IPMI and the hardware virtualization (note this is a baremetal SCALE install)

Code:

   13.384829] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.386841] pci 0000:00:1f.1: Adding to iommu group 21
[   13.387638] pci 0000:00:1f.1: Removing from iommu group 21
[   13.388890] pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000
[   13.389546] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.392942] pci 0000:00:1f.1: Adding to iommu group 21
[   13.394081] pci 0000:00:1f.1: Removing from iommu group 21
[   13.395915] pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000
[   13.397090] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.399487] pci 0000:00:1f.1: Adding to iommu group 21
[   13.400425] pci 0000:00:1f.1: Removing from iommu group 21
[   13.401765] pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000
[   13.403119] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.406114] pci 0000:00:1f.1: Adding to iommu group 21
[   13.406900] pci 0000:00:1f.1: Removing from iommu group 21
[   13.407583] pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000
[   13.408182] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.409195] pci 0000:00:1f.1: Adding to iommu group 21
[   13.409994] pci 0000:00:1f.1: Removing from iommu group 21
[   13.410875] pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000
[   13.411821] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.413660] pci 0000:00:1f.1: Adding to iommu group 21
[   13.415934] pci 0000:00:1f.1: Removing from iommu group 21
[   13.416846] pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000
[   13.417786] pci 0000:00:1f.1: reg 0x10: [mem 0xfd000000-0xfdffffff 64bit]
[   13.419584] pci 0000:00:1f.1: Adding to iommu group 21
[   13.420353] pci 0000:00:1f.1: Removing from iommu group 21
[   13.421149] EDAC MC0: Giving out device to module pnd2_edac controller Pondicherry2: DEV pnd2/dnv (INTERRUPT)
[   13.431781] intel_rapl_common: Found RAPL domain package
[   13.432374] intel_rapl_common: Found RAPL domain core
[   13.432987] intel_rapl_common: Found RAPL domain dram
[   13.501253] ipmi_si IPI0001:01: IPMI kcs interface initialized
[   13.508657] ipmi_ssif: IPMI SSIF Interface driver
[   39.882900] ioatdma: Intel(R) QuickData Technology Driver 5.00
[   39.904779] NTB Resource Split driver, version 1
[   39.924747] Software Queue-Pair Transport over NTB, version 4
[   39.986319] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[   40.885163] Adding 16777212k swap on /dev/mapper/sde4.  Priority:-2 extents:1 across:16777212k SSFS
[   41.144908] ixgbe 0000:02:00.0: registered PHC device on enp2s0
[   41.324789] ixgbe 0000:02:00.0 enp2s0: detected SFP+: 3
[   41.419881] pps pps0: new PPS source ptp1
[   41.437473] ixgbe 0000:08:00.0: registered PHC device on eno3
[   41.464808] ixgbe 0000:02:00.0 enp2s0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[   41.637548] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[   44.668976] RPC: Registered named UNIX socket transport module.
[   44.669810] RPC: Registered udp transport module.
[   44.670638] RPC: Registered tcp transport module.
[   44.671439] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   45.339205] ixgbe 0000:08:00.0 eno3: NIC Link is Up 1 Gbps, Flow Control: None
[   45.339684] IPv6: ADDRCONF(NETDEV_CHANGE): eno3: link becomes ready

The dmesg is after I ran manually a sudo zfs mount Backup/valhalla. The middlewared log is when I yesterday booted back into SCALE and imported the pool again after my attempt with Ubuntu.

no messeges in kernel logs is kind of odd as well since ZFS is within the kernel, so I would expect something being spit out on the manual mount attempt.

I'm not a fraction as experienced as @winnielinnie but what I may try in your place:

Since for whatever reason the dataset got moved to an encrypted pool (see the symbol in #6), which it should not, I would again see the culprit here. Also moving to boot pool does not seem to work.

0) Backup your current configuration
1) Restore an older configuration from backup *
2) Use any spare HDD / SSD you have lying around, create a pool with it
3) move the systemdataset there
4) reboot

* I do not use keys, probably you should be sure that you exported your config with password seeds ('m not sure about this option)

I am not sure, but if I followed this thread correctly you destroyed .system when it is not clear to me whether the dataset actually resided on boot pool during that time. Because #6 showed it does reside on the backup pool and I interpret #26 as wrongly showing it resides on boot pool. Or did you actually move the set before #26?

No, the weird part is it shows up in the pool, but I had a check that seems to be (at least for SCALE) normal. It shows being mounted in Backup/.system instead of "LEGACY". So no the .system dataset was not being moved or tinkled with, and I would admit that if I did at this point. The system dataset has not been changed or touched. For some reason it gets inherited into the pool, because after removing the .system directory, the pool needs to be manually unlocked after each reboot I have done so far. So I am sure if the mount point being pointed to Backup/.system was not a "cosmetic" glitch for not showing "LEGACY" and that it does need to reside there.

Your suggestion, would it not be then easier to reinstall the system and restore the backup ? Unless I am maybe missing some step.

 

[valhalla]

I still think that my theory may make sense. For reasons unknown, the Backup/valhalla dataset (which was encrypted with key X on System X) got snapshot to the Backup system which has key Y. For reasons unknown, Backup pool encrypted valhalla dataset with key Y. Now when you try to mount, key Y is unlocked but there seems to be something that still related to key X. Because the message kind of makes sense when you try to mount the dataset that is encrypted (and the system doesn't know it) it would not allow you to mount it. But it is difficult to proof that theory.

I also agree "but it should then prompt that it is locked and you need to unlock the dataset" as it does when you just do a zfs mount while the pool itself is locked. And that is why I am also scratching my head on how is that even possible.

 

[winnielinnie]

For reasons unknown, the Backup/valhalla dataset (which was encrypted with key X on System X) got snapshot to the Backup system which has key Y. For reasons unknown, Backup pool encrypted valhalla dataset with key Y. Now when you try to mount, key Y is unlocked but there seems to be something that still related to key X.

But the key (Master Key) is "available", which means the dataset is unlocked and decrypted. So when you replicated to the Backup pool, valhalla inherited the User Key of Backup. (Backup was obviously unlocked at the time of receiving the first stream.)

You can test this for yourself:

Unlock Backup, which is the encryptionroot for valhalla.

Now Backup's and valhalla's master keys are available. The datasets are decrypted.

Now you can change the user key for valhalla. (Either use a passphrase or string.) This will break inheritance, and you'll be able to lock/unlock valhalla on its own (without a higher up encryptionroot parent.)

 

[valhalla]

interesting fact is that it reboots your system without a warning when you change the key.

EDIT: Done over the gui

 

[winnielinnie]

Okay, how did you do this backup task?

Because you wouldn't be able to change the user key back to what it was, even if you tried.

 

[valhalla]

created replication tasks based on snapshots on the main system, which I have already for a few years, when I was running CORE. These were pushed to the backup system. via SSH. on the CORE systems that worked always flawlessly. The only thing that I have done with the backup system recently is removed the USB boot drive in favor of a SSD, re-created the pool with 2 additional drives, imported the SSH keys so they were the same as before and fired off the replication tasks on the normal scheduled base that they were on before. No other changes were made on the backup system as I like KISS. The tasks also ran on a daily bases without any hiccups.

EDIT: so basically a complete fresh install with a fresh Backup pool. Prior that, all the datasets were encrypted with the main system key and none of them were inherited by the Backup key at that time. And since replication tasks were working, I may have overlooked the inheriting part, which brings me to the point were I am now.

 

[valhalla]

interesting fact is that it reboots your system without a warning when you change the key.

EDIT: Done over the gui

of course after reboot, the dataset valhalla is still inherited by Backup pool.

 

[valhalla]

What I also tried is to clone the dataset via the GUI, but also there, it created the dataset, but fails to mount.

 

[winnielinnie]

Can you try to change the user key in an Ubuntu session?

You basically unlock ("load-key") the encryptionroot Backup. This makes Backup and Backup/valhalla's master keys available (unlocked). Then you can use the "change-key" command to try to change valhalla to use a simple, easy-to-remember passphrase. (Hopefully that won't crash your Ubuntu system.)

 

[valhalla]

I can certainly try. Let me fire that up.