SOLVED Where did my swap go?

[GeorgePatches]

I've been shuffling disks around in my pool the last few days. Detaching a disk from one mirrored vdev and then extending a different mirrored vdev, that sort of thing. Before I moved any disks, I had 22GiB of swap. 16 on the boot-pool and 2 on each of my 3 vdevs of my 'Array' pool. Now my system only reports 4GiB of swap after shuffling things around. I thought a reboot might fix it, it did not. If I do a 'gpart show', all the swap partitions are there as far as I can tell, but it's not picking them up. If I do a 'swapctl -l' I only see 2 /dev/mirror items listed, and I would expect to see 4.

Running TrueNAS Core 13.0-U4. Any thoughts or suggestions on how to restore this to normal?

 

[Patrick M. Hausen]

Did you reboot after the shuffleing? Please do so first whenever convenient. Then the output of these commands will probably help to diagnose if there is a problem or expected behaviour:

zpool status
swapinfo
gpart show

 

[GeorgePatches]

I've rebooted since I finished, twice in fact. :P

Here is the output of the requested commands.

Code:

root@truenas[~]# zpool status
  pool: Array
 state: ONLINE
  scan: resilvered 1.10T in 03:53:21 with 0 errors on Thu May  4 13:13:12 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        Array                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/6fdd405b-45a4-11ec-9f16-94de807ce975  ONLINE       0     0     0
            gptid/6faa39d7-45a4-11ec-9f16-94de807ce975  ONLINE       0     0     0
            gptid/57a8e772-ea7e-11ed-ba1f-14dae9c52267  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/768db535-45a4-11ec-9f16-94de807ce975  ONLINE       0     0     0
            gptid/4685dd28-e9d4-11ed-ba1f-14dae9c52267  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            gptid/ebba0e02-311d-11ed-8cee-14dae9c52267  ONLINE       0     0     0
            gptid/059e0ee6-e90c-11ed-ba1f-14dae9c52267  ONLINE       0     0     0
        logs
          gptid/66723965-45a4-11ec-9f16-94de807ce975    ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:32 with 0 errors on Thu May  4 03:45:32 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0

errors: No known data errors

Code:

root@truenas[~]# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap0.eli   2097152        0  2097152     0%
/dev/mirror/swap1.eli   2097152        0  2097152     0%
Total             4194304        0  4194304     0%

Code:

root@truenas[~]# gpart show
=>      40  28131248  nvd0  GPT  (13G)
        40        88        - free -  (44K)
       128  28131160     1  freebsd-zfs  (13G)

=>       40  468862048  ada0  GPT  (224G)
         40       1024     1  freebsd-boot  (512K)
       1064   33554432     3  freebsd-swap  (16G)
   33555496  435290112     2  freebsd-zfs  (208G)
  468845608      16480        - free -  (8.0M)

=>       40  468862048  ada1  GPT  (224G)
         40       1024     1  freebsd-boot  (512K)
       1064   33554432     3  freebsd-swap  (16G)
   33555496  435290112     2  freebsd-zfs  (208G)
  468845608      16480        - free -  (8.0M)

=>        40  3907029088  da0  GPT  (1.8T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)

=>        40  5860533088  da2  GPT  (2.7T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)

=>        40  3907029088  da3  GPT  (1.8T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)

=>        40  3907029088  da4  GPT  (1.8T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)

=>        40  3907029088  da5  GPT  (1.8T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)

=>        40  5860533088  da6  GPT  (2.7T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)

=>        40  3907029088  da1  GPT  (1.8T)
          40          88       - free -  (44K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)

 

[GeorgePatches]

Did you reboot after the shuffleing? Please do so first whenever convenient. Then the output of these commands will probably help to diagnose if there is a problem or expected behaviour:

zpool status
swapinfo
gpart show

Does it all look correct to you? It seems like it should have 22GB to me.

 

[Patrick M. Hausen]

Sorry about the delay. I also need gmirror status and gpart list, please.

 

[GeorgePatches]

Sorry about the delay. I also need gmirror status and gpart list, please.

Yes sir, here you go.

Code:

root@truenas[~]# gmirror status
        Name    Status  Components
mirror/swap0  COMPLETE  da1p1 (ACTIVE)
                        da6p1 (ACTIVE)
                        da3p1 (ACTIVE)
mirror/swap1  COMPLETE  da5p1 (ACTIVE)
                        da4p1 (ACTIVE)
                        da2p1 (ACTIVE)

Code:

root@truenas[~]# gpart list
Geom name: nvd0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 28131287
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: nvd0p1
   Mediasize: 14403153920 (13G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e2
   efimedia: HD(1,GPT,66723965-45a4-11ec-9f16-94de807ce975,0x80,0x1ad3f58)
   rawuuid: 66723965-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 14403153920
   offset: 65536
   type: freebsd-zfs
   index: 1
   end: 28131287
   start: 128
Consumers:
1. Name: nvd0
   Mediasize: 14403239936 (13G)
   Sectorsize: 512
   Mode: r1w1e3

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 468862087
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,3d1e0db1-2d77-11ed-a7d1-14dae9c52267,0x28,0x400)
   rawuuid: 3d1e0db1-2d77-11ed-a7d1-14dae9c52267
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: ada0p2
   Mediasize: 222868537344 (208G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17180413952
   Mode: r1w1e1
   efimedia: HD(2,GPT,3d2257f2-2d77-11ed-a7d1-14dae9c52267,0x2000428,0x19f20000)
   rawuuid: 3d2257f2-2d77-11ed-a7d1-14dae9c52267
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 222868537344
   offset: 17180413952
   type: freebsd-zfs
   index: 2
   end: 468845607
   start: 33555496
3. Name: ada0p3
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r0w0e0
   efimedia: HD(3,GPT,3d202bf8-2d77-11ed-a7d1-14dae9c52267,0x428,0x2000000)
   rawuuid: 3d202bf8-2d77-11ed-a7d1-14dae9c52267
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 17179869184
   offset: 544768
   type: freebsd-swap
   index: 3
   end: 33555495
   start: 1064
Consumers:
1. Name: ada0
   Mediasize: 240057409536 (224G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 468862087
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,3d261eae-2d77-11ed-a7d1-14dae9c52267,0x28,0x400)
   rawuuid: 3d261eae-2d77-11ed-a7d1-14dae9c52267
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: ada1p2
   Mediasize: 222868537344 (208G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17180413952
   Mode: r1w1e1
   efimedia: HD(2,GPT,3d2a4a4f-2d77-11ed-a7d1-14dae9c52267,0x2000428,0x19f20000)
   rawuuid: 3d2a4a4f-2d77-11ed-a7d1-14dae9c52267
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 222868537344
   offset: 17180413952
   type: freebsd-zfs
   index: 2
   end: 468845607
   start: 33555496
3. Name: ada1p3
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r0w0e0
   efimedia: HD(3,GPT,3d2848d7-2d77-11ed-a7d1-14dae9c52267,0x428,0x2000000)
   rawuuid: 3d2848d7-2d77-11ed-a7d1-14dae9c52267
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 17179869184
   offset: 544768
   type: freebsd-swap
   index: 3
   end: 33555495
   start: 1064
Consumers:
1. Name: ada1
   Mediasize: 240057409536 (224G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   efimedia: HD(1,GPT,57a1fd12-ea7e-11ed-ba1f-14dae9c52267,0x80,0x400000)
   rawuuid: 57a1fd12-ea7e-11ed-ba1f-14dae9c52267
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da0p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,57a8e772-ea7e-11ed-ba1f-14dae9c52267,0x400080,0xe8a08808)
   rawuuid: 57a8e772-ea7e-11ed-ba1f-14dae9c52267
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: da0
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3

Geom name: da2
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,ebacf6ee-311d-11ed-8cee-14dae9c52267,0x80,0x400000)
   rawuuid: ebacf6ee-311d-11ed-8cee-14dae9c52267
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da2p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,ebba0e02-311d-11ed-8cee-14dae9c52267,0x400080,0x15d10a308)
   rawuuid: ebba0e02-311d-11ed-8cee-14dae9c52267
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da2
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da4
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,4670a59c-e9d4-11ed-ba1f-14dae9c52267,0x80,0x400000)
   rawuuid: 4670a59c-e9d4-11ed-ba1f-14dae9c52267
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da4p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,4685dd28-e9d4-11ed-ba1f-14dae9c52267,0x400080,0xe8a08808)
   rawuuid: 4685dd28-e9d4-11ed-ba1f-14dae9c52267
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: da4
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da5
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da5p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,6decd7af-45a4-11ec-9f16-94de807ce975,0x80,0x400000)
   rawuuid: 6decd7af-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da5p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,6faa39d7-45a4-11ec-9f16-94de807ce975,0x400080,0xe8a08808)
   rawuuid: 6faa39d7-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: da5
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da3
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,76028949-45a4-11ec-9f16-94de807ce975,0x80,0x400000)
   rawuuid: 76028949-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da3p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,768db535-45a4-11ec-9f16-94de807ce975,0x400080,0xe8a08808)
   rawuuid: 768db535-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: da3
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da6
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da6p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,058c9d85-e90c-11ed-ba1f-14dae9c52267,0x80,0x400000)
   rawuuid: 058c9d85-e90c-11ed-ba1f-14dae9c52267
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da6p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,059e0ee6-e90c-11ed-ba1f-14dae9c52267,0x400080,0x15d10a308)
   rawuuid: 059e0ee6-e90c-11ed-ba1f-14dae9c52267
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da6
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,6a52fb7e-45a4-11ec-9f16-94de807ce975,0x80,0x400000)
   rawuuid: 6a52fb7e-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da1p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,6fdd405b-45a4-11ec-9f16-94de807ce975,0x400080,0xe8a08808)
   rawuuid: 6fdd405b-45a4-11ec-9f16-94de807ce975
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: da1
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

The gmirror command defintely shows the wonkiness. Is the fix to just fiddle with the other options of gmirror until I have this straightened out?

 

[Patrick M. Hausen]

Nope, this is all intentional on the part of TrueNAS and the root cause is that one of your vdevs is a three way mirror. TN tries to configure the swap devices with the same level of redundancy as the ZFS pool. Probably the one with the system dataset - not quite sure what it does with multiple pools that have different levels of redundancy and swap partitions on all devices.

So you should probably remove one of the disks (zpool detach ...) of your first vdev.

In FreeNAS 11 (IIRC) swap was always configured as two way mirrors across all disks available. Then in 12 (again, IIRC) that changed to "figure out how many disks can fail in this pool and configure swap accordingly". I did open an issue in JIRA at the time arguing that that was a regression (effectively removed half of my swap on a RAIDZ2 pool) but it was closed with "not a bug".

I have reconfigured all my pools to mirrors since.

tldr; make sure all vdevs have the same two way mirror topology and the issue should fix itself after a reboot.

 

[GeorgePatches]

That's really interesting. I just assumed that it would make swap on the vdevs that I had defined. Appreciate the explanation.

 

[GeorgePatches]

Totally fixed just like you said. That is freaking wild to me.

Code:

root@truenas[~]# gmirror status
        Name    Status  Components
mirror/swap0  COMPLETE  da6p1 (ACTIVE)
                        da5p1 (ACTIVE)
mirror/swap1  COMPLETE  da4p1 (ACTIVE)
                        da3p1 (ACTIVE)
mirror/swap2  COMPLETE  da2p1 (ACTIVE)
                        da0p1 (ACTIVE)
mirror/swap3  COMPLETE  ada1p3 (ACTIVE)
                        ada0p3 (ACTIVE)