Balance failed with status -2, now the file system shows csum errors

[simonwaid]

I tried to balance a raid1 array. The balance failed with status -2

Now, when copying data to the file system I get the following in dmesg:

[  986.872832] BTRFS critical (device sda1): corrupt leaf: root=7 block=5758685151232 slot=1, csum end range (5848702070784) goes beyond the start range (5848701284352) of the next csum item
[  986.872840] BTRFS info (device sda1): leaf 5758685151232 gen 82170 total ptrs 6 free space 8129 owner 7
[  986.872841] 	item 0 key (18446744073709551606 128 5848693952512) itemoff 11967 itemsize 4316
[  986.872842] 	item 1 key (18446744073709551606 128 5848698372096) itemoff 8355 itemsize 3612
[  986.872843] 	item 2 key (18446744073709551606 128 5848701284352) itemoff 8351 itemsize 4
[  986.872844] 	item 3 key (18446744073709551606 128 5848701288448) itemoff 8339 itemsize 12
[  986.872845] 	item 4 key (18446744073709551606 128 5848701300736) itemoff 8331 itemsize 8
[  986.872846] 	item 5 key (18446744073709551606 128 5848702070784) itemoff 8279 itemsize 52
[  986.872847] BTRFS error (device sda1): block=5758685151232 write time tree block corruption detected
[  986.895795] BTRFS: error (device sda1) in btrfs_commit_transaction:2436: errno=-5 IO failure (Error while writing out transaction)
[  986.895809] BTRFS info (device sda1: state E): forced readonly
[  986.895815] BTRFS warning (device sda1: state E): Skipping commit of aborted transaction.
[  986.895820] BTRFS: error (device sda1: state EA) in cleanup_transaction:1983: errno=-5 IO failure

[simonwaid]

btrfs progs version is 6.01 Kernel version is 6.0.7

[adam900710]

The csum item overlap bugs have a lot of fixes:

  ea7036de0d36 ("btrfs: fix fsync failure and transaction abort after writes to prealloc extents")
  27b9a8122ff71a ("Btrfs: fix csum tree corruption, duplicate and outdated checksums")
  b84b8390d6009c ("Btrfs: fix file read corruption after extent cloning and fsync")
  40e046acbd2f36 ("Btrfs: fix missing data checksums after replaying a log tree")
  e289f03ea79bbc ("btrfs: fix corrupt log due to concurrent fsync of inodes with shared extents")

The fixes spans between v5.8 to v5.13, thus your fs may be affected during that time.

Thankfully btrfs check --init-csum-tree should handle it well by rebuilding the whole csum tree.

But before that, please run btrfs check --readonly to make sure it's the only problem.

[simonwaid]

I created the file system a few days ago using btrfs progs version is 6.01. After copying data to the raid1 system I added 2 disks and started a balance, which was interrupted by a power outage. After mounting / unmounting, btrfs check showed errors in the extend tree caused by the power outage. btrfs check --repair solved the issues (or it seemed like) and btrfs check was clean. I started a new btrfs balance which failed with status -2 and resulted in corruption in both the extend tree as well as the csum tree.

The reason why I created a new file system a few days ago was, that btrfs balance had failed already 2 times with status -2 before, each time resulting in corruption of the extend tree (version 5.16.2 of btrfs progs).

To me all this looks like a bug in btrfs balance

[simonwaid]

Thanks for the hint with btrfs check --init-csum-tree. I'll do that now.

The current output of btrfs check is the following. However, I ran btrfs check --repair already to fix the corruption in the extend tree.

sudo btrfs check /dev/sda1 
Opening filesystem to check...
Checking filesystem on /dev/sda1
UUID: 73e1de51-7507-4ab9-ba63-7a9754c93663
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
there are no extents for csum range 5848701284352-5848701308928
csum exists for 5848701284352-5848701308928 but there is no extent record
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 5758609899520 bytes used, error(s) found
total csum bytes: 5616686120
total tree bytes: 7123337216
total fs tree bytes: 1100529664
total extent tree bytes: 101433344
btree space waste bytes: 441645203
file data blocks allocated: 5751486562304
 referenced 5751486562304

[simonwaid]

Update

btrfs check --init-csum-tree resultet in the following:

repair deleting extent record: key [11626074750976,169,0]
Repaired extent references for 11626074750976
ref mismatch on [11626074767360 16384] extent item 1, found 0
backref 11626074767360 root 7 not referenced back 0x5617b54204f0
incorrect global backref count on 11626074767360 found 1 wanted 0
backpointer mismatch on [11626074767360 16384]
owner ref check failed [11626074767360 16384]
repair deleting extent record: key [11626074767360,169,0]
kernel-shared/free-space-tree.c:736: remove_free_space_extent: Assertion `key.type == BTRFS_FREE_SPACE_EXTENT_KEY` failed, value 0
btrfs(remove_from_free_space_tree+0x325)[0x5617a1505eee]
btrfs(btrfs_run_delayed_refs+0x438)[0x5617a14fddcd]
btrfs(btrfs_commit_transaction+0x3f)[0x5617a150dfe2]
btrfs(+0x845ca)[0x5617a15515ca]
btrfs(+0x8772b)[0x5617a155472b]
btrfs(main+0x91)[0x5617a14e6072]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f1510e29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f1510e29e40]
btrfs(_start+

[simonwaid]

Now btrfs check

backref 14974568022016 root 7 not referenced back 0x55d301173db0
incorrect global backref count on 14974568022016 found 1 wanted 0
backpointer mismatch on [14974568022016 16384]
owner ref check failed [14974568022016 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
there is no free space entry for 30408704-30457856
cache appears valid but isn't 30408704
there is no free space entry for 894457348096-894464999424
cache appears valid but isn't 894457348096
there is no free space entry for 5758507810816-5758507827200
cache appears valid but isn't 5758507810816
there is no free space entry for 11625567354880-11625567895552
cache appears valid but isn't 11625567354880
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 5754480979968 bytes used, error(s) found
total csum bytes: 5609700716
total tree bytes: 6888980480
total fs tree bytes: 1099268096
total extent tree bytes: 110804992
btree space waste bytes: 210756027
file data blocks allocated: 5744333533184
 referenced 5744333533184

[simonwaid]

Running sudo btrfs check --repair /dev/sda1

Opening filesystem to check...
Checking filesystem on /dev/sda1
UUID: 73e1de51-7507-4ab9-ba63-7a9754c93663
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
ref mismatch on [11626074767360 16384] extent item 1, found 0
backref 11626074767360 root 7 not referenced back 0x558526cdfb90
incorrect global backref count on 11626074767360 found 1 wanted 0
backpointer mismatch on [11626074767360 16384]
owner ref check failed [11626074767360 16384]
repair deleting extent record: key [11626074767360,169,0]
kernel-shared/free-space-tree.c:736: remove_free_space_extent: Assertion `key.type == BTRFS_FREE_SPACE_EXTENT_KEY` failed, value 0
btrfs(remove_from_free_space_tree+0x325)[0x55851f147eee]
btrfs(btrfs_run_delayed_refs+0x438)[0x55851f13fdcd]
btrfs(btrfs_commit_transaction+0x3f)[0x55851f14ffe2]
btrfs(+0x845ca)[0x55851f1935ca]
btrfs(+0x8772b)[0x55851f19672b]
btrfs(main+0x91)[0x55851f128072]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f0c40629d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f0c40629e40]
btrfs(_start+0x25)[0x55851f127c05]
Abgebrochen