On 23.03.19 г. 1:30 ч., Jan Alexander Steffens wrote: > Hi all, > > we have a build server here that accumulated lots of uncleaned subvolumes. > Snapshots containing build roots are frequently created and destroyed. > There's also a borg backup script which creates and destroys a > snapshot of the root subvolume. > We noticed this only after a reboot for a kernel upgrade, after which > the cleaner almost ground the system to a halt. > > The server has a single RAID1 btrfs covering two unpartitioned HDDs. > > After letting the cleaner do its job, I started some balance > operations (the same as in > https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-balance.sh). > During these operation, btrfs raised two warnings: > Those two warnings are essentially identical. The first one is saying that the block group which is being relocated still has reserved space on it - something which shouldn't really happen. The 2nd one also barks about a block group which is being deleted yet has a non-zero reservation count. > > Later on, long after the balance operations finished, the > btrfs-cleaner thread crashed: > > Mar 22 22:18:29 soyuz kernel: ------------[ cut here ]------------ > Mar 22 22:18:29 soyuz kernel: kernel BUG at mm/slub.c:305! > Mar 22 22:18:29 soyuz kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI > Mar 22 22:18:31 soyuz kernel: CPU: 6 PID: 305 Comm: btrfs-cleaner > Tainted: G W OE 5.0.3-arch1-1-ARCH #1 > Mar 22 22:18:31 soyuz kernel: Hardware name: FUJITSU /D3417-B2, BIOS > V5.0.0.12 R1.8.0.SR.1 for D3417-B2x 05/16/2017 > Mar 22 22:18:31 soyuz kernel: RIP: 0010:__slab_free+0x1e4/0x380 > Mar 22 22:18:31 soyuz kernel: Code: 4c 89 ff e8 7e d8 5f 00 48 8b 54 > 24 18 48 8b 74 24 20 48 89 04 24 e9 ba fe ff ff e8 e6 fb ff ff 85 c0 > 0f 85 56 fe ff ff eb ad <0f> 0b 48 89 54 24 08 48 89 74 24 10 48 89 4c > 24 18 bf 01 00 00 00 > Mar 22 22:18:31 soyuz kernel: RSP: 0000:ffffa56fc7157d70 EFLAGS: 00010246 > Mar 22 22:18:31 soyuz kernel: RAX: ffff96b7c058f4b8 RBX: > ffff96b7c058f4b8 RCX: ffff96b7c058f4b8 > Mar 22 22:18:31 soyuz kernel: RDX: 00000000001a000c RSI: > ffff96b7c058f4b8 RDI: ffff96c265ff2c00 > Mar 22 22:18:31 soyuz kernel: RBP: ffffa56fc7157e08 R08: > 0000000000000001 R09: ffffffffc0484831 > Mar 22 22:18:31 soyuz kernel: R10: ffffa56fc7157de0 R11: > ffff96bc5e372be0 R12: ffffd9e595016380 > Mar 22 22:18:31 soyuz kernel: R13: ffff96c265ff2c00 R14: > ffff96b7c058f4b8 R15: ffffa56fc7157e40 > Mar 22 22:18:31 soyuz kernel: FS: 0000000000000000(0000) > GS:ffff96c26e380000(0000) knlGS:0000000000000000 > Mar 22 22:18:31 soyuz kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > Mar 22 22:18:31 soyuz kernel: CR2: 00007fa6a0b331e0 CR3: > 0000000ee3b60002 CR4: 00000000003606e0 > Mar 22 22:18:31 soyuz kernel: DR0: 0000000000000000 DR1: > 0000000000000000 DR2: 0000000000000000 > Mar 22 22:18:31 soyuz kernel: DR3: 0000000000000000 DR6: > 00000000fffe0ff0 DR7: 0000000000000400 > Mar 22 22:18:31 soyuz kernel: Call Trace: > Mar 22 22:18:31 soyuz kernel: ? radix_tree_node_ctor+0x40/0x40 > Mar 22 22:18:31 soyuz kernel: ? delete_node+0x37/0x1f0 > Mar 22 22:18:31 soyuz kernel: ? radix_tree_delete_item+0x39/0xc0 > Mar 22 22:18:31 soyuz kernel: kmem_cache_free+0x1d3/0x200 > Mar 22 22:18:31 soyuz kernel: btrfs_kill_all_delayed_nodes+0xb1/0x100 [btrfs] > Mar 22 22:18:31 soyuz kernel: ? btrfs_destroy_pinned_extent+0x110/0x110 [btrfs] > Mar 22 22:18:31 soyuz kernel: btrfs_clean_one_deleted_snapshot+0x7c/0xf0 [btrfs] > Mar 22 22:18:31 soyuz kernel: cleaner_kthread+0x10d/0x130 [btrfs] > Mar 22 22:18:31 soyuz kernel: kthread+0x112/0x130 > Mar 22 22:18:31 soyuz kernel: ? kthread_park+0x80/0x80 > Mar 22 22:18:31 soyuz kernel: ret_from_fork+0x35/0x40 > Mar 22 22:18:31 soyuz kernel: Modules linked in: lz4 lz4_compress zram And this is just a memory corruption. Unfortunately there isn't anything which can aid further debugging here. In case you can reproduce the problem it might make sense to apply debugging patches to help pinpoint the root cause. > > I'll reboot the system now to get the cleaner going again. Any suggestions? > > Greetings, > Jan Steffens > Arch Linux >
