This appears to be some kind of race when there are a lot of pending metadata writes in flight. I went and unmounted/remounted again (after taking about 30 minutes of 5MB/s writes flushing an rsync with a ton of tiny files) and after the remount the issue went away again. So I can only presume it is an issue only when there are a million or so tiny files pending write. Matt On 2/4/20 3:41 AM, Matt Corallo wrote: > Things settled a tiny bit after unmount (see last email for the errors > that generated) and remount, and a balance -mconvert,soft worked: > > [268093.588482] BTRFS info (device dm-2): balance: start > -mconvert=raid1,soft -sconvert=raid1,soft > ... > [288405.946776] BTRFS info (device dm-2): balance: ended with status: 0 > > However, the enospc issue still appears and seems tied to a few of the > previously-allocated metadata blocks: > > # btrfs balance start -musage=0 /bigraid > ... > > [289714.420418] BTRFS info (device dm-2): balance: start -musage=0 -susage=0 > [289714.508411] BTRFS info (device dm-2): 64 enospc errors during balance > [289714.508413] BTRFS info (device dm-2): balance: ended with status: -28 > > # cd /sys/fs/btrfs/e2843f83-aadf-418d-b36b-5642f906808f/allocation/ && > grep -Tr . > metadata/raid1/used_bytes: 255838797824 > metadata/raid1/total_bytes: 441307889664 > metadata/disk_used: 511677595648 > metadata/bytes_pinned: 0 > metadata/bytes_used: 255838797824 > metadata/total_bytes_pinned: 999424 > metadata/disk_total: 882615779328 > metadata/total_bytes: 441307889664 > metadata/bytes_reserved: 4227072 > metadata/bytes_readonly: 65536 > metadata/bytes_may_use: 433502945280 > metadata/flags: 4 > system/raid1/used_bytes: 1474560 > system/raid1/total_bytes: 33554432 > system/disk_used: 2949120 > system/bytes_pinned: 0 > system/bytes_used: 1474560 > system/total_bytes_pinned: 0 > system/disk_total: 67108864 > system/total_bytes: 33554432 > system/bytes_reserved: 0 > system/bytes_readonly: 0 > system/bytes_may_use: 0 > system/flags: 2 > global_rsv_reserved: 536870912 > data/disk_used: 13645423230976 > data/bytes_pinned: 0 > data/bytes_used: 13645423230976 > data/single/used_bytes: 13645423230976 > data/single/total_bytes: 13661217226752 > data/total_bytes_pinned: 0 > data/disk_total: 13661217226752 > data/total_bytes: 13661217226752 > data/bytes_reserved: 117518336 > data/bytes_readonly: 196608 > data/bytes_may_use: 15064711168 > data/flags: 1 > global_rsv_size: 536870912 > > > Somewhat more frightening, this also happens on the system blocks: > > [288405.946776] BTRFS info (device dm-2): balance: ended with status: 0 > [289589.506357] BTRFS info (device dm-2): balance: start -musage=5 -susage=5 > [289589.905675] BTRFS info (device dm-2): relocating block group > 9676759498752 flags system|raid1 > [289590.807033] BTRFS info (device dm-2): found 89 extents > [289591.300212] BTRFS info (device dm-2): 16 enospc errors during balance > [289591.300216] BTRFS info (device dm-2): balance: ended with status: -28 > > Matt > > On 2/3/20 9:40 PM, Chris Murphy wrote: >> A developer might find it useful to see this reproduced with mount >> option enospc_debug. And soon after enospc the output from: >> >> cd /sys/fs/btrfs/UUID/allocation/ && grep -Tr . >> >> yep, space then dot at the end >>
