echo w > /proc/sysrq-trigger
Interesting.
One material point which I failed to mention is that the btrfs volume is
on an encrypted volume (cryptsetup luksOpen /dev/vdc backups).
The first step, "mount -r /dev/vg/ext2fs-snapshot
/btrfs-backup-volume/local-snapshot", seemed to trigger the problem.
When I did the echo to sysrq-trigger, it seemed to stop blocking, but
that might have been a coincidence. After the echo, kernel output
exceeded 100KB, so I saved it to https://davidnewall.com/kern.1
During rsync (--archive --one-file-system --hard-links --inplace
--numeric-ids --delete /btrfs-backup-volume/local-snapshot/
/btrfs-backup-volume/data/), initially there was no problem, but, then
it (df) seemed to hang again. The rsync took a long time to complete,
and before it did finish, I did the echo to sysrq-trigger again; kernel
output is saved to https://davidnewall.com/kern.2
The rsync finished not long after the echo to sysrq-trigger, but that's
probably also a coincidence. After rsync completed, df still hung. I
did another echo to sysrq-trigger, and saved kernel output to
https://davidnewall.com/kern.3
I tried a minor change in procedure to see if it would bring the system
back to normal response. Normally I'd do "btrfs subvolume snapshot",
but I tried unmounting the lvm2 snapshot first (umount
/btrfs-backup-volume/local-snapshot). It did not complete within the
expected time, and another echo to sysrq-trigger resulted in
https://davidnewall.com/kern.4
Eventually the umount completed and system came back to normal response.
I did the btrfs subvolume snapshot, and it completed faster than I could
notice without causing any issues.
After unmounting the btrfs volume, I tried each step again, and
everything completed within expected times without causing any hang.
Something which I did previously mention, but I'll repeat because it
might well be important, is that the base ext2 filesystem is on a
drbd-replicated volume. I don't know if it's part of the problem, and I
observe that the hang condition was not triggered at the point of
creating the lvm2 snapshot.
I greatly appreciate your advice and help.
Thanks,
David
