Hi, I had a RAID1 disk failure recently, and a limitation in number of SATA connectors meant I could not do a live replace. I'm still in the progress of resolving the issue, but posting some feedback here on the issues I faced and what could have helped. **What all I did** - remove the dying disk and mount in degraded mode. This didn't work, but _didn't provide me a failure reason_ - swap the dead disk post boot with the new one. Tried a btrfs device add, but it failed as well because of too many failing reads from the dead disk (which wasn't even attached). - Started a btrfs device delete, which finally pointed me to the root cause: `ERROR: error removing device '/dev/sdb1': No space left on device` In short: 1. My 2.7TBx3 setup was storing 2.9TB of data 2. Removing any one the disks was not possible without losing some data However, getting to this error message took me hours of effort, despite having read the btrfs wiki. **Feedback** 1. The `btrfs device delete` command should have failed immediately instead of making me wait for ages. 2. A delete command should stop writes to a disk, unless avoidable. This would have left some data on the disk, and a subsequent delete could have cleaned it up much faster. It failed in the worst manner possible, because all deletion progress reversed once it failed: it balanced back to complete usage, causing more strain on a dying disk. 3. A degraded mount failure should provide some error message. *Status*: I cleared up some disk space, and ran another delete which is still ongoing. I don't have physical access to the system for a few more days, so adding the new disk will take me some time. --- ## Debug Details ``` uname -a Linux tatooine.captnemo.in 4.20.10-arch1-1-ARCH #1 SMP PREEMPT Fri Feb 15 17:49:06 UTC 2019 x86_64 GNU/Linux btrfs --version btrfs-progs v4.20.1 ``` A few more details are on a reddit post[0] I made to help debug the issue and a gist[1] has disk usage and filesystem details. [0]: https://www.reddit.com/r/archlinux/comments/asrlam/btrfscleaner_at_100_cpu_usage_on_raid1_setup/egwc047/ [1]: https://paste.ubuntu.com/p/GDNKNBqqFy/
