Chasing IO errors. BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2907: errno=-5 IO failure I've just had an odd one. Over the last few days I've noticed a file system blocking, if that is the correct term, and this morning go read only. This resulted in a lot of checksum errors. Having spotted the file system go read only in the logs and then noted the error message in the subject shortly after booting I assumed a hardware error and changed the SATA cable. That had no effect so I isolated the disk and mounted the respective file system degraded. Shortly after mounting the degraded file system I had the same error again. So I unmounted the file system edited fstab and swapped the disk which I though originally had the error with the one now showing an error. The file system is btrfs, kernel 5.2.9, RAID 1 with three WD reds of 3, 3 and 4 TB. btrfs is on top of luks. The original 'blocking' behaviour seemed to manifest itself as I upgraded the kernel to 5.2.5 or 5.2.7 a day or two ago. So I tried 5.1.21 to see if that made a difference when the error was showing. It did not. Yesterday I had a backup with rsync, started early in the morning that should take minutes to complete still running 8h later with two CPU cores maxed. Up until I had the file system go read only I had not noticed anything amiss in the logs, but to be honest, I'd not looked very hard. smartctl did not show anything amiss with the drives. Does this sound like a hardware error? I have ordered a replacement drive, if it is not needed as a replacement I will put it into a homebrew NAS. I've hit the issue again. Hopefully the system is up long enough to post this. I'm a bit worried that trying to track this down disconnecting a disk at a time I might hit the btrfs split brain issue. Pete
