On 2020/2/26 上午11:38, Jonathan H wrote: > On Tue, Feb 25, 2020 at 3:58 PM Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: >> >> >> >> On 2020/2/26 上午4:39, Jonathan H wrote: >>> Hello everyone, >>> >>> Previously, I was running an array with six disks all connected via >>> USB. I am running raid1c3 for metadata and raid6 for data, kernel >>> 5.5.4-arch1-1 and btrfs --version v5.4, and I use bees for >>> deduplication. Four of the six drives are stored in a single four-bay >>> enclosure. Due to my oversight, TLER was not enabled for any of the >>> drives, so when one of them started failing, the enclosure was reset >>> and all four drives were disconnected. >>> >>> After rebooting, the file system was still mountable. I saw some >>> transid errors in dmesg, >> >> This means the fs is already corrupted. >> If btrfs check is run before mount, it may provide some pretty good >> debugging > > Here is the output from "btrfs check --check-data-csum": http://ix.io/2cH7 It's great that your metadata is safe. > >> Also the exact message for the transid error and some context would help >> us to determine how serious the corruption is. > > Unfortunately, I can't reproduce the transid errors, since they seem > to have been fixed. However, I do recall that the wanted and found > generations numbers differed by less than a hundred? So your previous transid may be a bad copy on your failing disk and btrfs found it way to get next good copy. The biggest concern is no longer a concern now. > >>> I noticed that almost all of the files give an I/O error when read, >>> and similar kernel messages are generated, but with positive roots. >> >> Please give the exact dmesg. >> Including all the messages for the same bytenr. > > Here is the dmesg: http://ix.io/2cH7 More context would be welcomed. Anyway, even with more context, it may still lack the needed info as such csum failure message is rate limited. The mirror num 2 means it's the first rebuild try failed. Since only the first rebuild try failed, and there are some corrected data read, it looks btrfs can still rebuild the data. Since you have already observed some EIO, it looks like write hole is involved, screwing up the rebuild process. But it's still very strange, as I'm expecting more mirror number other than 2. For your 6 disks with 1 bad disk, we still have 5 ways to rebuild data, only showing mirror num 2 doesn't look correct to me. > >> Although I don't believe it's the hardware to blame, but you can still >> try to disable write cache on all related devices, as an experiment to >> rule out bad disk flush/fua behavior. > > I've disabled the write cashes for now. > BTW, since your free space cache is already corrupted, it's recommended to clear the space cache. For now, since it looks like write hole is involved, the only way to solve the problem is to remove all offending files (including a super large file in root 5). You can use `btrfs inspect logical-resolve <bytenr> <mnt>" to see all the involved files. The full <bytenr> are the bytenr shown in btrfs check --check-data-csum output. Thanks, Qu
Attachment:
signature.asc
Description: OpenPGP digital signature
