On 07. feb. 2016 00:32, Chris Murphy wrote:
It's probably unrelated the problem, but I would given the many bug
fixes (including in send/receive) since kernel 3.19, and progs 4.1,
that I'd get both systems using the same kernel and progs version. I
suspect most of upstream's testing before release for send/receive is
with matching kernel and progs versions. My understanding is most of
the send code is in the kernel, and most of the receive code is in
progs (of course, receive also implies writing to a Btrfs volume as
well which would be kernel code too). I really wouldn't intentionally
mix and match versions like this, unless you're trying to find bugs as
a result of mismatching versions.
Ok, that sounds like good advice. My thought was to keep the versions
different to reduce the risk of my data getting nuked on both systems
because of some obscure bug in one specific version, since btrfs is not
100% stable yet. Also, it was much easier to create a small read-only
"NAS OS" to run from a USB stick using Arch Linux than with Ubuntu.
Guess I'll have to re-evaluate this then.
Note that this is a logical address. The chunk tree will translate
that into separate physical sectors on the actual drives. This kind of
corruption suggests that it's not media, or even storage stack related
like a torn write or anything like that. I'm not sure how it can
happen, someone else who knows the sequence of data checksumming, data
allocation being split into two paths for writes, and metadata writes,
would have to speak up.
Being a logical or physical address is not the point here at all. The
file ended up corrupted because somehow both copies of the file had a
checksum mismatch on the exact same (4k) block of data. This should not
be possible. For now I can only see two explanations - a weird bug
somewhere in btrfs or corrupt RAM, because either the data block or the
checksum must have been corrupted somewhere between the calculation and
writing to the disk. Next step now is a few rounds of memtest.
Also, the file is still recoverable most likely. You can use btrfs
restore to extract it from the unmounted file system without
complaining about checksum mismatches. It's just that the normal read
path won't hand over data it thinks is corrupt.
I still haven't learned enough about the capabilities of btrfs, so I
wasn't aware of this. And since this was the backup server, I replaced
the file from the main server to see if this would break the incremental
send/receive (and that worked perfectly, since I kept the inode i guess).
This is not what I expect from a raid10!
Technically what you don't expect from raid10 is any notification that
the file may be corrupt at all. It'd be interesting to extract the
file with restore, and then compare hashes to a known good copy.
Well, I really don't expect this to be happening at all. If this is a
bug in btrfs, it could just as well have struck the same file on the
server. Without a backup I could not know if it was the data or the
checksum that was bogus. This being a 16GB edited family film video
file, I had the original source deleted, so I could very well end up
with some very annoying chop in the video in a worst case scenario.
Anyway, I just hope that if there is a bug in the code, this could help
find it.
--
Tom Arild Naess
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
