On 2016-06-27 17:57, Zygo Blaxell wrote:
On Mon, Jun 27, 2016 at 10:17:04AM -0600, Chris Murphy wrote:
On Mon, Jun 27, 2016 at 5:21 AM, Austin S. Hemmelgarn
<ahferroin7@xxxxxxxxx> wrote:
On 2016-06-25 12:44, Chris Murphy wrote:
On Fri, Jun 24, 2016 at 12:19 PM, Austin S. Hemmelgarn
<ahferroin7@xxxxxxxxx> wrote:
OK but hold on. During scrub, it should read data, compute checksums
*and* parity, and compare those to what's on-disk - > EXTENT_CSUM in
the checksum tree, and the parity strip in the chunk tree. And if
parity is wrong, then it should be replaced.
Except that's horribly inefficient. With limited exceptions involving
highly situational co-processors, computing a checksum of a parity block is
always going to be faster than computing parity for the stripe. By using
that to check parity, we can safely speed up the common case of near zero
errors during a scrub by a pretty significant factor.
OK I'm in favor of that. Although somehow md gets away with this by
computing and checking parity for its scrubs, and still manages to
keep drives saturated in the process - at least HDDs, I'm not sure how
it fares on SSDs.
A modest desktop CPU can compute raid6 parity at 6GB/sec, a less-modest
one at more than 10GB/sec. Maybe a bottleneck is within reach of an
array of SSDs vs. a slow CPU.
OK, great for people who are using modern desktop or server CPU's. Not
everyone has that luxury, and even on many such CPU's, it's _still_
faster to computer CRC32c checksums. On top of that, we don't appear to
be using the in-kernel parity-raid libraries (or if we are, I haven't
been able to find where we are calling the functions for it), so we
don't necessarily get assembly optimized or co-processor accelerated
computation of the parity itself. The other thing that I didn't mention
above though, is that computing parity checksums will always take less
time than computing parity, because you have to process significantly
less data. On a 4 disk RAID5 array, you're processing roughly 2/3 as
much data to do the parity checksums instead of parity itself, which
means that the parity computation would need to be 200% faster than the
CRC32c computation to break even, and this margin gets bigger and bigger
as you add more disks.
On small arrays, this obviously won't have much impact. Once you start
to scale past a few TB though, even a few hundred MB/s faster processing
means a significant decrease in processing time. Say you have a CPU
which gets about 12.0GB/s for RAID5 parity, and and about 12.25GB/s for
CRC32c (~2% is a conservative ratio assuming you use the CRC32c
instruction and assembly optimized RAID5 parity computations on a modern
x86_64 processor (the ratio on both the mobile Core i5 in my laptop and
the Xeon E3 in my home server is closer to 5%)). Assuming those
numbers, and that we're already checking checksums on non-parity blocks,
processing 120TB of data in a 4 disk array (which gives 40TB of parity
data, so 160TB total) gives:
For computing the parity to scrub:
120TB / 12.25GB = 9795.9 seconds for processing CRC32c csums of all the
regular data
120TB / 12GB = 10000 seconds for processing parity of all stripes
= 19795.9 seconds total
~ 5.4 hours total
For computing csums of the parity:
120TB / 12.25GB = 9795.9 seconds for processing CRC32c csums of all the
regular data
40TB / 12.25GB = 3265.3 seconds for processing CRC32c csums of all the
parity data
= 13061.2 seconds total
~ 3.6 hours total
The checksum based computation is approximately 34% faster than the
parity computation. Much of this of course is that you have to process
the regular data twice for the parity computation method (once for
csums, once for parity). You could probably do one pass computing both
values, but that would need to be done carefully; and, without
significant optimization, would likely not get you much benefit other
than cutting the number of loads in half.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
