On 2016-06-28 08:14, Steven Haigh wrote:
On 28/06/16 22:05, Austin S. Hemmelgarn wrote:
On 2016-06-27 17:57, Zygo Blaxell wrote:
On Mon, Jun 27, 2016 at 10:17:04AM -0600, Chris Murphy wrote:
On Mon, Jun 27, 2016 at 5:21 AM, Austin S. Hemmelgarn
<ahferroin7@xxxxxxxxx> wrote:
On 2016-06-25 12:44, Chris Murphy wrote:
On Fri, Jun 24, 2016 at 12:19 PM, Austin S. Hemmelgarn
<ahferroin7@xxxxxxxxx> wrote:
OK but hold on. During scrub, it should read data, compute checksums
*and* parity, and compare those to what's on-disk - > EXTENT_CSUM in
the checksum tree, and the parity strip in the chunk tree. And if
parity is wrong, then it should be replaced.
Except that's horribly inefficient. With limited exceptions involving
highly situational co-processors, computing a checksum of a parity
block is
always going to be faster than computing parity for the stripe. By
using
that to check parity, we can safely speed up the common case of near
zero
errors during a scrub by a pretty significant factor.
OK I'm in favor of that. Although somehow md gets away with this by
computing and checking parity for its scrubs, and still manages to
keep drives saturated in the process - at least HDDs, I'm not sure how
it fares on SSDs.
A modest desktop CPU can compute raid6 parity at 6GB/sec, a less-modest
one at more than 10GB/sec. Maybe a bottleneck is within reach of an
array of SSDs vs. a slow CPU.
OK, great for people who are using modern desktop or server CPU's. Not
everyone has that luxury, and even on many such CPU's, it's _still_
faster to computer CRC32c checksums. On top of that, we don't appear to
be using the in-kernel parity-raid libraries (or if we are, I haven't
been able to find where we are calling the functions for it), so we
don't necessarily get assembly optimized or co-processor accelerated
computation of the parity itself. The other thing that I didn't mention
above though, is that computing parity checksums will always take less
time than computing parity, because you have to process significantly
less data. On a 4 disk RAID5 array, you're processing roughly 2/3 as
much data to do the parity checksums instead of parity itself, which
means that the parity computation would need to be 200% faster than the
CRC32c computation to break even, and this margin gets bigger and bigger
as you add more disks.
On small arrays, this obviously won't have much impact. Once you start
to scale past a few TB though, even a few hundred MB/s faster processing
means a significant decrease in processing time. Say you have a CPU
which gets about 12.0GB/s for RAID5 parity, and and about 12.25GB/s for
CRC32c (~2% is a conservative ratio assuming you use the CRC32c
instruction and assembly optimized RAID5 parity computations on a modern
x86_64 processor (the ratio on both the mobile Core i5 in my laptop and
the Xeon E3 in my home server is closer to 5%)). Assuming those
numbers, and that we're already checking checksums on non-parity blocks,
processing 120TB of data in a 4 disk array (which gives 40TB of parity
data, so 160TB total) gives:
For computing the parity to scrub:
120TB / 12.25GB = 9795.9 seconds for processing CRC32c csums of all the
regular data
120TB / 12GB = 10000 seconds for processing parity of all stripes
= 19795.9 seconds total
~ 5.4 hours total
For computing csums of the parity:
120TB / 12.25GB = 9795.9 seconds for processing CRC32c csums of all the
regular data
40TB / 12.25GB = 3265.3 seconds for processing CRC32c csums of all the
parity data
= 13061.2 seconds total
~ 3.6 hours total
The checksum based computation is approximately 34% faster than the
parity computation. Much of this of course is that you have to process
the regular data twice for the parity computation method (once for
csums, once for parity). You could probably do one pass computing both
values, but that would need to be done carefully; and, without
significant optimization, would likely not get you much benefit other
than cutting the number of loads in half.
And it all means jack shit because you don't get the data to disk that
quick. Who cares if its 500% faster - if it still saturates the
throughput of the actual drives, what difference does it make?
It has less impact on everything else running on the system at the time
because it uses less CPU time and potentially less memory. This is the
exact same reason that you want your RAID parity computation performance
as good as possible, the less time the CPU spends on that, the more it
can spend on other things. On top of that, there are high-end systems
that do have SSD's that can get multiple GB/s of data transfer per
second, and NVDIMM's are starting to become popular in the server
market, and those give you data transfer speeds equivalent to regular
memory bandwidth (which can be well over 20GB/s on decent hardware (I've
got a relatively inexpensive system using DDR3-1866 RAM that has roughly
22-24GB/s of memory bandwidth)). Looking at this another way, the fact
that the storage device is the bottleneck right now is not a good excuse
to not worry about making everything else as efficient as possible.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
