On 2016-01-16 13:07, Rich Freeman wrote:
On Sat, Jan 16, 2016 at 9:10 AM, Duncan <1i5t5.duncan@xxxxxxx> wrote:
Al posted on Sat, 16 Jan 2016 12:27:16 +0000 as excerpted:
Is there any urgency for dedup? What's wrong with storing the hash on
disk with the block and having a separate process dedup the written data
over time;
There's actually uses for both inline and out-of-line[1] aka delayed
dedup. Btrfs already has a number of independent products doing various
forms of out-of-line dedup, so what's missing and being developed now is
the inline dedup option, which being directly in the write processing,
must be handled by btrfs itself -- it can't be primarily done by third
parties with just a few kernel calls, like out-of-line dedup can.
Does the out-of-line dedup option actually utilize stored hashes, or
is it forced to re-read all the data to compute hashes? If it is
collecting checksums/etc is this done efficiently?
AFAIK, duperemove has the option to store block hashes in a database to
save them between runs (I'm pretty sure that it invalidates hashes if
the file containing the block changed, but I'm not certain).
I think he is actually suggesting a hybrid approach where a bit of
effort is done during operations to greatly streamline out-of-line
deduplication. I'm not sure how close we are to that already, or if
any room for improvement remains.
There isn't any implementation I know of that does this. In theory, it
would be pretty easy if we could somehow get block checksums from BTRFS
in userspace.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
