Hi, Just FYI, these patches are also available on the following site, kernel: https://github.com/liubogithub/btrfs-work.git dedup-on-3.14-linux progs: https://github.com/liubogithub/btrfs-progs.git dedup thanks, -liubo On Thu, Apr 10, 2014 at 11:48:30AM +0800, Liu Bo wrote: > Hello, > > This the 10th attempt for in-band data dedupe, based on Linux _3.14_ kernel. > > Data deduplication is a specialized data compression technique for eliminating > duplicate copies of repeating data.[1] > > This patch set is also related to "Content based storage" in project ideas[2], > it introduces inband data deduplication for btrfs and dedup/dedupe is for short. > > * PATCH 1 is a speed-up improvement, which is about dedup and quota. > > * PATCH 2-5 is the preparation work for dedup implementation. > > * PATCH 6 shows how we implement dedup feature. > > * PATCH 7 fixes a backref walking bug with dedup. > > * PATCH 8 fixes a free space bug of dedup extents on error handling. > > * PATCH 9 adds the ioctl to control dedup feature. > > * PATCH 10 targets delayed refs' scalability problem of deleting refs, which is > uncovered by the dedup feature. > > * PATCH 11-16 fixes bugs of dedupe including race bug, deadlock, abnormal > transaction abortion and crash. > > * btrfs-progs patch(PATCH 17) offers all details about how to control the > dedup feature on progs side. > > I've tested this with xfstests by adding a inline dedup 'enable & on' in xfstests' > mount and scratch_mount. > > > ***NOTE*** > Known bugs: > * Mounting with options "flushoncommit" and enabling dedupe feature will end up > with _deadlock_. > > > TODO: > * a bit-to-bit comparison callback. > > All comments are welcome! > > > [1]: http://en.wikipedia.org/wiki/Data_deduplication > [2]: https://btrfs.wiki.kernel.org/index.php/Project_ideas#Content_based_storage > > v10: > - fix a typo in the subject line. > - update struct 'btrfs_ioctl_dedup_args' in the kernel side to fix > 'Inappropriate ioctl for device'. > > v9: > - fix a deadlock and a crash reported by users. > - fix the metadata ENOSPC problem with dedup again. > > v8: > - fix the race crash of dedup ref again. > - fix the metadata ENOSPC problem with dedup. > > v7: > - rebase onto the lastest btrfs > - break a big patch into smaller ones to make reviewers happy. > - kill mount options of dedup and use ioctl method instead. > - fix two crash due to the special dedup ref > > For former patch sets: > v6: http://thread.gmane.org/gmane.comp.file-systems.btrfs/27512 > v5: http://thread.gmane.org/gmane.comp.file-systems.btrfs/27257 > v4: http://thread.gmane.org/gmane.comp.file-systems.btrfs/25751 > v3: http://comments.gmane.org/gmane.comp.file-systems.btrfs/25433 > v2: http://comments.gmane.org/gmane.comp.file-systems.btrfs/24959 > > Liu Bo (16): > Btrfs: disable qgroups accounting when quota_enable is 0 > Btrfs: introduce dedup tree and relatives > Btrfs: introduce dedup tree operations > Btrfs: introduce dedup state > Btrfs: make ordered extent aware of dedup > Btrfs: online(inband) data dedup > Btrfs: skip dedup reference during backref walking > Btrfs: don't return space for dedup extent > Btrfs: add ioctl of dedup control > Btrfs: improve the delayed refs process in rm case > Btrfs: fix a crash of dedup ref > Btrfs: fix deadlock of dedup work > Btrfs: fix transactin abortion in __btrfs_free_extent > Btrfs: fix wrong pinned bytes in __btrfs_free_extent > Btrfs: use total_bytes instead of bytes_used for global_rsv > Btrfs: fix dedup enospc problem > > fs/btrfs/backref.c | 9 + > fs/btrfs/ctree.c | 2 +- > fs/btrfs/ctree.h | 86 ++++++ > fs/btrfs/delayed-ref.c | 26 +- > fs/btrfs/delayed-ref.h | 3 + > fs/btrfs/disk-io.c | 37 +++ > fs/btrfs/extent-tree.c | 235 +++++++++++++--- > fs/btrfs/extent_io.c | 22 +- > fs/btrfs/extent_io.h | 16 ++ > fs/btrfs/file-item.c | 244 +++++++++++++++++ > fs/btrfs/inode.c | 635 ++++++++++++++++++++++++++++++++++++++----- > fs/btrfs/ioctl.c | 167 ++++++++++++ > fs/btrfs/ordered-data.c | 44 ++- > fs/btrfs/ordered-data.h | 13 +- > fs/btrfs/qgroup.c | 3 + > fs/btrfs/relocation.c | 3 + > fs/btrfs/transaction.c | 41 +++ > fs/btrfs/transaction.h | 1 + > include/trace/events/btrfs.h | 3 +- > include/uapi/linux/btrfs.h | 12 + > 20 files changed, 1471 insertions(+), 131 deletions(-) > > -- > 1.8.2.1 > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
