[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: inquiry on btrfs send/receive

Hi Alex, Jan,

I was also interested in send/receive semantics & was thinking that if
we adhere to the semantics as in

it is impossible to track the "deleted" items (files,dirs, eXtended
attributes). I can develop a command which compare two subvolumes an
extract all of this kind of information. But this command would return
correct information *only if*
  A) a subvolume is a snapshot of the other one
  B.1) the reference snapshot is not touched OR
  B.2) I have the lastgen "when" the snapshot is taken

then transid/generation number could be made to work to determine the
differences between two subvolumes including the deletes that were
involved. I agree that file-clone is tricky as you pointed, but we
dont use it in our environment.

Do you still see issues with it that I am missing?

Also out of curiosity, would you mind sharing how you avoided looking
at transid? Would it be that all meta changes are logged in
tree-modification-log or equivalent?


PS: Alex/Jan, sorry for the multiple mails. including the linux-btrfs
list refused HTML msgs.

On Mon, Jun 4, 2012 at 6:52 PM, Alexander Block <ablock84@xxxxxxxxxxxxxx> wrote:
> On Mon, Jun 4, 2012 at 2:39 PM, Alex Lyakas
> <alex.bolshoy.btrfs@xxxxxxxxx> wrote:
> > Hi Jan, Alex,
> >
> > I have seen some discussions about btrfs send/receive functionality
> > being developed by you. I have also been interested in this. I spent
> > some time coding a prototype doing something like Alex described in
> > http://www.spinics.net/lists/linux-btrfs/msg16175.html, i.e., walking
> > over FS tree and pulling those items that have transid/generation
> > larger than a particular value. I realized though, that there are many
> > issues with that approach, and also probably there are many issues I
> > am not aware of. Some of the issues I realized:
> Well, you are, like me in the beginning, on a wrong track ;) Using the
> transid only is not the way send/receive will be implemented when it's
> done.
> >
> > # How does one track changes in generic INODE_ITEM properties, like
> > "mode" or "uid/gid"? Whenever such property gets changed, INODE_ITEM
> > gets stamped with a new transid, but do we need to compare it with the
> > previous version on the receive side to realize what has changed?
> > # File size - is it required, again, to compare vs previous size, to
> > realize file truncation? (file grow perhaps can be realized via new
> > # What should be done if INODE_ITEM::flags change (e.g., inode gets
> > nodatacow/nodatasum flags set). What should be done at receive side?
> > # How does one track deletion of INODE_ITEMs? Or, deletion and
> > re-creation of a INODE_ITEM with the same inode number? (I saw that
> > inode_cache mount option allows to re-use inode numbers, so I think it
> > can happen.) Does this mean that on receive side, it is required to
> > compare contents of each directory vs previous version?
> > # What should be done with INODE_ITEMs like block/char device, FIFO or a socket?
> > # XATTR_ITEMs: although they have a transid stamp, again, need to
> > track deletion/re-creation of them. Again by comparing?
> > # INODE_REFs: these seem most tough to me, because they don't have
> > transid stamps. How such scenario can be handled: an INODE_ITEM had
> > two INODE_REFs with names N1 and N2. But now on the send side, both
> > those INODE_REFs were deleted and INODE_REFs N3 and N4 were created.
> > Does that mean we need to always compare all INODE_REFs for each
> > INODE_ITEM, or we perhaps can use DIR_ITEMs/DIR_INDEXs of parent
> > INODE_ITEM to detect changes in INODE_REFs?
> >
> > All in all, it looks like the approach of navigating the FS tree and
> > trying to *understand* specifically which modifications were
> > performed, is quite error-prone. And I am sure there are modifications
> > I am not aware about.
> The problem with the transid only approach is, that there is no way
> to find out "what" has changed in an inode. You only know which inode
> has changed. You could probably determine which extents have
> changed, but this is unreliable as you've already read in my older mail.
> There is also absolutely no way to detect deleted/moved files/dirs.
> When send/receive is released and working, I may try to implement
> a mode that only relies on the transid, but this has low priority for me
> and also needs some changes to other parts of btrfs. If I implement
> that, it would however still be unable to detect what has changed and
> would also miss deleted/moved dirs. You could compare it to rsync
> without the --delete option (which I use frequently to transfer VM
> images).
> >
> > I was wondering, what state your work is in? Is it possible to look at
> > some code or prototype, to understand what approach have you taken, or
> > perhaps an overall description of the approach?
> Currently, most things work as expected. But, the code is not in a state
> to be released. Jan, Arne, David and Stefan are currently reviewing my
> code and I have a lot of TODO's due to the suggestions they all made.
> >
> > Jan, I saw that you provided some new code for backref resolving. Can
> > you give a hint of how is that related to the send/receive
> > functionality?
> It is not only related to the send/receive code. It is currently mainly related
> to the upcoming qgroups patches that Jan is preparing. It may also be
> related to other parts of btrfs (as far as I know, scrub is also using the
> backref resolving code). His patches are however a requirement for
> send/receive to work properly when I release my first patches).
> >
> > Thanks,
> > Alex.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]    [Yosemite Photos]    [Free Online Dating]     [Linux Kernel]     [Linux SCSI]     [XFree86]

Add to Google Powered by Linux