Btrfs devel - Dec 2012 - Incremental btrfs receive in opposite direction fails

Hi everyone!

(I use kernel 3.7.1 and mason/btrfs-progs.git head + the receive
lchown patch [1]. Is there a more up to date btrfs-progs repo for
send/receive enthusiasts?)

What I''m trying is basically:

# Transfer a snapshot of my old system from a backup to the new system
btrfs send /mnt/bak/.snap | btrfs receive /                 # OK

# Merge the old system''s files with the new system''s files
btrfs subvolume snapshot /.snap /.snap-merge                # OK
mkdir /.snap-merge/.old                                     # OK
mv /.snap-merge/* /.snap-merge/.old                         # OK
cp -a --reflink /* /.snap-merge                             # OK
btrfs subvolume snapshot -r /.snap-merge /.snap-merge-done  # OK

# Incrementally transfer the merged files back to the backup drive
btrfs send -p /.snap /.snap-merge-done | btrfs receive -p /mnt/bak

The last step fails due to the changed direction of the transfer.
"ERROR: could not find parent subvolume". It seems like btrfs receive
cannot find the parent/clone source because it is searching for its UUID
only by looking through the /mnt/bak snapshots'' received_uuid fields.

Is there a way for me to directly change the received_uuid of
/mnt/bak/.snap to make it identical to the UUID of /.snap? This looks
like the easiest way if I only need to do it once.

[1] https://patchwork.kernel.org/patch/1606091/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi,

On Sat, Dec 29, 2012 at 12:42 AM,  <junlion@tormail.org>
wrote:
> Hi everyone!
>
> (I use kernel 3.7.1 and mason/btrfs-progs.git head + the receive
> lchown patch [1]. Is there a more up to date btrfs-progs repo for
> send/receive enthusiasts?)
There is no special repo, but you may want, in addition to the patch
you mentioned, apply this one as well:
https://patchwork.kernel.org/patch/1604391/
Whether you need it or not depends on how your subvolumes are
organized in a file tree.

>
> What I''m trying is basically:
>
> # Transfer a snapshot of my old system from a backup to the new system
> btrfs send /mnt/bak/.snap | btrfs receive /                 # OK
>
> # Merge the old system''s files with the new system''s
files
> btrfs subvolume snapshot /.snap /.snap-merge                # OK
> mkdir /.snap-merge/.old                                     # OK
> mv /.snap-merge/* /.snap-merge/.old                         # OK
> cp -a --reflink /* /.snap-merge                             # OK
> btrfs subvolume snapshot -r /.snap-merge /.snap-merge-done  # OK
>
> # Incrementally transfer the merged files back to the backup drive
> btrfs send -p /.snap /.snap-merge-done | btrfs receive -p /mnt/bak
>
> The last step fails due to the changed direction of the transfer.
> "ERROR: could not find parent subvolume". It seems like btrfs
receive
> cannot find the parent/clone source because it is searching for its UUID
> only by looking through the /mnt/bak snapshots'' received_uuid
fields.
>
> Is there a way for me to directly change the received_uuid of
> /mnt/bak/.snap to make it identical to the UUID of /.snap? This looks
> like the easiest way if I only need to do it once.
There is no implemented way, but since you debugged this far, you can
put up some code that sends BTRFS_IOC_SET_RECEIVED_SUBVOL, which is
the one setting the received_uuid (and some other small stuff, please
check the kernel code for it).

Thanks,
Alex.

>
> [1] https://patchwork.kernel.org/patch/1606091/
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 2012-12-29 15:00 +0200, Alex Lyakas wrote:
> There is no special repo, but you may want, in addition to the patch
> you mentioned, apply this one as well:
> https://patchwork.kernel.org/patch/1604391/
Very useful! Somehow a few lines got wrapped though.
> > Is there a way for me to directly change the received_uuid of
> > /mnt/bak/.snap to make it identical to the UUID of /.snap? This looks
> > like the easiest way if I only need to do it once.
> There is no implemented way, but since you debugged this far, you can
> put up some code that sends BTRFS_IOC_SET_RECEIVED_SUBVOL, which is
> the one setting the received_uuid (and some other small stuff, please
> check the kernel code for it).
Ah, BTRFS_IOC_SET_RECEIVED_SUBVOL. Thanks for pointing me in the right
direction. After some hacking, it worked without eating my data... These
are the two commands to run before the failing one:

btrfs subvolume snapshot /mnt/bak/.snap /mnt/bak/.snap-mangled
uu / .snap /mnt/bak/.snap-mangled

... where uu is my crude little received_uuid and stransid display and
adjustment tool. I''ve attached its source.

Would be cool if someone modified btrfs send/receive to handle this use
case automatically, but it''s too difficult for me.

Hi,

I admit I haven''t completely understood what you''re trying to
achieve. You can
only receive an incremental stream if the internal (!) file system state on the
receiver''s side is the same as on the sender''s. Thus I
don''t see where setting a
uuid to trick btrfs receive could do you any good.

On Sun, December 30, 2012 at 07:40 (+0100), junlion
wrote:
> On 2012-12-29 15:00 +0200, Alex Lyakas wrote:
>> There is no special repo, but you may want, in addition to the patch
>> you mentioned, apply this one as well:
>> https://patchwork.kernel.org/patch/1604391/
> 
> Very useful! Somehow a few lines got wrapped though.
> 
>>> Is there a way for me to directly change the received_uuid of
>>> /mnt/bak/.snap to make it identical to the UUID of /.snap? This
looks
>>> like the easiest way if I only need to do it once.
>> There is no implemented way, but since you debugged this far, you can
>> put up some code that sends BTRFS_IOC_SET_RECEIVED_SUBVOL, which is
>> the one setting the received_uuid (and some other small stuff, please
>> check the kernel code for it).
> 
> Ah, BTRFS_IOC_SET_RECEIVED_SUBVOL. Thanks for pointing me in the right
> direction. After some hacking, it worked without eating my data... These
> are the two commands to run before the failing one:
> 
> btrfs subvolume snapshot /mnt/bak/.snap /mnt/bak/.snap-mangled
> uu / .snap /mnt/bak/.snap-mangled
> 
> ... where uu is my crude little received_uuid and stransid display and
> adjustment tool. I''ve attached its source.
Still in doubt, I''d like to recommend checking with Arne''s
fssum tool if both
file systems are really the same now. You can fetch and build from his
repository at

	git://git.kernel.org/pub/scm/linux/kernel/git/arne/far-progs.git

As a general rule, please send patches inline for people to quote them when
commenting.
> Would be cool if someone modified btrfs send/receive to handle this use
> case automatically, but it''s too difficult for me.
We could completely drop the check leading to the "could not find parent
subvolume" error message, if someone explained why removing it was
sensible.

Looking forward to your explanation!
-Jan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi,
> I admit I haven''t completely understood what you''re
trying to achieve. You can
> only receive an incremental stream if the internal (!) file system state on
the
> receiver''s side is the same as on the sender''s.
Understood, and the internal states are identical, except for the
snapshot metadata.

This executable example that shows the problem:


    # Create two throwaway filesystems /tmp/img1 and /tmp/img2
    # and mount them at /tmp/mp1 and /tmp/mp2
    for i in 1 2; do
        truncate -s 1G /tmp/img$i
        loop=`losetup -f --show /tmp/img$i`
        mkfs.btrfs "$loop"
        losetup -d "$loop"
        mkdir /tmp/mp$i
        mount /tmp/img$i /tmp/mp$i
    done

    # Create a subvol on the first fs, populate it
    btrfs sub create /tmp/mp1/foo
    echo 1 >/tmp/mp1/foo/1

    # Make an ro snapshot of the subvol, transfer it to the 2nd fs
    btrfs sub snap -r /tmp/mp1/foo /tmp/mp1/foo.1
    btrfs send /tmp/mp1/foo.1 | btrfs receive -v /tmp/mp2

    # Make an rw snapshot of the transferred snapshot on the 2nd fs,
    # add something to this rw snapshot
    btrfs sub snap /tmp/mp2/foo.1 /tmp/mp2/foo
    echo 2 >/tmp/mp2/foo/2

    # Make an ro snapshot of the added-to snapshot
    btrfs sub snap -r /tmp/mp2/foo /tmp/mp2/foo.2

    # Now try to transfer it back in the other direction
    btrfs send -p /tmp/mp2/foo.1 /tmp/mp2/foo.2 |
    btrfs receive -v /tmp/mp1
    # "ERROR: could not find parent subvolume"
    # even though both fs have the same foo.1


As I understand it, if "clone" is a clone/parent source, btrfs send
transmits
    clone.uuid
    clone.ctransid
and btrfs receive searches for a subvol "sub" with
    sub.received_uuid == clone.uuid and sub.stransid == clone.ctransid.

But because the direction of the transfer has changed, it can''t find
anything. It would work if btrfs send was modified to *additionally*
transmit
    clone.received_uuid
    clone.stransid
and btrfs receive to do a *fallback* search for a subvol with
    sub.uuid == clone.received_uuid and sub.ctransid == clone.stransid

However, since I''m really not familiar with btrfs code, it was easier
for me to just make a rw snapshot of mp1/foo.1 and modify its metadata:

    # can''t modify the info about a ro snapshot
    btrfs sub snap /tmp/mp1/foo.1 /tmp/mp1/foo.1rw

    # mp1/foo.1rw.received_uuid := mp2/foo.1.uuid
    # mp1/foo.1rw.stransid      := mp2/foo.1.ctransid
    uu /tmp/mp2 foo.1 /tmp/mp1/foo.1rw

    # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    # !! Now we''ve made it look as if mp1/foo.1 was received from !!
    # !! mp2/foo.1 when actually it was the other way around.     !!
    # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    
    # Let''s retry the original problematic command:
    btrfs send -p /tmp/mp2/foo.1 /tmp/mp2/foo.2 |
    btrfs receive -v /tmp/mp1

    # It works fine! Hurray!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

>     # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>     # !! Now we''ve made it look as if mp1/foo.1 was received from
!!
>     # !! mp2/foo.1 when actually it was the other way around.     !!
>     # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Better just ignore what I wrote there, it''s not completely correct.
(Because we do not modify mp1/foo.1 but snapshot it to mp1/foo.1rw
and modify that one.)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html