rsync - Feb 2024 - tests for clone-dest

WayneD's clone-dest patch seems to be working for me, it's available
here:

https://github.com/WayneD/rsync-patches/blob/master/clone-dest.diff

I've written some very basic tests, which I've attached to this email,
in
case they are of use to anyone else.  They're also available here:

https://github.com/josephmaher/rsync

Notes:

 	Tests need to be run as root.

 	clone-dest-btrs.test needs mkfs.btrfs (from btrfs-progs) and 
btrs-search-metadata (from python3-btrfs).

 	clone-dest-xfs.test needs mkfs.xfs and xfs_bmap (from xfsprogs).


Joseph
-------------- next part --------------
#!/bin/sh

# Copyright (C) 2024 by Joseph Maher <github at josephmaher.org>
# This program is distributable under the terms of the GNU GPL (see COPYING)
#
# Check that the --clone-dest option makes reflinks as requested

. "$suitedir/rsync.fns"

test -f /sbin/mkfs.btrfs || test_skipped "Can't find mkfs.btrfs (only
available on Linus with btrfs support)"
test -f /usr/bin/btrfs-search-metadata || test_skipped "Can't find
btrfs-search-metatadata from python3-btrfs (only available on Linux)"

# make a btrfs filesystem and mount it
truncate -s 115M $scratchdir/btrfs.image
/sbin/mkfs.btrfs $scratchdir/btrfs.image > /dev/null
mkdir -p $scratchdir/mnt/
mount -o loop $scratchdir/btrfs.image $scratchdir/mnt/ || test_skipped
"Can't mount btrfs image file, try running as root"

# set up some test files and rsync them
mkdir $scratchdir/mnt/1 $scratchdir/mnt/2 $scratchdir/mnt/3
# files should be at least 4K in size so they fill an extent block
dd if=/dev/urandom of=$scratchdir/mnt/1/a bs=4K count=1 status=none
# sometimes the extents get cached, sycn helps write them to disk
sync $scratchdir/mnt/1/a
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/1/b
sync $scratchdir/mnt/1/b
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/3/a
sync $scratchdir/mnt/3/a

clonedir=$(realpath $scratchdir/mnt/3)

checkit "$RSYNC -a --clone-dest='$clonedir'
'$scratchdir/mnt/1/' '$scratchdir/mnt/2/'"
"$scratchdir/mnt/1/" "$scratchdir/mnt/2/"
sync $scratchdir/mnt/2/a
sync $scratchdir/mnt/2/b

# check the extents are the same

get_extents() {
    result=$(btrfs-search-metadata file $1 | grep disk_bytenr | sed
's/.*disk_bytenr\ //' | sed 's/\ disk_num_bytes.*//')
    if [ ! -n "$result" ]; then
        echo "couldn't find extents for " $1
    fi
    echo "$result"
}

test "$(get_extents $scratchdir/mnt/2/a)" = "$(get_extents
$scratchdir/mnt/3/a)" || test_fail "clone-dest files have different
extents"

# clean up
umount $scratchdir/mnt/

# The script would have aborted on error, so getting here means we've won.
exit 0
-------------- next part --------------
#!/bin/sh

# Copyright (C) 2024 by Joseph Maher <github at josephmaher.org>
# This program is distributable under the terms of the GNU GPL (see COPYING)
#
# Check that the --clone-dest option makes reflinks as requested

. "$suitedir/rsync.fns"

test -f /sbin/mkfs.xfs || test_skipped "Can't find mkfs.xfs (only
available on Linus with xfs support)"

# make a btrfs filesystem and mount it
truncate -s 300M $scratchdir/xfs.image
/sbin/mkfs.xfs $scratchdir/xfs.image > /dev/null
mkdir -p $scratchdir/mnt/
mount -o loop $scratchdir/xfs.image $scratchdir/mnt/ || test_skipped
"Can't mount xfs image file, try running as root"

# set up some test files and rsync them
mkdir $scratchdir/mnt/1 $scratchdir/mnt/2 $scratchdir/mnt/3
# files should be at least 4K in size so they fill an extent block
dd if=/dev/urandom of=$scratchdir/mnt/1/a bs=4K count=1 status=none
# sometimes the extents get cached, sync helps write them to disk
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/1/b
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/3/a

clonedir=$(realpath $scratchdir/mnt/3)

checkit "$RSYNC -a --clone-dest='$clonedir'
'$scratchdir/mnt/1/' '$scratchdir/mnt/2/'"
"$scratchdir/mnt/1/" "$scratchdir/mnt/2/"

# check the extents are the same

get_extents() {
    echo "$(/sbin/xfs_bmap $1 | tail -n 1)"
}

test "$(get_extents $scratchdir/mnt/2/a)" = "$(get_extents
$scratchdir/mnt/3/a)" || test_fail "clone-dest files have different
extents"

# clean up
umount $scratchdir/mnt/

# The script would have aborted on error, so getting here means we've won.
exit 0

Here's a test for clone-test on bcachefs, also available here:

https://github.com/josephmaher/rsync

Notes:

         Test needs to be run as root.

         clone-dest-bcachefs needs mkfs.bcachefs


Joseph


On Wed, 14 Feb 2024, Joseph Maher via rsync wrote:
>
> WayneD's clone-dest patch seems to be working for me, it's
available here:
>
> https://github.com/WayneD/rsync-patches/blob/master/clone-dest.diff
>
> I've written some very basic tests, which I've attached to this
email, in
> case they are of use to anyone else.  They're also available here:
>
> https://github.com/josephmaher/rsync
>
> Notes:
>
> 	Tests need to be run as root.
>
> 	clone-dest-btrs.test needs mkfs.btrfs (from btrfs-progs) and 
> btrs-search-metadata (from python3-btrfs).
>
> 	clone-dest-xfs.test needs mkfs.xfs and xfs_bmap (from xfsprogs).
>
>
> Joseph
-------------- next part --------------
#!/bin/sh

# Copyright (C) 2024 by Joseph Maher <github at josephmaher.org>
# This program is distributable under the terms of the GNU GPL (see COPYING)
#
# Check that the --clone-dest option makes reflinks as requested

. "$suitedir/rsync.fns"

test -f /sbin/mkfs.bcachefs || test_skipped "Can't find mkfs.bcachefs
(only available on Linux with bacachefs support)"

# make a btrfs filesystem and mount it
truncate -s 25M $scratchdir/bcachefs.image
/sbin/mkfs.bcachefs $scratchdir/bcachefs.image > /dev/null
mkdir -p $scratchdir/mnt/
mount -o loop $scratchdir/bcachefs.image $scratchdir/mnt/ || test_skipped
"Can't mount bcachefs image file, try running as root"

# set up some test files and rsync them
mkdir $scratchdir/mnt/1 $scratchdir/mnt/2 $scratchdir/mnt/3
# files should be at least 4K in size so they fill an extent block
dd if=/dev/urandom of=$scratchdir/mnt/1/a bs=4K count=1 status=none
# sometimes the extents get cached, sync helps write them to disk
sync $scratchdir/mnt/1/a
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/1/b
sync $scratchdir/mnt/1/b
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/3/a
sync $scratchdir/mnt/3/a

clonedir=$(realpath $scratchdir/mnt/3)

checkit "$RSYNC -a --clone-dest='$clonedir'
'$scratchdir/mnt/1/' '$scratchdir/mnt/2/'"
"$scratchdir/mnt/1/" "$scratchdir/mnt/2/"
sync $scratchdir/mnt/2/a
sync $scratchdir/mnt/2/b

# check the extents are the same

get_extents() {
    #try and get the extend id from filefrag
    result=$(/usr/sbin/filefrag -v $1 | /usr/bin/sed '4!d' |
/usr/bin/awk '{ print $4 }')
    if [ ! -n "$result" ]; then
        echo "couldn't find extents for " $1
    fi
    echo "$result"
}

test "$(get_extents $scratchdir/mnt/2/a)" = "$(get_extents
$scratchdir/mnt/3/a)" || test_fail "clone-dest files have different
extents"

# clean up
umount $scratchdir/mnt/

# The script would have aborted on error, so getting here means we've won.
exit 0