rsync - Feb 2008 - hardlinks not working with inode number > 2^31

I've been using 3.0.0pre8 to move a Debian archive from one filesystem
to another. This archive contains a daily snapshot of the Debian ftp
site, with common files hardlinked to save space.

I noticed that it was using far more space than necessary. Upon
investigation it seems that all the source files that have inode numbers
greater than 2^31 aren't being hardlinked together at the destination.

XFS uses a sparse inode number distribution. Here is an ls -i of a
random directory:

3239109826 libtomcat4-java_4.0.3-3woody3_all.deb
  18105717 libtomcat4-java_4.1.31-3_all.deb
1092085178 libtomcat4-java_4.1.31-4_all.deb
  18105718 tomcat4-admin_4.1.31-3_all.deb
1092085181 tomcat4-admin_4.1.31-4_all.deb
3239109829 tomcat4-webapps_4.0.3-3woody3_all.deb
  18105719 tomcat4-webapps_4.1.31-3_all.deb
1092085183 tomcat4-webapps_4.1.31-4_all.deb
3239109831 tomcat4_4.0.3-3woody3.diff.gz
3239109832 tomcat4_4.0.3-3woody3.dsc
3239109833 tomcat4_4.0.3-3woody3_all.deb
3239109834 tomcat4_4.0.3.orig.tar.gz
  18105720 tomcat4_4.1.31-3.diff.gz
  18105721 tomcat4_4.1.31-3.dsc
  18105722 tomcat4_4.1.31-3_all.deb
1092085188 tomcat4_4.1.31-4.diff.gz
1092085190 tomcat4_4.1.31-4.dsc
1092085191 tomcat4_4.1.31-4_all.deb
1091849984 tomcat4_4.1.31.orig.tar.gz

The files with 3239109xxx inodes aren't getting hardlinked.
Note that this is still less than 2^32, so should fit in an unsigned 32 bit int.
Note also that on large (>1TB) XFS filesystems on 64bit systems, a mount
option "inode64" is recommended, as otherwise inodes are only
allocated
from the 1st 1TB space. This will reduce performance (the inodes are
then not always located near the data), and can also lead to not being
able to create new files if the 1st 1TB is full.

If rsync cannot currently cope with larger than 31bit inodes, than this
would need to be dealt with IMHO. It's not like larger than 1TB
filesystems are a rare thing nowadays, with single 3,5inch SATA disks
already available at 1TB sizes for very reasonable prices.


Paul Slootman

On Sat 02 Feb 2008, Paul Slootman wrote:
> 
> The files with 3239109xxx inodes aren't getting hardlinked.
> Note that this is still less than 2^32, so should fit in an unsigned 32 bit
int.
> Note also that on large (>1TB) XFS filesystems on 64bit systems, a mount
> option "inode64" is recommended, as otherwise inodes are only
allocated
> from the 1st 1TB space. This will reduce performance (the inodes are
> then not always located near the data), and can also lead to not being
> able to create new files if the 1st 1TB is full.
The XFS people have also helpfully supplied an 'ino64' mount option to
force inode numbers to be 64bit (an option purely intended for
debugging).  The problem can easily be reproduced this way:

# mount -t xfs -o ino64 /dev/mapper/vg0-xxx /mnt
# cd /mnt
# mkdir TEST
# cd TEST
# mkdir a b
# date > a/x
# ln a/x a/y
# ls -li a
total 8
4297145607 -rw-r--r-- 2 root root 29 2008-02-04 12:44 x
4297145607 -rw-r--r-- 2 root root 29 2008-02-04 12:44 y
# rsync -avH a/ b
sending incremental file list
./
x
y

sent 195 bytes  received 53 bytes  496.00 bytes/sec
total size is 58  speedup is 0.23
# ls -li b
total 8
4301447358 -rw-r--r-- 1 root root 29 2008-02-04 12:44 x
4301447359 -rw-r--r-- 1 root root 29 2008-02-04 12:44 y

# rm b/*

Running rsync with -vvvv:

# rsync -avvvvH a/ b
cmd=<NULL> machine=<NULL> user=<NULL> path=b
cmd[0]=. cmd[1]=b 
note: iconv_open("UTF-8", "UTF-8") succeeded.
(Client) Protocol versions: remote=30, negotiated=30
(Server) Protocol versions: remote=30, negotiated=30
sending incremental file list
[sender] make_file(.,*,2)
[sender] make_file(x,*,2)
[sender] make_file(y,*,2)
[sender] flist start=1, used=3, low=0, high=2
[sender] i=1 a ./ mode=040755 len=14 uid=0 gid=0 flags=5
[sender] i=2 a x mode=0100644 len=29 uid=0 gid=0 flags=0
[sender] i=3 a y mode=0100644 len=29 uid=0 gid=0 flags=0
send_file_list done
file list sent
send_files starting
server_recv(2) starting pid=3282
gid 0() maps to 0
recv_file_name(.)
recv_file_name(x)
recv_file_name(y)
received 3 names
[receiver] flist start=1, used=3, low=0, high=2
[receiver] i=1 0 ./ mode=040755 len=14 uid=0 gid=0 flags=5
[receiver] i=2 1 x mode=0100644 len=29 uid=0 gid=0 flags=20
[receiver] i=3 1 y mode=0100644 len=29 uid=0 gid=0 flags=20
recv_file_list done
get_local_name count=3 b
generator starting pid=3282
recv_files(3) starting
delta-transmission disabled for local transfer or --whole-file
recv_generator(.,0)
send_files(0, a/.)
./
set modtime of . to (1202125466) Mon Feb  4 12:44:26 2008
recv_generator(.,1)
recv_generator(x,2)
send_files(2, a/x)
count=0 n=0 rem=0
send_files mapped a/x of size 29
calling match_sums a/x
x
sending file_sum
false_alarms=0 hash_hits=0 matches=0
sender finished a/x
recv_generator(y,3)
send_files(3, a/y)
count=0 n=0 rem=0
send_files mapped a/y of size 29
calling match_sums a/y
y
sending file_sum
false_alarms=0 hash_hits=0 matches=0
sender finished a/y
generate_files phase=1
recv_files(.)
recv_files(x)
data recv 29 at 0
got file_sum
set modtime of .x.aRFEUX to (1202125463) Mon Feb  4 12:44:23 2008
renaming .x.aRFEUX to x
recv_files(y)
data recv 29 at 0
got file_sum
set modtime of .y.46W9Pc to (1202125463) Mon Feb  4 12:44:23 2008
renaming .y.46W9Pc to y
send_files phase=1
touch_up_dirs: . (0)
set modtime of . to (1202125466) Mon Feb  4 12:44:26 2008
recv_files phase=1
generate_files phase=2
send_files phase=2
send files finished
total: matches=0  hash_hits=0  false_alarms=0 data=58
recv_files phase=2
generate_files phase=3
recv_files finished
generate_files finished
client_run waiting on 3282

sent 195 bytes  received 53 bytes  496.00 bytes/sec
total size is 58  speedup is 0.23
_exit_cleanup(code=0, file=main.c, line=1058): entered
_exit_cleanup(code=0, file=main.c, line=1058): about to call exit(0)


----------------
Retrying with the filesystem not mounted ino64:

# rm -r a b
# mkdir a b
# date > a/x
# ln a/x a/y
# ls -li a
total 8
# ls -li a
total 8
574825 -rw-r--r-- 2 root root 29 2008-02-04 12:52 x
574825 -rw-r--r-- 2 root root 29 2008-02-04 12:52 y
# rsync -avvvvH a/ b
cmd=<NULL> machine=<NULL> user=<NULL> path=b
cmd[0]=. cmd[1]=b 
note: iconv_open("UTF-8", "UTF-8") succeeded.
(Client) Protocol versions: remote=30, negotiated=30
(Server) Protocol versions: remote=30, negotiated=30
sending incremental file list
[sender] make_file(.,*,2)
[sender] make_file(x,*,2)
[sender] make_file(y,*,2)
[sender] flist start=1, used=3, low=0, high=2
[sender] i=1 a ./ mode=040755 len=14 uid=0 gid=0 flags=5
[sender] i=2 a x mode=0100644 len=29 uid=0 gid=0 flags=0
[sender] i=3 a y mode=0100644 len=29 uid=0 gid=0 flags=0
send_file_list done
file list sent
send_files starting
server_recv(2) starting pid=3889
gid 0() maps to 0
recv_file_name(.)
recv_file_name(x)
recv_file_name(y)
received 3 names
[receiver] flist start=1, used=3, low=0, high=2
[receiver] i=1 0 ./ mode=040755 len=14 uid=0 gid=0 flags=5
[receiver] i=2 1 x mode=0100644 len=29 uid=0 gid=0 flags=20
[receiver] i=3 1 y mode=0100644 len=29 uid=0 gid=0 flags=20
recv_file_list done
get_local_name count=3 b
generator starting pid=3889
recv_files(3) starting
delta-transmission disabled for local transfer or --whole-file
recv_generator(.,0)
send_files(0, a/.)
./
set modtime of . to (1202125929) Mon Feb  4 12:52:09 2008
recv_generator(.,1)
recv_generator(x,2)
recv_generator(y,3)
send_files(3, a/y)
count=0 n=0 rem=0
send_files mapped a/y of size 29
calling match_sums a/y
y
sending file_sum
false_alarms=0 hash_hits=0 matches=0
sender finished a/y
generate_files phase=1
recv_files(.)
recv_files(y)
data recv 29 at 0
got file_sum
set modtime of .y.Sbo0BF to (1202125926) Mon Feb  4 12:52:06 2008
renaming .y.Sbo0BF to y
send_files(2, a/x)
x => y
send_files phase=1
touch_up_dirs: . (0)
set modtime of . to (1202125929) Mon Feb  4 12:52:09 2008
recv_files(x)
recv_files phase=1
generate_files phase=2
send_files phase=2
send files finished
total: matches=0  hash_hits=0  false_alarms=0 data=29
recv_files phase=2
generate_files phase=3
recv_files finished
generate_files finished
client_run waiting on 3889

sent 130 bytes  received 43 bytes  346.00 bytes/sec
total size is 58  speedup is 0.34
_exit_cleanup(code=0, file=main.c, line=1058): entered
_exit_cleanup(code=0, file=main.c, line=1058): about to call exit(0)
# ls -li b
total 8
2178311 -rw-r--r-- 2 root root 29 2008-02-04 12:52 x
2178311 -rw-r--r-- 2 root root 29 2008-02-04 12:52 y


Paul Slootman

It appears that the problem is not directly related to inode number
size, but to the way they are hashed to detect the same inode numbers.

If in hashtable_find() I change the following definition:

    int key64 = (tbl->node_size > sizeof (struct ht_int32_node));

to:

    int key64 = 1;

then the "key" is properly saved to a 64-bit value, instead of a
32-bit
value, and then it works.  This looks like an attempt to save space,
using 64 bits only when necessary. However the test seems flawed...


Paul Slootman