rsync - Aug 2007 - Which rsync version?

Hi all.

I once tried to rsync around 100 GB (10 million files), but version
2.6.6 needed too much RAM and was too slow.
Is one of the snapshots stable enough to try this again?

Greetings
Sven
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.samba.org/archive/rsync/attachments/20070802/7bbdf90d/attachment.bin

On Thu 02 Aug 2007, Sven.Hartrumpf@FernUni-Hagen.de
wrote:
> 
> I once tried to rsync around 100 GB (10 million files), but version
> 2.6.6 needed too much RAM and was too slow.
I believe 2.6.7 had a number of memory-saving options; each subsequent
version will of course have other improvements. 2.6.9 is the current
official release.

If those files don't have hardlinks (or you're not interested
in preserving hardlinks), you can greatly improve performance
by trying the 3.0.0 cvs snapshot.


Paul Slootman

On 8/3/07, Paul Slootman <paul@debian.org> wrote:
> On Fri 03 Aug 2007, Sven.Hartrumpf@FernUni-Hagen.de wrote:
>
> > Program received signal SIGPIPE, Broken pipe.
> > 0xffffe410 in __kernel_vsyscall ()
>
> Unfortunately this means you were not tracing the process that failed;
> you were tracing the process that forked another process and got a
> SIGPIPE when trying to write to that process after that process died due
> to the double free error...
Hints for debugging multiple processes:

http://sourceware.org/gdb/current/onlinedocs/gdb_5.html#SEC28

Matt

Fri, 3 Aug 2007 11:48:02 +0200, paul wrote:
> On Fri 03 Aug 2007, Sven.Hartrumpf@FernUni-Hagen.de wrote:
> >
> > I understand, but I have no idea how I can solve this problem.
> >
> > Is rsync starting a second rsync instance?
>
> Yes.
>
> http://www.delorie.com/gnu/docs/gdb/gdb_26.html may help.
Thanks. Now gdb's output it might be more useful?

(gdb) show follow-fork-mode
Debugger response to a program call of fork or vfork is "parent".
(gdb) set follow-fork-mode child
(gdb) show follow-fork-mode
Debugger response to a program call of fork or vfork is "child".
(gdb) run -a --progress -v /cl/ /data/cl
Starting program: /programs/linux/rsync-3.0.0-64/bin/rsync -a --progress -v /cl/
/data/cl
sending incremental file list
Program received signal SIGPIPE, Broken pipe.
[Switching to process 18620]
0x00002b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6
(gdb) bt
#0  0x00002b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6
#1  0x0000000000420acc in writefd_unbuffered (fd=7,
    buf=0x66cb20 "8n&/schulorganisation,property=source.txt",
len=2122)
    at io.c:1387
#2  0x000000000042126a in io_flush (flush_it_all=1) at io.c:1442
#3  0x0000000000409c8c in read_ndx_and_attrs (f_in=0,
    iflag_ptr=0x7fff0a2284c8, type_ptr=0x7fff0a2284cf "",
    buf=0x7fff0a2263d0 "", len_ptr=0x7fff0a2284c4) at rsync.c:184
#4  0x000000000040f18e in recv_files (f_in=0, local_name=0x0) at receiver.c:371
#5  0x0000000000416f39 in do_recv (f_in=0, f_out=1, local_name=0x0)
    at main.c:726
#6  0x00000000004177cd in start_server (f_in=0, f_out=1, argc=1,
    argv=0x7fff0a229638) at main.c:884
#7  0x0000000000418875 in child_main (argc=2122, argv=0xffffffffffffffff)
    at main.c:891
#8  0x000000000042af95 in local_child (argc=2, argv=0x7fff0a229630,
    f_in=0x7fff0a22b61c, f_out=0x7fff0a22b618,
    child_main=0x418860 <child_main>) at pipe.c:147
#9  0x000000000041817b in main (argc=2, argv=0x66c9b0) at main.c:445
(gdb)


Sven
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.samba.org/archive/rsync/attachments/20070803/cb5fe5a0/attachment.bin

Mon, 20 Aug 2007 22:11:12 -0700, wayned wrote:
> On Fri, Aug 03, 2007 at 09:20:22PM +0200, Sven.Hartrumpf wrote:
> > #0  0x00002b4167a93ee5 in free () from /lib64/libc.so.6
> > #1  0x0000000000431c9b in pool_free_old (p=0x66fb90,
> >     addr=<value optimized out>) at lib/pool_alloc.c:266
>
> I found and fixed a bug in pool_free_old() that could leave some freed
> extents on the list of currently allocated extents (in one code path).
> If that were to happen, rsync would try to double-free the memory.
>
> Hopefully the latest version in CVS (and the "nightly" tar file)
will
> avoid this crash now.
Yes, it does!

Many thanks.
Sven
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.samba.org/archive/rsync/attachments/20070821/0191a694/attachment.bin