rsync - May 2004 - Professional grade rsync?

Hi, folks.

We've gone where no man has gone before.  On HP-UX, rsync bombs at about
1.75 million directories, files and links (combined sum) in a single
transfer.

Is there a professional-grade alternative on HP-UX for folks willing to pay
for it?  It wouldn't even need to be network-aware, just from single-system
areas to the same box, but with the nifty delete and update features that
rsync has.  My searches turn up unison and some other tools (BSD mirror,
etc.), but rsync has beaten any other open-source solution hands down on the
scalability side of things.  Now, we need more ...

Thanks,

A. Daniel King, System Analyst
HP-UX, Linux, Solaris

Silly suggestion, perhaps, but...

Ever considered simply breaking your sync down into several separate 
sets?  IE instead of rsync /foo/baz user@machine:/foo, where /foo/baz 
contains ten directories each then spiraling down into hundreds of 
thousands more... rsync /foo/baz/1 user@machine:/foo/baz, then rsync 
/foo/baz/2 user@machine:/foo/baz, and so on.  Enterprising sorts can, of 
course, write wrappers in Perl or the shell of your choice to 
automatically sync everything in /foo/baz one directory and/or file at a 
time with ease.

I haven't personally run into your upper end limits on heavy-duty 
equipment, but I've used the technique described above to keep old 32MB 
and 64MB machines in successful service as rsync servers; the principle 
is the same - scale your single sync down to a level that your hardware 
can handle.

(It should also be possible to make rsync more memory-friendly when the 
size of the job exceeds the resources available, by causing it to do the 
exact same thing as the wrapper described above does, but I'm not going 
to kvetch.)

Jim Salter
JRS Systems
> Hi, folks.
> 
> We've gone where no man has gone before.  On HP-UX, rsync bombs at
about
> 1.75 million directories, files and links (combined sum) in a single
> transfer.
> 
> Is there a professional-grade alternative on HP-UX for folks willing to pay
> for it?  It wouldn't even need to be network-aware, just from
single-system
> areas to the same box, but with the nifty delete and update features that
> rsync has.  My searches turn up unison and some other tools (BSD mirror,
> etc.), but rsync has beaten any other open-source solution hands down on
the
> scalability side of things.  Now, we need more ...
> 
> Thanks,
> 
> A. Daniel King, System Analyst
> HP-UX, Linux, Solaris

I do a daily sync to about 20 locations of 2.4 million files. I think rsync is
very much 'professional' grade. I have done this sync of 2.4 million
files both
as a single rsync of the tree as well as splitting the tree up in to multiple
rsyncs. (essentially a 'for dir in ls;do rsync $dir dest:blaa;done)

Watch your memory usage on both sides of the rsync. How much RAM is rsync using?
How much RAM on both the src and dest box? You might also consider 2.6.2 -- this
has some improvements in memory usage and speed.

eric




"King, Daniel" wrote:
> 
> Hi, folks.
> 
> We've gone where no man has gone before.  On HP-UX, rsync bombs at
about
> 1.75 million directories, files and links (combined sum) in a single
> transfer.
> 
> Is there a professional-grade alternative on HP-UX for folks willing to pay
> for it?  It wouldn't even need to be network-aware, just from
single-system
> areas to the same box, but with the nifty delete and update features that
> rsync has.  My searches turn up unison and some other tools (BSD mirror,
> etc.), but rsync has beaten any other open-source solution hands down on
the
> scalability side of things.  Now, we need more ...
> 
> Thanks,
> 
> A. Daniel King, System Analyst
> HP-UX, Linux, Solaris
> --
> To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

On Fri, May 28, 2004 at 02:01:12PM -0400, King, Daniel
wrote:
> We've gone where no man has gone before.  On HP-UX, rsync bombs at
about
> 1.75 million directories, files and links (combined sum) in a single
> transfer.
Are you using 2.6.2?  It is more memory efficient than older versions,
so it should handle more files before overflowing memory.

..wayne..

Not yo my knowledge. Go for it. But if youre selling it you must publish code as
per the GPL. Enjoy.

TMC

On Fri, May 28, 2004 at 02:01:12PM -0400, King, Daniel
wrote:
> Hi, folks.
> 
> We've gone where no man has gone before.  On HP-UX, rsync bombs at
about
> 1.75 million directories, files and links (combined sum) in a single
> transfer.
> 
> Is there a professional-grade alternative on HP-UX for folks willing to pay
> for it?  It wouldn't even need to be network-aware, just from
single-system
> areas to the same box, but with the nifty delete and update features that
> rsync has.  My searches turn up unison and some other tools (BSD mirror,
> etc.), but rsync has beaten any other open-source solution hands down on
the
> scalability side of things.  Now, we need more ...
> 
> Thanks,
> 
> A. Daniel King, System Analyst
> HP-UX, Linux, Solaris
> -- 
> To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Tomasz M. Ciolek	
*******************************************************************************
 tmc at dreamcraft dot com dot au 
*******************************************************************************
   GPG Key ID:		0x41C4C2F0
   GPG Key Fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4 C2F0
   Key available on www.pgp.net	
*******************************************************************************