On 12Feb, 2008, at 09:53 , Martin Strand wrote:
> I'm trying to use rsync to get a live backup of 2 million files,
> about 50 gb, max depth of 5 directory levels.
> I'm on a gigabit lan so I'm passing -W, but it's still
incredibly
> slow. What else can I do to speed things up?
> Perhaps there's a good way to filter out files older than X so only
> newer files are checked?.
> Will rsync 3.0.0 make a big difference for large trees?
I say yes, although you did not quantify your "slow".
The following was not on a gigabit, but on a firewire (tcp/ip over
firewire), and I suspect that it would have worked better on a gigabit
(better drivers). It was reported some time ago using rsync-3.0.0pre7:
a new tranfer of about 30GB worth of data in about 2 hours:
Number of files: 407317
Total file size: 34860439736 bytes
Total transferred file size: 34860261591 bytes
Literal data: 34860261591 bytes
Matched data: 0 bytes
File list size: 10391700
File list generation time: 0.003 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 35084947192
Total bytes received: 7332354
sent 35084947192 bytes received 7332354 bytes 4927652.82 bytes/sec
A subsequent re-rsync took about 5 minutes (little data changes).
I would say that you can look at about the same sort of figures, or
even a bit less, with better hardware. This was after all just a
PowerBook G4 backed up to an iMac Intel.
Giuliano