Hi everyone,
I'm trying to rsync a large directory. After pondering on the
list-of-files, it's now more or less doing nothing. It's
running for over 4 hours now, not doing anything except
eating CPUtime on the destination for over half an hour. I don't
know what it was doing before that.
The stdout of the rsync reported:
4675350 files to consider
(which is about right)
On the source machine it's using about 470 Mb of memory.
On the destination machine it's using 800M of memory and growing.
The rsync process on the destination machine is not doing
ANY system calls. Oh! There ARE a lot of hardlinks involved.
The destination machine IS swapping a bit:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 803564 2820 660 6788 1272 120 1272 120 2896 172 3 34 63
1 0 0 803180 2848 660 6788 992 632 992 632 3358 163 2 35 63
1 0 0 802860 2844 604 6668 1160 360 1160 364 3156 180 1 30 69
1 0 0 802888 2816 604 6668 916 420 916 424 2795 179 4 28 68
1 0 0 802452 2760 604 6668 1348 0 1348 0 2805 180 1 34 65
0 1 0 802484 2840 604 6668 932 304 932 308 2601 188 2 31 67
1 0 0 802304 2832 604 6668 1164 364 1164 364 3165 149 1 27 72
1 0 0 802128 2820 604 6668 1036 364 1036 364 2909 133 3 29 68
1 0 0 801728 2804 604 6668 1424 164 1424 168 3292 165 2 27 71
3 0 0 801456 2808 600 6672 1016 264 1016 264 2682 154 7 24 69
1 1 0 801580 2828 600 7048 940 112 1008 128 2377 298 41 35 24
2 0 0 801528 2780 600 7228 1108 0 1288 0 2725 201 2 25 73
2 0 0 801488 2744 600 7636 656 0 1064 0 2237 294 1 25 74
1 1 0 801568 2852 600 7952 716 0 1032 128 2430 285 1 25 74
2 1 0 801616 2780 596 8464 540 0 784 0 1642 279 34 16 50
2 1 0 801576 3844 596 8568 656 0 1028 24 2308 419 12 32 57
1 1 0 801404 2808 596 8896 708 0 1036 0 2180 252 2 21 77
0 2 0 801336 3080 600 8744 844 136 1008 144 2335 207 1 21 78
0 1 1 801372 2820 608 8764 960 568 988 564 2799 176 1 28 71
1 0 0 801204 2772 608 8656 576 180 576 184 2141 150 1 21 78
Is this normal?
Is there a way that I can estimate how much longer this is going to
take. (A day: fine. A week: Not ok.) Will rsync use MORE memory? I
have 200m of swap free right now, but if I should expect it to double
its memory use once more, I will have to add some swap to make it
happy..... Right now I can monitor it, but I'll need some sleep
in the next 16 hours or so... ;-)
Roger.
--
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam - no windows, no gates, apache inside!"
****
On Tue, Aug 26, 2003 at 12:45:10PM +0200, Rogier Wolff wrote:> > Hi everyone, > > I'm trying to rsync a large directory. After pondering on the > list-of-files, it's now more or less doing nothing. It's > running for over 4 hours now, not doing anything except > eating CPUtime on the destination for over half an hour. I don't > know what it was doing before that. > > The stdout of the rsync reported: > 4675350 files to consider > > (which is about right) > > On the source machine it's using about 470 Mb of memory. > On the destination machine it's using 800M of memory and growing. > The rsync process on the destination machine is not doing > ANY system calls. Oh! There ARE a lot of hardlinks involved.It cannot grow without doing system calls. 470MB would be consistent with the file list size you site. Another 300MB+ additional could be due to a large file but i'd have to double check source which i'm not going to do at this moment. Some of that might be due to the hardlinks but again i'd have to check the source to see how much.> The destination machine IS swapping a bit: >[snip] Swap may be thrashing.> Is this normal? > > Is there a way that I can estimate how much longer this is going to > take.No.> (A day: fine. A week: Not ok.) Will rsync use MORE memory? I > have 200m of swap free right now, but if I should expect it to double > its memory use once more, I will have to add some swap to make itprobably not by much. But who can tell.> happy..... Right now I can monitor it, but I'll need some sleep > in the next 16 hours or so... ;-)You don't say what version of rsync, what OS, or what options were specified. One thing is certain. Reducing the file count per invocation will reduce the memory footprint although it won't be able to preserve hardlinks that cross invocation boundaries. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt