I just tried to copy several million files with rsync.
Unfortunately, it seems to freeze:
# rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/
/mnt/iscsi_backup/backuppc/
building file list ...
7891370 files to consider
And it waits here. It was already copied once sussessfully, and this time, I
just wanted to do an update.
Destination filesystem is iSCSI, on another device in LAN.
I can list the files there, create new files etc., so it's not a device
problem.
As there were many files, it used up lost of RAM:
# free
total used free shared buffers cached
Mem: 524460 520928 3532 0 344 6604
-/+ buffers/cache: 513980 10480
Swap: 1534072 1100884 433188
But still, there is a lot of swap left.
I can see that rsync is in "D" state, and doesn't use much CPU
time:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23311 root 5 -17 685m 440m 488 D 0.0 86.0 2:05.94 rsync
23310 root 0 -17 578m 608 608 S 0.0 0.1 8:41.55 rsync
So I made a strace, and rsync wasn't doing much:
[root@syn1 kernel]# strace -p 23311
Process 23311 attached - interrupt to quit
(ctrl + C after one minute)
Process 23311 detached
[root@syn1 kernel]# strace -p 23310
Process 23310 attached - interrupt to quit
select(6, [5], [], NULL, {4, 800000}) = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}) = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}) = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}
(ctrl + C after two minutes or so)
It's in that state for several hours now.
Will it eventually wake up and finish, or perhaps something doesn't work
properly?
Those "Timeout" messages in strace output worry me and make me think
something didn't go as planned.
I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it
looks like that to me).
--
Tomasz Chmielewski
http://wpkg.org
On Mon 21 Aug 2006, Tomasz Chmielewski wrote:> I just tried to copy several million files with rsync. > > Unfortunately, it seems to freeze: > > # rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/ /mnt/iscsi_backup/backuppc/ > building file list ... > 7891370 files to consider > > > And it waits here. It was already copied once sussessfully, and this time, I just wanted to do an update. > Destination filesystem is iSCSI, on another device in LAN. > I can list the files there, create new files etc., so it's not a device problem. > > > As there were many files, it used up lost of RAM: > > # free > total used free shared buffers cached > Mem: 524460 520928 3532 0 344 6604 > -/+ buffers/cache: 513980 10480 > Swap: 1534072 1100884 433188 > > But still, there is a lot of swap left. > > > I can see that rsync is in "D" state, and doesn't use much CPU time: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 23311 root 5 -17 685m 440m 488 D 0.0 86.0 2:05.94 rsync > 23310 root 0 -17 578m 608 608 S 0.0 0.1 8:41.55 rsyncYou have 512MB of RAM, and it's using 685MB of virtual memory.> So I made a strace, and rsync wasn't doing much: > > [root@syn1 kernel]# strace -p 23311 > Process 23311 attached - interrupt to quit > > (ctrl + C after one minute)It was busy swapping in, to handle the SIGSTOP from the strace....> Process 23311 detached > > > [root@syn1 kernel]# strace -p 23310 > Process 23310 attached - interrupt to quit > select(6, [5], [], NULL, {4, 800000}) = 0 (Timeout) > select(6, [5], [], NULL, {60, 0}) = 0 (Timeout) > select(6, [5], [], NULL, {60, 0}) = 0 (Timeout) > select(6, [5], [], NULL, {60, 0} > > (ctrl + C after two minutes or so) > > > It's in that state for several hours now. > Will it eventually wake up and finish, or perhaps something doesn't work properly?The other side is probably busy building its list of files for the comparison. Can you strace that process?> Those "Timeout" messages in strace output worry me and make me think something didn't go as planned. > > I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it looks like that to me).Did you upgrade to 2.6.8 on both sides? Newer versions are better at using less memory, although I don't know how much difference is between 2.6.6 and 2.6.8 in that respect. Paul Slootman
Hello I have a similar problem. I want to rsync a huge number of files and I'm running out of memory. I'm using Dirvish to backup my servers. For those who haven't heard about it it's sort of "front end" for rsync making a copy of the files for the first backup, and then copying modified files and hardlinking unmodified files every day. I have 600 000 files, and I'm keeping 120 backups on disk. Files are not changing too much, I have something like 120 GB of data and 180 GB of backup. That is roughly 600 000 files with 100 hardlinks on each, so 60 000 000 files.>From what I've been reading in the FAQ and the ML rsync will use 100bytes per files (6 GB), and I'll probably get a x3 "bonus" (18 GB) for using the --delete and -H options. How could I get these files rsynced ? By the way what is the memory usage on the receiver side ? For the first rsync there is no files receiver side, so it's no problem. But after that first rsync memory usage will be the same on sender and receiver side, as they'll both have to create their file list. Am I right ? My servers are running Debian Sarge i.e. rsync 2.6.4