rsync - Nov 2009 - rsyncing large sized folder

Hi

I have to rsync a folder say 0. This folder inside it contains sub folders
[0-999]. Inside this subfolders i have further subfolders [0-999]. so my
directory structure is: 

         /0/0/0/file
        /0/0/1/file
        /0/0/2/file
              |
              |
         /0/0/999/file
         /0/1/0/file
        /0/1/1/file
             |
            |
        /0/0/999/file      and so on till        /0/999/999/file.



Now i am syncing the parent folder /0 on another machine.I have copied all
the data to other machine and then run rsync.But rsync is taking a lot of
time and is not getting completed.Moreover i want to know will rsync take
this much time only the first time or also the next time when it is again
run when theres only a change in few files say 5 or 10.

Please help.    
-- 
View this message in context:
http://old.nabble.com/rsyncing-large-sized-folder-tp26473463p26473463.html
Sent from the Samba - rsync mailing list archive at Nabble.com.

love.wadhwa wrote:
> Hi
> 
> I have to rsync a folder say 0. This folder inside it contains sub folders
> [0-999]. Inside this subfolders i have further subfolders [0-999]. so my
> directory structure is: 
> 
>          /0/0/0/file
>         /0/0/1/file
>         /0/0/2/file
>               |
>               |
>          /0/0/999/file
>          /0/1/0/file
>         /0/1/1/file
>              |
>             |
>         /0/0/999/file      and so on till        /0/999/999/file.
> 
> 
> 
> Now i am syncing the parent folder /0 on another machine.I have copied all
> the data to other machine and then run rsync.But rsync is taking a lot of
> time and is not getting completed.Moreover i want to know will rsync take
> this much time only the first time or also the next time when it is again
> run when theres only a change in few files say 5 or 10.
So ... you have a million directories and at least a million files? It has to
exchange at least some metadata about all of these (size, date/time, names),
and also has to access all those inodes on disks at each end. It does not
surprise me that this will take a while. Even if there is no data to transfer,
there is a goodly amount of *meta*data that rsync must examine. And it matters
what flags you are passing as well ...

Perhaps if you say more about the flags, the nature of the OSes at each end,
and the kind of connection you have between the systems?

Best wishes -- Eliot Moss

Hi

I am running the following command to rsync the same.

rsync -avz   "folder to be rsynced"  username at hostname::module  

Nature of OS is linux and folders are rsynced by running the rsync daemon.

Also would like to know if it would take the same time or very less time for
rsync to sync the same folder the next time when there are few update done
at either end ?



Eliot Moss wrote:
> 
> love.wadhwa wrote:
>> Hi
>> 
>> I have to rsync a folder say 0. This folder inside it contains sub
>> folders
>> [0-999]. Inside this subfolders i have further subfolders [0-999]. so
my
>> directory structure is: 
>> 
>>          /0/0/0/file
>>         /0/0/1/file
>>         /0/0/2/file
>>               |
>>               |
>>          /0/0/999/file
>>          /0/1/0/file
>>         /0/1/1/file
>>              |
>>             |
>>         /0/0/999/file      and so on till        /0/999/999/file.
>> 
>> 
>> 
>> Now i am syncing the parent folder /0 on another machine.I have copied
>> all
>> the data to other machine and then run rsync.But rsync is taking a lot
of
>> time and is not getting completed.Moreover i want to know will rsync
take
>> this much time only the first time or also the next time when it is
again
>> run when theres only a change in few files say 5 or 10.
> 
> So ... you have a million directories and at least a million files? It has
> to
> exchange at least some metadata about all of these (size, date/time,
> names),
> and also has to access all those inodes on disks at each end. It does not
> surprise me that this will take a while. Even if there is no data to
> transfer,
> there is a goodly amount of *meta*data that rsync must examine. And it
> matters
> what flags you are passing as well ...
> 
> Perhaps if you say more about the flags, the nature of the OSes at each
> end,
> and the kind of connection you have between the systems?
> 
> Best wishes -- Eliot Moss
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 
-- 
View this message in context:
http://old.nabble.com/rsyncing-large-sized-folder-tp26473463p26524534.html
Sent from the Samba - rsync mailing list archive at Nabble.com.