This is my first time to really use rsync. I did small tests to get the arguments like I wanted and then kicked off the big rsync about 2 and a half hours ago. So far, it has not copied over any files. The command I used is: rsync \ --relative \ --recursive \ --copy-links \ host:/glob/that/matches/about/eighty/./directories \ /local/target/dir The list of directories are all full of symbolic links that point off to NFS mounted file systems. I don't expect it to complete today but I do have to stop it each day at the end of the work day. But it worries me that it has yet to copy over any files. Is it really making progress? Or will it take this long to really start copying files over each day I start it? I expect the total amount copied to be about 400G and about 4 million files. It is possible to break this up into pieces if that would help. Thank you for your help and advice, Perry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: <lists.samba.org/pipermail/rsync/attachments/20140113/224c32bb/attachment.pgp>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 First, don't run rsync without either --times or --archive. Without that rsync won't copy timestamps and it won't be able to tell what is changed when you run it again. Second, if rsync isn't copying anything then there are 2 reasons... 1. You already have most of the files copied and it is going through them looking for a file that needs updating 2. You are using rsync version 2 where all files had to be indexed before it copied anything. On 01/13/2014 03:06 PM, Perry Smith wrote:> This is my first time to really use rsync. I did small tests to > get the arguments like I wanted and then kicked off the big rsync > about 2 and a half hours ago. So far, it has not copied over any > files. > > The command I used is: > > rsync \ --relative \ --recursive \ --copy-links \ > host:/glob/that/matches/about/eighty/./directories \ > /local/target/dir > > The list of directories are all full of symbolic links that point > off to NFS mounted file systems. I don't expect it to complete > today but I do have to stop it each day at the end of the work day. > But it worries me that it has yet to copy over any files. > > Is it really making progress? Or will it take this long to really > start copying files over each day I start it? > > I expect the total amount copied to be about 400G and about 4 > million files. > > It is possible to break this up into pieces if that would help. > > Thank you for your help and advice, Perry > > >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: sanitarium.net PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - enigmail.net iEYEARECAAYFAlLUUUoACgkQVKC1jlbQAQdt+ACg483AanF2KnP/0kKHLsg7Qe6R SwQAoLT5fQRnbU+IxeBzsNY1yDWdAFEU =susu -----END PGP SIGNATURE-----
Perry Smith wrote:> This is my first time to really use rsync. I did small tests to get the arguments like I wanted and then kicked off the big rsync about 2 and a half hours ago. So far, it has not copied over any files. > > ----- > > Is it really making progress? Or will it take this long to really start copying files over each day I start it? > > I expect the total amount copied to be about 400G and about 4 million files----- This appears to be a classic case of using a hammer to drive in a screw. Um... rsync was designed to save network bandwidth by running on the host and doing file-stat intensive stuff ON the local hard disk (by running on the server and on your client). But your usage case does very badly because rsync needs direct access on both end --- THEN it optimizes the stuff transferred to minimize the amount needing to be copied over the network.... But you are not getting ANY benefit because it will do all of those stats over the network via NFS which is notoriously slow in many or most cases (especially with lots of stat calls). Your copy job would already be done if you did it with 'tar' and just copied over everything. On the receiving end tell tar not to overwrite newer stuff. Yes it will waste more network bandwidth, but it would very likely, already be done. As you have described the problem, there is no real reason to use rsync, as it is unable to optimize network bandwidth because all the stats are remote. Even "cp -au src/. dst/." will likely be faster than trying to use rsync.... talk about tool abuse! ;-) For rsync to do a reasonable job, you really need to tell whoever owns that server to put rsync ON that server so it can access the files locally, then it could do what it does best and build up a list of differences so it only needs to transfer the changed stuff. Certainly, even if you have rsync on the remote end -- for the 1st transfer, if you need to transfer most of the files, it would be better just to create a tar on the remote end, compress it, and copy that locally. How is it that you have so much data on a server you don't have any ability to run a local 'job' on? It really sounds like an impediment to you getting your work done.