'Lo. I've run into a frustrating issue when trying to synchronize a directory hierarchy over a reliable (but slow) connection to an unreliable remote. Basically, I have the following: http://mvn-repository.io7m.com/com/io7m/ This is a set of nested directories containing binaries and sources for projects I develop/maintain. Every time a new release is made, I deploy the binaries and sources to an exact copy of the above hierarchy on my local machine, and then rsync that (over SSH) to mvn-repository.io7m.com. $ rsync -avz --delete --progress local/ io7m.com:/home/io7m/mvn-repository.io7m.com/ The problem: The latest project produces one .jar file that's about 80mb. Evidently, the hosting provider I use for io7m.com is using some sort of process tracking system that kills processes that have been running for too long (I think it just measures CPU time). The result of this is that I get about 50% of the way through copying that (comparatively) large file, and then the remote rsync process is suddenly killed because it has been running for too long. This would be fine, except that it seems that rsync is utterly refusing all efforts to continue copying that file from wherever it left off. It always restarts copying of the file from nothing and tries to copy the full 80mb, resulting it being killed halfway through and causing much grinding of teeth. The documentation for --partial states that "Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.". Well, for whatever reason, it doesn't (or it at least fails to continue using it). I've tried --partial-dir, passing it an absolute path to the temporary directory in my home directory. It created a file in there the first time, but after being killed by the remote side and restarting, it ignored that file and instead created a new temporary file (with a random suffix) in the destination directory! Am I doing something wrong? $ rsync -avz --delete --progress --partial-dir=/home/io7m/tmp/rsync io7m.com:/home/io7m/mvn-repository.io7m.com/ I'm at a loss. How can I reliably get this directory hierarchy up onto the server? I don't care if I have to retry the command multiple times until the copy has fully succeeded, but I obviously can't do that if rsync keeps restarting the failed file from scratch every time. M
(until a better answer comes along) The killed rsync process should leave a temporary file .<file-name>.<random> If you rename the temporary to the real file name, rsync should continue from about where it left off. -----Original Message----- From: rsync-bounces at lists.samba.org [mailto:rsync-bounces at lists.samba.org] On Behalf Of net.rsync at io7m.com Sent: Tuesday, November 25, 2014 9:03 AM To: rsync at lists.samba.org Subject: Dealing with an unreliable remote 'Lo. I've run into a frustrating issue when trying to synchronize a directory hierarchy over a reliable (but slow) connection to an unreliable remote. Basically, I have the following: http://mvn-repository.io7m.com/com/io7m/ This is a set of nested directories containing binaries and sources for projects I develop/maintain. Every time a new release is made, I deploy the binaries and sources to an exact copy of the above hierarchy on my local machine, and then rsync that (over SSH) to mvn-repository.io7m.com. $ rsync -avz --delete --progress local/ io7m.com:/home/io7m/mvn-repository.io7m.com/ The problem: The latest project produces one .jar file that's about 80mb. Evidently, the hosting provider I use for io7m.com is using some sort of process tracking system that kills processes that have been running for too long (I think it just measures CPU time). The result of this is that I get about 50% of the way through copying that (comparatively) large file, and then the remote rsync process is suddenly killed because it has been running for too long. This would be fine, except that it seems that rsync is utterly refusing all efforts to continue copying that file from wherever it left off. It always restarts copying of the file from nothing and tries to copy the full 80mb, resulting it being killed halfway through and causing much grinding of teeth. The documentation for --partial states that "Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.". Well, for whatever reason, it doesn't (or it at least fails to continue using it). I've tried --partial-dir, passing it an absolute path to the temporary directory in my home directory. It created a file in there the first time, but after being killed by the remote side and restarting, it ignored that file and instead created a new temporary file (with a random suffix) in the destination directory! Am I doing something wrong? $ rsync -avz --delete --progress --partial-dir=/home/io7m/tmp/rsync io7m.com:/home/io7m/mvn-repository.io7m.com/ I'm at a loss. How can I reliably get this directory hierarchy up onto the server? I don't care if I have to retry the command multiple times until the copy has fully succeeded, but I obviously can't do that if rsync keeps restarting the failed file from scratch every time. M -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
On 2014-11-25T10:27:13 -0600 "Tony Abernethy" <tony at servasoftware.com> wrote:> (until a better answer comes along) > The killed rsync process should leave a temporary file .<file-name>.<random> > If you rename the temporary to the real file name, rsync should continue > from about where it left off.Thanks, that'll at least help to get the stubborn file onto the server. Obviously isn't a long term solution, though! M
you may have a look here: http://superuser.com/questions/192766/resume-transfer-of-a-single-file-by-rsync http://stackoverflow.com/questions/16572066/resuming-rsync-partial-p-partial-on-a-interrupted-transfer if you use inplace or append, for security reason you could even run another rsync "diff" to compare if source and destination is really identical, see: http://www.techrepublic.com/blog/linux-and-open-source/how-to-compare-the-content-of-two-or-more-directories-automatically/ regards roland> Gesendet: Dienstag, 25. November 2014 um 16:02 Uhr > Von: net.rsync at io7m.com > An: rsync at lists.samba.org > Betreff: Dealing with an unreliable remote > > 'Lo. > > I've run into a frustrating issue when trying to synchronize a > directory hierarchy over a reliable (but slow) connection to an > unreliable remote. Basically, I have the following: > > http://mvn-repository.io7m.com/com/io7m/ > > This is a set of nested directories containing binaries and sources for > projects I develop/maintain. Every time a new release is made, I deploy > the binaries and sources to an exact copy of the above hierarchy on my > local machine, and then rsync that (over SSH) to > mvn-repository.io7m.com. > > $ rsync -avz --delete --progress local/ io7m.com:/home/io7m/mvn-repository.io7m.com/ > > The problem: > > The latest project produces one .jar file that's about 80mb. > Evidently, the hosting provider I use for io7m.com is using some sort > of process tracking system that kills processes that have been running > for too long (I think it just measures CPU time). The result of this is > that I get about 50% of the way through copying that > (comparatively) large file, and then the remote rsync process is > suddenly killed because it has been running for too long. > > This would be fine, except that it seems that rsync is utterly refusing > all efforts to continue copying that file from wherever it left off. It > always restarts copying of the file from nothing and tries to copy the > full 80mb, resulting it being killed halfway through and causing much > grinding of teeth. > > The documentation for --partial states that "Using the --partial option > tells rsync to keep the partial file which should make a subsequent > transfer of the rest of the file much faster.". Well, for whatever > reason, it doesn't (or it at least fails to continue using it). > > I've tried --partial-dir, passing it an absolute path to the temporary > directory in my home directory. It created a file in there the first time, > but after being killed by the remote side and restarting, it ignored > that file and instead created a new temporary file (with a random suffix) > in the destination directory! Am I doing something wrong? > > $ rsync -avz --delete --progress --partial-dir=/home/io7m/tmp/rsync io7m.com:/home/io7m/mvn-repository.io7m.com/ > > I'm at a loss. How can I reliably get this directory hierarchy up onto > the server? I don't care if I have to retry the command multiple times > until the copy has fully succeeded, but I obviously can't do that if > rsync keeps restarting the failed file from scratch every time. > > M > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >
On 25.11.2014 15:02, net.rsync at io7m.com wrote:> 'Lo. > > I've run into a frustrating issue when trying to synchronize a > directory hierarchy over a reliable (but slow) connection to an > unreliable remote. Basically, I have the following: > > http://mvn-repository.io7m.com/com/io7m/ > > This is a set of nested directories containing binaries and sources for > projects I develop/maintain. Every time a new release is made, I deploy > the binaries and sources to an exact copy of the above hierarchy on my > local machine, and then rsync that (over SSH) to > mvn-repository.io7m.com. > > $ rsync -avz --delete --progress local/ io7m.com:/home/io7m/mvn-repository.io7m.com/ > > The problem: > > The latest project produces one .jar file that's about 80mb. > Evidently, the hosting provider I use for io7m.com is using some sort > of process tracking system that kills processes that have been running > for too long (I think it just measures CPU time). The result of this is > that I get about 50% of the way through copying that > (comparatively) large file, and then the remote rsync process is > suddenly killed because it has been running for too long. > > This would be fine, except that it seems that rsync is utterly refusing > all efforts to continue copying that file from wherever it left off. It > always restarts copying of the file from nothing and tries to copy the > full 80mb, resulting it being killed halfway through and causing much > grinding of teeth. > > The documentation for --partial states that "Using the --partial option > tells rsync to keep the partial file which should make a subsequent > transfer of the rest of the file much faster.". Well, for whatever > reason, it doesn't (or it at least fails to continue using it). > > I've tried --partial-dir, passing it an absolute path to the temporary > directory in my home directory. It created a file in there the first time, > but after being killed by the remote side and restarting, it ignored > that file and instead created a new temporary file (with a random suffix) > in the destination directory! Am I doing something wrong? > > $ rsync -avz --delete --progress --partial-dir=/home/io7m/tmp/rsync io7m.com:/home/io7m/mvn-repository.io7m.com/ > > I'm at a loss. How can I reliably get this directory hierarchy up onto > the server? I don't care if I have to retry the command multiple times > until the copy has fully succeeded, but I obviously can't do that if > rsync keeps restarting the failed file from scratch every time.--inplace --partial The later prevents rsync from discarting partial files, the former makes rsync "reuse" already transfered parts and overwriting/appending as needed. You only have to be careful if you use hardlinks, --inplace can have "bad interaction" with hardlinks. -- Matthias
On 2014-11-25T20:03:27 +0100 Matthias Schniedermeyer <ms at citd.de> wrote:> > --inplace --partial > > The later prevents rsync from discarting partial files, the former makes > rsync "reuse" already transfered parts and overwriting/appending as > needed. > > You only have to be careful if you use hardlinks, --inplace can have > "bad interaction" with hardlinks.Thank you! --inplace was what was needed. M
Possibly Parallel Threads
- Dealing with an unreliable remote
- Dealing with an unreliable remote
- Correct, resumable, large-file flags
- Correct, resumable, large-file flags
- GSM ISDN gateway to quadbri NT port under bristuffed Asterisk - where to set gain (I have unreliable inband dtmf recognition )?