hi! i want to use rsync for backup to central storage. from emc networker i know some nice feature - it`s telling about "changed files during save" - i.e. i know which files have changed their contents while being backed up. for example you can see if shutdown of some oracle database was forgotten and that database was backed up online accidentally. what`s the rsync equivalent to this? how can i see which files changed while rsync was transferring them ? i get some "rsync warning: some files vanished before they could be transferred ....." from time to time.... any hints how to see some more details here to know which files being at an probably inconsistent state on the mirror side ? regards roland
On 9/15/07, roland <devzero@web.de> wrote:> what`s the rsync equivalent to this? > how can i see which files changed while rsync was transferring them ?Handling of concurrent changes to source files is one of rsync's weaknesses. The rsync sender naively reads each source file from beginning to end and sends what it sees; it doesn't detect if source files change while being transferred. In many cases, the concurrent modification will leave the source file with an mtime different from the mtime rsync saw when it statted the file during file-list building (which gets set on the destination file), so a subsequent rsync run will fix the corrupted destination file. See this thread for more information: http://lists.samba.org/archive/rsync/2006-January/014534.html Matt
> Handling of concurrent changes to source files is one of rsync's > weaknesses.too bad, but good to know :)>The rsync sender naively reads each source file from > beginning to end and sends what it sees; it doesn't detect if source > files change while being transferred.that`s what i feared.> In many cases, the concurrent > modification will leave the source file with an mtime different from > the mtime rsync saw when it statted the file during file-list building > (which gets set on the destination file), >so a subsequent rsync run will fix the corrupted destination file.mhh - if the file being very large and stat`ed before transfer and then changing during transfer because that file is accidentally in use (how should the receiver/sender always know?) - how should a subsequent rsync run be able to fix this reliably ? the next run, rsync would detect that it changed and would transfer the differences again - and while doing this, the file could change again....and so on...... so, this is not a reliable workaround, imho. what about thinking about an option for stat`ing the file _after_ transfer again, so rsync could at least tell "warning: file changed during rsync transfer...." which would be better than nothing, imho.>See this thread for more information: > http://lists.samba.org/archive/rsync/2006-January/014534.htmlthanks for the pointer roland ps: sorry, couldn`t cc to hashproduct+rsync@gmail.com - my webmailer won`t let me> -----Urspr?ngliche Nachricht----- > Von: "Matt McCutchen" <hashproduct+rsync@gmail.com> > Gesendet: 16.09.07 17:56:45 > An: roland <devzero@web.de> > CC: rsync@lists.samba.org > Betreff: Re: File changed during save....> > On 9/15/07, roland <devzero@web.de> wrote: > > what`s the rsync equivalent to this? > > how can i see which files changed while rsync was transferring them ? > > Handling of concurrent changes to source files is one of rsync's > weaknesses. The rsync sender naively reads each source file from > beginning to end and sends what it sees; it doesn't detect if source > files change while being transferred. In many cases, the concurrent > modification will leave the source file with an mtime different from > the mtime rsync saw when it statted the file during file-list building > (which gets set on the destination file), so a subsequent rsync run > will fix the corrupted destination file. See this thread for more > information: > > http://lists.samba.org/archive/rsync/2006-January/014534.html > > Matt >_______________________________________________________________________ Jetzt neu! Sch?tzen Sie Ihren PC mit McAfee und WEB.DE. 3 Monate kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220
> Note that back-to-back rsyncs make the window of opportunity much > much smaller for things to change during transit.yes, but it still leaves room for corrupted transfers nobody would probably know about !?> -----Urspr?ngliche Nachricht----- > Von: <tony@servacorp.com> > Gesendet: 16.09.07 18:24:23 > An: "'roland'" <devzero@web.de> > CC: <rsync@lists.samba.org> > Betreff: RE: File changed during save....> > Matt McCutchen wrote: > > > > On 9/15/07, roland <devzero@web.de> wrote: > > > what`s the rsync equivalent to this? > > > how can i see which files changed while rsync was > > transferring them ? > > > > Handling of concurrent changes to source files is one of rsync's > > weaknesses. The rsync sender naively reads each source file from > > beginning to end and sends what it sees; it doesn't detect if source > > files change while being transferred. In many cases, the concurrent > > modification will leave the source file with an mtime different from > > the mtime rsync saw when it statted the file during file-list building > > (which gets set on the destination file), so a subsequent rsync run > > will fix the corrupted destination file. See this thread for more > > information: > > > > http://lists.samba.org/archive/rsync/2006-January/014534.html > > > > Matt > > -- > > Note that back-to-back rsyncs make the window of opportunity much > much smaller for things to change during transit. > >______________________________________________________________________ XXL-Speicher, PC-Virenschutz, Spartarife & mehr: Nur im WEB.DE Club! Jetzt testen! http://produkte.web.de/club/?mc=021130
sounds interesting - are you speaking about a special rsync version or about this helper script: http://marc.info/?l=rsync&m=115822570129821&w=2 ?> -----Urspr?ngliche Nachricht----- > Von: "Stephen Zemlicka" <stevezemlicka@gmail.com> > Gesendet: 16.09.07 19:43:54 > An: "'roland'" <devzero@web.de> > CC: <rsync@lists.samba.org> > Betreff: RE: File changed during save....> > If you're on windows, someone wrote a vss patch for rsync. I haven't used > it extensively though but it has worked for in-use outlook pst files so far. > I plan on testing it with exchange and sql databases in the near future. > > _____________________________ > Stephen Zemlicka > Integrated Computer Technologies > PH. 608-558-5926 > E-Mail stevezemlicka@gmail.com > > -----Original Message----- > From: rsync-bounces+stevezemlicka=gmail.com@lists.samba.org > [mailto:rsync-bounces+stevezemlicka=gmail.com@lists.samba.org] On Behalf Of > Tony Abernethy > Sent: Sunday, September 16, 2007 11:24 AM > To: 'Matt McCutchen'; 'roland' > Cc: rsync@lists.samba.org > Subject: RE: File changed during save.... > > Matt McCutchen wrote: > > > > On 9/15/07, roland <devzero@web.de> wrote: > > > what`s the rsync equivalent to this? > > > how can i see which files changed while rsync was > > transferring them ? > > > > Handling of concurrent changes to source files is one of rsync's > > weaknesses. The rsync sender naively reads each source file from > > beginning to end and sends what it sees; it doesn't detect if source > > files change while being transferred. In many cases, the concurrent > > modification will leave the source file with an mtime different from > > the mtime rsync saw when it statted the file during file-list building > > (which gets set on the destination file), so a subsequent rsync run > > will fix the corrupted destination file. See this thread for more > > information: > > > > http://lists.samba.org/archive/rsync/2006-January/014534.html > > > > Matt > > -- > > Note that back-to-back rsyncs make the window of opportunity much > much smaller for things to change during transit. > > -- > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > >_____________________________________________________________________ Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! http://smartsurfer.web.de/?mc=100071&distributionid=000000000066
devzero@web.de wrote:> > > Note that back-to-back rsyncs make the window of opportunity much > > much smaller for things to change during transit. > > yes, but it still leaves room for corrupted transfers nobody > would probably know about !? >With UNIX file semantics, Loop until nothing is transferred -- might be infinite. With WINDOWS file semantics, all bets are off. Seems that mtime and filesize are stored long before the new contents are actually written. What you can actually get away with depends on the exact nature of the contents of a file as it gets changed.