hi We have a box here connected to an antenna receiving rather large amounts of metheorological data from a satellite. the data is received and ransferred to another box and removed from the receiving server. I first thought of using rsync for this, but it seems --remove-source- files has no way of checking if the file is open or not, so if the receiving process is still writing to a file, rsync will transfer the file and unlink() it without bothering if the file has been modified any further, effectively resulting in a corrupt file. I solved this with a rather clumsy script doing md5sums for fils at both ends and checking if I have the exact same file at both sides some time after transfer. this works, but it's not really optimal. Does anyone know if rsync could be better on this somehow? roy -- Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 roy@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote:> > We have a box here connected to an antenna receiving rather large > amounts of metheorological data from a satellite. the data is received > and ransferred to another box and removed from the receiving server. I > first thought of using rsync for this, but it seems --remove-source- > files has no way of checking if the file is open or not, so if the > receiving process is still writing to a file, rsync will transfer the > file and unlink() it without bothering if the file has been modified any > further, effectively resulting in a corrupt file. I solved this with aI would solve this not using rsync, but by writing the active file in a separate directory, or by first writing to a special filename and renaming when done (like firefox adds .part to the filename while it is still downloading it). Paul Slootman
On 3. juni. 2009, at 12.34, Paul Slootman wrote:> On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: >> >> We have a box here connected to an antenna receiving rather large >> amounts of metheorological data from a satellite. the data is >> received >> and ransferred to another box and removed from the receiving >> server. I >> first thought of using rsync for this, but it seems --remove-source- >> files has no way of checking if the file is open or not, so if the >> receiving process is still writing to a file, rsync will transfer the >> file and unlink() it without bothering if the file has been >> modified any >> further, effectively resulting in a corrupt file. I solved this >> with a > > I would solve this not using rsync, but by writing the active file > in a > separate directory, or by first writing to a special filename and > renaming when done (like firefox adds .part to the filename while it > is > still downloading it).Why not just something like --ignore-open-files? roy -- Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 roy@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote:>> I would solve this not using rsync, but by writing the active file in a >> separate directory, or by first writing to a special filename and >> renaming when done (like firefox adds .part to the filename while it >> is >> still downloading it). > > Why not just something like --ignore-open-files?Because checking whether a process has a file open can be very costly in system resources. Paul