Displaying 1 result from an estimated 1 matches for "crawldir".
Did you mean:
caldir
2008 Sep 07
2
keep rsync from removing unfinished source files?
...peed and mass. speed has a fast Internet
connection and is running a crawler which downloads a lot of files to
disk. mass has a lot of disk space. I want to move the files from
speed to mass after they're done downloading. Ideally, I'd just run:
$ rsync --remove-source-files speed:/var/crawldir .
but I worry that rsync will unlink a source file that hasn't finished
downloading yet. (I looked at the source code and I didn't see
anything protecting against this.) Any suggestions?
Ideas I had were:
- a pause between downloading the file list and downloading the files
- an exclude...