Hello there I have a question about rsync, and some options. I have 10 servers here that all need data from one machine. It's a LARGE amount of files all pictures and files and such, now every time I rsync the directory over it takes like hours to create the file list. Now I'm fine with that, but I need to know if it can save the file list it generates and uses it over again for the other machines? Now my script is pretty basic: rsync -urvlopg --delete /local/dir rsync://remote1/MODULE repeat this ten times for remote1 through remote10. I want to speed up this process by making it so it only gets the file list once and copies the updates it needs to each machine. Right now when the script runs it will rsync generate file list, copy the files to the remote and then finish, go to the next line and do the same thing over again, so it takes hours to create the file list and hours to copy. I just want it to create the file list once and copy what it needs to each machine, is there a way to do this? -- Victor Farah - Systems Administrator netmediaservices.net
On Wed, May 14, 2008 12:55 pm, Victor Farah wrote:> Hello there > I have a question about rsync, and some options. > I have 10 servers here that all need data from one machine. It's a > LARGE amount of files all pictures and files and such, now every time I > rsync the directory over it takes like hours to create the file list. > Now I'm fine with that, but I need to know if it can save the file list > it generates and uses it over again for the other machines? > Now my script is pretty basic: > rsync -urvlopg --delete /local/dir rsync://remote1/MODULE > repeat this ten times for remote1 through remote10. I want to speed up > this process by making it so it only gets the file list once and copies > the updates it needs to each machine. Right now when the script runs it > will rsync generate file list, copy the files to the remote and then > finish, go to the next line and do the same thing over again, so it > takes hours to create the file list and hours to copy. I just want it > to create the file list once and copy what it needs to each machine, is > there a way to do this?rsync is stateless, so AFAIK it doesn't save the state. You can look into unison, which is stateful - it saves the state of each replica between runs. I believe there is a way to get it to act like rsync - one directional transfers. --Yan -- o__ ,>/'_ o__ (_)\(_) ,>/'_ o__ Yan Seiner (_)\(_) ,>/'_ o__ Personal Trainer (_)\(_) ,>/'_ o__ Professional Engineer (_)\(_) ,>/'_ Who says engineers have to be pencil necked geeks? (_)\(_) "I worry about my child and the Internet all the time, even though she's too young to have logged on yet. Here's what I worry about. I worry that 10 or 15 years from now, she will come to me and say 'Daddy, where were you when they took freedom of the press away from the Internet?'" --Mike Godwin, Electronic Frontier Foundation
On Wed, May 14, 2008 at 03:55:40PM -0400, Victor Farah wrote:> I need to know if it can save the file list it generates and > uses it over again for the other machines?As long as each of the machines you are updating is in the same state, you can use a batch file to update them all. Create the batch file when you update the first machine (add --write-batch=FILENAME to the command) and then just use --read-batch=FILENAME in place of a source arg on all the rest of the destinations. If the destination machines differ, the batch file might fail to update a file since it might not contain the right data needed to perform the update. The process is somewhat forgiving, though, as rsync can skip items in the batch file that aren't needed, or discard a failed update if the final checksum does not match. ..wayne..