I've been trying to rsync a very large list of files specified by --files-from to an rsync daemon and couldn't really figure out why it wasn't working, but I think I finally understand the error. I unfortunately don't have the error to paste here for you all to see, since it takes several hours to get it (and I'm working on a workaround to rsync the files so I don't have time to try and get the error again) - but if it's necessary to see it I can reply with it on demand. Anyways, the file specified in --files-from is about 500mb in size and contains several million files. It takes rsync a few hours for it to parse through all the files in the list. When it comes time for rsync to start transmitting file data to the rsync server, it gets an error that the connection timed out and it couldn't transmit any data. I'm assuming this is just a standard 60 second tcp connection timeout problem, but I see no way to implement keepalives or anything on the connection to prevent it from dropping. Any ideas? Is this really possible? Eli.
I wrote:> I'm assuming this is just a standard 60 second tcp connection > timeout problem, but I see no way to implement keepalives or > anything on the connection to prevent it from dropping. Any > ideas? Is this really possible?I missed the ability to set socket options at the daemon level in rsyncd.conf. Setting "socket options = SO_KEEPALIVE=1" in the daemon config file fixes the timeouts (duh!) :) Eli.
On Tue, Mar 08, 2005 at 12:35:07PM -0500, Eli wrote:> Anyways, the file specified in --files-from is about 500mb in size and > contains several million files. It takes rsync a few hours for it to > parse through all the files in the list. When it comes time for rsync > to start transmitting file data to the rsync server, it gets an error > that the connection timed out and it couldn't transmit any data.Since rsync is keeping the socket busy during the building of the list (it is trasmitting it during that time), I guess the sorting/cleaning of the list is taking enough time that the socket timeouted out on you. I'm glad that the setting of SO_KEEPALIVE did something to help fix your problem, but I'm having a hard time seeing why: the code should set that for all daemon socket connections (excluding only daemon-over-ssh). Are you using the --timeout=N option to rsync (which is an internal-to- rsync timeout value, unrelated to the TCP timeout). ..wayne..