Trying to rsync a rather large file from a windows server to a centos server and all but this is working fine. As it's a 20 gig file I am trying to send the diff of with a -c, I suspect over the low bandwidth this presents an issue. I also stage this file locally on another centos server and could calc the diff and create a patch and send that, comparing checksums etc... A quick look at bsdiff and bspatch and the mem requirements on my 20 gig file make that solution rather not acceptable. Anyone know a better solution to accomplish this? Thanks! jlc
On 1/20/2010 12:01 PM, Joseph L. Casale wrote:> Trying to rsync a rather large file from a windows server to a centos server > and all but this is working fine. > > As it's a 20 gig file I am trying to send the diff of with a -c, I suspect over > the low bandwidth this presents an issue. I also stage this file locally on another > centos server and could calc the diff and create a patch and send that, comparing > checksums etc... > > A quick look at bsdiff and bspatch and the mem requirements on my 20 gig file make > that solution rather not acceptable. > > Anyone know a better solution to accomplish this?Is your windows rsync version fairly current? I think older cygwin versions had a file size limit. -- Les Mikesell lesmikesell at gmail.com
On Wed, Jan 20, 2010 at 1:01 PM, Joseph L. Casale <jcasale at activenetwerx.com> wrote:> Trying to rsync a rather large file from a windows server to a centos server > and all but this is working fine. > > As it's a 20 gig file I am trying to send the diff of with a -c, I suspect over > the low bandwidth this presents an issue. I also stage this file locally on another > centos server and could calc the diff and create a patch and send that, comparing > checksums etc... > > A quick look at bsdiff and bspatch and the mem requirements on my 20 gig file make > that solution rather not acceptable. > > Anyone know a better solution to accomplish this? > > Thanks! > jlcI don't understand why the diff shenanigans. Rsync has that built-in, so you shouldn't need to be doing that as a separate step. If it is a file size limit, you could try to split(1) the file, then rsync the chunks. You might also try cygwin 1.7, which has improved the support for modern Windows OS dramatically.
On Wed, Jan 20, 2010 at 1:01 PM, Joseph L. Casale <jcasale at activenetwerx.com> wrote:> Trying to rsync a rather large file from a windows server to a centos server > and all but this is working fine. > > As it's a 20 gig file I am trying to send the diff of with a -c, I suspect over > the low bandwidth this presents an issue. I also stage this file locally on another > centos server and could calc the diff and create a patch and send that, comparing > checksums etc... > > A quick look at bsdiff and bspatch and the mem requirements on my 20 gig file make > that solution rather not acceptable. > > Anyone know a better solution to accomplish this? >I had to do a similar thing across a satellite link. I ended up splitting the files into 500M segments then rsyncing those. When completed, I ssh'ed into the remote and rejoined then checksummed.
On Wed, Jan 20, 2010 at 1:01 PM, Joseph L. Casale <jcasale at activenetwerx.com> wrote:> <snip> > As it's a 20 gig file I am trying to send the diff of with a -c, I suspect > over > <snip> >This might be too basic a question, what type of file system are you using on the CentOS system? For instance if it is the ext2/ext3 file system, there are file size limits depending on such things as the block size: http://en.wikipedia.org/wiki/Ext2#File_system_limits. Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20100120/90989b4d/attachment-0001.html>