I'm interested in large file system replication capability of rsync. Could some of you people who use it share how large their mirroring is? What would you say is the largest sized site being mirrored using rsync? Thanks! JP
I am rsync'ing 150gig's worth of data over a 9 meg pipe. 160,000 files. Everything works fine (latest rsync version), except large file support is quirky and doesnt work. Any file over 2 gigs just hangs, so I just exclude them and send them over using a different pice of software. Aharon On Wed, 26 Jun 2002, John Palmer wrote:> I'm interested in large file system replication capability of rsync. Could > some of you people who use it share how large their mirroring is? What would > you say is the largest sized site being mirrored using rsync? > > Thanks! > > JP > > >-- Sr Unix Administrator Tampa, Florida Websites: http://www.tamparacing.com http://www.tampaforums.com http://www.ls6.com http://www.lastgen.com http://www.xmbforum.com
I am currently syncing 1.3 terabytes of files using rsync. This is spread across about 12 filesystems all on the same server. Unfortunately we are planning to move away from rsync because it is taking too long to run and it takes up too much memory (some of the rsync processes take up 1.5 GB of RAM -- if we get two of those running at once, the server dies). We have been very happy with rsync but it has recently reached a critical mass where it can no longer handle the number of files we are trying to sync in a timely manner. But if you are looking for the largest rsync site, we might be a contender. :) FYI, we also use (and plan to continue to use) rsync for several smaller mirroring operations.> I'm interested in large file system replication capability of > rsync. Could > some of you people who use it share how large their mirroring > is? What would > you say is the largest sized site being mirrored using rsync? > > Thanks! > > JP
What are you moving to? -----Original Message----- From: Granzow, Doug (NCI) [mailto:granzowd@mail.nih.gov] Sent: Thursday, June 27, 2002 8:31 AM To: 'rsync@lists.samba.org' Subject: RE: Largest file system being synced I am currently syncing 1.3 terabytes of files using rsync. This is spread across about 12 filesystems all on the same server. Unfortunately we are planning to move away from rsync because it is taking too long to run and it takes up too much memory (some of the rsync processes take up 1.5 GB of RAM -- if we get two of those running at once, the server dies). We have been very happy with rsync but it has recently reached a critical mass where it can no longer handle the number of files we are trying to sync in a timely manner. But if you are looking for the largest rsync site, we might be a contender. :) FYI, we also use (and plan to continue to use) rsync for several smaller mirroring operations.> I'm interested in large file system replication capability of > rsync. Could > some of you people who use it share how large their mirroring > is? What would > you say is the largest sized site being mirrored using rsync? > > Thanks! > > JP-- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
We're planning to move to Veritas Volume Replicator. It has the advantage of working at the filesystem level, so whenever a write is done on the primary site, the same write is automatically done on the mirror site. For what I am doing here, it should (hopefully!) work a lot better than rsync because it runs continuously, and it doesn't have the startup overhead of building a file list, which is the biggest problem with rsync when you are dealing with large groups of files.> -----Original Message----- > From: Breedlove, Robert [mailto:RBreedlove@caiso.com] > Sent: Thursday, June 27, 2002 11:46 AM > To: Granzow, Doug (NCI); 'rsync@lists.samba.org' > Subject: RE: Largest file system being synced > > > What are you moving to? > > -----Original Message----- > From: Granzow, Doug (NCI) [mailto:granzowd@mail.nih.gov] > Sent: Thursday, June 27, 2002 8:31 AM > To: 'rsync@lists.samba.org' > Subject: RE: Largest file system being synced > > > > I am currently syncing 1.3 terabytes of files using rsync. > This is spread > across about 12 filesystems all on the same server. > Unfortunately we are > planning to move away from rsync because it is taking too > long to run and it > takes up too much memory (some of the rsync processes take up > 1.5 GB of RAM > -- if we get two of those running at once, the server dies). > We have been > very happy with rsync but it has recently reached a critical > mass where it > can no longer handle the number of files we are trying to > sync in a timely > manner. But if you are looking for the largest rsync site, > we might be a > contender. :) > > FYI, we also use (and plan to continue to use) rsync for > several smaller > mirroring operations. > > > > I'm interested in large file system replication capability of > > rsync. Could > > some of you people who use it share how large their mirroring > > is? What would > > you say is the largest sized site being mirrored using rsync? > > > > Thanks! > > > > JP