Hi,
thank you for your answer.
But what do you mean by "slow rsync"? Do you mean limiting bandwith? I
think this will not be an option with 100Gb of data to synchronise...
However, I like your idea with the LAN: I think I will bring my mirror to
the remote site :-)
Just for the record, I didn't made the standard mistake, having things one
directory level off from where they should be.
I was hoping doing a "scp -rp" would preserve all the necessary
information needed by rsync but apparently not... Maybe I should use tar?
Regards,
Cyrille
<Tony@ServaCorp.com> a ?crit sur 19/10/2005 11:00:07 :
> Standard mistake is to somehow get things one directory level off
> from where they should be.
>
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,
> do a fast rsync from the second disk space to the legitimate
target..>
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one
> level difference between them.
>
> I know I've done it a number of times, and it is no fun over remote
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>
>
> WARNING -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so
> I'm not posting to the list.---
> ?? hosts allow/deny ignore with auth users present ??
> The major sucurity concerns are NOT permanently losing all my data
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,
> and everything needs to assume
> that they will chose the most inopportune moment to do something I
don't
like.> Essentially the stunt is that /home/rsync-foo is rsync'd with other
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>
>
>
> ---Timing depends on what is different. Typically this one is about
> 10 to 15 minutes (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real 192m41.906s
>
>
> real 12m18.512s
> sent 13750 bytes received 7577551 bytes 10279.35 bytes/sec
> total size is 14013850399 speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [root@wf1 /root]# cat rsync-sjs-dwg
> #!/bin/sh
>
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>
> time rsync -avz --progress --partial --timeout=1750 --password-
> file=/etc/rsync.secrets/rsync-sjs-dwg \
> rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>
> ----rsync server (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast
> # etc/rsyncd.conf
> pid file = /etc/rsyncd.pid
> [rsync-sjs-dwg]
> comment = backup of sjs dwg
> path = /home/rsync-sjs-dwg
> use chroot = true
> read only = false
> list = false
> uid = root
> gid = root
> auth users = rsync-sjs-dwg
> secrets file = /etc/rsyncd.secrets
> strict modes = true
> hosts allow = 10.0.0.1 pete-162 wf
> hosts deny = *.*.*.*
> ---rsync "client" (This is a PRIMARY)
> [root@sys001 root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg SJS_Drawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /tmp/rsync/OPENED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> for name in title SJS_Drawings ; do
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /home/dwg/$name rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /tmp/rsync/CLOSED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>
>
>
>
>
> -----Original Message-----
> From: rsync-bounces+tony=servacorp.com@lists.samba.org [mailto:
> rsync-bounces+tony=servacorp.com@lists.samba.org]On Behalf Of Cyrille
Bollu> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync@lists.samba.org
> Subject: Initialising a mirror
>
> Dears,
>
> I would like to mirror data from 2 servers connected together via
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).
>
> I'm sure rsync is one of the best tool to keep these data in sync
> but how should I use it to initialise the mirror?
>
> I'm currently testing the solution with 10Gb of data to keep in
> sync. But on my lines it would take more than 40 hours to initially
> create the mirror! And in production the data to keep in sync will
> reach 100Gb... :-(
>
> So, I went with a laptop on the remote site and "scp -rp" the
data
> on it. I then got back to my site and "scp -rp" it on the mirror
to
> initialise it.
>
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.
>
> Does anybody has an idea about what I'm doing wrong?
>
> Does anybody has a solution to this problem?
>
> Regards,
>
> Cyrille
-------------- next part --------------
HTML attachment scrubbed and removed