Hello all, Since the rsync on Panther many things changed in my professional life. This project is abandoned although it should work. But this you already know. What's new? On my new job I have several servers to administrate. Servers that aren't backed up (sic). So, there's why I'm back to rsync. The backup plan I would like: 1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long names for windows ones). They run rsync (I don't need data encryption so no need to ssh unless it's simpler) and pull data they want to the backup server. This side is OK. 2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1 disks and rsync server. Here I would like the backups to be incremental and rotated, with a sort of time stamp. Here is where things get complicated. Solutions I've found: A - Simpler at the end but would need modifications to rsync: Go further on the backup module of rsync, building in the rotate mv/rm commands. Why simpler? Because a rsync client would only do one command to the rsync server saying "there is it", and the server do it's own internal cook with the stuff. No need to additional scripts. I know that rsync isn't an incremental backup tool at the basis but a syncing tool. But as it already do some of the work, why not doing it better? B - As I said before, additional scripts. Do the client open an ssh connection, do the pre_backup.sh then close ssh (as I understood even using ssh for rsync, it uses it own tunnel, and not an already opened one, no?) run rsync, then rerun ssh, do the post_backup.sh. That should work, but why scripting many times (I'm sure I'm not the only one in this case) when it could be done once for all? C - Between A and B solutions, rsync could, in a first time, have an option to run pre and post scripts when running on ssh. In this manner there would not be needed to open and close ssh 3 times. D - I looked at rsnapshot, but it wouldn't help on this case of rotating server side folders. I also found complicated scripts for rotating folders from the server, but the client would need putting a flag on the server saying syncing is done. All of this don't seem to be the good approach for me. To much complicated to respond to some lines of code missing on rsync (OK, missing for me but I know that this isn't the original goal of rsync and it does lovely what it was designed for). Well, I'm not an rsync expert, so please correct my expos? if I'm wrong. I think solution C would be easy to code, solution A would be the ideal for me but requires some more functions. I'm ready to help coding it. What do you think? Am I on the right path? Would this addition being integrated to the main rsync (after testing, of course)? Any other remarks or suggestions? Best regards, Vitorio
On Fri, 2008-07-18 at 12:40 +0200, macuserfr wrote:> What's new? On my new job I have several servers to > administrate. Servers that aren't backed up (sic). So, there's why I'm > back to rsync. > > The backup plan I would like: > > 1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long > names for windows ones). They run rsync (I don't need data encryption > so no need to ssh unless it's simpler) and pull data they want to the > backup server. This side is OK.I think you mean "push".> 2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1 > disks and rsync server. Here I would like the backups to be > incremental and rotated, with a sort of time stamp. Here is where > things get complicated.Just use an rsync daemon wired up to an rsnapshot installation in sync_first mode, where the daemon module points to the .sync dir in the snapshot root and the "post-xfer exec" command calls "rsnapshot LOWEST-INTERVAL". You can use a listening daemon or a single-use daemon over ssh as you prefer. This is the approach I've been recommending for pushing hard-linked backups for a while; it achieves everything you want without modifying any of the tools. I helped Eric Johansson with a similar setup a while ago: http://lists.samba.org/archive/rsync/2007-December/019470.html If you have multiple clients pushing backups, it's simplest to have a separate module and snapshot root for each client. To keep things manageable, you can write a script to automatically generate the individual rsnapshot.conf files by substituting the necessary values into a template. Matt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.samba.org/archive/rsync/attachments/20080718/c5445245/attachment.bin
Hmmm, I also tried this to use variable from the script calling
instead of from the script body, same result (script doesn't
run : / ) :
freenas:/mnt/data/serveuranm# cat ../timestamp.sh
#!/bin/tcsh
#Turn on debug info
set DEBUG = 0
#Check if we have the good arg number
if( $# < 1 || $# > 1 ) then
echo "Usage: $0 <name_of_backuped_host>"
exit
endif
#Name of the backup directory
set BACKUP = backup
#Name of the backup mount (partition)
set MOUNTNAME = data
#Check if the backup directory exists
if( ! ( -e $BACKUP ) ) then
echo "Directory backup is missing"
exit
endif
#Set up useful variables
set DATE = `date +"%Y-%m-%d-%H%M%S"`
set AVAIL = `df | grep $MOUNTNAME | awk -F' ' '{print $4}'`
set SIZE = `du -s $BACKUP/ | awk -F' ' '{print $1}'`
if ( $DEBUG ) then
echo "DATE = $DATE \
AVAIL = $AVAIL \
SIZE = $SIZE"
endif
if( ! ( -e lastest ) ) then
mv -f $BACKUP lastest
else
mv -f lastest $1$DATE
mv -f $BACKUP lastest
endif
while ( $AVAIL < $SIZE )
rm -Rf `ls -1 | grep $1 | head -n 1`
end
unset *
freenas:/mnt/data/serveuranm# cat /var/etc/rsyncd.conf
syslog facility = local4
list = no
port = 873
pid file = /var/run/rsyncd.pid
uid = rsync
[serveuranm]
comment = Sauvegarde du serveur ANM
path = /mnt/data/serveuranm/
list = true
max connections = 0
read only = false
uid = serveuranm
gid = rsync
post-xfer exec = /mnt/data/timestamp.sh $RSYNC_HOST_NAME
Thanks for any clue of what's wrong.
Best regards,
Vitorio
Le 28 juil. 08 ? 09:39, macuserfr a ?crit :
> Hi folks, that's me again...
>
> Well, following Matt's suggestion, I'm trying to setup my server to
> call a post transfer script. Guess what? It's not working and I
> don't know why. I've looked around other posts in this mail list
and
> googling around. Didn't found what's wrong with my setup. So rsync
> runs nicely but don't call the post-transfer script. The post
> transfer script runs fine when I run it manually with root, rsync
> user or the module defined user (serveuranm). No errors in client
> side nor in rsyncd log. There is maybe a path/user mistake somewhere.
>
> Client side:
> cwrsync with rsync 3.0.2 and modded cygwin1.dll in order to have
> long name support.
>
> cwrsync.cmd batch rsync file content:
>
> @ECHO OFF
> REM *****************************************************************
> REM
> REM CWRSYNC.CMD - Batch file template to start your rsync command (s).
> REM
> REM By Tevfik K. (http://itefix.no)
> REM *****************************************************************
>
> REM Make environment variable changes local to this batch file
> SETLOCAL
>
> REM ** CUSTOMIZE ** Specify where to find rsync and related files (C:
> \CWRSYNC)
> SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC
>
> REM Set CYGWIN variable to 'nontsec'. That makes sure that
permissions
> REM on your windows machine are not updated as a side effect of cygwin
> REM operations.
> SET CYGWIN=nontsec nodosfilewarning codepage:utf8
>
> SET LC_CTYPE="C-UTF-8"
>
> REM Set HOME variable to your windows home directory. That makes sure
> REM that ssh command creates known_hosts in a directory you have
> access.
> SET HOME=%HOMEDRIVE%%HOMEPATH%
>
> REM Make cwRsync home as a part of system PATH to find required DLLs
> SET CWOLDPATH=%PATH%
> SET PATH=%CWRSYNCHOME%\BIN;%PATH%
>
> REM Windows paths may contain a colon (:) as a part of drive
> designation and
> REM backslashes (example c:\, g:\). However, in rsync syntax, a
> colon in a
> REM path means searching for a remote host. Solution: use absolute
> path 'a la unix',
> REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in
> front of the
> REM drive letter:
> REM
> REM Example : C:\WORK\* --> /cygdrive/c/work/*
> REM
> REM Example 1 - rsync recursively to a unix server with an openssh
> server :
> REM
> REM rsync -r /cygdrive/c/work/ remotehost:/home/user/work/
> REM
> REM Example 2 - Local rsync recursively
> REM
> REM rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/
> REM
> REM Example 3 - rsync to an rsync server recursively :
> REM (Double colons?? YES!!)
> REM
> REM rsync -r /cygdrive/c/doc/ remotehost::module/doc
> REM
> REM Rsync is a very powerful tool. Please look at documentation for
> other options.
> REM
>
> REM ** CUSTOMIZE ** Enter your rsync command(s) here
> rsync -ab --link-dest="/lastest" --size-only --chmod=ugo=rwX /
> cygdrive/c/DossiersPartages 172.20.30.194::serveuranm/backup
>
> Server side:
>
> freenas:~# cat /var/etc/rsyncd.conf
> syslog facility = local4
> list = no
> port = 873
> pid file = /var/run/rsyncd.pid
> uid = rsync
>
> [serveuranm]
> comment = Sauvegarde du serveur ANM
> path = /mnt/data/serveuranm/
> list = true
> max connections = 0
> read only = false
> uid = serveuranm
> gid = rsync
> post-xfer exec = /mnt/data/timestamp.sh
>
> freenas:~# cat /var/log/rsyncd.log
> Jul 24 09:46:13 freenas rsyncd[18257]: connect from
> serveuranm.domaineanm.fr (172.20.30.197)
> Jul 24 09:46:13 freenas rsyncd[18258]: rsync to serveuranm/backup
> from serveuranm.domaineanm.fr (172.20.30.197)
> Jul 24 09:46:13 freenas rsyncd[18258]: receiving file list
> Jul 24 10:57:40 freenas rsyncd[18258]: sent 18697 bytes received
> 1935199 bytes total size 22859727296
> Jul 24 13:07:37 freenas rsyncd[23442]: connect from
> serveuranm.domaineanm.fr (172.20.30.197)
> Jul 24 13:07:37 freenas rsyncd[23443]: rsync to serveuranm/backup
> from serveuranm.domaineanm.fr (172.20.30.197)
> Jul 24 13:07:37 freenas rsyncd[23443]: receiving file list
> Jul 24 14:12:42 freenas rsyncd[23443]: sent 42471 bytes received
> 2353855 bytes total size 22859731464
>
> freenas:~# ls -l /mnt/data/
> total 24
> drwxrwxrwx 2 root wheel 512 Jul 2 17:36 .AppleDB
> drwxrwxrwx 2 ftp wheel 512 Jul 2 17:36 .AppleDesktop
> drwxrwxrwx 2 ftp wheel 512 Jul 2 17:36 .AppleDouble
> -rw-rw-rw- 1 ftp wheel 6148 Jul 2 17:36 .DS_Store
> drwxrwxr-x 2 root operator 512 Jan 8 2001 .snap
> drwxrwxrwx 3 ftp wheel 512 Jul 2 17:36 Network Trash
> Folder
> drwxrwxrwx 3 ftp wheel 512 Jul 2 17:36 Temporary Items
> drwxrwxrwx 6 serveuranm rsync 512 Jul 24 13:07 serveuranm
> -rwxr-xr-x 1 root wheel 734 Jul 22 18:02 timestamp.sh
>
> freenas:~# cat /mnt/data/timestamp.sh
> #!/bin/tcsh
> #Turn on debug info
> set DEBUG = 0
>
> #Name of the backup directory
> set BACKUP = backup
>
> #Name of the backup mount (partition)
> set MOUNTNAME = data
>
> #Check if the backup directory exists
> if( ! ( -e $BACKUP ) ) then
> echo "Directory backup is missing"
> exit
> endif
>
> #Set up useful variables
> set DATE = `date +"%Y-%m-%d-%H%M%S"`
> set AVAIL = `df | grep $MOUNTNAME | awk -F' ' '{print $4}'`
> set SIZE = `du -s $BACKUP/ | awk -F' ' '{print $1}'`
>
> if ( $DEBUG ) then
> echo "DATE = $DATE \
> AVAIL = $AVAIL \
> SIZE = $SIZE"
> endif
>
> if( ! ( -e lastest ) ) then
> mv -f $BACKUP lastest
> else
> mv -f lastest $1$DATE
> mv -f $BACKUP lastest
> endif
> while ( $AVAIL < $SIZE )
> rm -Rf `ls -1 | grep $1 | head -n 1`
> end
> unset *
>
> freenas:/mnt/data/serveuranm# cat /etc/passwd
> root:*:0:0:Charlie &:/root:/bin/tcsh
> toor:*:0:0:Bourne-again Superuser:/root:
> daemon:*:1:1:Owner of many system processes:/root:/usr/sbin/nologin
> operator:*:2:5:System &:/:/usr/sbin/nologin
> bin:*:3:7:Binaries Commands and Source:/:/usr/sbin/nologin
> tty:*:4:65533:Tty Sandbox:/:/usr/sbin/nologin
> kmem:*:5:65533:KMem Sandbox:/:/usr/sbin/nologin
> www:*:80:80:World Wide Web Owner:/nonexistent:/usr/sbin/nologin
> nobody:*:65534:65534:Unprivileged user:/nonexistent:/usr/sbin/nologin
> ftp:*:21:50:FTP user:/mnt:/sbin/nologin
> man:*:9:9:Mister Man Pages:/usr/share/man:/usr/sbin/nologin
> sshd:*:22:22:Secure Shell Daemon:/var/empty:/usr/sbin/nologin
> _dhcp:*:65:65:dhcp programs:/var/empty:/usr/sbin/nologin
> serveuranm:*:11212:1001:Sauvegarde du serveur ANM:/mnt/data/
> serveuranm/:/bin/tcsh
> rsync:*:11211:1001:Sauvegarde via rsync:/mnt/data/:/bin/tcsh
>
> Maybe there is a problem because the script is not on the path of
> the module, but I read that the post-xfer exec is not chrooted so I
> could find any path of the system. The script is readable and
> executable by all users.
>
> PS: I don't have strace installed on this server.
>
> Thanks for any help on this. I'm sure it's a dumb mistake but
I'm
> too close to see it. I've rechecked setup many times without seeing
> anything wrong.
>
> Best regards,
>
> Vitorio