I am using rsync/rsnapshot on Windows XP (via cygwin) to backup to a mapped share (/u) on a Linux server. I’m hoping to get this to work so that I don’t have to open an XP share to the network for Linux “pull” backups. My current setup doesn’t seem to give me incremental backups. See the “du” output… $ /usr/bin/du -csh /cygdrive/u/docs_bkup/daily.0/ /cygdrive/u/docs_bkup/daily.1/ /cygdrive/u/docs_bkup/daily.2/ 35M /cygdrive/u/docs_bkup/daily.0/ 9.0M /cygdrive/u/docs_bkup/daily.1/ 9.0M /cygdrive/u/docs_bkup/daily.2/ 54M total This looks like some kind of incremental but there are two problems: 1. I’ve not changed any files between rsyncs (so there shouldn’t even be 9M. 2. When I use the –i option, the files being copied are copied because of permissions… .f...p... /cygdrive/c/Documen…. (ok, 3 problems..) 3. I can change a file on in the “daily.#” areas and none of the other (supposedly linked) files get changed. Here are the main points of my rsnapshot.conf: snapshot_root /cygdrive/u/docs_bkup/ #cmd_cp /usr/bin/cp cmd_rm /usr/bin/rm cmd_rsync /usr/bin/rsync interval daily 3 # I’ve tried several configurations…this is the current one… rsync_short_args -arltgoDvzi rsync_long_args --delete --numeric-ids --relative --delete-excluded --modify-window=1 #If tested this both ways… # 0-> seems to copy less and du reports less space. link_dest 1 # just a test source – hard to see the destination, but it’s “.” backup /cygdrive/c/Documents and Settings/Link/My Documents/My Music . Any ideas or other things I can provide to help with this troubleshooting? Thanks, Link McGinnis Cedar Springs Christian Stores 504 N. Peters Rd Knoxville, TN 37931 865-246-2042 x225 -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.385 / Virus Database: 268.5.0/325 - Release Date: 4/26/2006 -------------- next part -------------- HTML attachment scrubbed and removed
On Thu, 2006-04-27 at 14:21 -0400, Link McGinnis wrote:> I am using rsync/rsnapshot on Windows XP (via cygwin) to backup to a > mapped share (/u) on a Linux server.> .f...p... /cygdrive/c/Documen?. > snapshot_root /cygdrive/u/docs_bkup/ > rsync_short_args -arltgoDvzi >You mean you have mounted a share from some Linux server as U: on your Windows machine and rsnapshot is transferring from C: to U:, both drives accessed through Cygwin? This arrangement strikes me as very awkward. For one thing, you don't get the benefit of incremental transfer. I can see how permissions on U: would get mangled in rsync to Cygwin to Windows to Samba (I'm guessing) to Linux conversion, causing U: to ignore rsync's attempt to set the permissions from the source on it. That means the permissions differ every time, so rsync doesn't try to hard link files, but even if it did, hard linking is unlikely to work through layers of Cygwin, Windows, and Samba. As a workaround, don't preserve permissions. You clearly tried to preserve everything except permissions, but keep in mind that -a, your first short option, includes -p. Delete the a: "-rltgoDvzi". I would also recommend not preserving user and group ownership. This should get rid of the itemized differences, but rsync is still likely to fail to hard link identical files. A better solution would be to have the Windows machine push the files to an rsync daemon running on the Linux server. However, rsnapshot only supports local snapshot roots, so you would have write your own script to invoke rsync. This isn't so bad. Just come up with a name for the destination directory and use --link-dest to link to the previous one. To set up an rsync daemon, write a configuration file according to the instructions in rsyncd.conf(5), and then invoke the daemon using rsync --daemon --config=<the-file>. This strategy will give you incremental transfer and hard linking, and you can safely tell rsync to preserve permissions. Nonetheless, the permissions rsync reads through Cygwin won't meaningfully represent Windows permissions. -- Matt McCutchen hashproduct@verizon.net http://hashproduct.metaesthetics.net/
Link,
I discovered a way that you can have your Windows machine push to an
rsync daemon and still have rsnapshot manage the backups so you don't
have to mess around with backup numbers and link-dest yourself. In
short, give the daemon a "post-xfer exec" that invokes rsnapshot. (CC
to the rsync list because others might wish to use this setup.)
Setting this up requires some fiddling and testing on the Linux server.
First, add a post-xfer exec line to call a script ./kick-rsnapshot, like
this:
[bkups]
path = bkups
read only = false
auth users = <all the users>
secrets file = bkups.secrets
post-xfer exec = ./kick-rsnapshot
Then, create the following kick-rsnapshot script and make it executable:
#!/bin/bash
if [ "$RSYNC_EXIT_STATUS" == "0" ]; then
rsnapshot -c rsnapshot.conf occasional
fi
Note that rsnapshot is only called if the push is successful. If the
push gets interrupted, you can just keep trying, and rsnapshot will
store the backup when you eventually succeed. If "file is vanished"
is
a problem, you can change the if:
if [ "$RSYNC_EXIT_STATUS" == "0" ] || [
"$RSYNC_EXIT_STATUS" == "24" ]; then
Now, write the configuration file rsnapshot.conf. If your system has
a /etc/rsnapshot.conf.default, you may wish to copy it to the daemon's
directory and edit from there.
Choose a snapshot root inside the daemon directory, _different_ from the
module directory. The default log and lock files are in system-wide
areas; either change them to be inside the daemon directory or comment
them out. Set up a single backup interval; I used "occasional".
Finally, create a single backup entry as follows:
backup /path/to/daemon/bkups/ ./
rsync_long_args=--link-dest=../../bkups/
This entry copies the module contents into the snapshot. Since the
module is also specified as a link-dest directory, all files except
directories get hard-linked.
Now test the setup by pushing to the rsync daemon. Since the backup
management is done on the Linux server, the command on Cygwin is a
one-liner:
rsync <options> /cygdrive/c/Documents and Settings/$USERNAME/My
Documents/My Music/ $user@$bkup_ip::bkups/
(Note the double colon, w
Rsync on cygwin connects directly to the rsync daemon and uploads the
files to the module. Then the daemon tells rsnapshot on the Linux
server to convert the module to a snapshot. Check to see whether an
occasional.0 with the user's files has appeared on the Linux server.
As long as a file remains unchanged, its appearances in the module and
in all the snapshots will be hard-linked together. When the source file
changes, the rsync daemon writes a new destination file and moves it
over the old hard-linked one, so old backups are not corrupted.
So that's how you would accommodate a single user, but you have multiple
users. Give each user his/her own module. You can put all the
usernames and passwords in the same secrets file and mention that file
once at the top; you can also move the "read only" and "post-xfer
exec"
lines to the top. Then, give each module a separate "auth users" line
naming just the user who will be pushing to the module.
I'm guessing only one user pushes at a time, and each push creates a
snapshot. Do you want each snapshot to contain (a) the data for all
users or (b) just the user who pushed? For (a), simply add a separate
"backup" entry for each module to rsnapshot.conf.
I think (b) makes more sense, but it takes more work. You need a
separate rsnapshot installation for each user with its own configuration
file and snapshot root, and then you should make kick-rsnapshot examine
$RSYNC_MODULE_NAME and/or $RSYNC_USER_NAME to determine which rsnapshot
installation to kick.
If you want to change an rsnapshot setting, you shouldn't have to modify
each configuration file separately. You might want to write a template
configuration file with placeholders like USER. Then you can do
something like this in kick-rsnapshot:
sed -e "s/USER/$RSYNC_USER_NAME/g" rsnapshot.conf.template | rsnapshot
-c /dev/stdin occasional
Devious, isn't it?
By the way, rsnapshot doesn't seem to like my version of cp. rsnapshot
calls "cp -al occasional.0/ occasional.1/", but cp refuses to create a
destination directory named with a trailing slash. If you have this
problem, write a script "mycp" that removes the slash (below) and
specify it as cmd_cp in rsnapshot.conf.
#!/bin/bash
/bin/cp -al $2 ${3%/}
> Do I run this on the samba server at startup so that it's running all
of the time? Or, am I supposed to start it each time I want to make a
backup? It seems to need to be attended (supply password).
Have the daemon start (call ./start) when the system starts and stop
(call ./stop) when the system shuts down. The Windows user pushing her
files does need to enter her rsync password, or she can store it in a
file and give rsync --password-file=<file>.
I realize this setup procedure is complicated, but I think the resulting
system will work well. If you need help carrying it out, perhaps we
could go on AIM (mattmccutchen) or I could SSH onto your machine and
guide you in the ytalk shell. Enjoy!
-- Matt McCutchen
Matt,> No, you had mounted an area on the Linux server as U:, which is to > say /cygdrive/u, on the Windows machine, presumably using Samba. You > had rsync doing what appeared to be a local copy from one area of the > local Cygwin filesystem to another: /cygdrive/c to /cygdrive/u. The > only network traffic was related to reads and writes on /cygdrive/u, and > no additional processes were executed on the Linux server. > > "Rsync over SSH" is used when you give a source or destination with a > hostname and a single colon. Rsync logs you in to the Linux server > using SSH and executes another rsync process there. The two rsync > processes talk to each other over the SSH connection. Rsync over SSH > involves pipes between rsync and SSH, and the Cygwin implementation of > those pipes was at one point buggy, causing rsync to hang.I'm not trying to dispute this - I assure you I know enough only to cause problems. But, looking back at the script that I was using, I established the rsync command with a "destination" variable of "remote_location="link@192.168.1.111:home/". This supports your "one colon" requirement for ssh. Before I setup the ssh key, this command forced me to supply a password. Then, once the key was established (ssh-keygen -t rsa), no password was needed. I assume that this meant that ssh was getting involved but I have not investigated logs or seen other evidence of it. Deciding on a method... My requirements for making this decision are: 1. User's shouldn't even know their PC is being backed up. These user's (mostly product buyers) don't understand technology and don't want to (I don't want them to either). Their backups have to be automatically scheduled through Window's "Scheduled Tasks" to run at night. This means the script that runs has to handle everything that a user may have needed to provide (paths, passwords, etc). 2. The backups could be placed either in the users' home directory as: 1. /home/link/bkup/daily.0 /daily.1, etc. Or in a centralized location as: 2. /samba/bkups/link/daily.0 /daily.1 I don't care either way. Responses to "Cons" within...> Global daemon: > + Daemon rotates the backups automaticallyThis is a big PRO.> - No encryptionok since these PCs are all on a private network (crossed fingers).> - Port might be blocked by firewallsI can modify the firewall to allow port access.> + User's command line is simple (especially if the daemon listens on the > privileged port 873, in which case the user can omit --port) > - Has to be started and stoppedOn the server side? Is the user responsible for this? They don't even log into this server (knowingly) so this could be a deal-breaker.> +/- Daemon accounts are used and must be maintained separately from > system onesNot a problem (as far as I can tell)> + Only one daemon is needed > +/- Centralized; backups most naturally kept in a single directory, > configurably owned by individual system users or by a single userAs in "/samba/bkups/<USER>/daily.0 ??? -> Good!> Single-use personal daemons over SSH:Is this "single-use daemon" running on the server? And the user's PC starts the rsync communication?> + Daemon rotates the backups automatically > + Encryption (SSH) > + Uses well-known port 22 > + Requires no starting and stopping > - User has to pass rsync "-e ssh"Can this be automated at the beginning of the backup? Does it require user-intervention?> - "rsyncd.conf" has to be in user's home directory on server _or_ user > has to pass "--rsync-path 'cd <dir> && rsync'" where <dir> is the > daemon's install directoryOK if it can be added to the automated script...> +/- Only system accounts are usedThis seems OK to me unless I don't understand the implications...> - A separate daemon-and-rsnapshot setup is needed for each user, but you > can write a script to create them quicklyThat is OK> +/- Each user owns his/her own setup and backups, but you can still keep > all backups in a single directory and/or provide centralized > configuration files >> Plain rsync over SSH: > - User must rotate backups by passing rsync > "--rsync-path ./rsync-and-kick-rsnapshot" _or_ running "ssh > backupmachine kick-rsnapshot" after each successful transferThis sounds like a deal-breaker. Can it be part of the script?> + Encryption (SSH) > + Uses well-known port 22 > + Requires no starting and stopping > +/- Only system accounts are usedThis seems to be the simplest setup but may require user-intervention. Or can this also be included in the automated script? I'm sorry I haven't been able to stay with this continually, but I wear a lot of hats. I hope this time is productive for you and that others will benefit from this discussion and setup. I'm documenting every forward progress and can provide that when the project is completed. Thanks again for all of your help. Link McGinnis -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.385 / Virus Database: 268.5.2/329 - Release Date: 5/2/2006