Hi I'm trying to build a backup system based on rsync. It should work with different destinations (local, share, remote etc) that's why I'm looking for ways how every operation can be controlled from the client side. I want to create full backups and incremental backups. For the incremental backups I use --compare-dest lastfullbackup. This works nicely except that the whole directory tree is created even if no files are copied. This is mentioned several times on the Internet and the usual answer is that there's no way around it because of the way that rsync works. I don't mind that they are created if there's a (rsync-)way to delete them again afterwards, something like rsync -a --delete --prune-empty-dirs dest dest Unfortunately at least one place has to be local, doing a remote-remote is not possible. The second problem is that I want to delete older backups. I can use rsync -a --delete empty_local_folder remote_folder_to_delete to get rid of the contents. However the folder itself still remains. So after a while I'd have a lot of empty folders. As I want this to work on various systems I can't (and don't want to) use remote shell hacks like "do_something && rsync ". Also the policy of what and when to delete should be configurable from the client. Are there pure rsync ways to solve these two problems? Thanks bye Fabi
Try having a look at this, for inspiration. It does require the capability of hard-links on the remote end though... and it does not resolve your "delete target directory" either, but it might save you some fiddeling around :-) https://blog.interlinked.org/tutorials/rsync_time_machine.html HC On 02-06-2016 10:27, Fabian Cenedese wrote:> Hi > > I'm trying to build a backup system based on rsync. It should > work with different destinations (local, share, remote etc) that's > why I'm looking for ways how every operation can be controlled > from the client side. > > I want to create full backups and incremental backups. For the > incremental backups I use --compare-dest lastfullbackup. This > works nicely except that the whole directory tree is created > even if no files are copied. This is mentioned several times > on the Internet and the usual answer is that there's no way > around it because of the way that rsync works. I don't mind > that they are created if there's a (rsync-)way to delete them > again afterwards, something like > > rsync -a --delete --prune-empty-dirs dest dest > > Unfortunately at least one place has to be local, doing a > remote-remote is not possible. > > The second problem is that I want to delete older backups. > I can use > > rsync -a --delete empty_local_folder remote_folder_to_delete > > to get rid of the contents. However the folder itself still remains. > So after a while I'd have a lot of empty folders. > > As I want this to work on various systems I can't (and don't > want to) use remote shell hacks like "do_something && rsync ". > Also the policy of what and when to delete should be configurable > from the client. > > Are there pure rsync ways to solve these two problems? > > Thanks > > bye Fabi > >
Fabian Cenedese <Cenedese at indel.ch> wrote:> Are there pure rsync ways to solve these two problems?Short answer - no I don't think there is. My feeling is that rsync (at least, rsync on it's own) isn't the right tool for the job. One think I would comment on though is that, IMO, making backup policies under the control of the client is "asking for trouble" for the general case. Your use case may well be different, but IME you can't trust clients to do the right thing - unless they are completely under your control, in which case there's little difference between client or server controlled policies. Personally I use a combination of rsync to maintain a clone of each client onto one server, and then StoreBackup (on that one server) to manage multiple generations of backups with file-level dedupe etc.
At 10:52 02.06.2016, Hans-Christian Jehg wrote:>Content-Transfer-Encoding: 7bit > >Try having a look at this, for inspiration. > >It does require the capability of hard-links on the remote end though... and it does not resolve your "delete target directory" either, but it might save you some fiddeling around :-) > >https://blog.interlinked.org/tutorials/rsync_time_machine.htmlThanks, I already saw that before. I looked for existing backup programs but none really did what I wanted that's why I'm trying to do it myself. This script is bash and also uses the "remote shell hacks" using SSH. As I want to run it also from Windows I'm looking for a rsync solution. bye Fabi
In <20160602082722.8F11829233B7 at macserver.private>, on 06/02/16 at 10:27 AM, Fabian Cenedese <Cenedese at indel.ch> said: Hi Fabian,>rsync -a --delete empty_local_folder remote_folder_to_delete>to get rid of the contents. However the folder itself still remains. So >after a while I'd have a lot of empty folders.This comes up on the list every now and then. Here's a snip from an archive message: Here is how you would write an "rsyncrm" script that can be invoked like "rsyncrm -ni remote::volume/directory". Note that this reveals the attributes of the current directory to the server. #!/bin/bash victim="${@: -1:1}" rsync -d --del --filter="R /$(basename -- "$victim")" --filter='- /*' \ "${@:1:$#-1}" . "$(dirname -- "$victim")" This works because it deletes the unwanted directory from the parent directory. Steven -- ---------------------------------------------------------------------- "Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. www.scoug.com www.arcanoae.com www.warpcave.com ----------------------------------------------------------------------
At 17:57 02.06.2016, Steven Levine wrote:>>rsync -a --delete empty_local_folder remote_folder_to_delete > >>to get rid of the contents. However the folder itself still remains. So >>after a while I'd have a lot of empty folders. > >This comes up on the list every now and then. > >Here's a snip from an archive message: > > Here is how you would write an "rsyncrm" script that can be invoked like > "rsyncrm -ni remote::volume/directory". Note that this reveals the > attributes of the current directory to the server. > > #!/bin/bash > victim="${@: -1:1}" > rsync -d --del --filter="R /$(basename -- "$victim")" --filter='- /*' \ > "${@:1:$#-1}" . "$(dirname -- "$victim")" > >This works because it deletes the unwanted directory from the parent >directory.Took me some time to find out what it does but it actually seems to work, thanks a lot. Now let's see if this method can also be used to really prune empty dirs in the destination. Is there no page with rsync tips and tricks? Would be nice to have such cool commands collected in one place. bye Fabi
At 10:27 02.06.2016, Fabian Cenedese wrote:>Content-Transfer-Encoding: 7bit > >Hi > >I'm trying to build a backup system based on rsync. It should >work with different destinations (local, share, remote etc) that's >why I'm looking for ways how every operation can be controlled >from the client side. > >I want to create full backups and incremental backups. For the >incremental backups I use --compare-dest lastfullbackup. This >works nicely except that the whole directory tree is created >even if no files are copied. This is mentioned several times >on the Internet and the usual answer is that there's no way >around it because of the way that rsync works. I don't mind >that they are created if there's a (rsync-)way to delete them >again afterwards, something like > >rsync -a --delete --prune-empty-dirs dest dest > >Unfortunately at least one place has to be local, doing a >remote-remote is not possible.So far I haven't found a solution to this problem. I've now seen this release note for the last version 3.1.2: - Don't create an empty backup dir for a transferred file that doesn't exist yet. I'm wondering now if that would help in my case with --compare-dest. But I don't have it compiled for Windows. Can anybody confirm if the newest rsync does not create empty folders anymore? Thanks bye Fabi