rsync - Jan 2015 - [Bug 11086] New: rsync --delete has stopped to delete destination files that don't exist at the source

https://bugzilla.samba.org/show_bug.cgi?id=11086

            Bug ID: 11086
           Summary: rsync --delete has stopped to delete destination files
                    that don't exist at the source
           Product: rsync
           Version: 3.1.1
          Hardware: x64
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: luminarior at gmail.com
        QA Contact: rsync-qa at samba.org

I have a system Linux root partition wich I backup to another disk
periodically. I use the following script for that action:

#!/bin/bash
NICE=15
PH="/mnt/b"
cd /

nice -n $NICE rm -v /root/.xsession-errors
nice -n $NICE rm -v /root/dead.letter

nice -n $NICE /usr/bin/rsync -avhHxq --progress --stats --force --delete
--delete-before --log-file='/var/log/backup2diskrsync.log'
--exclude='/usr/portage/distfiles/' --exclude='/home/ftp/'
--exclude='/lost+found/' --exclude='/var/tmp/ccache/'
--exclude='/root/.ccache/' --exclude='/var/cache/edb/'
--exclude='/home/myuser/.cache/chromium/'
--exclude='/home/myuser/.opera/cache/'
--exclude='/home/myuser/.opera/opcache/'
--exclude='/home/mysearch/search/fsearch/swapfile'
--exclude='/home/myuser/workspace/mysearch/swapfile' / $PH/

The problem is that I don't see any "delete" messages in log file
anymore. For
example, I have the following in /usr/src/ directory on backup disk:
# du -sh /mnt/b/usr/src/*
0       /mnt/b/usr/src/linux
944M    /mnt/b/usr/src/linux-3.14.14-gentoo
961M    /mnt/b/usr/src/linux-3.16.5-gentoo
964M    /mnt/b/usr/src/linux-3.17.7-gentoo
972M    /mnt/b/usr/src/linux-3.18.3-gentoo
971M    /mnt/b/usr/src/linux-3.18.5-gentoo
But I don't have this outdated long time ago directories on my root
partition:
# du -sh /usr/src/*
0       /usr/src/linux
943M    /usr/src/linux-3.18.5-gentoo

And that's just a one of examples of outdated files and directories. My
backup
storage partition will become full soon.

The rsync log (after # /bin/bash backup2diskwithrsync.sh under root):
2015/01/31 17:25:56 [12091] building file list
2015/01/31 17:25:58 [12091] rsync: readlink_stat("/home/myuser/.gvfs")
failed:
Permission denied (13)
2015/01/31 17:25:59 [12091] 877346 files to consider
2015/01/31 17:26:01 [12091] >f..t...... etc/openvpn/openvpn-status.log
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.err
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.info
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.err
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.info
2015/01/31 17:26:03 [12091] .d..t...... var/log/
2015/01/31 17:26:03 [12091] >f.st...... var/log/backup2diskrsync.log
2015/01/31 17:26:03 [12091] >f.st...... var/log/messages
2015/01/31 17:26:03 [12091] >f.st...... var/log/net-snmpd.log
2015/01/31 17:26:04 [12091] >f.st...... var/log/samba/log.nmbd
2015/01/31 17:26:05 [12091] >f..t......
var/tmp/kdecache-root/icon-cache.kcache
2015/01/31 17:26:05 [12091] >f..t......
var/tmp/kdecache-myuser/plasma_theme_G-Remix-T_v5.3.kcache
2015/01/31 17:26:06 [12091] >f..t......
var/tmp/kdecache-myuser/plasma_theme_internal-system-colors.kcache
2015/01/31 17:26:06 [12091] .d..t......
var/www/localhost/htdocs/rutorrent/share/users/myuserutorrent/settings/erasedata/
2015/01/31 17:26:07 [12091] Number of files: 877,346 (reg: 758,168, dir:
67,562, link: 45,877, dev: 5,678, special: 61)
2015/01/31 17:26:07 [12091] Number of created files: 0
2015/01/31 17:26:07 [12091] Number of deleted files: 0
2015/01/31 17:26:07 [12091] Number of regular files transferred: 12
2015/01/31 17:26:07 [12091] Total file size: 25.31G bytes
2015/01/31 17:26:07 [12091] Total transferred file size: 379.83M bytes
2015/01/31 17:26:07 [12091] Literal data: 379.83M bytes
2015/01/31 17:26:07 [12091] Matched data: 0 bytes
2015/01/31 17:26:07 [12091] File list size: 24.44M
2015/01/31 17:26:07 [12091] File list generation time: 3.099 seconds
2015/01/31 17:26:07 [12091] File list transfer time: 0.000 seconds
2015/01/31 17:26:07 [12091] Total bytes sent: 404.40M
2015/01/31 17:26:07 [12091] Total bytes received: 291
2015/01/31 17:26:07 [12091] sent 404.40M bytes  received 291 bytes  35.17M
bytes/sec
2015/01/31 17:26:07 [12091] total size is 25.31G  speedup is 62.60
2015/01/31 17:26:07 [12091] rsync error: some files/attrs were not transferred
(see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

Error log in the console (some files it doesn't like probably)
rsync: readlink_stat("/home/xor/.gvfs") failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1165) [sender=3.1.1]

And still there are a lot of garbage on backup partition wich had to be deleted
a looong time ago:
# du -sh /mnt/b/usr/src/*
0       /mnt/b/usr/src/linux
944M    /mnt/b/usr/src/linux-3.14.14-gentoo
961M    /mnt/b/usr/src/linux-3.16.5-gentoo
964M    /mnt/b/usr/src/linux-3.17.7-gentoo
972M    /mnt/b/usr/src/linux-3.18.3-gentoo
971M    /mnt/b/usr/src/linux-3.18.5-gentoo

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11086

--- Comment #1 from Kevin Korb <rsync at sanitarium.net> ---
This is your problem:
> 2015/01/31 17:25:58 [12091] rsync:
readlink_stat("/home/myuser/.gvfs") failed: Permission denied (13)
If rsync hits any error it aborts the deletion process which it would have said
without -q.  You can override that with --ignore-errors but better to
--exclude=.gvfs since it isn't a real file/dir/filesystem.

Also, if you are making backups you should really look into rsync --link-dest
or an rsync based backup system if you don't feel like scripting it.

Also, --delete-before is much slower than --delete-during which is the default
version of --delete.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11086

--- Comment #2 from ddd <luminarior at gmail.com> ---
Thank you a lot! I added --exclude='.gvfs' parameter and everything
works now.
Thumbs up!

I was trying to use --delete-before because of low space on the partition. Now
it's really enough space and I use default --delete option.
> Also, if you are making backups you should really look into rsync
--link-dest or an rsync based backup system if you don't feel like scripting
it.
Hmm, what --link-dest stands for? Is it somehow related to hard-links on the
same filesystem? I usually try to exclude hardlinks like
--exclude='/home/ftp/'
wich usually works for me.

--link-dest=DIR         hardlink to files in DIR when unchanged
I don't understand what that option is doing.
I use rsync for easy backups of the system partition from the primary disk to a
backup one 2 times a week. It's very useful when I delete smth by accident
like
one time I did "crontab -r" instead of "crontab -e". ) And
in case 1st system
drive fails I change SATA interface immediately and start OS from the backup
one. Rsync based backup system? I've seen some of those. They usually
don't do
what I need (like they backup to files, snapshots with dates in them instead of
simple backup to another disk wich consumes too much space and you'll have
to
use liveDVD to start from such snapshot wich is time-consuming) or too
overburdened with options, configs etc. My scripts is very simple and does
exactly what I need.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11086

--- Comment #3 from Kevin Korb <rsync at sanitarium.net> ---
What you have is a mirror not a backup system.  Sure, it can protect you from
accidental deletions or disk failures but it wouldn't protect you from
intentional data destruction (think virus or hacker).

Yes, I was referring to date+time based "snapshots".  That is what
rsync with
--link-dest and some scripting can do and that is what most of the rsync based
backup systems use.

I prefer to have many backups in a system.  That way I can restore data from
the past if I need to (which I have).  It is pretty trivial to boot from
something like SystemRescueCD and restore your whole system from such a backup.
 In fact, if your backups are on a different system and you setup a network
boot environment you can boot from the backup over the network.  I did that
once when my desktop became unstable and I was trying to figure out if it was a
hardware problem or if it was caused by recent updates.  When I could duplicate
a several day old problem on your computer with the software it was running a
month ago I knew it was the hardware that was bad.  Turned out there was a
leaking capacitor behind the heatsink on the video card.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11086

--- Comment #4 from ddd <luminarior at gmail.com> ---
Well, maybe I should look into those rsync-based snapshot systems when I'll
buy
another 3 TB HDD. ) Can you recommend any of those backup systems?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11086

--- Comment #5 from Kevin Korb <rsync at sanitarium.net> ---
Not really.  I wrote my own.  IIRC BackupPC is the most up to date one.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11086

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|NEW                         |RESOLVED

-- 
You are receiving this mail because:
You are the QA Contact for the bug.