rsync - Jan 2006 - Shell Script Does not complete if rsync returns Code 24

I have a simple backup shell script that I am using for backups.  

I have a problem which I think is a result of this error: 

rsync warning: some files vanished before they could be transferred (code
24) at main.c(789)

Any command after the rsync never gets executed if I get the above error.
The file system is very large and we have engineers working at all hours so
it is rare that this would complete with out the code 24 error.

I am using rsync-2.6.4-3.

I read the FAQ which describes how to write a wrapper.

http://samba.anu.edu.au/rsync/FAQ.html

The problem is that I have no clue what to do with this and or how to make
it work with my script.

Any help would be appreciated.

The script follows:

#!/bin/sh -e
# Assumes the existence of
#${DIR_BKUP_ROOT}/{Hour-1, Hour-2.....}

DIR_BKUP_ROOT="/data1/backup/Hourly"
backup=`cat /etc/snapshot/include.text`
excludefile=/etc/snapshot/exclude.text
INDEX=`cat /data1/backup/Hourly/last-bk.txt`
RM="/bin/rm"
CP="/bin/cp"
RSYNC="/usr/bin/rsync"
DATE="/bin/date"

[ -x $RM -a -x $CP -a -x $RSYNC -a -x $DATE ] || exit 1

if [ -f ${DIR_BKUP_ROOT}/bk-stats.txt ]; then
                  $RM -f ${DIR_BKUP_ROOT}/bk-stats.txt
fi

echo Last backup $NEXTHOURLY started on `date` >>
${DIR_BKUP_ROOT}/bk-stats.txt

case "$INDEX" in
        'Hour-1')
                NEXTHOURLY="Hour-2"
                PREVBACK="Hour-1"
                CLEANUP="Hour-3"
        ;;
        'Hour-2')
                NEXTHOURLY="Hour-3"
                PREVBACK="Hour-2"
                CLEANUP="Hour-4"
        ;;
        'Hour-3')
                NEXTHOURLY="Hour-4"
                PREVBACK="Hour-3"
                CLEANUP="Hour-5"
        ;;
        'Hour-4')
                NEXTHOURLY="Hour-5"
                PREVBACK="Hour-4"
                CLEANUP="Hour-1"
        ;;
        'Hour-5')
                NEXTHOURLY="Hour-1"
                PREVBACK="Hour-5"
                CLEANUP="Hour-2"
        ;;
        *)
                echo "ERROR: Invalid hour!"
                exit 2
        ;;
esac
#
if [ -f ${DIR_BKUP_ROOT}/last-bk.txt ]; then
                  $RM -f ${DIR_BKUP_ROOT}/last-bk.txt
fi
#
if [ -d ${DIR_BKUP_ROOT}/$NEXTHOURLY ]; then
                $RM -rf ${DIR_BKUP_ROOT}/$NEXTHOURLY
fi
#
echo $NEXTHOURLY >> ${DIR_BKUP_ROOT}/last-bk.txt
#
#
#if [ -f ${DIR_BKUP_ROOT}/last-bk.txt ]; then
$RSYNC -R -a --exclude-from=$excludefile
--link-dest=${DIR_BKUP_ROOT}/${PREVBACK} $backup
${DIR_BKUP_ROOT}/${NEXTHOURLY}/
#fi
echo  $NEXTHOURLY backup completed `date` >> ${DIR_BKUP_ROOT}/bk-stats.txt
if [ -d ${DIR_BKUP_ROOT}/$CLEANUP ]; then
                $RM -rf ${DIR_BKUP_ROOT}/$CLEANUP
fi
echo  The removal of $CLEANUP completed `date` >>
${DIR_BKUP_ROOT}/bk-stats.txt
exit 0


------------------------------------
CIO Systems
John Gallagher
www.ciosystems.com
------------------------------------

Hi,

On Wednesday 11 January 2006 04:28, John Gallagher wrote:
> The problem is that I have no clue what to do with this and or how to make
> it work with my script.
If you want hide errors, remove the -e from your sh invocation or add || true
at the end of the rsync invocation.

Though I would not do either in a backup script. Better would be to just ignore
some of rsync's return codes.

Regards,

-- 
Ren? Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany)
            http://www.exactcode.de | http://www.t2-project.org
            +49 (0)30  255 897 45

On Tue, Jan 10, 2006 at 07:28:00PM -0800, John Gallagher
wrote:
> The problem is that I have no clue what to do with this and or how to
> make it work with my script.
The FAQ says to name the script something like rsync-no24.  All you'd
need to do in your script is to run rsync-no24 in place of rsync.

..wayne..

John Gallagher wrote:
>>If you want to handle errors from rsync in your shell script, 
>>then remove the "-e" and test for errors after your call to
rsync.
>>
>>Linus
> 
> 
> Let me start by saying my shell scripting skills are very weak, as if that
> were not already apparent.
> 
> I understand the -e will exit when the return code is other than 0.  I
> believe this is what I want with the exception of the rsync code 24 files
> have vanished error.  The script is a modified version of one of the
> examples found on http://www.mikerubel.org/computers/rsync_snapshots/.  It
> takes almost 3 hours to complete the index of the directory, sync and then
> removal of the oldest backup (Snapshot).  
> 
> Are you saying that in order to use the wrapper example below I need to
> remove the -e?  How do I call the wrapper or do I need to incorporate this
> example into the original script?  This is where I am clueless.....
> 
> From: http://samba.anu.edu.au/rsync/FAQ.html
> The easiest way to do this is to create a shell script wrapper. For
> instance, name this something like "rsync-no24": 
> 
> #!/bin/sh
> rsync "$@"
> e=$?
> if test $e = 24; then
>     exit 0
> fi
> exit $e
First, this is not an rsync issue.

You made this statement in your original post:

<quote>
Any command after the rsync never gets executed if I get the above error.
The file system is very large and we have engineers working at all hours so
it is rare that this would complete with out the code 24 error.
</quote>

That tells me that you want your script (the one you posted in your original 
message) to continue processing if rsync takes the error 24. If that is the 
case, then you need to remove the -e on your shell invocation OR call a wrapper 
that treats the error 24 as normal. Since you seem to be going in the wrapper 
direction, leave the -e on you shell invocation (so other errors will cause the 
script to exit immediately) and modify this line:

RSYNC="/usr/bin/rsync"

with the path of your wrapper that ignores the error 24. For example:

RSYNC="/usr/local/bin/rsync-no24"

Your wrapper shell script will notice that rsync got an error 24, and return an 
exit status of zero so your main shell script will think everything is fine.

Linus

> First, this is not an rsync issue.
> 
> That tells me that you want your script (the one you posted 
> in your original
> message) to continue processing if rsync takes the error 24. 
> If that is the case, then you need to remove the -e on your 
> shell invocation OR call a wrapper that treats the error 24 
> as normal. Since you seem to be going in the wrapper 
> direction, leave the -e on you shell invocation (so other 
> errors will cause the script to exit immediately) and modify 
> this line:
> 
> RSYNC="/usr/bin/rsync"
> 
> with the path of your wrapper that ignores the error 24. For example:
> 
> RSYNC="/usr/local/bin/rsync-no24"
> 
> Your wrapper shell script will notice that rsync got an error 
> 24, and return an exit status of zero so your main shell 
> script will think everything is fine.
> 
> Linus
You are correct this is not a rsync issue, however the FAQ is posted on how
to filter out the Code 24 error.  What was not clear to me was how to call
the no24 wrapper.

It is very clear now and I have modified the script based on your
recommendations. 

Thank you!

John