In my new bash script, I'm doing what I think is a very simple rsync
command the way I'm used to doing it. I just do a lot of setup and
checking before I get to it.
When I run it, it gets very unhappy with me. It's probably something
very simple.
I need to build the rsync command in a string so that some things can go
away - like if my variables DRY_RUN and DELETE are undefined, they
should be gone when the string is evaluated. Without this, DRY_RUN
turns into "" and rsync still sees it as the first (null) parameter
and
gets totally lost. I brought this up in a previous thread here awhile ago.
The ls commands are only in the script to document the state of things
right before rsync runs.
The first version of COMMAND (commented out here) was intended to
protect the arguments if there were embedded blanks. It also fails when
I use it, but with slightly different error messages.
/home/bigbird/pgm/backup_periodic/ is where the script is stored and is
the current directory when it runs. This script is nested within a
couple of others, the first of which calls the the second using sudo, so
everything runs as root.
I'm running my script with
shopt -s -o xtrace # debug
shopt -s -o verbose # debug
That's what makes this output a little dense, but complete.
What the heck am I doing to get rsync so confused?
(The destination was built using my previous notebook (RIP). That's why
there's so much more in it. I'll move things around before I run
anything with --delete.)
BTW, when I first ran this, my log file did not exist and rsync said it
would ignore it because it didn't exist. In the past, this worked
fine. If it wasn't there, rsync just created it (which is the expected
behavior - at least to me).
TIA
Joe
kubuntu precise (12.04), 64-bit i5 notebook running in konsole under KDE.
bigbird at ramdass:~/bin$ rsync --version
rsync version 3.0.9 protocol version 30
ls -l ${LOGFILE1}
+ ls -l /home/bigbird/rlog/ramdass-01-internal_webcasts_log
-rw-rw-rw- 1 bigbird bigbird 0 Jan 24 05:11
/home/bigbird/rlog/ramdass-01-internal_webcasts_log
ls -l ${LOGFILE2}
+ ls -l /home/bigbird/rlog/ramdass-01-internal_webcasts_output
-rw-rw-rw- 1 root root 0 Jan 24 05:11
/home/bigbird/rlog/ramdass-01-internal_webcasts_output
ls -l ${MOUNT_POINT[0]}/${DIRECTORY[0]}/
+ ls -l /media/dataspace/data/webcasts/
total 12
drwxrwxr-x 5 bigbird bigbird 4096 Dec 14 13:24 Business
drwxrwxr-x 13 bigbird bigbird 4096 Jan 19 06:07 Health
drwxrwxr-x 10 bigbird bigbird 4096 Jan 10 12:38 Spiritual
ls -l ${MOUNT_POINT[1]}/${DIRECTORY[1]}
+ ls -l /media/dest/data/webcasts
total 44
drwxr-xr-x 6 bigbird bigbird 4096 Jun 4 2011 Business
drwxr-xr-x 2 bigbird bigbird 4096 May 5 2012 Comedy
drwxr-xr-x 2 bigbird bigbird 4096 Oct 20 2011 Computer
drwxr-xr-x 4 bigbird bigbird 4096 Jun 4 2011 Eco
drwxr-xr-x 3 bigbird bigbird 4096 Oct 26 2011 Education
drwxr-xr-x 13 bigbird bigbird 4096 Oct 31 2011 Health
drwxr-xr-x 2 bigbird bigbird 4096 Nov 4 2011 Links
drwxr-xr-x 2 bigbird bigbird 4096 Nov 6 2011 Music
drwxr-xr-x 2 bigbird bigbird 4096 Sep 26 2011 Nature
drwxr-xr-x 10 bigbird bigbird 4096 Nov 2 2011 Spiritual
-r--r--r-- 1 bigbird bigbird 1043 Dec 19 2010 webcasts-info.txt
##COMMAND="rsync ${DRY_RUN} -avushi ${DELETE} --stats --progress
--log-file=\"${LOGFILE1}\"
\"${MOUNT_POINT[0]}/${DIRECTORY[0]}/\"
\"${MOUNT_POINT[1]}/${DIRECTORY[1]}\" | tee
\"${LOGFILE2}\""
COMMAND="rsync ${DRY_RUN} -avushi ${DELETE} --stats --progress
--log-file=${LOGFILE1} ${MOUNT_POINT[0]}/${DIRECTORY[0]}/
${MOUNT_POINT[1]}/${DIRECTORY[1]} | tee ${LOGFILE2}"
+ COMMAND='rsync -n -avushi --stats --progress
--log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log
/media/dataspace/data/webcasts/ /media/dest/data/webcasts | tee
/home/bigbird/rlog/ramdass-01-internal_webcasts_output'
echo "RUNNING [[${COMMAND}]]" ## let user see what's being done
+ echo 'RUNNING [[rsync -n -avushi --stats --progress
--log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log
/media/dataspace/data/webcasts/ /media/dest/data/webcasts | tee
/home/bigbird/rlog/ramdass-01-internal_webcasts_output]]'
RUNNING [[rsync -n -avushi --stats --progress
--log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log
/media/dataspace/data/webcasts/ /media/dest/data/webcasts | tee
/home/bigbird/rlog/ramdass-01-internal_webcasts_output]]
${COMMAND} ## Do it!
+ rsync -n -avushi --stats --progress
--log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log
/media/dataspace/data/webcasts/ /media/dest/data/webcasts '|' tee
/home/bigbird/rlog/ramdass-01-internal_webcasts_output
sending incremental file list
rsync: link_stat "/home/bigbird/pgm/backup_periodic/|" failed: No such
file or directory (2)
rsync: link_stat "/home/bigbird/pgm/backup_periodic/tee" failed: No
such
file or directory (2)
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
ERROR: destination must be a directory when copying more than 1 file
rsync error: errors selecting input/output files, dirs (code 3) at
main.c(571) [Receiver=3.0.9]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605)
[sender=3.0.9]
RC=$?
+ RC=12
It seems to me that the | is interpreted literally. You need to escape it like this in your command variable: \| To make the logfile and source/destination paths "space proof" when you store the command in a variable you add escaped " around the paths like this: \"/path/to/logfile\". You should also check out running your command with eval, even if it is not needed in your exact scenario. Mvh *Hans-Kristian Bakke* Mob: 91 76 17 38 On 24 January 2013 12:08, Joe <josephj at main.nc.us> wrote:> In my new bash script, I'm doing what I think is a very simple rsync > command the way I'm used to doing it. I just do a lot of setup and > checking before I get to it. > > When I run it, it gets very unhappy with me. It's probably something > very simple. > > I need to build the rsync command in a string so that some things can go > away - like if my variables DRY_RUN and DELETE are undefined, they > should be gone when the string is evaluated. Without this, DRY_RUN > turns into "" and rsync still sees it as the first (null) parameter and > gets totally lost. I brought this up in a previous thread here awhile ago. > > The ls commands are only in the script to document the state of things > right before rsync runs. > The first version of COMMAND (commented out here) was intended to > protect the arguments if there were embedded blanks. It also fails when > I use it, but with slightly different error messages. > > /home/bigbird/pgm/backup_periodic/ is where the script is stored and is > the current directory when it runs. This script is nested within a > couple of others, the first of which calls the the second using sudo, so > everything runs as root. > > I'm running my script with > shopt -s -o xtrace # debug > shopt -s -o verbose # debug > That's what makes this output a little dense, but complete. > > What the heck am I doing to get rsync so confused? > > (The destination was built using my previous notebook (RIP). That's why > there's so much more in it. I'll move things around before I run > anything with --delete.) > > BTW, when I first ran this, my log file did not exist and rsync said it > would ignore it because it didn't exist. In the past, this worked > fine. If it wasn't there, rsync just created it (which is the expected > behavior - at least to me). > > TIA > Joe > > kubuntu precise (12.04), 64-bit i5 notebook running in konsole under KDE. > bigbird at ramdass:~/bin$ rsync --version > rsync version 3.0.9 protocol version 30 > > ls -l ${LOGFILE1} > + ls -l /home/bigbird/rlog/ramdass-01-internal_webcasts_log > -rw-rw-rw- 1 bigbird bigbird 0 Jan 24 05:11 > /home/bigbird/rlog/ramdass-01-internal_webcasts_log > ls -l ${LOGFILE2} > + ls -l /home/bigbird/rlog/ramdass-01-internal_webcasts_output > -rw-rw-rw- 1 root root 0 Jan 24 05:11 > /home/bigbird/rlog/ramdass-01-internal_webcasts_output > ls -l ${MOUNT_POINT[0]}/${DIRECTORY[0]}/ > + ls -l /media/dataspace/data/webcasts/ > total 12 > drwxrwxr-x 5 bigbird bigbird 4096 Dec 14 13:24 Business > drwxrwxr-x 13 bigbird bigbird 4096 Jan 19 06:07 Health > drwxrwxr-x 10 bigbird bigbird 4096 Jan 10 12:38 Spiritual > ls -l ${MOUNT_POINT[1]}/${DIRECTORY[1]} > + ls -l /media/dest/data/webcasts > total 44 > drwxr-xr-x 6 bigbird bigbird 4096 Jun 4 2011 Business > drwxr-xr-x 2 bigbird bigbird 4096 May 5 2012 Comedy > drwxr-xr-x 2 bigbird bigbird 4096 Oct 20 2011 Computer > drwxr-xr-x 4 bigbird bigbird 4096 Jun 4 2011 Eco > drwxr-xr-x 3 bigbird bigbird 4096 Oct 26 2011 Education > drwxr-xr-x 13 bigbird bigbird 4096 Oct 31 2011 Health > drwxr-xr-x 2 bigbird bigbird 4096 Nov 4 2011 Links > drwxr-xr-x 2 bigbird bigbird 4096 Nov 6 2011 Music > drwxr-xr-x 2 bigbird bigbird 4096 Sep 26 2011 Nature > drwxr-xr-x 10 bigbird bigbird 4096 Nov 2 2011 Spiritual > -r--r--r-- 1 bigbird bigbird 1043 Dec 19 2010 webcasts-info.txt > ##COMMAND="rsync ${DRY_RUN} -avushi ${DELETE} --stats --progress > --log-file=\"${LOGFILE1}\" \"${MOUNT_POINT[0]}/${DIRECTORY[0]}/\" > \"${MOUNT_POINT[1]}/${DIRECTORY[1]}\" | tee \"${LOGFILE2}\"" > COMMAND="rsync ${DRY_RUN} -avushi ${DELETE} --stats --progress > --log-file=${LOGFILE1} ${MOUNT_POINT[0]}/${DIRECTORY[0]}/ > ${MOUNT_POINT[1]}/${DIRECTORY[1]} | tee ${LOGFILE2}" > + COMMAND='rsync -n -avushi --stats --progress > --log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log > /media/dataspace/data/webcasts/ /media/dest/data/webcasts | tee > /home/bigbird/rlog/ramdass-01-internal_webcasts_output' > echo "RUNNING [[${COMMAND}]]" ## let user see what's being done > + echo 'RUNNING [[rsync -n -avushi --stats --progress > --log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log > /media/dataspace/data/webcasts/ /media/dest/data/webcasts | tee > /home/bigbird/rlog/ramdass-01-internal_webcasts_output]]' > RUNNING [[rsync -n -avushi --stats --progress > --log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log > /media/dataspace/data/webcasts/ /media/dest/data/webcasts | tee > /home/bigbird/rlog/ramdass-01-internal_webcasts_output]] > ${COMMAND} ## Do it! > + rsync -n -avushi --stats --progress > --log-file=/home/bigbird/rlog/ramdass-01-internal_webcasts_log > /media/dataspace/data/webcasts/ /media/dest/data/webcasts '|' tee > /home/bigbird/rlog/ramdass-01-internal_webcasts_output > sending incremental file list > rsync: link_stat "/home/bigbird/pgm/backup_periodic/|" failed: No such > file or directory (2) > rsync: link_stat "/home/bigbird/pgm/backup_periodic/tee" failed: No such > file or directory (2) > rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: > Broken pipe (32) > ERROR: destination must be a directory when copying more than 1 file > rsync error: errors selecting input/output files, dirs (code 3) at > main.c(571) [Receiver=3.0.9] > rsync: connection unexpectedly closed (9 bytes received so far) [sender] > rsync error: error in rsync protocol data stream (code 12) at io.c(605) > [sender=3.0.9] > RC=$? > + RC=12 > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20130124/582d229c/attachment.html>
On Fri 25 Jan 2013 00:08:52 NZDT +1300, Joe wrote:> In my new bash script, I'm doing what I think is a very simple rsync > command the way I'm used to doing it. I just do a lot of setup and > checking before I get to it. > > When I run it, it gets very unhappy with me. It's probably something > very simple.Your problem is with shell programming, not rsync, but that is frequently a requirement with rsync because of its stellar argument lists...> ##COMMAND="rsync ${DRY_RUN} -avushi ${DELETE} --stats --progress > --log-file=\"${LOGFILE1}\" \"${MOUNT_POINT[0]}/${DIRECTORY[0]}/\"ARRGGHHH. Do yourself a favour and increase the legibility and coding standard of your programs no end by using array variables that hold argument lists only but no commands, and preferably no arguments that need a path name. Like this rsyncargs=( -avushi -stats #--log-file= # don't do this, you will be paying for it ) LOGFILE="..." logargs=( .... ) Make it a rule to always program in a way that allows spaces in every argument. Anything going belly-up with a space in a file name is plain incompetent. Then run your commands rsync "${rsyncargs[@]}" --log-file="$LOGFILE" | tee "${logargs[@]}" You could set LOGFILE to /dev/null if desirable. Avoid having to eval huge strings, it's always risky and almost always bad, and avoid putting variables that need to be expanded later into argument lists held by other variables. That way you solve all your problems with when "|" is expanded, having to quote variables containing paths twice and so on, and your code stays legible and maintainable. If you need progress logging, I've found this to be easy, safe and effective (at the cost of a bit of performance - which you're not caring about as you're using bash): Log() { if [ -n "$logging" ]; then ( set -x "$@" ) else "$@" fi } Log yourcommand Don't bother trying to be compatible with other *sh-type shells on other *nix systems, trying to get any use out of such retarded constructs only leads to mental illness. HTH, Volker -- Volker Kuhlmann is list0570 with the domain in header. http://volker.dnsalias.net/ Please do not CC list postings to me.