rsync - Jun 2007 - add inverse recursive feature or do a mkdir -p/install -D -d ?

Hi,

Background :
I am using rsync to push/sync FTP uploaded files from master server
to slave servers. On slaves the full directory path of each file doesn't
exist so rsync is unable to sync it and there a no options in rsync to do
something like "mkdir -p/install -D -d" on destination before sync
file.

N.B. I don't want to do a full sync of the root/home directory every time
a file is uploaded somewhere in the hierarchy on the master server

Can you implement recursive make directory for the parents missing
on the destination ?

As alternative can you add an option to do a inverse recursive
syncronisation
of every parents directoy missing until full path is available on the
destination.

As workaround I have coded a little script rrsync.sh who do
an inverse recursive sync by calling rsync dry-run on every parents and
do a full rsync from the first parent available on destination.


TEST

$ mkdir -p test1/toto1/toto2/toto3/toto4
$ mkdir -p test2
$ touch test1/toto1/toto2/toto3/toto4/a

$ rsync -av test1/toto1/toto2/toto3/toto4 test2/toto1/toto2/toto3/toto4
building file list ... done
rsync: mkdir "/home/baco/test2/toto1/toto2/toto3/toto4" failed: No
such
file or directory (2)
rsync error: error in file IO (code 11) at main.c(529) [receiver=2.6.9]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453)
[sender=2.6.9]

$ rsync -av test1/toto1/toto2/toto3/toto4/a test2/toto1/toto2/toto3/toto4/a
building file list ... done
rsync: push_dir#3 "/home/baco/test2/toto1/toto2/toto3/toto4" failed:
No
such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at
main.c(565) [receiver=2.6.9]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453)
[sender=2.6.9]

$ ./rrsync.sh test1/toto1/toto2/toto3/toto4/a
test2/toto1/toto2/toto3/toto4/a
rsync -av test1/toto1 test2/toto1

$ ./rrsync.sh test1/toto1/toto2/toto3/toto4 test2/toto1/toto2/toto3/toto4
rsync -av test1/toto1 test2/toto1

$ rsync -av test1/toto1 test2/toto1
building file list ... done
created directory test2/toto1
toto1/
toto1/toto2/
toto1/toto2/toto3/
toto1/toto2/toto3/toto4/
toto1/toto2/toto3/toto4/a

sent 196 bytes  received 66 bytes  524.00 bytes/sec
total size is 0  speedup is 0.00

$ ./rrsync.sh test1/toto1/toto2/toto3/toto4 test2/toto1/toto2/toto3/toto4
rsync -av test1/toto1/toto2 test2/toto1/toto2

$ ./rrsync.sh test1/toto1/toto2/toto3/toto4/a
test2/toto1/toto2/toto3/toto4/a
rsync -av test1/toto1/toto2 test2/toto1/toto2


CODE

#!/bin/bash
# rrsync.sh by Guy Baconniere

usage="$0 <src> <dst>"
src=${1:?$usage}
dst=${2:?$usage}

function crushedpath() {
        path="${1:?give a path please}"
        path=$(echo -n "${path}" | sed -e 's,/$,,')
        pathdepth=$(echo -n "${path}" | sed -e 's,[^/],,g' |
wc -c)
        echo "${path}"
        for (( i=1; ${i} < ${pathdepth}; i=$(( ${i}+1 )) )); do
                fpath=$(( ${pathdepth} - ${i} ))
                subpath=$(echo -n "${path}" | cut -d "/"
-f1-${fpath})
                if [ ! -z "${subpath}" ]; then
                        echo "${subpath}"
                fi
        done
}

function doublecrushedpath() {
        src="${1:?give a src path please}"
        dst="${2:?give a dst path please}"
        src=$(echo -n "${src}" | sed -e 's,/$,,')
        dst=$(echo -n "${dst}" | sed -e 's,/$,,')
        srcdepth=$(echo -n "${src}" | sed -e 's,[^/],,g' | wc
-c)
        dstdepth=$(echo -n "${dst}" | sed -e 's,[^/],,g' | wc
-c)
        if [ $srcdepth -lt $dstdepth ]; then
                maxdepth=$srcdepth;
        else
                maxdepth=$dstdepth;
        fi
        echo "${src} ${dst}"
        for (( i=1; ${i} < ${maxdepth}; i=$(( ${i}+1 )) )); do
                fsrc=$(( ${srcdepth} - ${i} ))
                fdst=$(( ${dstdepth} - ${i} ))
                subsrc=$(echo -n "${src}" | cut -d "/"
-f1-${fsrc})
                subdst=$(echo -n "${dst}" | cut -d "/"
-f1-${fdst})
                if [ ! -z "${subsrc}" ] || [ ! -z
"${subdst}" ]; then
                        echo "${subsrc} ${subdst}"
                fi
        done
}

function rrsync() {
        src=${1:?src please}
        dst=${2:?dst please}
        doublecrushedpath "${src}" "${dst}" | while read
t_src t_dst; do
                if ! rsync -nlD -- "${t_src}" "${t_dst}"
2>&1 \
                | grep -q -E 'rsync: push_dir#3 "[^"]*"
failed: No such
file or directory'; then
                        echo rsync -av "${t_src}" "${t_dst}"
|| true
                        break
                fi
        done

}

#crushedpath "${dst}"
#doublecrushedpath "${src}" "${dst}"

rrsync "${src}" "${dst}"



--

Infomaniak Network SA
Guy Baconniere <baco@infomaniak.ch>
Unix System Administrator
Certified Linux Engineer (RHCE, LPIC-2)
Avenue de la Praille 26
1227 Carouge (Geneva)
Switzerland (CH)
Phone +41 (0)22 820 3541
Fax +41 (0)22 820 3546
AS29222 / BACO-RIPE

On Wed, Jun 06, 2007 at 12:31:39PM +0200, Infomaniak Network SA / Guy Baconniere
wrote:
> $ rsync -av test1/toto1/toto2/toto3/toto4 test2/toto1/toto2/toto3/toto4
> building file list ... done
> rsync: mkdir "/home/baco/test2/toto1/toto2/toto3/toto4" failed:
No such
> file or directory (2)
You can use --relative (-R) to send the path info as part of the
sync:

  rsync -avR test1/./toto1/toto2/toto3/toto4 test2/

or

  cd test1
  rsync -avR toto1/toto2/toto3/toto4 ../test2/

The "/./" idiom in the first example was added in rsync 2.6.7, and
indicates the cut-off point between directories that are outside
the transfer, and those that are included in the transfer.

..wayne..