rsync - Sep 2012 - collecting the differences between a local host and remote archive to third local location (a usb device)

Hi everyone,

I'm trying to archive to a remote host but the size of the data to copy is
prohibitively large to consider doing the sync over the wire (it's TBs of
raw video), so i was wondering if it was possible to use a usb attached storage
device as the transport medium.

Is it possible for rsync to compare the two systems (over the network) finding
any newer or different files on the local system as compared to the remote
archive (ignoring local deletes), then copy only those new/differences to a
locally attached portable drive so that the portable drive can be physically
taken across town to the other system, plugged in and then the new data be
ingested so that a subsequent rsync between the the two systems would see those
files as synchronised?

I thought the --compare-dest option might be the way to go, as per this post:

http://lists.samba.org/archive/rsync/2006-June/015827.html

but it seems to not behave the way i thought it would.

Thanks in advance, and apologies if I'm asking a bleeding obvious question.
:)

Kind regards,
Ben

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20120927/9587d4a7/attachment.html>

Hi,

So in some ways, you would like to use rsync as a "diff/patch" tool.
There
are some similar questions out there
(here<http://serverfault.com/questions/62364/get-rsync-to-generate-a-patch-file-instead-of-copying-across-files>
 or there
<http://lists.samba.org/archive/rsync/2005-January/011439.html>),
and the answers are quite similar too. Basically, two methods :
I. Bash scripting (1. Do a dry run and export the files list, 2. Get only
these files on your USB, 3. Push the new/updated files on the original
system)
II. Batch-mode (see this
example<http://www.linuxmantra.com/2011/02/rsync-in-batch-mode.html>
)

Hope that will help,
Greg

On Thu, Sep 27, 2012 at 9:33 AM, Benjamin Ward <ben at forward.net.au>
wrote:
>
> Hi everyone,
>
> I'm trying to archive to a remote host but the size of the data to copy
is
> prohibitively large to consider doing the sync over the wire (it's TBs
of
> raw video), so i was wondering if it was possible to use a usb attached
> storage device as the transport medium.
>
> Is it possible for rsync to compare the two systems (over the network)
> finding any newer or different files on the local system as compared to the
> remote archive (ignoring local deletes), then copy only those
> new/differences to a locally attached portable drive so that the portable
> drive can be physically taken across town to the other system, plugged in
> and then the new data be ingested so that a subsequent rsync between the
> the two systems would see those files as synchronised?
>
> I thought the --compare-dest option might be the way to go, as per this
> post:
>
> http://lists.samba.org/archive/rsync/2006-June/015827.html
>
> but it seems to not behave the way i thought it would.
>
> Thanks in advance, and apologies if I'm asking a bleeding obvious
> question. :)
>
> Kind regards,
> Ben
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20120927/7ac9959c/attachment.html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have never actually used it personally but --only-write-batch can do
that.

On 09/27/12 03:33, Benjamin Ward wrote:
> 
> Hi everyone,
> 
> I'm trying to archive to a remote host but the size of the data to
> copy is prohibitively large to consider doing the sync over the
> wire (it's TBs of raw video), so i was wondering if it was possible
> to use a usb attached storage device as the transport medium.
> 
> Is it possible for rsync to compare the two systems (over the
> network) finding any newer or different files on the local system
> as compared to the remote archive (ignoring local deletes), then
> copy only those new/differences to a locally attached portable
> drive so that the portable drive can be physically taken across
> town to the other system, plugged in and then the new data be
> ingested so that a subsequent rsync between the the two systems
> would see those files as synchronised?
> 
> I thought the --compare-dest option might be the way to go, as per
> this post:
> 
> http://lists.samba.org/archive/rsync/2006-June/015827.html
> 
> but it seems to not behave the way i thought it would.
> 
> Thanks in advance, and apologies if I'm asking a bleeding obvious 
> question. :)
> 
> Kind regards, Ben
> 
> 
> 
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBkgY4ACgkQVKC1jlbQAQesmACgleHk921/Shg7AIqQ/KknIXmW
0HwAn3LlIDWQKvyNioECqfVkCNWRSnUP
=4kBs
-----END PGP SIGNATURE-----

Benjamin Ward wrote:
> but it seems to not behave the way i thought it would.
> 
How did you think it would work?

I use it  and will describe what it does w/my example:

I'm using: (according to ps)

/usr/bin/rsync --archive --human-readable --acls --xattrs --hard-links 
--no-inc-recursive --prune-empty-dirs --whole-file --one-file-system 
--compare-dest=/home/. --exclude=**/.recycle/** 
--exclude=CPAN-ishtar-build-cache/** /home/.snapdir/@GMT-2012.10.11-02.23.25/. 
/home.diff/


This takes yesturdays snapshot and compares it to today file system 
(compare-dest=/home) and puts the differences (the files
I would need to bring "dest" "up to date", onto /home.diff
(or your usb drive)...

Since I'm running the above on a single machine there is no network so I use
whole-file and no-inc-recursive.

Is that what you were wanting?   Sometimes it's a matter of figuring out, in
to
which slot, each of the compared
items go into...

I wonder if rdiff would do this nicely...?
http://librsync.sourcefrog.net/doc/rdiff.html

On Thu, Sep 27, 2012 at 12:33 AM, Benjamin Ward <ben at forward.net.au>
wrote:
>
> Hi everyone,
>
> I'm trying to archive to a remote host but the size of the data to copy
is
> prohibitively large to consider doing the sync over the wire (it's TBs
of
> raw video), so i was wondering if it was possible to use a usb attached
> storage device as the transport medium.
>
> Is it possible for rsync to compare the two systems (over the network)
> finding any newer or different files on the local system as compared to the
> remote archive (ignoring local deletes), then copy only those
> new/differences to a locally attached portable drive so that the portable
> drive can be physically taken across town to the other system, plugged in
> and then the new data be ingested so that a subsequent rsync between the
> the two systems would see those files as synchronised?
>
> I thought the --compare-dest option might be the way to go, as per this
> post:
>
> http://lists.samba.org/archive/rsync/2006-June/015827.html
>
> but it seems to not behave the way i thought it would.
>
> Thanks in advance, and apologies if I'm asking a bleeding obvious
> question. :)
>
> Kind regards,
> Ben
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20121011/6395e094/attachment.html>