rsync - Dec 2011 - sync prob with big files

Hi list.

We have 2 NAS-Devices, each mounted on the same (virtual) box. For our
backup we use bacula. Bacula ist writing its files to the first NAS
(192.168.1.9). These are big files, up to 160 GB.

For implementing a good backup strategy we decided to mirror (sync)
another NAS-Device (192.168.1.8) to have redundant bacula backups. If
sync is done the Device will be kept at another location too.

I do have problems with syncing. rsync is exiting with error-code 1.
Sometimes early, sometimes late in progress. This is what I am using in
a shell script:

rsync --omit-dir-times --size-only -avzAX \
--bwlimit=$KILOBYTES_PER_SECOND --devices --specials --numeric-ids \
--delete $BACULA_BACKUP_DIR $MOUNT_DIR/$SYNC_DIR

The bandwidth limit is 2500 kps. Without the limit it was working so far
since my boss told me to stop using it this way because he was afraid of
rsync taking all the network resources.

BTW, both NAS-Devices are mounted aus type cifs. My rsync version is
rsync  version 3.0.7  protocol version 30

Is this a known issue or does this make sense to anyone in any way?

Thanks for your help.
-fuz

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

OK, several things here....

1. according to the man page error code 1 is a syntax error meaning it
is simply failing to run.  Try expanding those variables to make sure
they are coming out the way you think they are.  Also, capture any
error that rsync outputs (this is actually easier without the -v).

2.  --devices and --specials are both included within -a

3.  Can CIFS actually handle ACLs, XATTRs, devices, and specials?  In
my very limited non-Windows guy experience it can't.  OTOH, if both
source and target are CIFS then the source shouldn't have anything
that the target can't handle.

4.  You probably shouldn't use --size-only on a regular basis.  It is
generally used to correct previous copy methods that didn't sync
timestamps.  But since that is included in -a you shouldn't need to
keep using it.  It is possible for rsync to miss changes to a file in
this mode because not all changes modify the file size.

On 12/07/11 10:15, fuzzy_4711 wrote:
> Hi list.
> 
> We have 2 NAS-Devices, each mounted on the same (virtual) box. For
> our backup we use bacula. Bacula ist writing its files to the first
> NAS (192.168.1.9). These are big files, up to 160 GB.
> 
> For implementing a good backup strategy we decided to mirror
> (sync) another NAS-Device (192.168.1.8) to have redundant bacula
> backups. If sync is done the Device will be kept at another
> location too.
> 
> I do have problems with syncing. rsync is exiting with error-code
> 1. Sometimes early, sometimes late in progress. This is what I am
> using in a shell script:
> 
> rsync --omit-dir-times --size-only -avzAX \ 
> --bwlimit=$KILOBYTES_PER_SECOND --devices --specials --numeric-ids
> \ --delete $BACULA_BACKUP_DIR $MOUNT_DIR/$SYNC_DIR
> 
> The bandwidth limit is 2500 kps. Without the limit it was working
> so far since my boss told me to stop using it this way because he
> was afraid of rsync taking all the network resources.
> 
> BTW, both NAS-Devices are mounted aus type cifs. My rsync version
> is rsync  version 3.0.7  protocol version 30
> 
> Is this a known issue or does this make sense to anyone in any
> way?
> 
> Thanks for your help. -fuz
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7flPkACgkQVKC1jlbQAQf76gCfdI1ZggftZhsxmt9TDfTV6qfX
L5AAnRvaYr1rNc9wxyCtMJ3kuamxQbVi
=hXfm
-----END PGP SIGNATURE-----

fuzzy_4711 schrieb:
> Hi list.
>
> rsync --omit-dir-times --size-only -avzAX \
> --bwlimit=$KILOBYTES_PER_SECOND --devices --specials --numeric-ids \
> --delete $BACULA_BACKUP_DIR $MOUNT_DIR/$SYNC_DIR
I miss -c (or --checksum) there.
You never know whether the filesize changed, or whether the time is correct.
I am much to paranoid to do it without -c, I'd recomment using it at 
least on weekends when noone is working to make sure the data is right, 
since calculating the checksum does some HD accessing such a huge file.

Jou

So you are using rsync on boxA that replicates a file mounted via CIFS
on boxB to a file mounted via CIFS on boxC?


How do you know this isn't a CIFS problem instead of a rsync one? Try
rsyncing the file from boxB to /tmp on boxA and if that's successful,
rsync it to boxC. Maybe there's a problem with one of the CIFS mounts?

Jason

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1