CentOS - Jun 2016 - remote backup

Il 07/06/2016 21:35, Keith Keller ha scritto:
> On 2016-06-04, Alessandro Baggi <alessandro.baggi at gmail.com>
wrote:
>> i've need to backup a partition of ~200GB with a local connection
of 8/2
>> mbps.
>>
>> Tool like bacula, amanda can't help me due to low bandwidth in
local server.
>>
>> I'm thinking rsync will be a good choice.
>
> If you want pseudo-snapshots (not real point-in-time snapshots) you can
> use rsnapshot or backuppc, both of which use rsync under the hood.  You
> get the advantages of rsync along with having an archive of previous
> backups.
>
> --keith
>
Thank you for your reply and sorry for late.

My needs is only get a copy of large dataset a make sure that it is not 
broken after transfer. After transfer, this data will be stored on local 
backup server where there is bacula installation.

For file transfer, to save time and bandwidth I will use rsync but I 
don't know how to check if those file will be corrupted.

How I can perform this check?
I can make an md5 for each file but for a great number of file this can 
be a problem.

Thanks in advance.

On 06/09/2016 08:18 AM, Alessandro Baggi wrote:
> How I can perform this check? 

Run rsync with the -c argument.

On 2016-06-09, Gordon Messmer <gordon.messmer at gmail.com>
wrote:
> On 06/09/2016 08:18 AM, Alessandro Baggi wrote:
>> How I can perform this check? 
>
>
> Run rsync with the -c argument.
Will this be very slow if Alessandro has a large number of files?  OTOH
if he really needs to ensure integrity there likely isn't a better
option.

--keith

-- 
kkeller at wombat.san-francisco.ca.us

On Thursday, June 09, 2016 05:18:03 PM Alessandro Baggi
wrote:
> Thank you for your reply and sorry for late.
> 
> My needs is only get a copy of large dataset a make sure that it is not
> broken after transfer. After transfer, this data will be stored on local
> backup server where there is bacula installation.
> 
> For file transfer, to save time and bandwidth I will use rsync but I
> don't know how to check if those file will be corrupted.
> 
> How I can perform this check?
> I can make an md5 for each file but for a great number of file this can
> be a problem.
Is there any chance you could switch to running ZFS or maybe BTRFS? They are 
ridiculously more efficient at sending reliable, incremental updates to a file 
system over a low bandwidth link and the load of rsync "at scale" can
be
enormous. 

In our case, with about half a billion files, doing rsync over a local Gb LAN 
took well over 24 hours - simply doing the discovery stage of rsync was nearly 
all the overhead due to IOPs limitations. 

Switching our primary backup method to using ZFS and send/receive of 
incremental snapshots cut the time to backup/replicate to under 30 minutes, 
with no significant change in server load. 

And don't let the name "incremental snapshots" fool you - the end
result is
identical to doing a full backup / copy, with all files being verified as binary
perfect as of the moment the snapshot was made. 

Really, if you can do this and you care about your data, you want to do this, 
even if you don't know it yet. The learning curve is significant but the 
results are well worth it.