rsync - Dec 2018 - Aw: Re: rsync remote raw block device with --inplace

> There have been addons to rsync in the past to do that but rsync really
> isn't the correct tool for the job.
why not correct tool ?

if rsync can greatly keep two large files in sync between source and destination
(using --inplace), why should it (generally spoken) not also be used to keep two
blockdevices in sync ? 

maybe these links are interesting in that context:

https://lists.samba.org/archive/rsync/2010-June/025164.html

https://github.com/dop251/diskrsync

roland
> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
> Von: "Kevin Korb via rsync" <rsync at lists.samba.org>
> An: rsync at lists.samba.org
> Betreff: Re: rsync remote raw block device with --inplace
>
> There have been addons to rsync in the past to do that but rsync really
> isn't the correct tool for the job.  Neither is dd.
> 
> The right tool is something that understands the filesystem within the
> block device such as ntfsclone (what I use) or partimage (if you have
> ever used Clonezilla this is what it uses).  These will know how to skip
> all the empty parts of the filesystem and will still be capable of
> restoring a complete image in a bare metal restore.  You can still use
> dd to snag a copy of the MBR since that is outside of any filesystems.
> 
> Also, if you do have to resort to a plain image use ddrescue instead of
> dd.  It has a status screen and it can resume as long as you used a log
> file when you ran it.
> 
> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
> > It would be very nice to be able to rsync the raw data content of,
e.g.,
> > a non-mounted disk partition, particularly in combination with
--inplace.
> > 
> > Our reality: several dual-boot machines running Windows during the day
> > and Linux at night, during backups.  Windows is very tedious and iffy
to
> > re-reinstall without a raw disk image to start from.  Disks fail, and
> > the ensuing downtime must be minimized. 
> > 
> > We're using dd for this.  Most of the nightly work is redundant
and
> > wasteful of elapsed time and storage.  Storage is cheap, but it's
not
> > *that* cheap.  Elapsed time is priceless.
> > 
> > Rsync refuses to back up raw devices, and even raw character devices,
> > with the message "skipping non-regular file" (I think the
relevant
> > message is in generator.c). 
> > 
> > In Linux, anyway, the "raw" command allows a block device to
be bound as
> > a character device, and then even a "cat" command can read
the raw data
> > of the block device.  So why does rsync refuse to copy such content,
or
> > why is it a bad idea, or what rsync doctrine conflicts with it?  I
agree
> > there are security concerns here, but rsync already disallows some of
> > its functions unless the super user is requesting them. 
> > 
> > 
> 
> -- 
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			https://sanitarium.net/
> 	PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync would only benefit you in terms of reducing network bandwidth
(assuming it is even networking).  The resulting file will be the size
of the block device just like you get with dd[rescue].  You can tell
rsync to write the file sparsely which would mean that any parts of the
disk that have never been written to would probably still contain nulls
and would not be allocated but you will still be copying any deleted
file data that still exists.

Ntfsclone and partimage understand the filesystem.  They will just skip
the blocks that don't contain any current data.  This will keep your
backup file as small as possible.  If you ever have to do a restore they
will put all the file data back where it was and either skip the parts
they didn't save or fill them with nulls.

On 12/30/18 3:50 PM, devzero at web.de wrote:
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.
> 
> why not correct tool ?
> 
> if rsync can greatly keep two large files in sync between source and
destination
> (using --inplace), why should it (generally spoken) not also be used to
keep two
> blockdevices in sync ? 
> 
> maybe these links are interesting in that context:
> 
> https://lists.samba.org/archive/rsync/2010-June/025164.html
> 
> https://github.com/dop251/diskrsync
> 
> roland
> 
>> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
>> Von: "Kevin Korb via rsync" <rsync at lists.samba.org>
>> An: rsync at lists.samba.org
>> Betreff: Re: rsync remote raw block device with --inplace
>>
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.  Neither is dd.
>>
>> The right tool is something that understands the filesystem within the
>> block device such as ntfsclone (what I use) or partimage (if you have
>> ever used Clonezilla this is what it uses).  These will know how to
skip
>> all the empty parts of the filesystem and will still be capable of
>> restoring a complete image in a bare metal restore.  You can still use
>> dd to snag a copy of the MBR since that is outside of any filesystems.
>>
>> Also, if you do have to resort to a plain image use ddrescue instead of
>> dd.  It has a status screen and it can resume as long as you used a log
>> file when you ran it.
>>
>> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
>>> It would be very nice to be able to rsync the raw data content of,
e.g.,
>>> a non-mounted disk partition, particularly in combination with
--inplace.
>>>
>>> Our reality: several dual-boot machines running Windows during the
day
>>> and Linux at night, during backups.  Windows is very tedious and
iffy to
>>> re-reinstall without a raw disk image to start from.  Disks fail,
and
>>> the ensuing downtime must be minimized. 
>>>
>>> We're using dd for this.  Most of the nightly work is redundant
and
>>> wasteful of elapsed time and storage.  Storage is cheap, but
it's not
>>> *that* cheap.  Elapsed time is priceless.
>>>
>>> Rsync refuses to back up raw devices, and even raw character
devices,
>>> with the message "skipping non-regular file" (I think the
relevant
>>> message is in generator.c). 
>>>
>>> In Linux, anyway, the "raw" command allows a block device
to be bound as
>>> a character device, and then even a "cat" command can
read the raw data
>>> of the block device.  So why does rsync refuse to copy such
content, or
>>> why is it a bad idea, or what rsync doctrine conflicts with it?  I
agree
>>> there are security concerns here, but rsync already disallows some
of
>>> its functions unless the super user is requesting them. 
>>>
>>>
>>
>> -- 
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> 	Kevin Korb			Phone:    (407) 252-6853
>> 	Systems Administrator		Internet:
>> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
>> 	Orlando, Florida		kmk at sanitarium.net (personal)
>> 	Web page:			https://sanitarium.net/
>> 	PGP public key available on web site.
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>> -- 
>> Please use reply-all for most replies to avoid omitting the mailing
list.
>> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20181230/0b6b02ca/signature.sig>

It was broucht up before indeed: 
https://lists.samba.org/archive/rsync/2012-June/027680.html

On 12/30/18 9:50 PM, devzero--- via rsync wrote:
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.
> why not correct tool ?
>
> if rsync can greatly keep two large files in sync between source and
destination
> (using --inplace), why should it (generally spoken) not also be used to
keep two
> blockdevices in sync ?
>
> maybe these links are interesting in that context:
>
> https://lists.samba.org/archive/rsync/2010-June/025164.html
>
> https://github.com/dop251/diskrsync
>
> roland
>
>> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
>> Von: "Kevin Korb via rsync" <rsync at lists.samba.org>
>> An: rsync at lists.samba.org
>> Betreff: Re: rsync remote raw block device with --inplace
>>
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.  Neither is dd.
>>
>> The right tool is something that understands the filesystem within the
>> block device such as ntfsclone (what I use) or partimage (if you have
>> ever used Clonezilla this is what it uses).  These will know how to
skip
>> all the empty parts of the filesystem and will still be capable of
>> restoring a complete image in a bare metal restore.  You can still use
>> dd to snag a copy of the MBR since that is outside of any filesystems.
>>
>> Also, if you do have to resort to a plain image use ddrescue instead of
>> dd.  It has a status screen and it can resume as long as you used a log
>> file when you ran it.
>>
>> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
>>> It would be very nice to be able to rsync the raw data content of,
e.g.,
>>> a non-mounted disk partition, particularly in combination with
--inplace.
>>>
>>> Our reality: several dual-boot machines running Windows during the
day
>>> and Linux at night, during backups.  Windows is very tedious and
iffy to
>>> re-reinstall without a raw disk image to start from.  Disks fail,
and
>>> the ensuing downtime must be minimized.
>>>
>>> We're using dd for this.  Most of the nightly work is redundant
and
>>> wasteful of elapsed time and storage.  Storage is cheap, but
it's not
>>> *that* cheap.  Elapsed time is priceless.
>>>
>>> Rsync refuses to back up raw devices, and even raw character
devices,
>>> with the message "skipping non-regular file" (I think the
relevant
>>> message is in generator.c).
>>>
>>> In Linux, anyway, the "raw" command allows a block device
to be bound as
>>> a character device, and then even a "cat" command can
read the raw data
>>> of the block device.  So why does rsync refuse to copy such
content, or
>>> why is it a bad idea, or what rsync doctrine conflicts with it?  I
agree
>>> there are security concerns here, but rsync already disallows some
of
>>> its functions unless the super user is requesting them.
>>>
>>>
>> -- 
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> 	Kevin Korb			Phone:    (407) 252-6853
>> 	Systems Administrator		Internet:
>> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
>> 	Orlando, Florida		kmk at sanitarium.net (personal)
>> 	Web page:			https://sanitarium.net/
>> 	PGP public key available on web site.
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>> -- 
>> Please use reply-all for most replies to avoid omitting the mailing
list.
>> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html

maybe this could also be useful:

https://github.com/RyanHow/block2file

> Gesendet: Sonntag, 30. Dezember 2018 um 22:52 Uhr
> Von: "Rolf Fokkens via rsync" <rsync at lists.samba.org>
> An: rsync at lists.samba.org
> Betreff: Re: Aw: Re: rsync remote raw block device with --inplace
>
> It was broucht up before indeed: 
> https://lists.samba.org/archive/rsync/2012-June/027680.html
> 
> On 12/30/18 9:50 PM, devzero--- via rsync wrote:
> >> There have been addons to rsync in the past to do that but rsync
really
> >> isn't the correct tool for the job.
> > why not correct tool ?
> >
> > if rsync can greatly keep two large files in sync between source and
destination
> > (using --inplace), why should it (generally spoken) not also be used
to keep two
> > blockdevices in sync ?
> >
> > maybe these links are interesting in that context:
> >
> > https://lists.samba.org/archive/rsync/2010-June/025164.html
> >
> > https://github.com/dop251/diskrsync
> >
> > roland
> >
> >> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
> >> Von: "Kevin Korb via rsync" <rsync at
lists.samba.org>
> >> An: rsync at lists.samba.org
> >> Betreff: Re: rsync remote raw block device with --inplace
> >>
> >> There have been addons to rsync in the past to do that but rsync
really
> >> isn't the correct tool for the job.  Neither is dd.
> >>
> >> The right tool is something that understands the filesystem within
the
> >> block device such as ntfsclone (what I use) or partimage (if you
have
> >> ever used Clonezilla this is what it uses).  These will know how
to skip
> >> all the empty parts of the filesystem and will still be capable of
> >> restoring a complete image in a bare metal restore.  You can still
use
> >> dd to snag a copy of the MBR since that is outside of any
filesystems.
> >>
> >> Also, if you do have to resort to a plain image use ddrescue
instead of
> >> dd.  It has a status screen and it can resume as long as you used
a log
> >> file when you ran it.
> >>
> >> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
> >>> It would be very nice to be able to rsync the raw data content
of, e.g.,
> >>> a non-mounted disk partition, particularly in combination with
--inplace.
> >>>
> >>> Our reality: several dual-boot machines running Windows during
the day
> >>> and Linux at night, during backups.  Windows is very tedious
and iffy to
> >>> re-reinstall without a raw disk image to start from.  Disks
fail, and
> >>> the ensuing downtime must be minimized.
> >>>
> >>> We're using dd for this.  Most of the nightly work is
redundant and
> >>> wasteful of elapsed time and storage.  Storage is cheap, but
it's not
> >>> *that* cheap.  Elapsed time is priceless.
> >>>
> >>> Rsync refuses to back up raw devices, and even raw character
devices,
> >>> with the message "skipping non-regular file" (I
think the relevant
> >>> message is in generator.c).
> >>>
> >>> In Linux, anyway, the "raw" command allows a block
device to be bound as
> >>> a character device, and then even a "cat" command
can read the raw data
> >>> of the block device.  So why does rsync refuse to copy such
content, or
> >>> why is it a bad idea, or what rsync doctrine conflicts with
it?  I agree
> >>> there are security concerns here, but rsync already disallows
some of
> >>> its functions unless the super user is requesting them.
> >>>
> >>>
> >> -- 
> >>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >> 	Kevin Korb			Phone:    (407) 252-6853
> >> 	Systems Administrator		Internet:
> >> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> >> 	Orlando, Florida		kmk at sanitarium.net (personal)
> >> 	Web page:			https://sanitarium.net/
> >> 	PGP public key available on web site.
> >>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >>
> >> -- 
> >> Please use reply-all for most replies to avoid omitting the
mailing list.
> >> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> >> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html