rsync - Dec 2020 - Is there any way to restore/create hardlinks lost in incremental backups?

I run a simple self written incremental backup system using rsync's
--link-dest option.

Occasionally, because I've moved things around or because I've done
something else that breaks things, the hard links aren't created as
they should be and I get a very space consuming backup increment.

Is there any easy way that one can restore hard links in the *middle*
of a series?  For example say I have:-

    day1/pictures
    day2/pictures
    day3/pictures
    day4/pictures
    day5/pictures

and I notice that day4/pictures is using as much space as
day1/pictures but all the others are relatively small, i.e.
day2 day3 and day5 have correctly hard linked to the previous day but
day4 hasn't.

It needs a tool that can scan day4, check a file is identical with the
one in day3 then hardlink it without losing the link from day5.

There's jdupes but that does lose the link from day5 so you'd have to
apply it to all the directories after the one that's lost the links.



-- 
Chris Green
?

Hi.

Is it possible that, if day4 is consuming too much space, that day3 was an
incomplete backup?

The rsync wrapper I wrote goes to a little trouble to make sure that
incomplete backups aren't allowed.  It's called Backup.rsync, and can be
found at:
https://stromberg.dnsalias.org/~strombrg/Backup.remote.html
It does this by mv'ing backups to a magic name scheme only after they fully
finish, to distinguish them from partial backups.  If a backup is found
that doesn't have that magic name scheme, it is assumed to be partia, and
is reused as the starting point for the next snapshot.

Feel free to use it, or raid it for ideas.

HTH


On Thu, Dec 10, 2020 at 9:29 AM Chris Green via rsync <rsync at
lists.samba.org>
wrote:
> I run a simple self written incremental backup system using rsync's
> --link-dest option.
>
> Occasionally, because I've moved things around or because I've done
> something else that breaks things, the hard links aren't created as
> they should be and I get a very space consuming backup increment.
>
> Is there any easy way that one can restore hard links in the *middle*
> of a series?  For example say I have:-
>
>     day1/pictures
>     day2/pictures
>     day3/pictures
>     day4/pictures
>     day5/pictures
>
> and I notice that day4/pictures is using as much space as
> day1/pictures but all the others are relatively small, i.e.
> day2 day3 and day5 have correctly hard linked to the previous day but
> day4 hasn't.
>
> It needs a tool that can scan day4, check a file is identical with the
> one in day3 then hardlink it without losing the link from day5.
>
> There's jdupes but that does lose the link from day5 so you'd have
to
> apply it to all the directories after the one that's lost the links.
>
>
>
> --
> Chris Green
> ?
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20201210/1864d98a/attachment.htm>

On Thu 10 Dec 2020, Chris Green via rsync wrote:
> 
> Occasionally, because I've moved things around or because I've done
> something else that breaks things, the hard links aren't created as
> they should be and I get a very space consuming backup increment.
> 
> Is there any easy way that one can restore hard links in the *middle*
> of a series?  For example say I have:-
> 
>     day1/pictures
>     day2/pictures
>     day3/pictures
>     day4/pictures
>     day5/pictures
> 
> and I notice that day4/pictures is using as much space as
> day1/pictures but all the others are relatively small, i.e.
> day2 day3 and day5 have correctly hard linked to the previous day but
> day4 hasn't.
> 
> It needs a tool that can scan day4, check a file is identical with the
> one in day3 then hardlink it without losing the link from day5.
If you have these files that are hardlinked:

    day1/pictures/1.jpg
    day2/pictures/1.jpg
    day3/pictures/1.jpg

And these are hardlinked, but to a different inode:

    day4/pictures/1.jpg
    day5/pictures/1.jpg

then there is no way of linking the second group to the first in one
step; you will have to individually link day3/pictures/1.jpg to
day4/pictures/1.jpg and then day3/pictures/1.jpg (or
day4/pictures/1.jpg) to day5/pictures/1.jpg.

It's not like a group of directory entries that are hardlinked to one
inode are some sort of actual group; they just happen to be directory
entries that point to the same inode number. There is no other relation
between those directory entries.

So you will have to incrementally process each next day against the
previous day.


If I make a significant change in such a directory structure (e.g.
renaming a directory) I try to remember to do the same thing on the
backup which some say is wrong, but it saves a lot of space, like you
discovered :)


Paul

You could rsync the current day4 dir to a day4.new dir, and list all the
prior days as --link-dest options. Make sure that you're using the same
xatt/acl options as your official backup command (the options may or may
not be present) so that you are preserving the same level of info as the
backup.  You also have the choice of copying the whole day4 dir or just the
day4/pictures dir, as you see fit.

For example:

rsync -aiv --link-dest=../day1 --link-dest=../day2
--link-dest=../day3 day4/ day4.new/
mv day4 day4.bad
mv day4.new day4

If you only want to reprocess the pictures subdir, just tweak the
"day4/"
arg to be "day4/pictures" (no trailing slash) and change the mv
commands to
deal with just that subdir.

..wayne..


On Thu, Dec 10, 2020 at 9:29 AM Chris Green via rsync <rsync at
lists.samba.org>
wrote:
> I run a simple self written incremental backup system using rsync's
> --link-dest option.
>
> Occasionally, because I've moved things around or because I've done
> something else that breaks things, the hard links aren't created as
> they should be and I get a very space consuming backup increment.
>
> Is there any easy way that one can restore hard links in the *middle*
> of a series?  For example say I have:-
>
>     day1/pictures
>     day2/pictures
>     day3/pictures
>     day4/pictures
>     day5/pictures
>
> and I notice that day4/pictures is using as much space as
> day1/pictures but all the others are relatively small, i.e.
> day2 day3 and day5 have correctly hard linked to the previous day but
> day4 hasn't.
>
> It needs a tool that can scan day4, check a file is identical with the
> one in day3 then hardlink it without losing the link from day5.
>
> There's jdupes but that does lose the link from day5 so you'd have
to
> apply it to all the directories after the one that's lost the links.
>
>
>
> --
> Chris Green
> ?
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20201213/cc0b75b2/attachment.htm>