rsync - Jun 2014 - [Bug 10637] rsync --link-dest should break hard links when encountering "Too many links"

samba-bugs at samba.org wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=10637
>
> --- Comment #1 from Karl O. Pinc <kop at meme.com> 2014-05-28
19:05:04 UTC ---
> Yum is also rsync happy.  That's where our --link-dest backups always
break due
> to too many hard links.
----------

What would be "too many"? -- a few million?  I have files in a test
setup that have over 7000 files hard linked (only an 8 byte file, but since
minimum block size is 4K, that would be 28MB w/o the hard link v. 4k with.

It would NOT be good for rsync to start breaking
links w/standard options.  Maybe a new option
to allow link breaking?

On 06/06/2014 12:08:46 AM, L. A. Walsh wrote:
> samba-bugs at samba.org wrote:
> 
> > https://bugzilla.samba.org/show_bug.cgi?id=10637
> >
> > --- Comment #1 from Karl O. Pinc <kop at meme.com> 2014-05-28
19:05:04
> UTC ---
> > Yum is also rsync happy.  That's where our --link-dest backups
> always break due
> > to too many hard links.
> 
> ----------
> 
> What would be "too many"? -- a few million?  I have files in a
test
> setup that have over 7000 files hard linked (only an 8 byte file, but
> since
> minimum block size is 4K, that would be 28MB w/o the hard link v. 4k
> with.
Too many would be when the link(2) call returns EMLINK
because your filesystem has hit it's hard link limit.

The ext2/3 limit is 32,000, the ext4 limit is twice that.
(EXT3_LINK_MAX, EXT4_LINK_MAX)

This is low enough that I can't fill a 4TB drive with
--link-dest backups without EMLINK error.
> 
> It would NOT be good for rsync to start breaking
> links w/standard options.  Maybe a new option
> to allow link breaking?
Rsync has no choice.  If you hit the filesystem link
limit then you can't make more hardlinks.  The only
question is what link to break.  The proposal is
to break the --link-dest links.  Currently rsync
instead almost always breaks the -H links.  Because
there's only 1 --link-dest link but almost always
many -H links you're simply more likely to run
out when linking -H links.  If you're preserving
hard links you probably really want an accurate
copy of your hard links and would much rather,
since you're forced to sacrifice something,
forgo the --link-dest for that particular file.

It would be nice to have an option, after this
proposal is implemented, to control whether rsync
reports an error when breaking --link-dest links.
It might even be nice to have the default be to _not_
report errors when breaking --link-dest links.

However, once started down the path of controlling
which errors rsync reports and which it does not,
it's probably better to implement an option that
controls/filters error reporting in general instead
of having separate options for each kind of error
that rsync might report.

Regards,

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein