rsync - Jan 2013 - Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

There have been a lot of posts on the list lately about issues with hard
links.  It has been very interesting, but I don't understand it very
thoroughly.  I haven't used hard links for anything yet.  I've used
symlinks - not for backups, of course - and have seen them get broken or
deleted in backups.

Is there a tutorial anywhere that will explain how this works (assuming
that the reader understands the basic concepts of backups and knows how
to program, but doesn't really understand how to use links to create
things like incremental or differential backups)?  It seems like there
are a lot of fine points to consider, some of which can really bite you
if you don't take them into account.

I'm working on my own personal backup system using bash and rsync.  When
it's done it will be pretty good, but it would be *much* better if I
rewrote it to have more backup versions using hard links to save space.

I am writing my own because (aside from learning a lot) I have only seen
two types of backup utilities - those that are very simplistic and won't
let me do what I want and those which are enterprise level and I can't
figure out how to get them to do anything without extensive study.  (I
did experiment with areca (I think it uses rsync libraries under the
hood) which would probably do everything I want, but I got stuck too
many times and couldn't get enough support on their forum to keep going.)

TIA
    Joe

This may help:  (man ln)

A hard link to a file is
indistinguishable from the original directory entry; any changes to a
file are effectively independent of the name used to reference the file.
Hard links may not normally refer to directories and may not span file
systems.

Assuming you do many backups and many of the files do not change, 
hard links are your friend.

Backing up soft links: 
Do you back up the link or what the link points to?
(Even that simple thing has interesting ways to get complicated.)

-----Original Message-----
From: rsync-bounces at lists.samba.org [mailto:rsync-bounces at lists.samba.org]
On Behalf Of Joe
Sent: Tuesday, January 22, 2013 1:32 AM
To: rsync at lists.samba.org
Subject: Is there a howto/tutorial on backups/rsync that covers the use of hard
and soft links?

There have been a lot of posts on the list lately about issues with hard links. 
It has been very interesting, but I don't understand it very thoroughly.  I
haven't used hard links for anything yet.  I've used symlinks - not for
backups, of course - and have seen them get broken or deleted in backups.

Is there a tutorial anywhere that will explain how this works (assuming that the
reader understands the basic concepts of backups and knows how to program, but
doesn't really understand how to use links to create things like incremental
or differential backups)?  It seems like there are a lot of fine points to
consider, some of which can really bite you if you don't take them into
account.

I'm working on my own personal backup system using bash and rsync.  When
it's done it will be pretty good, but it would be *much* better if I rewrote
it to have more backup versions using hard links to save space.

I am writing my own because (aside from learning a lot) I have only seen two
types of backup utilities - those that are very simplistic and won't let me
do what I want and those which are enterprise level and I can't figure out
how to get them to do anything without extensive study.  (I did experiment with
areca (I think it uses rsync libraries under the
hood) which would probably do everything I want, but I got stuck too many times
and couldn't get enough support on their forum to keep going.)

TIA
    Joe
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here is one I wrote up for a LUG presentation that is specifically
about doing it yourself:
http://sanitarium.net/golug/rsync_backups_2010.html

On 01/22/13 02:31, Joe wrote:
> There have been a lot of posts on the list lately about issues with
> hard links.  It has been very interesting, but I don't understand
> it very thoroughly.  I haven't used hard links for anything yet.
> I've used symlinks - not for backups, of course - and have seen
> them get broken or deleted in backups.
> 
> Is there a tutorial anywhere that will explain how this works
> (assuming that the reader understands the basic concepts of backups
> and knows how to program, but doesn't really understand how to use
> links to create things like incremental or differential backups)?
> It seems like there are a lot of fine points to consider, some of
> which can really bite you if you don't take them into account.
> 
> I'm working on my own personal backup system using bash and rsync.
> When it's done it will be pretty good, but it would be *much*
> better if I rewrote it to have more backup versions using hard
> links to save space.
> 
> I am writing my own because (aside from learning a lot) I have only
> seen two types of backup utilities - those that are very simplistic
> and won't let me do what I want and those which are enterprise
> level and I can't figure out how to get them to do anything without
> extensive study.  (I did experiment with areca (I think it uses
> rsync libraries under the hood) which would probably do everything
> I want, but I got stuck too many times and couldn't get enough
> support on their forum to keep going.)
> 
> TIA Joe
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlD+0AEACgkQVKC1jlbQAQdbMACgykxmhD+tZCGGfuoN3WLpWYye
u0oAnji+YSLgaSZWZ1l/F9jHxsR3zV48
=u7q7
-----END PGP SIGNATURE-----