Gordon Messmer wrote:> On 11/09/2015 09:59 AM, John R Pierce wrote: >> On 11/9/2015 9:50 AM, Gordon Messmer wrote: >>> I don't see the distinction you're making. >> >> a incremental backup copies everything since the last incremental >> a differential copies everything since the last full. > > I guess that makes sense, but in backup systems based on rsync and hard > links (such as rsnapshot), *every* backup on the backup volume is a > "full" backup, so incremental and differential are the same thing. > >> rsync is NOT a backup system, its just a incremental file copy > > ..which can be used as a component of a backup system, such as rsnapshot > or backuppc.Actually, we use rsync for backups. We have a script that creates a new daily directory... and uses hard links to previous dates. That way, it looks like a full b/u... but you can go to a previous date to restore an older version of the file (aka ACK! I saved that file full of garbage to my Great American Novel filename! <g>). And if you aren't familiar with hard links, which rsync happily creates, they were certainly hard enough to wrap my head around, until I got it... and really like them. Just note that they *must* be on one filesystem, as opposed to symlinks, which can cross filesystems. mark
On Mon, 9 Nov 2015 13:42:08 -0500 m.roth at 5-cent.us wrote:> And if you aren't familiar with hard links, which rsync happily creates, > they were certainly hard enough to wrap my head around, until I got it...More than one filename for a particular file. What's difficult about that?> and really like them. Just note that they *must* be on one filesystem, as > opposed to symlinks, which can cross filesystems.Obviously, since a hard link is part of the file and directory structure of the filesystem. -- MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
On Mon, November 9, 2015 12:42 pm, m.roth at 5-cent.us wrote:> Gordon Messmer wrote: >> On 11/09/2015 09:59 AM, John R Pierce wrote: >>> On 11/9/2015 9:50 AM, Gordon Messmer wrote: >>>> I don't see the distinction you're making. >>> >>> a incremental backup copies everything since the last incremental >>> a differential copies everything since the last full. >> >> I guess that makes sense, but in backup systems based on rsync and hard >> links (such as rsnapshot), *every* backup on the backup volume is a >> "full" backup, so incremental and differential are the same thing. >> >>> rsync is NOT a backup system, its just a incremental file copy >> >> ..which can be used as a component of a backup system, such as rsnapshot >> or backuppc. > > Actually, we use rsync for backups. We have a script that creates a new > daily directory... and uses hard links to previous dates. That way, it > looks like a full b/u... but you can go to a previous date to restore an > older version of the file (aka ACK! I saved that file full of garbage to > my Great American Novel filename! <g>).I wonder how filesystem behaves when almost every file has some 400 hard links to it. (thinking in terms of a year worth of daily backups). Valeri> > And if you aren't familiar with hard links, which rsync happily creates, > they were certainly hard enough to wrap my head around, until I got it... > and really like them. Just note that they *must* be on one filesystem, as > opposed to symlinks, which can cross filesystems. > > mark > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 11/09/2015 11:10 AM, Frank Cox wrote:>> >And if you aren't familiar with hard links, which rsync happily creates, >> >they were certainly hard enough to wrap my head around, until I got it... > More than one filename for a particular file. What's difficult about that?I think the difficult part is that so many people don't understand that EVERY regular file is a hard link. It doesn't mean "more than one" at all. A hard link is the association between a directory entry (filename) and an inode in the filesystem.
On 11/09/2015 11:34 AM, Valeri Galtsev wrote:> I wonder how filesystem behaves when almost every file has some 400 hard > links to it. (thinking in terms of a year worth of daily backups).Why do you think that would be a problem? Most inodes have one hard link. When that link is removed, the link count in the inode is decremented (inodes are reference-counted, you can see their ref count in "ls -l" output). When the link count reaches 0 and no open file descriptors exist, the inode is removed. Creating more hard links just increases the ref count. That's it. It's not a weird special case.
On 11/9/2015 11:34 AM, Valeri Galtsev wrote:> I wonder how filesystem behaves when almost every file has some 400 hard > links to it. (thinking in terms of a year worth of daily backups).XFS handles this fine. I have a backuppc storage pool with backups of 27 servers going back a year... now, I just have 30 days of incrementals, and 12 months of fulls, but in backuppc's implementation the distinction between incremental and full is quite blurred as both are fully deduped across the whole pool via use of hard links. * Pool is 5510.40GB comprising 9993293 files and 4369 directories (as of 11/9 02:08), * Pool hashing gives 3452 repeated files with longest chain 54, * Nightly cleanup removed 737 files of size 1.64GB (around 11/9 02:08), * Pool file system was recently at 35% (11/9 11:44), today's max is 35% (11/9 01:00) and yesterday's max was 36%. There are 27 hosts that have been backed up, for a total of: * 441 full backups of total size 71125.43GB (prior to pooling and compression), * 623 incr backups of total size 20775.88GB (prior to pooling and compression). so 90+TB of backups take 5.5TB of actual space. -- john r pierce, recycling bits in santa cruz
Valeri Galtsev wrote:> > On Mon, November 9, 2015 12:42 pm, m.roth at 5-cent.us wrote: >> Gordon Messmer wrote: >>> On 11/09/2015 09:59 AM, John R Pierce wrote: >>>> On 11/9/2015 9:50 AM, Gordon Messmer wrote: >>>>> I don't see the distinction you're making. >>>> >>>> a incremental backup copies everything since the last incremental >>>> a differential copies everything since the last full. >>> >>> I guess that makes sense, but in backup systems based on rsync and hard >>> links (such as rsnapshot), *every* backup on the backup volume is a >>> "full" backup, so incremental and differential are the same thing. >>> >>>> rsync is NOT a backup system, its just a incremental file copy<snip>>> Actually, we use rsync for backups. We have a script that creates a new >> daily directory... and uses hard links to previous dates. That way, it >> looks like a full b/u... but you can go to a previous date to restore an >> older version of the file (aka ACK! I saved that file full of garbage to >> my Great American Novel filename! <g>). > > I wonder how filesystem behaves when almost every file has some 400 hard > links to it. (thinking in terms of a year worth of daily backups).That, I can't answer - what we have is "disaster recovery", not "archive", so we only keep them for no more than five weeks. On the other hand... a reasonable approach would be for, over maybe two months old, to keep the first of the month, and rm everything else for the month. mark