Hi list, is there any working solution for deduplication of data for centos? We are trying to find a solution for our backup server which runs a bash script invoking xdelta(3). But having this functionality in fs is much more friendly... We have looked into lessfs, sdfs and ddar. Are these filesystems ready to use (on centos)? ddar is sthg different, I know. Thx Rainer
From: Rainer Traut <tr.ml at gmx.de>> is there any working solution for deduplication of data for centos? > We are trying to find a solution for our backup server which runs a bash > script invoking xdelta(3). But having this functionality in fs is much > more friendly... > > We have looked into lessfs, sdfs and ddar. > Are these filesystems ready to use (on centos)? > ddar is sthg different, I know.Never tried but what about zfs? JD
On 08/27/12 4:55 AM, Rainer Traut wrote:> is there any working solution for deduplication of data for centos? > We are trying to find a solution for our backup server which runs a bash > script invoking xdelta(3). But having this functionality in fs is much > more friendly...BackupPC does exactly this. its not a generalized solution to deduplication of a file system, instead, its a backup system, designed to backup multiple targets, that implements deduplication on the backup tree it maintains. -- john r pierce N 37, W 122 santa cruz ca mid-left coast
On Mon, Aug 27, 2012 at 6:55 AM, Rainer Traut <tr.ml at gmx.de> wrote:> > is there any working solution for deduplication of data for centos? > We are trying to find a solution for our backup server which runs a bash > script invoking xdelta(3). But having this functionality in fs is much > more friendly... >Below forwarded on behalf of mroth: Les, A favor, please? Could you post this for me? Spamhouse is bouncing me again, this time because *they* have a bug (see below). I tried asking Karanbir, but I guess he's not online yet.... Thanks in advance. John R Pierce wrote:> On 08/27/12 4:55 AM, Rainer Traut wrote: >> is there any working solution for deduplication of data for centos? Weare trying to find a solution for our backup server which runs a bash script invoking xdelta(3). But having this functionality in fs is much more friendly...> > BackupPC does exactly this. its not a generalized solution todeduplication of a file system, instead, its a backup system, designed to backup multiple targets, that implements deduplication on the backup tree it maintains. I've tried, twice, to suggest that a workaround that doesn't involve a new, and possibly experimental f/s would be to use rsync with hard links, which is what we do. There's no way we have enough disk space for 5 weeks of terabytes of data.... However, the reason I haven't been able to suggest it is that I'm being blocked by spamhost. And when I go there, it asserts I'm listed in the CBL. And when I go *THERE*, it tells me I'm not. Oh, and now, when I try to go to the CBL, it's down. I don't suppose the CentOS list has a whitelist.... mark
----- Original Message -----> From: "Rainer Traut" <tr.ml at gmx.de> > To: centos at centos.org > Sent: Monday, August 27, 2012 4:55:03 AM > Subject: [CentOS] Deduplication data for CentOS? > > Hi list, > > is there any working solution for deduplication of data for centos? > We are trying to find a solution for our backup server which runs a > bash > script invoking xdelta(3). But having this functionality in fs is > much > more friendly... > > We have looked into lessfs, sdfs and ddar. > Are these filesystems ready to use (on centos)? > ddar is sthg different, I know. > > Thx > RainerAlthough not open source, CrashplanPROe only costs $365 for a perpetual five client license. I use it to backup some of my Linux boxes. It has very good deduplication, compression, and encryption. For example I have 1.7TB of data on one linux system and another system that has 1.5TB. I NFS mount one of the systems to another and only use one Crashplan client to backup both data sets to a single backup archive. The backup archive is only 1.2TB and that also spans 90 days worth of file modification and deletion I can recover. David.
On Mon, Aug 27, 2012 at 7:55 AM, Rainer Traut <tr.ml at gmx.de> wrote:> Hi list, > > is there any working solution for deduplication of data for centos? > We are trying to find a solution for our backup server which runs a bash > script invoking xdelta(3). But having this functionality in fs is much > more friendly... > > We have looked into lessfs, sdfs and ddar. > Are these filesystems ready to use (on centos)? > ddar is sthg different, I know. > > Thx > RainerThis is something I have been thinking about peripherally for a while now. What are your impressions of SDFS (OpenDedupe)? I had been hoping it would be pretty good. Any issues with it on CentOS? ? Brian Mathis
Sorry for the top posting. Dedup is just a hype. After a while the table that manage the deduped data will be just too big. Don't use it for long term. Sent from Samsung Galaxy ^^
Rainer Traut <tr.ml at ...> writes:> > Hi list, > > is there any working solution for deduplication of data for centos? > We are trying to find a solution for our backup server which runs a bash > script invoking xdelta(3). But having this functionality in fs is much > more friendly... > > We have looked into lessfs, sdfs and ddar. > Are these filesystems ready to use (on centos)? > ddar is sthg different, I know. > > Thx > Rainer >Not sure if it's already been mentioned but storeBackup uses rsync and hardlinks to minimise storage - and it break up big files and backs up the fragments separately. May help ... http://www.nongnu.org/storebackup/en/node2.html