similar to: Use rsync's checksums to deduplicate across backups

Hi list, is there any working solution for deduplication of data for centos? We are trying to find a solution for our backup server which runs a bash script invoking xdelta(3). But having this functionality in fs is much more friendly... We have looked into lessfs, sdfs and ddar. Are these filesystems ready to use (on centos)? ddar is sthg different, I know. Thx Rainer

Hi All: We have been looking at implementing deduplication on a backup server. >From what I have been able to find the available documentation is pretty thin. I ended up trying to install LessFS on this CentOS 5.7 box but we have now encountered problems with fuse version. Has anyone out there been able to get LessFS running on CentOS 5.7 and can provide some pointers? If not LessFS can you

Hi, I'm currently running GlusterFS over XFS, and it works quite well. I'm wondering if it's possible to add data deduplication into the mix by: glusterfs --> lessfs --> xfs or glusterfs --> opendedupe --> xfs Has anybody tried doing this? We're running VM images on gluster, and I figure we could get a bit of space saving bu deduplicating the data. Gerald

My full backup script errorred out the last two times I ran it. I''ve got a full Bash trace of it, so I know exactly what was done. There are a moderate number of snapshots on the zp1 pool, and I''m intending to replicate the whole thing into the backup pool. After housekeeping, I take make a current snapshot on the data pool (zp1). Since this is a new full backup, I then

Hi, I have a gluster 3.10 volume with a dir with ~1 million small files in them, say mounted at /mnt/dir with FUSE, and I'm observing something weird: When I list and stat them all using rsync, then the lstat() calls that rsync does are incredibly fast (23 microseconds per call on average, definitely faster than a network roundtrip between my 3-machine bricks connected via Ethernet). But

First, it fails because the destination directory doesn''t exist. Then it fails because it DOES exist. I really expected one of those to work. So, what am I confused about now? (Running 2008.11) # zpool import -R /backups/bup-ruin bup-ruin # zfs send -R "zp1 at bup-20090222-054457UTC" | zfs receive -dv bup-ruin/fsfs/zp1" cannot receive: specified fs (bup-ruin/fsfs/zp1)

So, I was running my full backup last night, backing up my main data pool zp1, and it seems to have hung. Any suggestions for additional data gathering? -bash-3.2$ zpool status zp1 pool: zp1 state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using ''zpool

I have set up the configurations and profiles for some of my programs so that I can use them across distros, and easily configure new installations with all my old settings. For instance, if I use Firefox in Fedora, add some bookmarks, then go to suse or Mandriva the new bookmarks are there, as is history, etc. Same for Thunderbird. To do this I copy the relevant folder from /home/user to my

At the risk of ridicule for my deficient linear algebra skills, I ask for help using the solve.QP function to do portfolio optimization. I am trying to following a textbook example and need help converting the problem into the format required by solve.QP. Below is my sample code if anyone is willing to go through it. This problem will not solve because it is not set up properly. I hope I

On 15/09/17 02:45, Sam McLeod wrote: > Out of interest have you tried testing performance > with performance.stat-prefetch enabled? Not yet, because I'm still struggling to understand the current more basic setup's performance behaviour (with it being off), but it's definitely on my list and I'll report the outcome.

Here is a short proposal for the hybrid storage cache idea with introduction/motivation and a bird''s eye view of an approach to implement a hybrid storage cache for btrfs. Please note that there is currently no available patches. We would like to get as much input before as possible before we start designing and implementing a solution. 1. Introduction The emerge of Solid State

I did a quick test on one of my lab clusters with no tuning except for quota being enabled: [root at dell-per730-03 ~]# gluster v info Volume Name: vmstore Type: Replicate Volume ID: 0d2e4c49-334b-47c9-8e72-86a4c040a7bd Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: 192.168.50.1:/rhgs/brick1/vmstore Brick2:

Folks I'm looking for a solution for backups because ZFS has failed on me too many times. In my environment, I have a large amount of data (around 2tb) that I periodically back up. I keep the last 5 "snapshots". I use rsync so that when I overwrite the oldest backup, most of the data is already there and the backup completes quickly, because only a small number of files have

On 17/09/17 18:03, Niklas Hamb?chen wrote: > So far the only difference between `ls` and `bup index` I could observe > is that `bup index` chdir()s into the directory to index, ls doesn't. > > But when I `cd` into the dir and run `ls` without directory argument, it > is still much faster than bup index for each stat(). Hmm, bup uses the fchdir() syscall to go into the target

Hi, I feel that i'm having a small problem with my upsmon configuration (maybe even simply a permissions problem) since I upgrade today to 2.0.2 I have apcsmart set up and talk to my ups no problem (upsc get answer). I have upsd running no problem! But when I try to start upsmon, it's unable to talk to my upsd. Having try a few option I found that saying RUN_AS_USER=root makes my upsmon

On 15/09/17 03:46, Niklas Hamb?chen wrote: >> Out of interest have you tried testing performance >> with performance.stat-prefetch enabled? I have now tested with `performance.stat-prefetch: on` but am not observing a difference. So far the only difference between `ls` and `bup index` I could observe is that `bup index` chdir()s into the directory to index, ls doesn't. But when

Hi all ! I am currently trying to find a smart way to long-term-backup my data. I currently do link-dest-updates to an external hdd. However, I am concerned about data corruption in the backup or the original, i.e. corruption of the data inside the files. Thus i am wondering if there is a way to check the integrity of the backups on the external disk. i.e. to avoid data degradation for

Hi, Jernej Simon?i? wrote: > As long as you have cygwin with development tools installed, it's just > ./configure && make. autotools magic. I myself have few clue of contemporary MS-Windows. > https://eternallybored.org/misc/xorriso/xorriso-1.4.2-cygwin32.zip Thank you for flying xorriso. > > > oscdimg.exe -a -m -o -j1 > > >

Hi everyone! I played around with rsync sources a little and wrote a small patch that computes the checksums from parts of the files only. I'm just writing to ask if the rsync developers would have any interest in the sort of functionality described below. If you do, I'm willing to work with you to produce a cleaned up patch for git. For background: This started as a way to satisfy an

On 02/09/2017 01:11 PM, Leonard den Ottolander wrote: > On Thu, 2017-02-09 at 12:58 -0600, Johnny Hughes wrote: >> At the time of extraction, the <name>.metadata file is created (again, >> not by us, but by the Red Hat team that distributes source), and all the >> non-text sha1sums are in there as well as all the text sources. > > Aha, <name>.metadata, well,