similar to: Dedup on read-only snapshots

Hey list, I wonder if it is possible to deduplicate read-only snapshots. Background: I''m using an bash/rsync script[1] to backup my whole system on a nightly basis to an attached USB3 drive into a scratch area, then take a snapshot of this area. I''d like to have these snapshots immutable, so they should be read-only. Since rsync won''t discover moved files but

Dear All, I have a very slow deduplication going on on an external usb disk. I'm using it for backups - I rsync the relevent files to the disk and then take a snapshot. I then deduplicate with bedup dedup <disk-mount-point> What I am finding is that it is reporting a deduplication between the data on the disk and its snapshot e.g.: Deduplicated: -

Hi, The following series of patches implements in btrfs an ioctl to do out-of-band deduplication of file extents. To be clear, this means that the file system is mounted and running, but the dedupe is not done during file writes, but after the fact when some userspace software initiates a dedupe. The primary patch is loosely based off of one sent by Josef Bacik back in January, 2011.

Hi. I am seriously considering employing btrfs on my systems, particularly due to some space-saving features that it has (namely, deduplication and compression). In fact, I was (a few moments ago) trying to back up some of my systems to a 2TB HD that has an ext4 filesystem and, in the middle of the last one, I got the error message that the backup HD was full. Given that what I backup there are

Hello, some questions regarding btrfs deduplication. - What is the state of it? Is it "safe" to use? https://btrfs.wiki.kernel.org/index.php/Deduplication does not yield much information. - https://pypi.python.org/pypi/bedup says: "bedup looks for new and changed files, making sure that multiple copies of identical files share space on disk. It integrates deeply with btrfs so

Welcome, I''ve been using btrfs for over a 3 months to store my personal data on my NAS server. Almost all interactions with files on the server are done using unison synchronizer. After another use of bedup (https://github.com/g2p/bedup) on my btrfs volume I experienced huge perfomance loss with synchronization. It now takes over 3 hours what have taken only 15 minutes! File

Hello, I was bored this weekend so I hacked up online dedup for Btrfs. It''s working quite well so I think it can be more widely tested. There are two ways to use it 1) Compatible mode - this is a bit slower but will handle being used by older kernels. We use the csum tree to find duplicate blocks. Since it is relatively easy to have crc32c collisions this also involves reading the

Glück Auf! I know its been discussed more then ones, but as a user I really would like to see the patch for allowing this in the kernel. Some users tested this patch successfully for weeks or months in 2 or 3 kernel versions since then, true? I''d say by creating a snapshot, it''s nothing else in the end. More then one file or tree sharing the same data on disc, or am I wrong?

Originally from https://github.com/markfasheh/duperemove/blob/master/btrfs-extent-same.c Signed-off-by: Gabriel de Perthuis <g2p.code+btrfs@gmail.com> --- .gitignore | 1 + Makefile | 2 +- btrfs-extent-same.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+), 1 deletion(-) create mode 100644 btrfs-extent-same.c diff

Hello all: I have two machines with btrfs, that give me the "blocked for more than 120 seconds" message. After that I cannot write anything to disk, i am unable to unmount the btrfs filesystem and i can only reboot with sysrq-trigger. It always happens when i write many files with rsync over network. When i used 3.2rc6 it happened randomly on both machines after 50-500gb of writes.

Hi all, I have two multi-disk btrfs filesystems on a Arch linux 3.4.0 system. After a power failure, both filesystems refuse to mount [ 10.402284] Btrfs loaded [ 10.402714] device fsid 1e7c18a4-02d6-44b1-8eaf-c01378009cd3 devid 4 transid 65282 /dev/sdc [ 10.403108] btrfs: force zlib compression [ 10.403130] btrfs: enabling inode map caching [ 10.403152] btrfs: disk space caching is

Dear All : Under Sun Storage 7000 system, can we see per share ratio after enable dedup function ? We would like deep to see each share dedup ratio. On Web GUI, only show dedup ratio entire storage pool. Thanks a lot, -- Rex -- This message posted from opensolaris.org

This article raises the concern that SSD controllers (in particular SandForce) do internal dedup, and in particular that this could defeat ditto-block style replication of critical metadata as done by filesystems including ZFS. http://storagemojo.com/2011/06/27/de-dup-too-much-of-good-thing/ Along with discussion of risk evaluation, it also suggests that filesystems could vary each copy in some

Greetings Gentlemen, I''m currently testing a new setup for a ZFS based storage system with dedup enabled. The system is setup on OI 148, which seems quite stable w/ dedup enabled (compared to the OpenSolaris snv_136 build I used before). One issue I ran into, however, is quite baffling: With iozone set to 32 threads, ZFS''s ARC seems to consume all available memory, making

Is there an easy way to find out what datasets have dedup''d data in them. Even better would be to discover which files in a particular dataset are dedup''d. I ran # zdb -DDDD which gave output like: index 1055c9f21af63 refcnt 2 single DVA[0]=<0:1e274ec3000:2ac00:STD:1> [L0 deduplicated block] sha256 uncompressed LE contiguous unique unencrypted 1-copy size=20000L/20000P

I'm new to spatial analysis and am exploring numerous packages, mostly enjoying sp, gstat, and spBayes. Is there a function that allows the user to dedup a data set with multiple values at the same coordinates and replace those duplicated values with the mean at those coordinates? I've written some cumbersome code that works, but would prefer an efficient R function if it exists.

Hello, I have a 5.5TB Btrfs filesystem on top of a md-raid 5 device. Now if i run some file operations like find, i get these messages. kernel is 2.6.38.5-1 on arch linux May 5 14:15:12 mail kernel: [13559.089713] parent transid verify failed on 3062073683968 wanted 5181 found 5188 May 5 14:15:12 mail kernel: [13559.089834] parent transid verify failed on 3062073683968 wanted 5181 found 5188

Hi all Having managed ZFS for about two years, I want to post a wishlist. INCLUDED IN ZFS - Mirror existing single-drive filesystem, as in ''zfs attach'' - RAIDz-stuff - single and hopefully multiple-parity RAID configuration with block-level checksumming - Background scrub/fsck - Pool-like management with multiple RAIDs/mirrors (VDEVs) - Autogrow as in ZFS autoexpand NOT

I tried the deduplication feature but the performance of my fileserver dived from writing 50MB/s via CIFS to 4MB/s. what happens to the deduped blocks when you set dedup=off? are they written back to disk? is the deduptable deleted or is it still there? thanks -- This message posted from opensolaris.org

We want ''btrfs subvolume list'' only to list readonly subvolumes, this patch set introduces a new option ''r'' to implement it. You can use the command like that: btrfs subvolume list -r <mnt> Changelog v2 -> v3: - re-implement list_subvols() - re-implement this read-only subvolume list function based on the new list_subvols() Changelog v1 -> v2: -