similar to: Deduplication

Hey list, I wonder if it is possible to deduplicate read-only snapshots. Background: I''m using an bash/rsync script[1] to backup my whole system on a nightly basis to an attached USB3 drive into a scratch area, then take a snapshot of this area. I''d like to have these snapshots immutable, so they should be read-only. Since rsync won''t discover moved files but

Welcome, I''ve been using btrfs for over a 3 months to store my personal data on my NAS server. Almost all interactions with files on the server are done using unison synchronizer. After another use of bedup (https://github.com/g2p/bedup) on my btrfs volume I experienced huge perfomance loss with synchronization. It now takes over 3 hours what have taken only 15 minutes! File

Hi. I am seriously considering employing btrfs on my systems, particularly due to some space-saving features that it has (namely, deduplication and compression). In fact, I was (a few moments ago) trying to back up some of my systems to a 2TB HD that has an ext4 filesystem and, in the middle of the last one, I got the error message that the backup HD was full. Given that what I backup there are

Dear All, I have a very slow deduplication going on on an external usb disk. I'm using it for backups - I rsync the relevent files to the disk and then take a snapshot. I then deduplicate with bedup dedup <disk-mount-point> What I am finding is that it is reporting a deduplication between the data on the disk and its snapshot e.g.: Deduplicated: -

In reading this blog post: http://blogs.sun.com/bobn/entry/taking_zfs_deduplication_for_a a question came to mind..... To understand the context of the question, consider the opening paragraph from the above post; Here is my test case: I have 2 directories of photos, totaling about 90MB > each. And here''s the trick - they are almost complete duplicates of each > other. I downloaded

Hello; Dedup on ZFS is an absolutely wonderful feature! Is there a way to conduct dedup replication across boxes from one dedup ZFS data set to another? Warmest Regards Steven Sim

I haven''t seen any mention of it in this forum yet, so FWIW you might be interested in the details of ZFS deduplication mentioned in this recently-filed case. Case log: http://arc.opensolaris.org/caselog/PSARC/2009/571/ Discussion: http://www.opensolaris.org/jive/thread.jspa?threadID=115507 Very nice -- I like the interaction with "copies", and (like a few others) I think

Hi Guys, I am wondering about mail deduplication. I am looking into the possibility of seperating out all of the message bodies with multiple parts inside mail that is recived from `dovecot` and hashing them all. The idea is that by hashing all of the parts inside the email, I will be able to ensure that each part of the email will only be saved once. This means that attachments & common

According to https://github.com/g2p/bedup/tree/wip/dedup-syscall "The clone call is considered a write operation and won''t work on read-only snapshots." Is this fixed on newer kernels? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at

?But could not we for example do split dwarf, but for example do dedup of types ? I do not mean right now, but in a theory ? Best regards, George | Developer | Access Softek, Inc ________________________________ От: David Blaikie <dblaikie at gmail.com> Отправлено: 16 декабря 2017 г. 22:25 Кому: George Rimar Копия: Sean Silva; llvm-dev at lists.llvm.org; Rui Ueyama; Rafael Espindola Тема:

Just a quick update, I''ve dropped the hashing stuff in favor of doing a memcmp in the kernel to make sure the data is still the same. The thing that takes a while is reading the data up from disk, so doing a memcmp of the entire buffer isn''t that big of a deal, not to mention there''s a possiblity for malicious users if there is a problem with the hashing algorithms we

Hi, The following series of patches implements in btrfs an ioctl to do out-of-band deduplication of file extents. To be clear, this means that the file system is mounted and running, but the dedupe is not done during file writes, but after the fact when some userspace software initiates a dedupe. The primary patch is loosely based off of one sent by Josef Bacik back in January, 2011.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, A feature of Cyrus-IMAPd I really missed after migrated to Dovecot is their optional "duplicate suppression", which eliminates duplicate message at deliver time, if their envelope sender, recipient and message-id matches. For example, if one subscribes to a mailing list, and someone hit "Reply All" to reply to him, there

Here are patches to do offline deduplication for Btrfs. It works well for the cases it''s expected to, I''m looking for feedback on the ioctl interface and such, I''m well aware there are missing features for the userspace app (like being able to set a different blocksize). If this interface is acceptable I will flesh out the userspace app a little more, but I believe the

Hi list, is there any working solution for deduplication of data for centos? We are trying to find a solution for our backup server which runs a bash script invoking xdelta(3). But having this functionality in fs is much more friendly... We have looked into lessfs, sdfs and ddar. Are these filesystems ready to use (on centos)? ddar is sthg different, I know. Thx Rainer

>Wasn't our (lld/ELF's) position on debug info size that we should focus on providing a great split-dwarf workflow and not try go too far out of our way to deduplicate >or otherwise reduce debug info size inside LLD? I recall there being some patches that made linking of large debug binaries like 1.5GB+ clang faster, but we decided to >reject those changes because split-dwarf was

Hello, I''ve stumbled upon this article: http://storagemojo.com/2011/06/27/de-dup-too-much-of-good-thing/ Reportedly Sandforce SF1200 SSD controller does internally block-level data de-duplication. This effectively removes the additional protection given by writing multiple metadata copies. This technique may be used, or can be used in the future by manufactureres of other drives too. I

Dear ZFS-Users, I want to switch to ZFS, but still want to encrypt my data. Native Encryption for ZFS was added in "ZFS Pool Version Number 30<http://en.wikipedia.org/wiki/ZFS#Release_history>", but I''m using ZFS on FreeBSD with Version 28. My question is how would encfs (fuse encryption) affect zfs specific features like data Integrity and deduplication? Regards

Howdy All, I''ve a bit of a strange problem here. I have a filesystem with one snapshot that simply refuses to be destroyed. The snapshots just prior to it and just after it were destroyed without problem. While running the zfs destroy command on this particular snapshot, the server becomes more-or-less hung. It''s pingable but will not open a new shell (local or via ssh) however

I created a zfs pool with dedup with the following settings: zpool create data c8t1d0 zfs create data/shared zfs set dedup=on data/shared The thing I was wondering about was it seems like ZFS only dedup at the file level and not the block. When I make multiple copies of a file to the store I see an increase in the deup ratio, but when I copy similar files the ratio stays at 1.00x. -- This