similar to: Offline Deduplication for Btrfs

Just a quick update, I''ve dropped the hashing stuff in favor of doing a memcmp in the kernel to make sure the data is still the same. The thing that takes a while is reading the data up from disk, so doing a memcmp of the entire buffer isn''t that big of a deal, not to mention there''s a possiblity for malicious users if there is a problem with the hashing algorithms we

Originally from https://github.com/markfasheh/duperemove/blob/master/btrfs-extent-same.c Signed-off-by: Gabriel de Perthuis <g2p.code+btrfs@gmail.com> --- .gitignore | 1 + Makefile | 2 +- btrfs-extent-same.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+), 1 deletion(-) create mode 100644 btrfs-extent-same.c diff

Hey list, I wonder if it is possible to deduplicate read-only snapshots. Background: I''m using an bash/rsync script[1] to backup my whole system on a nightly basis to an attached USB3 drive into a scratch area, then take a snapshot of this area. I''d like to have these snapshots immutable, so they should be read-only. Since rsync won''t discover moved files but

Hello, I was bored this weekend so I hacked up online dedup for Btrfs. It''s working quite well so I think it can be more widely tested. There are two ways to use it 1) Compatible mode - this is a bit slower but will handle being used by older kernels. We use the csum tree to find duplicate blocks. Since it is relatively easy to have crc32c collisions this also involves reading the

Hello, some questions regarding btrfs deduplication. - What is the state of it? Is it "safe" to use? https://btrfs.wiki.kernel.org/index.php/Deduplication does not yield much information. - https://pypi.python.org/pypi/bedup says: "bedup looks for new and changed files, making sure that multiple copies of identical files share space on disk. It integrates deeply with btrfs so

Hi, The following series of patches implements in btrfs an ioctl to do out-of-band deduplication of file extents. To be clear, this means that the file system is mounted and running, but the dedupe is not done during file writes, but after the fact when some userspace software initiates a dedupe. The primary patch is loosely based off of one sent by Josef Bacik back in January, 2011.

Glück Auf! I know its been discussed more then ones, but as a user I really would like to see the patch for allowing this in the kernel. Some users tested this patch successfully for weeks or months in 2 or 3 kernel versions since then, true? I''d say by creating a snapshot, it''s nothing else in the end. More then one file or tree sharing the same data on disc, or am I wrong?

We''re going to use this flag EXTENT_DEFRAG to indicate which range belongs to defragment so that we can implement snapshow-aware defrag: We set the EXTENT_DEFRAG flag when dirtying the extents that need defragmented, so later on writeback thread can differentiate between normal writeback and writeback started by defragmentation. This patch is used for the latter one. Originally patch

I''ve been thinking about this for a while, from a perspective of how to make it work by allocating i-node numbers from a global pool, but yesterday I realized that offering the feature would be a bad idea because it violates the semantics of file systems. I will be happy to expand on that point if anyone disagrees with it. dln -- "It is merely a matter of persistence." --

I''m looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance? Specifically, on ext2 (journal means more writes, so I don''t use ext3 on SSDs,

Welcome, I''ve been using btrfs for over a 3 months to store my personal data on my NAS server. Almost all interactions with files on the server are done using unison synchronizer. After another use of bedup (https://github.com/g2p/bedup) on my btrfs volume I experienced huge perfomance loss with synchronization. It now takes over 3 hours what have taken only 15 minutes! File

While testing raid-auto-repair patches I''m going to send out later, I just found the very last bug in my current scrub patch series: Changelog v4->v5: - fixed a deadlock when fixup is taking longer while scrub is about to end Original message follows: ------------------------ This patch set introduces two new features for scrub. They share the backref iteration code which is the

Hello, Trying to put together a list of TODO items for btrfs so we can update the wiki page fully. So far these things are on the list * Proper ENOSPC handling * O_DIRECT support (without checksumming) * AIO support * Subvolume quotas and inherited space usage information * Snapshot removal * QA Suite for automated regression testing * Reserved space for online fsck and the ability to add

Hi, There''s been discussion before on this list on the very small number of hard links supported by btrfs.[1][2] In those threads, an often asked question has been if there''s a real world use case the limit breaks. Also it has been pointed out that a fix for this would need a disk format change. As discussed in bug #15762 [3], there are certainly real-world use cases this

Hi gang, I was a little surprised to see that patch go by recently which fixed an endian bug. I went to see how sparse checking looked and it was.. broken. I got it going again in my Fedora environment. Most of the patches are just cleanups, but there *were* three real bugs lurking in all that sparse warning spam. So I maintain that it''s worth our time to keep it going and fix

I realise that the hard link limit is in the queue to fix, and I read the recent thread as well as the older (october I think) thread. I just wanted to note that BackupPC *does* in fact run into the hard link limit, and its due to the dpkg configuration scripts. BackupPC hard links files with the same content together by scanning new files and linking them together, whether or not they started

Hi all Having managed ZFS for about two years, I want to post a wishlist. INCLUDED IN ZFS - Mirror existing single-drive filesystem, as in ''zfs attach'' - RAIDz-stuff - single and hopefully multiple-parity RAID configuration with block-level checksumming - Background scrub/fsck - Pool-like management with multiple RAIDs/mirrors (VDEVs) - Autogrow as in ZFS autoexpand NOT

Hi. I am seriously considering employing btrfs on my systems, particularly due to some space-saving features that it has (namely, deduplication and compression). In fact, I was (a few moments ago) trying to back up some of my systems to a 2TB HD that has an ext4 filesystem and, in the middle of the last one, I got the error message that the backup HD was full. Given that what I backup there are

After running ceph on XFS for some time, I decided to try btrfs again. Performance with the current "for-linux-min" branch and big metadata is much better. The only problem (?) I''m still seeing is a warning that seems to occur from time to time: [87703.784552] ------------[ cut here ]------------ [87703.789759] WARNING: at fs/btrfs/inode.c:2103

Hello, I''ve stumbled upon this article: http://storagemojo.com/2011/06/27/de-dup-too-much-of-good-thing/ Reportedly Sandforce SF1200 SSD controller does internally block-level data de-duplication. This effectively removes the additional protection given by writing multiple metadata copies. This technique may be used, or can be used in the future by manufactureres of other drives too. I