similar to: [RFC] Online dedup for Btrfs

Here are patches to do offline deduplication for Btrfs. It works well for the cases it''s expected to, I''m looking for feedback on the ioctl interface and such, I''m well aware there are missing features for the userspace app (like being able to set a different blocksize). If this interface is acceptable I will flesh out the userspace app a little more, but I believe the

Just a quick update, I''ve dropped the hashing stuff in favor of doing a memcmp in the kernel to make sure the data is still the same. The thing that takes a while is reading the data up from disk, so doing a memcmp of the entire buffer isn''t that big of a deal, not to mention there''s a possiblity for malicious users if there is a problem with the hashing algorithms we

We need an smb_mb() before waitqueue_active to avoid missing wakeups. Before Mitch was hitting a deadlock between the ordered flushers and the transaction commit because the ordered flushers were waiting for more refs and were never woken up, so those smp_mb()''s are the most important. Everything else I added for correctness sake and to avoid getting bitten by this again somewhere else.

In fixing how we deal with bad inodes, we had a regression in the orphan cleanup code, since it expects to get a bad inode back. So fix it to deal with getting -ESTALE back by deleting the orphan item manually and moving on. Thanks, Reported-by: Simon Kirby <sim@hostway.ca> Signed-off-by: Josef Bacik <josef@redhat.com> --- fs/btrfs/inode.c | 36

An user reported that he has hit an annoying deadlock while playing with ceph based on btrfs. Current updating device tree requires space from METADATA chunk, so we -may- need to do a recursive chunk allocation when adding/updating dev extent, that is where the deadlock comes from. If we use SYSTEM metadata to update device tree, we can avoid the recursive stuff. Reported-by: Jim Schutt

I noticed while running xfstests 83 that if we didn''t have enough space to delete our inode the orphan cleanup would just loop. This is because it keeps finding the same orphan item and keeps trying to kill it but can''t because we don''t get an error back from iput for deleting the inode. So keep track of the last guy we tried to kill, if it''s the same as the

Hello, I just updated and pushed btrfs-next, if it is missing something let me know. I had to kick out the printk formatting patch because it didn''t compile and I didn''t take Miao''s background enospc flushing stuff since some of it didn''t apply and 5/5 hasn''t been updated to force waiting on background flushers. Let me know if I missed anything

We''re going to use this flag EXTENT_DEFRAG to indicate which range belongs to defragment so that we can implement snapshow-aware defrag: We set the EXTENT_DEFRAG flag when dirtying the extents that need defragmented, so later on writeback thread can differentiate between normal writeback and writeback started by defragmentation. This patch is used for the latter one. Originally patch

Hello all. What we have: SL6 - kernel 2.6.32-131.2.1.el6.x86_64 btrfs on mdadm RAID5 with 8 HDD - 27T partition. I see this at top: 1182 root 20 0 0 0 0 R 100.0 0.0 16:39.73 [btrfs-delalloc-] And LA is grow. What is this and how can I fix it? -- Best regards, Proskurin Kirill -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of

Btrfs uses an rwsem to control access to its extent tree. Threads will hold a read lock on this rwsem while they scan the extent tree, and if need_resched() they will drop the lock and schedule. The transaction commit needs to take a write lock for this rwsem for a very short period to switch out the commit roots. If there are a lot of threads doing this caching operation we can starve out the

If we have particularly full nodes, we could call btrfs_node_blockptr up to 32 times, which is 32 pairs of kmap/kunmap, which _sucks_. So go ahead and map the extent buffer while we look for readahead targets. Thanks, Signed-off-by: Josef Bacik <josef@redhat.com> --- fs/btrfs/ctree.c | 23 +++++++++++++++++++++-- 1 files changed, 21 insertions(+), 2 deletions(-) diff --git

We''ve been seeing warnings coming out of the orphan commit stuff forever from ceph. Turns out it''s because we''re racing with checking if the orphan block reserve is set, because we clear it outside of the spin_lock. So leave the normal fastpath checks where they are, but take the spin_lock and _recheck_ to make sure we haven''t had an orphan block rsv added in

Hello, Trying to put together a list of TODO items for btrfs so we can update the wiki page fully. So far these things are on the list * Proper ENOSPC handling * O_DIRECT support (without checksumming) * AIO support * Subvolume quotas and inherited space usage information * Snapshot removal * QA Suite for automated regression testing * Reserved space for online fsck and the ability to add

Liu fixed part of this problem and unfortunately I steered him in slightly the wrong direction and so didn''t completely fix the problem. The problem is we limit the size of the delalloc range we are looking for to max bytes and then we try to lock that range. If we fail to lock the pages in that range we will shrink the max bytes to a single page and re loop. However if our first page

When the binary search returns 0 (exact match), the target key will necessarily be at slot 0 of all nodes below the current one, so in this case the binary search is not needed because it will always return 0, and we waste time doing it, holding node locks for longer than necessary, etc. Below follow histograms with the times spent on the current approach of doing a binary search when the

Back story: I started my pool with a 200gb partition at the end of my drive (sdc5) , until I was able to clear out the data at the beginning of my drive. When I was ready, I ran `btrfs dev add /dev/sdc4 /` then `btrfs dev del /dev/sdc5 /`, $ sudo btrfs fi resize max / Resize ''/'' of ''max'' ERROR: unable to resize ''/'' - Invalid argument in

On 06/30/2011 09:13 PM, Josef Bacik wrote: > On 06/30/2011 10:12 AM, Proskurin Kirill wrote: >> On 06/29/2011 08:14 PM, Josef Bacik wrote: >>>> Ok - I upgrade to 2.6.39-2 but it is seems to all things get worse. >>>> Now I see [btrfs-transacti]& btrfs-endio-wri] 80-100% all the time and >>>> io performance looks like lower then before.

After running ceph on XFS for some time, I decided to try btrfs again. Performance with the current "for-linux-min" branch and big metadata is much better. The only problem (?) I''m still seeing is a warning that seems to occur from time to time: [87703.784552] ------------[ cut here ]------------ [87703.789759] WARNING: at fs/btrfs/inode.c:2103

This was a request from the systemd guys. They need a quick and easy way to get all devices attached to a Btrfs filesystem in order to check if any of the disks are SSD for...something, I didn''t ask :). I''ve tested this with the btrfs-progs patch that accompanies this patch. Thanks, Signed-off-by: Josef Bacik <josef@redhat.com> --- fs/btrfs/ioctl.c | 64

When testing with max_extents=4k, we enospc out really really early. The reason for this is we really overwhelm the system with our worst case calculation. When we try to flush delalloc, we don''t want everybody to wait around forever, so we wake up the waiters when we''ve done some of the work in hopes that its enough work to get everything they need done. The problem with this