Btrfs devel - Jun 2011 - Applications using fsync cause hangs for several seconds every few minutes

Hello list,

I''ve been using btrfs on my personal machines for about two years now,
and on this machine for about a year with absolutely no problems.
Infact, it has held up better than ext4 with regards to reliability.

However, recently, perhaps with 2.6.39, or after I quickly started
filling up my disk again, it has become impossible for me to work for
long periods on my machine.

Every few minutes, (I guess) when applications do fsync (firefox,
xchat, vim, etc), all applications that use fsync() hang for several
seconds, and applications that use general IO suffer extreme
slowdowns. iotop shows various combinations of the processes listed
below doing writes, and the total write as 2-3MB/s.

[btrfs-dealloc-]
[btrfs-submit-0]
[btrfs-transacti]
[btrfs-endio-wri]
[flush-btrfs-1]

In some extreme cases, I''ve had hangs for 5 whole minutes. I''m
really
beginning to appreciate how little I/O GNOME Shell does since it
remains completely responsive throughout this. I have a feeling that
the cause for this is extreme fragmentation.

My hard disk is a 500GB SATA hdd, my btrfs partition details are:

# btrfs filesystem show
Label: ''gentoo''  uuid: 6f539d7f-f70f-4216-a4a9-6f7a2117a04a
	Total devices 1 FS bytes used 246.37GB
	devid    1 size 345.13GB used 345.13GB path /dev/sda7

Btrfs v0.19-35-g1b444cd-dirty

What can I do to debug this issue? What other information should I
supply? Could someone guide me on how to figure out why my machine is
unusable now?

Thanks in advance,

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hello,

Nirbheek Chauhan <nirbheek <at> gentoo.org>
writes:
> However, recently, perhaps with 2.6.39, or after I quickly started
> filling up my disk again, it has become impossible for me to work for
> long periods on my machine.
>
> Every few minutes, (I guess) when applications do fsync (firefox,
> xchat, vim, etc), all applications that use fsync() hang for several
> seconds, and applications that use general IO suffer extreme
> slowdowns. iotop shows various combinations of the processes listed
> below doing writes, and the total write as 2-3MB/s.
>
> [btrfs-dealloc-]
> [btrfs-submit-0]
> [btrfs-transacti]
> [btrfs-endio-wri]
> [flush-btrfs-1]
I''m using btrfs under a 2.6.39-ARCH kernel and run into the same issue.

In my case the [btrfs-submit-0] and [btrfs-transacti] shows up in iotop
and produce 99% of IO at the time a application is frozen. For something
like 10 to 30 seconds.

After a fresh boot everything runs quiet acceptable and goes worse over
the day. If I reboot the unmount process takes a lot of time something
up to 8 minutes. After the reboot everything is back to normal. At least
for a while.

I use snapshots on a daily base and have 20 active snapshots all the
time. Maybe this is the reason for the performance impact?
> What can I do to debug this issue? What other information should I
> supply? Could someone guide me on how to figure out why my machine is
> unusable now?
I have no solution for this problem. In fact a reboot helps me even if
only temporary. I also changed the IO scheduler, switched to laptop mode
and remounted the device. Neither of these helped.

Maybe someone else has any suggestion?

Thanks.

Jan Stilow
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 2011-06-07 at 04:28 +0530, Nirbheek Chauhan
wrote:
> Every few minutes, (I guess) when applications do fsync (firefox,
> xchat, vim, etc), all applications that use fsync() hang for several
> seconds, and applications that use general IO suffer extreme
> slowdowns. iotop shows various combinations of the processes listed
> below doing writes, and the total write as 2-3MB/s. 
I have experienced this too. It /seemed/ to help removing a lot of
snapshots (i have hundreds that i didn''t really need).

Would it be stupid to try disabling fsync like described at 
 http://ubuntuforums.org/archive/index.php/t-1103926.html ?

I don''t know of the consequences... but it would prove your theory?

~mck

-- 
“Don’t worry about people stealing your ideas. If your ideas are any
good, you’ll have to ram them down people’s throats.” - Howard Aiken 
| http://semb.wever.org | http://sesat.no
| http://tech.finn.no       | Java XSS Filter

On 06/06/2011 06:58 PM, Nirbheek Chauhan wrote:
> Hello list,
> 
> I''ve been using btrfs on my personal machines for about two years
now,
> and on this machine for about a year with absolutely no problems.
> Infact, it has held up better than ext4 with regards to reliability.
> 
> However, recently, perhaps with 2.6.39, or after I quickly started
> filling up my disk again, it has become impossible for me to work for
> long periods on my machine.
> 
> Every few minutes, (I guess) when applications do fsync (firefox,
> xchat, vim, etc), all applications that use fsync() hang for several
> seconds, and applications that use general IO suffer extreme
> slowdowns. iotop shows various combinations of the processes listed
> below doing writes, and the total write as 2-3MB/s.
> 
> [btrfs-dealloc-]
> [btrfs-submit-0]
> [btrfs-transacti]
> [btrfs-endio-wri]
> [flush-btrfs-1]
> 
> In some extreme cases, I''ve had hangs for 5 whole minutes.
I''m really
> beginning to appreciate how little I/O GNOME Shell does since it
> remains completely responsive throughout this. I have a feeling that
> the cause for this is extreme fragmentation.
> 
> My hard disk is a 500GB SATA hdd, my btrfs partition details are:
> 
> # btrfs filesystem show
> Label: ''gentoo''  uuid:
6f539d7f-f70f-4216-a4a9-6f7a2117a04a
> 	Total devices 1 FS bytes used 246.37GB
> 	devid    1 size 345.13GB used 345.13GB path /dev/sda7
> 
> Btrfs v0.19-35-g1b444cd-dirty
> 
> What can I do to debug this issue? What other information should I
> supply? Could someone guide me on how to figure out why my machine is
> unusable now?
> 
> Thanks in advance,
> 
Hello,

I''ve been looking into this and I have a suspicion.  Would you run with
this
patch and see if the problem goes away?  If so I''m on the right track
and I''ll
have more test patches for you to try :).  Thanks,

Josef

diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 19450bc..2e30350 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -150,7 +150,6 @@ static noinline int run_scheduled_bios(struct
btrfs_device *device)
 	 * another device without first sending all of these down.
 	 * So, setup a plug here and finish it off before we return
 	 */
-	blk_start_plug(&plug);

 	bdi = blk_get_backing_dev_info(device->bdev);
 	fs_info = device->dev_root->fs_info;
@@ -290,7 +289,6 @@ loop_lock:
 	spin_unlock(&device->io_lock);

 done:
-	blk_finish_plug(&plug);
 	return 0;
 }

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, Jul 18, 2011 at 11:47 PM, Josef Bacik <josef@redhat.com>
wrote:
> On 06/06/2011 06:58 PM, Nirbheek Chauhan wrote:
>> What can I do to debug this issue? What other information should I
>> supply? Could someone guide me on how to figure out why my machine is
>> unusable now?
>
> I''ve been looking into this and I have a suspicion.  Would you run
with this
> patch and see if the problem goes away?  If so I''m on the right
track
> and I''ll
> have more test patches for you to try :).  Thanks,
>
Hello!

Thanks for taking a look! I''m currently travelling and as soon as I
get a bit closer to my backups, I''ll try the patch out and report back
for further testing :)

Cheers,

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, 2011-07-18 at 14:17 -0400, Josef Bacik wrote:
> I''ve been looking into this and I have a suspicion.  Would you run
> with this patch and see if the problem goes away? 
Didn''t help me.

2.6.39 is not usable. 3.0.0 is ok for a few hours then too becomes
unusable. This is discussed in future threads, eg "Btrfs slowdown".

dumbing out fsync (libeatmydata) only gives marginal improvements...

~mck

-- 
Bombing for peace is like F***ing for virginity | www.semb.wever.org |
www.sesat.no | tech.finn.no | http://xss-http-filter.sf.net