Hi,
I''m using btrfs as rootfs on my Fedora 12 (rawhide) test system.
Every yum activity is very slow, like 15 minutes for installation of 11
packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64,
btrfs-progs-0.19-7.fc12.x86_64.
Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk
on Intel ICH controller.
/dev/sda2 on / type btrfs (rw)
/dev/sda2 19G 6,7G 13G 36% /
Attached is output from vmstat 1 during yum transaction. I also
have 200MB of "perf record" output, which I can drill down. First
screen
of profile is below:
# Samples: 9024736
#
# Overhead Command Shared
Object Symbol
# ........ ...............
.................................................... ......
#
15.44% perf.2.6.31-33. [kernel]
[k] kmem_cache_open
9.74% perf.2.6.31-33. [kernel]
[k] print_lock_contention_bug
6.34% perf.2.6.31-33. [kernel]
[k] free_kmem_cache_cpus
4.49% perf.2.6.31-33. [kernel]
[k] is_module_text_address
4.01% perf.2.6.31-33. [kernel]
[k] lock_set_class
3.45% yum-complete-tr [kernel]
[k] print_lock_contention_bug
2.50% perf.2.6.31-33. [kernel]
[k] _spin_lock
2.48% yum-complete-tr [kernel]
[k] kmem_cache_open
2.47% perf.2.6.31-33. [kernel]
[k] zlib_tr_flush_block [zlib_deflate]
1.65% yum-complete-tr [kernel]
[k] lock_set_class
1.62% perf.2.6.31-33. [kernel]
[k] schedule_hrtimeout_range
1.53% perf.2.6.31-33. [kernel]
[k] sysfs_slab_add
1.50% yum-complete-tr [kernel]
[k] zlib_tr_flush_block [zlib_deflate]
1.46% yum-complete-tr [kernel]
[k] free_kmem_cache_cpus
1.40% yum-complete-tr [kernel]
[k] _spin_lock
1.34% perf.2.6.31-33. [kernel]
[k] __mutex_lock_common
1.27% perf.2.6.31-33. [kernel]
[k] btrfs_getxattr [btrfs]
1.06% perf.2.6.31-33. [kernel]
[k] lock_contended
0.78% perf.2.6.31-33. [kernel]
[k] _raw_read_unlock
0.75% perf.2.6.31-33. [kernel]
[k] btrfs_listxattr [btrfs]
0.74% perf.2.6.31-33. [kernel]
[k] run_scheduled_bios [btrfs]
0.62% perf.2.6.31-33. [kernel]
[k] lockdep_count_backward_deps
0.61% perf.2.6.31-33. [kernel]
[k] show_slab_objects
0.58% yum-complete-tr [kernel]
[k] sysfs_slab_add
0.48% yum-complete-tr /usr/lib64/libpython2.6.so.1.0
[.] 0x0000000005dab0
0.46% yum-complete-tr [kernel]
[k] btrfs_listxattr [btrfs]
0.45% yum-complete-tr /usr/lib64/librpm.so.0.0.0
[.] 0x0000000004877f
0.44% perf.2.6.31-33. [kernel]
[k] idr_for_each
0.44% perf.2.6.31-33. [kernel]
[k] clear_lock_stats
0.44% yum-complete-tr [kernel]
[k] lock_contended
0.43% perf.2.6.31-33. [kernel]
[k] btrfs_iget [btrfs]
0.41% yum-complete-tr [kernel]
[k] is_module_text_address
0.37% yum-complete-tr /usr/lib64/libpython2.6.so.1.0
[.] PyEval_EvalFrameEx
0.36% yum-complete-tr [kernel]
[k] run_scheduled_bios [btrfs]
0.36% yum-complete-tr [kernel]
[k] show_slab_objects
0.33% perf.2.6.31-33. [kernel]
[k] lock_acquire
0.33% perf.2.6.31-33. [kernel]
[k] __module_text_address
0.29% yum-complete-tr [kernel]
[k] btrfs_getxattr [btrfs]
0.28% ldconfig [kernel]
[k] free_kmem_cache_cpus
0.27% perf.2.6.31-33. [kernel]
[k] btrfs_set_device_sector_size [btrfs]
0.26% yum-complete-tr [kernel]
[k] _raw_read_unlock
0.25% perf.2.6.31-33. [kernel]
[k] _spin_lock_bh
0.25% yum-complete-tr [kernel]
[k] btrfs_drop_extents [btrfs]
0.23% perf.2.6.31-33. [kernel]
[k] btrfs_comp_cpu_keys [btrfs]
0.23% yum-complete-tr /lib64/libc-2.10.90.so
[.] __GI_memcpy
--
Tomasz Torcz 72->|
80->|
xmpp: zdzichubg@chrome.pl 72->|
80->|
On Mon, Sep 28, 2009 at 02:18:27PM +0200, Tomasz Torcz wrote:> > Hi, > > I''m using btrfs as rootfs on my Fedora 12 (rawhide) test system. > Every yum activity is very slow, like 15 minutes for installation of 11 > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64. > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk > on Intel ICH controller.You''re doing quite a lot of reads, and some writes. Could you please capture the output of sysrq-w at 5s intervals during the upgrade? -chris> > /dev/sda2 on / type btrfs (rw) > > /dev/sda2 19G 6,7G 13G 36% / > > Attached is output from vmstat 1 during yum transaction. I also > have 200MB of "perf record" output, which I can drill down. First screen > of profile is below: > > # Samples: 9024736 > # > # Overhead Command Shared Object Symbol > # ........ ............... .................................................... ...... > # > 15.44% perf.2.6.31-33. [kernel] [k] kmem_cache_open > 9.74% perf.2.6.31-33. [kernel] [k] print_lock_contention_bug > 6.34% perf.2.6.31-33. [kernel] [k] free_kmem_cache_cpus > 4.49% perf.2.6.31-33. [kernel] [k] is_module_text_address > 4.01% perf.2.6.31-33. [kernel] [k] lock_set_class > 3.45% yum-complete-tr [kernel] [k] print_lock_contention_bug > 2.50% perf.2.6.31-33. [kernel] [k] _spin_lock > 2.48% yum-complete-tr [kernel] [k] kmem_cache_open > 2.47% perf.2.6.31-33. [kernel] [k] zlib_tr_flush_block [zlib_deflate] > 1.65% yum-complete-tr [kernel] [k] lock_set_class > 1.62% perf.2.6.31-33. [kernel] [k] schedule_hrtimeout_range > 1.53% perf.2.6.31-33. [kernel] [k] sysfs_slab_add > 1.50% yum-complete-tr [kernel] [k] zlib_tr_flush_block [zlib_deflate] > 1.46% yum-complete-tr [kernel] [k] free_kmem_cache_cpus > 1.40% yum-complete-tr [kernel] [k] _spin_lock > 1.34% perf.2.6.31-33. [kernel] [k] __mutex_lock_common > 1.27% perf.2.6.31-33. [kernel] [k] btrfs_getxattr [btrfs] > 1.06% perf.2.6.31-33. [kernel] [k] lock_contended > 0.78% perf.2.6.31-33. [kernel] [k] _raw_read_unlock > 0.75% perf.2.6.31-33. [kernel] [k] btrfs_listxattr [btrfs] > 0.74% perf.2.6.31-33. [kernel] [k] run_scheduled_bios [btrfs] > 0.62% perf.2.6.31-33. [kernel] [k] lockdep_count_backward_deps > 0.61% perf.2.6.31-33. [kernel] [k] show_slab_objects > 0.58% yum-complete-tr [kernel] [k] sysfs_slab_add > 0.48% yum-complete-tr /usr/lib64/libpython2.6.so.1.0 [.] 0x0000000005dab0 > 0.46% yum-complete-tr [kernel] [k] btrfs_listxattr [btrfs] > 0.45% yum-complete-tr /usr/lib64/librpm.so.0.0.0 [.] 0x0000000004877f > 0.44% perf.2.6.31-33. [kernel] [k] idr_for_each > 0.44% perf.2.6.31-33. [kernel] [k] clear_lock_stats > 0.44% yum-complete-tr [kernel] [k] lock_contended > 0.43% perf.2.6.31-33. [kernel] [k] btrfs_iget [btrfs] > 0.41% yum-complete-tr [kernel] [k] is_module_text_address > 0.37% yum-complete-tr /usr/lib64/libpython2.6.so.1.0 [.] PyEval_EvalFrameEx > 0.36% yum-complete-tr [kernel] [k] run_scheduled_bios [btrfs] > 0.36% yum-complete-tr [kernel] [k] show_slab_objects > 0.33% perf.2.6.31-33. [kernel] [k] lock_acquire > 0.33% perf.2.6.31-33. [kernel] [k] __module_text_address > 0.29% yum-complete-tr [kernel] [k] btrfs_getxattr [btrfs] > 0.28% ldconfig [kernel] [k] free_kmem_cache_cpus > 0.27% perf.2.6.31-33. [kernel] [k] btrfs_set_device_sector_size [btrfs] > 0.26% yum-complete-tr [kernel] [k] _raw_read_unlock > 0.25% perf.2.6.31-33. [kernel] [k] _spin_lock_bh > 0.25% yum-complete-tr [kernel] [k] btrfs_drop_extents [btrfs] > 0.23% perf.2.6.31-33. [kernel] [k] btrfs_comp_cpu_keys [btrfs] > 0.23% yum-complete-tr /lib64/libc-2.10.90.so [.] __GI_memcpy > > > > -- > Tomasz Torcz 72->| 80->| > xmpp: zdzichubg@chrome.pl 72->| 80->| >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Sep 28, 2009 at 09:35:43AM -0400, Chris Mason wrote:> On Mon, Sep 28, 2009 at 02:18:27PM +0200, Tomasz Torcz wrote: > > > > Hi, > > > > I''m using btrfs as rootfs on my Fedora 12 (rawhide) test system. > > Every yum activity is very slow, like 15 minutes for installation of 11 > > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64. > > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk > > on Intel ICH controller. > > You''re doing quite a lot of reads, and some writes. Could you please > capture the output of sysrq-w at 5s intervals during the upgrade?It got quite big (over 100 KiB, over 1 MiB after unpacking), so I put it at http://pipebreaker.pl/dump/sysrq-w_every_5_sec.txt.bz2 -- Tomasz Torcz Only gods can safely risk perfection, xmpp: zdzichubg@chrome.pl it''s a dangerous thing for a man. -- Alia -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Sep 29, 2009 at 08:18:22AM +0200, Tomasz Torcz wrote:> On Mon, Sep 28, 2009 at 09:35:43AM -0400, Chris Mason wrote: > > On Mon, Sep 28, 2009 at 02:18:27PM +0200, Tomasz Torcz wrote: > > > > > > Hi, > > > > > > I''m using btrfs as rootfs on my Fedora 12 (rawhide) test system. > > > Every yum activity is very slow, like 15 minutes for installation of 11 > > > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64. > > > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk > > > on Intel ICH controller. > > > > You''re doing quite a lot of reads, and some writes. Could you please > > capture the output of sysrq-w at 5s intervals during the upgrade? > > It got quite big (over 100 KiB, over 1 MiB after unpacking), so I > put it at http://pipebreaker.pl/dump/sysrq-w_every_5_sec.txt.bz2Ok, you''ve got 158 sysrq-w runs in here, and 119 of them involve fsync. This is good because it is what Josef expected the problem to be. The first thing I would try is mount -o ssd. I think what is happening is that we are seeking around while fsyncing files. mount -o ssd isn''t really the right long term answer but it will tell us if I''ve got it right. If that works, I''d try the code in the master branch of btrfs-unstable. This has a change that makes the allocator a little smarter, even without mount -o ssd. If you''re comparing w/ext3 and wondering why btrfs is sooooooo much slower it might be because btrfs has barriers on by default and ext3 doesn''t. You could mount -o nobarrier for btrfs or mount -o barrier=1 for ext3 for a proper comparison. (Assuming your dmesg doesn''t have messages from btrfs about disabling barriers). -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Sep 29, 2009 at 09:25:38AM -0400, Chris Mason wrote:> On Tue, Sep 29, 2009 at 08:18:22AM +0200, Tomasz Torcz wrote: > > On Mon, Sep 28, 2009 at 09:35:43AM -0400, Chris Mason wrote: > > > > Every yum activity is very slow, like 15 minutes for installation of 11 > > > > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64. > > > > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk > > > > on Intel ICH controller. > > > > > > You''re doing quite a lot of reads, and some writes. Could you please > > > capture the output of sysrq-w at 5s intervals during the upgrade? > > > > It got quite big (over 100 KiB, over 1 MiB after unpacking), so I > > put it at http://pipebreaker.pl/dump/sysrq-w_every_5_sec.txt.bz2 > > Ok, you''ve got 158 sysrq-w runs in here, and 119 of them involve fsync. > This is good because it is what Josef expected the problem to be. > > The first thing I would try is mount -o ssd. I think what is happening > is that we are seeking around while fsyncing files. mount -o ssd isn''t > really the right long term answer but it will tell us if I''ve got it > right.Tried that. Not noticable faster. Dumps are here: http://pipebreaker.pl/dump/sysrq_w-with_ssd.txt.bz2 http://pipebreaker.pl/dump/vmstat-with.ssd.txt.bz2> > If you''re comparing w/ext3 and wondering why btrfs is sooooooo much > slower it might be because btrfs has barriers on by default and ext3 > doesn''t. You could mount -o nobarrier for btrfs or mount -o barrier=1 > for ext3 for a proper comparison. > > (Assuming your dmesg doesn''t have messages from btrfs about disabling > barriers).I wouldn''t expect barriers to work here (reminder, this is PATA drive on ICH7 sata controller), but I will test tomorrow with nobarrier. Then I probably check his "yum upgrade" under seekwatcher on friday. -- Tomasz Torcz Morality must always be based on practicality. xmpp: zdzichubg@chrome.pl -- Baron Vladimir Harkonnen -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote:> > > > If you''re comparing w/ext3 and wondering why btrfs is sooooooo much > > slower it might be because btrfs has barriers on by default and ext3 > > doesn''t. You could mount -o nobarrier for btrfs or mount -o barrier=1 > > for ext3 for a proper comparison. > > > > (Assuming your dmesg doesn''t have messages from btrfs about disabling > > barriers). > > I wouldn''t expect barriers to work here (reminder, this is PATA drive > on ICH7 sata controller), but I will test tomorrow with nobarrier. > Then I probably check his "yum upgrade" under seekwatcher on friday.Nobarrier result are here: http://pipebreaker.pl/dump/sysrq_w-nobarrier.txt.bz2 http://pipebreaker.pl/dump/vmstat-nobarrier.txt.bz2 My observation: still slow. Just a note on my unscientific methodology: I''ve got two very similar machines with Rawhide, one on ext4, second on btrfs. Every day I do "yum upgrade" on both. Ext4 one finishes much faster thatn btrfs. This is especially visible while yum is in "cleanup phase". On ext4 most packages are cleaned''up in under one second time. On btrfs this takes few seconds. Whole transaction ends in couple of minutes on ext4. On btrfs it takes 5x longer. -- Tomasz Torcz RIP is irrevelant. Spoofing is futile. xmpp: zdzichubg@chrome.pl Your routes will be aggreggated. -- Alex Yuriev -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 30 2009, Tomasz Torcz wrote:> I wouldn''t expect barriers to work here (reminder, this is PATA drive > on ICH7 sata controller), but I will test tomorrow with nobarrier. > Then I probably check his "yum upgrade" under seekwatcher on friday.Barriers should work fine on that setup. In fact, barriers were first implemented for PATA back in the day :-) -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 10/01/2009 05:04 AM, Jens Axboe wrote:> On Wed, Sep 30 2009, Tomasz Torcz wrote: > >> I wouldn''t expect barriers to work here (reminder, this is PATA drive >> on ICH7 sata controller), but I will test tomorrow with nobarrier. >> Then I probably check his "yum upgrade" under seekwatcher on friday. >> > Barriers should work fine on that setup. In fact, barriers were first > implemented for PATA back in the day :-) > >I think that we first tested them on ich5 & old promise controllers with old (250GB) P-ATA disks :-) ric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Oct 01, 2009 at 11:01:46AM +0200, Tomasz Torcz wrote:> On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote: > > > > > > If you''re comparing w/ext3 and wondering why btrfs is sooooooo much > > > slower it might be because btrfs has barriers on by default and ext3 > > > doesn''t. You could mount -o nobarrier for btrfs or mount -o barrier=1 > > > for ext3 for a proper comparison.Last update: things got noticably faster with 2.6.31.4-88.fc12.x86_64, which contains backport of latest mainline btrfs (thank you, Josef!). No more comlains from me (for now ;). -- Tomasz Torcz Morality must always be based on practicality. xmpp: zdzichubg@chrome.pl -- Baron Vladimir Harkonnen -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Oct 21, 2009 at 11:27:33AM +0200, Tomasz Torcz wrote:> On Thu, Oct 01, 2009 at 11:01:46AM +0200, Tomasz Torcz wrote: > > On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote: > > > > > > > > If you''re comparing w/ext3 and wondering why btrfs is sooooooo much > > > > slower it might be because btrfs has barriers on by default and ext3 > > > > doesn''t. You could mount -o nobarrier for btrfs or mount -o barrier=1 > > > > for ext3 for a proper comparison. > > Last update: things got noticably faster with 2.6.31.4-88.fc12.x86_64, > which contains backport of latest mainline btrfs (thank you, Josef!). > No more comlains from me (for now ;).Fantastic. Are you running with or without barriers now? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Oct 26, 2009 at 05:55:45PM +0900, Chris Mason wrote:> On Wed, Oct 21, 2009 at 11:27:33AM +0200, Tomasz Torcz wrote: > > On Thu, Oct 01, 2009 at 11:01:46AM +0200, Tomasz Torcz wrote: > > > On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote: > > > > > > > > > > If you''re comparing w/ext3 and wondering why btrfs is sooooooo much > > > > > slower it might be because btrfs has barriers on by default and ext3 > > > > > doesn''t. You could mount -o nobarrier for btrfs or mount -o barrier=1 > > > > > for ext3 for a proper comparison. > > > > Last update: things got noticably faster with 2.6.31.4-88.fc12.x86_64, > > which contains backport of latest mainline btrfs (thank you, Josef!). > > No more comlains from me (for now ;). > > Fantastic. Are you running with or without barriers now?I stick to default settings, that means barriers I think. Overall, my test hardware is so slow by itself, that barriers are not really noticable. (It''s 3 GHz Pentium 4 in 64 bit with single IDE 20GB drive). -- Tomasz Torcz RIP is irrevelant. Spoofing is futile. xmpp: zdzichubg@chrome.pl Your routes will be aggreggated. -- Alex Yuriev -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html