Jacek Konieczny
2012-Jun-14 09:41 UTC
Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
Hello,
After upgrading my DomUs to Linux kernel 3.3.0 I have noticed some
applications are _inceribly_ slow. Eventually I fount out that are some
filesystem writes blocking for long time.
The problem occurs no matter if I use file-backed VBD or
LVM-device-backed VBD, it doesn''t matter if I run the domU under Xen
4.1.2 or under Xen 3.4.3. It does not matter if I write through
a file system in domU or if I write directly to the xvd device.
But the same operations work well under kernels 3.0.x, but fail in
3.3.x.
Any ideas what is causing that? How can I prevent this in any other way
than downgrading the domU kernel?
[root@vpbx15 ~]# time dd if=/dev/zero bs=1024 count=10000 of=/dev/xvdc1
10000+0 records in
10000+0 records out
10240000 bytes (10 MB) copied, 52.9366 s, 193 kB/s
88.23s real 0.00s user 0.36s system
while blocked:
[root@vpbx15 ~]# cat /proc/`pidof dd`/stack
[<c016891a>] check_preempt_curr+0x6a/0x80
[<c0109f3c>] xen_clocksource_read+0x2c/0x60
[<c0179f10>] ktime_get_ts+0xf0/0x120
[<c01bd285>] sleep_on_page+0x5/0x10
[<c01bd280>] sleep_on_page+0x0/0x10
[<c01bd4cd>] wait_on_page_bit+0x8d/0xa0
[<c015c8a0>] wake_bit_function+0x0/0x60
[<c01bd6e1>] grab_cache_page_write_begin+0x91/0xd0
[<c0220d1d>] block_write_begin+0x2d/0x90
[<c01bcab7>] generic_file_buffered_write+0xe7/0x250
[<c0225e10>] blkdev_get_block+0x0/0x70
[<c01696d8>] get_parent_ip+0x8/0x20
[<c01be43e>] __generic_file_aio_write+0x27e/0x500
[<c0109880>] xen_read_cr2_direct+0x0/0x10
[<c0430a81>] do_page_fault+0x161/0x4d0
[<c0224feb>] blkdev_aio_write+0x3b/0xa0
[<c01f5a95>] do_sync_write+0xc5/0x110
[<c01f625a>] rw_verify_area+0x6a/0x130
[<c01f59d0>] do_sync_write+0x0/0x110
[<c01f6620>] vfs_write+0xa0/0x160
[<c01f6901>] sys_write+0x41/0x80
[<c0433f5c>] sysenter_do_call+0x12/0x2c
[<ffffffff>] 0xffffffff
Greets,
Jacek
Fajar A. Nugraha
2012-Jun-14 09:49 UTC
Re: Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
On Thu, Jun 14, 2012 at 4:41 PM, Jacek Konieczny <jajcus@jajcus.net> wrote:> Hello, > > After upgrading my DomUs to Linux kernel 3.3.0 I have noticed some > applications are _inceribly_ slow. Eventually I fount out that are some > filesystem writes blocking for long time. > > The problem occurs no matter if I use file-backed VBD or > LVM-device-backed VBD, it doesn''t matter if I run the domU under Xen > 4.1.2 or under Xen 3.4.3. It does not matter if I write through > a file system in domU or if I write directly to the xvd device. > But the same operations work well under kernels 3.0.x, but fail in > 3.3.x. > > Any ideas what is causing that? How can I prevent this in any other way > than downgrading the domU kernel?Asking in xen-devel might provide you with more help. However, before doing so, you should test with latest -rc kernel (3.5-rc2) to make sure the problem still exists in latest development version. -- Fajar
Jacek Konieczny
2012-Jun-14 10:21 UTC
Re: Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
On Thu, Jun 14, 2012 at 04:49:12PM +0700, Fajar A. Nugraha wrote:> Asking in xen-devel might provide you with more help. However, before > doing so, you should test with latest -rc kernel (3.5-rc2) to make > sure the problem still exists in latest development version.Unfortunately, due to different specific requirements I am not able to currently try anything newer than 3.3.x, so I am wondering which change before 3.0 and 3.3.0 could cause that problem… Before trying every kernel version or git-bisect I was hoping for some hints that could save me some work… Greets, Jacek _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Fajar A. Nugraha
2012-Jun-14 10:28 UTC
Re: Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
On Thu, Jun 14, 2012 at 5:21 PM, Jacek Konieczny <jajcus@jajcus.net> wrote:> On Thu, Jun 14, 2012 at 04:49:12PM +0700, Fajar A. Nugraha wrote: >> Asking in xen-devel might provide you with more help. However, before >> doing so, you should test with latest -rc kernel (3.5-rc2) to make >> sure the problem still exists in latest development version. > > Unfortunately, due to different specific requirements I am not able to > currently try anything newer than 3.3.x, so I am wondering which change > before 3.0 and 3.3.0 could cause that problem… > > Before trying every kernel version or git-bisect I was hoping for some > hints that could save me some work…The one most familiar with xen support in upstream kernel is probably Konrad, and asking in xen-devel would (hopefully) catch other dev''s attention as well. However even if you find the problem, I doubt there''s much you can do about it as kernel 3.3 is now EOL, and fixes would probably be on or against latest -rc kernel. A workaround you can probably try is look at vendor-supported kernels, e.g. Ubuntu''s 3.2 kernel, as those usually have some backport fixes from later versions. -- Fajar
G.Bakalarski@icm.edu.pl
2012-Jun-14 10:33 UTC
Re: Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
> On Thu, Jun 14, 2012 at 04:49:12PM +0700, Fajar A. Nugraha wrote: >> Asking in xen-devel might provide you with more help. However, before >> doing so, you should test with latest -rc kernel (3.5-rc2) to make >> sure the problem still exists in latest development version. > > Unfortunately, due to different specific requirements I am not able to > currently try anything newer than 3.3.x, so I am wondering which change > before 3.0 and 3.3.0 could cause that problem? > > Before trying every kernel version or git-bisect I was hoping for some > hints that could save me some work? >We are using 3.2 from Debian Wheezy. It is close to 3.2.10 probably. It performs quite well ... You may try a kernel from recent Apline Linux (they announced recent release based on 3.3.x - if my memory is not corrupted)... GB
Jacek Konieczny
2012-Jun-28 12:41 UTC
[SOLVED] Re: Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
On Thu, Jun 14, 2012 at 11:41:38AM +0200, Jacek Konieczny wrote:> After upgrading my DomUs to Linux kernel 3.3.0 I have noticed some > applications are _inceribly_ slow. Eventually I fount out that are some > filesystem writes blocking for long time.[...]> [root@vpbx15 ~]# time dd if=/dev/zero bs=1024 count=10000 of=/dev/xvdc1 > 10240000 bytes (10 MB) copied, 52.9366 s, 193 kB/s > 88.23s real 0.00s user 0.36s system[...]> while blocked: > > [root@vpbx15 ~]# cat /proc/`pidof dd`/stack > [<c016891a>] check_preempt_curr+0x6a/0x80 > [<c0109f3c>] xen_clocksource_read+0x2c/0x60 > [<c0179f10>] ktime_get_ts+0xf0/0x120I have managed to upgrade my image to kernel 3.4.4 – that still didn't help, but when I switched to a 64-bit kernel, the problem disappeared. I have been using 64-bit hypervisor for years now, but the domU kernels were always 32-bit. Upgrading dom0 kernel to 64-bit did not change anything. I am not 100% sure if it is the kernel architecture changes or some difference in kernel configurations (some options which where enabled in 32-bit kernel where automatically disabled when compiling for 64-bit), but I cannot find any such obvious setting change. Anyway, using 64-bit kernel fixes the problem for me and that is good enough. Since I am able to use a 64-bit kernel I have no reason to use 32 bit Linux kernels under Xen any more. Greets, Jacek _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Jacek Konieczny
2012-Jun-28 12:41 UTC
[SOLVED] Re: [Xen-users] Very slow VBD writes in domU under Linux 3.3, ok under 3.0.x
On Thu, Jun 14, 2012 at 11:41:38AM +0200, Jacek Konieczny wrote:> After upgrading my DomUs to Linux kernel 3.3.0 I have noticed some > applications are _inceribly_ slow. Eventually I fount out that are some > filesystem writes blocking for long time.[...]> [root@vpbx15 ~]# time dd if=/dev/zero bs=1024 count=10000 of=/dev/xvdc1 > 10240000 bytes (10 MB) copied, 52.9366 s, 193 kB/s > 88.23s real 0.00s user 0.36s system[...]> while blocked: > > [root@vpbx15 ~]# cat /proc/`pidof dd`/stack > [<c016891a>] check_preempt_curr+0x6a/0x80 > [<c0109f3c>] xen_clocksource_read+0x2c/0x60 > [<c0179f10>] ktime_get_ts+0xf0/0x120I have managed to upgrade my image to kernel 3.4.4 – that still didn't help, but when I switched to a 64-bit kernel, the problem disappeared. I have been using 64-bit hypervisor for years now, but the domU kernels were always 32-bit. Upgrading dom0 kernel to 64-bit did not change anything. I am not 100% sure if it is the kernel architecture changes or some difference in kernel configurations (some options which where enabled in 32-bit kernel where automatically disabled when compiling for 64-bit), but I cannot find any such obvious setting change. Anyway, using 64-bit kernel fixes the problem for me and that is good enough. Since I am able to use a 64-bit kernel I have no reason to use 32 bit Linux kernels under Xen any more. Greets, Jacek _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel