Hi. Originally we had tried 3.0.4-0 with loop back mounted file systems. For some reason the dom0 crashed (after running for a day or so). It did this a couple of times on one host server for us, and again once on another server. We have just tried 3.0.4-1 with tap:aio file systems on the domUs. After that we''ve gone a couple of days with no kernel crash. Good so far. However it seems that performance is a lot slower. e.g. on a domU: :/$ time sudo du -s 2020684 . real 9m25.646s user 0m0.044s sys 0m0.144s On a laptop with a puny 5400 rpm drive: $ time sudo du -s 86923472 . real 0m18.376s user 0m0.532s sys 0m10.737s And things like bonnie seem to make no real progress. And on startup things ''seem'' slower. When I run xentop the VBD* columns are all zero (cf. when we were using loop back mounted files and they showed disk IO). With xen 3.0.4 how does one identify how much IO each domU is doing? Also how does one associate a tapdisk process to a domU? Run xm block-list for each domU (seems awkward). Are there any options to tune disk IO with blk tap? On the dom0 or domU? Has anyone else experienced slower disk IO with 3.0.4/blktap? Any other information I can provide? Regards, Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Feb 14, 2007 at 12:26:32PM +1300, Peter wrote:> Originally we had tried 3.0.4-0 with loop back mounted file systems. > For some reason the dom0 crashed (after running for a day or so). It > did this a couple of times on one host server for us, and again once on > another server. > > We have just tried 3.0.4-1 with tap:aio file systems on the domUs. > After that we''ve gone a couple of days with no kernel crash. Good so far.You don''t say whether the underlying file you are pointing to is sparse or pre-allocated (non sparse) ? In the sparse case it is expected that performance is terrible - because every write requires the undering FS to allocate some more blocks - which in turn causes a journal sync. If you use non-sparse then all the blocks are pre-allocated so you don''t get the journal bottleneck.> However it seems that performance is a lot slower. > > e.g. on a domU: > :/$ time sudo du -s > 2020684 . > > real 9m25.646s > user 0m0.044s > sys 0m0.144s > > > On a laptop with a puny 5400 rpm drive: $ time sudo du -s > 86923472 . > > real 0m18.376s > user 0m0.532s > sys 0m10.737s > > And things like bonnie seem to make no real progress. > > And on startup things ''seem'' slower.Slower than what? I''d certainly expect tap:aio: to be slower than file: because file: is not actually flushing your data to disk - it hangs around in memory and is flushed by the host kernel VM as needed. Since this isn''t remotely safe for your data, its not even worth comparing tap:aio with file:.> Has anyone else experienced slower disk IO with 3.0.4/blktap?Yes, when using sparse files ontop of a journalled fs. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi.
We have hit bugs with 3.0.3 with blktap, so can''t use it there. (I saw
some fixes on the devel list that I think address these problems in 3.0.4).
3.0.4 has crashed within a hour to a day after enabling any loop back
mounted file systems. So we can''t use loop back mounted devices on
3.0.4. 3.0.3 is fine. We''d reported this via bug tracker but we
don''t
have a reproducible test case for it.
The performance on 3.0.4 (with sparse blktap files) is a lot slower
(unusably so) compared to 3.0.3 (with sparse loopback mounted files).
On the 3.0.4 server we notice that IO overall (even on dom0) is very
slow for some reason hdparm -t /dev/md1 returns about 3-8MB/s. c.f.
normally it returns 46-55 MB/s.
Any ideas what could be the culprit? Is anyone else seeing IO issues?
If IO is good on your 3.0.4 server, then care to describe your setup?
With blktap we don''t get any vbd output in xentop. And I cannot see
how
to associate the tapdisk processes in ps auxf with its associated domU.
How would one do this?
And I''ve just received the following bug (below) from 3.0.4 (tapdisk
related apparently).
Regards, Peter
------------[ cut here ]------------
kernel BUG at fs/aio.c:511!
invalid opcode: 0000 [#1]
Modules linked in: xt_physdev iptable_filter ip_tables x_tables
dm_mirror dm_mod tg3 ext3 jbd raid1
CPU: 0
EIP: 0061:[<c0173291>] Not tainted VLI
EFLAGS: 00010086 (2.6.16.33-xen0 #2)
EIP is at __aio_put_req+0x26/0x10a
eax: cb71bdc0 ebx: cb71bdc0 ecx: cb71be30 edx: ffffffff
esi: c5f62180 edi: c58b9f68 ebp: c5f62180 esp: c58b9f18
ds: 007b es: 007b ss: 0069
Process tapdisk (pid: 18570, threadinfo=c58b8000 task=c6470540)
Stack: <0>00001000 c5f62180 c5f62180 cb71bdc0 c0173393 c5f62180 cb71bdc0
00000000
cb71bdc0 c0174620 cb71bdc0 0805d400 00000001 0805d400 c5f62180
0805f184
c0174731 c58b9f68 0805d400 00000040 00000142 00000000 00000000
00000000
Call Trace:
[<c0173393>] aio_put_req+0x1e/0x63
[<c0174620>] io_submit_one+0x154/0x1d2
[<c0174731>] sys_io_submit+0x93/0xde
[<c01049e9>] syscall_call+0x7/0xb
Code: e9 87 14 0e 00 56 53 83 ec 08 8b 5c 24 18 8b 74 24 14 8b 53 0c 83
ea 01 85 d2 89 53 0c 78 0c 31 c0 85 d2 74 10 83 c4 08 5b 5e c3 <0f> 0b
ff 01 ea b3 34 c0 eb ea 8d 4b 70 8b 43 70 8b 51 04 89 50
<7>exit_aio:ioctx still alive: 2 1 0
blk_tap: Can''t get UE info!
Daniel P. Berrange wrote:> On Wed, Feb 14, 2007 at 12:26:32PM +1300, Peter wrote:
>> Originally we had tried 3.0.4-0 with loop back mounted file systems.
>> For some reason the dom0 crashed (after running for a day or so). It
>> did this a couple of times on one host server for us, and again once on
>> another server.
>>
>> We have just tried 3.0.4-1 with tap:aio file systems on the domUs.
>> After that we''ve gone a couple of days with no kernel crash.
Good so far.
>
> You don''t say whether the underlying file you are pointing to is
sparse
> or pre-allocated (non sparse) ? In the sparse case it is expected that
> performance is terrible - because every write requires the undering FS
> to allocate some more blocks - which in turn causes a journal sync. If
> you use non-sparse then all the blocks are pre-allocated so you
don''t get
> the journal bottleneck.
>
>> However it seems that performance is a lot slower.
>>
>> e.g. on a domU:
>> :/$ time sudo du -s
>> 2020684 .
>>
>> real 9m25.646s
>> user 0m0.044s
>> sys 0m0.144s
>>
>>
>> On a laptop with a puny 5400 rpm drive: $ time sudo du -s
>> 86923472 .
>>
>> real 0m18.376s
>> user 0m0.532s
>> sys 0m10.737s
>>
>> And things like bonnie seem to make no real progress.
>>
>> And on startup things ''seem'' slower.
>
> Slower than what? I''d certainly expect tap:aio: to be slower than
file:
> because file: is not actually flushing your data to disk - it hangs around
> in memory and is flushed by the host kernel VM as needed. Since this
isn''t
> remotely safe for your data, its not even worth comparing tap:aio with
file:.
>
>> Has anyone else experienced slower disk IO with 3.0.4/blktap?
>
> Yes, when using sparse files ontop of a journalled fs.
>
> Regards,
> Dan.
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Fri, Feb 16, 2007 at 02:18:49PM +1300, Peter wrote:> Hi. > > We have hit bugs with 3.0.3 with blktap, so can''t use it there. (I saw > some fixes on the devel list that I think address these problems in 3.0.4). > > 3.0.4 has crashed within a hour to a day after enabling any loop back > mounted file systems. So we can''t use loop back mounted devices on > 3.0.4. 3.0.3 is fine. We''d reported this via bug tracker but we don''t > have a reproducible test case for it. > > The performance on 3.0.4 (with sparse blktap files) is a lot slower > (unusably so) compared to 3.0.3 (with sparse loopback mounted files).As I said in my previous mail - it is meaningless to compare performance of tap:aio with file: One writes data to disk when you ask it to (blktap) the other postpones the writes indefinitely (file). Of course the one which doesn''t actually write your data to disk is faster - but that''s not useful because upon crash you can say goodbye to your data. It is doubly meaningless to compare the performance when using sparse files. Every write to a sparse file which requires extra blocks to be allocated will hit the journal - this destroys performance. If you want to look at performance, use non-sparse files with blktap, and compare against phy: Any comparison against file: is completely bogus. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users