Xen devel - Nov 2006 - Direct I/O to domU seeing a 30% performance hit

I was asked to test direct I/O to a PV domU. Since, I had a system with 
two NICs, I gave one to a domU and one dom0. (Each is running the same 
kernel: xen 3.0.3 x86_64.)

I''m running netperf from an outside system to the domU and dom0 and I
am
seeing 30% less throughput for the domU vs dom0.

Is this to be expected? If so, why? If not, does anyone have a guess as 
to what I might be doing wrong or what the issue might be?

Thanks,

John Byrne






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I was asked to test direct I/O to a PV domU. Since, I had a system
with
> two NICs, I gave one to a domU and one dom0. (Each is running the same
> kernel: xen 3.0.3 x86_64.)
> 
> I''m running netperf from an outside system to the domU and dom0
and I
am
> seeing 30% less throughput for the domU vs dom0.
That doesn''t make sense -- they should be identical. Is there other
stuff going on in the system that could be taking CPU time away from the
domU?

Ian

 
> Is this to be expected? If so, why? If not, does anyone have a guess
as
> to what I might be doing wrong or what the issue might be?
> 
> Thanks,
> 
> John Byrne
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

There have been a couple of network receive throughput
performance regressions to domUs over time that were
subsequently fixed. I think one may have crept in to
3.0.3.

Are you seeing any dropped packets on the vif associated
with your domU in your dom0? If so, propagating changeset
11861 from unstable may help:

changeset:   11861:637eace6d5c6
user:        kfraser@localhost.localdomain
date:        Mon Oct 23 11:20:37 2006 +0100
summary:     [NET] back: Fix packet queuing so that packets are drained if the


In the past, we also had receive throughput issues to
domUs that were due to socket buffer size logic but those
were fixed a while ago.

Can you send netstat -i output from dom0?

Emmanuel.


On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne
wrote:
> 
> I was asked to test direct I/O to a PV domU. Since, I had a system with 
> two NICs, I gave one to a domU and one dom0. (Each is running the same 
> kernel: xen 3.0.3 x86_64.)
> 
> I''m running netperf from an outside system to the domU and dom0
and I am
> seeing 30% less throughput for the domU vs dom0.
> 
> Is this to be expected? If so, why? If not, does anyone have a guess as 
> to what I might be doing wrong or what the issue might be?
> 
> Thanks,
> 
> John Byrne
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> There have been a couple of network receive throughput 
> performance regressions to domUs over time that were 
> subsequently fixed. I think one may have crept in to 3.0.3.
The report was (I believe) with a NIC directly assigned to the domU, so
not using netfront/back at all.

John: please can you give more details on your config.

Ian
 
> Are you seeing any dropped packets on the vif associated with 
> your domU in your dom0? If so, propagating changeset
> 11861 from unstable may help:
> 
> changeset:   11861:637eace6d5c6
> user:        kfraser@localhost.localdomain
> date:        Mon Oct 23 11:20:37 2006 +0100
> summary:     [NET] back: Fix packet queuing so that packets 
> are drained if the
> 
> 
> In the past, we also had receive throughput issues to domUs 
> that were due to socket buffer size logic but those were 
> fixed a while ago.
> 
> Can you send netstat -i output from dom0?
> 
> Emmanuel.
> 
> 
> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
> > 
> > I was asked to test direct I/O to a PV domU. Since, I had a system 
> > with two NICs, I gave one to a domU and one dom0. (Each is 
> running the 
> > same
> > kernel: xen 3.0.3 x86_64.)
> > 
> > I''m running netperf from an outside system to the domU and 
> dom0 and I 
> > am seeing 30% less throughput for the domU vs dom0.
> > 
> > Is this to be expected? If so, why? If not, does anyone 
> have a guess 
> > as to what I might be doing wrong or what the issue might be?
> > 
> > Thanks,
> > 
> > John Byrne
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian,

(Sorry. I sent things from the wrong e-mail address, so you''ve probably
been getting bounces. This one should work.)

My config is attached,

Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
performance of the two should be different. The obvious asymmetry is the 
disk. Since the disk isn''t direct, any disk I/O by the domU would 
certainly impact dom0, but I don''t think there should be much, if any.
I
did run a dom0 test with the domU started, but idle and there was no 
real change to dom0''s numbers.

What''s the best way to gather information about what is going on with 
the domains without perturbing them? (Or, at least, perturbing everyone 
equally.)

As to the test, I am running netperf 2.4.1 on an outside machine to the 
dom0 and the domU. (So the doms are running the netserver portion.) I 
was originally running it in the doms to the outside machine, but when 
the bad numbers showed up I moved it to the outside machine because I 
wondered if the bad numbers were due to something happening to the 
system time in domU. The numbers is the "outside" test to domU look
worse.

Thanks,

John Byrne


Ian Pratt wrote:
>  
>> There have been a couple of network receive throughput 
>> performance regressions to domUs over time that were 
>> subsequently fixed. I think one may have crept in to 3.0.3.
> 
> The report was (I believe) with a NIC directly assigned to the domU, so
> not using netfront/back at all.
> 
> John: please can you give more details on your config.
> 
> Ian
>  
>> Are you seeing any dropped packets on the vif associated with 
>> your domU in your dom0? If so, propagating changeset
>> 11861 from unstable may help:
>>
>> changeset:   11861:637eace6d5c6
>> user:        kfraser@localhost.localdomain
>> date:        Mon Oct 23 11:20:37 2006 +0100
>> summary:     [NET] back: Fix packet queuing so that packets 
>> are drained if the
>>
>>
>> In the past, we also had receive throughput issues to domUs 
>> that were due to socket buffer size logic but those were 
>> fixed a while ago.
>>
>> Can you send netstat -i output from dom0?
>>
>> Emmanuel.
>>
>>
>> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
>>> I was asked to test direct I/O to a PV domU. Since, I had a system 
>>> with two NICs, I gave one to a domU and one dom0. (Each is 
>> running the 
>>> same
>>> kernel: xen 3.0.3 x86_64.)
>>>
>>> I''m running netperf from an outside system to the domU and
>> dom0 and I 
>>> am seeing 30% less throughput for the domU vs dom0.
>>>
>>> Is this to be expected? If so, why? If not, does anyone 
>> have a guess 
>>> as to what I might be doing wrong or what the issue might be?
>>>
>>> Thanks,
>>>
>>> John Byrne
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
> performance of the two should be different. The obvious asymmetry is
the
> disk. Since the disk isn''t direct, any disk I/O by the domU would
> certainly impact dom0, but I don''t think there should be much, if
any.
I
> did run a dom0 test with the domU started, but idle and there was no
> real change to dom0''s numbers.
> 
> What''s the best way to gather information about what is going on
with
> the domains without perturbing them? (Or, at least, perturbing
everyone
> equally.)
> 
> As to the test, I am running netperf 2.4.1 on an outside machine to
the
> dom0 and the domU. (So the doms are running the netserver portion.) I
> was originally running it in the doms to the outside machine, but when
> the bad numbers showed up I moved it to the outside machine because I
> wondered if the bad numbers were due to something happening to the
> system time in domU. The numbers is the "outside" test to domU
look
worse.


It might be worth checking that there''s no interrupt sharing happening.
While running the test against the domU, see how much CPU dom0 burns in
the same period using ''xm vcpu-list''.

To keep things simple, have dom0 and domU as uniprocessor guests.

Ian

> Ian Pratt wrote:
> >
> >> There have been a couple of network receive throughput
> >> performance regressions to domUs over time that were
> >> subsequently fixed. I think one may have crept in to 3.0.3.
> >
> > The report was (I believe) with a NIC directly assigned to the domU,
so
> > not using netfront/back at all.
> >
> > John: please can you give more details on your config.
> >
> > Ian
> >
> >> Are you seeing any dropped packets on the vif associated with
> >> your domU in your dom0? If so, propagating changeset
> >> 11861 from unstable may help:
> >>
> >> changeset:   11861:637eace6d5c6
> >> user:        kfraser@localhost.localdomain
> >> date:        Mon Oct 23 11:20:37 2006 +0100
> >> summary:     [NET] back: Fix packet queuing so that packets
> >> are drained if the
> >>
> >>
> >> In the past, we also had receive throughput issues to domUs
> >> that were due to socket buffer size logic but those were
> >> fixed a while ago.
> >>
> >> Can you send netstat -i output from dom0?
> >>
> >> Emmanuel.
> >>
> >>
> >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
> >>> I was asked to test direct I/O to a PV domU. Since, I had a
system
> >>> with two NICs, I gave one to a domU and one dom0. (Each is
> >> running the
> >>> same
> >>> kernel: xen 3.0.3 x86_64.)
> >>>
> >>> I''m running netperf from an outside system to the
domU and
> >> dom0 and I
> >>> am seeing 30% less throughput for the domU vs dom0.
> >>>
> >>> Is this to be expected? If so, why? If not, does anyone
> >> have a guess
> >>> as to what I might be doing wrong or what the issue might be?
> >>>
> >>> Thanks,
> >>>
> >>> John Byrne
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >>
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ian and John,

I''m also doing some performance analysis about Linux native, dom0 and
domU
(para-virtualized). Here are some brief comparison for 256K sequential 
read/write. The testing is done using for JBOD based on 8 Maxtor SAS Atlas 2 
15K drives with LSI SAS HBA.

256K Sequential Read
Linux Native: 559.6MB/s
Xen Domain0: 423.3MB/s
Xen DomainU: 555.9MB/s

256K Sequential Write
Linux Native: 668.9MB/s
Xen Domain0: 708.7MB/s
Xen DomainU: 373.5MB/s

Just two questions:

It seems para-virtualized DomU outperform Dom0 in terms of sequential read 
and is very to Linux native performance. However, DomU does show poor (only 
50%) sequential write performance compared with Linux native and Dom0.

Could you explain some reason behind this?

Thanks,

Liang


----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "John Byrne" <john.l.byrne@hp.com>
Cc: "xen-devel" <xen-devel@lists.xensource.com>; "Emmanuel
Ackaouy"
<ack@xensource.com>
Sent: Tuesday, November 07, 2006 10:20 AM
Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30% performance hit

> Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
> performance of the two should be different. The obvious asymmetry is
the
> disk. Since the disk isn''t direct, any disk I/O by the domU would
> certainly impact dom0, but I don''t think there should be much, if
any.
I
> did run a dom0 test with the domU started, but idle and there was no
> real change to dom0''s numbers.
>
> What''s the best way to gather information about what is going on
with
> the domains without perturbing them? (Or, at least, perturbing
everyone
> equally.)
>
> As to the test, I am running netperf 2.4.1 on an outside machine to
the
> dom0 and the domU. (So the doms are running the netserver portion.) I
> was originally running it in the doms to the outside machine, but when
> the bad numbers showed up I moved it to the outside machine because I
> wondered if the bad numbers were due to something happening to the
> system time in domU. The numbers is the "outside" test to domU
look
worse.


It might be worth checking that there''s no interrupt sharing happening.
While running the test against the domU, see how much CPU dom0 burns in
the same period using ''xm vcpu-list''.

To keep things simple, have dom0 and domU as uniprocessor guests.

Ian

> Ian Pratt wrote:
> >
> >> There have been a couple of network receive throughput
> >> performance regressions to domUs over time that were
> >> subsequently fixed. I think one may have crept in to 3.0.3.
> >
> > The report was (I believe) with a NIC directly assigned to the domU,
so
> > not using netfront/back at all.
> >
> > John: please can you give more details on your config.
> >
> > Ian
> >
> >> Are you seeing any dropped packets on the vif associated with
> >> your domU in your dom0? If so, propagating changeset
> >> 11861 from unstable may help:
> >>
> >> changeset:   11861:637eace6d5c6
> >> user:        kfraser@localhost.localdomain
> >> date:        Mon Oct 23 11:20:37 2006 +0100
> >> summary:     [NET] back: Fix packet queuing so that packets
> >> are drained if the
> >>
> >>
> >> In the past, we also had receive throughput issues to domUs
> >> that were due to socket buffer size logic but those were
> >> fixed a while ago.
> >>
> >> Can you send netstat -i output from dom0?
> >>
> >> Emmanuel.
> >>
> >>
> >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
> >>> I was asked to test direct I/O to a PV domU. Since, I had a
system
> >>> with two NICs, I gave one to a domU and one dom0. (Each is
> >> running the
> >>> same
> >>> kernel: xen 3.0.3 x86_64.)
> >>>
> >>> I''m running netperf from an outside system to the
domU and
> >> dom0 and I
> >>> am seeing 30% less throughput for the domU vs dom0.
> >>>
> >>> Is this to be expected? If so, why? If not, does anyone
> >> have a guess
> >>> as to what I might be doing wrong or what the issue might be?
> >>>
> >>> Thanks,
> >>>
> >>> John Byrne
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >>
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I''m also doing some performance analysis about Linux native, dom0
and
domU
> (para-virtualized). Here are some brief comparison for 256K sequential
> read/write. The testing is done using for JBOD based on 8 Maxtor SAS
Atlas
> 2
> 15K drives with LSI SAS HBA.
> 
> 256K Sequential Read
> Linux Native: 559.6MB/s
> Xen Domain0: 423.3MB/s
> Xen DomainU: 555.9MB/s
This doesn''t make a lot of sense. Only thing I can think of is that
there must be some extra prefetching going on in the domU case. It still
doesn''t explain why the dom0 result is so much worse than native. 

It might be worth repeating with both native and dom0 boot with
maxcpus=1.

Are you using near-identical kernels in both cases? Same drivers, same
part of the disk for the tests, etc?

How are you doing the measurement? A timed ''dd''?

Ian

 
> 256K Sequential Write
> Linux Native: 668.9MB/s
> Xen Domain0: 708.7MB/s
> Xen DomainU: 373.5MB/s
> 
> Just two questions:
> 
> It seems para-virtualized DomU outperform Dom0 in terms of sequential
read
> and is very to Linux native performance. However, DomU does show poor
(only
> 50%) sequential write performance compared with Linux native and Dom0.
> 
> Could you explain some reason behind this?
> 
> Thanks,
> 
> Liang
> 
> 
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "John Byrne" <john.l.byrne@hp.com>
> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> <ack@xensource.com>
> Sent: Tuesday, November 07, 2006 10:20 AM
> Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30% performance
hit
> 
> 
> > Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
> > performance of the two should be different. The obvious asymmetry is
> the
> > disk. Since the disk isn''t direct, any disk I/O by the domU
would
> > certainly impact dom0, but I don''t think there should be
much, if
any.
> I
> > did run a dom0 test with the domU started, but idle and there was no
> > real change to dom0''s numbers.
> >
> > What''s the best way to gather information about what is going
on
with
> > the domains without perturbing them? (Or, at least, perturbing
> everyone
> > equally.)
> >
> > As to the test, I am running netperf 2.4.1 on an outside machine to
> the
> > dom0 and the domU. (So the doms are running the netserver portion.)
I
> > was originally running it in the doms to the outside machine, but
when
> > the bad numbers showed up I moved it to the outside machine because
I
> > wondered if the bad numbers were due to something happening to the
> > system time in domU. The numbers is the "outside" test to
domU look
> worse.
> 
> 
> It might be worth checking that there''s no interrupt sharing
happening.
> While running the test against the domU, see how much CPU dom0 burns
in
> the same period using ''xm vcpu-list''.
> 
> To keep things simple, have dom0 and domU as uniprocessor guests.
> 
> Ian
> 
> 
> > Ian Pratt wrote:
> > >
> > >> There have been a couple of network receive throughput
> > >> performance regressions to domUs over time that were
> > >> subsequently fixed. I think one may have crept in to 3.0.3.
> > >
> > > The report was (I believe) with a NIC directly assigned to the
domU,
> so
> > > not using netfront/back at all.
> > >
> > > John: please can you give more details on your config.
> > >
> > > Ian
> > >
> > >> Are you seeing any dropped packets on the vif associated with
> > >> your domU in your dom0? If so, propagating changeset
> > >> 11861 from unstable may help:
> > >>
> > >> changeset:   11861:637eace6d5c6
> > >> user:        kfraser@localhost.localdomain
> > >> date:        Mon Oct 23 11:20:37 2006 +0100
> > >> summary:     [NET] back: Fix packet queuing so that packets
> > >> are drained if the
> > >>
> > >>
> > >> In the past, we also had receive throughput issues to domUs
> > >> that were due to socket buffer size logic but those were
> > >> fixed a while ago.
> > >>
> > >> Can you send netstat -i output from dom0?
> > >>
> > >> Emmanuel.
> > >>
> > >>
> > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
> > >>> I was asked to test direct I/O to a PV domU. Since, I had
a
system
> > >>> with two NICs, I gave one to a domU and one dom0. (Each
is
> > >> running the
> > >>> same
> > >>> kernel: xen 3.0.3 x86_64.)
> > >>>
> > >>> I''m running netperf from an outside system to
the domU and
> > >> dom0 and I
> > >>> am seeing 30% less throughput for the domU vs dom0.
> > >>>
> > >>> Is this to be expected? If so, why? If not, does anyone
> > >> have a guess
> > >>> as to what I might be doing wrong or what the issue might
be?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> John Byrne
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xensource.com
> > >> http://lists.xensource.com/xen-devel
> > >>
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ian,

I already set dom0_max_vcpus=1 for domain0 when I was doing testing. Also, 
Linux native kernel and domU kernel are all compiled as Uni-Processor 
mode.All the testing for Linux native, domain0 and domainU are exactly the 
same. All used Linux kernel 2.6.16.29.

Regards,

Liang

----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "Liang Yang" <multisyncfe991@hotmail.com>; "John
Byrne"
<john.l.byrne@hp.com>
Cc: "xen-devel" <xen-devel@lists.xensource.com>; "Emmanuel
Ackaouy"
<ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
Sent: Tuesday, November 07, 2006 11:06 AM
Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen DomU. Re: 
[Xen-devel] Direct I/O to domU seeing a 30% performance hit

> I''m also doing some performance analysis about Linux native, dom0
and
domU
> (para-virtualized). Here are some brief comparison for 256K sequential
> read/write. The testing is done using for JBOD based on 8 Maxtor SAS
Atlas
> 2
> 15K drives with LSI SAS HBA.
>
> 256K Sequential Read
> Linux Native: 559.6MB/s
> Xen Domain0: 423.3MB/s
> Xen DomainU: 555.9MB/s
This doesn''t make a lot of sense. Only thing I can think of is that
there must be some extra prefetching going on in the domU case. It still
doesn''t explain why the dom0 result is so much worse than native.

It might be worth repeating with both native and dom0 boot with
maxcpus=1.

Are you using near-identical kernels in both cases? Same drivers, same
part of the disk for the tests, etc?

How are you doing the measurement? A timed ''dd''?

Ian

> 256K Sequential Write
> Linux Native: 668.9MB/s
> Xen Domain0: 708.7MB/s
> Xen DomainU: 373.5MB/s
>
> Just two questions:
>
> It seems para-virtualized DomU outperform Dom0 in terms of sequential
read
> and is very to Linux native performance. However, DomU does show poor
(only
> 50%) sequential write performance compared with Linux native and Dom0.
>
> Could you explain some reason behind this?
>
> Thanks,
>
> Liang
>
>
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "John Byrne" <john.l.byrne@hp.com>
> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> <ack@xensource.com>
> Sent: Tuesday, November 07, 2006 10:20 AM
> Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30% performance
hit
>
>
> > Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
> > performance of the two should be different. The obvious asymmetry is
> the
> > disk. Since the disk isn''t direct, any disk I/O by the domU
would
> > certainly impact dom0, but I don''t think there should be
much, if
any.
> I
> > did run a dom0 test with the domU started, but idle and there was no
> > real change to dom0''s numbers.
> >
> > What''s the best way to gather information about what is going
on
with
> > the domains without perturbing them? (Or, at least, perturbing
> everyone
> > equally.)
> >
> > As to the test, I am running netperf 2.4.1 on an outside machine to
> the
> > dom0 and the domU. (So the doms are running the netserver portion.)
I
> > was originally running it in the doms to the outside machine, but
when
> > the bad numbers showed up I moved it to the outside machine because
I
> > wondered if the bad numbers were due to something happening to the
> > system time in domU. The numbers is the "outside" test to
domU look
> worse.
>
>
> It might be worth checking that there''s no interrupt sharing
happening.
> While running the test against the domU, see how much CPU dom0 burns
in
> the same period using ''xm vcpu-list''.
>
> To keep things simple, have dom0 and domU as uniprocessor guests.
>
> Ian
>
>
> > Ian Pratt wrote:
> > >
> > >> There have been a couple of network receive throughput
> > >> performance regressions to domUs over time that were
> > >> subsequently fixed. I think one may have crept in to 3.0.3.
> > >
> > > The report was (I believe) with a NIC directly assigned to the
domU,
> so
> > > not using netfront/back at all.
> > >
> > > John: please can you give more details on your config.
> > >
> > > Ian
> > >
> > >> Are you seeing any dropped packets on the vif associated with
> > >> your domU in your dom0? If so, propagating changeset
> > >> 11861 from unstable may help:
> > >>
> > >> changeset:   11861:637eace6d5c6
> > >> user:        kfraser@localhost.localdomain
> > >> date:        Mon Oct 23 11:20:37 2006 +0100
> > >> summary:     [NET] back: Fix packet queuing so that packets
> > >> are drained if the
> > >>
> > >>
> > >> In the past, we also had receive throughput issues to domUs
> > >> that were due to socket buffer size logic but those were
> > >> fixed a while ago.
> > >>
> > >> Can you send netstat -i output from dom0?
> > >>
> > >> Emmanuel.
> > >>
> > >>
> > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
> > >>> I was asked to test direct I/O to a PV domU. Since, I had
a
system
> > >>> with two NICs, I gave one to a domU and one dom0. (Each
is
> > >> running the
> > >>> same
> > >>> kernel: xen 3.0.3 x86_64.)
> > >>>
> > >>> I''m running netperf from an outside system to
the domU and
> > >> dom0 and I
> > >>> am seeing 30% less throughput for the domU vs dom0.
> > >>>
> > >>> Is this to be expected? If so, why? If not, does anyone
> > >> have a guess
> > >>> as to what I might be doing wrong or what the issue might
be?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> John Byrne
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xensource.com
> > >> http://lists.xensource.com/xen-devel
> > >>
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ian,

I forget to mention the benchmark tool I was using is IOMeter which support 
setting up larger queue depth of
outstanding I/O to saturate the disk. Single dd will not saturate the disk.

I already set dom0_max_vcpus=1 for domain0 when I was doing testing. Also,
Linux native kernel and domU kernel are all compiled as Uni-Processor
mode.All the testing for Linux native, domain0 and domainU are exactly the
same. All used Linux kernel 2.6.16.29.

Regards,

Liang

----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "Liang Yang" <multisyncfe991@hotmail.com>; "John
Byrne"
<john.l.byrne@hp.com>
Cc: "xen-devel" <xen-devel@lists.xensource.com>; "Emmanuel
Ackaouy"
<ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
Sent: Tuesday, November 07, 2006 11:06 AM
Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen DomU. Re: 
[Xen-devel] Direct I/O to domU seeing a 30% performance hit

> I''m also doing some performance analysis about Linux native, dom0
and
domU
> (para-virtualized). Here are some brief comparison for 256K sequential
> read/write. The testing is done using for JBOD based on 8 Maxtor SAS
Atlas
> 2
> 15K drives with LSI SAS HBA.
>
> 256K Sequential Read
> Linux Native: 559.6MB/s
> Xen Domain0: 423.3MB/s
> Xen DomainU: 555.9MB/s
This doesn''t make a lot of sense. Only thing I can think of is that
there must be some extra prefetching going on in the domU case. It still
doesn''t explain why the dom0 result is so much worse than native.

It might be worth repeating with both native and dom0 boot with
maxcpus=1.

Are you using near-identical kernels in both cases? Same drivers, same
part of the disk for the tests, etc?

How are you doing the measurement? A timed ''dd''?

Ian

> 256K Sequential Write
> Linux Native: 668.9MB/s
> Xen Domain0: 708.7MB/s
> Xen DomainU: 373.5MB/s
>
> Just two questions:
>
> It seems para-virtualized DomU outperform Dom0 in terms of sequential
read
> and is very to Linux native performance. However, DomU does show poor
(only
> 50%) sequential write performance compared with Linux native and Dom0.
>
> Could you explain some reason behind this?
>
> Thanks,
>
> Liang
>
>
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "John Byrne" <john.l.byrne@hp.com>
> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> <ack@xensource.com>
> Sent: Tuesday, November 07, 2006 10:20 AM
> Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30% performance
hit
>
>
> > Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
> > performance of the two should be different. The obvious asymmetry is
> the
> > disk. Since the disk isn''t direct, any disk I/O by the domU
would
> > certainly impact dom0, but I don''t think there should be
much, if
any.
> I
> > did run a dom0 test with the domU started, but idle and there was no
> > real change to dom0''s numbers.
> >
> > What''s the best way to gather information about what is going
on
with
> > the domains without perturbing them? (Or, at least, perturbing
> everyone
> > equally.)
> >
> > As to the test, I am running netperf 2.4.1 on an outside machine to
> the
> > dom0 and the domU. (So the doms are running the netserver portion.)
I
> > was originally running it in the doms to the outside machine, but
when
> > the bad numbers showed up I moved it to the outside machine because
I
> > wondered if the bad numbers were due to something happening to the
> > system time in domU. The numbers is the "outside" test to
domU look
> worse.
>
>
> It might be worth checking that there''s no interrupt sharing
happening.
> While running the test against the domU, see how much CPU dom0 burns
in
> the same period using ''xm vcpu-list''.
>
> To keep things simple, have dom0 and domU as uniprocessor guests.
>
> Ian
>
>
> > Ian Pratt wrote:
> > >
> > >> There have been a couple of network receive throughput
> > >> performance regressions to domUs over time that were
> > >> subsequently fixed. I think one may have crept in to 3.0.3.
> > >
> > > The report was (I believe) with a NIC directly assigned to the
domU,
> so
> > > not using netfront/back at all.
> > >
> > > John: please can you give more details on your config.
> > >
> > > Ian
> > >
> > >> Are you seeing any dropped packets on the vif associated with
> > >> your domU in your dom0? If so, propagating changeset
> > >> 11861 from unstable may help:
> > >>
> > >> changeset:   11861:637eace6d5c6
> > >> user:        kfraser@localhost.localdomain
> > >> date:        Mon Oct 23 11:20:37 2006 +0100
> > >> summary:     [NET] back: Fix packet queuing so that packets
> > >> are drained if the
> > >>
> > >>
> > >> In the past, we also had receive throughput issues to domUs
> > >> that were due to socket buffer size logic but those were
> > >> fixed a while ago.
> > >>
> > >> Can you send netstat -i output from dom0?
> > >>
> > >> Emmanuel.
> > >>
> > >>
> > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
> > >>> I was asked to test direct I/O to a PV domU. Since, I had
a
system
> > >>> with two NICs, I gave one to a domU and one dom0. (Each
is
> > >> running the
> > >>> same
> > >>> kernel: xen 3.0.3 x86_64.)
> > >>>
> > >>> I''m running netperf from an outside system to
the domU and
> > >> dom0 and I
> > >>> am seeing 30% less throughput for the domU vs dom0.
> > >>>
> > >>> Is this to be expected? If so, why? If not, does anyone
> > >> have a guess
> > >>> as to what I might be doing wrong or what the issue might
be?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> John Byrne
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xensource.com
> > >> http://lists.xensource.com/xen-devel
> > >>
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I already set dom0_max_vcpus=1 for domain0 when I was doing testing.
Also,
> Linux native kernel and domU kernel are all compiled as Uni-Processor
> mode.All the testing for Linux native, domain0 and domainU are exactly
the
> same. All used Linux kernel 2.6.16.29.
Please could you post a ''diff'' of the two kernel configs.

It might be worth diff''ing the boot messages in both cases too.

Thanks,
Ian

 
> Regards,
> 
> Liang
> 
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "Liang Yang" <multisyncfe991@hotmail.com>; "John
Byrne"
> <john.l.byrne@hp.com>
> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> <ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
> Sent: Tuesday, November 07, 2006 11:06 AM
> Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen
DomU.
> Re:
> [Xen-devel] Direct I/O to domU seeing a 30% performance hit
> 
> 
> > I''m also doing some performance analysis about Linux native,
dom0
and
> domU
> > (para-virtualized). Here are some brief comparison for 256K
sequential
> > read/write. The testing is done using for JBOD based on 8 Maxtor SAS
> Atlas
> > 2
> > 15K drives with LSI SAS HBA.
> >
> > 256K Sequential Read
> > Linux Native: 559.6MB/s
> > Xen Domain0: 423.3MB/s
> > Xen DomainU: 555.9MB/s
> 
> This doesn''t make a lot of sense. Only thing I can think of is
that
> there must be some extra prefetching going on in the domU case. It
still
> doesn''t explain why the dom0 result is so much worse than native.
> 
> It might be worth repeating with both native and dom0 boot with
> maxcpus=1.
> 
> Are you using near-identical kernels in both cases? Same drivers, same
> part of the disk for the tests, etc?
> 
> How are you doing the measurement? A timed ''dd''?
> 
> Ian
> 
> 
> > 256K Sequential Write
> > Linux Native: 668.9MB/s
> > Xen Domain0: 708.7MB/s
> > Xen DomainU: 373.5MB/s
> >
> > Just two questions:
> >
> > It seems para-virtualized DomU outperform Dom0 in terms of
sequential
> read
> > and is very to Linux native performance. However, DomU does show
poor
> (only
> > 50%) sequential write performance compared with Linux native and
Dom0.
> >
> > Could you explain some reason behind this?
> >
> > Thanks,
> >
> > Liang
> >
> >
> > ----- Original Message -----
> > From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> > To: "John Byrne" <john.l.byrne@hp.com>
> > Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> > <ack@xensource.com>
> > Sent: Tuesday, November 07, 2006 10:20 AM
> > Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30% performance
> hit
> >
> >
> > > Both dom0 and the domU are SLES 10, so I don''t know why
the "idle"
> > > performance of the two should be different. The obvious asymmetry
is
> > the
> > > disk. Since the disk isn''t direct, any disk I/O by the
domU would
> > > certainly impact dom0, but I don''t think there should be
much, if
> any.
> > I
> > > did run a dom0 test with the domU started, but idle and there was
no
> > > real change to dom0''s numbers.
> > >
> > > What''s the best way to gather information about what is
going on
> with
> > > the domains without perturbing them? (Or, at least, perturbing
> > everyone
> > > equally.)
> > >
> > > As to the test, I am running netperf 2.4.1 on an outside machine
to
> > the
> > > dom0 and the domU. (So the doms are running the netserver
portion.)
> I
> > > was originally running it in the doms to the outside machine, but
> when
> > > the bad numbers showed up I moved it to the outside machine
because
> I
> > > wondered if the bad numbers were due to something happening to
the
> > > system time in domU. The numbers is the "outside" test
to domU
look
> > worse.
> >
> >
> > It might be worth checking that there''s no interrupt sharing
> happening.
> > While running the test against the domU, see how much CPU dom0 burns
> in
> > the same period using ''xm vcpu-list''.
> >
> > To keep things simple, have dom0 and domU as uniprocessor guests.
> >
> > Ian
> >
> >
> > > Ian Pratt wrote:
> > > >
> > > >> There have been a couple of network receive throughput
> > > >> performance regressions to domUs over time that were
> > > >> subsequently fixed. I think one may have crept in to
3.0.3.
> > > >
> > > > The report was (I believe) with a NIC directly assigned to
the
> domU,
> > so
> > > > not using netfront/back at all.
> > > >
> > > > John: please can you give more details on your config.
> > > >
> > > > Ian
> > > >
> > > >> Are you seeing any dropped packets on the vif associated
with
> > > >> your domU in your dom0? If so, propagating changeset
> > > >> 11861 from unstable may help:
> > > >>
> > > >> changeset:   11861:637eace6d5c6
> > > >> user:        kfraser@localhost.localdomain
> > > >> date:        Mon Oct 23 11:20:37 2006 +0100
> > > >> summary:     [NET] back: Fix packet queuing so that
packets
> > > >> are drained if the
> > > >>
> > > >>
> > > >> In the past, we also had receive throughput issues to
domUs
> > > >> that were due to socket buffer size logic but those were
> > > >> fixed a while ago.
> > > >>
> > > >> Can you send netstat -i output from dom0?
> > > >>
> > > >> Emmanuel.
> > > >>
> > > >>
> > > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne
wrote:
> > > >>> I was asked to test direct I/O to a PV domU. Since,
I had a
> system
> > > >>> with two NICs, I gave one to a domU and one dom0.
(Each is
> > > >> running the
> > > >>> same
> > > >>> kernel: xen 3.0.3 x86_64.)
> > > >>>
> > > >>> I''m running netperf from an outside system
to the domU and
> > > >> dom0 and I
> > > >>> am seeing 30% less throughput for the domU vs dom0.
> > > >>>
> > > >>> Is this to be expected? If so, why? If not, does
anyone
> > > >> have a guess
> > > >>> as to what I might be doing wrong or what the issue
might be?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> John Byrne
> > > >> _______________________________________________
> > > >> Xen-devel mailing list
> > > >> Xen-devel@lists.xensource.com
> > > >> http://lists.xensource.com/xen-devel
> > > >>
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Attached is the diff of the two kernel configs.

----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "Liang Yang" <yangliang_mr@hotmail.com>; "John
Byrne"
<john.l.byrne@hp.com>
Cc: "xen-devel" <xen-devel@lists.xensource.com>; "Emmanuel
Ackaouy"
<ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
Sent: Tuesday, November 07, 2006 11:15 AM
Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen DomU. Re: 
[Xen-devel] Direct I/O to domU seeing a 30% performance hit

> I already set dom0_max_vcpus=1 for domain0 when I was doing testing.
Also,
> Linux native kernel and domU kernel are all compiled as Uni-Processor
> mode.All the testing for Linux native, domain0 and domainU are exactly
the
> same. All used Linux kernel 2.6.16.29.
Please could you post a ''diff'' of the two kernel configs.

It might be worth diff''ing the boot messages in both cases too.

Thanks,
Ian

> Regards,
>
> Liang
>
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "Liang Yang" <multisyncfe991@hotmail.com>; "John
Byrne"
> <john.l.byrne@hp.com>
> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> <ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
> Sent: Tuesday, November 07, 2006 11:06 AM
> Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen
DomU.
> Re:
> [Xen-devel] Direct I/O to domU seeing a 30% performance hit
>
>
> > I''m also doing some performance analysis about Linux native,
dom0
and
> domU
> > (para-virtualized). Here are some brief comparison for 256K
sequential
> > read/write. The testing is done using for JBOD based on 8 Maxtor SAS
> Atlas
> > 2
> > 15K drives with LSI SAS HBA.
> >
> > 256K Sequential Read
> > Linux Native: 559.6MB/s
> > Xen Domain0: 423.3MB/s
> > Xen DomainU: 555.9MB/s
>
> This doesn''t make a lot of sense. Only thing I can think of is
that
> there must be some extra prefetching going on in the domU case. It
still
> doesn''t explain why the dom0 result is so much worse than native.
>
> It might be worth repeating with both native and dom0 boot with
> maxcpus=1.
>
> Are you using near-identical kernels in both cases? Same drivers, same
> part of the disk for the tests, etc?
>
> How are you doing the measurement? A timed ''dd''?
>
> Ian
>
>
> > 256K Sequential Write
> > Linux Native: 668.9MB/s
> > Xen Domain0: 708.7MB/s
> > Xen DomainU: 373.5MB/s
> >
> > Just two questions:
> >
> > It seems para-virtualized DomU outperform Dom0 in terms of
sequential
> read
> > and is very to Linux native performance. However, DomU does show
poor
> (only
> > 50%) sequential write performance compared with Linux native and
Dom0.
> >
> > Could you explain some reason behind this?
> >
> > Thanks,
> >
> > Liang
> >
> >
> > ----- Original Message -----
> > From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> > To: "John Byrne" <john.l.byrne@hp.com>
> > Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> > <ack@xensource.com>
> > Sent: Tuesday, November 07, 2006 10:20 AM
> > Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30% performance
> hit
> >
> >
> > > Both dom0 and the domU are SLES 10, so I don''t know why
the "idle"
> > > performance of the two should be different. The obvious asymmetry
is
> > the
> > > disk. Since the disk isn''t direct, any disk I/O by the
domU would
> > > certainly impact dom0, but I don''t think there should be
much, if
> any.
> > I
> > > did run a dom0 test with the domU started, but idle and there was
no
> > > real change to dom0''s numbers.
> > >
> > > What''s the best way to gather information about what is
going on
> with
> > > the domains without perturbing them? (Or, at least, perturbing
> > everyone
> > > equally.)
> > >
> > > As to the test, I am running netperf 2.4.1 on an outside machine
to
> > the
> > > dom0 and the domU. (So the doms are running the netserver
portion.)
> I
> > > was originally running it in the doms to the outside machine, but
> when
> > > the bad numbers showed up I moved it to the outside machine
because
> I
> > > wondered if the bad numbers were due to something happening to
the
> > > system time in domU. The numbers is the "outside" test
to domU
look
> > worse.
> >
> >
> > It might be worth checking that there''s no interrupt sharing
> happening.
> > While running the test against the domU, see how much CPU dom0 burns
> in
> > the same period using ''xm vcpu-list''.
> >
> > To keep things simple, have dom0 and domU as uniprocessor guests.
> >
> > Ian
> >
> >
> > > Ian Pratt wrote:
> > > >
> > > >> There have been a couple of network receive throughput
> > > >> performance regressions to domUs over time that were
> > > >> subsequently fixed. I think one may have crept in to
3.0.3.
> > > >
> > > > The report was (I believe) with a NIC directly assigned to
the
> domU,
> > so
> > > > not using netfront/back at all.
> > > >
> > > > John: please can you give more details on your config.
> > > >
> > > > Ian
> > > >
> > > >> Are you seeing any dropped packets on the vif associated
with
> > > >> your domU in your dom0? If so, propagating changeset
> > > >> 11861 from unstable may help:
> > > >>
> > > >> changeset:   11861:637eace6d5c6
> > > >> user:        kfraser@localhost.localdomain
> > > >> date:        Mon Oct 23 11:20:37 2006 +0100
> > > >> summary:     [NET] back: Fix packet queuing so that
packets
> > > >> are drained if the
> > > >>
> > > >>
> > > >> In the past, we also had receive throughput issues to
domUs
> > > >> that were due to socket buffer size logic but those were
> > > >> fixed a while ago.
> > > >>
> > > >> Can you send netstat -i output from dom0?
> > > >>
> > > >> Emmanuel.
> > > >>
> > > >>
> > > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne
wrote:
> > > >>> I was asked to test direct I/O to a PV domU. Since,
I had a
> system
> > > >>> with two NICs, I gave one to a domU and one dom0.
(Each is
> > > >> running the
> > > >>> same
> > > >>> kernel: xen 3.0.3 x86_64.)
> > > >>>
> > > >>> I''m running netperf from an outside system
to the domU and
> > > >> dom0 and I
> > > >>> am seeing 30% less throughput for the domU vs dom0.
> > > >>>
> > > >>> Is this to be expected? If so, why? If not, does
anyone
> > > >> have a guess
> > > >>> as to what I might be doing wrong or what the issue
might be?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> John Byrne
> > > >> _______________________________________________
> > > >> Xen-devel mailing list
> > > >> Xen-devel@lists.xensource.com
> > > >> http://lists.xensource.com/xen-devel
> > > >>
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

It looks like you''re using totally different disk schedulers:

90c87
< CONFIG_DEFAULT_CFQ=y
---
> # CONFIG_DEFAULT_CFQ is not set
92c89
< CONFIG_DEFAULT_IOSCHED="cfq"
---
> CONFIG_DEFAULT_IOSCHED="anticipatory"
Try changing them both to the same thing, and seeing what happens...

 -George

On 11/7/06, Liang Yang <multisyncfe991@hotmail.com>
wrote:
> Attached is the diff of the two kernel configs.
>
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "Liang Yang" <yangliang_mr@hotmail.com>; "John
Byrne"
> <john.l.byrne@hp.com>
> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> <ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
> Sent: Tuesday, November 07, 2006 11:15 AM
> Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen DomU.
Re:
> [Xen-devel] Direct I/O to domU seeing a 30% performance hit
>
>
> > I already set dom0_max_vcpus=1 for domain0 when I was doing testing.
> Also,
> > Linux native kernel and domU kernel are all compiled as Uni-Processor
> > mode.All the testing for Linux native, domain0 and domainU are exactly
> the
> > same. All used Linux kernel 2.6.16.29.
>
> Please could you post a ''diff'' of the two kernel configs.
>
> It might be worth diff''ing the boot messages in both cases too.
>
> Thanks,
> Ian
>
>
> > Regards,
> >
> > Liang
> >
> > ----- Original Message -----
> > From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> > To: "Liang Yang" <multisyncfe991@hotmail.com>;
"John Byrne"
> > <john.l.byrne@hp.com>
> > Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> > <ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
> > Sent: Tuesday, November 07, 2006 11:06 AM
> > Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen
> DomU.
> > Re:
> > [Xen-devel] Direct I/O to domU seeing a 30% performance hit
> >
> >
> > > I''m also doing some performance analysis about Linux
native, dom0
> and
> > domU
> > > (para-virtualized). Here are some brief comparison for 256K
> sequential
> > > read/write. The testing is done using for JBOD based on 8 Maxtor
SAS
> > Atlas
> > > 2
> > > 15K drives with LSI SAS HBA.
> > >
> > > 256K Sequential Read
> > > Linux Native: 559.6MB/s
> > > Xen Domain0: 423.3MB/s
> > > Xen DomainU: 555.9MB/s
> >
> > This doesn''t make a lot of sense. Only thing I can think of
is that
> > there must be some extra prefetching going on in the domU case. It
> still
> > doesn''t explain why the dom0 result is so much worse than
native.
> >
> > It might be worth repeating with both native and dom0 boot with
> > maxcpus=1.
> >
> > Are you using near-identical kernels in both cases? Same drivers, same
> > part of the disk for the tests, etc?
> >
> > How are you doing the measurement? A timed ''dd''?
> >
> > Ian
> >
> >
> > > 256K Sequential Write
> > > Linux Native: 668.9MB/s
> > > Xen Domain0: 708.7MB/s
> > > Xen DomainU: 373.5MB/s
> > >
> > > Just two questions:
> > >
> > > It seems para-virtualized DomU outperform Dom0 in terms of
> sequential
> > read
> > > and is very to Linux native performance. However, DomU does show
> poor
> > (only
> > > 50%) sequential write performance compared with Linux native and
> Dom0.
> > >
> > > Could you explain some reason behind this?
> > >
> > > Thanks,
> > >
> > > Liang
> > >
> > >
> > > ----- Original Message -----
> > > From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> > > To: "John Byrne" <john.l.byrne@hp.com>
> > > Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
> > > <ack@xensource.com>
> > > Sent: Tuesday, November 07, 2006 10:20 AM
> > > Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30%
performance
> > hit
> > >
> > >
> > > > Both dom0 and the domU are SLES 10, so I don''t know
why the "idle"
> > > > performance of the two should be different. The obvious
asymmetry
> is
> > > the
> > > > disk. Since the disk isn''t direct, any disk I/O by
the domU would
> > > > certainly impact dom0, but I don''t think there
should be much, if
> > any.
> > > I
> > > > did run a dom0 test with the domU started, but idle and
there was
> no
> > > > real change to dom0''s numbers.
> > > >
> > > > What''s the best way to gather information about
what is going on
> > with
> > > > the domains without perturbing them? (Or, at least,
perturbing
> > > everyone
> > > > equally.)
> > > >
> > > > As to the test, I am running netperf 2.4.1 on an outside
machine
> to
> > > the
> > > > dom0 and the domU. (So the doms are running the netserver
> portion.)
> > I
> > > > was originally running it in the doms to the outside
machine, but
> > when
> > > > the bad numbers showed up I moved it to the outside machine
> because
> > I
> > > > wondered if the bad numbers were due to something happening
to the
> > > > system time in domU. The numbers is the "outside"
test to domU
> look
> > > worse.
> > >
> > >
> > > It might be worth checking that there''s no interrupt
sharing
> > happening.
> > > While running the test against the domU, see how much CPU dom0
burns
> > in
> > > the same period using ''xm vcpu-list''.
> > >
> > > To keep things simple, have dom0 and domU as uniprocessor guests.
> > >
> > > Ian
> > >
> > >
> > > > Ian Pratt wrote:
> > > > >
> > > > >> There have been a couple of network receive
throughput
> > > > >> performance regressions to domUs over time that
were
> > > > >> subsequently fixed. I think one may have crept in
to 3.0.3.
> > > > >
> > > > > The report was (I believe) with a NIC directly assigned
to the
> > domU,
> > > so
> > > > > not using netfront/back at all.
> > > > >
> > > > > John: please can you give more details on your config.
> > > > >
> > > > > Ian
> > > > >
> > > > >> Are you seeing any dropped packets on the vif
associated with
> > > > >> your domU in your dom0? If so, propagating
changeset
> > > > >> 11861 from unstable may help:
> > > > >>
> > > > >> changeset:   11861:637eace6d5c6
> > > > >> user:        kfraser@localhost.localdomain
> > > > >> date:        Mon Oct 23 11:20:37 2006 +0100
> > > > >> summary:     [NET] back: Fix packet queuing so that
packets
> > > > >> are drained if the
> > > > >>
> > > > >>
> > > > >> In the past, we also had receive throughput issues
to domUs
> > > > >> that were due to socket buffer size logic but those
were
> > > > >> fixed a while ago.
> > > > >>
> > > > >> Can you send netstat -i output from dom0?
> > > > >>
> > > > >> Emmanuel.
> > > > >>
> > > > >>
> > > > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John
Byrne wrote:
> > > > >>> I was asked to test direct I/O to a PV domU.
Since, I had a
> > system
> > > > >>> with two NICs, I gave one to a domU and one
dom0. (Each is
> > > > >> running the
> > > > >>> same
> > > > >>> kernel: xen 3.0.3 x86_64.)
> > > > >>>
> > > > >>> I''m running netperf from an outside
system to the domU and
> > > > >> dom0 and I
> > > > >>> am seeing 30% less throughput for the domU vs
dom0.
> > > > >>>
> > > > >>> Is this to be expected? If so, why? If not,
does anyone
> > > > >> have a guess
> > > > >>> as to what I might be doing wrong or what the
issue might be?
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> John Byrne
> > > > >> _______________________________________________
> > > > >> Xen-devel mailing list
> > > > >> Xen-devel@lists.xensource.com
> > > > >> http://lists.xensource.com/xen-devel
> > > > >>
> > > > >
> > > > > _______________________________________________
> > > > > Xen-devel mailing list
> > > > > Xen-devel@lists.xensource.com
> > > > > http://lists.xensource.com/xen-devel
> > > > >
> > >
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> >
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I already changed the Linux I/O scheduler for all block devices to 
Anticipatory before I start the testing. This is done runtime.

Liang

----- Original Message ----- 
From: "George Dunlap" <gdunlap@xensource.com>
To: "Liang Yang" <multisyncfe991@hotmail.com>
Cc: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>; "John
Byrne"
<john.l.byrne@hp.com>; "xen-devel"
<xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy" <ack@xensource.com>
Sent: Tuesday, November 07, 2006 11:40 AM
Subject: Re: Performance data of Linux native vs. Xen Dom0 and Xen DomU. Re: 
[Xen-devel] Direct I/O to domU seeing a 30% performance hit

> It looks like you''re using totally different disk schedulers:
>
> 90c87
> < CONFIG_DEFAULT_CFQ=y
> ---
>> # CONFIG_DEFAULT_CFQ is not set
> 92c89
> < CONFIG_DEFAULT_IOSCHED="cfq"
> ---
>> CONFIG_DEFAULT_IOSCHED="anticipatory"
>
> Try changing them both to the same thing, and seeing what happens...
>
> -George
>
> On 11/7/06, Liang Yang <multisyncfe991@hotmail.com> wrote:
>> Attached is the diff of the two kernel configs.
>>
>> ----- Original Message -----
>> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
>> To: "Liang Yang" <yangliang_mr@hotmail.com>; "John
Byrne"
>> <john.l.byrne@hp.com>
>> Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
>> <ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
>> Sent: Tuesday, November 07, 2006 11:15 AM
>> Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen
DomU.
>> Re:
>> [Xen-devel] Direct I/O to domU seeing a 30% performance hit
>>
>>
>> > I already set dom0_max_vcpus=1 for domain0 when I was doing
testing.
>> Also,
>> > Linux native kernel and domU kernel are all compiled as
Uni-Processor
>> > mode.All the testing for Linux native, domain0 and domainU are
exactly
>> the
>> > same. All used Linux kernel 2.6.16.29.
>>
>> Please could you post a ''diff'' of the two kernel
configs.
>>
>> It might be worth diff''ing the boot messages in both cases
too.
>>
>> Thanks,
>> Ian
>>
>>
>> > Regards,
>> >
>> > Liang
>> >
>> > ----- Original Message -----
>> > From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
>> > To: "Liang Yang" <multisyncfe991@hotmail.com>;
"John Byrne"
>> > <john.l.byrne@hp.com>
>> > Cc: "xen-devel" <xen-devel@lists.xensource.com>;
"Emmanuel Ackaouy"
>> > <ack@xensource.com>; <ian.pratt@cl.cam.ac.uk>
>> > Sent: Tuesday, November 07, 2006 11:06 AM
>> > Subject: RE: Performance data of Linux native vs. Xen Dom0 and Xen
>> DomU.
>> > Re:
>> > [Xen-devel] Direct I/O to domU seeing a 30% performance hit
>> >
>> >
>> > > I''m also doing some performance analysis about Linux
native, dom0
>> and
>> > domU
>> > > (para-virtualized). Here are some brief comparison for 256K
>> sequential
>> > > read/write. The testing is done using for JBOD based on 8
Maxtor SAS
>> > Atlas
>> > > 2
>> > > 15K drives with LSI SAS HBA.
>> > >
>> > > 256K Sequential Read
>> > > Linux Native: 559.6MB/s
>> > > Xen Domain0: 423.3MB/s
>> > > Xen DomainU: 555.9MB/s
>> >
>> > This doesn''t make a lot of sense. Only thing I can think
of is that
>> > there must be some extra prefetching going on in the domU case. It
>> still
>> > doesn''t explain why the dom0 result is so much worse than
native.
>> >
>> > It might be worth repeating with both native and dom0 boot with
>> > maxcpus=1.
>> >
>> > Are you using near-identical kernels in both cases? Same drivers,
same
>> > part of the disk for the tests, etc?
>> >
>> > How are you doing the measurement? A timed ''dd''?
>> >
>> > Ian
>> >
>> >
>> > > 256K Sequential Write
>> > > Linux Native: 668.9MB/s
>> > > Xen Domain0: 708.7MB/s
>> > > Xen DomainU: 373.5MB/s
>> > >
>> > > Just two questions:
>> > >
>> > > It seems para-virtualized DomU outperform Dom0 in terms of
>> sequential
>> > read
>> > > and is very to Linux native performance. However, DomU does
show
>> poor
>> > (only
>> > > 50%) sequential write performance compared with Linux native
and
>> Dom0.
>> > >
>> > > Could you explain some reason behind this?
>> > >
>> > > Thanks,
>> > >
>> > > Liang
>> > >
>> > >
>> > > ----- Original Message -----
>> > > From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
>> > > To: "John Byrne" <john.l.byrne@hp.com>
>> > > Cc: "xen-devel"
<xen-devel@lists.xensource.com>; "Emmanuel Ackaouy"
>> > > <ack@xensource.com>
>> > > Sent: Tuesday, November 07, 2006 10:20 AM
>> > > Subject: RE: [Xen-devel] Direct I/O to domU seeing a 30%
performance
>> > hit
>> > >
>> > >
>> > > > Both dom0 and the domU are SLES 10, so I don''t
know why the "idle"
>> > > > performance of the two should be different. The obvious
asymmetry
>> is
>> > > the
>> > > > disk. Since the disk isn''t direct, any disk I/O
by the domU would
>> > > > certainly impact dom0, but I don''t think there
should be much, if
>> > any.
>> > > I
>> > > > did run a dom0 test with the domU started, but idle and
there was
>> no
>> > > > real change to dom0''s numbers.
>> > > >
>> > > > What''s the best way to gather information about
what is going on
>> > with
>> > > > the domains without perturbing them? (Or, at least,
perturbing
>> > > everyone
>> > > > equally.)
>> > > >
>> > > > As to the test, I am running netperf 2.4.1 on an outside
machine
>> to
>> > > the
>> > > > dom0 and the domU. (So the doms are running the
netserver
>> portion.)
>> > I
>> > > > was originally running it in the doms to the outside
machine, but
>> > when
>> > > > the bad numbers showed up I moved it to the outside
machine
>> because
>> > I
>> > > > wondered if the bad numbers were due to something
happening to the
>> > > > system time in domU. The numbers is the
"outside" test to domU
>> look
>> > > worse.
>> > >
>> > >
>> > > It might be worth checking that there''s no interrupt
sharing
>> > happening.
>> > > While running the test against the domU, see how much CPU
dom0 burns
>> > in
>> > > the same period using ''xm vcpu-list''.
>> > >
>> > > To keep things simple, have dom0 and domU as uniprocessor
guests.
>> > >
>> > > Ian
>> > >
>> > >
>> > > > Ian Pratt wrote:
>> > > > >
>> > > > >> There have been a couple of network receive
throughput
>> > > > >> performance regressions to domUs over time that
were
>> > > > >> subsequently fixed. I think one may have crept
in to 3.0.3.
>> > > > >
>> > > > > The report was (I believe) with a NIC directly
assigned to the
>> > domU,
>> > > so
>> > > > > not using netfront/back at all.
>> > > > >
>> > > > > John: please can you give more details on your
config.
>> > > > >
>> > > > > Ian
>> > > > >
>> > > > >> Are you seeing any dropped packets on the vif
associated with
>> > > > >> your domU in your dom0? If so, propagating
changeset
>> > > > >> 11861 from unstable may help:
>> > > > >>
>> > > > >> changeset:   11861:637eace6d5c6
>> > > > >> user:        kfraser@localhost.localdomain
>> > > > >> date:        Mon Oct 23 11:20:37 2006 +0100
>> > > > >> summary:     [NET] back: Fix packet queuing so
that packets
>> > > > >> are drained if the
>> > > > >>
>> > > > >>
>> > > > >> In the past, we also had receive throughput
issues to domUs
>> > > > >> that were due to socket buffer size logic but
those were
>> > > > >> fixed a while ago.
>> > > > >>
>> > > > >> Can you send netstat -i output from dom0?
>> > > > >>
>> > > > >> Emmanuel.
>> > > > >>
>> > > > >>
>> > > > >> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John
Byrne wrote:
>> > > > >>> I was asked to test direct I/O to a PV
domU. Since, I had a
>> > system
>> > > > >>> with two NICs, I gave one to a domU and one
dom0. (Each is
>> > > > >> running the
>> > > > >>> same
>> > > > >>> kernel: xen 3.0.3 x86_64.)
>> > > > >>>
>> > > > >>> I''m running netperf from an
outside system to the domU and
>> > > > >> dom0 and I
>> > > > >>> am seeing 30% less throughput for the domU
vs dom0.
>> > > > >>>
>> > > > >>> Is this to be expected? If so, why? If not,
does anyone
>> > > > >> have a guess
>> > > > >>> as to what I might be doing wrong or what
the issue might be?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>>
>> > > > >>> John Byrne
>> > > > >> _______________________________________________
>> > > > >> Xen-devel mailing list
>> > > > >> Xen-devel@lists.xensource.com
>> > > > >> http://lists.xensource.com/xen-devel
>> > > > >>
>> > > > >
>> > > > > _______________________________________________
>> > > > > Xen-devel mailing list
>> > > > > Xen-devel@lists.xensource.com
>> > > > > http://lists.xensource.com/xen-devel
>> > > > >
>> > >
>> > >
>> > > _______________________________________________
>> > > Xen-devel mailing list
>> > > Xen-devel@lists.xensource.com
>> > > http://lists.xensource.com/xen-devel
>> >
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>
>>
>>
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian,

I had screwed up and had a process running in dom0 chewing up CPU in 
dom0. I thought I had taken care of it.

After fixing that, most of the numbers for dom0, domU, and the base SLES 
kernel are within a couple of tenths of percent of each other. However, 
there are some fairly large differences in some of the runs where the 
socket buffers are small.

Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

262142 262142   4096    60.00     941.03		base
262142 262142   4096    60.00     939.95		dom0
262142 262142   4096    60.00     937.22		domU

  16384  16384  32768    60.00     379.68		base
  16384  16384  32768    60.00     350.15		dom0
  16384  16384  32768    60.00     367.89		domU

In the latter case, the divergence from the base performance is much 
larger. I assume that when the socket buffers are small, the extra 
overhead for the interrupts is showing up more because more interrupts 
are required.

Overall, though, the numbers are now acceptable. Thanks for your help. 
It allowed me to spot my goof. (Sorry about wasting your time though.)

One last question: is there an easy way to break out the amount of CPU 
time spent in the hypervisor?

Thanks,

John Byrne


Ian Pratt wrote:
>> Both dom0 and the domU are SLES 10, so I don''t know why the
"idle"
>> performance of the two should be different. The obvious asymmetry is
> the
>> disk. Since the disk isn''t direct, any disk I/O by the domU
would
>> certainly impact dom0, but I don''t think there should be much,
if any.
> I
>> did run a dom0 test with the domU started, but idle and there was no
>> real change to dom0''s numbers.
>>
>> What''s the best way to gather information about what is going
on with
>> the domains without perturbing them? (Or, at least, perturbing
> everyone
>> equally.)
>>
>> As to the test, I am running netperf 2.4.1 on an outside machine to
> the
>> dom0 and the domU. (So the doms are running the netserver portion.) I
>> was originally running it in the doms to the outside machine, but when
>> the bad numbers showed up I moved it to the outside machine because I
>> wondered if the bad numbers were due to something happening to the
>> system time in domU. The numbers is the "outside" test to
domU look
> worse.
> 
> 
> It might be worth checking that there''s no interrupt sharing
happening.
> While running the test against the domU, see how much CPU dom0 burns in
> the same period using ''xm vcpu-list''.
> 
> To keep things simple, have dom0 and domU as uniprocessor guests.
> 
> Ian
> 
> 
>> Ian Pratt wrote:
>>>> There have been a couple of network receive throughput
>>>> performance regressions to domUs over time that were
>>>> subsequently fixed. I think one may have crept in to 3.0.3.
>>> The report was (I believe) with a NIC directly assigned to the
domU,
> so
>>> not using netfront/back at all.
>>>
>>> John: please can you give more details on your config.
>>>
>>> Ian
>>>
>>>> Are you seeing any dropped packets on the vif associated with
>>>> your domU in your dom0? If so, propagating changeset
>>>> 11861 from unstable may help:
>>>>
>>>> changeset:   11861:637eace6d5c6
>>>> user:        kfraser@localhost.localdomain
>>>> date:        Mon Oct 23 11:20:37 2006 +0100
>>>> summary:     [NET] back: Fix packet queuing so that packets
>>>> are drained if the
>>>>
>>>>
>>>> In the past, we also had receive throughput issues to domUs
>>>> that were due to socket buffer size logic but those were
>>>> fixed a while ago.
>>>>
>>>> Can you send netstat -i output from dom0?
>>>>
>>>> Emmanuel.
>>>>
>>>>
>>>> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne wrote:
>>>>> I was asked to test direct I/O to a PV domU. Since, I had a
system
>>>>> with two NICs, I gave one to a domU and one dom0. (Each is
>>>> running the
>>>>> same
>>>>> kernel: xen 3.0.3 x86_64.)
>>>>>
>>>>> I''m running netperf from an outside system to the
domU and
>>>> dom0 and I
>>>>> am seeing 30% less throughput for the domU vs dom0.
>>>>>
>>>>> Is this to be expected? If so, why? If not, does anyone
>>>> have a guess
>>>>> as to what I might be doing wrong or what the issue might
be?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> John Byrne
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> One last question: is there an easy way to break out the 
> amount of CPU time spent in the hypervisor?
It may be possible to configure the CPU perf counters to record the
amount of time you spend in ring0. Otherwise, use xen-oprofile for an
estimate.

Ian
 
> Thanks,
> 
> John Byrne
> 
> 
> Ian Pratt wrote:
> >> Both dom0 and the domU are SLES 10, so I don''t know why
the "idle"
> >> performance of the two should be different. The obvious 
> asymmetry is
> > the
> >> disk. Since the disk isn''t direct, any disk I/O by the
domU would
> >> certainly impact dom0, but I don''t think there should be 
> much, if any.
> > I
> >> did run a dom0 test with the domU started, but idle and 
> there was no 
> >> real change to dom0''s numbers.
> >>
> >> What''s the best way to gather information about what is 
> going on with 
> >> the domains without perturbing them? (Or, at least, perturbing
> > everyone
> >> equally.)
> >>
> >> As to the test, I am running netperf 2.4.1 on an outside machine
to
> > the
> >> dom0 and the domU. (So the doms are running the netserver 
> portion.) I 
> >> was originally running it in the doms to the outside machine, but 
> >> when the bad numbers showed up I moved it to the outside machine 
> >> because I wondered if the bad numbers were due to 
> something happening 
> >> to the system time in domU. The numbers is the "outside"
> test to domU 
> >> look
> > worse.
> > 
> > 
> > It might be worth checking that there''s no interrupt 
> sharing happening.
> > While running the test against the domU, see how much CPU 
> dom0 burns 
> > in the same period using ''xm vcpu-list''.
> > 
> > To keep things simple, have dom0 and domU as uniprocessor guests.
> > 
> > Ian
> > 
> > 
> >> Ian Pratt wrote:
> >>>> There have been a couple of network receive throughput 
> performance 
> >>>> regressions to domUs over time that were subsequently
fixed. I
> >>>> think one may have crept in to 3.0.3.
> >>> The report was (I believe) with a NIC directly assigned 
> to the domU,
> > so
> >>> not using netfront/back at all.
> >>>
> >>> John: please can you give more details on your config.
> >>>
> >>> Ian
> >>>
> >>>> Are you seeing any dropped packets on the vif associated 
> with your 
> >>>> domU in your dom0? If so, propagating changeset
> >>>> 11861 from unstable may help:
> >>>>
> >>>> changeset:   11861:637eace6d5c6
> >>>> user:        kfraser@localhost.localdomain
> >>>> date:        Mon Oct 23 11:20:37 2006 +0100
> >>>> summary:     [NET] back: Fix packet queuing so that
packets
> >>>> are drained if the
> >>>>
> >>>>
> >>>> In the past, we also had receive throughput issues to
domUs that
> >>>> were due to socket buffer size logic but those were 
> fixed a while 
> >>>> ago.
> >>>>
> >>>> Can you send netstat -i output from dom0?
> >>>>
> >>>> Emmanuel.
> >>>>
> >>>>
> >>>> On Mon, Nov 06, 2006 at 09:55:17PM -0800, John Byrne
wrote:
> >>>>> I was asked to test direct I/O to a PV domU. Since, I 
> had a system 
> >>>>> with two NICs, I gave one to a domU and one dom0.
(Each is
> >>>> running the
> >>>>> same
> >>>>> kernel: xen 3.0.3 x86_64.)
> >>>>>
> >>>>> I''m running netperf from an outside system to
the domU and
> >>>> dom0 and I
> >>>>> am seeing 30% less throughput for the domU vs dom0.
> >>>>>
> >>>>> Is this to be expected? If so, why? If not, does
anyone
> >>>> have a guess
> >>>>> as to what I might be doing wrong or what the issue
might be?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> John Byrne
> >>>> _______________________________________________
> >>>> Xen-devel mailing list
> >>>> Xen-devel@lists.xensource.com
> >>>> http://lists.xensource.com/xen-devel
> >>>>
> >>> _______________________________________________
> >>> Xen-devel mailing list
> >>> Xen-devel@lists.xensource.com
> >>> http://lists.xensource.com/xen-devel
> >>>
> > 
> > 
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel