Xen devel - Oct 2013 - Re: [HVM} xen_platform_pci=0 doesn't prevent platform device creation and disk and nic take over by PV drivers.

Wednesday, October 9, 2013, 2:00:02 PM, you wrote:
> create ^
> title it xen_platform_pci=0 doesn''t work with qemu-xen
> owner it Anthony Perard <anthony.perard@citrix.com>
> thanks
Perhaps in general looking at the libxl and xend .. and .. qemu-xen and
qemu-xen-traditonal compatibility shoud be added too.

Perhaps i''m a bit blunt ..
but for users it''s quite a mess and PITA now for quite some time, the
transition of both is now smeared out over quite some major and minor releases.

With some features not working since .. some features are working again but have
a gap of not working,
since old features becoming usable again are (understandably) not backported.

This seems to be at a level that it is even undocumentable ... you would need a
spreadsheet
with every minor version to try to document that ... it seems to be leading to a
lot of user questions as well.

I''m not blaming any one personally or anything like that .. just
expressing my worries and the threat i see for Xen as a project in general.
Namely users sop

I understand that new features like the arm port are interesting .. but i think
that has the cost of both
IanC and Stefano time to work on libxl and qemu-upstream seems to be quite
limited, and perhaps developer time in general is.

But getting these major transitions smeared out over several major and minor
releases doesn''t seem to be very desirable.
This will be enlarged since distro''s like debian are going to stop
packaging the xen-forks. But since they will be using a "stable"
release for these
projects like qemu, i will take some time for xen patches to for instance qemu
to trickle down into the qemu-upstream stable.

(BTW debian unstable already seems to be not packaging qemu-traditional anymore
but rely on the upstream qemu package entirely,
leading to questions on IRC already )

KVM seems to have it''s act quite together with their virtio drivers at
the moment and i must say i''m getting
attracted to try to switch (although i generally like Xen so i''m still
in denial :-) ).

Just my few cents and a call out as a user ..
(and perhaps a topic for the developer conference to address .. also in the
light of the short discussion about a long term stable release
which in my opinion can''t be done before both are resolved)

Don''t know how Pasi thinks about this and if he has any comments, since
he seems to be quite involved in (helping) the user community ..

--
Sander


> On Wed, 2013-10-09 at 00:09 +0200, Sander Eikelenboom wrote:
>> Hi,
>> 
>> While trying to get to the bottom of my passthrough problem with the
rom bar,
>> i noticed that specifiying "xen_platform_pci=0" in the config
file does not prevent
>> the platform device to appear in my HVM guest and consequently the disk
and nic are taken over.
>> 
>> Running latest xen-unstable, together with upstream qemu and upstream
seabios.
>> 
>> --
>> 
>> Sander
>>

On Wed, 9 Oct 2013, Sander Eikelenboom wrote:
> Wednesday, October 9, 2013, 2:00:02 PM, you wrote:
> 
> > create ^
> > title it xen_platform_pci=0 doesn''t work with qemu-xen
> > owner it Anthony Perard <anthony.perard@citrix.com>
> > thanks
> 
> Perhaps in general looking at the libxl and xend .. and .. qemu-xen and
qemu-xen-traditonal compatibility shoud be added too.
> 
> Perhaps i''m a bit blunt ..
> but for users it''s quite a mess and PITA now for quite some time,
the transition of both is now smeared out over quite some major and minor
releases.
> 
> With some features not working since .. some features are working again but
have a gap of not working,
> since old features becoming usable again are (understandably) not
backported.
> 
> This seems to be at a level that it is even undocumentable ... you would
need a spreadsheet
> with every minor version to try to document that ... it seems to be leading
to a lot of user questions as well.
> 
> I''m not blaming any one personally or anything like that .. just
expressing my worries and the threat i see for Xen as a project in general.
> Namely users sop
> 
> I understand that new features like the arm port are interesting .. but i
think that has the cost of both
> IanC and Stefano time to work on libxl and qemu-upstream seems to be quite
limited, and perhaps developer time in general is.
> 
> But getting these major transitions smeared out over several major and
minor releases doesn''t seem to be very desirable.
> This will be enlarged since distro''s like debian are going to stop
packaging the xen-forks. But since they will be using a "stable"
release for these
> projects like qemu, i will take some time for xen patches to for instance
qemu to trickle down into the qemu-upstream stable.
> 
> (BTW debian unstable already seems to be not packaging qemu-traditional
anymore but rely on the upstream qemu package entirely,
> leading to questions on IRC already )
> 
> KVM seems to have it''s act quite together with their virtio
drivers at the moment and i must say i''m getting
> attracted to try to switch (although i generally like Xen so i''m
still in denial :-) ).
> 
> Just my few cents and a call out as a user ..
> (and perhaps a topic for the developer conference to address .. also in the
light of the short discussion about a long term stable release
> which in my opinion can''t be done before both are resolved)
> 
> Don''t know how Pasi thinks about this and if he has any comments,
since he seems to be quite involved in (helping) the user community ..
Thank you for the feedback. We need constructive criticism like this to
improve the project and the way we handle things. Let me tell you where
I stand.

In the case of xl, we really thought that we filled all the gaps. Just
when we were about to remove xend for good, people stepped up telling us
to wait because actually we missed something.
I don''t think we could have done anything better, given the
circumstances. If we don''t know that we have a bug or a missing
feature,
we can''t do anything about it.

In the case of QEMU, we could probably have done a better job at
tracking missing features. However in general once we know that we have
an issue, we usually resolve it in a timely manner.

Anthony wrote a blog post
(http://blog.xen.org/index.php/2013/09/20/qemu-vs-qemu-traditional-update/)
on this, but having a detailed well formatted wiki page listing
everything we know is missing in upstream QEMU would go a long way
toward avoiding situations like this in the future.

The other thing that could help is better testing. OSSTests doesn''t
cover the xen_platform_pci option, otherwise we would have found and
fixed the issues months ago.

Anybody can write a new OSSTest, not just developers. Give a look at
Wei''s recent blog post:

http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/

Do you care about a particular feature? Do you want to make sure it
doesn''t break in xen-unstable? Write an OSSTest for it!

On 09/10/13 13:49, Sander Eikelenboom wrote:
> Wednesday, October 9, 2013, 2:00:02 PM, you wrote:
>
>> create ^
>> title it xen_platform_pci=0 doesn''t work with qemu-xen
>> owner it Anthony Perard <anthony.perard@citrix.com>
>> thanks
> Perhaps in general looking at the libxl and xend .. and .. qemu-xen and
qemu-xen-traditonal compatibility shoud be added too.
>
> Perhaps i''m a bit blunt ..
> but for users it''s quite a mess and PITA now for quite some time,
the transition of both is now smeared out over quite some major and minor
releases.
>
> With some features not working since .. some features are working again but
have a gap of not working,
> since old features becoming usable again are (understandably) not
backported.
Well thank you for being blunt -- we''d much rather you complain than 
just silently disappear. :-)  I can certainly see where you''re coming 
from.  But on the flip side -- how are developers supposed to know what 
is broken if nobody tests it, or reports it?  We had several test days 
before the 4.3 release; if anyone had reported this feature broken, it 
would have been fixed immediately.

Something similar with xend -- many of the Xen developers were 
considering removing xend for 4.4, because xl seemed to be mature enough 
for everyone.  When we discussed it on the list, immediately a number of 
people stood up and listed features they wanted that were missing from 
xl, or ways in which xl didn''t meet their needs.

We''re not mind-readers, and our software doesn''t have a
call-home
feature to let us know who is using what feature.  The core developers 
have already been looking at libxl, xend, and qemu-xen for some time; we 
made the switch because it looked like we had feature parity for the 
important features.  Us taking a longer look at this point isn''t going 
to help things: we''ve seen all we''re going to see.

If you have features that are missing from xl/qemu-xen that are not on 
my 4.4 development list, and not in xenbugs, the best thing you can do 
is report them, so we can put them on our list.  Better yet, as Stefano 
said, write a test case for them, to make sure that they are never 
broken in any release ever again. :-)

As for the future -- it is unfortunate that with a major transition, 
like from xend->libxl and qemu-traditional->qemu-xen, there is going to 
be some "catch-up" as bugs are ironed out and important functionality 
"faulted in".  This same thing happens in other open source projects; 
KDE, Gnome, Ubuntu''s move to Unity, all come to mind as things that
have
had this happen. Even Linux: the Linux driver for the SD card reader 
broke on my 8-year-old laptop 4 years ago, and still hasn''t been fixed.
KVM isn''t an exception: they just haven''t had to do any major 
refactorings yet (as far as I know). Maybe that''s because they have 
superior process, developers, or design; or maybe it''s just because
it''s
a younger project by about 4 years, and there''s a major refactoring 
coming up right around the corner. :-)

I don''t forsee any more big transitions like that for Xen any time on 
the horizon: once libxl and qemu-xen settle down, things should be 
stable for quite a while.  We''re putting a lot more effort into writing
test cases, and making it easy for others to do so; as our test suite 
grows and becomes more comprehensive, the number of features that can 
get broken without us noticing will get smaller and smaller.

  -George

Wednesday, October 9, 2013, 5:03:56 PM, you wrote:
> On Wed, 9 Oct 2013, Sander Eikelenboom wrote:
>> Wednesday, October 9, 2013, 2:00:02 PM, you wrote:
>> 
>> > create ^
>> > title it xen_platform_pci=0 doesn''t work with qemu-xen
>> > owner it Anthony Perard <anthony.perard@citrix.com>
>> > thanks
>> 
>> Perhaps in general looking at the libxl and xend .. and .. qemu-xen and
qemu-xen-traditonal compatibility shoud be added too.
>> 
>> Perhaps i''m a bit blunt ..
>> but for users it''s quite a mess and PITA now for quite some
time, the transition of both is now smeared out over quite some major and minor
releases.
>> 
>> With some features not working since .. some features are working again
but have a gap of not working,
>> since old features becoming usable again are (understandably) not
backported.
>> 
>> This seems to be at a level that it is even undocumentable ... you
would need a spreadsheet
>> with every minor version to try to document that ... it seems to be
leading to a lot of user questions as well.
>> 
>> I''m not blaming any one personally or anything like that ..
just expressing my worries and the threat i see for Xen as a project in general.
>> Namely users sop
>> 
>> I understand that new features like the arm port are interesting .. but
i think that has the cost of both
>> IanC and Stefano time to work on libxl and qemu-upstream seems to be
quite limited, and perhaps developer time in general is.
>> 
>> But getting these major transitions smeared out over several major and
minor releases doesn''t seem to be very desirable.
>> This will be enlarged since distro''s like debian are going to
stop packaging the xen-forks. But since they will be using a "stable"
release for these
>> projects like qemu, i will take some time for xen patches to for
instance qemu to trickle down into the qemu-upstream stable.
>> 
>> (BTW debian unstable already seems to be not packaging qemu-traditional
anymore but rely on the upstream qemu package entirely,
>> leading to questions on IRC already )
>> 
>> KVM seems to have it''s act quite together with their virtio
drivers at the moment and i must say i''m getting
>> attracted to try to switch (although i generally like Xen so
i''m still in denial :-) ).
>> 
>> Just my few cents and a call out as a user ..
>> (and perhaps a topic for the developer conference to address .. also in
the light of the short discussion about a long term stable release
>> which in my opinion can''t be done before both are resolved)
>> 
>> Don''t know how Pasi thinks about this and if he has any
comments, since he seems to be quite involved in (helping) the user community ..
> Thank you for the feedback. We need constructive criticism like this to
> improve the project and the way we handle things. Let me tell you where
> I stand.
> In the case of xl, we really thought that we filled all the gaps. Just
> when we were about to remove xend for good, people stepped up telling us
> to wait because actually we missed something.
> I don''t think we could have done anything better, given the
> circumstances. If we don''t know that we have a bug or a missing
feature,
> we can''t do anything about it.
I know it''s hard to test all the (possible combination) of options.
However a systematic screening of the documented options allowed in config files
would have revealed
more before missing features in libxl even before user testing.

But i agree it has some chicken and egg problem to it, since early adopter user
testing usually comes after the .0 release
and the rest after the .1 or .2.

However (with hindsight) when refactoring such major components it''s
perhaps better to adjust the release schedule and
focus developer attention to that refactoring for that release and make the
refactoring job set the pace and be the one and only true blocker
and for the .1 at least be liberal with backporting missing features as well (to
prevent a feature gap as much as possible).

That would (hopefully) prevent such a refactoring from dragging on, which is
also a developer burden.
The earlier it is done and over with, the more time to work on the exciting new
features and less nagging users :-p

I know that for the external projects you are not in control for both the patch
acceptance and release pace,
so f.e. for the linux kernel there wasn''t a way to prevent it being
smeared over multiple releases.

However i think qemu is perhaps different in the sense that on distro''s
users will be more inclined to compile a newer kernel
than to compile a new qemu package from upstream sources. Also (PV)HVM seems to
only gain importance ...

So i think it is even more important to keep xen in qemu-upstream in good shape
and to direct some test effort into it, especially prior to an upcoming qemu
release (rc''s).
> In the case of QEMU, we could probably have done a better job at
> tracking missing features. However in general once we know that we have
> an issue, we usually resolve it in a timely manner.
I must say in general the responsiveness is quite good, although as i mentioned
earlier,
i think it is somewhat reduced for the qemu and libxl parts since the arm port
is on it''s way.

For instance there wasn''t much response to the thread about lacking pci
multifunction passthrough
options. (Neither to my personal problem about the rom bar of passthroughed pci
devices.)

I don''t say they have to be solved right away, the more since they are
in the interaction between 2 or more components
(libxl, qemu, hvmloader, seabios), but at least *some* response on how to
proceed would be nice.

Another thing i stumbled across in the qemu source in include/exec/memory.h:

From:   Avi Kivity
Subject:        [Qemu-devel] [PATCH 16/23] memory: temporarily add
memory_region_get_ram_addr()
Date:   Mon, 19 Dec 2011 16:13:37 +0200
/**
 * memory_region_get_ram_addr: Get the ram address associated with a memory
 *                             region
 *
 * DO NOT USE THIS FUNCTION.  This is a temporary workaround while the Xen
 * code is being reworked.
 */
ram_addr_t memory_region_get_ram_addr(MemoryRegion *mr);


It seems it is temporary for quite some time ;-)


> Anthony wrote a blog post
> (http://blog.xen.org/index.php/2013/09/20/qemu-vs-qemu-traditional-update/)
> on this, but having a detailed well formatted wiki page listing
> everything we know is missing in upstream QEMU would go a long way
> toward avoiding situations like this in the future.
> The other thing that could help is better testing. OSSTests
doesn''t
> cover the xen_platform_pci option, otherwise we would have found and
> fixed the issues months ago.
Yes .. although it seemed to be in the bugzilla for quite some time ..
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1839

I know there was a discussion some time ago about bugzilla''s not being
used by developers,
but if that is the case it better can be closed with a message to just send it
to the mailing
list instead. Now a user has the expectation he has reported it and the chance
he won''t post it again to
xen-devel and thus the project is losing a perfectly sound bug report
(and perhaps a user since it''s not addressed nor replied to).

I know that most bugzilla''s are actually distro bugzilla''s,
and are out of reach, however this one
is on xensource. I don''t know how much contact there is with the
various distro package maintainers to forward
the bugzilla reports that should be fixed upstream ?

> Anybody can write a new OSSTest, not just developers. Give a look at
> Wei''s recent blog post:
>
http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/
> Do you care about a particular feature? Do you want to make sure it
> doesn''t break in xen-unstable? Write an OSSTest for it!
Well one of the features happens to be the pci passthrough.

Since i use it, i can say it works in general for single devices and when the
device is not requiring it''s expansion rom to function.
But that last issue i just stumbled upon.

But for such an OSSTEST the test machine should have certain hardware to
passthrough,
or is there a possibility to create a fake/dummy test pci device on the host and
pass
that through and test its behavior (if all mem ranges, ioports and rom are
read/writable as expected)
(did a fast google but didn''t found something immediately useful.


--
Sander

Wednesday, October 9, 2013, 5:32:04 PM, you wrote:
> On 09/10/13 13:49, Sander Eikelenboom wrote:
>> Wednesday, October 9, 2013, 2:00:02 PM, you wrote:
>>
>>> create ^
>>> title it xen_platform_pci=0 doesn''t work with qemu-xen
>>> owner it Anthony Perard <anthony.perard@citrix.com>
>>> thanks
>> Perhaps in general looking at the libxl and xend .. and .. qemu-xen and
qemu-xen-traditonal compatibility shoud be added too.
>>
>> Perhaps i''m a bit blunt ..
>> but for users it''s quite a mess and PITA now for quite some
time, the transition of both is now smeared out over quite some major and minor
releases.
>>
>> With some features not working since .. some features are working again
but have a gap of not working,
>> since old features becoming usable again are (understandably) not
backported.
> Well thank you for being blunt -- we''d much rather you complain
than
> just silently disappear. :-)  I can certainly see where you''re
coming
> from.  But on the flip side -- how are developers supposed to know what 
> is broken if nobody tests it, or reports it?  We had several test days 
> before the 4.3 release; if anyone had reported this feature broken, it 
> would have been fixed immediately.
Well if you do care, you can''t silently disappear :-)

I know there is a almost unlimited combination of config options, so not
everything can be tested in every combination (not by hand, and probably not by
machine either).
However during such a refactoring one would expect that at least every option
that can be disabled and enabled would be tested on it''s own.
This one clearly wasn''t nor do the docs say anything about it missing
...

I have no idea how much test resources there are available to the project, since
there are already quite a lot of
dependencies to test.

However this makes me wonder, does the current test system have something like
the randconfig on linux ?
Or is it always running tests on the same config ?

Hmm have to dive into those docs about OSStest then ...
> Something similar with xend -- many of the Xen developers were 
> considering removing xend for 4.4, because xl seemed to be mature enough 
> for everyone.  When we discussed it on the list, immediately a number of 
> people stood up and listed features they wanted that were missing from 
> xl, or ways in which xl didn''t meet their needs.
> We''re not mind-readers, and our software doesn''t have a
call-home
> feature to let us know who is using what feature.  The core developers 
> have already been looking at libxl, xend, and qemu-xen for some time; we 
> made the switch because it looked like we had feature parity for the 
> important features.  Us taking a longer look at this point isn''t
going
> to help things: we''ve seen all we''re going to see.
The other part of it is that it''s not documented that something is
missing either.
So (most) users aren''t mind-readers as well as code readers ;-)

The docs for example don''t say all the nifty multifunction and vslot
functionality of
pci passthrough isn''t working.
> If you have features that are missing from xl/qemu-xen that are not on 
> my 4.4 development list, and not in xenbugs, the best thing you can do 
> is report them, so we can put them on our list.  Better yet, as Stefano 
> said, write a test case for them, to make sure that they are never 
> broken in any release ever again. :-)
> As for the future -- it is unfortunate that with a major transition, 
like from xend->>libxl and qemu-traditional->qemu-xen, there is going
to 
> be some "catch-up" as bugs are ironed out and important
functionality
> "faulted in".  This same thing happens in other open source
projects;
> KDE, Gnome, Ubuntu''s move to Unity, all come to mind as things
that have
> had this happen. Even Linux: the Linux driver for the SD card reader 
> broke on my 8-year-old laptop 4 years ago, and still hasn''t been
fixed.
> KVM isn''t an exception: they just haven''t had to do any
major
> refactorings yet (as far as I know). Maybe that''s because they
have
> superior process, developers, or design; or maybe it''s just
because it''s
> a younger project by about 4 years, and there''s a major
refactoring
> coming up right around the corner. :-)
Yes i know Xen is running into the "Law of the handicap of a head
start" now,
and KVM has had the opportunity to learn from that and eventually they will run
into that as well. How ever i think there could perhaps be some lessons in here
regarding to the release process when doing major code refactoring projects.
> I don''t forsee any more big transitions like that for Xen any time
on
> the horizon: once libxl and qemu-xen settle down, things should be 
> stable for quite a while.  We''re putting a lot more effort into
writing
> test cases, and making it easy for others to do so; as our test suite 
> grows and becomes more comprehensive, the number of features that can 
> get broken without us noticing will get smaller and smaller.
I could see at least one such transition lurking on the qemu-side of things.
The machine model transition from I440FX/PIIX4 to Q35 (or making the xen code
agnostic
as seems to be done with KVM)

And the dicussion on how to proceed with PVH could lead to another one ...
>   -George
--
Sander

On Wed, Oct 9, 2013 at 1:49 PM, Sander Eikelenboom <linux@eikelenboom.it>
wrote:
>
> Wednesday, October 9, 2013, 2:00:02 PM, you wrote:
>
>> create ^
>> title it xen_platform_pci=0 doesn''t work with qemu-xen
>> owner it Anthony Perard <anthony.perard@citrix.com>
>> thanks
>
> Perhaps in general looking at the libxl and xend .. and .. qemu-xen and
qemu-xen-traditonal compatibility shoud be added too.
>
> Perhaps i''m a bit blunt ..
> but for users it''s quite a mess and PITA now for quite some time,
the transition of both is now smeared out over quite some major and minor
releases.
>
> With some features not working since .. some features are working again but
have a gap of not working,
> since old features becoming usable again are (understandably) not
backported.
>
> This seems to be at a level that it is even undocumentable ... you would
need a spreadsheet
> with every minor version to try to document that ... it seems to be leading
to a lot of user questions as well.
>
> I''m not blaming any one personally or anything like that .. just
expressing my worries and the threat i see for Xen as a project in general.
> Namely users sop
>
> I understand that new features like the arm port are interesting .. but i
think that has the cost of both
> IanC and Stefano time to work on libxl and qemu-upstream seems to be quite
limited, and perhaps developer time in general is.
FWIW, the ARM stuff isn''t just "interesting" -- it was a
critical
strategic opportunity for the future of the Xen project.  The work
that was done, particularly the political stuff done by Stefano to get
Xen into Linaro, was a major achievement at just the right time.  A
delay of even a few months may have meant that Xen had basically no
presence in ARM server space, which would in turn have had an impact
on the commercial viability of the Xen project as a whole.  The vast
majority of the money used to support Xen comes from the server
market.  Growing this market means more commercial interest in Xen,
which means more developers; missing this market could have knock-on
effects in the x86 space, which would have had a much bigger impact on
PCI pass-through in the long term.

 -George