There are basically three issues we''re waiting to sort out (see below for more info): * XSA-55 * cpu hotplug in qemu-upsream * The MMIO hole issue Unfortuantely, there is considerable uncertainty about how long it will take for each of those to complete. We''re hoping to be able to release maybe on the 19th, This information will be mirrored on the Xen 4.3 Roadmap wiki page: http://wiki.xen.org/wiki/Xen_Roadmap/4.3 The key goals we''re focusing on now, in order, are as follows: 1. Have a bug-free 4.3 release 2. Have an awesome 4.3 release 3. Have a 4.3 release that happens on schedule (ready by June 15th) The most important thing in making a case is to answer the question, "If there are bugs in this patch, will they be discovered before the June 17th release?" The second most important thing is to consider the cost/benefit analysis of bugs that are found: what is the risk of introducing a bug which will delay the release, vs the benefit it will have in making the release better? = Timeline We are planning on a 9-month release cycle. Based on that, below are our estimated dates: * Feature freeze: 25 March 2013 * Code freezing point: 15 April 2013 * First RC: 6 May 2013 <== WE ARE HERE * Release: 19 June 2013 The RCs and release will of course depend on stability and bugs, and will therefore be fairly unpredictable. Each new feature will be considered on a case-by-case basis. The June 17th release is both an estimate and a goal. At this point, Xen 4.3 can be released whenever it''s actually ready. In fact, the sooner we release, the sooner we can open the tree up for new development and get on to 4.4 -- so keep fixing those bugs! Last updated: 10 June 2013 == Completed = * Default to QEMU upstream (partial) - pci pass-thru (external) - enable dirtybit tracking during migration (external) - xl cd-{insert,eject} (external) * openvswitch toostack integration To label "tech-preview" unless we get good testing (>10 individuals) * NUMA scheduler affinity * Install into /usr/local by default * Allow config file to specify multiple usb devices for HVM domains * Persistent grants for blk (external) - Linux - qemu * Allow XSM to override IS_PRIV checks in the hypervisor * vTPM updates * Scalability: 16TiB of RAM * CPUID-based idle (don''t rely on ACPI info f/ dom0) * Serial console improvements -EHCI debug port == Bugs resolved since last update = * Revert Jan''s debugging patch (commit bd9be94) resolution: reverted * Migration w/ qemu-upstream causes stuck clock resolutoin: Fixed * xl does not handle migrate interruption gracefully resolution: Not for 4.3 * libxl / xl does not handle failure of remote qemu gracefully resolution: Not for 4.3 == Open bugs = * cpu hotplug broken in qemu-xen > Upstream qemu has qmp interface; either implement our own way or backport > Issue in SeaBIOS: can wake up "unplugged" cpus! owner: Anthony status: In progress priority: blocker * XSA-55 status: v6 posted priority: blocker (?) * qemu-upstream MMIO hole issue > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > "You can reproduce it when a VM has a big pci hole size (such as 512MB), e.g. create a VM with a virtual device which has a 512MB pci BAR." priority: high status: Original fix broken; investigating proper fix priority: blocker (?) * xl pci-detach broken for pv guests? > http://bugs.xenproject.org/xen/bug/12 status: Still investigating priority: Would be blocker if accurate * Update 4.3-rc to 4.3 in README; add tag bragging about 4.3 owner: George status: v1 sent, v2 pending * perl 5.18 fails to compile qemu-traditional docs? > http://www.gossamer-threads.com/lists/xen/devel/284141 status: discussion in progress priority: minor * Scan through qemu-upstream changesets looking for important fixes (particularly for stubdoms) for qemu-traditional - cpu hotplug owner: Anthony * qemu-upstream not freeing pirq > http://www.gossamer-threads.com/lists/xen/devel/281498 priority: high status: patches posted * AMD: IOMMU accidental clearing owner: Jan Beulich, Suravee Suthikulpanit status: * __update_vcpu_system_time if system_time comes out negative status: Not for 4.3 == Not yet complete = * ARM v7 server port (basic) * ARM v8 server port (basic) - Goal: Good Arndale support for 4.3 - Regressions here will not slip the release, so development is ongoing == External projects (Not affected by feature freeze) = * Race in PV shutdown between tool detection and shutdown watch > http://www.gossamer-threads.com/lists/xen/devel/282467 > Nothing to do with ACPI status: Probably a bug in Linux xen drivers * Multi-page blk rings (external) - blkback in kernel roger@citrix - qemu blkback status: Overall blk architecture being discussed prognosis: Fair * Xen EFI feature: Xen can boot from grub.efi owner: Daniel Kiper status: Just begun prognosis: Fair * libvirt/libxl integration (external) - Update libvirt to 4.2 status: Patch accepted - Migration owner: cyliu@suse (?) status: first draft implemented, not yet submitted prognosis: ? - Itemize other things that need work To begin with, we need someone to go and make some lists: - Features available in libvirt/KVM not available in libvirt/libxl See http://libvirt.org/hvsupport.html - Features available in xl/Xen but not available in libvirt/Xen
On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> * XSA-55 > status: v6 posted > priority: blocker (?)How is this series coming -- is it likely to be checked in sometime in the next few days? Alternately -- how many patches on this list are addressing theoretical things an evil compiler *might* do, and how many are addressing observed bugs? Would focusing on observed bugs speed up the process at all? Just doing my job to explore the possibilities here... -George
George Dunlap writes ("Re: Xen 4.3 development update"):> On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: > > * XSA-55 > > status: v6 posted > > priority: blocker (?) > > How is this series coming -- is it likely to be checked in sometime in > the next few days?We''re still short of review. Most recently the series expanded following some further reports from the reporter, Matthew Daley.> Alternately -- how many patches on this list are addressing > theoretical things an evil compiler *might* do, and how many are > addressing observed bugs? Would focusing on observed bugs speed up > the process at all?I don''t think it would. Mostly the perhaps-theoretical considerations influence the choice of fix approach, rather than being dealt with in individual patches. The main exception is the signed/unsigned patch. We haven''t actually found any specific miscompilations of our code without that patch, but we haven''t looked. I have seen reports of security-relevant miscompilations of other code with similar "issues". Ian.
On 10/06/13 17:10, George Dunlap wrote:> On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: >> * XSA-55 >> status: v6 posted >> priority: blocker (?) > How is this series coming -- is it likely to be checked in sometime in > the next few days? > > Alternately -- how many patches on this list are addressing > theoretical things an evil compiler *might* do, and how many are > addressing observed bugs? Would focusing on observed bugs speed up > the process at all? > > Just doing my job to explore the possibilities here... > > -GeorgeI am currently performing a very detailed review. Going is slow, and have already identified areas needing improvement. ~Andrew> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
>>> On 10.06.13 at 18:04, George Dunlap <George.Dunlap@eu.citrix.com> wrote: > * XSA-55 > status: v6 posted > priority: blocker (?)I view this as an unconditional blocker - shipping with a published security vulnerability of this kind would be very bad. Hence, no matter how long it takes, we shouldn''t ship without this taken care of.> * AMD: IOMMU accidental clearing > owner: Jan Beulich, Suravee Suthikulpanit > status:v5 posted, but I agree that at most the erratum workaround would be a reasonable candidate to go in before the release. In the most recently posted shape it would even apply without patch 1 (albeit with fuzz, but that means swapping the patch order is no big deal anymore). I meanwhile doubt, however, that having the erratum workaround is that important considering the other flaws. IOW I vote for either taking both (along with the risk), or postpone them together (aiming at a backport to 4.3.1). Jan
On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> * qemu-upstream MMIO hole issue > > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > > "You can reproduce it when a VM has a big pci hole size (such as > 512MB), e.g. create a VM with a virtual device which has a 512MB > pci BAR." > priority: high > status: Original fix broken; investigating proper fix > priority: blocker (?)Any chance someone could spend a bit of time digging into SeaBIOS / qemu to see if: 1. SeaBIOS has any logic for querying the PCI space and changing the size of the PCI hole 2. If SeaBIOS can do this when running on qemu, and if so, why it''s not working? This document seems to make it clear that on real hardware the BIOS is expected to resize the PCI hole based on the available devices; for example, "Installing PCI expansion cards may also increase the size of the hole. The impact will depend on the amount of Memory Mapped I/O (MMIO) space the PCI card(s) require." http://techfiles.de/dmelanchthon/files/memory_hole.pdf If SeaBIOS runs on real hardware, it should be able to do the same thing. -George
On Tue, 11 Jun 2013, George Dunlap wrote:> On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: > > * qemu-upstream MMIO hole issue > > > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > > > "You can reproduce it when a VM has a big pci hole size (such as > > 512MB), e.g. create a VM with a virtual device which has a 512MB > > pci BAR." > > priority: high > > status: Original fix broken; investigating proper fix > > priority: blocker (?) > > Any chance someone could spend a bit of time digging into SeaBIOS / > qemu to see if: > 1. SeaBIOS has any logic for querying the PCI space and changing the > size of the PCI hole > 2. If SeaBIOS can do this when running on qemu, and if so, why it''s not working?AFAICT SeaBIOS is going to use the address space between BUILD_PCIMEM_START (0xe0000000) and BUILD_PCIMEM_END (0xfec00000) as PCI hole, when it runs out, it is going to return error. See src/pciinit.c:pci_setup
On Tue, 2013-06-11 at 13:11 +0100, George Dunlap wrote:> On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: > > * qemu-upstream MMIO hole issue > > > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > > > "You can reproduce it when a VM has a big pci hole size (such as > > 512MB), e.g. create a VM with a virtual device which has a 512MB > > pci BAR." > > priority: high > > status: Original fix broken; investigating proper fix > > priority: blocker (?) > > Any chance someone could spend a bit of time digging into SeaBIOS / > qemu to see if: > 1. SeaBIOS has any logic for querying the PCI space and changing the > size of the PCI hole > 2. If SeaBIOS can do this when running on qemu, and if so, why it''s not working?Under QEMU SeaBIOS will enable a bunch of extra magic code which talks to qemu to figure this stuff out and construct e.g. an e820. Under Xen the e820 is created by hvmloader and passed to SeaBIOS which doesn''t further mess with it. This is similar to how I understand things work on real hardware -- coreboot brings up the platform and takes care of all this stuff before launching SeaBIOS to provide a "traditional" BIOS like layout. Coreboot supplies the e820 (or some precursor which SeaBIOS then turns into an actual e820). I''d have thought the code in tools/firmware/hvmloader/pci.c would be the bit which would take care of this for us. It sets pci_mem_start/end which are then consumed by acpi/build.c into the ACPI info struct (shared information with the AML code). I don''t know enough AML to locate the code which actually responds to this bit though. I''m a bit surprised that pci_mem_start is not referenced in hvmloader/e820.c though. Or maybe this issue isn''t related to the e820 hole but some other representation of it? (I''ve not really been following all that closely...) I''m wondering how this works for the traditional QEMU + ROMBIOS combo...> This document seems to make it clear that on real hardware the BIOS is > expected to resize the PCI hole based on the available devices; for > example, "Installing PCI expansion cards may also increase the size of > the hole. The impact will depend on the amount of Memory Mapped I/O > (MMIO) space the PCI card(s) require." > > http://techfiles.de/dmelanchthon/files/memory_hole.pdf > > If SeaBIOS runs on real hardware, it should be able to do the same thing.I think this is coreboot''s job on real h/w, but I''m not 100% sure. Ian.
On Tue, 2013-06-11 at 14:53 +0100, Stefano Stabellini wrote:> On Tue, 11 Jun 2013, George Dunlap wrote: > > On Mon, Jun 10, 2013 at 5:04 PM, George Dunlap > > <George.Dunlap@eu.citrix.com> wrote: > > > * qemu-upstream MMIO hole issue > > > > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > > > > "You can reproduce it when a VM has a big pci hole size (such as > > > 512MB), e.g. create a VM with a virtual device which has a 512MB > > > pci BAR." > > > priority: high > > > status: Original fix broken; investigating proper fix > > > priority: blocker (?) > > > > Any chance someone could spend a bit of time digging into SeaBIOS / > > qemu to see if: > > 1. SeaBIOS has any logic for querying the PCI space and changing the > > size of the PCI hole > > 2. If SeaBIOS can do this when running on qemu, and if so, why it''s not working? > > AFAICT SeaBIOS is going to use the address space between > BUILD_PCIMEM_START (0xe0000000) and BUILD_PCIMEM_END (0xfec00000) as PCI > hole, when it runs out, it is going to return error. > > See src/pciinit.c:pci_setupWhich starts with: if (CONFIG_COREBOOT || usingXen()) { // PCI setup already done by coreboot or Xen - just do probe. pci_probe_devices(); return; } So doesn''t apply to our case (or the native hardware case) Ian.
>>> On 10.06.13 at 18:04, George Dunlap <George.Dunlap@eu.citrix.com> wrote: > == Open bugs =While Intel so far hasn''t been able to reproduce this, we just now verified that a problem we''ve been seeing with our 4.2 code base (with the APIC virtualization changes backported from -unstable) on an IvyBridge EP system is also reproducible with 4.3: Win2008 hangs relatives quickly during install, in apparently different ways when run a single- or multi-vCPU guest. The fact that Intel hasn''t been able to repro this may still be due to differences in the exact hardware we and they are using (we can''t, e.g., exclude stepping or microcode changes yet), which we''re in the process of eliminating as a factor. We''re also in the process of verifying whether suppressing use of all the APIC virtualization related hardware features would make the problem go away. This is not the least because on an IvyBridge workstation (not having those capabilities) the problem isn''t seen by us. Jan
There are basically two issues we''re waiting to sort out (see below for more info): * cpu hotplug in qemu-upsream * The MMIO hole issue Anthony has patches for cpu hot-plug, and we have plans for how to work around the MMIO hole issue, so we''re getting really close. Also note that we have slipped the release date a week, to the 26th, to make sure we have enough time to catch any potential bugs in the recent check-ins. This information will be mirrored on the Xen 4.3 Roadmap wiki page: http://wiki.xen.org/wiki/Xen_Roadmap/4.3 The key goals we''re focusing on now, in order, are as follows: 1. Have a bug-free 4.3 release 2. Have an awesome 4.3 release 3. Have a 4.3 release that happens on schedule (ready by June 15th) The most important thing in making a case is to answer the question, "If there are bugs in this patch, will they be discovered before the June 17th release?" The second most important thing is to consider the cost/benefit analysis of bugs that are found: what is the risk of introducing a bug which will delay the release, vs the benefit it will have in making the release better? = Timeline We are planning on a 9-month release cycle. Based on that, below are our estimated dates: * Feature freeze: 25 March 2013 * Code freezing point: 15 April 2013 * First RC: 6 May 2013 <== WE ARE HERE * Release: 26 June 2013 The RCs and release will of course depend on stability and bugs, and will therefore be fairly unpredictable. Each new feature will be considered on a case-by-case basis. The June 17th release is both an estimate and a goal. At this point, Xen 4.3 can be released whenever it''s actually ready. In fact, the sooner we release, the sooner we can open the tree up for new development and get on to 4.4 -- so keep fixing those bugs! Last updated: 17 June 2013 == Completed = * Default to QEMU upstream (partial) - pci pass-thru (external) - enable dirtybit tracking during migration (external) - xl cd-{insert,eject} (external) * openvswitch toostack integration To label "tech-preview" unless we get good testing (>10 individuals) * NUMA scheduler affinity * Install into /usr/local by default * Allow config file to specify multiple usb devices for HVM domains * Persistent grants for blk (external) - Linux - qemu * Allow XSM to override IS_PRIV checks in the hypervisor * vTPM updates * Scalability: 16TiB of RAM * CPUID-based idle (don''t rely on ACPI info f/ dom0) * Serial console improvements -EHCI debug port == Bugs resolved since last update = * tmem map_domain_page issue resolution: fixed * xl pci-detach broken for pv guests? resolution: Guest kernel bug * XSA-55 resolution: patches committed * AMD IOMMU MSI interrupts missing? resolution: fixed == Open bugs = * cpu hotplug broken in qemu-xen > Upstream qemu has qmp interface; either implement our own way or backport > Issue in SeaBIOS: can wake up "unplugged" cpus! owner: Anthony status: In progress priority: blocker * qemu-upstream MMIO hole issue > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > "You can reproduce it when a VM has a big pci hole size (such as 512MB), e.g. create a VM with a virtual device which has a 512MB pci BAR." priority: high status: Original fix broken; investigating proper fix priority: blocker (?) * Win2k8 fails install on many IvyBridge EP systems > Not reproducible by Intel > Reproducible on 4.2 priority: Probably not a blocker * Update 4.3-rc to 4.3 in README; add tag bragging about 4.3 owner: George status: v1 sent, v2 pending * perl 5.18 fails to compile qemu-traditional docs? > http://www.gossamer-threads.com/lists/xen/devel/284141 status: discussion in progress priority: minor * Scan through qemu-upstream changesets looking for important fixes (particularly for stubdoms) for qemu-traditional - cpu hotplug owner: Anthony * qemu-upstream not freeing pirq > http://www.gossamer-threads.com/lists/xen/devel/281498 priority: high, probably not a blocker status: patches posted * AMD: IOMMU accidental clearing owner: Jan Beulich, Suravee Suthikulpanit status: * __update_vcpu_system_time if system_time comes out negative status: Not for 4.3
On Mon, Jun 17, 2013 at 11:58 AM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> * Win2k8 fails install on many IvyBridge EP systems > > Not reproducible by Intel > > Reproducible on 4.2 > priority: Probably not a blockerI''ve marked this "not a blocker" since it''s not a regression and seems to only affect a small number of systems in a particular configuration. Feel free to come back on that one if you disagree. -George
>>> On 17.06.13 at 12:58, George Dunlap <George.Dunlap@eu.citrix.com> wrote: > * Win2k8 fails install on many IvyBridge EP systems > > Not reproducible by Intel > > Reproducible on 4.2Reproducible only on 4.2 with APIC virtualization commits backported. So this is a regression due to new functionality (which to date cannot be worked around).> priority: Probably not a blockerBut nevertheless I agree that this is unlikely to be a blocker.> * AMD: IOMMU accidental clearing > owner: Jan Beulich, Suravee Suthikulpanit > status:I admit that from the title I don''t recall what this is referring to, and hence what the state of it is. Jan
On Mon, Jun 17, 2013 at 11:58 AM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> * perl 5.18 fails to compile qemu-traditional docs? > > http://www.gossamer-threads.com/lists/xen/devel/284141 > status: discussion in progress > priority: minorIs there an update to this? Should I just mark it as "Not for 4.3"? -George
On Mon, 17 Jun 2013 11:58:23 +0100, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> There are basically two issues we''re waiting to sort out (see below > for more info): > * cpu hotplug in qemu-upsream > * The MMIO hole issueDoes that mean that the PCI hole mis-map issue is going to be fixed in 4.3? Or is this a different issue? I''ve been thinking about a domU workaround for this. I haven''t yet had a chance to try it due to too many other things happening, but I was pondering something along the following lines. For Linux domU, something like memmap=2G$2G on the kernel command line. For Windows domU, use something like: bcdedit /set badmemorylist $page1 $page2 ... to mark the whole 2GB-4GB area as not usable. It''s a gross hack, but it might just get things working for now. Gordan
On 17/06/13 12:17, Gordan Bobic wrote:> On Mon, 17 Jun 2013 11:58:23 +0100, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: >> There are basically two issues we''re waiting to sort out (see below >> for more info): >> * cpu hotplug in qemu-upsream >> * The MMIO hole issue > > Does that mean that the PCI hole mis-map issue is going to be fixed > in 4.3? Or is this a different issue?Well "fixed" in the sense that guests shouldn''t mysteriously crash anymore. See my proposed 4.3 fix here: http://marc.info/?l=xen-devel&m=137121936531168&w=2 -George
On Mon, 2013-06-17 at 12:16 +0100, George Dunlap wrote:> On Mon, Jun 17, 2013 at 11:58 AM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: > > * perl 5.18 fails to compile qemu-traditional docs? > > > http://www.gossamer-threads.com/lists/xen/devel/284141 > > status: discussion in progress > > priority: minor > > Is there an update to this? Should I just mark it as "Not for 4.3"?Nothing since "<1369905414.13087.50.camel@zakaz.uk.xensource.com>": 8<------ Seems like we either need to backport: commit 3179d694a8dcaa091131e3db644d445c0130713e Author: Michael Tokarev <mjt@tls.msk.ru> Date: Tue Mar 20 02:25:57 2012 +0400 Support utf8 chars in pod docs Or just disable the build of docs for qemu-xen-trad, it seems to me that they are rather useless... 8<---------- The former seems like something we could still do at this stage (risk =obvious build failure, I think). I''m not sure how the latter can be achieved so I''ve no idea about feasibility. Ian, what do you think? Perl 5.18 was released this May, so it is pretty bleeding edge. On the other hand it appears that at least Arch has picked it up and people who run bleeding edge stuff make good testers... Ian.
Il 17/06/2013 12:58, George Dunlap ha scritto:> There are basically two issues we''re waiting to sort out (see below > for more info): > * cpu hotplug in qemu-upsream > * The MMIO hole issue > > Anthony has patches for cpu hot-plug, and we have plans for how to > work around the MMIO hole issue, so we''re getting really close. > > Also note that we have slipped the release date a week, to the 26th, > to make sure we have enough time to catch any potential bugs in the > recent check-ins. > > This information will be mirrored on the Xen 4.3 Roadmap wiki page: > http://wiki.xen.org/wiki/Xen_Roadmap/4.3 > > The key goals we''re focusing on now, in order, are as follows: > 1. Have a bug-free 4.3 release > 2. Have an awesome 4.3 release > 3. Have a 4.3 release that happens on schedule (ready by June 15th) > > The most important thing in making a case is to answer the question, > "If there are bugs in this patch, will they be discovered before the > June 17th release?" The second most important thing is to consider the > cost/benefit analysis of bugs that are found: what is the risk of > introducing a bug which will delay the release, vs the benefit it will > have in making the release better? > > = Timeline > > We are planning on a 9-month release cycle. Based on that, below are > our estimated dates: > * Feature freeze: 25 March 2013 > * Code freezing point: 15 April 2013 > * First RC: 6 May 2013 <== WE ARE HERE > * Release: 26 June 2013 > > The RCs and release will of course depend on stability and bugs, and > will therefore be fairly unpredictable. Each new feature will be > considered on a case-by-case basis. > > The June 17th release is both an estimate and a goal. At this point, > Xen 4.3 can be released whenever it''s actually ready. In fact, the > sooner we release, the sooner we can open the tree up for new > development and get on to 4.4 -- so keep fixing those bugs! > > Last updated: 17 June 2013 > > == Completed => > * Default to QEMU upstream (partial) > - pci pass-thru (external) > - enable dirtybit tracking during migration (external)> - xl cd-{insert,eject}Tested and working. No more bugs found.> > * openvswitch toostack integration > To label "tech-preview" unless we get good testing (>10 individuals)Tested and working. It also works with hvm domUs that didn''t work on initial version of vif script. No bugs found for now.> > * NUMA scheduler affinity > > * Install into /usr/local by default > > * Allow config file to specify multiple usb devices for HVM domains > > * Persistent grants for blk (external) > - Linux > - qemu > > * Allow XSM to override IS_PRIV checks in the hypervisor > > * vTPM updates > > * Scalability: 16TiB of RAM > > * CPUID-based idle (don''t rely on ACPI info f/ dom0) > > * Serial console improvements > -EHCI debug port > > == Bugs resolved since last update => > * tmem map_domain_page issue > resolution: fixed > > * xl pci-detach broken for pv guests? > resolution: Guest kernel bug > > * XSA-55 > resolution: patches committed > > * AMD IOMMU MSI interrupts missing? > resolution: fixed > > == Open bugs => > * cpu hotplug broken in qemu-xen > > Upstream qemu has qmp interface; either implement our own way or backport > > Issue in SeaBIOS: can wake up "unplugged" cpus! > owner: Anthony > status: In progress > priority: blocker > > * qemu-upstream MMIO hole issue > > http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html > > "You can reproduce it when a VM has a big pci hole size (such as > 512MB), e.g. create a VM with a virtual device which has a 512MB > pci BAR." > priority: high > status: Original fix broken; investigating proper fix > priority: blocker (?) > > * Win2k8 fails install on many IvyBridge EP systems > > Not reproducible by Intel > > Reproducible on 4.2 > priority: Probably not a blocker > > * Update 4.3-rc to 4.3 in README; add tag bragging about 4.3 > owner: George > status: v1 sent, v2 pending > > * perl 5.18 fails to compile qemu-traditional docs? > > http://www.gossamer-threads.com/lists/xen/devel/284141 > status: discussion in progress > priority: minor > > * Scan through qemu-upstream changesets looking for important fixes > (particularly for stubdoms) for qemu-traditional > - cpu hotplug > owner: Anthony > > * qemu-upstream not freeing pirq > > http://www.gossamer-threads.com/lists/xen/devel/281498 > priority: high, probably not a blocker > status: patches posted > > * AMD: IOMMU accidental clearing > owner: Jan Beulich, Suravee Suthikulpanit > status: > > * __update_vcpu_system_time if system_time comes out negative > status: Not for 4.3 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On 17/06/13 13:35, Fabio Fantoni wrote:> Il 17/06/2013 12:58, George Dunlap ha scritto: >> There are basically two issues we''re waiting to sort out (see below >> for more info): >> * cpu hotplug in qemu-upsream >> * The MMIO hole issue >> >> Anthony has patches for cpu hot-plug, and we have plans for how to >> work around the MMIO hole issue, so we''re getting really close. >> >> Also note that we have slipped the release date a week, to the 26th, >> to make sure we have enough time to catch any potential bugs in the >> recent check-ins. >> >> This information will be mirrored on the Xen 4.3 Roadmap wiki page: >> http://wiki.xen.org/wiki/Xen_Roadmap/4.3 >> >> The key goals we''re focusing on now, in order, are as follows: >> 1. Have a bug-free 4.3 release >> 2. Have an awesome 4.3 release >> 3. Have a 4.3 release that happens on schedule (ready by June 15th) >> >> The most important thing in making a case is to answer the question, >> "If there are bugs in this patch, will they be discovered before the >> June 17th release?" The second most important thing is to consider the >> cost/benefit analysis of bugs that are found: what is the risk of >> introducing a bug which will delay the release, vs the benefit it will >> have in making the release better? >> >> = Timeline >> >> We are planning on a 9-month release cycle. Based on that, below are >> our estimated dates: >> * Feature freeze: 25 March 2013 >> * Code freezing point: 15 April 2013 >> * First RC: 6 May 2013 <== WE ARE HERE >> * Release: 26 June 2013 >> >> The RCs and release will of course depend on stability and bugs, and >> will therefore be fairly unpredictable. Each new feature will be >> considered on a case-by-case basis. >> >> The June 17th release is both an estimate and a goal. At this point, >> Xen 4.3 can be released whenever it''s actually ready. In fact, the >> sooner we release, the sooner we can open the tree up for new >> development and get on to 4.4 -- so keep fixing those bugs! >> >> Last updated: 17 June 2013 >> >> == Completed =>> >> * Default to QEMU upstream (partial) >> - pci pass-thru (external) >> - enable dirtybit tracking during migration (external) > >> - xl cd-{insert,eject} > > Tested and working. No more bugs found. > >> >> * openvswitch toostack integration >> To label "tech-preview" unless we get good testing (>10 individuals) > > Tested and working. It also works with hvm domUs that didn''t work on > initial version of vif script. No bugs found for now.Great, thanks for testing. -George
On Mon, Jun 17, 2013 at 11:58:23AM +0100, George Dunlap wrote:> There are basically two issues we''re waiting to sort out (see below > for more info): > * cpu hotplug in qemu-upsream > * The MMIO hole issueThere is also the issue of migration not working with ''xl'' with the PVHVM Linux guests. I am able to reproduce this (this is the issue Vasiliy Tolstov experienced) when doing a simple migration: xl migrate latest localhost which hangs at 50% and stops. I hadn''t been able yet to capture the output from the guest so I don''t know if I am hitting the same _exact_ error Vasiliy is hitting. If I do ''xm migrate latest localhost'' it works nicely. Note that I am using qemu-traditional for this (xl).
On 17/06/13 14:24, Konrad Rzeszutek Wilk wrote:> On Mon, Jun 17, 2013 at 11:58:23AM +0100, George Dunlap wrote: >> There are basically two issues we''re waiting to sort out (see below >> for more info): >> * cpu hotplug in qemu-upsream >> * The MMIO hole issue > There is also the issue of migration not working with ''xl'' with > the PVHVM Linux guests. I am able to reproduce this (this is > the issue Vasiliy Tolstov experienced) when doing a simple > migration: > > xl migrate latest localhost > > which hangs at 50% and stops. I hadn''t been able yet to capture the > output from the guest so I don''t know if I am hitting the same > _exact_ error Vasiliy is hitting. > > If I do ''xm migrate latest localhost'' it works nicely. > > Note that I am using qemu-traditional for this (xl).You mentioned this on IRC, but you haven''t actually given anything near a real bug report. :-) xl localhost migrate is tested in osstest, and we just had that bug with the timing for PVHVM guests that we fixed a few weeks ago -- that involved doing 100+ localhost migrates. Without some more specification, and/or someone else reporting the issue, I can only assume it''s something local to your configuration. -George
(Jacek: can you confirm that this fixes this other bug for you? George: can I have an ack wrt the release?) Perl 5.18 is unhappy with the qemu-xen-traditional documents, giving this error: qemu.pod around line 91: Non-ASCII character seen before =encoding in ''Schuetz.''. Assuming UTF-8 POD document had syntax errors at /usr/bin/core_perl/pod2man line 71. make[3]: *** [qemu.1] Error 255 We do not want these docs. They are not really relevant to Xen users. So instead of backporting the utf-8 fix from qemu upstream (3179d694a8dcaa091131e3db644d445c0130713e), we just disable the docs build. We do this in xen-hooks.mak because qemu-xen-traditional''s configure script lacks the ability to explicitly disable the docs build. (The docs build for upstream-based qemu-xen was already disabled by xen.git#a0d110801d701c43e7b8c73dbd6b2444a10a7cdb) Reported-by: jacek burghardt <jaceksburghardt@gmail.com> Cc: George Dunlap <George.Dunlap@eu.citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> diff --git a/xen-hooks.mak b/xen-hooks.mak index b55f45b..58d61c9 100644 --- a/xen-hooks.mak +++ b/xen-hooks.mak @@ -83,4 +83,6 @@ datadir := $(subst qemu,xen/qemu,$(datadir)) docdir := $(subst qemu,xen/qemu,$(docdir)) mandir := $(subst share/man,share/xen/man,$(mandir)) +BUILD_DOCS+ configdir := $(XEN_SCRIPT_DIR)
George, --On 17 June 2013 14:29:22 +0100 George Dunlap <george.dunlap@eu.citrix.com> wrote:>> There is also the issue of migration not working with ''xl'' with >> the PVHVM Linux guests. I am able to reproduce this (this is >> the issue Vasiliy Tolstov experienced) when doing a simple >> migration: >> >> xl migrate latest localhost >> >> which hangs at 50% and stops. I hadn''t been able yet to capture the >> output from the guest so I don''t know if I am hitting the same >> _exact_ error Vasiliy is hitting. >> >> If I do ''xm migrate latest localhost'' it works nicely. >> >> Note that I am using qemu-traditional for this (xl). > > You mentioned this on IRC, but you haven''t actually given anything near a > real bug report. :-) > > xl localhost migrate is tested in osstest, and we just had that bug with > the timing for PVHVM guests that we fixed a few weeks ago -- that > involved doing 100+ localhost migrates.Was that on qemu-xen/qemu-upstream rather than qemu-traditional? -- Alex Bligh
On 17/06/13 16:47, Ian Jackson wrote:> (Jacek: can you confirm that this fixes this other bug for you? > George: can I have an ack wrt the release?) > > Perl 5.18 is unhappy with the qemu-xen-traditional documents, giving > this error: > qemu.pod around line 91: Non-ASCII character seen before =encoding in > ''Schuetz.''. Assuming UTF-8 POD document had syntax errors at /usr/bin/core_perl/pod2man line 71. > make[3]: *** [qemu.1] Error 255 > > We do not want these docs. They are not really relevant to Xen users. > So instead of backporting the utf-8 fix from qemu upstream > (3179d694a8dcaa091131e3db644d445c0130713e), we just disable the docs > build. > > We do this in xen-hooks.mak because qemu-xen-traditional''s configure > script lacks the ability to explicitly disable the docs build. > > (The docs build for upstream-based qemu-xen was already disabled by > xen.git#a0d110801d701c43e7b8c73dbd6b2444a10a7cdb) > > Reported-by: jacek burghardt <jaceksburghardt@gmail.com> > Cc: George Dunlap <George.Dunlap@eu.citrix.com> > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Re the release: Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
George Dunlap writes ("Re: [PATCH] qemu-xen-traditional: disable docs"):> On 17/06/13 16:47, Ian Jackson wrote: > > (Jacek: can you confirm that this fixes this other bug for you? > > George: can I have an ack wrt the release?) > > > > Perl 5.18 is unhappy with the qemu-xen-traditional documents, giving...> Re the release: > > Acked-by: George Dunlap <george.dunlap@eu.citrix.com>Thanks, I have applied this to qemu-xen-traditional and updated xen.git''s Config.mk. Ian.
On Mon, Jun 17, 2013 at 05:16:38PM +0100, Alex Bligh wrote:> George, > > --On 17 June 2013 14:29:22 +0100 George Dunlap > <george.dunlap@eu.citrix.com> wrote: > > >>There is also the issue of migration not working with ''xl'' with > >>the PVHVM Linux guests. I am able to reproduce this (this is > >>the issue Vasiliy Tolstov experienced) when doing a simple > >>migration: > >> > >> xl migrate latest localhost > >> > >>which hangs at 50% and stops. I hadn''t been able yet to capture the > >>output from the guest so I don''t know if I am hitting the same > >>_exact_ error Vasiliy is hitting. > >> > >>If I do ''xm migrate latest localhost'' it works nicely. > >> > >>Note that I am using qemu-traditional for this (xl). > > > >You mentioned this on IRC, but you haven''t actually given anything near a > >real bug report. :-)And to double check whether this is an issue with the machine I can''t reproduce this on another Intel box. Hmmm...> > > >xl localhost migrate is tested in osstest, and we just had that bug with > >the timing for PVHVM guests that we fixed a few weeks ago -- that > >involved doing 100+ localhost migrates. > > Was that on qemu-xen/qemu-upstream rather than qemu-traditional? > > -- > Alex Bligh > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
>>> On 17.06.13 at 13:13, "Jan Beulich" <JBeulich@suse.com> wrote: >>>> On 17.06.13 at 12:58, George Dunlap <George.Dunlap@eu.citrix.com> wrote: >> * Win2k8 fails install on many IvyBridge EP systems >> > Not reproducible by Intel >> > Reproducible on 4.2 > > Reproducible only on 4.2 with APIC virtualization commits backported. > So this is a regression due to new functionality (which to date cannot > be worked around).Update: This is a bad interaction between Viridian mode and APIC-V. Disabling either makes things work again. Yang narrowed this down to the EOI handling, and created a non-postable patch that deals with the issue (non-postable because it does things in ways only suitable for temporary testing). I do think, though, that this isn''t the way to go anyway - instead we likely will want to suppress Windows using the APIC related MSRs when APIC register virtualization is in use. Yang seems to agree with this, but this approach wasn''t tested yet (partly because for it to be complete we''d also want to populate Viridian CPUID leaf 6, but the APIC related bit there is insufficiently specified, so we first need to find out from MS when exactly this bit is supposed to get set). In any case, with the workaround of disabling Viridian support now known, I guess we can indeed defer fixing this until 4.3.1. Jan
Jan Beulich wrote on 2013-06-20:>>>> On 17.06.13 at 13:13, "Jan Beulich" <JBeulich@suse.com> wrote: >>>>> On 17.06.13 at 12:58, George Dunlap <George.Dunlap@eu.citrix.com> > wrote: >>> * Win2k8 fails install on many IvyBridge EP systems >>>> Not reproducible by Intel >>>> Reproducible on 4.2 >> >> Reproducible only on 4.2 with APIC virtualization commits backported. >> So this is a regression due to new functionality (which to date cannot >> be worked around). > > Update: This is a bad interaction between Viridian mode and APIC-V. > Disabling either makes things work again. Yang narrowed this down > to the EOI handling, and created a non-postable patch that deals > with the issue (non-postable because it does things in ways only > suitable for temporary testing). I do think, though, that this isn''t the > way to go anyway - instead we likely will want to suppress Windows > using the APIC related MSRs when APIC register virtualization is in > use. Yang seems to agree with this, but this approach wasn''t testedJust do a testing and it is working to not set CPUID3A_MSR_APIC_ACCESS and CPUID4A_MSR_BASED_APIC.> yet (partly because for it to be complete we''d also want to > populate Viridian CPUID leaf 6, but the APIC related bit there is > insufficiently specified, so we first need to find out from MS when > exactly this bit is supposed to get set).I think CPUID leaf 6 just indicate which hardware feature is used by hypervisor. It should have nothing to do with this issue. We can consider it separately.> > In any case, with the workaround of disabling Viridian support now > known, I guess we can indeed defer fixing this until 4.3.1. > > JanBest regards, Yang
>>> On 21.06.13 at 10:08, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote: > Jan Beulich wrote on 2013-06-20: >>>>> On 17.06.13 at 13:13, "Jan Beulich" <JBeulich@suse.com> wrote: >>>>>> On 17.06.13 at 12:58, George Dunlap <George.Dunlap@eu.citrix.com> >> wrote: >>>> * Win2k8 fails install on many IvyBridge EP systems >>>>> Not reproducible by Intel >>>>> Reproducible on 4.2 >>> >>> Reproducible only on 4.2 with APIC virtualization commits backported. >>> So this is a regression due to new functionality (which to date cannot >>> be worked around). >> >> Update: This is a bad interaction between Viridian mode and APIC-V. >> Disabling either makes things work again. Yang narrowed this down >> to the EOI handling, and created a non-postable patch that deals >> with the issue (non-postable because it does things in ways only >> suitable for temporary testing). I do think, though, that this isn''t the >> way to go anyway - instead we likely will want to suppress Windows >> using the APIC related MSRs when APIC register virtualization is in >> use. Yang seems to agree with this, but this approach wasn''t tested > Just do a testing and it is working to not set CPUID3A_MSR_APIC_ACCESS and > CPUID4A_MSR_BASED_APIC.But after some more thought I think we ought to still use your original patch (suitably converted to proper shape), and keep the leaf 3 bit set (i.e. only turn off the respective leaf 4 bit). This _should_ make Windows not use the MSRs, but would still cope with it nevertheless doing so for whatever reason.>> yet (partly because for it to be complete we''d also want to >> populate Viridian CPUID leaf 6, but the APIC related bit there is >> insufficiently specified, so we first need to find out from MS when >> exactly this bit is supposed to get set). > I think CPUID leaf 6 just indicate which hardware feature is used by > hypervisor. It should have nothing to do with this issue. We can consider it > separately.Sure, this would be a separate patch, but still belonging here as potentially influencing Windows'' decision on how to set up certain aspects of its operation, including APIC handling. I''m in the process of putting all these together, in case you haven''t already. Jan
We''ve checked in fixes to the two outstanding blockers. We''ve also had a number of minor fixes checked in. At the point of writing this e-mail, there are no known outstanding blockers. We''ve slipped the release again in order to be able to finish up the coding for the two blockers, as well as to be able to have a test day (tomorrow). But at this point, unless there''s a real howler discovered, I think we''re just going to publish even if there are a few remaining bugs turned up. This information will be mirrored on the Xen 4.3 Roadmap wiki page: http://wiki.xen.org/wiki/Xen_Roadmap/4.3 = Timeline We are planning on a 9-month release cycle. Based on that, below are our estimated dates: * Feature freeze: 25 March 2013 * Code freezing point: 15 April 2013 * First RC: 6 May 2013 <== WE ARE HERE * Release: 26 June 2013 Last updated: 27 June 2013 == Completed = * Default to QEMU upstream (partial) - pci pass-thru (external) - enable dirtybit tracking during migration (external) - xl cd-{insert,eject} (external) * openvswitch toostack integration To label "tech-preview" unless we get good testing (>10 individuals) * NUMA scheduler affinity * Install into /usr/local by default * Allow config file to specify multiple usb devices for HVM domains * Persistent grants for blk (external) - Linux - qemu * Allow XSM to override IS_PRIV checks in the hypervisor * vTPM updates * Scalability: 16TiB of RAM * CPUID-based idle (don''t rely on ACPI info f/ dom0) * Serial console improvements -EHCI debug port == Bugs resolved since last update = * perl 5.18 fails to compile qemu-traditional docs? resolution: qemu-traditional docs build disabled * cpu hotplug broken in qemu-xen resolution: fixed * qemu-upstream MMIO hole issue resolution: work-around patch * Update 4.3-rc to 4.3 in README; add tag bragging about 4.3 resolution: Done * Win2k8 fails install on many IvyBridge EP systems resolution: fixed == Open bugs = * qemu-upstream not freeing pirq > http://www.gossamer-threads.com/lists/xen/devel/281498 priority: high, probably not a blocker status: patches posted * AMD: IOMMU accidental clearing owner: Jan Beulich, Suravee Suthikulpanit status: * __update_vcpu_system_time if system_time comes out negative status: Not for 4.3 == Not yet complete = * ARM v7 server port (basic) * ARM v8 server port (basic) - Goal: Good Arndale support for 4.3 - Regressions here will not slip the release, so development is ongoing == External projects (Not affected by feature freeze) = * xl pci-detach broken for pv guests? > http://bugs.xenproject.org/xen/bug/12 > kernel doesn''t know about moving into states 7 and 8 status: External * Race in PV shutdown between tool detection and shutdown watch > http://www.gossamer-threads.com/lists/xen/devel/282467 > Nothing to do with ACPI status: Probably a bug in Linux xen drivers * Multi-page blk rings (external) - blkback in kernel roger@citrix - qemu blkback status: Overall blk architecture being discussed prognosis: Fair * Xen EFI feature: Xen can boot from grub.efi owner: Daniel Kiper status: Just begun prognosis: Fair * libvirt/libxl integration (external) - Update libvirt to 4.2 status: Patch accepted - Migration owner: cyliu@suse (?) status: first draft implemented, not yet submitted prognosis: ? - Itemize other things that need work To begin with, we need someone to go and make some lists: - Features available in libvirt/KVM not available in libvirt/libxl See http://libvirt.org/hvsupport.html - Features available in xl/Xen but not available in libvirt/Xen
>>> On 27.06.13 at 15:54, George Dunlap <George.Dunlap@eu.citrix.com> wrote: > * AMD: IOMMU accidental clearing > owner: Jan Beulich, Suravee Suthikulpanit > status:If that''s referring to Andrew''s "AMD/intremap: Prevent use of per-device vector maps..." - this got committed earlier today. Jan
On 27/06/13 15:05, Jan Beulich wrote:>>>> On 27.06.13 at 15:54, George Dunlap <George.Dunlap@eu.citrix.com> wrote: >> * AMD: IOMMU accidental clearing >> owner: Jan Beulich, Suravee Suthikulpanit >> status: > If that''s referring to Andrew''s "AMD/intremap: Prevent use of > per-device vector maps..." - this got committed earlier today.Missed that one -- thanks for the update. -George
On 6/27/2013 9:05 AM, Jan Beulich wrote:>>>> On 27.06.13 at 15:54, George Dunlap <George.Dunlap@eu.citrix.com> wrote: >> * AMD: IOMMU accidental clearing >> owner: Jan Beulich, Suravee Suthikulpanit >> status: > If that''s referring to Andrew''s "AMD/intremap: Prevent use of > per-device vector maps..." - this got committed earlier today. > > Jan >I think he meant the one iommu/amd: Fix logic for clearing the IOMMU interrupt bits as discussed here (http://lists.xen.org/archives/html/xen-devel/2013-06/msg01770.html). Suravee> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
>>> On 28.06.13 at 17:20, Suravee Suthikulanit <suravee.suthikulpanit@amd.com> wrote: > On 6/27/2013 9:05 AM, Jan Beulich wrote: >>>>> On 27.06.13 at 15:54, George Dunlap <George.Dunlap@eu.citrix.com> wrote: >>> * AMD: IOMMU accidental clearing >>> owner: Jan Beulich, Suravee Suthikulpanit >>> status: >> If that''s referring to Andrew''s "AMD/intremap: Prevent use of >> per-device vector maps..." - this got committed earlier today. > > I think he meant the one iommu/amd: Fix logic for clearing the IOMMU > interrupt bits as discussed here > (http://lists.xen.org/archives/html/xen-devel/2013-06/msg01770.html).But that one as well as the erratum 787 we settled on postponing until after 4.3, at least as far as I recall. George - can you clarify? Jan
I would to get clarification so is the memory bug fixed then ? So am I able to assign more 3.6. GB of memory? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 28/06/13 16:36, jacek burghardt wrote:> > I would to get clarification so is the memory bug fixed then ? So am I > able to assign more 3.6. GB of memory? >You mean for an HVM domain with a PCI device passed through? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel