thr3ads.net - Xen devel - [QEMU][RFC V2 00/10] QEMU disaggregation in Xen environment. [Aug 2012]

If this information is useful, please help other people find it:
Share via:

Julien Grall

2012-Aug-22 12:30 UTC

[QEMU][RFC V2 00/10] QEMU disaggregation in Xen environment.

Hello,

This patch series only concerns QEMU. Another serie will come for Xen.

I''m currently working on QEMU disaggregation in Xen environment. The
goal is to be able to running multiple QEMU for a same domain
(http://lists.xen.org/archives/html/xen-devel/2012-03/msg00299.html).

I have already sent a version of patch series few months ago:
    - QEMU: https://lists.gnu.org/archive/html/qemu-devel/2012-03/msg04401.html
    - Xen: http://lists.xen.org/archives/html/xen-devel/2012-03/msg01947.html
With the different feedbacks, I have improved both QEMU and Xen modifications.
As before, I will sent two patch series, one for QEMU the other for Xen.

Modifications between V1 and V2:
    - introduce new machine options
    - use memory listener to avoid Xen specific code in QEMU core
    (depends of "[PATCH V5 0/8] memory: unify ioport registration"
patch series)
    - implement disaggregation
    - add wrapper for older Xen version

Julien Grall (10):
  xen: add new machine options to support QEMU disaggregation in Xen
    environment
  xen: modify QEMU status path in XenStore
  xen: add wrappers for new Xen disaggregation hypercalls
  xen-hvm: register qemu as ioreq server and retrieve shared pages
  xen-memory: register memory/IO range in Xen
  xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG
  xen: specify which device is part of default devices
  xen: audio is not a part of default devices
  xen-memory: handle node "device_model" for physical mapping
  xen: emulate IDE outside default device set

 arch_init.c     |    6 +
 hw/ide/qdev.c   |    8 ++-
 hw/pc_piix.c    |   40 +++++---
 hw/pci.c        |    6 +
 hw/xen.h        |   31 ++++++
 hw/xen_common.h |   58 +++++++++++
 qemu-config.c   |   12 ++
 xen-all.c       |  304 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 xen-stub.c      |    5 +
 9 files changed, 447 insertions(+), 23 deletions(-)

-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 01/10] xen: add new machine options to support QEMU disaggregation in Xen environment

- xen_dmid: specify the id of QEMU. It will be used to
    retrieve/store information inside XenStore.
    - xen_default_dev (on/off): as default devices need to be create in
    each QEMU (due to code dependency), this option specifies if it will
    register range/PCI of default device via xen hypercall.
    (Root bridge, south bridge, ...).
    - xen_emulate_ide (on/off): enable/disable emulation in QEMU.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 qemu-config.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/qemu-config.c b/qemu-config.c
index c05ffbc..7740442 100644
--- a/qemu-config.c
+++ b/qemu-config.c
@@ -612,6 +612,18 @@ static QemuOptsList qemu_machine_opts = {
             .name = "dump-guest-core",
             .type = QEMU_OPT_BOOL,
             .help = "Include guest memory in  a core dump",
+        }, {
+            .name = "xen_dmid",
+            .type = QEMU_OPT_NUMBER,
+            .help = "Xen device model id",
+        }, {
+            .name = "xen_default_dev",
+            .type = QEMU_OPT_BOOL,
+            .help = "emulate Xen default device (South Bridge, IDE,
...)"
+        }, {
+            .name = "xen_emulate_ide",
+            .type = QEMU_OPT_BOOL,
+            .help = "emulate IDE with XEN"
         },
         { /* End of list */ }
     },
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 02/10] xen: modify QEMU status path in XenStore

QEMU will now write its status in another XenStore path because multiple
QEMU can run for a same domain. If xen_dmid machine option is not specified,
it means that an old version of Xen is used, so status is written in the old
path.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 xen-all.c |   16 +++++++++++++++-
 1 files changed, 15 insertions(+), 1 deletions(-)

diff --git a/xen-all.c b/xen-all.c
index 61def2e..df6927d 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -36,6 +36,7 @@
 
 static MemoryRegion ram_memory, ram_640k, ram_lo, ram_hi;
 static MemoryRegion *framebuffer;
+static uint32_t xen_dmid = ~0;
 
 /* Compatibility with older version */
 #if __XEN_LATEST_INTERFACE_VERSION__ < 0x0003020a
@@ -958,7 +959,14 @@ static void xenstore_record_dm_state(struct xs_handle *xs,
const char *state)
         exit(1);
     }
 
-    snprintf(path, sizeof (path),
"/local/domain/0/device-model/%u/state", xen_domid);
+    if (xen_dmid == ~0) {
+        snprintf(path, sizeof(path),
"/local/domain/0/device-model/%u/state",
+                 xen_domid);
+    } else {
+        snprintf(path, sizeof(path),
"/local/domain/0/dms/%u/%u/state",
+                 xen_domid, xen_dmid);
+    }
+
     if (!xs_write(xs, XBT_NULL, path, state, strlen(state))) {
         fprintf(stderr, "error recording dm state\n");
         exit(1);
@@ -1077,6 +1085,12 @@ int xen_hvm_init(void)
     unsigned long ioreq_pfn;
     unsigned long bufioreq_evtchn;
     XenIOState *state;
+    QemuOptsList *list = qemu_find_opts("machine");
+
+    if (!QTAILQ_EMPTY(&list->head)) {
+        xen_dmid = qemu_opt_get_number(QTAILQ_FIRST(&list->head),
+                                       "xen_dmid", ~0);
+    }
 
     state = g_malloc0(sizeof (XenIOState));
 
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 03/10] xen: add wrappers for new Xen disaggregation hypercalls

QEMU disaggregation is not supported on old Xen versions.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 hw/xen_common.h |   58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 58 insertions(+), 0 deletions(-)

diff --git a/hw/xen_common.h b/hw/xen_common.h
index 727757a..b2525ad 100644
--- a/hw/xen_common.h
+++ b/hw/xen_common.h
@@ -152,6 +152,64 @@ static inline int xen_xc_hvm_inject_msi(XenXC xen_xc,
domid_t dom,
 }
 #endif
 
+/* Xen before 4.3 */
+#if CONFIG_XEN_CTRL_INTERFACE_VERSION < 430
+static inline int xen_xc_hvm_register_pcidev(XenXC xen_xc, domid_t dom,
+        unsigned int serverid, uint8_t domain,
+        uint8_t bus, uint8_t device, uint8_t function)
+{
+    return 0;
+}
+
+static inline int xen_xc_hvm_map_io_range_to_ioreq_server(XenXC xen_xc,
+        domid_t dom, unsigned int serverid, int is_mmio,
+        uint64_t start, uint64_t end)
+{
+    return 1;
+}
+
+static inline int xen_xc_hvm_unmap_io_range_from_ioreq_server(XenXC xen_xc,
+        domid_t dom, unsigned int serverid, int is_mmio, uint64_t start)
+{
+    return 1;
+}
+
+static inline int xen_xc_hvm_register_ioreq_server(XenXC xen_xc, domid_t dom)
+{
+    return 0;
+}
+
+#else
+static inline int xen_xc_hvm_register_pcidev(XenXC xen_xc, domid_t dom,
+        unsigned int serverid, uint8_t domain,
+        uint8_t bus, uint8_t device, uint8_t function)
+{
+    return xc_hvm_register_pcidev(xen_xc, dom, serverid, domain,
+                                  bus, device, function);
+}
+
+static inline int xen_xc_hvm_map_io_range_to_ioreq_server(XenXC xen_xc,
+        domid_t dom, unsigned int serverid, int is_mmio,
+        uint64_t start, uint64_t end)
+{
+    return xc_hvm_map_io_range_to_ioreq_server(xen_xc, dom, serverid, is_mmio,
+                                               start, end);
+}
+
+static inline int xen_xc_hvm_unmap_io_range_from_ioreq_server(XenXC xen_xc,
+        domid_t dom, unsigned int serverid, int is_mmio, uint64_t start)
+{
+    return xc_hvm_unmap_io_range_from_ioreq_server(xen_xc, dom, serverid,
+                                                   is_mmio, start);
+}
+
+static inline int xen_xc_hvm_register_ioreq_server(XenXC xen_xc, domid_t dom)
+{
+    return xc_hvm_register_ioreq_server(xen_xc, dom);
+}
+
+#endif
+
 void destroy_hvm_domain(bool reboot);
 
 /* shutdown/destroy current domain because of an error */
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 04/10] xen-hvm: register qemu as ioreq server and retrieve shared pages

With QEMU disaggregation in Xen environment, each QEMU needs to ask Xen
for an ioreq server id. This id will be use to retrieve its private share
pages.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 xen-all.c |   80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 files changed, 76 insertions(+), 4 deletions(-)

diff --git a/xen-all.c b/xen-all.c
index df6927d..5f05838 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -36,6 +36,7 @@
 
 static MemoryRegion ram_memory, ram_640k, ram_lo, ram_hi;
 static MemoryRegion *framebuffer;
+static unsigned int serverid;
 static uint32_t xen_dmid = ~0;
 
 /* Compatibility with older version */
@@ -64,6 +65,67 @@ static inline ioreq_t *xen_vcpu_ioreq(shared_iopage_t
*shared_page, int vcpu)
 #define HVM_PARAM_BUFIOREQ_EVTCHN 26
 #endif
 
+#if __XEN_LATEST_INTERFACE_VERSION__ < 0x00040300
+static inline unsigned long xen_buffered_iopage(void)
+{
+    unsigned long pfn;
+
+    xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_BUFIOREQ_PFN, &pfn);
+
+    return pfn;
+}
+
+static inline unsigned long xen_iopage(void)
+{
+    unsigned long pfn;
+
+    xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_IOREQ_PFN, &pfn);
+
+    return pfn;
+}
+
+static inline evtchn_port_or_error_t xen_buffered_channel(void)
+{
+    unsigned long evtchn;
+    int rc;
+
+    rc = xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_BUFIOREQ_EVTCHN,
+                          &evtchn);
+
+    if (rc < 0) {
+        return rc;
+    } else {
+        return evtchn;
+    }
+}
+#else
+static inline unsigned long xen_buffered_iopage(void)
+{
+    unsigned long pfn;
+
+    xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_IO_PFN_FIRST, &pfn);
+    pfn += (serverid - 1) * 2 + 2;
+
+    return pfn;
+}
+
+static inline unsigned long xen_iopage(void)
+{
+    unsigned long pfn;
+
+    xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_IO_PFN_FIRST, &pfn);
+    pfn += (serverid - 1) * 2 + 1;
+
+    return pfn;
+}
+
+static inline evtchn_port_or_error_t xen_buffered_channel(void)
+{
+    return xc_hvm_get_ioreq_server_buf_channel(xen_xc, xen_domid, serverid);
+}
+
+#endif
+
 #define BUFFER_IO_MAX_DELAY  100
 
 typedef struct XenPhysmap {
@@ -1112,7 +1174,15 @@ int xen_hvm_init(void)
     state->suspend.notify = xen_suspend_notifier;
     qemu_register_suspend_notifier(&state->suspend);
 
-    xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_IOREQ_PFN, &ioreq_pfn);
+    rc = xen_xc_hvm_register_ioreq_server(xen_xc, xen_domid);
+
+    if (rc < 0) {
+        hw_error("registered server returned error %d", rc);
+    }
+
+    serverid = rc;
+
+    ioreq_pfn = xen_iopage();
     DPRINTF("shared page at pfn %lx\n", ioreq_pfn);
     state->shared_page = xc_map_foreign_range(xen_xc, xen_domid,
XC_PAGE_SIZE,
                                               PROT_READ|PROT_WRITE, ioreq_pfn);
@@ -1121,7 +1191,7 @@ int xen_hvm_init(void)
                  errno, xen_xc);
     }
 
-    xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_BUFIOREQ_PFN,
&ioreq_pfn);
+    ioreq_pfn = xen_buffered_iopage();
     DPRINTF("buffered io page at pfn %lx\n", ioreq_pfn);
     state->buffered_io_page = xc_map_foreign_range(xen_xc, xen_domid,
XC_PAGE_SIZE,
                                                    PROT_READ|PROT_WRITE,
ioreq_pfn);
@@ -1142,12 +1212,14 @@ int xen_hvm_init(void)
         state->ioreq_local_port[i] = rc;
     }
 
-    rc = xc_get_hvm_param(xen_xc, xen_domid, HVM_PARAM_BUFIOREQ_EVTCHN,
-            &bufioreq_evtchn);
+    rc = xen_buffered_channel();
     if (rc < 0) {
         fprintf(stderr, "failed to get HVM_PARAM_BUFIOREQ_EVTCHN\n");
         return -1;
     }
+
+    bufioreq_evtchn = rc;
+
     rc = xc_evtchn_bind_interdomain(state->xce_handle, xen_domid,
             (uint32_t)bufioreq_evtchn);
     if (rc == -1) {
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 05/10] xen-memory: register memory/IO range in Xen

Add Memory listener on IO and modify the one on memory.
Becareful, the first listener is not called is the range is still register with
register_ioport*. So Xen will never know that this QEMU is handle the range.

IO request works as before, the only thing is QEMU will never receive IO request
that it can''t handle.

y Changes to be committed:

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 xen-all.c |  113 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 113 insertions(+), 0 deletions(-)

diff --git a/xen-all.c b/xen-all.c
index 5f05838..14e5d3d 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -152,6 +152,7 @@ typedef struct XenIOState {
 
     struct xs_handle *xenstore;
     MemoryListener memory_listener;
+    MemoryListener io_listener;
     QLIST_HEAD(, XenPhysmap) physmap;
     target_phys_addr_t free_phys_offset;
     const XenPhysmap *log_for_dirtybit;
@@ -195,6 +196,31 @@ void xen_hvm_inject_msi(uint64_t addr, uint32_t data)
     xen_xc_hvm_inject_msi(xen_xc, xen_domid, addr, data);
 }
 
+static void xen_map_iorange(target_phys_addr_t addr, uint64_t size,
+                            int is_mmio, const char *name)
+{
+    /* Don''t register xen.ram */
+    if (is_mmio && !strncmp(name, "xen.ram", 7)) {
+        return;
+    }
+
+    DPRINTF("map %s %s 0x"TARGET_FMT_plx" -
0x"TARGET_FMT_plx"\n",
+            (is_mmio) ? "mmio" : "io", name, addr, addr +
size - 1);
+
+    xen_xc_hvm_map_io_range_to_ioreq_server(xen_xc, xen_domid, serverid,
+                                            is_mmio, addr, addr + size - 1);
+}
+
+static void xen_unmap_iorange(target_phys_addr_t addr, uint64_t size,
+                              int is_mmio)
+{
+    DPRINTF("unmap %s %s 0x"TARGET_FMT_plx" -
0x"TARGET_FMT_plx"\n",
+            (is_mmio) ? "mmio" : "io", name, addr, addr +
size - 1);
+
+    xen_xc_hvm_unmap_io_range_from_ioreq_server(xen_xc, xen_domid, serverid,
+                                                is_mmio, addr);
+}
+
 static void xen_suspend_notifier(Notifier *notifier, void *data)
 {
     xc_set_hvm_param(xen_xc, xen_domid, HVM_PARAM_ACPI_S_STATE, 3);
@@ -527,12 +553,16 @@ static void xen_region_add(MemoryListener *listener,
                            MemoryRegionSection *section)
 {
     xen_set_memory(listener, section, true);
+    xen_map_iorange(section->offset_within_address_space,
+                    section->size, 1, section->mr->name);
 }
 
 static void xen_region_del(MemoryListener *listener,
                            MemoryRegionSection *section)
 {
     xen_set_memory(listener, section, false);
+    xen_unmap_iorange(section->offset_within_address_space,
+                      section->size, 1);
 }
 
 static void xen_region_nop(MemoryListener *listener,
@@ -651,6 +681,86 @@ static MemoryListener xen_memory_listener = {
     .priority = 10,
 };
 
+static void xen_io_begin(MemoryListener *listener)
+{
+}
+
+static void xen_io_commit(MemoryListener *listener)
+{
+}
+
+static void xen_io_region_add(MemoryListener *listener,
+                              MemoryRegionSection *section)
+{
+    xen_map_iorange(section->offset_within_address_space,
+                    section->size, 0, section->mr->name);
+}
+
+static void xen_io_region_del(MemoryListener *listener,
+                              MemoryRegionSection *section)
+{
+    xen_unmap_iorange(section->offset_within_address_space,
+                      section->size, 0);
+}
+
+static void xen_io_region_nop(MemoryListener *listener,
+                              MemoryRegionSection *section)
+{
+}
+
+static void xen_io_log_start(MemoryListener *listener,
+                             MemoryRegionSection *section)
+{
+}
+
+static void xen_io_log_stop(MemoryListener *listener,
+                            MemoryRegionSection *section)
+{
+}
+
+static void xen_io_log_sync(MemoryListener *listener,
+                            MemoryRegionSection *section)
+{
+}
+
+static void xen_io_log_global_start(MemoryListener *listener)
+{
+}
+
+static void xen_io_log_global_stop(MemoryListener *listener)
+{
+}
+
+static void xen_io_eventfd_add(MemoryListener *listener,
+                               MemoryRegionSection *section,
+                               bool match_data, uint64_t data,
+                               EventNotifier *e)
+{
+}
+
+static void xen_io_eventfd_del(MemoryListener *listener,
+                               MemoryRegionSection *section,
+                               bool match_data, uint64_t data,
+                               EventNotifier *e)
+{
+}
+
+static MemoryListener xen_io_listener = {
+    .begin = xen_io_begin,
+    .commit = xen_io_commit,
+    .region_add = xen_io_region_add,
+    .region_del = xen_io_region_del,
+    .region_nop = xen_io_region_nop,
+    .log_start = xen_io_log_start,
+    .log_stop = xen_io_log_stop,
+    .log_sync = xen_io_log_sync,
+    .log_global_start = xen_io_log_global_start,
+    .log_global_stop = xen_io_log_global_stop,
+    .eventfd_add = xen_io_eventfd_add,
+    .eventfd_del = xen_io_eventfd_del,
+    .priority = 10,
+};
+
 /* VCPU Operations, MMIO, IO ring ... */
 
 static void xen_reset_vcpu(void *opaque)
@@ -1239,6 +1349,9 @@ int xen_hvm_init(void)
     memory_listener_register(&state->memory_listener,
get_system_memory());
     state->log_for_dirtybit = NULL;
 
+    state->io_listener = xen_io_listener;
+    memory_listener_register(&state->io_listener, get_system_io());
+
     /* Initialize backend core & drivers */
     if (xen_be_init() != 0) {
         fprintf(stderr, "%s: xen backend core setup failed\n",
__FUNCTION__);
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 06/10] xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG

With QEMU disaggregation QEMU needs to specify which PCI device it''s
able to
handle. It will use the device place in the topology (domain, bus, device,
function).
When Xen will trap an access for the config space, it will forge a new
ioreq and forward it to the right QEMU.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 hw/pci.c   |    6 ++++++
 hw/xen.h   |    1 +
 xen-all.c  |   38 ++++++++++++++++++++++++++++++++++++++
 xen-stub.c |    5 +++++
 4 files changed, 50 insertions(+), 0 deletions(-)

diff --git a/hw/pci.c b/hw/pci.c
index 4d95984..0112edf 100644
--- a/hw/pci.c
+++ b/hw/pci.c
@@ -33,6 +33,7 @@
 #include "qmp-commands.h"
 #include "msi.h"
 #include "msix.h"
+#include "xen.h"
 
 //#define DEBUG_PCI
 #ifdef DEBUG_PCI
@@ -781,6 +782,11 @@ static PCIDevice *do_pci_register_device(PCIDevice
*pci_dev, PCIBus *bus,
     pci_dev->devfn = devfn;
     pstrcpy(pci_dev->name, sizeof(pci_dev->name), name);
     pci_dev->irq_state = 0;
+
+    if (xen_enabled() && xen_register_pcidev(pci_dev)) {
+        return NULL;
+    }
+
     pci_config_alloc(pci_dev);
 
     pci_config_set_vendor_id(pci_dev->config, pc->vendor_id);
diff --git a/hw/xen.h b/hw/xen.h
index e5926b7..663731a 100644
--- a/hw/xen.h
+++ b/hw/xen.h
@@ -35,6 +35,7 @@ int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num);
 void xen_piix3_set_irq(void *opaque, int irq_num, int level);
 void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len);
 void xen_hvm_inject_msi(uint64_t addr, uint32_t data);
+int xen_register_pcidev(PCIDevice *pci_dev);
 void xen_cmos_set_s3_resume(void *opaque, int irq, int level);
 
 qemu_irq *xen_interrupt_controller_init(void);
diff --git a/xen-all.c b/xen-all.c
index 14e5d3d..485c312 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -174,6 +174,16 @@ void xen_piix3_set_irq(void *opaque, int irq_num, int
level)
                               irq_num & 3, level);
 }
 
+int xen_register_pcidev(PCIDevice *pci_dev)
+{
+    DPRINTF("register pci %x:%x.%x %s\n", 0, (pci_dev->devfn
>> 3) & 0x1f,
+            pci_dev->devfn & 0x7, pci_dev->name);
+
+    return xen_xc_hvm_register_pcidev(xen_xc, xen_domid, serverid,
+                                      0, 0, pci_dev->devfn >> 3,
+                                      pci_dev->devfn & 0x7);
+}
+
 void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len)
 {
     int i;
@@ -943,6 +953,29 @@ static void cpu_ioreq_move(ioreq_t *req)
     }
 }
 
+#if __XEN_LATEST_INTERFACE_VERSION__ >= 0x00040300
+static void cpu_ioreq_config_space(ioreq_t *req)
+{
+    uint64_t cf8 = req->addr;
+    uint32_t tmp = req->size;
+    uint16_t size = req->size & 0xff;
+    uint16_t off = req->size >> 16;
+
+    if ((size + off + 0xcfc) > 0xd00) {
+        hw_error("Invalid ioreq config space size = %u off = %u\n",
+                 size, off);
+    }
+
+    req->addr = 0xcfc + off;
+    req->size = size;
+
+    do_outp(0xcf8, 4, cf8);
+    cpu_ioreq_pio(req);
+    req->addr = cf8;
+    req->size = tmp;
+}
+#endif
+
 static void handle_ioreq(ioreq_t *req)
 {
     if (!req->data_is_ptr && (req->dir == IOREQ_WRITE) &&
@@ -962,6 +995,11 @@ static void handle_ioreq(ioreq_t *req)
         case IOREQ_TYPE_INVALIDATE:
             xen_invalidate_map_cache();
             break;
+#if __XEN_LATEST_INTERFACE_VERSION__ >= 0x00040300
+        case IOREQ_TYPE_PCI_CONFIG:
+            cpu_ioreq_config_space(req);
+            break;
+#endif
         default:
             hw_error("Invalid ioreq type 0x%x\n", req->type);
     }
diff --git a/xen-stub.c b/xen-stub.c
index 8ff2b79..0128965 100644
--- a/xen-stub.c
+++ b/xen-stub.c
@@ -25,6 +25,11 @@ void xen_piix3_set_irq(void *opaque, int irq_num, int level)
 {
 }
 
+int xen_register_pcidev(PCIDevice *pci_dev)
+{
+    return 1;
+}
+
 void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len)
 {
 }
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 07/10] xen: specify which device is part of default devices

One major problem of QEMU disaggregation is that some devices needs to
be "emulate" in each QEMU, but only one need to register it in Xen.

This patch introduces helpers that can be used in QEMU code (for
instance hw/pc_piix.c) to specify if the device is part of default sets.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 hw/pc_piix.c |    2 ++
 hw/xen.h     |   20 ++++++++++++++++++++
 xen-all.c    |   29 +++++++++++++++++++++++++++--
 3 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/hw/pc_piix.c b/hw/pc_piix.c
index 0c0096f..6cb0a2a 100644
--- a/hw/pc_piix.c
+++ b/hw/pc_piix.c
@@ -342,9 +342,11 @@ static void pc_xen_hvm_init(ram_addr_t ram_size,
     if (xen_hvm_init() != 0) {
         hw_error("xen hardware virtual machine initialisation
failed");
     }
+    xen_set_register_default_dev(1,  NULL);
     pc_init_pci_no_kvmclock(ram_size, boot_device,
                             kernel_filename, kernel_cmdline,
                             initrd_filename, cpu_model);
+    xen_set_register_default_dev(0, NULL);
     xen_vcpu_init();
 }
 #endif
diff --git a/hw/xen.h b/hw/xen.h
index 663731a..3c8724f 100644
--- a/hw/xen.h
+++ b/hw/xen.h
@@ -21,6 +21,7 @@ extern uint32_t xen_domid;
 extern enum xen_mode xen_mode;
 
 extern int xen_allowed;
+extern int xen_register_default_dev;
 
 static inline int xen_enabled(void)
 {
@@ -31,6 +32,25 @@ static inline int xen_enabled(void)
 #endif
 }
 
+static inline int xen_is_registered_default_dev(void)
+{
+#if defined(CONFIG_XEN)
+    return xen_register_default_dev;
+#else
+    return 1;
+#endif
+}
+
+static inline void xen_set_register_default_dev(int val, int *old)
+{
+#if defined(CONFIG_XEN)
+    if (old) {
+        *old = xen_register_default_dev;
+    }
+    xen_register_default_dev = val;
+#endif
+}
+
 int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num);
 void xen_piix3_set_irq(void *opaque, int irq_num, int level);
 void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len);
diff --git a/xen-all.c b/xen-all.c
index 485c312..afa9bcc 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -39,6 +39,10 @@ static MemoryRegion *framebuffer;
 static unsigned int serverid;
 static uint32_t xen_dmid = ~0;
 
+/* Use to tell if we register pci/mmio/pio of default devices */
+int xen_register_default_dev = 0;
+static int xen_emulate_default_dev = 1;
+
 /* Compatibility with older version */
 #if __XEN_LATEST_INTERFACE_VERSION__ < 0x0003020a
 static inline uint32_t xen_vcpu_eport(shared_iopage_t *shared_page, int i)
@@ -176,6 +180,10 @@ void xen_piix3_set_irq(void *opaque, int irq_num, int
level)
 
 int xen_register_pcidev(PCIDevice *pci_dev)
 {
+    if (xen_register_default_dev && !xen_emulate_default_dev) {
+        return 0;
+    }
+
     DPRINTF("register pci %x:%x.%x %s\n", 0, (pci_dev->devfn
>> 3) & 0x1f,
             pci_dev->devfn & 0x7, pci_dev->name);
 
@@ -214,6 +222,18 @@ static void xen_map_iorange(target_phys_addr_t addr,
uint64_t size,
         return;
     }
 
+    /* Handle the registration of all default io range */
+    if (xen_register_default_dev) {
+        /* Register ps/2 only if we emulate VGA */
+        if (!strcmp(name, "i8042-data") || !strcmp(name,
"i8042-cmd")) {
+            if (display_type == DT_NOGRAPHIC) {
+                return;
+            }
+        } else if (!xen_emulate_default_dev && strcmp(name,
"serial")) {
+            return;
+        }
+    }
+
     DPRINTF("map %s %s 0x"TARGET_FMT_plx" -
0x"TARGET_FMT_plx"\n",
             (is_mmio) ? "mmio" : "io", name, addr, addr +
size - 1);
 
@@ -1300,6 +1320,8 @@ int xen_hvm_init(void)
     if (!QTAILQ_EMPTY(&list->head)) {
         xen_dmid = qemu_opt_get_number(QTAILQ_FIRST(&list->head),
                                        "xen_dmid", ~0);
+        xen_emulate_default_dev =
qemu_opt_get_bool(QTAILQ_FIRST(&list->head),
+                                                   
"xen_default_dev", 1);
     }
 
     state = g_malloc0(sizeof (XenIOState));
@@ -1395,9 +1417,12 @@ int xen_hvm_init(void)
         fprintf(stderr, "%s: xen backend core setup failed\n",
__FUNCTION__);
         exit(1);
     }
-    xen_be_register("console", &xen_console_ops);
-    xen_be_register("vkbd", &xen_kbdmouse_ops);
     xen_be_register("qdisk", &xen_blkdev_ops);
+
+    if (xen_emulate_default_dev) {
+        xen_be_register("console", &xen_console_ops);
+        xen_be_register("vkbd", &xen_kbdmouse_ops);
+    }
     xen_read_physmap(state);
 
     return 0;
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 08/10] xen: audio is not a part of default devices

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 arch_init.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 9b46bfc..1077b16 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -44,6 +44,7 @@
 #include "exec-memory.h"
 #include "hw/pcspk.h"
 #include "qemu/page_cache.h"
+#include "hw/xen.h"
 
 #ifdef DEBUG_ARCH_INIT
 #define DPRINTF(fmt, ...) \
@@ -976,6 +977,9 @@ void select_soundhw(const char *optarg)
 void audio_init(ISABus *isa_bus, PCIBus *pci_bus)
 {
     struct soundhw *c;
+    int register_default_dev;
+
+    xen_set_register_default_dev(0, &register_default_dev);
 
     for (c = soundhw; c->name; ++c) {
         if (c->enabled) {
@@ -990,6 +994,8 @@ void audio_init(ISABus *isa_bus, PCIBus *pci_bus)
             }
         }
     }
+
+    xen_set_register_default_dev(register_default_dev, NULL);
 }
 #else
 void select_soundhw(const char *optarg)
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 09/10] xen-memory: handle node "device_model" for physical mapping

Retrieve only physical mapping where device model corresponds to dmid.
When a new physical mapping is added, specify the device model id of the
current QEMU.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 xen-all.c |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/xen-all.c b/xen-all.c
index afa9bcc..f424cce 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -440,6 +440,14 @@ go_physmap:
                                    XEN_DOMCTL_MEM_CACHEATTR_WB);
 
     snprintf(path, sizeof(path),
+            
"/local/domain/0/device-model/%d/physmap/%"PRIx64"/device_model",
+             xen_domid, (uint64_t)phys_offset);
+    snprintf(value, sizeof(value), "%u", xen_dmid);
+    if (!xs_write(state->xenstore, 0, path, value, strlen(value))) {
+        return -1;
+    }
+
+    snprintf(path, sizeof(path),
            
"/local/domain/0/device-model/%d/physmap/%"PRIx64"/start_addr",
             xen_domid, (uint64_t)phys_offset);
     snprintf(value, sizeof(value), "%"PRIx64, (uint64_t)start_addr);
@@ -1266,6 +1274,7 @@ static void xen_read_physmap(XenIOState *state)
     unsigned int len, num, i;
     char path[80], *value = NULL;
     char **entries = NULL;
+    uint32_t dmid = ~0;
 
     snprintf(path, sizeof(path),
             "/local/domain/0/device-model/%d/physmap", xen_domid);
@@ -1274,6 +1283,17 @@ static void xen_read_physmap(XenIOState *state)
         return;
 
     for (i = 0; i < num; i++) {
+        snprintf(path, sizeof(path),
+                
"/local/domain/0/device-model/%d/physmap/%s/device_model",
+                 xen_domid, entries[i]);
+        value = xs_read(state->xenstore, 0, path, &len);
+        if (value) {
+            dmid = strtoul(value, NULL, 10);
+            free(value);
+            if (dmid != xen_dmid) {
+                continue;
+            }
+        }
         physmap = g_malloc(sizeof (XenPhysmap));
         physmap->phys_offset = strtoull(entries[i], NULL, 16);
         snprintf(path, sizeof(path),
-- 
Julien Grall

Julien Grall

2012-Aug-22 12:30 UTC

head link

[QEMU][RFC V2 10/10] xen: emulate IDE outside default device set

IDE can be emulate in a different QEMU that the default.

This patch also fixes ide_get_geometry. When QEMU didn''t emulate IDE,
it try to derefence a NULL bus.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 hw/ide/qdev.c |    8 +++++++-
 hw/pc_piix.c  |   38 ++++++++++++++++++++++++--------------
 hw/xen.h      |   10 ++++++++++
 xen-all.c     |    8 +++++++-
 4 files changed, 48 insertions(+), 16 deletions(-)

diff --git a/hw/ide/qdev.c b/hw/ide/qdev.c
index 5ea9b8f..473acd7 100644
--- a/hw/ide/qdev.c
+++ b/hw/ide/qdev.c
@@ -115,7 +115,13 @@ IDEDevice *ide_create_drive(IDEBus *bus, int unit,
DriveInfo *drive)
 int ide_get_geometry(BusState *bus, int unit,
                      int16_t *cyls, int8_t *heads, int8_t *secs)
 {
-    IDEState *s = &DO_UPCAST(IDEBus, qbus, bus)->ifs[unit];
+    IDEState *s = NULL;
+
+    if (!bus) {
+        return -1;
+    }
+
+    s = &DO_UPCAST(IDEBus, qbus, bus)->ifs[unit];
 
     if (s->drive_kind != IDE_HD || !s->bs) {
         return -1;
diff --git a/hw/pc_piix.c b/hw/pc_piix.c
index 6cb0a2a..b904100 100644
--- a/hw/pc_piix.c
+++ b/hw/pc_piix.c
@@ -148,6 +148,7 @@ static void pc_init1(MemoryRegion *system_memory,
     MemoryRegion *pci_memory;
     MemoryRegion *rom_memory;
     void *fw_cfg = NULL;
+    int register_default_dev = 0;
 
     pc_cpus_init(cpu_model);
 
@@ -242,23 +243,32 @@ static void pc_init1(MemoryRegion *system_memory,
             pci_nic_init_nofail(nd, "e1000", NULL);
     }
 
-    ide_drive_get(hd, MAX_IDE_BUS);
-    if (pci_enabled) {
-        PCIDevice *dev;
-        if (xen_enabled()) {
-            dev = pci_piix3_xen_ide_init(pci_bus, hd, piix3_devfn + 1);
+    if (!xen_enabled() || xen_is_emulated_ide()) {
+        xen_set_register_default_dev(0, &register_default_dev);
+        ide_drive_get(hd, MAX_IDE_BUS);
+        if (pci_enabled) {
+            PCIDevice *dev;
+            if (xen_enabled()) {
+                dev = pci_piix3_xen_ide_init(pci_bus, hd, piix3_devfn + 1);
+            } else {
+                dev = pci_piix3_ide_init(pci_bus, hd, piix3_devfn + 1);
+            }
+            idebus[0] = qdev_get_child_bus(&dev->qdev,
"ide.0");
+            idebus[1] = qdev_get_child_bus(&dev->qdev,
"ide.1");
         } else {
-            dev = pci_piix3_ide_init(pci_bus, hd, piix3_devfn + 1);
+            for (i = 0; i < MAX_IDE_BUS; i++) {
+                ISADevice *dev;
+                dev = isa_ide_init(isa_bus, ide_iobase[i], ide_iobase2[i],
+                                   ide_irq[i],
+                                   hd[MAX_IDE_DEVS * i],
+                                   hd[MAX_IDE_DEVS * i + 1]);
+                idebus[i] = qdev_get_child_bus(&dev->qdev,
"ide.0");
+            }
         }
-        idebus[0] = qdev_get_child_bus(&dev->qdev, "ide.0");
-        idebus[1] = qdev_get_child_bus(&dev->qdev, "ide.1");
+        xen_set_register_default_dev(register_default_dev, NULL);
     } else {
-        for(i = 0; i < MAX_IDE_BUS; i++) {
-            ISADevice *dev;
-            dev = isa_ide_init(isa_bus, ide_iobase[i], ide_iobase2[i],
-                               ide_irq[i],
-                               hd[MAX_IDE_DEVS * i], hd[MAX_IDE_DEVS * i + 1]);
-            idebus[i] = qdev_get_child_bus(&dev->qdev,
"ide.0");
+        for (i = 0; i < MAX_IDE_BUS; i++) {
+            idebus[i] = NULL;
         }
     }
 
diff --git a/hw/xen.h b/hw/xen.h
index 3c8724f..3c89fb9 100644
--- a/hw/xen.h
+++ b/hw/xen.h
@@ -22,6 +22,7 @@ extern enum xen_mode xen_mode;
 
 extern int xen_allowed;
 extern int xen_register_default_dev;
+extern int xen_emulate_ide;
 
 static inline int xen_enabled(void)
 {
@@ -32,6 +33,15 @@ static inline int xen_enabled(void)
 #endif
 }
 
+static inline int xen_is_emulated_ide(void)
+{
+#if defined(CONFIG_XEN)
+    return xen_emulate_ide;
+#else
+    return 1;
+#endif
+}
+
 static inline int xen_is_registered_default_dev(void)
 {
 #if defined(CONFIG_XEN)
diff --git a/xen-all.c b/xen-all.c
index f424cce..f091908 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -43,6 +43,8 @@ static uint32_t xen_dmid = ~0;
 int xen_register_default_dev = 0;
 static int xen_emulate_default_dev = 1;
 
+int xen_emulate_ide = 0;
+
 /* Compatibility with older version */
 #if __XEN_LATEST_INTERFACE_VERSION__ < 0x0003020a
 static inline uint32_t xen_vcpu_eport(shared_iopage_t *shared_page, int i)
@@ -1342,6 +1344,8 @@ int xen_hvm_init(void)
                                        "xen_dmid", ~0);
         xen_emulate_default_dev =
qemu_opt_get_bool(QTAILQ_FIRST(&list->head),
                                                    
"xen_default_dev", 1);
+        xen_emulate_ide = qemu_opt_get_bool(QTAILQ_FIRST(&list->head),
+                                            "xen_emulate_ide", 1);
     }
 
     state = g_malloc0(sizeof (XenIOState));
@@ -1437,12 +1441,14 @@ int xen_hvm_init(void)
         fprintf(stderr, "%s: xen backend core setup failed\n",
__FUNCTION__);
         exit(1);
     }
-    xen_be_register("qdisk", &xen_blkdev_ops);
 
     if (xen_emulate_default_dev) {
         xen_be_register("console", &xen_console_ops);
         xen_be_register("vkbd", &xen_kbdmouse_ops);
     }
+    if (xen_emulate_ide) {
+        xen_be_register("qdisk", &xen_blkdev_ops);
+    }
     xen_read_physmap(state);
 
     return 0;
-- 
Julien Grall

Stefano Stabellini

2012-Aug-23 14:40 UTC

head link

Re: [QEMU][RFC V2 01/10] xen: add new machine options to support QEMU disaggregation in Xen environment

On Wed, 22 Aug 2012, Julien Grall wrote:>     - xen_dmid: specify the id of QEMU. It will be used to
>     retrieve/store information inside XenStore.
>     - xen_default_dev (on/off): as default devices need to be create in
>     each QEMU (due to code dependency), this option specifies if it will
>     register range/PCI of default device via xen hypercall.
>     (Root bridge, south bridge, ...).
>     - xen_emulate_ide (on/off): enable/disable emulation in QEMU.
> 
> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
>  qemu-config.c |   12 ++++++++++++
>  1 files changed, 12 insertions(+), 0 deletions(-)
> 
> diff --git a/qemu-config.c b/qemu-config.c
> index c05ffbc..7740442 100644
> --- a/qemu-config.c
> +++ b/qemu-config.c
> @@ -612,6 +612,18 @@ static QemuOptsList qemu_machine_opts = {
>              .name = "dump-guest-core",
>              .type = QEMU_OPT_BOOL,
>              .help = "Include guest memory in  a core dump",
> +        }, {
> +            .name = "xen_dmid",
> +            .type = QEMU_OPT_NUMBER,
> +            .help = "Xen device model id",
> +        }, {
> +            .name = "xen_default_dev",
> +            .type = QEMU_OPT_BOOL,
> +            .help = "emulate Xen default device (South Bridge, IDE,
...)"                                                ^devices
It would be good to document exactly which ones are these (the default
devices)
> +        }, {
> +            .name = "xen_emulate_ide",
> +            .type = QEMU_OPT_BOOL,
> +            .help = "emulate IDE with XEN"
>          },
Is this really a Xen specific option? Couldn''t be used for example by
people that wants to use virtio-scsi on KVM without IDE emulation?

>          { /* End of list */ }
>      },
> -- 
> Julien Grall
>

Stefano Stabellini

2012-Aug-23 14:41 UTC

head link

Re: [QEMU][RFC V2 05/10] xen-memory: register memory/IO range in Xen

On Wed, 22 Aug 2012, Julien Grall wrote:> Add Memory listener on IO and modify the one on memory.
> Becareful, the first listener is not called is the range is still register
with
> register_ioport*. So Xen will never know that this QEMU is handle the
range.
I don''t understand what you mean here. Could you please elaborate?

> IO request works as before, the only thing is QEMU will never receive IO
request
> that it can''t handle.
> 
> y Changes to be committed:
Just remove this line from the commit message.

> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
>  xen-all.c |  113
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 113 insertions(+), 0 deletions(-)
> 
> diff --git a/xen-all.c b/xen-all.c
> index 5f05838..14e5d3d 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -152,6 +152,7 @@ typedef struct XenIOState {
>  
>      struct xs_handle *xenstore;
>      MemoryListener memory_listener;
> +    MemoryListener io_listener;
>      QLIST_HEAD(, XenPhysmap) physmap;
>      target_phys_addr_t free_phys_offset;
>      const XenPhysmap *log_for_dirtybit;
> @@ -195,6 +196,31 @@ void xen_hvm_inject_msi(uint64_t addr, uint32_t data)
>      xen_xc_hvm_inject_msi(xen_xc, xen_domid, addr, data);
>  }
>  
> +static void xen_map_iorange(target_phys_addr_t addr, uint64_t size,
> +                            int is_mmio, const char *name)
> +{
> +    /* Don''t register xen.ram */
> +    if (is_mmio && !strncmp(name, "xen.ram", 7)) {
> +        return;
> +    }
> +
> +    DPRINTF("map %s %s 0x"TARGET_FMT_plx" -
0x"TARGET_FMT_plx"\n",
> +            (is_mmio) ? "mmio" : "io", name, addr,
addr + size - 1);
> +
> +    xen_xc_hvm_map_io_range_to_ioreq_server(xen_xc, xen_domid, serverid,
> +                                            is_mmio, addr, addr + size -
1);
> +}
> +
> +static void xen_unmap_iorange(target_phys_addr_t addr, uint64_t size,
> +                              int is_mmio)
> +{
> +    DPRINTF("unmap %s %s 0x"TARGET_FMT_plx" -
0x"TARGET_FMT_plx"\n",
> +            (is_mmio) ? "mmio" : "io", name, addr,
addr + size - 1);
> +
> +    xen_xc_hvm_unmap_io_range_from_ioreq_server(xen_xc, xen_domid,
serverid,
> +                                                is_mmio, addr);
> +}
> +
>  static void xen_suspend_notifier(Notifier *notifier, void *data)
>  {
>      xc_set_hvm_param(xen_xc, xen_domid, HVM_PARAM_ACPI_S_STATE, 3);
> @@ -527,12 +553,16 @@ static void xen_region_add(MemoryListener *listener,
>                             MemoryRegionSection *section)
>  {
>      xen_set_memory(listener, section, true);
> +    xen_map_iorange(section->offset_within_address_space,
> +                    section->size, 1, section->mr->name);
>  }
>  
>  static void xen_region_del(MemoryListener *listener,
>                             MemoryRegionSection *section)
>  {
>      xen_set_memory(listener, section, false);
> +    xen_unmap_iorange(section->offset_within_address_space,
> +                      section->size, 1);
>  }
>  
>  static void xen_region_nop(MemoryListener *listener,
> @@ -651,6 +681,86 @@ static MemoryListener xen_memory_listener = {
>      .priority = 10,
>  };
>  
> +static void xen_io_begin(MemoryListener *listener)
> +{
> +}
> +
> +static void xen_io_commit(MemoryListener *listener)
> +{
> +}
> +
> +static void xen_io_region_add(MemoryListener *listener,
> +                              MemoryRegionSection *section)
> +{
> +    xen_map_iorange(section->offset_within_address_space,
> +                    section->size, 0, section->mr->name);
> +}
> +
> +static void xen_io_region_del(MemoryListener *listener,
> +                              MemoryRegionSection *section)
> +{
> +    xen_unmap_iorange(section->offset_within_address_space,
> +                      section->size, 0);
> +}
> +
> +static void xen_io_region_nop(MemoryListener *listener,
> +                              MemoryRegionSection *section)
> +{
> +}
> +
> +static void xen_io_log_start(MemoryListener *listener,
> +                             MemoryRegionSection *section)
> +{
> +}
> +
> +static void xen_io_log_stop(MemoryListener *listener,
> +                            MemoryRegionSection *section)
> +{
> +}
> +
> +static void xen_io_log_sync(MemoryListener *listener,
> +                            MemoryRegionSection *section)
> +{
> +}
> +
> +static void xen_io_log_global_start(MemoryListener *listener)
> +{
> +}
> +
> +static void xen_io_log_global_stop(MemoryListener *listener)
> +{
> +}
> +
> +static void xen_io_eventfd_add(MemoryListener *listener,
> +                               MemoryRegionSection *section,
> +                               bool match_data, uint64_t data,
> +                               EventNotifier *e)
> +{
> +}
> +
> +static void xen_io_eventfd_del(MemoryListener *listener,
> +                               MemoryRegionSection *section,
> +                               bool match_data, uint64_t data,
> +                               EventNotifier *e)
> +{
> +}
> +
> +static MemoryListener xen_io_listener = {
> +    .begin = xen_io_begin,
> +    .commit = xen_io_commit,
> +    .region_add = xen_io_region_add,
> +    .region_del = xen_io_region_del,
> +    .region_nop = xen_io_region_nop,
> +    .log_start = xen_io_log_start,
> +    .log_stop = xen_io_log_stop,
> +    .log_sync = xen_io_log_sync,
> +    .log_global_start = xen_io_log_global_start,
> +    .log_global_stop = xen_io_log_global_stop,
> +    .eventfd_add = xen_io_eventfd_add,
> +    .eventfd_del = xen_io_eventfd_del,
> +    .priority = 10,
> +};
> +
>  /* VCPU Operations, MMIO, IO ring ... */
>  
>  static void xen_reset_vcpu(void *opaque)
> @@ -1239,6 +1349,9 @@ int xen_hvm_init(void)
>      memory_listener_register(&state->memory_listener,
get_system_memory());
>      state->log_for_dirtybit = NULL;
>  
> +    state->io_listener = xen_io_listener;
> +    memory_listener_register(&state->io_listener, get_system_io());
> +
>      /* Initialize backend core & drivers */
>      if (xen_be_init() != 0) {
>          fprintf(stderr, "%s: xen backend core setup failed\n",
__FUNCTION__);
> -- 
> Julien Grall
>

Stefano Stabellini

2012-Aug-23 14:41 UTC

head link

Re: [QEMU][RFC V2 06/10] xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG

On Wed, 22 Aug 2012, Julien Grall wrote:> With QEMU disaggregation QEMU needs to specify which PCI device
it''s able to
> handle. It will use the device place in the topology (domain, bus, device,
> function).
> When Xen will trap an access for the config space, it will forge a new
> ioreq and forward it to the right QEMU.
> 
> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
>  hw/pci.c   |    6 ++++++
>  hw/xen.h   |    1 +
>  xen-all.c  |   38 ++++++++++++++++++++++++++++++++++++++
>  xen-stub.c |    5 +++++
>  4 files changed, 50 insertions(+), 0 deletions(-)
> 
> diff --git a/hw/pci.c b/hw/pci.c
> index 4d95984..0112edf 100644
> --- a/hw/pci.c
> +++ b/hw/pci.c
> @@ -33,6 +33,7 @@
>  #include "qmp-commands.h"
>  #include "msi.h"
>  #include "msix.h"
> +#include "xen.h"
>  
>  //#define DEBUG_PCI
>  #ifdef DEBUG_PCI
> @@ -781,6 +782,11 @@ static PCIDevice *do_pci_register_device(PCIDevice
*pci_dev, PCIBus *bus,
>      pci_dev->devfn = devfn;
>      pstrcpy(pci_dev->name, sizeof(pci_dev->name), name);
>      pci_dev->irq_state = 0;
> +
> +    if (xen_enabled() && xen_register_pcidev(pci_dev)) {
> +        return NULL;
Is this an error condition? If so we should print an error message,
right?

> +    }
> +
>      pci_config_alloc(pci_dev);
>  
>      pci_config_set_vendor_id(pci_dev->config, pc->vendor_id);
> diff --git a/hw/xen.h b/hw/xen.h
> index e5926b7..663731a 100644
> --- a/hw/xen.h
> +++ b/hw/xen.h
> @@ -35,6 +35,7 @@ int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int
irq_num);
>  void xen_piix3_set_irq(void *opaque, int irq_num, int level);
>  void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int
len);
>  void xen_hvm_inject_msi(uint64_t addr, uint32_t data);
> +int xen_register_pcidev(PCIDevice *pci_dev);
>  void xen_cmos_set_s3_resume(void *opaque, int irq, int level);
>  
>  qemu_irq *xen_interrupt_controller_init(void);
> diff --git a/xen-all.c b/xen-all.c
> index 14e5d3d..485c312 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -174,6 +174,16 @@ void xen_piix3_set_irq(void *opaque, int irq_num, int
level)
>                                irq_num & 3, level);
>  }
>  
> +int xen_register_pcidev(PCIDevice *pci_dev)
> +{
> +    DPRINTF("register pci %x:%x.%x %s\n", 0, (pci_dev->devfn
>> 3) & 0x1f,
> +            pci_dev->devfn & 0x7, pci_dev->name);
> +
> +    return xen_xc_hvm_register_pcidev(xen_xc, xen_domid, serverid,
> +                                      0, 0, pci_dev->devfn >> 3,
> +                                      pci_dev->devfn & 0x7);
> +}
> +
>  void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int
len)
>  {
>      int i;
> @@ -943,6 +953,29 @@ static void cpu_ioreq_move(ioreq_t *req)
>      }
>  }
>  
> +#if __XEN_LATEST_INTERFACE_VERSION__ >= 0x00040300
> +static void cpu_ioreq_config_space(ioreq_t *req)
> +{
> +    uint64_t cf8 = req->addr;
> +    uint32_t tmp = req->size;
> +    uint16_t size = req->size & 0xff;
> +    uint16_t off = req->size >> 16;
> +
> +    if ((size + off + 0xcfc) > 0xd00) {
> +        hw_error("Invalid ioreq config space size = %u off =
%u\n",
> +                 size, off);
> +    }
> +
> +    req->addr = 0xcfc + off;
> +    req->size = size;
> +
> +    do_outp(0xcf8, 4, cf8);
> +    cpu_ioreq_pio(req);
> +    req->addr = cf8;
> +    req->size = tmp;
> +}
> +#endif
> +
>  static void handle_ioreq(ioreq_t *req)
>  {
>      if (!req->data_is_ptr && (req->dir == IOREQ_WRITE)
&&
> @@ -962,6 +995,11 @@ static void handle_ioreq(ioreq_t *req)
>          case IOREQ_TYPE_INVALIDATE:
>              xen_invalidate_map_cache();
>              break;
> +#if __XEN_LATEST_INTERFACE_VERSION__ >= 0x00040300
> +        case IOREQ_TYPE_PCI_CONFIG:
> +            cpu_ioreq_config_space(req);
> +            break;
> +#endif
>          default:
>              hw_error("Invalid ioreq type 0x%x\n", req->type);
>      }
> diff --git a/xen-stub.c b/xen-stub.c
> index 8ff2b79..0128965 100644
> --- a/xen-stub.c
> +++ b/xen-stub.c
> @@ -25,6 +25,11 @@ void xen_piix3_set_irq(void *opaque, int irq_num, int
level)
>  {
>  }
>  
> +int xen_register_pcidev(PCIDevice *pci_dev)
> +{
> +    return 1;
> +}
> +
>  void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int
len)
>  {
>  }
> -- 
> Julien Grall
>

Stefano Stabellini

2012-Aug-23 14:42 UTC

head link

Re: [QEMU][RFC V2 07/10] xen: specify which device is part of default devices

On Wed, 22 Aug 2012, Julien Grall wrote:> One major problem of QEMU disaggregation is that some devices needs to
> be "emulate" in each QEMU, but only one need to register it in
Xen.
> 
> This patch introduces helpers that can be used in QEMU code (for
> instance hw/pc_piix.c) to specify if the device is part of default sets.
> 
> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
>  hw/pc_piix.c |    2 ++
>  hw/xen.h     |   20 ++++++++++++++++++++
>  xen-all.c    |   29 +++++++++++++++++++++++++++--
>  3 files changed, 49 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/pc_piix.c b/hw/pc_piix.c
> index 0c0096f..6cb0a2a 100644
> --- a/hw/pc_piix.c
> +++ b/hw/pc_piix.c
> @@ -342,9 +342,11 @@ static void pc_xen_hvm_init(ram_addr_t ram_size,
>      if (xen_hvm_init() != 0) {
>          hw_error("xen hardware virtual machine initialisation
failed");
>      }
> +    xen_set_register_default_dev(1,  NULL);
>      pc_init_pci_no_kvmclock(ram_size, boot_device,
>                              kernel_filename, kernel_cmdline,
>                              initrd_filename, cpu_model);
> +    xen_set_register_default_dev(0, NULL);
>      xen_vcpu_init();
>  }
Honestly I don''t like this interface, I would rather have an explicit
list of "default devices" and then go through them in
xen_register_pcidev and xen_map_iorange.

>  #endif
> diff --git a/hw/xen.h b/hw/xen.h
> index 663731a..3c8724f 100644
> --- a/hw/xen.h
> +++ b/hw/xen.h
> @@ -21,6 +21,7 @@ extern uint32_t xen_domid;
>  extern enum xen_mode xen_mode;
>  
>  extern int xen_allowed;
> +extern int xen_register_default_dev;
>  
>  static inline int xen_enabled(void)
>  {
> @@ -31,6 +32,25 @@ static inline int xen_enabled(void)
>  #endif
>  }
>  
> +static inline int xen_is_registered_default_dev(void)
> +{
> +#if defined(CONFIG_XEN)
> +    return xen_register_default_dev;
> +#else
> +    return 1;
> +#endif
> +}
> +
> +static inline void xen_set_register_default_dev(int val, int *old)
> +{
> +#if defined(CONFIG_XEN)
> +    if (old) {
> +        *old = xen_register_default_dev;
> +    }
> +    xen_register_default_dev = val;
> +#endif
> +}
> +
>  int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num);
>  void xen_piix3_set_irq(void *opaque, int irq_num, int level);
>  void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int
len);
> diff --git a/xen-all.c b/xen-all.c
> index 485c312..afa9bcc 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -39,6 +39,10 @@ static MemoryRegion *framebuffer;
>  static unsigned int serverid;
>  static uint32_t xen_dmid = ~0;
>  
> +/* Use to tell if we register pci/mmio/pio of default devices */
> +int xen_register_default_dev = 0;
> +static int xen_emulate_default_dev = 1;
>
>  /* Compatibility with older version */
>  #if __XEN_LATEST_INTERFACE_VERSION__ < 0x0003020a
>  static inline uint32_t xen_vcpu_eport(shared_iopage_t *shared_page, int i)
> @@ -176,6 +180,10 @@ void xen_piix3_set_irq(void *opaque, int irq_num, int
level)
>  
>  int xen_register_pcidev(PCIDevice *pci_dev)
>  {
> +    if (xen_register_default_dev && !xen_emulate_default_dev) {
> +        return 0;
> +    }
>
>      DPRINTF("register pci %x:%x.%x %s\n", 0, (pci_dev->devfn
>> 3) & 0x1f,
>              pci_dev->devfn & 0x7, pci_dev->name);
>  
> @@ -214,6 +222,18 @@ static void xen_map_iorange(target_phys_addr_t addr,
uint64_t size,
>          return;
>      }
>  
> +    /* Handle the registration of all default io range */
> +    if (xen_register_default_dev) {
> +        /* Register ps/2 only if we emulate VGA */
> +        if (!strcmp(name, "i8042-data") || !strcmp(name,
"i8042-cmd")) {
> +            if (display_type == DT_NOGRAPHIC) {
> +                return;
> +            }
> +        } else if (!xen_emulate_default_dev && strcmp(name,
"serial")) {
> +            return;
> +        }
> +    }
It seems to be that you are acting upon xen_register_default_dev only in
xen_map_iorange and xen_register_pcidev, so it shouldn''t be difficult
to
have a real list of "default devices" instead.

>      DPRINTF("map %s %s 0x"TARGET_FMT_plx" -
0x"TARGET_FMT_plx"\n",
>              (is_mmio) ? "mmio" : "io", name, addr,
addr + size - 1);
>  
> @@ -1300,6 +1320,8 @@ int xen_hvm_init(void)
>      if (!QTAILQ_EMPTY(&list->head)) {
>          xen_dmid = qemu_opt_get_number(QTAILQ_FIRST(&list->head),
>                                         "xen_dmid", ~0);
> +        xen_emulate_default_dev =
qemu_opt_get_bool(QTAILQ_FIRST(&list->head),
> +                                                   
"xen_default_dev", 1);
>      }
>  
>      state = g_malloc0(sizeof (XenIOState));
> @@ -1395,9 +1417,12 @@ int xen_hvm_init(void)
>          fprintf(stderr, "%s: xen backend core setup failed\n",
__FUNCTION__);
>          exit(1);
>      }
> -    xen_be_register("console", &xen_console_ops);
> -    xen_be_register("vkbd", &xen_kbdmouse_ops);
>      xen_be_register("qdisk", &xen_blkdev_ops);
> +
> +    if (xen_emulate_default_dev) {
> +        xen_be_register("console", &xen_console_ops);
> +        xen_be_register("vkbd", &xen_kbdmouse_ops);
> +    }
>      xen_read_physmap(state);

Stefano Stabellini

2012-Aug-23 14:42 UTC

head link

Re: [QEMU][RFC V2 08/10] xen: audio is not a part of default devices

On Wed, 22 Aug 2012, Julien Grall wrote:> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
>  arch_init.c |    6 ++++++
>  1 files changed, 6 insertions(+), 0 deletions(-)
> 
> diff --git a/arch_init.c b/arch_init.c
> index 9b46bfc..1077b16 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -44,6 +44,7 @@
>  #include "exec-memory.h"
>  #include "hw/pcspk.h"
>  #include "qemu/page_cache.h"
> +#include "hw/xen.h"
>  
>  #ifdef DEBUG_ARCH_INIT
>  #define DPRINTF(fmt, ...) \
> @@ -976,6 +977,9 @@ void select_soundhw(const char *optarg)
>  void audio_init(ISABus *isa_bus, PCIBus *pci_bus)
>  {
>      struct soundhw *c;
> +    int register_default_dev;
> +
> +    xen_set_register_default_dev(0, &register_default_dev);
>  
>      for (c = soundhw; c->name; ++c) {
>          if (c->enabled) {
> @@ -990,6 +994,8 @@ void audio_init(ISABus *isa_bus, PCIBus *pci_bus)
>              }
>          }
>      }
> +
> +    xen_set_register_default_dev(register_default_dev, NULL);
>  }
>  #else
>  void select_soundhw(const char *optarg)
and this is why it is better to have a list rather than a stateful
register_default_dev integer.
This stuff is really easy to break.

Stefano Stabellini

2012-Aug-23 14:43 UTC

head link

Re: [QEMU][RFC V2 10/10] xen: emulate IDE outside default device set

On Wed, 22 Aug 2012, Julien Grall wrote:> IDE can be emulate in a different QEMU that the default.
> 
> This patch also fixes ide_get_geometry. When QEMU didn''t emulate
IDE,                                                    
^doesn''t> it try to derefence a NULL bus.      ^tries to dereference

> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
>  hw/ide/qdev.c |    8 +++++++-
>  hw/pc_piix.c  |   38 ++++++++++++++++++++++++--------------
>  hw/xen.h      |   10 ++++++++++
>  xen-all.c     |    8 +++++++-
>  4 files changed, 48 insertions(+), 16 deletions(-)
> 
> diff --git a/hw/ide/qdev.c b/hw/ide/qdev.c
> index 5ea9b8f..473acd7 100644
> --- a/hw/ide/qdev.c
> +++ b/hw/ide/qdev.c
> @@ -115,7 +115,13 @@ IDEDevice *ide_create_drive(IDEBus *bus, int unit,
DriveInfo *drive)
>  int ide_get_geometry(BusState *bus, int unit,
>                       int16_t *cyls, int8_t *heads, int8_t *secs)
>  {
> -    IDEState *s = &DO_UPCAST(IDEBus, qbus, bus)->ifs[unit];
> +    IDEState *s = NULL;
> +
> +    if (!bus) {
> +        return -1;
> +    }
> +
> +    s = &DO_UPCAST(IDEBus, qbus, bus)->ifs[unit];
>  
>      if (s->drive_kind != IDE_HD || !s->bs) {
>          return -1;
> diff --git a/hw/pc_piix.c b/hw/pc_piix.c
> index 6cb0a2a..b904100 100644
> --- a/hw/pc_piix.c
> +++ b/hw/pc_piix.c
> @@ -148,6 +148,7 @@ static void pc_init1(MemoryRegion *system_memory,
>      MemoryRegion *pci_memory;
>      MemoryRegion *rom_memory;
>      void *fw_cfg = NULL;
> +    int register_default_dev = 0;
>  
>      pc_cpus_init(cpu_model);
>  
> @@ -242,23 +243,32 @@ static void pc_init1(MemoryRegion *system_memory,
>              pci_nic_init_nofail(nd, "e1000", NULL);
>      }
>  
> -    ide_drive_get(hd, MAX_IDE_BUS);
> -    if (pci_enabled) {
> -        PCIDevice *dev;
> -        if (xen_enabled()) {
> -            dev = pci_piix3_xen_ide_init(pci_bus, hd, piix3_devfn + 1);
> +    if (!xen_enabled() || xen_is_emulated_ide()) {
> +        xen_set_register_default_dev(0, &register_default_dev);
> +        ide_drive_get(hd, MAX_IDE_BUS);
> +        if (pci_enabled) {
> +            PCIDevice *dev;
> +            if (xen_enabled()) {
> +                dev = pci_piix3_xen_ide_init(pci_bus, hd, piix3_devfn +
1);
> +            } else {
> +                dev = pci_piix3_ide_init(pci_bus, hd, piix3_devfn + 1);
> +            }
> +            idebus[0] = qdev_get_child_bus(&dev->qdev,
"ide.0");
> +            idebus[1] = qdev_get_child_bus(&dev->qdev,
"ide.1");
>          } else {
> -            dev = pci_piix3_ide_init(pci_bus, hd, piix3_devfn + 1);
> +            for (i = 0; i < MAX_IDE_BUS; i++) {
> +                ISADevice *dev;
> +                dev = isa_ide_init(isa_bus, ide_iobase[i], ide_iobase2[i],
> +                                   ide_irq[i],
> +                                   hd[MAX_IDE_DEVS * i],
> +                                   hd[MAX_IDE_DEVS * i + 1]);
> +                idebus[i] = qdev_get_child_bus(&dev->qdev,
"ide.0");
> +            }
>          }
> -        idebus[0] = qdev_get_child_bus(&dev->qdev,
"ide.0");
> -        idebus[1] = qdev_get_child_bus(&dev->qdev,
"ide.1");
> +        xen_set_register_default_dev(register_default_dev, NULL);
>      } else {
> -        for(i = 0; i < MAX_IDE_BUS; i++) {
> -            ISADevice *dev;
> -            dev = isa_ide_init(isa_bus, ide_iobase[i], ide_iobase2[i],
> -                               ide_irq[i],
> -                               hd[MAX_IDE_DEVS * i], hd[MAX_IDE_DEVS * i +
1]);
> -            idebus[i] = qdev_get_child_bus(&dev->qdev,
"ide.0");
> +        for (i = 0; i < MAX_IDE_BUS; i++) {
> +            idebus[i] = NULL;
>          }
>      }
I think it would be better to have a non-xen specific option to
enable/disable ide emulation, something like hpet/no_hpet.

> diff --git a/hw/xen.h b/hw/xen.h
> index 3c8724f..3c89fb9 100644
> --- a/hw/xen.h
> +++ b/hw/xen.h
> @@ -22,6 +22,7 @@ extern enum xen_mode xen_mode;
>  
>  extern int xen_allowed;
>  extern int xen_register_default_dev;
> +extern int xen_emulate_ide;
>  
>  static inline int xen_enabled(void)
>  {
> @@ -32,6 +33,15 @@ static inline int xen_enabled(void)
>  #endif
>  }
>  
> +static inline int xen_is_emulated_ide(void)
> +{
> +#if defined(CONFIG_XEN)
> +    return xen_emulate_ide;
> +#else
> +    return 1;
> +#endif
> +}
> +
>  static inline int xen_is_registered_default_dev(void)
>  {
>  #if defined(CONFIG_XEN)
> diff --git a/xen-all.c b/xen-all.c
> index f424cce..f091908 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -43,6 +43,8 @@ static uint32_t xen_dmid = ~0;
>  int xen_register_default_dev = 0;
>  static int xen_emulate_default_dev = 1;
>  
> +int xen_emulate_ide = 0;
> +
>  /* Compatibility with older version */
>  #if __XEN_LATEST_INTERFACE_VERSION__ < 0x0003020a
>  static inline uint32_t xen_vcpu_eport(shared_iopage_t *shared_page, int i)
> @@ -1342,6 +1344,8 @@ int xen_hvm_init(void)
>                                         "xen_dmid", ~0);
>          xen_emulate_default_dev =
qemu_opt_get_bool(QTAILQ_FIRST(&list->head),
>                                                     
"xen_default_dev", 1);
> +        xen_emulate_ide =
qemu_opt_get_bool(QTAILQ_FIRST(&list->head),
> +                                            "xen_emulate_ide",
1);
>      }
>  
>      state = g_malloc0(sizeof (XenIOState));
> @@ -1437,12 +1441,14 @@ int xen_hvm_init(void)
>          fprintf(stderr, "%s: xen backend core setup failed\n",
__FUNCTION__);
>          exit(1);
>      }
> -    xen_be_register("qdisk", &xen_blkdev_ops);
>  
>      if (xen_emulate_default_dev) {
>          xen_be_register("console", &xen_console_ops);
>          xen_be_register("vkbd", &xen_kbdmouse_ops);
>      }
> +    if (xen_emulate_ide) {
> +        xen_be_register("qdisk", &xen_blkdev_ops);
> +    }
>      xen_read_physmap(state);
>  
>      return 0;
> -- 
> Julien Grall
>

Julien Grall

2012-Aug-23 17:18 UTC

head link

Re: [QEMU][RFC V2 05/10] xen-memory: register memory/IO range in Xen

On 08/23/2012 03:41 PM, Stefano Stabellini wrote:> On Wed, 22 Aug 2012, Julien Grall wrote:
>    
>> Add Memory listener on IO and modify the one on memory.
>> Becareful, the first listener is not called is the range is still
register with
>> register_ioport*. So Xen will never know that this QEMU is handle the
range.
>>      
> I don''t understand what you mean here. Could you please elaborate?
>    
I made a patch series to remove all ioport_register_* function in
differents devices (dma, cirrus, ...):
https://lists.gnu.org/archive/html/qemu-devel/2012-08/msg04007.html

ioport_register_* don'' t use the new memory API, so listener is not
called when a new range is registered.

I will rework the commit message and send it back.

-- 
Julien

Julien Grall

2012-Aug-24 12:59 UTC

head link

Re: [QEMU][RFC V2 06/10] xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG

On 08/23/2012 03:41 PM, Stefano Stabellini wrote:> On Wed, 22 Aug 2012, Julien Grall wrote:
>    
>> With QEMU disaggregation QEMU needs to specify which PCI device
it''s able to
>> handle. It will use the device place in the topology (domain, bus,
device,
>> function).
>> When Xen will trap an access for the config space, it will forge a new
>> ioreq and forward it to the right QEMU.
>>
>> Signed-off-by: Julien Grall<julien.grall@citrix.com>
>> ---
>>   hw/pci.c   |    6 ++++++
>>   hw/xen.h   |    1 +
>>   xen-all.c  |   38 ++++++++++++++++++++++++++++++++++++++
>>   xen-stub.c |    5 +++++
>>   4 files changed, 50 insertions(+), 0 deletions(-)
>>
>> diff --git a/hw/pci.c b/hw/pci.c
>> index 4d95984..0112edf 100644
>> --- a/hw/pci.c
>> +++ b/hw/pci.c
>> @@ -33,6 +33,7 @@
>>   #include "qmp-commands.h"
>>   #include "msi.h"
>>   #include "msix.h"
>> +#include "xen.h"
>>
>>   //#define DEBUG_PCI
>>   #ifdef DEBUG_PCI
>> @@ -781,6 +782,11 @@ static PCIDevice *do_pci_register_device(PCIDevice
*pci_dev, PCIBus *bus,
>>       pci_dev->devfn = devfn;
>>       pstrcpy(pci_dev->name, sizeof(pci_dev->name), name);
>>       pci_dev->irq_state = 0;
>> +
>> +    if (xen_enabled()&&  xen_register_pcidev(pci_dev)) {
>> +        return NULL;
>>      
> Is this an error condition? If so we should print an error message,
> right?
>    
Yes, it means that the BDF is already registered by another QEMU.

Xen devel - Aug 2012 - [QEMU][RFC V2 00/10] QEMU disaggregation in Xen environment.

[QEMU][RFC V2 00/10] QEMU disaggregation in Xen environment.

[QEMU][RFC V2 01/10] xen: add new machine options to support QEMU disaggregation in Xen environment

[QEMU][RFC V2 02/10] xen: modify QEMU status path in XenStore

[QEMU][RFC V2 03/10] xen: add wrappers for new Xen disaggregation hypercalls

[QEMU][RFC V2 04/10] xen-hvm: register qemu as ioreq server and retrieve shared pages

[QEMU][RFC V2 05/10] xen-memory: register memory/IO range in Xen

[QEMU][RFC V2 06/10] xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG

[QEMU][RFC V2 07/10] xen: specify which device is part of default devices

[QEMU][RFC V2 08/10] xen: audio is not a part of default devices

[QEMU][RFC V2 09/10] xen-memory: handle node "device_model" for physical mapping

[QEMU][RFC V2 10/10] xen: emulate IDE outside default device set

Re: [QEMU][RFC V2 01/10] xen: add new machine options to support QEMU disaggregation in Xen environment

Re: [QEMU][RFC V2 05/10] xen-memory: register memory/IO range in Xen

Re: [QEMU][RFC V2 06/10] xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG

Re: [QEMU][RFC V2 07/10] xen: specify which device is part of default devices

Re: [QEMU][RFC V2 08/10] xen: audio is not a part of default devices

Re: [QEMU][RFC V2 10/10] xen: emulate IDE outside default device set

Re: [QEMU][RFC V2 05/10] xen-memory: register memory/IO range in Xen

Re: [QEMU][RFC V2 06/10] xen-pci: register PCI device in Xen and handle IOREQ_TYPE_PCI_CONFIG