Hi- I am trying to pass an LSI SAS2008-based HBA (IBM M1015) through to an HVM Solaris VM, using Xen 4.2 unstable and the qemu-traditional device model. On boot I see the following error: MPT BIOS Fault 09h encountered at adapter PCI(00h,05h,00h) A list search yielded (http://comments.gmane.org/gmane.comp.emulators.xen.devel/128172), however there was no solution for an HVM VM. I''ve attached the log file for booting. The expansion/option ROM gets installed at 0xf7a00000 and is first accessed and mapped with the line: pt_iomem_map: e_phys=f3000001 maddr=f7a00000 type=8 len=524288 index=6 first_map=1 However the following log line seems to immediately map the same address space to a strange e_phys location: pt_iomem_map: e_phys=ffffffff maddr=f7a00000 type=8 len=524288 index=6 first_map=0 Any help or suggestions would be appreciated. I have tested the same card using PCI pass-through on ESXi and it functioned properly there. Thanks! David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, 2012-07-18 at 10:47 +0100, David Erickson wrote:> Hi- > I am trying to pass an LSI SAS2008-based HBA (IBM M1015) through to an > HVM Solaris VM, using Xen 4.2 unstable and the qemu-traditional device > model. On boot I see the following error: > > MPT BIOS Fault 09h encountered at adapter PCI(00h,05h,00h) > > A list search yielded > (http://comments.gmane.org/gmane.comp.emulators.xen.devel/128172), > however there was no solution for an HVM VM. I''ve attached the log > file for booting. The expansion/option ROM gets installed at > 0xf7a00000 and is first accessed and mapped with the line: > > pt_iomem_map: e_phys=f3000001 maddr=f7a00000 type=8 len=524288 index=6 > first_map=1 > > However the following log line seems to immediately map the same > address space to a strange e_phys location: > > pt_iomem_map: e_phys=ffffffff maddr=f7a00000 type=8 len=524288 index=6 > first_map=0 > > Any help or suggestions would be appreciated.SeaBIOS (used by qemu-xen) should be a lot better than ROMBIOS (used with qemu-xen-traditional) at supporting Option ROMS, so it would be worth trying that. Ian.
On Wed, Jul 18, 2012 at 2:59 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Wed, 2012-07-18 at 10:47 +0100, David Erickson wrote: >> Hi- >> I am trying to pass an LSI SAS2008-based HBA (IBM M1015) through to an >> HVM Solaris VM, using Xen 4.2 unstable and the qemu-traditional device >> model. On boot I see the following error: >> >> MPT BIOS Fault 09h encountered at adapter PCI(00h,05h,00h) >> >> A list search yielded >> (http://comments.gmane.org/gmane.comp.emulators.xen.devel/128172), >> however there was no solution for an HVM VM. I''ve attached the log >> file for booting. The expansion/option ROM gets installed at >> 0xf7a00000 and is first accessed and mapped with the line: >> >> pt_iomem_map: e_phys=f3000001 maddr=f7a00000 type=8 len=524288 index=6 >> first_map=1 >> >> However the following log line seems to immediately map the same >> address space to a strange e_phys location: >> >> pt_iomem_map: e_phys=ffffffff maddr=f7a00000 type=8 len=524288 index=6 >> first_map=0 >> >> Any help or suggestions would be appreciated. > > SeaBIOS (used by qemu-xen) should be a lot better than ROMBIOS (used > with qemu-xen-traditional) at supporting Option ROMS, so it would be > worth trying that.Ya I gave this a shot and it seems like with this device model and bios it doesn''t really pass the card through at all, in that there is no message about the expansion/option rom at boot, and once I''m into Solaris and run scanpci, nothing shows up at all (it is at least enumerable under traditional). There isn''t a lot of logging to indicate a problem either, here is the qemu-dm-solaris.log: xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22 = Invalid argument): Internal error xen be: qdisk-768: xc_gnttab_set_max_grants failed: Invalid argument xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22 = Invalid argument): Internal error xen be: qdisk-5632: xc_gnttab_set_max_grants failed: Invalid argument xen be: qdisk-768: error: unknown operation (3) xen-hotplug.log prints a couple of the following lines, but it also prints them for traditional: RTNETLINK answers: Operation not supported xl pci-list thinks it has attached it: derickso@xen:/var/log/xen$ sudo xl pci-list solaris Vdev Device 00.0 0000:02:00.0 I''ve also attached the xl dmesg from both a rombios and seabios boot if that is helpful. Thanks, David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Jul 18, 2012 at 10:55 AM, David Erickson <halcyon1981@gmail.com> wrote:> On Wed, Jul 18, 2012 at 2:59 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> On Wed, 2012-07-18 at 10:47 +0100, David Erickson wrote: >>> Hi- >>> I am trying to pass an LSI SAS2008-based HBA (IBM M1015) through to an >>> HVM Solaris VM, using Xen 4.2 unstable and the qemu-traditional device >>> model. On boot I see the following error: >>> >>> MPT BIOS Fault 09h encountered at adapter PCI(00h,05h,00h) >>> >>> A list search yielded >>> (http://comments.gmane.org/gmane.comp.emulators.xen.devel/128172), >>> however there was no solution for an HVM VM. I''ve attached the log >>> file for booting. The expansion/option ROM gets installed at >>> 0xf7a00000 and is first accessed and mapped with the line: >>> >>> pt_iomem_map: e_phys=f3000001 maddr=f7a00000 type=8 len=524288 index=6 >>> first_map=1 >>> >>> However the following log line seems to immediately map the same >>> address space to a strange e_phys location: >>> >>> pt_iomem_map: e_phys=ffffffff maddr=f7a00000 type=8 len=524288 index=6 >>> first_map=0 >>> >>> Any help or suggestions would be appreciated. >> >> SeaBIOS (used by qemu-xen) should be a lot better than ROMBIOS (used >> with qemu-xen-traditional) at supporting Option ROMS, so it would be >> worth trying that. > > Ya I gave this a shot and it seems like with this device model and > bios it doesn''t really pass the card through at all, in that there is > no message about the expansion/option rom at boot, and once I''m into > Solaris and run scanpci, nothing shows up at all (it is at least > enumerable under traditional). There isn''t a lot of logging to > indicate a problem either, here is the qemu-dm-solaris.log: > > xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed > (22 = Invalid argument): Internal error > xen be: qdisk-768: xc_gnttab_set_max_grants failed: Invalid argument > xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed > (22 = Invalid argument): Internal error > xen be: qdisk-5632: xc_gnttab_set_max_grants failed: Invalid argument > xen be: qdisk-768: error: unknown operation (3) > > xen-hotplug.log prints a couple of the following lines, but it also > prints them for traditional: > RTNETLINK answers: Operation not supported > > xl pci-list thinks it has attached it: > derickso@xen:/var/log/xen$ sudo xl pci-list solaris > Vdev Device > 00.0 0000:02:00.0 > > I''ve also attached the xl dmesg from both a rombios and seabios boot > if that is helpful.I should also mention I''ve seen the following messages that probably aren''t good when using the xen-qemu device model and pass-thru. On startup: libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an error message from QMP server: Parameter ''driver'' expects a driver name On destroy: libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an error message from QMP server: Device ''pci-pt-02_00.0'' not found -David
> I should also mention I''ve seen the following messages that probably > aren''t good when using the xen-qemu device model and pass-thru. On > startup: > > libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an > error message from QMP server: Parameter ''driver'' expects a driver > nameI did a tiny bit more debugging and that driver it is looking for is named "xen-pci-passthrough", and it is failing to locate it in the tools/qemu-xen-dir/hw/qdev.c qdev_find_info method.
David, I can confirm that I have seen the same error messages on using pass through with the qemu-xen device model. From: David Erickson <halcyon1981@gmail.com> To: Ian Campbell <Ian.Campbell@citrix.com> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> Sent: Wednesday, July 18, 2012 12:14 PM Subject: Re: [Xen-devel] LSI SAS2008 Option Rom Failure> I should also mention I''ve seen the following messages that probably > aren''t good when using the xen-qemu device model and pass-thru. On > startup: > > libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an > error message from QMP server: Parameter ''driver'' expects a driver > nameI did a tiny bit more debugging and that driver it is looking for is named "xen-pci-passthrough", and it is failing to locate it in the tools/qemu-xen-dir/hw/qdev.c qdev_find_info method. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
I should also mention I''ve seen the following messages that probably aren''t good when using the xen-qemu device model and pass-thru. On startup: libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an error message from QMP server: Parameter ''driver'' expects a driver name I solved this by using the latest version of qemu-devel git: http://wiki.xen.org/wiki/QEMU_Upstream _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Jul 18, 2012 at 2:02 PM, ivo <shandivo@gmail.com> wrote:> I should also mention I''ve seen the following messages that probably aren''t > good when using the xen-qemu device model and pass-thru. On startup: > > libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an error > message from QMP server: Parameter ''driver'' expects a driver name > > I solved this by using the latest version of qemu-devel git: > > > http://wiki.xen.org/wiki/QEMU_UpstreamHi Ivo- Can you mention how you built it? I checked out xen-unstable, tried modifying the Config.mk to change the QEMU_UPSTREAM_URL variable, however when I built with this new url it complained when it tried to checkout the qemu traditional because the revision doesn''t exist at that url. So instead I built normally, then went into tools/qemu-xen-dir and added the git remote for qemu unstable, checked out the master branch, ran ./configure, then backed up to the main xen-unstable.hg folder and did sudo make dist and sudo make install. When I then started the VM with the config containing device model qemu-xen nothing changed. I then added the line: device_model_override="/home/derickso/xen-unstable.hg/tools/qemu-xen-dir/i386-softmmu/qemu-system-i386" And started the VM, but got a bunch of errors: derickso@xen:~/xen-unstable.hg/tools/qemu-xen-dir/i386-softmmu$ sudo xl -v create /etc/xen/ubuntu.conf -V Parsing config from /etc/xen/ubuntu.conf xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x9dcc8 xc: detail: elf_parse_binary: memory: 0x100000 -> 0x19dcc8 xc: info: VIRTUAL MEMORY ARRANGEMENT: Loader: 0000000000100000->000000000019dcc8 TOTAL: 0000000000000000->000000007f800000 ENTRY ADDRESS: 0000000000100000 xc: info: PHYSICAL MEMORY ALLOCATION: 4KB PAGES: 0x0000000000000200 2MB PAGES: 0x00000000000003fb 1GB PAGES: 0x0000000000000000 xc: detail: elf_load_binary: phdr 0 at 0x0x7f77f2409000 -> 0x0x7f77f249db4d libxl: error: libxl_dm.c:1096:device_model_spawn_outcome: domain 2 device model: spawn failed (rc=-3) libxl: error: libxl_qmp.c:641:libxl__qmp_initialize: Connection error: No such file or directory libxl: error: libxl_exec.c:227:libxl__wait_for_offspring: Device Model not ready vncviewer: ConnectToTcpAddr: connect: Connection refused Unable to connect to VNC server Any suggestions? Thanks, David
On Wed, Jul 18, 2012 at 4:52 PM, David Erickson <halcyon1981@gmail.com> wrote:> On Wed, Jul 18, 2012 at 2:02 PM, ivo <shandivo@gmail.com> wrote: >> I should also mention I''ve seen the following messages that probably aren''t >> good when using the xen-qemu device model and pass-thru. On startup: >> >> libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an error >> message from QMP server: Parameter ''driver'' expects a driver name >> >> I solved this by using the latest version of qemu-devel git: >> >> >> http://wiki.xen.org/wiki/QEMU_Upstream > > Hi Ivo- > Can you mention how you built it? I checked out xen-unstable, tried > modifying the Config.mk to change the QEMU_UPSTREAM_URL variable, > however when I built with this new url it complained when it tried to > checkout the qemu traditional because the revision doesn''t exist at > that url. So instead I built normally, then went into > tools/qemu-xen-dir and added the git remote for qemu unstable, checked > out the master branch, ran ./configure, then backed up to the main > xen-unstable.hg folder and did sudo make dist and sudo make install. > When I then started the VM with the config containing device model > qemu-xen nothing changed. I then added the line: > > device_model_override="/home/derickso/xen-unstable.hg/tools/qemu-xen-dir/i386-softmmu/qemu-system-i386" > > And started the VM, but got a bunch of errors: > > derickso@xen:~/xen-unstable.hg/tools/qemu-xen-dir/i386-softmmu$ sudo > xl -v create /etc/xen/ubuntu.conf -V > Parsing config from /etc/xen/ubuntu.conf > xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x9dcc8 > xc: detail: elf_parse_binary: memory: 0x100000 -> 0x19dcc8 > xc: info: VIRTUAL MEMORY ARRANGEMENT: > Loader: 0000000000100000->000000000019dcc8 > TOTAL: 0000000000000000->000000007f800000 > ENTRY ADDRESS: 0000000000100000 > xc: info: PHYSICAL MEMORY ALLOCATION: > 4KB PAGES: 0x0000000000000200 > 2MB PAGES: 0x00000000000003fb > 1GB PAGES: 0x0000000000000000 > xc: detail: elf_load_binary: phdr 0 at 0x0x7f77f2409000 -> 0x0x7f77f249db4d > libxl: error: libxl_dm.c:1096:device_model_spawn_outcome: domain 2 > device model: spawn failed (rc=-3) > libxl: error: libxl_qmp.c:641:libxl__qmp_initialize: Connection error: > No such file or directory > libxl: error: libxl_exec.c:227:libxl__wait_for_offspring: Device Model not ready > vncviewer: ConnectToTcpAddr: connect: Connection refused > Unable to connect to VNC server > > Any suggestions?I also alternatively tried putting the following into xen-unstable.hg/.config: QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git QEMU_UPSTREAM_REVISION = master Then executed make world, which doesn''t have the above problems of checkout, but during the build has the following error in the qemu-xen-dir-remote: ERROR ERROR: User requested feature xen ERROR: configure was not able to find it ERROR Thanks, David
On Thu, Jul 19, 2012 at 1:52 AM, David Erickson <halcyon1981@gmail.com>wrote:> Hi Ivo- > Can you mention how you built it? I checked out xen-unstable, tried > modifying the Config.mk to change the QEMU_UPSTREAM_URL variable, > however when I built with this new url it complained when it tried to > checkout the qemu traditional because the revision doesn''t exist at > that url. So instead I built normally, then went into > tools/qemu-xen-dir and added the git remote for qemu unstable, checked > out the master branch, ran ./configure, then backed up to the main > xen-unstable.hg folder and did sudo make dist and sudo make install. > When I then started the VM with the config containing device model > qemu-xen nothing changed. I then added the line: > > > device_model_override="/home/derickso/xen-unstable.hg/tools/qemu-xen-dir/i386-softmmu/qemu-system-i386" > > And started the VM, but got a bunch of errors: > > derickso@xen:~/xen-unstable.hg/tools/qemu-xen-dir/i386-softmmu$ sudo > xl -v create /etc/xen/ubuntu.conf -V > Parsing config from /etc/xen/ubuntu.conf > xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x9dcc8 > xc: detail: elf_parse_binary: memory: 0x100000 -> 0x19dcc8 > xc: info: VIRTUAL MEMORY ARRANGEMENT: > Loader: 0000000000100000->000000000019dcc8 > TOTAL: 0000000000000000->000000007f800000 > ENTRY ADDRESS: 0000000000100000 > xc: info: PHYSICAL MEMORY ALLOCATION: > 4KB PAGES: 0x0000000000000200 > 2MB PAGES: 0x00000000000003fb > 1GB PAGES: 0x0000000000000000 > xc: detail: elf_load_binary: phdr 0 at 0x0x7f77f2409000 -> 0x0x7f77f249db4d > libxl: error: libxl_dm.c:1096:device_model_spawn_outcome: domain 2 > device model: spawn failed (rc=-3) > libxl: error: libxl_qmp.c:641:libxl__qmp_initialize: Connection error: > No such file or directory > libxl: error: libxl_exec.c:227:libxl__wait_for_offspring: Device Model not > ready > vncviewer: ConnectToTcpAddr: connect: Connection refused > Unable to connect to VNC server > > Any suggestions? > > Thanks, > David >I have just followed the wiki, using xen-unstable rev. 25567:165eb54e57c0 Create a file in the root of xen-unstable named .config and add this: QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git QEMU_UPSTREAM_REVISION = master Next, just ./configure and make as always. Check the output messages (tools directory) to be sure qemu git is used. Later add this to your xl config file: device_model_version = ''qemu-xen'' It worked flawlessy in my Ubuntu-12.04. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, Jul 19, 2012 at 2:18 AM, David Erickson <halcyon1981@gmail.com>wrote:> > I also alternatively tried putting the following into > xen-unstable.hg/.config: > QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git > QEMU_UPSTREAM_REVISION = master > > Then executed make world, which doesn''t have the above problems of > checkout, but during the build has the following error in the > qemu-xen-dir-remote: > > ERROR > ERROR: User requested feature xen > ERROR: configure was not able to find it > ERROR > > Thanks, > David >I never had problems with qemu-upstream. Try with a clean xen-unstable dir. (re-download it) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Jul 18, 2012 at 5:24 PM, ivo <shandivo@gmail.com> wrote:> > > On Thu, Jul 19, 2012 at 2:18 AM, David Erickson <halcyon1981@gmail.com> > wrote: >> >> I also alternatively tried putting the following into >> xen-unstable.hg/.config: >> QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git >> QEMU_UPSTREAM_REVISION = master >> >> Then executed make world, which doesn''t have the above problems of >> checkout, but during the build has the following error in the >> qemu-xen-dir-remote: >> >> ERROR >> ERROR: User requested feature xen >> ERROR: configure was not able to find it >> ERROR >> >> Thanks, >> David > > > I never had problems with qemu-upstream. Try with a clean xen-unstable dir. > (re-download it)Weird, I also am on Ubuntu 12.04, I did a clean checkout of that revision: hg clone -r 25567 http://xenbits.xen.org/hg/xen-unstable.hg cp .config xen-unstable.hg cd xen-unstable.hg cat .config: PYTHON_PREFIX_ARGQEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git QEMU_UPSTREAM_REVISION = master ./configure make world then it cruises along for awhile and errors at: make[3]: Entering directory `/home/derickso/xen-unstable.hg/tools'' if test -d git://git.qemu.org/qemu.git ; then \ mkdir -p qemu-xen-dir; \ else \ export GIT=git; \ /home/derickso/xen-unstable.hg/tools/../scripts/git-checkout.sh git://git.qemu.org/qemu.git master qemu-xen-dir ; \ fi if test -d git://git.qemu.org/qemu.git ; then \ source=git://git.qemu.org/qemu.git; \ else \ source=.; \ fi; \ cd qemu-xen-dir; \ $source/configure --enable-xen --target-list=i386-softmmu \ --source-path=$source \ --extra-cflags="-I/home/derickso/xen-unstable.hg/tools/../tools/include \ -I/home/derickso/xen-unstable.hg/tools/../tools/libxc \ -I/home/derickso/xen-unstable.hg/tools/../tools/xenstore \ -I/home/derickso/xen-unstable.hg/tools/../tools/xenstore/compat \ " \ --extra-ldflags="-L/home/derickso/xen-unstable.hg/tools/../tools/libxc \ -L/home/derickso/xen-unstable.hg/tools/../tools/xenstore" \ --bindir=/usr/lib/xen/bin \ --datadir=/usr/share/qemu-xen \ --disable-kvm \ --python=python \ ; \ make all ERROR ERROR: User requested feature xen ERROR: configure was not able to find it ERROR make[4]: Entering directory `/home/derickso/xen-unstable.hg/tools/qemu-xen-dir-remote'' Please call configure before running make! make[4]: *** [config-host.mak] Error 1 make[4]: Leaving directory `/home/derickso/xen-unstable.hg/tools/qemu-xen-dir-remote'' make[3]: *** [subdir-all-qemu-xen-dir] Error 2 make[3]: Leaving directory `/home/derickso/xen-unstable.hg/tools'' make[2]: *** [subdirs-install] Error 2 make[2]: Leaving directory `/home/derickso/xen-unstable.hg/tools'' make[1]: *** [install-tools] Error 2 make[1]: Leaving directory `/home/derickso/xen-unstable.hg'' make: *** [world] Error 2
On Wed, Jul 18, 2012 at 5:56 PM, David Erickson <halcyon1981@gmail.com> wrote:> On Wed, Jul 18, 2012 at 5:24 PM, ivo <shandivo@gmail.com> wrote: >> >> >> On Thu, Jul 19, 2012 at 2:18 AM, David Erickson <halcyon1981@gmail.com> >> wrote: >>> >>> I also alternatively tried putting the following into >>> xen-unstable.hg/.config: >>> QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git >>> QEMU_UPSTREAM_REVISION = master >>> >>> Then executed make world, which doesn''t have the above problems of >>> checkout, but during the build has the following error in the >>> qemu-xen-dir-remote: >>> >>> ERROR >>> ERROR: User requested feature xen >>> ERROR: configure was not able to find it >>> ERROR >>> >>> Thanks, >>> David >> >> >> I never had problems with qemu-upstream. Try with a clean xen-unstable dir. >> (re-download it) > > Weird, I also am on Ubuntu 12.04, I did a clean checkout of that revision: > > > hg clone -r 25567 http://xenbits.xen.org/hg/xen-unstable.hg > cp .config xen-unstable.hg > cd xen-unstable.hg > cat .config: > PYTHON_PREFIX_ARG> QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git > QEMU_UPSTREAM_REVISION = master > ./configure > make world > > then it cruises along for awhile and errors at: > make[3]: Entering directory `/home/derickso/xen-unstable.hg/tools'' > if test -d git://git.qemu.org/qemu.git ; then \ > mkdir -p qemu-xen-dir; \ > else \ > export GIT=git; \ > /home/derickso/xen-unstable.hg/tools/../scripts/git-checkout.sh > git://git.qemu.org/qemu.git master qemu-xen-dir ; \ > fi > if test -d git://git.qemu.org/qemu.git ; then \ > source=git://git.qemu.org/qemu.git; \ > else \ > source=.; \ > fi; \ > cd qemu-xen-dir; \ > $source/configure --enable-xen --target-list=i386-softmmu \ > --source-path=$source \ > --extra-cflags="-I/home/derickso/xen-unstable.hg/tools/../tools/include \ > -I/home/derickso/xen-unstable.hg/tools/../tools/libxc \ > -I/home/derickso/xen-unstable.hg/tools/../tools/xenstore \ > -I/home/derickso/xen-unstable.hg/tools/../tools/xenstore/compat \ > " \ > --extra-ldflags="-L/home/derickso/xen-unstable.hg/tools/../tools/libxc \ > -L/home/derickso/xen-unstable.hg/tools/../tools/xenstore" \ > --bindir=/usr/lib/xen/bin \ > --datadir=/usr/share/qemu-xen \ > --disable-kvm \ > --python=python \ > ; \ > make all > ERROR > ERROR: User requested feature xen > ERROR: configure was not able to find it > ERROR > make[4]: Entering directory > `/home/derickso/xen-unstable.hg/tools/qemu-xen-dir-remote'' > Please call configure before running make! > make[4]: *** [config-host.mak] Error 1 > make[4]: Leaving directory > `/home/derickso/xen-unstable.hg/tools/qemu-xen-dir-remote'' > make[3]: *** [subdir-all-qemu-xen-dir] Error 2 > make[3]: Leaving directory `/home/derickso/xen-unstable.hg/tools'' > make[2]: *** [subdirs-install] Error 2 > make[2]: Leaving directory `/home/derickso/xen-unstable.hg/tools'' > make[1]: *** [install-tools] Error 2 > make[1]: Leaving directory `/home/derickso/xen-unstable.hg'' > make: *** [world] Error 2Have you tried a clean build lately? I wonder if something in qemu''s upstream has changed since you built that is breaking it.
Ok solved the compilation problem, the issue is the upstream configure
script is testing for xen by compiling code that includes xs.h,
however xs.h now throws a deprecated warning, and the configure script
compiles with the flag that makes any warning an error. The solution
is to replace all instances of xs.h in
xen-unstable.hg/tools/qemu-xen-dir/configure with xenstore.h after the
make has failed, then run it again and it will work.
This also resolved the option/expansion ROM issue I was seeing, but
unfortunately simultaneously caused the card to no longer function
within the VM. The error I''m seeing now from dom0 is:
cat /var/log/xen/qemu-dm-ubuntu.log
xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed
(22 = Invalid argument): Internal error
xen be: qdisk-5632: xc_gnttab_set_max_grants failed: Invalid argument
[00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X
is already enabled.
[00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X
is already enabled.
[00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X
is already enabled.
[00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X
is already enabled.
I''ve attached the output of xl dmesg and the screenshot of the dmesg
error within my test Ubuntu domu (I don''t know how to debug Solaris as
well so I figure Ubuntu is a good place to verify pass through
functionality first). My domU VM conf file:
builder=''hvm''
device_model_version="qemu-xen"
bios="seabios"
memory = 2048
vcpus=2
name = "ubuntu"
vif = [''bridge=xenbr0, type=ioemu'']
disk = [
''file:/home/derickso/ubuntu-11.10-desktop-amd64.iso,ioemu:hdc:cdrom,r''
]
boot="d"
vnc=1
vnclisten="0.0.0.0"
vncdisplay=1
#pci=[''02:00.0'']
pci=[''02:00.0,msitranslate=1,power_mgmt=1,permissive=1'']
xen_platform_pci=1
I tried both ways for configuring the PCI pass through, and
disabling/enabling xen_platform_pci, didn''t make a difference.
Suggestions welcome!
Thanks,
David
On Wed, Jul 18, 2012 at 6:32 PM, jacek burghardt
<jaceksburghardt@gmail.com> wrote:> There is solution
>
http://xen.1045712.n5.nabble.com/Upstream-Qemu-With-Xen-configuration-problem-td4561779.html
> I wish somone create patch so latest qemu can be compiled with xen
unstable.
> The qemu has script that does all the work compiling it but the latest
> version is missing it.
> On Wed, Jul 18, 2012 at 7:02 PM, David Erickson
<halcyon1981@gmail.com>
> wrote:
>>
>> On Wed, Jul 18, 2012 at 5:56 PM, David Erickson
<halcyon1981@gmail.com>
>> wrote:
>> > On Wed, Jul 18, 2012 at 5:24 PM, ivo <shandivo@gmail.com>
wrote:
>> >>
>> >>
>> >> On Thu, Jul 19, 2012 at 2:18 AM, David Erickson
<halcyon1981@gmail.com>
>> >> wrote:
>> >>>
>> >>> I also alternatively tried putting the following into
>> >>> xen-unstable.hg/.config:
>> >>> QEMU_UPSTREAM_URL = git://git.qemu.org/qemu.git
>> >>> QEMU_UPSTREAM_REVISION = master
>> >>>
>> >>> Then executed make world, which doesn''t have the
above problems of
>> >>> checkout, but during the build has the following error in
the
>> >>> qemu-xen-dir-remote:
>> >>>
>> >>> ERROR
>> >>> ERROR: User requested feature xen
>> >>> ERROR: configure was not able to find it
>> >>> ERROR
>> >>>
>> >>> Thanks,
>> >>> David
>> >>
>> >>
>> >> I never had problems with qemu-upstream. Try with a clean
xen-unstable
>> >> dir.
>> >> (re-download it)
>> >
>> > Weird, I also am on Ubuntu 12.04, I did a clean checkout of that
>> > revision:
>> >
>> >
>> > hg clone -r 25567 http://xenbits.xen.org/hg/xen-unstable.hg
>> > cp .config xen-unstable.hg
>> > cd xen-unstable.hg
>> > cat .config:
>> > PYTHON_PREFIX_ARG>> > QEMU_UPSTREAM_URL =
git://git.qemu.org/qemu.git
>> > QEMU_UPSTREAM_REVISION = master
>> > ./configure
>> > make world
>> >
>> > then it cruises along for awhile and errors at:
>> > make[3]: Entering directory
`/home/derickso/xen-unstable.hg/tools''
>> > if test -d git://git.qemu.org/qemu.git ; then \
>> > mkdir -p qemu-xen-dir; \
>> > else \
>> > export GIT=git; \
>> >
>> > /home/derickso/xen-unstable.hg/tools/../scripts/git-checkout.sh
>> > git://git.qemu.org/qemu.git master qemu-xen-dir ; \
>> > fi
>> > if test -d git://git.qemu.org/qemu.git ; then \
>> > source=git://git.qemu.org/qemu.git; \
>> > else \
>> > source=.; \
>> > fi; \
>> > cd qemu-xen-dir; \
>> > $source/configure --enable-xen --target-list=i386-softmmu
\
>> > --source-path=$source \
>> >
>> >
--extra-cflags="-I/home/derickso/xen-unstable.hg/tools/../tools/include \
>> >
-I/home/derickso/xen-unstable.hg/tools/../tools/libxc \
>> >
-I/home/derickso/xen-unstable.hg/tools/../tools/xenstore
>> > \
>> >
>> > -I/home/derickso/xen-unstable.hg/tools/../tools/xenstore/compat \
>> > " \
>> >
>> >
--extra-ldflags="-L/home/derickso/xen-unstable.hg/tools/../tools/libxc \
>> >
>> > -L/home/derickso/xen-unstable.hg/tools/../tools/xenstore" \
>> > --bindir=/usr/lib/xen/bin \
>> > --datadir=/usr/share/qemu-xen \
>> > --disable-kvm \
>> > --python=python \
>> > ; \
>> > make all
>> > ERROR
>> > ERROR: User requested feature xen
>> > ERROR: configure was not able to find it
>> > ERROR
>> > make[4]: Entering directory
>> >
`/home/derickso/xen-unstable.hg/tools/qemu-xen-dir-remote''
>> > Please call configure before running make!
>> > make[4]: *** [config-host.mak] Error 1
>> > make[4]: Leaving directory
>> >
`/home/derickso/xen-unstable.hg/tools/qemu-xen-dir-remote''
>> > make[3]: *** [subdir-all-qemu-xen-dir] Error 2
>> > make[3]: Leaving directory
`/home/derickso/xen-unstable.hg/tools''
>> > make[2]: *** [subdirs-install] Error 2
>> > make[2]: Leaving directory
`/home/derickso/xen-unstable.hg/tools''
>> > make[1]: *** [install-tools] Error 2
>> > make[1]: Leaving directory
`/home/derickso/xen-unstable.hg''
>> > make: *** [world] Error 2
>>
>> Have you tried a clean build lately? I wonder if something in
qemu''s
>> upstream has changed since you built that is breaking it.
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
On Thu, 2012-07-19 at 04:56 +0100, David Erickson wrote:> Ok solved the compilation problem, the issue is the upstream configure > script is testing for xen by compiling code that includes xs.h, > however xs.h now throws a deprecated warning, and the configure script > compiles with the flag that makes any warning an error. The solution > is to replace all instances of xs.h in > xen-unstable.hg/tools/qemu-xen-dir/configure with xenstore.h after the > make has failed, then run it again and it will work.I think Anthony sent a patch upstream for this, but I suppose it hasn''t been committed yet -- Anthony is that right?> This also resolved the option/expansion ROM issue I was seeing, but > unfortunately simultaneously caused the card to no longer function > within the VM. The error I''m seeing now from dom0 is: > > cat /var/log/xen/qemu-dm-ubuntu.log > xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed > (22 = Invalid argument): Internal error > xen be: qdisk-5632: xc_gnttab_set_max_grants failed: Invalid argumentAFAIK this one is harmless but could be fixed by updating your dom0 kernel.> [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X > is already enabled. > [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X > is already enabled. > [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X > is already enabled. > [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X > is already enabled.Anthony/Stefano, are MSIs supposed to work with upstream Qemu (the upstream branch)? David also previous tried our branch of upstream qemu and had problems there too -- I thought we''d backported the basic PCI passthrough support but he was seeing libxl: error: libxl_qmp.c:288:qmp_handle_error_response: received an error message from QMP server: Parameter ''driver'' expects a driver name (see upthread for more details) David, it might be worth trying a simpler devie pass through first just to validate the basic functionality is working for you. Usually a NIC or a USB controller is pretty good simple case. If you try a NIC then you could also try PXE booting from it, which will exercise the option ROM support, which should be simpler than the option ROM used by a RAID controller.> > I''ve attached the output of xl dmesg and the screenshot of the dmesg > error within my test Ubuntu domu (I don''t know how to debug Solaris as > well so I figure Ubuntu is a good place to verify pass through > functionality first).Yes, starting with Ubuntu is a good idea. The last line of the dmesg is: (XEN) vmsi.c:108:d32767 Unsupported delivery mode 3 which sounds interesting, might be something Stefano knows about? Ian.
On 19/07/12 08:22, Ian Campbell wrote:> On Thu, 2012-07-19 at 04:56 +0100, David Erickson wrote: >> Ok solved the compilation problem, the issue is the upstream configure >> script is testing for xen by compiling code that includes xs.h, >> however xs.h now throws a deprecated warning, and the configure script >> compiles with the flag that makes any warning an error. The solution >> is to replace all instances of xs.h in >> xen-unstable.hg/tools/qemu-xen-dir/configure with xenstore.h after the >> make has failed, then run it again and it will work. > > I think Anthony sent a patch upstream for this, but I suppose it hasn''t > been committed yet -- Anthony is that right?Actually, no. I saw this issue (with the configure) only this week but haven''t done anything for it, yet. Instead of modify configure, you could try to pass --disable-werror to configure. That actually why I haven''t see the issue before.>> This also resolved the option/expansion ROM issue I was seeing, but >> unfortunately simultaneously caused the card to no longer function >> within the VM. The error I''m seeing now from dom0 is: >> >> cat /var/log/xen/qemu-dm-ubuntu.log >> xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed >> (22 = Invalid argument): Internal error >> xen be: qdisk-5632: xc_gnttab_set_max_grants failed: Invalid argument > > AFAIK this one is harmless but could be fixed by updating your dom0 > kernel. > >> [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X >> is already enabled. >> [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X >> is already enabled. >> [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X >> is already enabled. >> [00:05.0] pci_msix_write: Error: Can''t update msix entry 0 since MSI-X >> is already enabled. > > Anthony/Stefano, are MSIs supposed to work with upstream Qemu (the > upstream branch)?I think there should work, yes.> David also previous tried our branch of upstream qemu and had problems > there too -- I thought we''d backported the basic PCI passthrough support > but he was seeing > libxl: error: libxl_qmp.c:288:qmp_handle_error_response: > received an > error message from QMP server: Parameter ''driver'' expects a > driver > name > (see upthread for more details)This is because the pci passthrough is not compiled in QEMU. I could maybe try to improve this error message. -- Anthony PERARD
On Thu, 2012-07-19 at 11:57 +0100, Anthony PERARD wrote:> On 19/07/12 08:22, Ian Campbell wrote: > > On Thu, 2012-07-19 at 04:56 +0100, David Erickson wrote: > >> Ok solved the compilation problem, the issue is the upstream configure > >> script is testing for xen by compiling code that includes xs.h, > >> however xs.h now throws a deprecated warning, and the configure script > >> compiles with the flag that makes any warning an error. The solution > >> is to replace all instances of xs.h in > >> xen-unstable.hg/tools/qemu-xen-dir/configure with xenstore.h after the > >> make has failed, then run it again and it will work. > > > > I think Anthony sent a patch upstream for this, but I suppose it hasn''t > > been committed yet -- Anthony is that right? > > Actually, no. I saw this issue (with the configure) only this week but > haven''t done anything for it, yet. > > Instead of modify configure, you could try to pass --disable-werror to > configure. That actually why I haven''t see the issue before.OK.> > David also previous tried our branch of upstream qemu and had problems > > there too -- I thought we''d backported the basic PCI passthrough support > > but he was seeing > > libxl: error: libxl_qmp.c:288:qmp_handle_error_response: > > received an > > error message from QMP server: Parameter ''driver'' expects a > > driver > > name > > (see upthread for more details) > > This is because the pci passthrough is not compiled in QEMU. I could > maybe try to improve this error message.Should this be enabled in our build system and/or referenced in the wiki?
On 19/07/12 12:07, Ian Campbell wrote:> On Thu, 2012-07-19 at 11:57 +0100, Anthony PERARD wrote: >> On 19/07/12 08:22, Ian Campbell wrote: >>> David also previous tried our branch of upstream qemu and had problems >>> there too -- I thought we''d backported the basic PCI passthrough support >>> but he was seeing >>> libxl: error: libxl_qmp.c:288:qmp_handle_error_response: >>> received an >>> error message from QMP server: Parameter ''driver'' expects a >>> driver >>> name >>> (see upthread for more details) >> >> This is because the pci passthrough is not compiled in QEMU. I could >> maybe try to improve this error message. > > Should this be enabled in our build system and/or referenced in the > wiki?This should not be needed as it is enable by default if possible. But I don''t know what went wrong here :(. Otherwise, to be sure that pci passthrough is compiled in, use the --enable-xen-pci-passthrough configure option. -- Anthony PERARD
On Thu, 2012-07-19 at 12:31 +0100, Anthony PERARD wrote:> On 19/07/12 12:07, Ian Campbell wrote: > > On Thu, 2012-07-19 at 11:57 +0100, Anthony PERARD wrote: > >> On 19/07/12 08:22, Ian Campbell wrote: > >>> David also previous tried our branch of upstream qemu and had problems > >>> there too -- I thought we''d backported the basic PCI passthrough support > >>> but he was seeing > >>> libxl: error: libxl_qmp.c:288:qmp_handle_error_response: > >>> received an > >>> error message from QMP server: Parameter ''driver'' expects a > >>> driver > >>> name > >>> (see upthread for more details) > >> > >> This is because the pci passthrough is not compiled in QEMU. I could > >> maybe try to improve this error message. > > > > Should this be enabled in our build system and/or referenced in the > > wiki? > > This should not be needed as it is enable by default if possible. But I > don''t know what went wrong here :(.Missing dependency? e.g. libpci-devel?> Otherwise, to be sure that pci passthrough is compiled in, use the > --enable-xen-pci-passthrough configure option.We should enable this option in our build system so that we get a consistent set of features not dependent on the vagaries of the underlying system. If there are libraries which we require we should check for them in out configure script. Ian.>
On Thu, Jul 19, 2012 at 12:38 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> >> This should not be needed as it is enable by default if possible. But I >> don''t know what went wrong here :(. > > Missing dependency? e.g. libpci-devel?The only check are, if configured for linux, if xen is enable and if xen is >= 3.4. We don''t use libpci with QEMU upstream. -- Anthony PERARD
On Thu, 2012-07-19 at 12:44 +0100, Anthony PERARD wrote:> On Thu, Jul 19, 2012 at 12:38 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > > >> This should not be needed as it is enable by default if possible. But I > >> don''t know what went wrong here :(. > > > > Missing dependency? e.g. libpci-devel? > > The only check are, if configured for linux, if xen is enable and if > xen is >= 3.4. We don''t use libpci with QEMU upstream.Hrm, perhaps David can provide some logs. What files from the qemu source tree would give a hint? Is there a config.log?
On Thu, 19 Jul 2012, Ian Campbell wrote:> On Thu, 2012-07-19 at 12:31 +0100, Anthony PERARD wrote: > > On 19/07/12 12:07, Ian Campbell wrote: > > > On Thu, 2012-07-19 at 11:57 +0100, Anthony PERARD wrote: > > >> On 19/07/12 08:22, Ian Campbell wrote: > > >>> David also previous tried our branch of upstream qemu and had problems > > >>> there too -- I thought we''d backported the basic PCI passthrough support > > >>> but he was seeing > > >>> libxl: error: libxl_qmp.c:288:qmp_handle_error_response: > > >>> received an > > >>> error message from QMP server: Parameter ''driver'' expects a > > >>> driver > > >>> name > > >>> (see upthread for more details) > > >> > > >> This is because the pci passthrough is not compiled in QEMU. I could > > >> maybe try to improve this error message. > > > > > > Should this be enabled in our build system and/or referenced in the > > > wiki? > > > > This should not be needed as it is enable by default if possible. But I > > don''t know what went wrong here :(. > > Missing dependency? e.g. libpci-devel? > > > Otherwise, to be sure that pci passthrough is compiled in, use the > > --enable-xen-pci-passthrough configure option. > > We should enable this option in our build system so that we get a > consistent set of features not dependent on the vagaries of the > underlying system. If there are libraries which we require we should > check for them in out configure script.I didn''t backport pci passthrough to qemu-upstream-unstable so that option wouldn''t be available there
On Thu, 2012-07-19 at 12:48 +0100, Stefano Stabellini wrote:> On Thu, 19 Jul 2012, Ian Campbell wrote: > > On Thu, 2012-07-19 at 12:31 +0100, Anthony PERARD wrote: > > > On 19/07/12 12:07, Ian Campbell wrote: > > > > On Thu, 2012-07-19 at 11:57 +0100, Anthony PERARD wrote: > > > >> On 19/07/12 08:22, Ian Campbell wrote: > > > >>> David also previous tried our branch of upstream qemu and had problems > > > >>> there too -- I thought we''d backported the basic PCI passthrough support > > > >>> but he was seeing > > > >>> libxl: error: libxl_qmp.c:288:qmp_handle_error_response: > > > >>> received an > > > >>> error message from QMP server: Parameter ''driver'' expects a > > > >>> driver > > > >>> name > > > >>> (see upthread for more details) > > > >> > > > >> This is because the pci passthrough is not compiled in QEMU. I could > > > >> maybe try to improve this error message. > > > > > > > > Should this be enabled in our build system and/or referenced in the > > > > wiki? > > > > > > This should not be needed as it is enable by default if possible. But I > > > don''t know what went wrong here :(. > > > > Missing dependency? e.g. libpci-devel? > > > > > Otherwise, to be sure that pci passthrough is compiled in, use the > > > --enable-xen-pci-passthrough configure option. > > > > We should enable this option in our build system so that we get a > > consistent set of features not dependent on the vagaries of the > > underlying system. If there are libraries which we require we should > > check for them in out configure script. > > I didn''t backport pci passthrough to qemu-upstream-unstable so that > option wouldn''t be available thereAha, I thought you had, nevermind then.
On Thu, 19 Jul 2012, Ian Campbell wrote:> Yes, starting with Ubuntu is a good idea. > > The last line of the dmesg is: > (XEN) vmsi.c:108:d32767 Unsupported delivery mode 3 > which sounds interesting, might be something Stefano knows about?What is your Xen version? Make sure you have the commit: Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Date: Tue Jul 3 13:39:01 2012 +0100 xen: event channel remapping for emulated MSIs Linux PV on HVM guests remap all the MSIs onto event channels, including MSIs corresponding to QEMU''s emulated devices. This patch makes sure that we handle correctly the case of emulated MSI that have been remapped, sending a pirq to the guest instead. Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Tested-by: Deep Debroy <ddebroy@gmail.com> Committed-by: Keir Fraser <keir@xen.org>
That may be the problem, to make sure my environment matched ivo''s for compiling I was using rev 25567 which is prior to that commit. We''re headed out of town right now but I will try the tip when I get back and see if it solves that problem. I''ll also try sending a NIC through per Ian''s suggestion. Also another quick question, it seems that upstream changed the way that NICs are assigned at create time, I was seeing VIF''s (ala XenServer style) created in my ifconfig but seemingly not properly attached to my VM, or at least it wasn''t DHCP''ing correctly. Is there different syntax or something I need to change network wise when using qemu upstream? Thanks all for the help! David On Thu, Jul 19, 2012 at 5:00 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Thu, 19 Jul 2012, Ian Campbell wrote: >> Yes, starting with Ubuntu is a good idea. >> >> The last line of the dmesg is: >> (XEN) vmsi.c:108:d32767 Unsupported delivery mode 3 >> which sounds interesting, might be something Stefano knows about? > > What is your Xen version? > Make sure you have the commit: > > Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > Date: Tue Jul 3 13:39:01 2012 +0100 > > xen: event channel remapping for emulated MSIs > > Linux PV on HVM guests remap all the MSIs onto event channels, > including MSIs corresponding to QEMU''s emulated devices. This patch > makes sure that we handle correctly the case of emulated MSI that have > been remapped, sending a pirq to the guest instead. > > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > Tested-by: Deep Debroy <ddebroy@gmail.com> > Committed-by: Keir Fraser <keir@xen.org>
Just got back in town, following up on the prior discussion. I
successfully compiled the latest code (25688 and qemu upstream
5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having
problems during initialization of the card in the guest, in particular
the unsupported delivery mode 3 which seems to cause interrupt related
problems during init. I''ve again attached the qemu-dm-log, and xl
dmesg log files, and additionally screenshots of the guest dmesg and
also for comparison starting the same livecd natively on the box.
As a second question, I am not getting a NIC inside the guest for
network access when using QEMU upstream, I see vif''s added to my host
machine:
vif1.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vif1.0-emu Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:1272 (1.2 KB)
But nothing corresponding inside the guest. My guest''s config file has:
vif = [''bridge=xenbr0, type=ioemu'']
Any help appreciated on both questions.
Thanks,
David
On Thu, Jul 19, 2012 at 8:27 AM, David Erickson <halcyon1981@gmail.com>
wrote:> That may be the problem, to make sure my environment matched ivo''s
for
> compiling I was using rev 25567 which is prior to that commit.
We''re
> headed out of town right now but I will try the tip when I get back
> and see if it solves that problem. I''ll also try sending a NIC
> through per Ian''s suggestion. Also another quick question, it
seems
> that upstream changed the way that NICs are assigned at create time, I
> was seeing VIF''s (ala XenServer style) created in my ifconfig but
> seemingly not properly attached to my VM, or at least it wasn''t
> DHCP''ing correctly. Is there different syntax or something I need
to
> change network wise when using qemu upstream?
>
> Thanks all for the help!
> David
>
> On Thu, Jul 19, 2012 at 5:00 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>> On Thu, 19 Jul 2012, Ian Campbell wrote:
>>> Yes, starting with Ubuntu is a good idea.
>>>
>>> The last line of the dmesg is:
>>> (XEN) vmsi.c:108:d32767 Unsupported delivery mode 3
>>> which sounds interesting, might be something Stefano knows about?
>>
>> What is your Xen version?
>> Make sure you have the commit:
>>
>> Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>> Date: Tue Jul 3 13:39:01 2012 +0100
>>
>> xen: event channel remapping for emulated MSIs
>>
>> Linux PV on HVM guests remap all the MSIs onto event channels,
>> including MSIs corresponding to QEMU''s emulated devices.
This patch
>> makes sure that we handle correctly the case of emulated MSI that
have
>> been remapped, sending a pirq to the guest instead.
>>
>> Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
>> Tested-by: Deep Debroy <ddebroy@gmail.com>
>> Committed-by: Keir Fraser <keir@xen.org>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
On Tue, 2012-07-31 at 08:35 +0100, David Erickson wrote:> Just got back in town, following up on the prior discussion. I > successfully compiled the latest code (25688 and qemu upstream > 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having > problems during initialization of the card in the guest, in particular > the unsupported delivery mode 3 which seems to cause interrupt related > problems during init.The log has "(XEN) vmsi.c:108:d32767 Unsupported delivery mode 3" mode 3 in this context appears to be dest__reserved_1, which corresponds to the IOAPIC datasheet I''m looking at[0], which shows that DELMOD=3 is reserved. The specification update[1] doesn''t seem to add anything in this regard. So either I''ve missed another update, we are mis-emulating something resulting in an invalid mode or Linux/the LSI ROM are using an unspecified IOAPIC mode. Perhaps Jan or Andrew have some idea which it is. [0] http://www.intel.com/design/chipsets/datashts/290566.htm [1] http://www.intel.com/design/chipsets/specupdt/290710.htm> I''ve again attached the qemu-dm-log, and xl > dmesg log files, and additionally screenshots of the guest dmesg and > also for comparison starting the same livecd natively on the box. > > As a second question, I am not getting a NIC inside the guest for > network access when using QEMU upstream, I see vif''s added to my host > machine: > > vif1.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:32 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > vif1.0-emu Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff > inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:500 > RX bytes:0 (0.0 B) TX bytes:1272 (1.2 KB) > > But nothing corresponding inside the guest. My guest''s config file has: > vif = [''bridge=xenbr0, type=ioemu'']Do you have the necessary driver inside the guest? IIRC the default emulated NIC is the rtl8139. It looks like you''ve asked for EMU only so I''m not sure why you have got a vif interface, that might be a bug. You should also check that your domU kernel is not unplugging the emulated NIC -- there should be log messages to that effect if it is. I don''t recall which guest kernel you have but you could try "xen_emul_unplug=never" on the command line.> > Any help appreciated on both questions. > > Thanks, > David > > > On Thu, Jul 19, 2012 at 8:27 AM, David Erickson <halcyon1981@gmail.com> wrote: > > That may be the problem, to make sure my environment matched ivo''s for > > compiling I was using rev 25567 which is prior to that commit. We''re > > headed out of town right now but I will try the tip when I get back > > and see if it solves that problem. I''ll also try sending a NIC > > through per Ian''s suggestion. Also another quick question, it seems > > that upstream changed the way that NICs are assigned at create time, I > > was seeing VIF''s (ala XenServer style) created in my ifconfig but > > seemingly not properly attached to my VM, or at least it wasn''t > > DHCP''ing correctly. Is there different syntax or something I need to > > change network wise when using qemu upstream? > > > > Thanks all for the help! > > David > > > > On Thu, Jul 19, 2012 at 5:00 AM, Stefano Stabellini > > <stefano.stabellini@eu.citrix.com> wrote: > >> On Thu, 19 Jul 2012, Ian Campbell wrote: > >>> Yes, starting with Ubuntu is a good idea. > >>> > >>> The last line of the dmesg is: > >>> (XEN) vmsi.c:108:d32767 Unsupported delivery mode 3 > >>> which sounds interesting, might be something Stefano knows about? > >> > >> What is your Xen version? > >> Make sure you have the commit: > >> > >> Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > >> Date: Tue Jul 3 13:39:01 2012 +0100 > >> > >> xen: event channel remapping for emulated MSIs > >> > >> Linux PV on HVM guests remap all the MSIs onto event channels, > >> including MSIs corresponding to QEMU''s emulated devices. This patch > >> makes sure that we handle correctly the case of emulated MSI that have > >> been remapped, sending a pirq to the guest instead. > >> > >> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > >> Tested-by: Deep Debroy <ddebroy@gmail.com> > >> Committed-by: Keir Fraser <keir@xen.org>
>>> On 31.07.12 at 11:04, Ian Campbell <Ian.Campbell@citrix.com> wrote: > On Tue, 2012-07-31 at 08:35 +0100, David Erickson wrote: >> Just got back in town, following up on the prior discussion. I >> successfully compiled the latest code (25688 and qemu upstream >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >> problems during initialization of the card in the guest, in particular >> the unsupported delivery mode 3 which seems to cause interrupt related >> problems during init. > > The log has "(XEN) vmsi.c:108:d32767 Unsupported delivery mode 3" > > mode 3 in this context appears to be dest__reserved_1, which corresponds > to the IOAPIC datasheet I''m looking at[0], which shows that DELMOD=3 is > reserved. > > The specification update[1] doesn''t seem to add anything in this regard. > > So either I''ve missed another update, we are mis-emulating something > resulting in an invalid mode or Linux/the LSI ROM are using an > unspecified IOAPIC mode. Perhaps Jan or Andrew have some idea which it > is.At a first glance I would assume this to be the same problem that was recently fixed for running with qemu-traditional by Stefano. See http://xenbits.xen.org/gitweb/?p=staging/qemu-xen-unstable.git;a=commit;h=ce6d9b1b2f9c6a5ca2500e03d0ef8b453bc4bf53 (and the earlier thread leading there; didn''t check whether this applies to upstream qemu at all). Later he also provided a full patch, but I don''t even see this in our traditional tree, yet - Ian(J)? Jan
On Tue, 2012-07-31 at 10:35 +0100, Jan Beulich wrote:> >>> On 31.07.12 at 11:04, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Tue, 2012-07-31 at 08:35 +0100, David Erickson wrote: > >> Just got back in town, following up on the prior discussion. I > >> successfully compiled the latest code (25688 and qemu upstream > >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having > >> problems during initialization of the card in the guest, in particular > >> the unsupported delivery mode 3 which seems to cause interrupt related > >> problems during init. > > > > The log has "(XEN) vmsi.c:108:d32767 Unsupported delivery mode 3" > > > > mode 3 in this context appears to be dest__reserved_1, which corresponds > > to the IOAPIC datasheet I''m looking at[0], which shows that DELMOD=3 is > > reserved. > > > > The specification update[1] doesn''t seem to add anything in this regard. > > > > So either I''ve missed another update, we are mis-emulating something > > resulting in an invalid mode or Linux/the LSI ROM are using an > > unspecified IOAPIC mode. Perhaps Jan or Andrew have some idea which it > > is. > > At a first glance I would assume this to be the same problem > that was recently fixed for running with qemu-traditional by > Stefano. See http://xenbits.xen.org/gitweb/?p=staging/qemu-xen-unstable.git;a=commit;h=ce6d9b1b2f9c6a5ca2500e03d0ef8b453bc4bf53 > (and the earlier thread leading there; didn''t check whether this > applies to upstream qemu at all). Later he also provided a full > patch, but I don''t even see this in our traditional tree, yet - > Ian(J)?Isn''t the link above our traditional tree? Anyhow, it''s a plausible theory. David, as a workaround I think you can add "pci_msitranslate=0" to the guest configuration file. Ian.> > Jan >
>>> On 31.07.12 at 11:41, Ian Campbell <Ian.Campbell@citrix.com> wrote: > On Tue, 2012-07-31 at 10:35 +0100, Jan Beulich wrote: >> >>> On 31.07.12 at 11:04, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> > On Tue, 2012-07-31 at 08:35 +0100, David Erickson wrote: >> >> Just got back in town, following up on the prior discussion. I >> >> successfully compiled the latest code (25688 and qemu upstream >> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >> >> problems during initialization of the card in the guest, in particular >> >> the unsupported delivery mode 3 which seems to cause interrupt related >> >> problems during init. >> > >> > The log has "(XEN) vmsi.c:108:d32767 Unsupported delivery mode 3" >> > >> > mode 3 in this context appears to be dest__reserved_1, which corresponds >> > to the IOAPIC datasheet I''m looking at[0], which shows that DELMOD=3 is >> > reserved. >> > >> > The specification update[1] doesn''t seem to add anything in this regard. >> > >> > So either I''ve missed another update, we are mis-emulating something >> > resulting in an invalid mode or Linux/the LSI ROM are using an >> > unspecified IOAPIC mode. Perhaps Jan or Andrew have some idea which it >> > is. >> >> At a first glance I would assume this to be the same problem >> that was recently fixed for running with qemu-traditional by >> Stefano. See > http://xenbits.xen.org/gitweb/?p=staging/qemu-xen-unstable.git;a=commit;h=ce6 > d9b1b2f9c6a5ca2500e03d0ef8b453bc4bf53 >> (and the earlier thread leading there; didn''t check whether this >> applies to upstream qemu at all). Later he also provided a full >> patch, but I don''t even see this in our traditional tree, yet - >> Ian(J)? > > Isn''t the link above our traditional tree?Yes, as I wrote. The implication was that this and/or the more complete fix may need porting to the upstream one. Jan
On Tue, Jul 31, 2012 at 10:47 AM, Jan Beulich <JBeulich@suse.com> wrote:> > Yes, as I wrote. The implication was that this and/or the more > complete fix may need porting to the upstream one.I did not upstream the code for the MSI translation to QEMU upstream. So nothing to fix :). -- Anthony PERARD
On Tue, 31 Jul 2012, Jan Beulich wrote:> >>> On 31.07.12 at 11:04, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Tue, 2012-07-31 at 08:35 +0100, David Erickson wrote: > >> Just got back in town, following up on the prior discussion. I > >> successfully compiled the latest code (25688 and qemu upstream > >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having > >> problems during initialization of the card in the guest, in particular > >> the unsupported delivery mode 3 which seems to cause interrupt related > >> problems during init. > > > > The log has "(XEN) vmsi.c:108:d32767 Unsupported delivery mode 3" > > > > mode 3 in this context appears to be dest__reserved_1, which corresponds > > to the IOAPIC datasheet I''m looking at[0], which shows that DELMOD=3 is > > reserved. > > > > The specification update[1] doesn''t seem to add anything in this regard. > > > > So either I''ve missed another update, we are mis-emulating something > > resulting in an invalid mode or Linux/the LSI ROM are using an > > unspecified IOAPIC mode. Perhaps Jan or Andrew have some idea which it > > is. > > At a first glance I would assume this to be the same problem > that was recently fixed for running with qemu-traditional by > Stefano. See http://xenbits.xen.org/gitweb/?p=staging/qemu-xen-unstable.git;a=commit;h=ce6d9b1b2f9c6a5ca2500e03d0ef8b453bc4bf53 > (and the earlier thread leading there; didn''t check whether this > applies to upstream qemu at all). Later he also provided a full > patch, but I don''t even see this in our traditional tree, yet - > Ian(J)?This is the patch and doesn''t seem to be in qemu-xen-traditional: http://marc.info/?l=xen-devel&m=134096922121958 However qemu-xen-upstream doesn''t have msi_translate at all, so it cannot be the issue.
On Tue, 31 Jul 2012, David Erickson wrote:> Just got back in town, following up on the prior discussion. I > successfully compiled the latest code (25688 and qemu upstream > 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having > problems during initialization of the card in the guest, in particular > the unsupported delivery mode 3 which seems to cause interrupt related > problems during init. I''ve again attached the qemu-dm-log, and xl > dmesg log files, and additionally screenshots of the guest dmesg and > also for comparison starting the same livecd natively on the box."unsupported delivery mode 3" means that the Linux guest is trying to remap the MSI onto an event channel but Xen is still trying to deliver the MSI using the emulated code path anyway. Adding #define XEN_PT_LOGGING_ENABLED 1 at the top of hw/xen_pt.h and posting the additional QEMU logs could be helpful. The full Xen logs might also be useful. I would add some more tracing to the hypervisor too: diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c index b5975d1..08f4ab7 100644 --- a/xen/drivers/passthrough/io.c +++ b/xen/drivers/passthrough/io.c @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( { struct pirq *pirq = dpci_pirq(pirq_dpci); + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, + pirq->pirq, + hvm_domain_use_pirq(d, pirq), + pirq->arch.hvm.emuirq); + if ( hvm_domain_use_pirq(d, pirq) ) send_guest_pirq(d, pirq); else
>> >> As a second question, I am not getting a NIC inside the guest for >> network access when using QEMU upstream, I see vif''s added to my host >> machine: >> >> vif1.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:32 >> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> >> vif1.0-emu Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff >> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:500 >> RX bytes:0 (0.0 B) TX bytes:1272 (1.2 KB) >> >> But nothing corresponding inside the guest. My guest''s config file has: >> vif = [''bridge=xenbr0, type=ioemu''] > > Do you have the necessary driver inside the guest? IIRC the default > emulated NIC is the rtl8139. > > It looks like you''ve asked for EMU only so I''m not sure why you have got > a vif interface, that might be a bug. > > You should also check that your domU kernel is not unplugging the > emulated NIC -- there should be log messages to that effect if it is. I > don''t recall which guest kernel you have but you could try > "xen_emul_unplug=never" on the command line.I believe I should have the driver, the test guest I am using is just an Ubuntu 11.10 desktop live cd. Which command line would I put "xen_emul_unplug=never"? the guest''s kernel boot line? Also I was somewhat surprised to see xen messages in the guest''s kernel dmesg given that it is running HVM, unless it has somehow detected it is running on top of Xen. If there is an alternative vif line I should use I''m happy to change it. Thanks- David
On Tue, Jul 31, 2012 at 2:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Tue, 2012-07-31 at 10:35 +0100, Jan Beulich wrote: >> >>> On 31.07.12 at 11:04, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> > On Tue, 2012-07-31 at 08:35 +0100, David Erickson wrote: >> >> Just got back in town, following up on the prior discussion. I >> >> successfully compiled the latest code (25688 and qemu upstream >> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >> >> problems during initialization of the card in the guest, in particular >> >> the unsupported delivery mode 3 which seems to cause interrupt related >> >> problems during init. >> > >> > The log has "(XEN) vmsi.c:108:d32767 Unsupported delivery mode 3" >> > >> > mode 3 in this context appears to be dest__reserved_1, which corresponds >> > to the IOAPIC datasheet I''m looking at[0], which shows that DELMOD=3 is >> > reserved. >> > >> > The specification update[1] doesn''t seem to add anything in this regard. >> > >> > So either I''ve missed another update, we are mis-emulating something >> > resulting in an invalid mode or Linux/the LSI ROM are using an >> > unspecified IOAPIC mode. Perhaps Jan or Andrew have some idea which it >> > is. >> >> At a first glance I would assume this to be the same problem >> that was recently fixed for running with qemu-traditional by >> Stefano. See http://xenbits.xen.org/gitweb/?p=staging/qemu-xen-unstable.git;a=commit;h=ce6d9b1b2f9c6a5ca2500e03d0ef8b453bc4bf53 >> (and the earlier thread leading there; didn''t check whether this >> applies to upstream qemu at all). Later he also provided a full >> patch, but I don''t even see this in our traditional tree, yet - >> Ian(J)? > > Isn''t the link above our traditional tree? > > Anyhow, it''s a plausible theory. > > David, as a workaround I think you can add "pci_msitranslate=0" to the > guest configuration file. >Gave this a go, same problem, will try the further logging suggested below next.
On Tue, 2012-07-31 at 16:28 +0100, David Erickson wrote:> >> > >> As a second question, I am not getting a NIC inside the guest for > >> network access when using QEMU upstream, I see vif''s added to my host > >> machine: > >> > >> vif1.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff > >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:32 > >> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > >> vif1.0-emu Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff > >> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link > >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:500 > >> RX bytes:0 (0.0 B) TX bytes:1272 (1.2 KB) > >> > >> But nothing corresponding inside the guest. My guest''s config file has: > >> vif = [''bridge=xenbr0, type=ioemu''] > > > > Do you have the necessary driver inside the guest? IIRC the default > > emulated NIC is the rtl8139. > > > > It looks like you''ve asked for EMU only so I''m not sure why you have got > > a vif interface, that might be a bug. > > > > You should also check that your domU kernel is not unplugging the > > emulated NIC -- there should be log messages to that effect if it is. I > > don''t recall which guest kernel you have but you could try > > "xen_emul_unplug=never" on the command line. > > I believe I should have the driver, the test guest I am using is just > an Ubuntu 11.10 desktop live cd. Which command line would I put > "xen_emul_unplug=never"? the guest''s kernel boot line?Yes.> Also I was > somewhat surprised to see xen messages in the guest''s kernel dmesg > given that it is running HVM, unless it has somehow detected it is > running on top of Xen.Modern kernels have a feature called "PVHVM" which allows for PV features, such as device drivers, even for HVM guests. I don''t know if 11.10 has that turned on / functional or not.> If there is an alternative vif line I should > use I''m happy to change it.You could try dropping the type=ioemu. Ian
On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Tue, 31 Jul 2012, David Erickson wrote: >> Just got back in town, following up on the prior discussion. I >> successfully compiled the latest code (25688 and qemu upstream >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >> problems during initialization of the card in the guest, in particular >> the unsupported delivery mode 3 which seems to cause interrupt related >> problems during init. I''ve again attached the qemu-dm-log, and xl >> dmesg log files, and additionally screenshots of the guest dmesg and >> also for comparison starting the same livecd natively on the box. > > "unsupported delivery mode 3" means that the Linux guest is trying to > remap the MSI onto an event channel but Xen is still trying to deliver > the MSI using the emulated code path anyway. > > Adding > > #define XEN_PT_LOGGING_ENABLED 1 > > at the top of hw/xen_pt.h and posting the additional QEMU logs could > be helpful. > > The full Xen logs might also be useful. I would add some more tracing to > the hypervisor too: > > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c > index b5975d1..08f4ab7 100644 > --- a/xen/drivers/passthrough/io.c > +++ b/xen/drivers/passthrough/io.c > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( > { > struct pirq *pirq = dpci_pirq(pirq_dpci); > > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, > + pirq->pirq, > + hvm_domain_use_pirq(d, pirq), > + pirq->arch.hvm.emuirq); > + > if ( hvm_domain_use_pirq(d, pirq) ) > send_guest_pirq(d, pirq); > elseHi Stefano- I made the modifications (it looks like that DEFINE hasn''t been used in awhile, caused a few compilation issues, I had to prefix most of the logged variables with s->hostaddr.), and am attaching the qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs, where do I find those at? Thanks, David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, Jul 31, 2012 at 8:35 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Tue, 2012-07-31 at 16:28 +0100, David Erickson wrote: >> >> >> >> As a second question, I am not getting a NIC inside the guest for >> >> network access when using QEMU upstream, I see vif''s added to my host >> >> machine: >> >> >> >> vif1.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff >> >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:32 >> >> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> >> >> >> vif1.0-emu Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff >> >> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link >> >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:500 >> >> RX bytes:0 (0.0 B) TX bytes:1272 (1.2 KB) >> >> >> >> But nothing corresponding inside the guest. My guest''s config file has: >> >> vif = [''bridge=xenbr0, type=ioemu''] >> > >> > Do you have the necessary driver inside the guest? IIRC the default >> > emulated NIC is the rtl8139. >> > >> > It looks like you''ve asked for EMU only so I''m not sure why you have got >> > a vif interface, that might be a bug. >> > >> > You should also check that your domU kernel is not unplugging the >> > emulated NIC -- there should be log messages to that effect if it is. I >> > don''t recall which guest kernel you have but you could try >> > "xen_emul_unplug=never" on the command line. >> >> I believe I should have the driver, the test guest I am using is just >> an Ubuntu 11.10 desktop live cd. Which command line would I put >> "xen_emul_unplug=never"? the guest''s kernel boot line? > > Yes. > >> Also I was >> somewhat surprised to see xen messages in the guest''s kernel dmesg >> given that it is running HVM, unless it has somehow detected it is >> running on top of Xen. > > Modern kernels have a feature called "PVHVM" which allows for PV > features, such as device drivers, even for HVM guests. I don''t know if > 11.10 has that turned on / functional or not.I took a closer look at the guests''s dmesg output and indeed it says "Booting paravirtualized kernel on Xen HVM"> >> If there is an alternative vif line I should >> use I''m happy to change it. > > You could try dropping the type=ioemu.Gave this a try, still am seeing the vif interfaces on the host, nothing in the guest. Also I combed through the dmesg on the guest and didn''t see anything about any network interfaces, and nothing about being plugged/unplugged either. There is the following line in my xen-hotplug.log, if it is relevant: @xen:/var/log/xen$ sudo cat xen-hotplug.log RTNETLINK answers: Operation not supported
On Tue, 31 Jul 2012, David Erickson wrote:> On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > On Tue, 31 Jul 2012, David Erickson wrote: > >> Just got back in town, following up on the prior discussion. I > >> successfully compiled the latest code (25688 and qemu upstream > >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having > >> problems during initialization of the card in the guest, in particular > >> the unsupported delivery mode 3 which seems to cause interrupt related > >> problems during init. I''ve again attached the qemu-dm-log, and xl > >> dmesg log files, and additionally screenshots of the guest dmesg and > >> also for comparison starting the same livecd natively on the box. > > > > "unsupported delivery mode 3" means that the Linux guest is trying to > > remap the MSI onto an event channel but Xen is still trying to deliver > > the MSI using the emulated code path anyway. > > > > Adding > > > > #define XEN_PT_LOGGING_ENABLED 1 > > > > at the top of hw/xen_pt.h and posting the additional QEMU logs could > > be helpful. > > > > The full Xen logs might also be useful. I would add some more tracing to > > the hypervisor too: > > > > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c > > index b5975d1..08f4ab7 100644 > > --- a/xen/drivers/passthrough/io.c > > +++ b/xen/drivers/passthrough/io.c > > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( > > { > > struct pirq *pirq = dpci_pirq(pirq_dpci); > > > > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, > > + pirq->pirq, > > + hvm_domain_use_pirq(d, pirq), > > + pirq->arch.hvm.emuirq); > > + > > if ( hvm_domain_use_pirq(d, pirq) ) > > send_guest_pirq(d, pirq); > > else > > Hi Stefano- > I made the modifications (it looks like that DEFINE hasn''t been used > in awhile, caused a few compilation issues, I had to prefix most of > the logged variables with s->hostaddr.), and am attaching the > qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs, > where do I find those at?Thanks for the logs! You can get the full Xen logs from the serial console but you can also grab the last few lines with "xl dmesg", like you did and it seems to be enough in this case. The initial MSI remapping has been done: [00:05.0] msi_msix_setup: requested pirq 4 for MSI-X (vec: 0, entry: 0) [00:05.0] msi_msix_update: Updating MSI-X with pirq 4 gvec 0 gflags 0x3037 (entry: 0) But the guest is not issuing the EVTCHNOP_bind_pirq hypercall that is necessary to be able to receive event notifications (emuirq=-1 in the Xen logs). Now we need to figure out why: we still need more logs, this time on the guest side. What is the kernel version that you are using in the guest? Could you please add "debug loglevel=9" to the guest kernel command line and then post the guest dmesg again? It would be great if you could use the emulated serial to get the logs rather than a picture. You can do that by adding serial=''pty'' to the VM config file and console=ttyS0 to the guest command line. This additional Xen change could also tell us if the EVTCHNOP_bind_pirq has been done: diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c index 53777f8..d65a97a 100644 --- a/xen/common/event_channel.c +++ b/xen/common/event_channel.c @@ -405,6 +405,8 @@ static long evtchn_bind_pirq(evtchn_bind_pirq_t *bind) #ifdef CONFIG_X86 if ( is_hvm_domain(d) && domain_pirq_to_irq(d, pirq) > 0 ) map_domain_emuirq_pirq(d, pirq, IRQ_PT); + printk("DEBUG %s %d pirq=%d irq=%d emuirq=%d\n", __func__, __LINE__, + pirq, domain_pirq_to_irq(d, pirq), domain_pirq_to_emuirq(d, pirq)); #endif out:
On Wed, Aug 1, 2012 at 4:13 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Tue, 31 Jul 2012, David Erickson wrote: >> On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini >> <stefano.stabellini@eu.citrix.com> wrote: >> > On Tue, 31 Jul 2012, David Erickson wrote: >> >> Just got back in town, following up on the prior discussion. I >> >> successfully compiled the latest code (25688 and qemu upstream >> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >> >> problems during initialization of the card in the guest, in particular >> >> the unsupported delivery mode 3 which seems to cause interrupt related >> >> problems during init. I''ve again attached the qemu-dm-log, and xl >> >> dmesg log files, and additionally screenshots of the guest dmesg and >> >> also for comparison starting the same livecd natively on the box. >> > >> > "unsupported delivery mode 3" means that the Linux guest is trying to >> > remap the MSI onto an event channel but Xen is still trying to deliver >> > the MSI using the emulated code path anyway. >> > >> > Adding >> > >> > #define XEN_PT_LOGGING_ENABLED 1 >> > >> > at the top of hw/xen_pt.h and posting the additional QEMU logs could >> > be helpful. >> > >> > The full Xen logs might also be useful. I would add some more tracing to >> > the hypervisor too: >> > >> > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c >> > index b5975d1..08f4ab7 100644 >> > --- a/xen/drivers/passthrough/io.c >> > +++ b/xen/drivers/passthrough/io.c >> > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( >> > { >> > struct pirq *pirq = dpci_pirq(pirq_dpci); >> > >> > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, >> > + pirq->pirq, >> > + hvm_domain_use_pirq(d, pirq), >> > + pirq->arch.hvm.emuirq); >> > + >> > if ( hvm_domain_use_pirq(d, pirq) ) >> > send_guest_pirq(d, pirq); >> > else >> >> Hi Stefano- >> I made the modifications (it looks like that DEFINE hasn''t been used >> in awhile, caused a few compilation issues, I had to prefix most of >> the logged variables with s->hostaddr.), and am attaching the >> qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs, >> where do I find those at? > > Thanks for the logs! > You can get the full Xen logs from the serial console but you can also > grab the last few lines with "xl dmesg", like you did and it seems to be > enough in this case. > > > The initial MSI remapping has been done: > > [00:05.0] msi_msix_setup: requested pirq 4 for MSI-X (vec: 0, entry: 0) > [00:05.0] msi_msix_update: Updating MSI-X with pirq 4 gvec 0 gflags 0x3037 (entry: 0) > > But the guest is not issuing the EVTCHNOP_bind_pirq hypercall that is > necessary to be able to receive event notifications (emuirq=-1 in the > Xen logs). > > Now we need to figure out why: we still need more logs, this time on the > guest side. > What is the kernel version that you are using in the guest? > Could you please add "debug loglevel=9" to the guest kernel command line > and then post the guest dmesg again? > It would be great if you could use the emulated serial to get the logs > rather than a picture. You can do that by adding serial=''pty'' to the VM > config file and console=ttyS0 to the guest command line. > This additional Xen change could also tell us if the EVTCHNOP_bind_pirq > has been done: > > > diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c > index 53777f8..d65a97a 100644 > --- a/xen/common/event_channel.c > +++ b/xen/common/event_channel.c > @@ -405,6 +405,8 @@ static long evtchn_bind_pirq(evtchn_bind_pirq_t *bind) > #ifdef CONFIG_X86 > if ( is_hvm_domain(d) && domain_pirq_to_irq(d, pirq) > 0 ) > map_domain_emuirq_pirq(d, pirq, IRQ_PT); > + printk("DEBUG %s %d pirq=%d irq=%d emuirq=%d\n", __func__, __LINE__, > + pirq, domain_pirq_to_irq(d, pirq), domain_pirq_to_emuirq(d, pirq)); > #endif > > out:The guest is an Ubuntu 11.10 livecd, kernel version 3.0.0-12-generic. I''ve also attached all the logs, thanks for the tip on the serial console, very useful. Additionally I''ve attached logs for booting a solaris livecd (my ultimate goal is to use this HBA card in Solaris), with the serial console tip I was able to capture its kernel boot as well. Lastly I still also have the issue where no NIC is being presented enabled in the guests, my assumption is this is because of the error in xen-hotplug.log: "RTNETLINK answers: Operation not supported", is there some way to debug this? I checked my dom0''s dmesg and this is what I get when I boot the Ubuntu livecd VM: [ 2506.619039] device vif8.0 entered promiscuous mode [ 2506.624304] ADDRCONF(NETDEV_UP): vif8.0: link is not ready [ 2506.777865] device vif8.0-emu entered promiscuous mode [ 2506.783073] xenbr0: port 3(vif8.0-emu) entering forwarding state [ 2506.783079] xenbr0: port 3(vif8.0-emu) entering forwarding state [ 2506.895379] pciback 0000:02:00.0: restoring config space at offset 0xf (was 0x100, writing 0x10a) [ 2506.895393] pciback 0000:02:00.0: restoring config space at offset 0xc (was 0x0, writing 0xf7a00000) [ 2506.895410] pciback 0000:02:00.0: restoring config space at offset 0x7 (was 0x4, writing 0xf7a80004) [ 2506.895420] pciback 0000:02:00.0: restoring config space at offset 0x5 (was 0x4, writing 0xf7ac0004) [ 2506.895428] pciback 0000:02:00.0: restoring config space at offset 0x4 (was 0x1, writing 0xd001) [ 2506.895435] pciback 0000:02:00.0: restoring config space at offset 0x3 (was 0x0, writing 0x10) [ 2507.056216] xen-pciback: vpci: 0000:02:00.0: assign to virtual slot 0 [ 2507.056398] pciback 0000:02:00.0: device has been assigned to another domain! Over-writting the ownership, but beware. [ 2517.191338] vif8.0-emu: no IPv6 routers present [ 2521.799320] xenbr0: port 3(vif8.0-emu) entering forwarding state and here is my ifconfig while ubuntu is booted (without VMs it doesn''t have the vifs): eth0 Link encap:Ethernet HWaddr 00:e0:81:cb:db:74 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:71977 errors:0 dropped:0 overruns:0 frame:0 TX packets:126629 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5472933 (5.4 MB) TX bytes:57934759 (57.9 MB) Interrupt:16 Memory:f7900000-f7920000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:4002 errors:0 dropped:0 overruns:0 frame:0 TX packets:4002 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:26667388 (26.6 MB) TX bytes:26667388 (26.6 MB) vif8.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) vif8.0-emu Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:172 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 B) TX bytes:26220 (26.2 KB) xenbr0 Link encap:Ethernet HWaddr 00:e0:81:cb:db:74 inet addr:192.168.1.7 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::2e0:81ff:fecb:db74/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:71245 errors:0 dropped:0 overruns:0 frame:0 TX packets:98995 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3595409 (3.5 MB) TX bytes:55909320 (55.9 MB) And the line from my ubuntu.conf: vif = [''bridge=xenbr0''] Thanks- David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, 2012-08-01 at 18:52 +0100, David Erickson wrote:> my assumption is this is because of the error > in xen-hotplug.log: "RTNETLINK answers: Operation not supported",That''s a benign warning AFAIK.> and here is my ifconfig while ubuntu is booted (without VMs it doesn''t > have the vifs):This all looks fine. I think you need to be investigating the network configuration inside the guest. Does the eth* device exist, is it configured etc In your Ubuntu boot log I see: [ 0.000000] Hypervisor detected: Xen HVM [ 0.000000] Xen version 4.2. [ 0.000000] Xen Platform PCI: I/O protocol version 1 [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. which means you will need the xen-netfront driver to be loaded, I don''t see any logs to that effect. What does lsmod say? What about "ifconfig -a". Does the driver even exist on the live cd under /lib/modules somewhere? It''s a bit odd that you still have vifX.Y-emu in dom0 given that the emulated device is supposed to have been unplugged. I wonder if that is a (separate) bug with emu device unplug. Which qemu was this again? If you added xen_emul_unplug=never to your guest command line then you would avoid this unplug and you should have an emulated NIC available instead. Ian.
On Thu, Aug 2, 2012 at 12:45 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Wed, 2012-08-01 at 18:52 +0100, David Erickson wrote: >> my assumption is this is because of the error >> in xen-hotplug.log: "RTNETLINK answers: Operation not supported", > > That''s a benign warning AFAIK. > >> and here is my ifconfig while ubuntu is booted (without VMs it doesn''t >> have the vifs): > > This all looks fine. I think you need to be investigating the network > configuration inside the guest. Does the eth* device exist, is it > configured etc > > In your Ubuntu boot log I see: > [ 0.000000] Hypervisor detected: Xen HVM > [ 0.000000] Xen version 4.2. > [ 0.000000] Xen Platform PCI: I/O protocol version 1 > [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. > [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. > > which means you will need the xen-netfront driver to be loaded, I don''t > see any logs to that effect. What does lsmod say? What about "ifconfig > -a". Does the driver even exist on the live cd under /lib/modules > somewhere?Hi Ian- I''ve attached a log with the list of modules matching front, xen-netfront.ko is definitely there, but lsmod doesn''t show it loaded. And ifconfig shows nothing other than the loopback interface.> It''s a bit odd that you still have vifX.Y-emu in dom0 given that the > emulated device is supposed to have been unplugged. I wonder if that is > a (separate) bug with emu device unplug. Which qemu was this again?Upstream, commit 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a from 7/30> If you added xen_emul_unplug=never to your guest command line then you > would avoid this unplug and you should have an emulated NIC available > instead.Verified this did work. Thanks, David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, 2012-08-02 at 17:24 +0100, David Erickson wrote:> On Thu, Aug 2, 2012 at 12:45 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > I''ve attached a log with the list of modules matching front, > xen-netfront.ko is definitely there, but lsmod doesn''t show it loaded.#So did you try manually loading it? [...]> > If you added xen_emul_unplug=never to your guest command line then you > > would avoid this unplug and you should have an emulated NIC available > > instead. > > Verified this did work.Great.> > Thanks, > David
On Thu, Aug 2, 2012 at 9:45 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Thu, 2012-08-02 at 17:24 +0100, David Erickson wrote: >> On Thu, Aug 2, 2012 at 12:45 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> I''ve attached a log with the list of modules matching front, >> xen-netfront.ko is definitely there, but lsmod doesn''t show it loaded.# > > So did you try manually loading it?I did, I got: insmod: error inserting ''/lib/modules/3.0.0-12-generic/kernel/drivers/net/xen-netfront.ko'': -1 Unknown symbol in module Presumably it depends on other not-loaded kernel modules, do you know which?
On Thu, Aug 2, 2012 at 10:10 AM, David Erickson <halcyon1981@gmail.com> wrote:> On Thu, Aug 2, 2012 at 9:45 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> On Thu, 2012-08-02 at 17:24 +0100, David Erickson wrote: >>> On Thu, Aug 2, 2012 at 12:45 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >>> I''ve attached a log with the list of modules matching front, >>> xen-netfront.ko is definitely there, but lsmod doesn''t show it loaded.# >> >> So did you try manually loading it? > > I did, I got: > insmod: error inserting > ''/lib/modules/3.0.0-12-generic/kernel/drivers/net/xen-netfront.ko'': -1 > Unknown symbol in module > > Presumably it depends on other not-loaded kernel modules, do you know which?As a followup I ran the following: ubuntu@ubuntu:~$ sudo modprobe xen-netfront ubuntu@ubuntu:~$ [ 238.408574] vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632 [ 238.433304] vbd vbd-5632: failed to write error node for device/vbd/5632 (19 xenbus_dev_probe on device/vbd/5632) ubuntu@ubuntu:~$ sudo lsmod Module Size Used by xen_blkfront 26261 0 xen_netfront 26671 0 xenbus_probe_frontend 13232 2 xen_blkfront,xen_netfront,[permanent] bnep 18436 2 rfcomm 47946 0 bluetooth 166112 10 bnep,rfcomm dm_crypt 23199 0 lp 17799 0 ppdev 17113 0 parport_pc 36962 1 parport 46562 3 lp,ppdev,parport_pc psmouse 73882 0 serio_raw 13166 0 i2c_piix4 13301 0 xen_platform_pci 12885 0 [permanent] binfmt_misc 17540 1 squashfs 36799 1 overlayfs 28267 1 nls_utf8 12557 1 isofs 40253 1 dm_raid45 78155 0 xor 12894 1 dm_raid45 dm_mirror 22203 0 dm_region_hash 20918 1 dm_mirror dm_log 18564 3 dm_raid45,dm_mirror,dm_region_hash btrfs 648895 0 zlib_deflate 27139 1 btrfs libcrc32c 12644 1 btrfs floppy 70365 0 mpt2sas 152860 0 scsi_transport_sas 40558 1 mpt2sas raid_class 13622 1 mpt2sas ubuntu@ubuntu:~$ ifconfig -a eth0 Link encap:Ethernet HWaddr 00:16:3e:68:15:49 inet addr:192.168.1.126 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::216:3eff:fe68:1549/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:54 errors:0 dropped:0 overruns:0 frame:0 TX packets:51 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5952 (5.9 KB) TX bytes:9522 (9.5 KB) Interrupt:75 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:96 errors:0 dropped:0 overruns:0 frame:0 TX packets:96 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:7392 (7.3 KB) TX bytes:7392 (7.3 KB) So it looks like it loaded (with some errors, are those problems?), but still not sure why it didn''t auto load on boot.
On Wed, Aug 1, 2012 at 10:52 AM, David Erickson <halcyon1981@gmail.com> wrote:> On Wed, Aug 1, 2012 at 4:13 AM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: >> On Tue, 31 Jul 2012, David Erickson wrote: >>> On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini >>> <stefano.stabellini@eu.citrix.com> wrote: >>> > On Tue, 31 Jul 2012, David Erickson wrote: >>> >> Just got back in town, following up on the prior discussion. I >>> >> successfully compiled the latest code (25688 and qemu upstream >>> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >>> >> problems during initialization of the card in the guest, in particular >>> >> the unsupported delivery mode 3 which seems to cause interrupt related >>> >> problems during init. I''ve again attached the qemu-dm-log, and xl >>> >> dmesg log files, and additionally screenshots of the guest dmesg and >>> >> also for comparison starting the same livecd natively on the box. >>> > >>> > "unsupported delivery mode 3" means that the Linux guest is trying to >>> > remap the MSI onto an event channel but Xen is still trying to deliver >>> > the MSI using the emulated code path anyway. >>> > >>> > Adding >>> > >>> > #define XEN_PT_LOGGING_ENABLED 1 >>> > >>> > at the top of hw/xen_pt.h and posting the additional QEMU logs could >>> > be helpful. >>> > >>> > The full Xen logs might also be useful. I would add some more tracing to >>> > the hypervisor too: >>> > >>> > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c >>> > index b5975d1..08f4ab7 100644 >>> > --- a/xen/drivers/passthrough/io.c >>> > +++ b/xen/drivers/passthrough/io.c >>> > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( >>> > { >>> > struct pirq *pirq = dpci_pirq(pirq_dpci); >>> > >>> > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, >>> > + pirq->pirq, >>> > + hvm_domain_use_pirq(d, pirq), >>> > + pirq->arch.hvm.emuirq); >>> > + >>> > if ( hvm_domain_use_pirq(d, pirq) ) >>> > send_guest_pirq(d, pirq); >>> > else >>> >>> Hi Stefano- >>> I made the modifications (it looks like that DEFINE hasn''t been used >>> in awhile, caused a few compilation issues, I had to prefix most of >>> the logged variables with s->hostaddr.), and am attaching the >>> qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs, >>> where do I find those at? >> >> Thanks for the logs! >> You can get the full Xen logs from the serial console but you can also >> grab the last few lines with "xl dmesg", like you did and it seems to be >> enough in this case. >> >> >> The initial MSI remapping has been done: >> >> [00:05.0] msi_msix_setup: requested pirq 4 for MSI-X (vec: 0, entry: 0) >> [00:05.0] msi_msix_update: Updating MSI-X with pirq 4 gvec 0 gflags 0x3037 (entry: 0) >> >> But the guest is not issuing the EVTCHNOP_bind_pirq hypercall that is >> necessary to be able to receive event notifications (emuirq=-1 in the >> Xen logs). >> >> Now we need to figure out why: we still need more logs, this time on the >> guest side. >> What is the kernel version that you are using in the guest? >> Could you please add "debug loglevel=9" to the guest kernel command line >> and then post the guest dmesg again? >> It would be great if you could use the emulated serial to get the logs >> rather than a picture. You can do that by adding serial=''pty'' to the VM >> config file and console=ttyS0 to the guest command line. >> This additional Xen change could also tell us if the EVTCHNOP_bind_pirq >> has been done: >> >> >> diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c >> index 53777f8..d65a97a 100644 >> --- a/xen/common/event_channel.c >> +++ b/xen/common/event_channel.c >> @@ -405,6 +405,8 @@ static long evtchn_bind_pirq(evtchn_bind_pirq_t *bind) >> #ifdef CONFIG_X86 >> if ( is_hvm_domain(d) && domain_pirq_to_irq(d, pirq) > 0 ) >> map_domain_emuirq_pirq(d, pirq, IRQ_PT); >> + printk("DEBUG %s %d pirq=%d irq=%d emuirq=%d\n", __func__, __LINE__, >> + pirq, domain_pirq_to_irq(d, pirq), domain_pirq_to_emuirq(d, pirq)); >> #endif >> >> out: > > The guest is an Ubuntu 11.10 livecd, kernel version 3.0.0-12-generic. > I''ve also attached all the logs, thanks for the tip on the serial > console, very useful. > > Additionally I''ve attached logs for booting a solaris livecd (my > ultimate goal is to use this HBA card in Solaris), with the serial > console tip I was able to capture its kernel boot as well.I''m attaching another log from Solaris'' kernel debugger, I''m not sure how helpful it is but I found it interesting that it didn''t detect an Intel IOMMU/ACPI table and unloaded it, then tried AMD - and I''m new to Solaris but comparing this log to one without PCI Passthrough, the npe module never gets loaded without PCI passthrough, so I assume it failed while setting up the AMD IOMMU module then loaded the npe module to report the error. -D _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, 2012-08-02 at 18:12 +0100, David Erickson wrote:> ubuntu@ubuntu:~$ sudo modprobe xen-netfront > ubuntu@ubuntu:~$ [ 238.408574] vbd vbd-5632: 19 xenbus_dev_probe on > device/vbd/5632 > [ 238.433304] vbd vbd-5632: failed to write error node for > device/vbd/5632 (19 xenbus_dev_probe on device/vbd/5632) > > ubuntu@ubuntu:~$ sudo lsmod > Module Size Used by > xen_blkfront 26261 0 > xen_netfront 26671 0 > xenbus_probe_frontend 13232 2 xen_blkfront,xen_netfront,[permanent][...]> ubuntu@ubuntu:~$ ifconfig -a > eth0 Link encap:Ethernet HWaddr 00:16:3e:68:15:49[...]> So it looks like it loadedGreat.> (with some errors, are those problems?),Strangely those were vbd (aka disk) errors.> but still not sure why it didn''t auto load on boot.xenbus_probe_frontend is a module. It should either be built in or Ubuntu''s tools (primarily the one which builds initrds, but perhaps also something in the live-cd suite) need to learn to load it manually at the appropriate times. It''s a tiny module so we would generally recommend building it in. You should file this as a bug against Ubuntu, I think. I''d have sworn this had been reported to them before but perhaps it has regressed. Ian.
On Thu, Aug 2, 2012 at 10:38 AM, David Erickson <halcyon1981@gmail.com> wrote:> On Wed, Aug 1, 2012 at 10:52 AM, David Erickson <halcyon1981@gmail.com> wrote: >> On Wed, Aug 1, 2012 at 4:13 AM, Stefano Stabellini >> <stefano.stabellini@eu.citrix.com> wrote: >>> On Tue, 31 Jul 2012, David Erickson wrote: >>>> On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini >>>> <stefano.stabellini@eu.citrix.com> wrote: >>>> > On Tue, 31 Jul 2012, David Erickson wrote: >>>> >> Just got back in town, following up on the prior discussion. I >>>> >> successfully compiled the latest code (25688 and qemu upstream >>>> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having >>>> >> problems during initialization of the card in the guest, in particular >>>> >> the unsupported delivery mode 3 which seems to cause interrupt related >>>> >> problems during init. I''ve again attached the qemu-dm-log, and xl >>>> >> dmesg log files, and additionally screenshots of the guest dmesg and >>>> >> also for comparison starting the same livecd natively on the box. >>>> > >>>> > "unsupported delivery mode 3" means that the Linux guest is trying to >>>> > remap the MSI onto an event channel but Xen is still trying to deliver >>>> > the MSI using the emulated code path anyway. >>>> > >>>> > Adding >>>> > >>>> > #define XEN_PT_LOGGING_ENABLED 1 >>>> > >>>> > at the top of hw/xen_pt.h and posting the additional QEMU logs could >>>> > be helpful. >>>> > >>>> > The full Xen logs might also be useful. I would add some more tracing to >>>> > the hypervisor too: >>>> > >>>> > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c >>>> > index b5975d1..08f4ab7 100644 >>>> > --- a/xen/drivers/passthrough/io.c >>>> > +++ b/xen/drivers/passthrough/io.c >>>> > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( >>>> > { >>>> > struct pirq *pirq = dpci_pirq(pirq_dpci); >>>> > >>>> > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, >>>> > + pirq->pirq, >>>> > + hvm_domain_use_pirq(d, pirq), >>>> > + pirq->arch.hvm.emuirq); >>>> > + >>>> > if ( hvm_domain_use_pirq(d, pirq) ) >>>> > send_guest_pirq(d, pirq); >>>> > else >>>> >>>> Hi Stefano- >>>> I made the modifications (it looks like that DEFINE hasn''t been used >>>> in awhile, caused a few compilation issues, I had to prefix most of >>>> the logged variables with s->hostaddr.), and am attaching the >>>> qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs, >>>> where do I find those at? >>> >>> Thanks for the logs! >>> You can get the full Xen logs from the serial console but you can also >>> grab the last few lines with "xl dmesg", like you did and it seems to be >>> enough in this case. >>> >>> >>> The initial MSI remapping has been done: >>> >>> [00:05.0] msi_msix_setup: requested pirq 4 for MSI-X (vec: 0, entry: 0) >>> [00:05.0] msi_msix_update: Updating MSI-X with pirq 4 gvec 0 gflags 0x3037 (entry: 0) >>> >>> But the guest is not issuing the EVTCHNOP_bind_pirq hypercall that is >>> necessary to be able to receive event notifications (emuirq=-1 in the >>> Xen logs). >>> >>> Now we need to figure out why: we still need more logs, this time on the >>> guest side. >>> What is the kernel version that you are using in the guest? >>> Could you please add "debug loglevel=9" to the guest kernel command line >>> and then post the guest dmesg again? >>> It would be great if you could use the emulated serial to get the logs >>> rather than a picture. You can do that by adding serial=''pty'' to the VM >>> config file and console=ttyS0 to the guest command line. >>> This additional Xen change could also tell us if the EVTCHNOP_bind_pirq >>> has been done: >>> >>> >>> diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c >>> index 53777f8..d65a97a 100644 >>> --- a/xen/common/event_channel.c >>> +++ b/xen/common/event_channel.c >>> @@ -405,6 +405,8 @@ static long evtchn_bind_pirq(evtchn_bind_pirq_t *bind) >>> #ifdef CONFIG_X86 >>> if ( is_hvm_domain(d) && domain_pirq_to_irq(d, pirq) > 0 ) >>> map_domain_emuirq_pirq(d, pirq, IRQ_PT); >>> + printk("DEBUG %s %d pirq=%d irq=%d emuirq=%d\n", __func__, __LINE__, >>> + pirq, domain_pirq_to_irq(d, pirq), domain_pirq_to_emuirq(d, pirq)); >>> #endif >>> >>> out: >> >> The guest is an Ubuntu 11.10 livecd, kernel version 3.0.0-12-generic. >> I''ve also attached all the logs, thanks for the tip on the serial >> console, very useful. >> >> Additionally I''ve attached logs for booting a solaris livecd (my >> ultimate goal is to use this HBA card in Solaris), with the serial >> console tip I was able to capture its kernel boot as well. > > I''m attaching another log from Solaris'' kernel debugger, I''m not sure > how helpful it is but I found it interesting that it didn''t detect an > Intel IOMMU/ACPI table and unloaded it, then tried AMD - and I''m new > to Solaris but comparing this log to one without PCI Passthrough, the > npe module never gets loaded without PCI passthrough, so I assume it > failed while setting up the AMD IOMMU module then loaded the npe > module to report the error.Just following up, is there anything I can do to further help debug and figure out what is causing the problem here? I''m assuming since it isn''t working properly in PV or HVM guests there may be multiple bugs. Is there an easy way to run Ubuntu as HVM only (more friendly than Solaris) to try and isolate if that is a separate bug from what is being seen with the PV Ubuntu VM? Thanks, David
On Fri, 3 Aug 2012, David Erickson wrote:> On Thu, Aug 2, 2012 at 10:38 AM, David Erickson <halcyon1981@gmail.com> wrote: > > On Wed, Aug 1, 2012 at 10:52 AM, David Erickson <halcyon1981@gmail.com> wrote: > >> On Wed, Aug 1, 2012 at 4:13 AM, Stefano Stabellini > >> <stefano.stabellini@eu.citrix.com> wrote: > >>> On Tue, 31 Jul 2012, David Erickson wrote: > >>>> On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini > >>>> <stefano.stabellini@eu.citrix.com> wrote: > >>>> > On Tue, 31 Jul 2012, David Erickson wrote: > >>>> >> Just got back in town, following up on the prior discussion. I > >>>> >> successfully compiled the latest code (25688 and qemu upstream > >>>> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having > >>>> >> problems during initialization of the card in the guest, in particular > >>>> >> the unsupported delivery mode 3 which seems to cause interrupt related > >>>> >> problems during init. I''ve again attached the qemu-dm-log, and xl > >>>> >> dmesg log files, and additionally screenshots of the guest dmesg and > >>>> >> also for comparison starting the same livecd natively on the box. > >>>> > > >>>> > "unsupported delivery mode 3" means that the Linux guest is trying to > >>>> > remap the MSI onto an event channel but Xen is still trying to deliver > >>>> > the MSI using the emulated code path anyway. > >>>> > > >>>> > Adding > >>>> > > >>>> > #define XEN_PT_LOGGING_ENABLED 1 > >>>> > > >>>> > at the top of hw/xen_pt.h and posting the additional QEMU logs could > >>>> > be helpful. > >>>> > > >>>> > The full Xen logs might also be useful. I would add some more tracing to > >>>> > the hypervisor too: > >>>> > > >>>> > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c > >>>> > index b5975d1..08f4ab7 100644 > >>>> > --- a/xen/drivers/passthrough/io.c > >>>> > +++ b/xen/drivers/passthrough/io.c > >>>> > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert( > >>>> > { > >>>> > struct pirq *pirq = dpci_pirq(pirq_dpci); > >>>> > > >>>> > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__, > >>>> > + pirq->pirq, > >>>> > + hvm_domain_use_pirq(d, pirq), > >>>> > + pirq->arch.hvm.emuirq); > >>>> > + > >>>> > if ( hvm_domain_use_pirq(d, pirq) ) > >>>> > send_guest_pirq(d, pirq); > >>>> > else > >>>> > >>>> Hi Stefano- > >>>> I made the modifications (it looks like that DEFINE hasn''t been used > >>>> in awhile, caused a few compilation issues, I had to prefix most of > >>>> the logged variables with s->hostaddr.), and am attaching the > >>>> qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs, > >>>> where do I find those at? > >>> > >>> Thanks for the logs! > >>> You can get the full Xen logs from the serial console but you can also > >>> grab the last few lines with "xl dmesg", like you did and it seems to be > >>> enough in this case. > >>> > >>> > >>> The initial MSI remapping has been done: > >>> > >>> [00:05.0] msi_msix_setup: requested pirq 4 for MSI-X (vec: 0, entry: 0) > >>> [00:05.0] msi_msix_update: Updating MSI-X with pirq 4 gvec 0 gflags 0x3037 (entry: 0) > >>> > >>> But the guest is not issuing the EVTCHNOP_bind_pirq hypercall that is > >>> necessary to be able to receive event notifications (emuirq=-1 in the > >>> Xen logs). > >>> > >>> Now we need to figure out why: we still need more logs, this time on the > >>> guest side. > >>> What is the kernel version that you are using in the guest? > >>> Could you please add "debug loglevel=9" to the guest kernel command line > >>> and then post the guest dmesg again? > >>> It would be great if you could use the emulated serial to get the logs > >>> rather than a picture. You can do that by adding serial=''pty'' to the VM > >>> config file and console=ttyS0 to the guest command line. > >>> This additional Xen change could also tell us if the EVTCHNOP_bind_pirq > >>> has been done: > >>> > >>> > >>> diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c > >>> index 53777f8..d65a97a 100644 > >>> --- a/xen/common/event_channel.c > >>> +++ b/xen/common/event_channel.c > >>> @@ -405,6 +405,8 @@ static long evtchn_bind_pirq(evtchn_bind_pirq_t *bind) > >>> #ifdef CONFIG_X86 > >>> if ( is_hvm_domain(d) && domain_pirq_to_irq(d, pirq) > 0 ) > >>> map_domain_emuirq_pirq(d, pirq, IRQ_PT); > >>> + printk("DEBUG %s %d pirq=%d irq=%d emuirq=%d\n", __func__, __LINE__, > >>> + pirq, domain_pirq_to_irq(d, pirq), domain_pirq_to_emuirq(d, pirq)); > >>> #endif > >>> > >>> out: > >> > >> The guest is an Ubuntu 11.10 livecd, kernel version 3.0.0-12-generic. > >> I''ve also attached all the logs, thanks for the tip on the serial > >> console, very useful. > >> > >> Additionally I''ve attached logs for booting a solaris livecd (my > >> ultimate goal is to use this HBA card in Solaris), with the serial > >> console tip I was able to capture its kernel boot as well. > > > > I''m attaching another log from Solaris'' kernel debugger, I''m not sure > > how helpful it is but I found it interesting that it didn''t detect an > > Intel IOMMU/ACPI table and unloaded it, then tried AMD - and I''m new > > to Solaris but comparing this log to one without PCI Passthrough, the > > npe module never gets loaded without PCI passthrough, so I assume it > > failed while setting up the AMD IOMMU module then loaded the npe > > module to report the error.unfortunately I don''t know much about solaris so it doesn''t help me very much> Just following up, is there anything I can do to further help debug > and figure out what is causing the problem here? I''m assuming since > it isn''t working properly in PV or HVM guests there may be multiple > bugs. Is there an easy way to run Ubuntu as HVM only (more friendly > than Solaris) to try and isolate if that is a separate bug from what > is being seen with the PV Ubuntu VM?I didn''t get any specific output on the PV MSI setup, probably because it is only printed when CONFIG_DEBUG is setup. Would you be able to rebuild your Ubuntu kernel with the appended patch? diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index 6e96e65..039f29c 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -90,6 +90,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) struct msi_desc *msidesc; struct msi_msg msg; + printk("DEBUG %s %d %s %s nvec=%d\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),nvec); list_for_each_entry(msidesc, &dev->msi_list, list) { __read_msi_msg(msidesc, &msg); pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) | @@ -102,7 +103,9 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) xen_msi_compose_msg(dev, pirq, &msg); __write_msi_msg(msidesc, &msg); dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); + printk("DEBUG %s %d %s %s pirq=%d\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),pirq); } else { + printk("DEBUG %s %d %s %s pirq=%d already bound\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),pirq); dev_dbg(&dev->dev, "xen: msi already bound to pirq=%d\n", pirq); } @@ -115,9 +118,11 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) dev_dbg(&dev->dev, "xen: msi --> pirq=%d --> irq=%d\n", pirq, irq); } + printk("DEBUG %s %d %s %s pirq=%d irq=%d\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),pirq,irq); return 0; error: + printk("DEBUG %s %d %s %s error\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev)); dev_err(&dev->dev, "Xen PCI frontend has not registered MSI/MSI-X support!\n"); return -ENODEV;