Isabelle, Francois
2009-Apr-06 15:52 UTC
[Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform
Hi all, I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry?>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform?(XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro .. (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 (XEN) [VT-D]dmar.c:485: Host address width 39 (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 .. (XEN) Intel VT-d Snoop Control supported. (XEN) Intel VT-d DMA Passthrough not supported. (XEN) Intel VT-d Queued Invalidation supported. (XEN) Intel VT-d Interrupt Remapping supported. (XEN) DMAR_IQA_REG = 7f79d000 (XEN) DMAR_IQH_REG = 0 (XEN) DMAR_IQT_REG = 20 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) queue invalidate wait descriptor was not executed (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... François Isabelle _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhao, Yu
2009-Apr-07 02:31 UTC
Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform
Hi Francois, Do you have this problem all the times? Can you please grab some logs and send them to me? The dmesg from a native kernel if you can''t boot up any version of Xen, and the whole Xen hypervisor log would help us to figure out the problem. Thanks, Yu Isabelle, Francois wrote:> Hi all, > > I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? > >>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? > > > (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 > (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 > (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro > > .. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 > (XEN) [VT-D]dmar.c:485: Host address width 39 > (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD > (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 > (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 > (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 > .. > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping supported. > (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) queue invalidate wait descriptor was not executed > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > François Isabelle > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2009-Apr-07 04:40 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform
> (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20Here the IQT is such a big number 0x20... this seems pretty strange. Hi François, And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. We don''t have the same host at hand. :-( So we need try these on your host. BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu Sent: Tuesday, April 07, 2009 10:31 AM To: Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform Hi Francois, Do you have this problem all the times? Can you please grab some logs and send them to me? The dmesg from a native kernel if you can''t boot up any version of Xen, and the whole Xen hypervisor log would help us to figure out the problem. Thanks, Yu Isabelle, Francois wrote:> Hi all, > > I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? > >>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? > > > (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 > (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 > (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro > > .. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 > (XEN) [VT-D]dmar.c:485: Host address width 39 > (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD > (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 > (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 > (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 > .. > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping supported. > (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) queue invalidate wait descriptor was not executed > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > François Isabelle > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2009-Apr-07 07:05 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform
>> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >Here the IQT is such a big number 0x20... this seems pretty strange.Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: Tuesday, April 07, 2009 12:40 PM To: Zhao, Yu; Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform> (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20Here the IQT is such a big number 0x20... this seems pretty strange. Hi François, And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. We don''t have the same host at hand. :-( So we need try these on your host. BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu Sent: Tuesday, April 07, 2009 10:31 AM To: Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform Hi Francois, Do you have this problem all the times? Can you please grab some logs and send them to me? The dmesg from a native kernel if you can''t boot up any version of Xen, and the whole Xen hypervisor log would help us to figure out the problem. Thanks, Yu Isabelle, Francois wrote:> Hi all, > > I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? > >>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? > > > (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 > (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 > (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro > > .. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 > (XEN) [VT-D]dmar.c:485: Host address width 39 > (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD > (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 > (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 > (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 > .. > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping supported. > (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) queue invalidate wait descriptor was not executed > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > François Isabelle > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Isabelle, Francois
2009-Apr-07 13:12 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform
Hi. Here is what you both asked for:>> Do you have this problem all the times? Can you please grab some logs >> and send them to me?Yes Yu, all the times. I will package the logs as attachment to this post.>> iommu=xxxDexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. Here are some parts of the log I don''t understand...>(XEN) [VT-D]dmar.c:485: Host address width 39Shouldn''t this be 38 ?>(XEN) Intel VT-d Snoop Control supported. >(XEN) Intel VT-d DMA Passthrough not supported.Shouldn''t that be enabled to support DMA from domU?>(XEN) Intel VT-d Queued Invalidation supported. >(XEN) Intel VT-d Interrupt Remapping supported.>(XEN) HPET broadcast init failed, turn to PIT broadcast.Does that mean the HPET is not functional? Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? Thank you both for your replies. François Isabelle -----Message d''origine----- De : Cui, Dexuan [mailto:dexuan.cui@intel.com] Envoyé : 7 avril 2009 03:05 À : Zhao, Yu; Isabelle, Francois Cc : xen-devel@lists.xensource.com Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform>> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >Here the IQT is such a big number 0x20... this seems pretty strange.Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: Tuesday, April 07, 2009 12:40 PM To: Zhao, Yu; Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform> (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20Here the IQT is such a big number 0x20... this seems pretty strange. Hi François, And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. We don''t have the same host at hand. :-( So we need try these on your host. BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu Sent: Tuesday, April 07, 2009 10:31 AM To: Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform Hi Francois, Do you have this problem all the times? Can you please grab some logs and send them to me? The dmesg from a native kernel if you can''t boot up any version of Xen, and the whole Xen hypervisor log would help us to figure out the problem. Thanks, Yu Isabelle, Francois wrote:> Hi all, > > I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? > >>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? > > > (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 > (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 > (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro > > .. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 > (XEN) [VT-D]dmar.c:485: Host address width 39 > (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD > (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 > (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 > (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 > .. > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping supported. > (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) queue invalidate wait descriptor was not executed > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > François Isabelle > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2009-Apr-07 14:21 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform
>>> iommu=xxx > Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting.When you only use iommu=no-intremap and see the panic, can you attached the xen log? And can you also try iommu=passthrough,no-intremap and attach the log?>>(XEN) [VT-D]dmar.c:485: Host address width 39 >Shouldn''t this be 38 ?The Host Address Width (HAW) of the platform is computed as (N+1), where N is the value reported in the Host Address Width field of DMA Remapping Reporting Structure. Please see acpi_parse_dmar() or VT-d spec for details.>>(XEN) Intel VT-d Snoop Control supported. >>(XEN) Intel VT-d DMA Passthrough not supported. > Shouldn''t that be enabled to support DMA from domU?If the "DMA Passthrough" (please don''t mistake it for DMA remapping:-) of VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. If you host supports DMA passthrough (you could check your BIOS setup menu), in Xen we can enable DMA passthrough by iommu=passthrough (Note, even with this parameter, we only enable passthrough for Dom0 considering secrity)>>(XEN) HPET broadcast init failed, turn to PIT broadcast. > Does that mean the HPET is not functional? >From hpet_fsb_cap_lookup(), looks your HPET timers may work, but they don''t support FSB routing, so won''t be used in power management code.>Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all?I think we should have almost the same performance with Interrupt Remapping enabled or disabled. 82576 should work in DomU. Thanks, -- Dexuan -----Original Message----- From: Isabelle, Francois [mailto:Francois.Isabelle@ca.kontron.com] Sent: Tuesday, April 07, 2009 9:12 PM To: Cui, Dexuan; Zhao, Yu Cc: xen-devel@lists.xensource.com Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform Hi. Here is what you both asked for:>> Do you have this problem all the times? Can you please grab some logs >> and send them to me?Yes Yu, all the times. I will package the logs as attachment to this post.>> iommu=xxxDexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. Here are some parts of the log I don''t understand...>(XEN) [VT-D]dmar.c:485: Host address width 39Shouldn''t this be 38 ?>(XEN) Intel VT-d Snoop Control supported. >(XEN) Intel VT-d DMA Passthrough not supported.Shouldn''t that be enabled to support DMA from domU?>(XEN) Intel VT-d Queued Invalidation supported. >(XEN) Intel VT-d Interrupt Remapping supported.>(XEN) HPET broadcast init failed, turn to PIT broadcast.Does that mean the HPET is not functional? Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? Thank you both for your replies. François Isabelle -----Message d''origine----- De : Cui, Dexuan [mailto:dexuan.cui@intel.com] Envoyé : 7 avril 2009 03:05 À : Zhao, Yu; Isabelle, Francois Cc : xen-devel@lists.xensource.com Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform>> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >Here the IQT is such a big number 0x20... this seems pretty strange.Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: Tuesday, April 07, 2009 12:40 PM To: Zhao, Yu; Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform> (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20Here the IQT is such a big number 0x20... this seems pretty strange. Hi François, And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. We don''t have the same host at hand. :-( So we need try these on your host. BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu Sent: Tuesday, April 07, 2009 10:31 AM To: Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform Hi Francois, Do you have this problem all the times? Can you please grab some logs and send them to me? The dmesg from a native kernel if you can''t boot up any version of Xen, and the whole Xen hypervisor log would help us to figure out the problem. Thanks, Yu Isabelle, Francois wrote:> Hi all, > > I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? > >>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? > > > (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 > (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 > (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro > > .. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 > (XEN) [VT-D]dmar.c:485: Host address width 39 > (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD > (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 > (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 > (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 > .. > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping supported. > (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) queue invalidate wait descriptor was not executed > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > François Isabelle > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Isabelle, Francois
2009-Apr-07 15:06 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform
>> When you only use iommu=no-intremap and see the panic, can you attached >>the xen log? >>And can you also try iommu=passthrough,no-intremap and attach the log?See attached files.>> VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping.The BIOS does not have an option for this on my board. But I did mistook it for DMA remapping. And thank you for answering my questions. François Isabelle -----Message d''origine----- De : Cui, Dexuan [mailto:dexuan.cui@intel.com] Envoyé : 7 avril 2009 10:22 À : Isabelle, Francois; Zhao, Yu Cc : xen-devel@lists.xensource.com Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform>>> iommu=xxx > Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting.When you only use iommu=no-intremap and see the panic, can you attached the xen log? And can you also try iommu=passthrough,no-intremap and attach the log?>>(XEN) [VT-D]dmar.c:485: Host address width 39 >Shouldn''t this be 38 ?The Host Address Width (HAW) of the platform is computed as (N+1), where N is the value reported in the Host Address Width field of DMA Remapping Reporting Structure. Please see acpi_parse_dmar() or VT-d spec for details.>>(XEN) Intel VT-d Snoop Control supported. >>(XEN) Intel VT-d DMA Passthrough not supported. > Shouldn''t that be enabled to support DMA from domU?If the "DMA Passthrough" (please don''t mistake it for DMA remapping:-) of VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. If you host supports DMA passthrough (you could check your BIOS setup menu), in Xen we can enable DMA passthrough by iommu=passthrough (Note, even with this parameter, we only enable passthrough for Dom0 considering secrity)>>(XEN) HPET broadcast init failed, turn to PIT broadcast. > Does that mean the HPET is not functional? >From hpet_fsb_cap_lookup(), looks your HPET timers may work, but they don''t support FSB routing, so won''t be used in power management code.>Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all?I think we should have almost the same performance with Interrupt Remapping enabled or disabled. 82576 should work in DomU. Thanks, -- Dexuan -----Original Message----- From: Isabelle, Francois [mailto:Francois.Isabelle@ca.kontron.com] Sent: Tuesday, April 07, 2009 9:12 PM To: Cui, Dexuan; Zhao, Yu Cc: xen-devel@lists.xensource.com Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform Hi. Here is what you both asked for:>> Do you have this problem all the times? Can you please grab some logs >> and send them to me?Yes Yu, all the times. I will package the logs as attachment to this post.>> iommu=xxxDexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. Here are some parts of the log I don''t understand...>(XEN) [VT-D]dmar.c:485: Host address width 39Shouldn''t this be 38 ?>(XEN) Intel VT-d Snoop Control supported. >(XEN) Intel VT-d DMA Passthrough not supported.Shouldn''t that be enabled to support DMA from domU?>(XEN) Intel VT-d Queued Invalidation supported. >(XEN) Intel VT-d Interrupt Remapping supported.>(XEN) HPET broadcast init failed, turn to PIT broadcast.Does that mean the HPET is not functional? Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? Thank you both for your replies. François Isabelle -----Message d''origine----- De : Cui, Dexuan [mailto:dexuan.cui@intel.com] Envoyé : 7 avril 2009 03:05 À : Zhao, Yu; Isabelle, Francois Cc : xen-devel@lists.xensource.com Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform>> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >Here the IQT is such a big number 0x20... this seems pretty strange.Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: Tuesday, April 07, 2009 12:40 PM To: Zhao, Yu; Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform> (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20Here the IQT is such a big number 0x20... this seems pretty strange. Hi François, And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. We don''t have the same host at hand. :-( So we need try these on your host. BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu Sent: Tuesday, April 07, 2009 10:31 AM To: Isabelle, Francois Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform Hi Francois, Do you have this problem all the times? Can you please grab some logs and send them to me? The dmesg from a native kernel if you can''t boot up any version of Xen, and the whole Xen hypervisor log would help us to figure out the problem. Thanks, Yu Isabelle, Francois wrote:> Hi all, > > I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? > >>From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? > > > (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 > (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 > (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro > > .. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 > (XEN) [VT-D]dmar.c:485: Host address width 39 > (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD > (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 > (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 > (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 > (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 > (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 > (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 > .. > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping supported. > (XEN) DMAR_IQA_REG = 7f79d000 > (XEN) DMAR_IQH_REG = 0 > (XEN) DMAR_IQT_REG = 20 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) queue invalidate wait descriptor was not executed > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > François Isabelle > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhao, Yu
2009-Apr-08 05:54 UTC
Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform
Francois, Thank you for the log. (XEN) [VT-D]iommu.c:1237:d32767 domain_context_mapping:PCI: bdf = 0:1d.0 Did you see 1 second delay after above line was printed? Or the system got panic immediately following it? (XEN) DMAR_IQA_REG = 7f79d000 (XEN) DMAR_IQH_REG = 0 (XEN) DMAR_IQT_REG = 20 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) queue invalidate wait descriptor was not executed (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... Isabelle, Francois wrote:>>> When you only use iommu=no-intremap and see the panic, can you attached >>the xen log? >>> And can you also try iommu=passthrough,no-intremap and attach the log? > > See attached files. > >>> VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. > The BIOS does not have an option for this on my board. But I did mistook it for DMA remapping. > > > And thank you for answering my questions. > > François Isabelle > > > > > -----Message d''origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoyé : 7 avril 2009 10:22 > À : Isabelle, Francois; Zhao, Yu > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > >>>> iommu=xxx >> Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. > When you only use iommu=no-intremap and see the panic, can you attached the xen log? > And can you also try iommu=passthrough,no-intremap and attach the log? > >>> (XEN) [VT-D]dmar.c:485: Host address width 39 >> Shouldn''t this be 38 ? > The Host Address Width (HAW) of the platform is computed as (N+1), where N is the value reported in the Host Address Width field of DMA Remapping Reporting Structure. > Please see acpi_parse_dmar() or VT-d spec for details. > >>> (XEN) Intel VT-d Snoop Control supported. >>> (XEN) Intel VT-d DMA Passthrough not supported. >> Shouldn''t that be enabled to support DMA from domU? > If the "DMA Passthrough" (please don''t mistake it for DMA remapping:-) of VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. If you host supports DMA passthrough (you could check your BIOS setup menu), in Xen we can enable DMA passthrough by iommu=passthrough (Note, even with this parameter, we only enable passthrough for Dom0 considering secrity) > > >>> (XEN) HPET broadcast init failed, turn to PIT broadcast. >> Does that mean the HPET is not functional? >>From hpet_fsb_cap_lookup(), looks your HPET timers may work, but they don''t support FSB routing, so won''t be used in power management code. > >> Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? > I think we should have almost the same performance with Interrupt Remapping enabled or disabled. > 82576 should work in DomU. > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: Isabelle, Francois [mailto:Francois.Isabelle@ca.kontron.com] > Sent: Tuesday, April 07, 2009 9:12 PM > To: Cui, Dexuan; Zhao, Yu > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > > Hi. > > Here is what you both asked for: > >>> Do you have this problem all the times? Can you please grab some logs >>> and send them to me? > > Yes Yu, all the times. I will package the logs as attachment to this post. > >>> iommu=xxx > Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. > > Here are some parts of the log I don''t understand... > >> (XEN) [VT-D]dmar.c:485: Host address width 39 > Shouldn''t this be 38 ? > >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. > Shouldn''t that be enabled to support DMA from domU? > >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. > >> (XEN) HPET broadcast init failed, turn to PIT broadcast. > Does that mean the HPET is not functional? > > Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? > > Thank you both for your replies. > > François Isabelle > > -----Message d''origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoyé : 7 avril 2009 03:05 > À : Zhao, Yu; Isabelle, Francois > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > >>> (XEN) DMAR_IQA_REG = 7f79d000 >>> (XEN) DMAR_IQH_REG = 0 >>> (XEN) DMAR_IQT_REG = 20 >> Here the IQT is such a big number 0x20... this seems pretty strange. > Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. > So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan > Sent: Tuesday, April 07, 2009 12:40 PM > To: Zhao, Yu; Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform > >> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 > Here the IQT is such a big number 0x20... this seems pretty strange. > > Hi François, > And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. > We don''t have the same host at hand. :-( So we need try these on your host. > BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? > > PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu > Sent: Tuesday, April 07, 2009 10:31 AM > To: Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform > > Hi Francois, > > Do you have this problem all the times? Can you please grab some logs > and send them to me? The dmesg from a native kernel if you can''t boot up > any version of Xen, and the whole Xen hypervisor log would help us to > figure out the problem. > > Thanks, > Yu > > > Isabelle, Francois wrote: >> Hi all, >> >> I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? >> >> >From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? >> >> >> (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 >> (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 >> (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro >> >> .. >> (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs >> (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 >> (XEN) [VT-D]dmar.c:485: Host address width 39 >> (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD >> (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 >> (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 >> (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 >> (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 >> .. >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. >> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) queue invalidate wait descriptor was not executed >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >> François Isabelle >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Isabelle, Francois
2009-Apr-09 12:40 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform
>> Did you see 1 second delay after above line was printed? Or the systemgot panic immediately following it? Yes, maybe a little less but it''s not immediate for sure. François Isabelle -----Message d''origine----- De : Zhao, Yu [mailto:yu.zhao@intel.com] Envoyé : 8 avril 2009 01:55 À : Isabelle, Francois Cc : Cui, Dexuan; xen-devel@lists.xensource.com Objet : Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform Francois, Thank you for the log. (XEN) [VT-D]iommu.c:1237:d32767 domain_context_mapping:PCI: bdf = 0:1d.0 Did you see 1 second delay after above line was printed? Or the system got panic immediately following it? (XEN) DMAR_IQA_REG = 7f79d000 (XEN) DMAR_IQH_REG = 0 (XEN) DMAR_IQT_REG = 20 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) queue invalidate wait descriptor was not executed (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... Isabelle, Francois wrote:>>> When you only use iommu=no-intremap and see the panic, can you attached >>the xen log? >>> And can you also try iommu=passthrough,no-intremap and attach the log? > > See attached files. > >>> VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. > The BIOS does not have an option for this on my board. But I did mistook it for DMA remapping. > > > And thank you for answering my questions. > > François Isabelle > > > > > -----Message d''origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoyé : 7 avril 2009 10:22 > À : Isabelle, Francois; Zhao, Yu > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > >>>> iommu=xxx >> Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. > When you only use iommu=no-intremap and see the panic, can you attached the xen log? > And can you also try iommu=passthrough,no-intremap and attach the log? > >>> (XEN) [VT-D]dmar.c:485: Host address width 39 >> Shouldn''t this be 38 ? > The Host Address Width (HAW) of the platform is computed as (N+1), where N is the value reported in the Host Address Width field of DMA Remapping Reporting Structure. > Please see acpi_parse_dmar() or VT-d spec for details. > >>> (XEN) Intel VT-d Snoop Control supported. >>> (XEN) Intel VT-d DMA Passthrough not supported. >> Shouldn''t that be enabled to support DMA from domU? > If the "DMA Passthrough" (please don''t mistake it for DMA remapping:-) of VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. If you host supports DMA passthrough (you could check your BIOS setup menu), in Xen we can enable DMA passthrough by iommu=passthrough (Note, even with this parameter, we only enable passthrough for Dom0 considering secrity) > > >>> (XEN) HPET broadcast init failed, turn to PIT broadcast. >> Does that mean the HPET is not functional? >>From hpet_fsb_cap_lookup(), looks your HPET timers may work, but they don''t support FSB routing, so won''t be used in power management code. > >> Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? > I think we should have almost the same performance with Interrupt Remapping enabled or disabled. > 82576 should work in DomU. > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: Isabelle, Francois [mailto:Francois.Isabelle@ca.kontron.com] > Sent: Tuesday, April 07, 2009 9:12 PM > To: Cui, Dexuan; Zhao, Yu > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > > Hi. > > Here is what you both asked for: > >>> Do you have this problem all the times? Can you please grab some logs >>> and send them to me? > > Yes Yu, all the times. I will package the logs as attachment to this post. > >>> iommu=xxx > Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. > > Here are some parts of the log I don''t understand... > >> (XEN) [VT-D]dmar.c:485: Host address width 39 > Shouldn''t this be 38 ? > >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. > Shouldn''t that be enabled to support DMA from domU? > >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. > >> (XEN) HPET broadcast init failed, turn to PIT broadcast. > Does that mean the HPET is not functional? > > Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? > > Thank you both for your replies. > > François Isabelle > > -----Message d''origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoyé : 7 avril 2009 03:05 > À : Zhao, Yu; Isabelle, Francois > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > >>> (XEN) DMAR_IQA_REG = 7f79d000 >>> (XEN) DMAR_IQH_REG = 0 >>> (XEN) DMAR_IQT_REG = 20 >> Here the IQT is such a big number 0x20... this seems pretty strange. > Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. > So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan > Sent: Tuesday, April 07, 2009 12:40 PM > To: Zhao, Yu; Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform > >> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 > Here the IQT is such a big number 0x20... this seems pretty strange. > > Hi François, > And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. > We don''t have the same host at hand. :-( So we need try these on your host. > BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? > > PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu > Sent: Tuesday, April 07, 2009 10:31 AM > To: Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform > > Hi Francois, > > Do you have this problem all the times? Can you please grab some logs > and send them to me? The dmesg from a native kernel if you can''t boot up > any version of Xen, and the whole Xen hypervisor log would help us to > figure out the problem. > > Thanks, > Yu > > > Isabelle, Francois wrote: >> Hi all, >> >> I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? >> >> >From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? >> >> >> (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 >> (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 >> (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro >> >> .. >> (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs >> (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 >> (XEN) [VT-D]dmar.c:485: Host address width 39 >> (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD >> (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 >> (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 >> (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 >> (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 >> .. >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. >> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) queue invalidate wait descriptor was not executed >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >> François Isabelle >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Isabelle, Francois
2009-Apr-16 13:50 UTC
RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform
Just to let you know.. It turned out that our DMAR table was wrongly providing IOMMU DRHD table for inactive channel (isochronous HD audio is disabled); our AMI BIOS was clearing the "non isoch" bit in the VTISOCHCTRL register. Thank you. François Isabelle -----Message d''origine----- De : Zhao, Yu [mailto:yu.zhao@intel.com] Envoyé : 8 avril 2009 01:55 À : Isabelle, Francois Cc : Cui, Dexuan; xen-devel@lists.xensource.com Objet : Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform Francois, Thank you for the log. (XEN) [VT-D]iommu.c:1237:d32767 domain_context_mapping:PCI: bdf = 0:1d.0 Did you see 1 second delay after above line was printed? Or the system got panic immediately following it? (XEN) DMAR_IQA_REG = 7f79d000 (XEN) DMAR_IQH_REG = 0 (XEN) DMAR_IQT_REG = 20 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) queue invalidate wait descriptor was not executed (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... Isabelle, Francois wrote:>>> When you only use iommu=no-intremap and see the panic, can you attached >>the xen log? >>> And can you also try iommu=passthrough,no-intremap and attach the log? > > See attached files. > >>> VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. > The BIOS does not have an option for this on my board. But I did mistook it for DMA remapping. > > > And thank you for answering my questions. > > François Isabelle > > > > > -----Message d''origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoyé : 7 avril 2009 10:22 > À : Isabelle, Francois; Zhao, Yu > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > >>>> iommu=xxx >> Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. > When you only use iommu=no-intremap and see the panic, can you attached the xen log? > And can you also try iommu=passthrough,no-intremap and attach the log? > >>> (XEN) [VT-D]dmar.c:485: Host address width 39 >> Shouldn''t this be 38 ? > The Host Address Width (HAW) of the platform is computed as (N+1), where N is the value reported in the Host Address Width field of DMA Remapping Reporting Structure. > Please see acpi_parse_dmar() or VT-d spec for details. > >>> (XEN) Intel VT-d Snoop Control supported. >>> (XEN) Intel VT-d DMA Passthrough not supported. >> Shouldn''t that be enabled to support DMA from domU? > If the "DMA Passthrough" (please don''t mistake it for DMA remapping:-) of VT-d Context Entry is used, the DMA requests related to the Context Entry can bypass DMA remapping. This can be faster than DMA remapping. If you host supports DMA passthrough (you could check your BIOS setup menu), in Xen we can enable DMA passthrough by iommu=passthrough (Note, even with this parameter, we only enable passthrough for Dom0 considering secrity) > > >>> (XEN) HPET broadcast init failed, turn to PIT broadcast. >> Does that mean the HPET is not functional? >>From hpet_fsb_cap_lookup(), looks your HPET timers may work, but they don''t support FSB routing, so won''t be used in power management code. > >> Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? > I think we should have almost the same performance with Interrupt Remapping enabled or disabled. > 82576 should work in DomU. > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: Isabelle, Francois [mailto:Francois.Isabelle@ca.kontron.com] > Sent: Tuesday, April 07, 2009 9:12 PM > To: Cui, Dexuan; Zhao, Yu > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > > Hi. > > Here is what you both asked for: > >>> Do you have this problem all the times? Can you please grab some logs >>> and send them to me? > > Yes Yu, all the times. I will package the logs as attachment to this post. > >>> iommu=xxx > Dexuan, ''iommu=no-intremap'' is not enough to make it run, '' iommu=no-qinval,no-intremap'' allows booting. > > Here are some parts of the log I don''t understand... > >> (XEN) [VT-D]dmar.c:485: Host address width 39 > Shouldn''t this be 38 ? > >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. > Shouldn''t that be enabled to support DMA from domU? > >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. > >> (XEN) HPET broadcast init failed, turn to PIT broadcast. > Does that mean the HPET is not functional? > > Can we expect a good level of performance with interrupt remapping disabled? Will a domU be able to load a 82576 driver and use the NIC at all? > > Thank you both for your replies. > > François Isabelle > > -----Message d''origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoyé : 7 avril 2009 03:05 > À : Zhao, Yu; Isabelle, Francois > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xxplatform > >>> (XEN) DMAR_IQA_REG = 7f79d000 >>> (XEN) DMAR_IQH_REG = 0 >>> (XEN) DMAR_IQT_REG = 20 >> Here the IQT is such a big number 0x20... this seems pretty strange. > Oh, actually this 0x20 is normal -- actually QT = IQT_REG >> 4 accoriding the format of IQT_REG in VT-d spec. So QT = 0x2 -- this is ok. > So looks the panic somehow happens on the first invalidation: enable_intremap() -> iommu_flush_iec_global(). I expect your host can boot fine with iommu=no-intremap -- if this still doesn''t work, could you please try iommu=no-qinval,no-intremap ? > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan > Sent: Tuesday, April 07, 2009 12:40 PM > To: Zhao, Yu; Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform > >> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 > Here the IQT is such a big number 0x20... this seems pretty strange. > > Hi François, > And, could you please try the xen parameter "iommu=no-intremap" ? -- If you still see the panic, please also add the parameter "nosmp" to see if there would any change. > We don''t have the same host at hand. :-( So we need try these on your host. > BTW, can you boot the same host using the latest native linux-2.6.29-rc8 when you iommu_intel=on and interrupt remapping is enabled? > > PS, what''s your BIOS version and date -- could you try to find the latest BIOS for the host? I personally tend to think this is a BIOS issue. > > Thanks, > -- Dexuan > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhao, Yu > Sent: Tuesday, April 07, 2009 10:31 AM > To: Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55xx platform > > Hi Francois, > > Do you have this problem all the times? Can you please grab some logs > and send them to me? The dmesg from a native kernel if you can''t boot up > any version of Xen, and the whole Xen hypervisor log would help us to > figure out the problem. > > Thanks, > Yu > > > Isabelle, Francois wrote: >> Hi all, >> >> I''m working with a board on which XEN panics with the following message (see below). From what I can see on the lists, most of the problems on these platforms seem to be related to broken DMAR tables and RMRR, but this platform passes the VT-d firmware toolkit (http://edc.intel.com/Platforms/Xeon-5500/) successfully so I''m tented to blame XEN for it... should I file a bug entry? >> >> >From what I can see in the ''queue invalidation'' code, a timeout is reached. As anyone seen this on a similar platform? >> >> >> (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 >> (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782bc49 >> (XEN) Command line: console=com1 com1=115200,8n1 iommu=1 loglvl=all loglvl_guest=all ro >> >> .. >> (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs >> (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 >> (XEN) [VT-D]dmar.c:485: Host address width 39 >> (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD >> (XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000 >> (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7 >> (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7 >> (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0 sec = 5 sub = 5 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:4.0 sec = 6 sub = e >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0 sec = 3 sub = 3 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0 sec = 18 sub = 18 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:8.0 sec = f sub = 17 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0 sec = 4 sub = 4 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:a.0 sec = 2 sub = 2 >> .. >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. >> (XEN) DMAR_IQA_REG = 7f79d000 >> (XEN) DMAR_IQH_REG = 0 >> (XEN) DMAR_IQT_REG = 20 >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) queue invalidate wait descriptor was not executed >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >> François Isabelle >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi. I noticed you submitted 82576 quirks for XEN, I was wondering if we could expect similar patches for the 10GE 82599 chip? Right now the Beta driver I have access to crashes under XEN. Have you got a chance to test this network controller yet ? Thank you François Isabelle _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The quirk submitted for the 82576 was for SR-IOV enablement on systems for which there is no native SR-IOV support. Unless Yu Zhao has submitted other quirks for the device that I missed... Intel has not yet delivered or published any SR-IOV driver enablement for the 82599. It''s in the works is about all I can say. The Beta driver you have is for Linux and has little or no testing on Xen. Normally a driver that works in Linux will work in Xen but as noted it is Beta level only and I''d not be surprised to hear of issues with it. Regards, Greg Rose Intel Lan Access Division -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Isabelle, Francois Sent: Wednesday, June 03, 2009 5:02 AM To: Zhao, Yu Cc: xen-devel@lists.xensource.com Subject: [Xen-devel] Testing Intel 82599 under XEN Hi. I noticed you submitted 82576 quirks for XEN, I was wondering if we could expect similar patches for the 10GE 82599 chip? Right now the Beta driver I have access to crashes under XEN. Have you got a chance to test this network controller yet ? Thank you François Isabelle _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel