thr3ads.net - Xen devel - S3 crash with VTD Queue Invalidation enabled [Jun 2013]

If this information is useful, please help other people find it:
Share via:

Ben Guthro

2013-Jun-03 18:29 UTC

S3 crash with VTD Queue Invalidation enabled

I am seeing a crash on some vPro systems in the S3 path -
specifically a Lenovo ThinkPad x220t (Sandybridge)

Once I managed to not suspend the console, I got a panic in
queue_invalidate_wait()
(I added a dump_execution_state() here, to get some more info)

(XEN) Entering ACPI S3 state.
(XEN) ----[ Xen-4.2.2  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c480149091>] invalidate_sync+0x258/0x291
(XEN) RFLAGS: 0000000000010086   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff830137a665c0   rcx: 0000000000000000
(XEN) rdx: ffff82c48030a0a0   rsi: 000000000000000a   rdi: ffff82c4802766e0
(XEN) rbp: ffff82c4802bfd30   rsp: ffff82c4802bfce0   r8:  0000000000000004
(XEN) r9:  0000000000000002   r10: 0000000000000020   r11: 0000000000000010
(XEN) r12: 0000000bf34a77bc   r13: 0000000000000000   r14: ffff830137a665f8
(XEN) r15: 0000000137a5c002   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) cr3: 00000000ba2cd000   cr2: ffff880024181ff0
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c4802bfce0:
(XEN)    0000000000000002 0000000000000002 0101010000000002 0000000000000082
(XEN)    00000001802bfd30 ffff830137a665c0 0000000000000000 0000000000000000
(XEN)    0000000000000000 1000000000000000 ffff82c4802bfd90 ffff82c48014919d
(XEN)    ffff82c400000000 0000000000000000 ffff82c4802bfd60 0000000000000000
(XEN)    ffff82c4802bfd90 ffff830137a665c0 ffff830137a66540 0000000000000000
(XEN)    ffff830137a66670 ffff82c4802679e0 ffff82c4802bfde0 ffff82c480145a60
(XEN)    0000000000000000 ffff82c4802bfdc0 ffff82c480125d36 ffff82c3ffd7a00c
(XEN)    0000000000000000 0000000000000003 0000000000000003 ffff82c48030a100
(XEN)    ffff82c4802bfe20 ffff82c480145b08 ffff830137a4e620 ffff82c3ffd7a00c
(XEN)    0000000000000000 0000000000000003 0000000000000003 ffff82c48030a100
(XEN)    ffff82c4802bfe30 ffff82c480141e12 ffff82c4802bfe80 ffff82c48019f315
(XEN)    ffff82c4802bfe60 0000000000000282 0000000000000003 ffff83010cc0a010
(XEN)    ffff8300ba0fd000 0000000000000000 0000000000000003 ffff82c48030a100
(XEN)    ffff82c4802bfea0 ffff82c480105ed4 ffff8300ba0fd188 ffff82c48030a170
(XEN)    ffff82c4802bfec0 ffff82c480127a1e ffff82c480125b8a ffff82c48030a190
(XEN)    ffff82c4802bfef0 ffff82c480127d89 ffff82c4802bff18 ffff82c4802bff18
(XEN)    ffff82c4802bff18 00000000ffffffff ffff82c4802bff10 ffff82c48015a42f
(XEN)    ffff8300ba59a000 ffff8300ba0fd000 ffff82c4802bfda8 0000000000001403
(XEN)    0000000000000003 0000000000003403 ffffffff81a6b278 ffff8800049f3d28
(XEN)    0000000000000000 0000000000000246 0000000000000404 0000000000000003
(XEN) Xen call trace:
(XEN)    [<ffff82c480149091>] invalidate_sync+0x258/0x291
(XEN)    [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
(XEN)    [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
(XEN)    [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
(XEN)    [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
(XEN)    [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
(XEN)    [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
(XEN)    [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
(XEN)    [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
(XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
(XEN)
(XEN)
(XEN) DMAR_IQA_REG = 137a5c002
(XEN) DMAR_IQH_REG = 120
(XEN) DMAR_IQT_REG = 140
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) queue invalidate wait descriptor was not executed
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...



This particular dump was with Xen 4.2.2, and Linux 3.8.8
I have tested the following other combinations, with no difference in behavior:

Xen-unstable git cs da3bca931fbcf0cbdfec971aca234e7ec0f41e16, with
Linux 3.10-rc3 cs 58f8bbd2e39c3732c55698494338ee19a92c53a0

Xen-4.2.2 / linux-3.8.8
Xen-4.2.2 / linux-3.8.13
Xen-4.2.3-pre / linux-3.8.13

Booting with iommu=no-qinval or iommu=off works around the problem,
but I was wondering if there was a more elegant solution, possibly
detecting, and disabling this feature if not working properly?


Thanks in advance for any insight.

Ben

Andrew Cooper

2013-Jun-03 19:22 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On 03/06/13 19:29, Ben Guthro wrote:> I am seeing a crash on some vPro systems in the S3 path -
> specifically a Lenovo ThinkPad x220t (Sandybridge)
>
> Once I managed to not suspend the console, I got a panic in
> queue_invalidate_wait()
> (I added a dump_execution_state() here, to get some more info)
>
> (XEN) Entering ACPI S3 state.
> (XEN) ----[ Xen-4.2.2  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e008:[<ffff82c480149091>] invalidate_sync+0x258/0x291
> (XEN) RFLAGS: 0000000000010086   CONTEXT: hypervisor
> (XEN) rax: 0000000000000000   rbx: ffff830137a665c0   rcx: 0000000000000000
> (XEN) rdx: ffff82c48030a0a0   rsi: 000000000000000a   rdi: ffff82c4802766e0
> (XEN) rbp: ffff82c4802bfd30   rsp: ffff82c4802bfce0   r8:  0000000000000004
> (XEN) r9:  0000000000000002   r10: 0000000000000020   r11: 0000000000000010
> (XEN) r12: 0000000bf34a77bc   r13: 0000000000000000   r14: ffff830137a665f8
> (XEN) r15: 0000000137a5c002   cr0: 000000008005003b   cr4: 00000000000426f0
> (XEN) cr3: 00000000ba2cd000   cr2: ffff880024181ff0
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=ffff82c4802bfce0:
> (XEN)    0000000000000002 0000000000000002 0101010000000002
0000000000000082
> (XEN)    00000001802bfd30 ffff830137a665c0 0000000000000000
0000000000000000
> (XEN)    0000000000000000 1000000000000000 ffff82c4802bfd90
ffff82c48014919d
> (XEN)    ffff82c400000000 0000000000000000 ffff82c4802bfd60
0000000000000000
> (XEN)    ffff82c4802bfd90 ffff830137a665c0 ffff830137a66540
0000000000000000
> (XEN)    ffff830137a66670 ffff82c4802679e0 ffff82c4802bfde0
ffff82c480145a60
> (XEN)    0000000000000000 ffff82c4802bfdc0 ffff82c480125d36
ffff82c3ffd7a00c
> (XEN)    0000000000000000 0000000000000003 0000000000000003
ffff82c48030a100
> (XEN)    ffff82c4802bfe20 ffff82c480145b08 ffff830137a4e620
ffff82c3ffd7a00c
> (XEN)    0000000000000000 0000000000000003 0000000000000003
ffff82c48030a100
> (XEN)    ffff82c4802bfe30 ffff82c480141e12 ffff82c4802bfe80
ffff82c48019f315
> (XEN)    ffff82c4802bfe60 0000000000000282 0000000000000003
ffff83010cc0a010
> (XEN)    ffff8300ba0fd000 0000000000000000 0000000000000003
ffff82c48030a100
> (XEN)    ffff82c4802bfea0 ffff82c480105ed4 ffff8300ba0fd188
ffff82c48030a170
> (XEN)    ffff82c4802bfec0 ffff82c480127a1e ffff82c480125b8a
ffff82c48030a190
> (XEN)    ffff82c4802bfef0 ffff82c480127d89 ffff82c4802bff18
ffff82c4802bff18
> (XEN)    ffff82c4802bff18 00000000ffffffff ffff82c4802bff10
ffff82c48015a42f
> (XEN)    ffff8300ba59a000 ffff8300ba0fd000 ffff82c4802bfda8
0000000000001403
> (XEN)    0000000000000003 0000000000003403 ffffffff81a6b278
ffff8800049f3d28
> (XEN)    0000000000000000 0000000000000246 0000000000000404
0000000000000003
> (XEN) Xen call trace:
> (XEN)    [<ffff82c480149091>] invalidate_sync+0x258/0x291
> (XEN)    [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
> (XEN)    [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
> (XEN)    [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
> (XEN)    [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
> (XEN)    [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
> (XEN)    [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
> (XEN)    [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
> (XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
> (XEN)
> (XEN)
> (XEN) DMAR_IQA_REG = 137a5c002
> (XEN) DMAR_IQH_REG = 120
> (XEN) DMAR_IQT_REG = 140
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) queue invalidate wait descriptor was not executed
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>
>
>
> This particular dump was with Xen 4.2.2, and Linux 3.8.8
> I have tested the following other combinations, with no difference in
behavior:
>
> Xen-unstable git cs da3bca931fbcf0cbdfec971aca234e7ec0f41e16, with
> Linux 3.10-rc3 cs 58f8bbd2e39c3732c55698494338ee19a92c53a0
>
> Xen-4.2.2 / linux-3.8.8
> Xen-4.2.2 / linux-3.8.13
> Xen-4.2.3-pre / linux-3.8.13
>
> Booting with iommu=no-qinval or iommu=off works around the problem,
> but I was wondering if there was a more elegant solution, possibly
> detecting, and disabling this feature if not working properly?
>
>
> Thanks in advance for any insight.
>
> Ben
This was likely broken by XSA-36

My fix for the crash path is:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb56feb315f02c2b521a86

You want to inspect the use of iommu_enabled and iommu_intremap.

~Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Jan Beulich

2013-Jun-04 08:54 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 03.06.13 at 21:22, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> On 03/06/13 19:29, Ben Guthro wrote:
>> (XEN) Xen call trace:
>> (XEN)    [<ffff82c480149091>] invalidate_sync+0x258/0x291
>> (XEN)    [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>> (XEN)    [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>> (XEN)    [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>> (XEN)    [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>> (XEN)    [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
>> (XEN)    [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>> (XEN)    [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>> (XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
> 
> This was likely broken by XSA-36
> 
> My fix for the crash path is:
>
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
> 56feb315f02c2b521a86
> 
> You want to inspect the use of iommu_enabled and iommu_intremap.
According to the comment in vtd_suspend(),
iommu_disable_x2apic_IR() is supposed to run after
iommu_suspend() (and indeed lapic_suspend() gets called
immediately after iommu_suspend() by device_power_down()),
and hence that shouldn''t be the reason here. But, Ben, to be
sure, dumping the state of the various IOMMU related enabling
variables would be a good idea.

Is this perhaps having some similarity with
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
We''re clearly running single-CPU only here and there...

Jan

Ben Guthro

2013-Jun-04 12:25 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 03.06.13 at 21:22, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>> On 03/06/13 19:29, Ben Guthro wrote:
>>> (XEN) Xen call trace:
>>> (XEN)    [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>> (XEN)    [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>> (XEN)    [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>> (XEN)    [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>> (XEN)    [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>> (XEN)    [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>>> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
>>> (XEN)    [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>> (XEN)    [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>> (XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>
>> This was likely broken by XSA-36
>>
>> My fix for the crash path is:
>>
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>> 56feb315f02c2b521a86
>>
>> You want to inspect the use of iommu_enabled and iommu_intremap.
>
> According to the comment in vtd_suspend(),
> iommu_disable_x2apic_IR() is supposed to run after
> iommu_suspend() (and indeed lapic_suspend() gets called
> immediately after iommu_suspend() by device_power_down()),
> and hence that shouldn''t be the reason here. But, Ben, to be
> sure, dumping the state of the various IOMMU related enabling
> variables would be a good idea.
I assume you are referring to the variables below, defined at the top of iommu.c
At the time of the crash, they look like this:

(XEN) iommu_enabled = 1
(XEN) force_iommu; = 0
(XEN) iommu_verbose; = 0
(XEN) iommu_workaround_bios_bug; = 0
(XEN) iommu_passthrough; = 0
(XEN) iommu_snoop = 0
(XEN) iommu_qinval = 1
(XEN) iommu_intremap = 1
(XEN) iommu_hap_pt_share = 0
(XEN) iommu_debug; = 0
(XEN) amd_iommu_perdev_intremap = 1

If that gives any additional insight, please let me know.
I''m not sure I gleaned anything particularly significant from it
though.

Or - perhaps you are referring to other enabling variables?
>
> Is this perhaps having some similarity with
> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
> We''re clearly running single-CPU only here and there...
We certainly should be, as we have gone through the
disable_nonboot_cpus() by this point - and I can verify that from the
logs.

Ben

Jan Beulich

2013-Jun-04 14:01 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 03.06.13 at 21:22, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>> (XEN) Xen call trace:
>>>> (XEN)    [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>>> (XEN)    [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>>> (XEN)    [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>>> (XEN)    [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>>> (XEN)    [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>>> (XEN)    [<ffff82c48019f315>]
enter_state_helper+0x1a0/0x3cb
>>>> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
>>>> (XEN)    [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>>> (XEN)    [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>>> (XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>
>>> This was likely broken by XSA-36
>>>
>>> My fix for the crash path is:
>>>
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>> 56feb315f02c2b521a86
>>>
>>> You want to inspect the use of iommu_enabled and iommu_intremap.
>>
>> According to the comment in vtd_suspend(),
>> iommu_disable_x2apic_IR() is supposed to run after
>> iommu_suspend() (and indeed lapic_suspend() gets called
>> immediately after iommu_suspend() by device_power_down()),
>> and hence that shouldn''t be the reason here. But, Ben, to be
>> sure, dumping the state of the various IOMMU related enabling
>> variables would be a good idea.
> 
> I assume you are referring to the variables below, defined at the top of 
> iommu.c
> At the time of the crash, they look like this:
> 
> (XEN) iommu_enabled = 1
> (XEN) force_iommu; = 0
> (XEN) iommu_verbose; = 0
> (XEN) iommu_workaround_bios_bug; = 0
> (XEN) iommu_passthrough; = 0
> (XEN) iommu_snoop = 0
> (XEN) iommu_qinval = 1
> (XEN) iommu_intremap = 1
> (XEN) iommu_hap_pt_share = 0
> (XEN) iommu_debug; = 0
> (XEN) amd_iommu_perdev_intremap = 1
> 
> If that gives any additional insight, please let me know.
> I''m not sure I gleaned anything particularly significant from it
though.
> 
> Or - perhaps you are referring to other enabling variables?
These were exactly the ones (or really you picked a superset of
what I wanted to know the state of). To me this pretty clearly
means that Andrew''s original thought here is not applicable, as
at this point we can''t possibly have shut down qinval yet.
>> Is this perhaps having some similarity with
>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html? 
>> We''re clearly running single-CPU only here and there...
> 
> We certainly should be, as we have gone through the
> disable_nonboot_cpus() by this point - and I can verify that from the
> logs.
I''m much more tending towards the connection here, noting that
Andrew''s original thread didn''t really lead anywhere (i.e. we
still
don''t know what the panic he saw was actually caused by).

Jan

Ben Guthro

2013-Jun-04 19:20 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>>> On 03.06.13 at 21:22, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>>> (XEN) Xen call trace:
>>>>> (XEN)    [<ffff82c480149091>]
invalidate_sync+0x258/0x291
>>>>> (XEN)    [<ffff82c48014919d>]
flush_iotlb_qi+0xd3/0xef
>>>>> (XEN)    [<ffff82c480145a60>]
iommu_flush_all+0xb5/0xde
>>>>> (XEN)    [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>>>> (XEN)    [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>>>> (XEN)    [<ffff82c48019f315>]
enter_state_helper+0x1a0/0x3cb
>>>>> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
>>>>> (XEN)    [<ffff82c480127a1e>]
do_tasklet_work+0x8d/0xc7
>>>>> (XEN)    [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>>>> (XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>>
>>>> This was likely broken by XSA-36
>>>>
>>>> My fix for the crash path is:
>>>>
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>>> 56feb315f02c2b521a86
>>>>
>>>> You want to inspect the use of iommu_enabled and
iommu_intremap.
>>>
>>> According to the comment in vtd_suspend(),
>>> iommu_disable_x2apic_IR() is supposed to run after
>>> iommu_suspend() (and indeed lapic_suspend() gets called
>>> immediately after iommu_suspend() by device_power_down()),
>>> and hence that shouldn''t be the reason here. But, Ben, to
be
>>> sure, dumping the state of the various IOMMU related enabling
>>> variables would be a good idea.
>>
>> I assume you are referring to the variables below, defined at the top
of
>> iommu.c
>> At the time of the crash, they look like this:
>>
>> (XEN) iommu_enabled = 1
>> (XEN) force_iommu; = 0
>> (XEN) iommu_verbose; = 0
>> (XEN) iommu_workaround_bios_bug; = 0
>> (XEN) iommu_passthrough; = 0
>> (XEN) iommu_snoop = 0
>> (XEN) iommu_qinval = 1
>> (XEN) iommu_intremap = 1
>> (XEN) iommu_hap_pt_share = 0
>> (XEN) iommu_debug; = 0
>> (XEN) amd_iommu_perdev_intremap = 1
>>
>> If that gives any additional insight, please let me know.
>> I''m not sure I gleaned anything particularly significant from
it though.
>>
>> Or - perhaps you are referring to other enabling variables?
>
> These were exactly the ones (or really you picked a superset of
> what I wanted to know the state of). To me this pretty clearly
> means that Andrew''s original thought here is not applicable, as
> at this point we can''t possibly have shut down qinval yet.
>
>>> Is this perhaps having some similarity with
>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>> We''re clearly running single-CPU only here and there...
>>
>> We certainly should be, as we have gone through the
>> disable_nonboot_cpus() by this point - and I can verify that from the
>> logs.
>
> I''m much more tending towards the connection here, noting that
> Andrew''s original thread didn''t really lead anywhere
(i.e. we still
> don''t know what the panic he saw was actually caused by).
>
I''m starting to think you''re on to something here.
I''ve put a bunch of trace throughout the functions in qinval.c

It seems that everything is functioning properly, up until we go
through the disable_nonboot_cpus() path.
Prior to this, I see the qinval.c functions being executed on all
cpus, and both drhd units
Afterward, it gets stuck in queue_invalidate_wait on the first drhd
unit.. and eventually panics.

I''m not exactly sure what to make of this yet.

Ben Guthro

2013-Jun-04 19:49 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net>
wrote:> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net>
wrote:
>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>> On 03.06.13 at 21:22, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>>>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>>>> (XEN) Xen call trace:
>>>>>> (XEN)    [<ffff82c480149091>]
invalidate_sync+0x258/0x291
>>>>>> (XEN)    [<ffff82c48014919d>]
flush_iotlb_qi+0xd3/0xef
>>>>>> (XEN)    [<ffff82c480145a60>]
iommu_flush_all+0xb5/0xde
>>>>>> (XEN)    [<ffff82c480145b08>]
vtd_suspend+0x23/0xf1
>>>>>> (XEN)    [<ffff82c480141e12>]
iommu_suspend+0x3c/0x3e
>>>>>> (XEN)    [<ffff82c48019f315>]
enter_state_helper+0x1a0/0x3cb
>>>>>> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
>>>>>> (XEN)    [<ffff82c480127a1e>]
do_tasklet_work+0x8d/0xc7
>>>>>> (XEN)    [<ffff82c480127d89>]
do_tasklet+0x6b/0x9b
>>>>>> (XEN)    [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>>>
>>>>> This was likely broken by XSA-36
>>>>>
>>>>> My fix for the crash path is:
>>>>>
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>>>> 56feb315f02c2b521a86
>>>>>
>>>>> You want to inspect the use of iommu_enabled and
iommu_intremap.
>>>>
>>>> According to the comment in vtd_suspend(),
>>>> iommu_disable_x2apic_IR() is supposed to run after
>>>> iommu_suspend() (and indeed lapic_suspend() gets called
>>>> immediately after iommu_suspend() by device_power_down()),
>>>> and hence that shouldn''t be the reason here. But, Ben,
to be
>>>> sure, dumping the state of the various IOMMU related enabling
>>>> variables would be a good idea.
>>>
>>> I assume you are referring to the variables below, defined at the
top of
>>> iommu.c
>>> At the time of the crash, they look like this:
>>>
>>> (XEN) iommu_enabled = 1
>>> (XEN) force_iommu; = 0
>>> (XEN) iommu_verbose; = 0
>>> (XEN) iommu_workaround_bios_bug; = 0
>>> (XEN) iommu_passthrough; = 0
>>> (XEN) iommu_snoop = 0
>>> (XEN) iommu_qinval = 1
>>> (XEN) iommu_intremap = 1
>>> (XEN) iommu_hap_pt_share = 0
>>> (XEN) iommu_debug; = 0
>>> (XEN) amd_iommu_perdev_intremap = 1
>>>
>>> If that gives any additional insight, please let me know.
>>> I''m not sure I gleaned anything particularly significant
from it though.
>>>
>>> Or - perhaps you are referring to other enabling variables?
>>
>> These were exactly the ones (or really you picked a superset of
>> what I wanted to know the state of). To me this pretty clearly
>> means that Andrew''s original thought here is not applicable,
as
>> at this point we can''t possibly have shut down qinval yet.
>>
>>>> Is this perhaps having some similarity with
>>>>
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>> We''re clearly running single-CPU only here and
there...
>>>
>>> We certainly should be, as we have gone through the
>>> disable_nonboot_cpus() by this point - and I can verify that from
the
>>> logs.
>>
>> I''m much more tending towards the connection here, noting that
>> Andrew''s original thread didn''t really lead anywhere
(i.e. we still
>> don''t know what the panic he saw was actually caused by).
>>
>
> I''m starting to think you''re on to something here.
hmm - maybe not.
I get the same crash with "maxcpus=1"


> I''ve put a bunch of trace throughout the functions in qinval.c
>
> It seems that everything is functioning properly, up until we go
> through the disable_nonboot_cpus() path.
> Prior to this, I see the qinval.c functions being executed on all
> cpus, and both drhd units
> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
> unit.. and eventually panics.
>
> I''m not exactly sure what to make of this yet.

Ben Guthro

2013-Jun-04 21:09 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net>
wrote:> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net>
wrote:
>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>> On 03.06.13 at 21:22, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>>>>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>>>>> (XEN) Xen call trace:
>>>>>>> (XEN)    [<ffff82c480149091>]
invalidate_sync+0x258/0x291
>>>>>>> (XEN)    [<ffff82c48014919d>]
flush_iotlb_qi+0xd3/0xef
>>>>>>> (XEN)    [<ffff82c480145a60>]
iommu_flush_all+0xb5/0xde
>>>>>>> (XEN)    [<ffff82c480145b08>]
vtd_suspend+0x23/0xf1
>>>>>>> (XEN)    [<ffff82c480141e12>]
iommu_suspend+0x3c/0x3e
>>>>>>> (XEN)    [<ffff82c48019f315>]
enter_state_helper+0x1a0/0x3cb
>>>>>>> (XEN)    [<ffff82c480105ed4>]
continue_hypercall_tasklet_handler+0x51/0xbf
>>>>>>> (XEN)    [<ffff82c480127a1e>]
do_tasklet_work+0x8d/0xc7
>>>>>>> (XEN)    [<ffff82c480127d89>]
do_tasklet+0x6b/0x9b
>>>>>>> (XEN)    [<ffff82c48015a42f>]
idle_loop+0x67/0x6f
>>>>>>
>>>>>> This was likely broken by XSA-36
>>>>>>
>>>>>> My fix for the crash path is:
>>>>>>
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>>>>> 56feb315f02c2b521a86
>>>>>>
>>>>>> You want to inspect the use of iommu_enabled and
iommu_intremap.
>>>>>
>>>>> According to the comment in vtd_suspend(),
>>>>> iommu_disable_x2apic_IR() is supposed to run after
>>>>> iommu_suspend() (and indeed lapic_suspend() gets called
>>>>> immediately after iommu_suspend() by device_power_down()),
>>>>> and hence that shouldn''t be the reason here. But,
Ben, to be
>>>>> sure, dumping the state of the various IOMMU related
enabling
>>>>> variables would be a good idea.
>>>>
>>>> I assume you are referring to the variables below, defined at
the top of
>>>> iommu.c
>>>> At the time of the crash, they look like this:
>>>>
>>>> (XEN) iommu_enabled = 1
>>>> (XEN) force_iommu; = 0
>>>> (XEN) iommu_verbose; = 0
>>>> (XEN) iommu_workaround_bios_bug; = 0
>>>> (XEN) iommu_passthrough; = 0
>>>> (XEN) iommu_snoop = 0
>>>> (XEN) iommu_qinval = 1
>>>> (XEN) iommu_intremap = 1
>>>> (XEN) iommu_hap_pt_share = 0
>>>> (XEN) iommu_debug; = 0
>>>> (XEN) amd_iommu_perdev_intremap = 1
>>>>
>>>> If that gives any additional insight, please let me know.
>>>> I''m not sure I gleaned anything particularly
significant from it though.
>>>>
>>>> Or - perhaps you are referring to other enabling variables?
>>>
>>> These were exactly the ones (or really you picked a superset of
>>> what I wanted to know the state of). To me this pretty clearly
>>> means that Andrew''s original thought here is not
applicable, as
>>> at this point we can''t possibly have shut down qinval yet.
>>>
>>>>> Is this perhaps having some similarity with
>>>>>
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>> We''re clearly running single-CPU only here and
there...
>>>>
>>>> We certainly should be, as we have gone through the
>>>> disable_nonboot_cpus() by this point - and I can verify that
from the
>>>> logs.
>>>
>>> I''m much more tending towards the connection here, noting
that
>>> Andrew''s original thread didn''t really lead
anywhere (i.e. we still
>>> don''t know what the panic he saw was actually caused by).
>>>
>>
>> I''m starting to think you''re on to something here.
>
> hmm - maybe not.
> I get the same crash with "maxcpus=1"
>
>
>
>> I''ve put a bunch of trace throughout the functions in qinval.c
>>
>> It seems that everything is functioning properly, up until we go
>> through the disable_nonboot_cpus() path.
>> Prior to this, I see the qinval.c functions being executed on all
>> cpus, and both drhd units
>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>> unit.. and eventually panics.
>>
>> I''m not exactly sure what to make of this yet.
querying status of the hardware all seems to be working correctly...
it just doesn''t work with querying the QINVAL_STAT_DONE state, as far
as I can tell.

Other register state is:

(XEN)  VER = 10
(XEN)  CAP = c0000020e60262
(XEN)  n_fault_reg = 1
(XEN)  fault_recording_offset = 200
(XEN)  fault_recording_reg_l = 0
(XEN)  fault_recording_reg_h = 0
(XEN)  ECAP = f0101a
(XEN)  GCMD = 0
(XEN)  GSTS = c7000000
(XEN)  RTADDR = 137a31000
(XEN)  CCMD = 800000000000000
(XEN)  FSTS = 0
(XEN)  FECTL = 0
(XEN)  FEDATA = 4128
(XEN)  FEADDR = fee0000c
(XEN)  FEUADDR = 0

(with code lifted from print_iommu_regs() )


None of this looks suspicious to my untrained eye - but I''m including
it here in case someone else sees something I don''t.

Jan Beulich

2013-Jun-05 08:24 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net> wrote:
> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net>
wrote:
>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>> On 04.06.13 at 14:25, Ben Guthro
<ben@guthro.net> wrote:
>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>> Is this perhaps having some similarity with
>>>>>>
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>> We''re clearly running single-CPU only here and
there...
>>>>>
>>>>> We certainly should be, as we have gone through the
>>>>> disable_nonboot_cpus() by this point - and I can verify
that from the
>>>>> logs.
>>>>
>>>> I''m much more tending towards the connection here,
noting that
>>>> Andrew''s original thread didn''t really lead
anywhere (i.e. we still
>>>> don''t know what the panic he saw was actually caused
by).
>>>>
>>>
>>> I''m starting to think you''re on to something
here.
>>
>> hmm - maybe not.
>> I get the same crash with "maxcpus=1"
>>
>>
>>
>>> I''ve put a bunch of trace throughout the functions in
qinval.c
>>>
>>> It seems that everything is functioning properly, up until we go
>>> through the disable_nonboot_cpus() path.
>>> Prior to this, I see the qinval.c functions being executed on all
>>> cpus, and both drhd units
>>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>>> unit.. and eventually panics.
>>>
>>> I''m not exactly sure what to make of this yet.
> 
> querying status of the hardware all seems to be working correctly...
> it just doesn''t work with querying the QINVAL_STAT_DONE state, as
far
> as I can tell.
> 
> Other register state is:
> 
> (XEN)  VER = 10
> (XEN)  CAP = c0000020e60262
> (XEN)  n_fault_reg = 1
> (XEN)  fault_recording_offset = 200
> (XEN)  fault_recording_reg_l = 0
> (XEN)  fault_recording_reg_h = 0
> (XEN)  ECAP = f0101a
> (XEN)  GCMD = 0
> (XEN)  GSTS = c7000000
> (XEN)  RTADDR = 137a31000
> (XEN)  CCMD = 800000000000000
> (XEN)  FSTS = 0
> (XEN)  FECTL = 0
> (XEN)  FEDATA = 4128
> (XEN)  FEADDR = fee0000c
> (XEN)  FEUADDR = 0
> 
> (with code lifted from print_iommu_regs() )
> 
> 
> None of this looks suspicious to my untrained eye - but I''m
including
> it here in case someone else sees something I don''t.
Xiantao, you certainly will want to give some advice here. I won''t
be able to look into this more deeply right away.

Jan

Ben Guthro

2013-Jun-05 13:54 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net>
wrote:
>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net>
wrote:
>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>> On 04.06.13 at 14:25, Ben Guthro
<ben@guthro.net> wrote:
>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>> Is this perhaps having some similarity with
>>>>>>>
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>>> We''re clearly running single-CPU only here
and there...
>>>>>>
>>>>>> We certainly should be, as we have gone through the
>>>>>> disable_nonboot_cpus() by this point - and I can verify
that from the
>>>>>> logs.
>>>>>
>>>>> I''m much more tending towards the connection here,
noting that
>>>>> Andrew''s original thread didn''t really
lead anywhere (i.e. we still
>>>>> don''t know what the panic he saw was actually
caused by).
>>>>>
>>>>
>>>> I''m starting to think you''re on to something
here.
>>>
>>> hmm - maybe not.
>>> I get the same crash with "maxcpus=1"
>>>
>>>
>>>
>>>> I''ve put a bunch of trace throughout the functions in
qinval.c
>>>>
>>>> It seems that everything is functioning properly, up until we
go
>>>> through the disable_nonboot_cpus() path.
>>>> Prior to this, I see the qinval.c functions being executed on
all
>>>> cpus, and both drhd units
>>>> Afterward, it gets stuck in queue_invalidate_wait on the first
drhd
>>>> unit.. and eventually panics.
>>>>
>>>> I''m not exactly sure what to make of this yet.
>>
>> querying status of the hardware all seems to be working correctly...
>> it just doesn''t work with querying the QINVAL_STAT_DONE state,
as far
>> as I can tell.
>>
>> Other register state is:
>>
>> (XEN)  VER = 10
>> (XEN)  CAP = c0000020e60262
>> (XEN)  n_fault_reg = 1
>> (XEN)  fault_recording_offset = 200
>> (XEN)  fault_recording_reg_l = 0
>> (XEN)  fault_recording_reg_h = 0
>> (XEN)  ECAP = f0101a
>> (XEN)  GCMD = 0
>> (XEN)  GSTS = c7000000
>> (XEN)  RTADDR = 137a31000
>> (XEN)  CCMD = 800000000000000
>> (XEN)  FSTS = 0
>> (XEN)  FECTL = 0
>> (XEN)  FEDATA = 4128
>> (XEN)  FEADDR = fee0000c
>> (XEN)  FEUADDR = 0
>>
>> (with code lifted from print_iommu_regs() )
>>
>>
>> None of this looks suspicious to my untrained eye - but I''m
including
>> it here in case someone else sees something I don''t.
>
> Xiantao, you certainly will want to give some advice here. I won''t
> be able to look into this more deeply right away.
Thanks Jan. Xiantao - I''d appreciate any insight you may have.

One curious thing I have found, that seems buggy to me, is that
{dis,en}able_qinval() is being called prior to the platform quirks
being executed.
It appears they are being called through iommu_{en,dis}able_x2apic_IR()

However, when I try to put a BUG(), or dump_execution_state in that
code, it would not dump a stack.

I was going to put a platform quirk in, to detect, and disable qinval
on this platform, but it seems that may be too late in the process.

Jan Beulich

2013-Jun-05 15:14 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 05.06.13 at 15:54, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net>
wrote:
>>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net>
wrote:
>>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro
<ben@guthro.net> wrote:
>>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>>> On 04.06.13 at 14:25, Ben Guthro
<ben@guthro.net> wrote:
>>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>> Is this perhaps having some similarity with
>>>>>>>>
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>>>> We''re clearly running single-CPU only
here and there...
>>>>>>>
>>>>>>> We certainly should be, as we have gone through the
>>>>>>> disable_nonboot_cpus() by this point - and I can
verify that from the
>>>>>>> logs.
>>>>>>
>>>>>> I''m much more tending towards the connection
here, noting that
>>>>>> Andrew''s original thread didn''t
really lead anywhere (i.e. we still
>>>>>> don''t know what the panic he saw was actually
caused by).
>>>>>>
>>>>>
>>>>> I''m starting to think you''re on to
something here.
>>>>
>>>> hmm - maybe not.
>>>> I get the same crash with "maxcpus=1"
>>>>
>>>>
>>>>
>>>>> I''ve put a bunch of trace throughout the functions
in qinval.c
>>>>>
>>>>> It seems that everything is functioning properly, up until
we go
>>>>> through the disable_nonboot_cpus() path.
>>>>> Prior to this, I see the qinval.c functions being executed
on all
>>>>> cpus, and both drhd units
>>>>> Afterward, it gets stuck in queue_invalidate_wait on the
first drhd
>>>>> unit.. and eventually panics.
>>>>>
>>>>> I''m not exactly sure what to make of this yet.
>>>
>>> querying status of the hardware all seems to be working
correctly...
>>> it just doesn''t work with querying the QINVAL_STAT_DONE
state, as far
>>> as I can tell.
>>>
>>> Other register state is:
>>>
>>> (XEN)  VER = 10
>>> (XEN)  CAP = c0000020e60262
>>> (XEN)  n_fault_reg = 1
>>> (XEN)  fault_recording_offset = 200
>>> (XEN)  fault_recording_reg_l = 0
>>> (XEN)  fault_recording_reg_h = 0
>>> (XEN)  ECAP = f0101a
>>> (XEN)  GCMD = 0
>>> (XEN)  GSTS = c7000000
With

#define DMA_GCMD_QIE    (((u64)1) << 26)

and

#define DMA_GSTS_QIES   (((u64)1) <<26)

this means qinval is still enabled at this point.
>>> (XEN)  RTADDR = 137a31000
>>> (XEN)  CCMD = 800000000000000
>>> (XEN)  FSTS = 0
>>> (XEN)  FECTL = 0
>>> (XEN)  FEDATA = 4128
>>> (XEN)  FEADDR = fee0000c
>>> (XEN)  FEUADDR = 0
>>>
>>> (with code lifted from print_iommu_regs() )
>>>
>>>
>>> None of this looks suspicious to my untrained eye - but
I''m including
>>> it here in case someone else sees something I don''t.
>>
>> Xiantao, you certainly will want to give some advice here. I
won''t
>> be able to look into this more deeply right away.
> 
> Thanks Jan. Xiantao - I''d appreciate any insight you may have.
> 
> One curious thing I have found, that seems buggy to me, is that
> {dis,en}able_qinval() is being called prior to the platform quirks
> being executed.
> It appears they are being called through iommu_{en,dis}able_x2apic_IR()
That''s because this setup needs to happen when interrupt (i.e.
APIC) initialization is happening, not when the IOMMUs get set
up (which is a process that assumes interrupts can already be
requested).

In effect we have

lapic_suspend() -> iommu_disable_x2apic_IR() ->
disable_intremap()/disable_qinval()

after

iommu_suspend() -> vtd_suspend() -> disable_qinval()

but the latter tail call only when !iommu_intremap, and you
have been running with interrupt remapping enabled, so only
the former code path would result in qinval getting disabled,
which is after the point of the hang.

Depending on whether ATS is in use, more than one invalidation
can be done in the processing here - could you therefore check
whether there''s any sign of ATS use ("iommu=verbose" should
make you see respective messages), and if so see whether
disabling it ("ats=off") makes a difference?

Jan

Ben Guthro

2013-Jun-05 15:25 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 05.06.13 at 15:54, Ben Guthro <ben@guthro.net> wrote:
>> On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net>
wrote:
>>>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro
<ben@guthro.net> wrote:
>>>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro
<ben@guthro.net> wrote:
>>>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>>>> On 04.06.13 at 14:25, Ben Guthro
<ben@guthro.net> wrote:
>>>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>>> Is this perhaps having some similarity with
>>>>>>>>>
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>>>>> We''re clearly running single-CPU
only here and there...
>>>>>>>>
>>>>>>>> We certainly should be, as we have gone through
the
>>>>>>>> disable_nonboot_cpus() by this point - and I
can verify that from the
>>>>>>>> logs.
>>>>>>>
>>>>>>> I''m much more tending towards the
connection here, noting that
>>>>>>> Andrew''s original thread didn''t
really lead anywhere (i.e. we still
>>>>>>> don''t know what the panic he saw was
actually caused by).
>>>>>>>
>>>>>>
>>>>>> I''m starting to think you''re on to
something here.
>>>>>
>>>>> hmm - maybe not.
>>>>> I get the same crash with "maxcpus=1"
>>>>>
>>>>>
>>>>>
>>>>>> I''ve put a bunch of trace throughout the
functions in qinval.c
>>>>>>
>>>>>> It seems that everything is functioning properly, up
until we go
>>>>>> through the disable_nonboot_cpus() path.
>>>>>> Prior to this, I see the qinval.c functions being
executed on all
>>>>>> cpus, and both drhd units
>>>>>> Afterward, it gets stuck in queue_invalidate_wait on
the first drhd
>>>>>> unit.. and eventually panics.
>>>>>>
>>>>>> I''m not exactly sure what to make of this yet.
>>>>
>>>> querying status of the hardware all seems to be working
correctly...
>>>> it just doesn''t work with querying the
QINVAL_STAT_DONE state, as far
>>>> as I can tell.
>>>>
>>>> Other register state is:
>>>>
>>>> (XEN)  VER = 10
>>>> (XEN)  CAP = c0000020e60262
>>>> (XEN)  n_fault_reg = 1
>>>> (XEN)  fault_recording_offset = 200
>>>> (XEN)  fault_recording_reg_l = 0
>>>> (XEN)  fault_recording_reg_h = 0
>>>> (XEN)  ECAP = f0101a
>>>> (XEN)  GCMD = 0
>>>> (XEN)  GSTS = c7000000
>
> With
>
> #define DMA_GCMD_QIE    (((u64)1) << 26)
>
> and
>
> #define DMA_GSTS_QIES   (((u64)1) <<26)
>
> this means qinval is still enabled at this point.
>
>>>> (XEN)  RTADDR = 137a31000
>>>> (XEN)  CCMD = 800000000000000
>>>> (XEN)  FSTS = 0
>>>> (XEN)  FECTL = 0
>>>> (XEN)  FEDATA = 4128
>>>> (XEN)  FEADDR = fee0000c
>>>> (XEN)  FEUADDR = 0
>>>>
>>>> (with code lifted from print_iommu_regs() )
>>>>
>>>>
>>>> None of this looks suspicious to my untrained eye - but
I''m including
>>>> it here in case someone else sees something I don''t.
>>>
>>> Xiantao, you certainly will want to give some advice here. I
won''t
>>> be able to look into this more deeply right away.
>>
>> Thanks Jan. Xiantao - I''d appreciate any insight you may have.
>>
>> One curious thing I have found, that seems buggy to me, is that
>> {dis,en}able_qinval() is being called prior to the platform quirks
>> being executed.
>> It appears they are being called through iommu_{en,dis}able_x2apic_IR()
>
> That''s because this setup needs to happen when interrupt (i.e.
> APIC) initialization is happening, not when the IOMMUs get set
> up (which is a process that assumes interrupts can already be
> requested).
>
> In effect we have
>
> lapic_suspend() -> iommu_disable_x2apic_IR() ->
> disable_intremap()/disable_qinval()
>
> after
>
> iommu_suspend() -> vtd_suspend() -> disable_qinval()
>
> but the latter tail call only when !iommu_intremap, and you
> have been running with interrupt remapping enabled, so only
> the former code path would result in qinval getting disabled,
> which is after the point of the hang.
>
> Depending on whether ATS is in use, more than one invalidation
> can be done in the processing here - could you therefore check
> whether there''s any sign of ATS use ("iommu=verbose"
should
> make you see respective messages), and if so see whether
> disabling it ("ats=off") makes a difference?
ATS does not appear to be running:

(XEN) [VT-D]dmar.c:737: Host address width 36
(XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
(XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
ffff82c3ffd57000
(XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
(XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
(XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
(XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
ffff82c3ffd56000
(XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
(XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
(XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
(XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
(XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:625:   RMRR region: base_addr ba8d5000 end_address ba8ebfff
(XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
(XEN) [VT-D]dmar.c:625:   RMRR region: base_addr bb800000 end_address bf9fffff

I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
was found.

Jan Beulich

2013-Jun-05 15:38 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>> Depending on whether ATS is in use, more than one invalidation
>> can be done in the processing here - could you therefore check
>> whether there''s any sign of ATS use ("iommu=verbose"
should
>> make you see respective messages), and if so see whether
>> disabling it ("ats=off") makes a difference?
> 
> ATS does not appear to be running:
> 
> (XEN) [VT-D]dmar.c:737: Host address width 36
> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
ffff82c3ffd57000
> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
ffff82c3ffd56000
> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr ba8d5000 end_address 
> ba8ebfff
> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr bb800000 end_address 
> bf9fffff
> 
> I would expect a line with "found ACPI_DMAR_ATSR" to be printed,
if it
> was found.
Right. So one less variable.

Jan

Ben Guthro

2013-Jun-05 20:27 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>> Depending on whether ATS is in use, more than one invalidation
>>> can be done in the processing here - could you therefore check
>>> whether there''s any sign of ATS use
("iommu=verbose" should
>>> make you see respective messages), and if so see whether
>>> disabling it ("ats=off") makes a difference?
>>
>> ATS does not appear to be running:
>>
>> (XEN) [VT-D]dmar.c:737: Host address width 36
>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
ffff82c3ffd57000
>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
ffff82c3ffd56000
>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr ba8d5000 end_address
>> ba8ebfff
>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr bb800000 end_address
>> bf9fffff
>>
>> I would expect a line with "found ACPI_DMAR_ATSR" to be
printed, if it
>> was found.
>
> Right. So one less variable.
Some more info.
Ross Philipson provided me with a handy utility to dump a bunch more
info about the DMAR tables, and with some more trace, this appears to
be tied to the IGD.

Early in the boot process, I see queue_invalidate_wait() called for
DRHD unit 0, and 1
(unit 0 is wired up to the IGD, unit 1 is everything else)

Up until i915 does the following, I see that unit being flushed with
queue_invalidate_wait() :

[    0.704537] ENERGY_PERF_BIAS: Set to ''normal'', was
''performance''
[    0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
(XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
(XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
[    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
bit banging on pin 5
[    2.253551] fbcon: inteldrmfb (fb0) is primary device
[    3.111838] Console: switching to colour frame buffer device 170x48
[    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[    3.171634] i915 0000:00:02.0: registered panic notifier
[    3.173339] acpi device:00: registered as cooling_device1
[    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
[    3.173962] input: Video Bus as
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
[    3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[    3.174258] ahci 0000:00:1f.2: version 3.0
[    3.174270] xen: registering gsi 19 triggering 0 polarity 1
[    3.174274] Already setup the GSI :19


After that - the unit never seems to be flushed.

...until we enter into the S3 hypercall, which loops over all DRHD
units, and explicitly flushes all of them via iommu_flush_all()

It is at that point that it hangs up when talking to the device that
the IGD is plumbed up to.


Does this point to something in the i915 driver doing something that
is incompatible with Xen?

Ben Guthro

2013-Jun-05 23:53 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net>
wrote:> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net>
wrote:
>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>> Depending on whether ATS is in use, more than one invalidation
>>>> can be done in the processing here - could you therefore check
>>>> whether there''s any sign of ATS use
("iommu=verbose" should
>>>> make you see respective messages), and if so see whether
>>>> disabling it ("ats=off") makes a difference?
>>>
>>> ATS does not appear to be running:
>>>
>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg
= ffff82c3ffd57000
>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg
= ffff82c3ffd56000
>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
>>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr ba8d5000
end_address
>>> ba8ebfff
>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr bb800000
end_address
>>> bf9fffff
>>>
>>> I would expect a line with "found ACPI_DMAR_ATSR" to be
printed, if it
>>> was found.
>>
>> Right. So one less variable.
>
> Some more info.
> Ross Philipson provided me with a handy utility to dump a bunch more
> info about the DMAR tables, and with some more trace, this appears to
> be tied to the IGD.
>
> Early in the boot process, I see queue_invalidate_wait() called for
> DRHD unit 0, and 1
> (unit 0 is wired up to the IGD, unit 1 is everything else)
>
> Up until i915 does the following, I see that unit being flushed with
> queue_invalidate_wait() :
>
> [    0.704537] ENERGY_PERF_BIAS: Set to ''normal'', was
''performance''
> [    0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
> bit banging on pin 5
> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
> [    3.111838] Console: switching to colour frame buffer device 170x48
> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
> [    3.171634] i915 0000:00:02.0: registered panic notifier
> [    3.173339] acpi device:00: registered as cooling_device1
> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom: no  post:
no)
> [    3.173962] input: Video Bus as
> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on
minor 0
> [    3.174258] ahci 0000:00:1f.2: version 3.0
> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
> [    3.174274] Already setup the GSI :19
>
>
> After that - the unit never seems to be flushed.
>
> ...until we enter into the S3 hypercall, which loops over all DRHD
> units, and explicitly flushes all of them via iommu_flush_all()
>
> It is at that point that it hangs up when talking to the device that
> the IGD is plumbed up to.
>
>
> Does this point to something in the i915 driver doing something that
> is incompatible with Xen?
I actually separated it from the S3 hypercall, adding a new debug key
''F'' - to just call iommu_flush_all()
I can crash it on demand with this.

Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
it does not occur.
So, that pretty much narrows it down to the IGD, in my mind.

Jan Beulich

2013-Jun-06 06:58 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net>
wrote:
>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>> Depending on whether ATS is in use, more than one
invalidation
>>>>> can be done in the processing here - could you therefore
check
>>>>> whether there''s any sign of ATS use
("iommu=verbose" should
>>>>> make you see respective messages), and if so see whether
>>>>> disabling it ("ats=off") makes a difference?
>>>>
>>>> ATS does not appear to be running:
>>>>
>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000
iommu->reg = ffff82c3ffd57000
>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000
iommu->reg = ffff82c3ffd56000
>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>>>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
>>>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
>>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr ba8d5000
end_address
>>>> ba8ebfff
>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr bb800000
end_address
>>>> bf9fffff
>>>>
>>>> I would expect a line with "found ACPI_DMAR_ATSR" to
be printed, if it
>>>> was found.
>>>
>>> Right. So one less variable.
>>
>> Some more info.
>> Ross Philipson provided me with a handy utility to dump a bunch more
>> info about the DMAR tables, and with some more trace, this appears to
>> be tied to the IGD.
>>
>> Early in the boot process, I see queue_invalidate_wait() called for
>> DRHD unit 0, and 1
>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>
>> Up until i915 does the following, I see that unit being flushed with
>> queue_invalidate_wait() :
>>
>> [    0.704537] ENERGY_PERF_BIAS: Set to ''normal'', was
''performance''
>> [    0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>> bit banging on pin 5
>> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
>> [    3.111838] Console: switching to colour frame buffer device 170x48
>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>> [    3.171634] i915 0000:00:02.0: registered panic notifier
>> [    3.173339] acpi device:00: registered as cooling_device1
>> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom: no 
post: no)
>> [    3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0
on
> minor 0
>> [    3.174258] ahci 0000:00:1f.2: version 3.0
>> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
>> [    3.174274] Already setup the GSI :19
>>
>>
>> After that - the unit never seems to be flushed.
>>
>> ...until we enter into the S3 hypercall, which loops over all DRHD
>> units, and explicitly flushes all of them via iommu_flush_all()
>>
>> It is at that point that it hangs up when talking to the device that
>> the IGD is plumbed up to.
>>
>>
>> Does this point to something in the i915 driver doing something that
>> is incompatible with Xen?
> 
> I actually separated it from the S3 hypercall, adding a new debug key
> ''F'' - to just call iommu_flush_all()
> I can crash it on demand with this.
> 
> Booting with "i915.modeset=0 single" (to prevent both KMS, and
Xorg) -
> it does not occur.
> So, that pretty much narrows it down to the IGD, in my mind.
Indeed, I agree. Yet I can''t in any way comment on what or why.
Xiantao (perhaps some graphics person would good to be Cc-ed
here too)?

Jan

Zhang, Xiantao

2013-Jun-06 15:06 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Thursday, June 06, 2013 2:59 PM
> To: Ben Guthro
> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
> 
> >>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
> > On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net>
wrote:
> >> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich
<JBeulich@suse.com> wrote:
> >>>>>> On 05.06.13 at 17:25, Ben Guthro
<ben@guthro.net> wrote:
> >>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich
<JBeulich@suse.com>
> wrote:
> >>>>> Depending on whether ATS is in use, more than one
invalidation
> >>>>> can be done in the processing here - could you
therefore check
> >>>>> whether there''s any sign of ATS use
("iommu=verbose" should
> >>>>> make you see respective messages), and if so see
whether
> >>>>> disabling it ("ats=off") makes a difference?
> >>>>
> >>>> ATS does not appear to be running:
> >>>>
> >>>> (XEN) [VT-D]dmar.c:737: Host address width 36
> >>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
> >>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000
iommu->reg > ffff82c3ffd57000
> >>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap =
f0101a
> >>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
> >>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
> >>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000
iommu->reg > ffff82c3ffd56000
> >>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap =
f0105a
> >>>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
> >>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
> >>>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
> >>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
> >>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
> >>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr ba8d5000
> end_address
> >>>> ba8ebfff
> >>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
> >>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr bb800000
> end_address
> >>>> bf9fffff
> >>>>
> >>>> I would expect a line with "found
ACPI_DMAR_ATSR" to be printed, if it
> >>>> was found.
> >>>
> >>> Right. So one less variable.
> >>
> >> Some more info.
> >> Ross Philipson provided me with a handy utility to dump a bunch
more
> >> info about the DMAR tables, and with some more trace, this appears
to
> >> be tied to the IGD.
> >>
> >> Early in the boot process, I see queue_invalidate_wait() called
for
> >> DRHD unit 0, and 1
> >> (unit 0 is wired up to the IGD, unit 1 is everything else)
> >>
> >> Up until i915 does the following, I see that unit being flushed
with
> >> queue_invalidate_wait() :
> >>
> >> [    0.704537] ENERGY_PERF_BIAS: Set to
''normal'', was ''performance''
> >> [    0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
> >> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling
back to
> >> bit banging on pin 5
> >> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
> >> [    3.111838] Console: switching to colour frame buffer device
170x48
> >> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer
device
> >> [    3.171634] i915 0000:00:02.0: registered panic notifier
> >> [    3.173339] acpi device:00: registered as cooling_device1
> >> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom: no 
post: no)
> >> [    3.173962] input: Video Bus as
> >>
> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
> >> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for
0000:00:02.0 on
> > minor 0
> >> [    3.174258] ahci 0000:00:1f.2: version 3.0
> >> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
> >> [    3.174274] Already setup the GSI :19
> >>
> >>
> >> After that - the unit never seems to be flushed.
> >>
> >> ...until we enter into the S3 hypercall, which loops over all DRHD
> >> units, and explicitly flushes all of them via iommu_flush_all()
> >>
> >> It is at that point that it hangs up when talking to the device
that
> >> the IGD is plumbed up to.
> >>
> >>
> >> Does this point to something in the i915 driver doing something
that
> >> is incompatible with Xen?
> >
> > I actually separated it from the S3 hypercall, adding a new debug key
> > ''F'' - to just call iommu_flush_all()
> > I can crash it on demand with this.
> >
> > Booting with "i915.modeset=0 single" (to prevent both KMS,
and Xorg) -
> > it does not occur.
> > So, that pretty much narrows it down to the IGD, in my mind.
> 
> Indeed, I agree. Yet I can''t in any way comment on what or why.
> Xiantao (perhaps some graphics person would good to be Cc-ed
> here too)?Hi, Jan/Ben
Thanks for your analysis! Could you try to enable  "snb_igd_quirk"  to
have a try ?  thanks!
Xiantao

Ben Guthro

2013-Jun-06 15:07 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Jun 6, 2013, at 11:06 AM, "Zhang, Xiantao"
<xiantao.zhang@intel.com> wrote:
>
>
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Thursday, June 06, 2013 2:59 PM
>> To: Ben Guthro
>> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
>> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>>
>>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net>
wrote:
>>> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net>
wrote:
>>>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>>> On 05.06.13 at 17:25, Ben Guthro
<ben@guthro.net> wrote:
>>>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich
<JBeulich@suse.com>
>> wrote:
>>>>>>> Depending on whether ATS is in use, more than one
invalidation
>>>>>>> can be done in the processing here - could you
therefore check
>>>>>>> whether there''s any sign of ATS use
("iommu=verbose" should
>>>>>>> make you see respective messages), and if so see
whether
>>>>>>> disabling it ("ats=off") makes a
difference?
>>>>>>
>>>>>> ATS does not appear to be running:
>>>>>>
>>>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed90000
>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000
iommu->reg >> ffff82c3ffd57000
>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap =
f0101a
>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address = fed91000
>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000
iommu->reg >> ffff82c3ffd56000
>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap =
f0105a
>>>>>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
>>>>>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
>>>>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr
ba8d5000
>> end_address
>>>>>> ba8ebfff
>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>>>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr
bb800000
>> end_address
>>>>>> bf9fffff
>>>>>>
>>>>>> I would expect a line with "found
ACPI_DMAR_ATSR" to be printed, if it
>>>>>> was found.
>>>>>
>>>>> Right. So one less variable.
>>>>
>>>> Some more info.
>>>> Ross Philipson provided me with a handy utility to dump a bunch
more
>>>> info about the DMAR tables, and with some more trace, this
appears to
>>>> be tied to the IGD.
>>>>
>>>> Early in the boot process, I see queue_invalidate_wait() called
for
>>>> DRHD unit 0, and 1
>>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>>
>>>> Up until i915 does the following, I see that unit being flushed
with
>>>> queue_invalidate_wait() :
>>>>
>>>> [    0.704537] ENERGY_PERF_BIAS: Set to
''normal'', was ''performance''
>>>> [    0.704537] ENERGY_PERF_BIAS: View and update with
x86_energy_p
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling
back to
>>>> bit banging on pin 5
>>>> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
>>>> [    3.111838] Console: switching to colour frame buffer device
170x48
>>>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer
device
>>>> [    3.171634] i915 0000:00:02.0: registered panic notifier
>>>> [    3.173339] acpi device:00: registered as cooling_device1
>>>> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom:
no  post: no)
>>>> [    3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>>> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for
0000:00:02.0 on
>>> minor 0
>>>> [    3.174258] ahci 0000:00:1f.2: version 3.0
>>>> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>>> [    3.174274] Already setup the GSI :19
>>>>
>>>>
>>>> After that - the unit never seems to be flushed.
>>>>
>>>> ...until we enter into the S3 hypercall, which loops over all
DRHD
>>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>>
>>>> It is at that point that it hangs up when talking to the device
that
>>>> the IGD is plumbed up to.
>>>>
>>>>
>>>> Does this point to something in the i915 driver doing something
that
>>>> is incompatible with Xen?
>>>
>>> I actually separated it from the S3 hypercall, adding a new debug
key
>>> ''F'' - to just call iommu_flush_all()
>>> I can crash it on demand with this.
>>>
>>> Booting with "i915.modeset=0 single" (to prevent both
KMS, and Xorg) -
>>> it does not occur.
>>> So, that pretty much narrows it down to the IGD, in my mind.
>>
>> Indeed, I agree. Yet I can''t in any way comment on what or
why.
>> Xiantao (perhaps some graphics person would good to be Cc-ed
>> here too)?
> Hi, Jan/Ben
> Thanks for your analysis! Could you try to enable 
"snb_igd_quirk"  to have a try ?  thanks!
> Xiantao
>

Thanks for your reply. I tried this param yesterday, but it did not
change the behavior.

Zhang, Xiantao

2013-Jun-06 15:13 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

> -----Original Message-----
> From: Ben Guthro [mailto:ben.guthro@gmail.com]
> Sent: Thursday, June 06, 2013 11:08 PM
> To: Zhang, Xiantao
> Cc: Jan Beulich; Ben Guthro; Andrew Cooper; xen-devel
> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
> 
> On Jun 6, 2013, at 11:06 AM, "Zhang, Xiantao"
<xiantao.zhang@intel.com>
> wrote:
> 
> >
> >
> >> -----Original Message-----
> >> From: Jan Beulich [mailto:JBeulich@suse.com]
> >> Sent: Thursday, June 06, 2013 2:59 PM
> >> To: Ben Guthro
> >> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
> >> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation
enabled
> >>
> >>>>> On 06.06.13 at 01:53, Ben Guthro
<ben@guthro.net> wrote:
> >>> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro
<ben@guthro.net> wrote:
> >>>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich
<JBeulich@suse.com>
> wrote:
> >>>>>>>> On 05.06.13 at 17:25, Ben Guthro
<ben@guthro.net> wrote:
> >>>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich
<JBeulich@suse.com>
> >> wrote:
> >>>>>>> Depending on whether ATS is in use, more than
one invalidation
> >>>>>>> can be done in the processing here - could you
therefore check
> >>>>>>> whether there''s any sign of ATS use
("iommu=verbose" should
> >>>>>>> make you see respective messages), and if so
see whether
> >>>>>>> disabling it ("ats=off") makes a
difference?
> >>>>>>
> >>>>>> ATS does not appear to be running:
> >>>>>>
> >>>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
> >>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address =
fed90000
> >>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address =
fed90000 iommu->reg > >> ffff82c3ffd57000
> >>>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262
ecap = f0101a
> >>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
> >>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address =
fed91000
> >>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address =
fed91000 iommu->reg > >> ffff82c3ffd56000
> >>>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262
ecap = f0105a
> >>>>>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
> >>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
> >>>>>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
> >>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
> >>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
> >>>>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr
ba8d5000
> >> end_address
> >>>>>> ba8ebfff
> >>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
> >>>>>> (XEN) [VT-D]dmar.c:625:   RMRR region: base_addr
bb800000
> >> end_address
> >>>>>> bf9fffff
> >>>>>>
> >>>>>> I would expect a line with "found
ACPI_DMAR_ATSR" to be printed, if it
> >>>>>> was found.
> >>>>>
> >>>>> Right. So one less variable.
> >>>>
> >>>> Some more info.
> >>>> Ross Philipson provided me with a handy utility to dump a
bunch more
> >>>> info about the DMAR tables, and with some more trace, this
appears to
> >>>> be tied to the IGD.
> >>>>
> >>>> Early in the boot process, I see queue_invalidate_wait()
called for
> >>>> DRHD unit 0, and 1
> >>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
> >>>>
> >>>> Up until i915 does the following, I see that unit being
flushed with
> >>>> queue_invalidate_wait() :
> >>>>
> >>>> [    0.704537] ENERGY_PERF_BIAS: Set to
''normal'', was ''performance''
> >>>> [    0.704537] ENERGY_PERF_BIAS: View and update with
x86_energy_p
> >>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >>>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out,
falling back to
> >>>> bit banging on pin 5
> >>>> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
> >>>> [    3.111838] Console: switching to colour frame buffer
device 170x48
> >>>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame
buffer device
> >>>> [    3.171634] i915 0000:00:02.0: registered panic
notifier
> >>>> [    3.173339] acpi device:00: registered as
cooling_device1
> >>>> [    3.173401] ACPI: Video Device [VID] (multi-head: yes 
rom: no  post: no)
> >>>> [    3.173962] input: Video Bus as
> >>
> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
> >>>> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for
0000:00:02.0 on
> >>> minor 0
> >>>> [    3.174258] ahci 0000:00:1f.2: version 3.0
> >>>> [    3.174270] xen: registering gsi 19 triggering 0
polarity 1
> >>>> [    3.174274] Already setup the GSI :19
> >>>>
> >>>>
> >>>> After that - the unit never seems to be flushed.
> >>>>
> >>>> ...until we enter into the S3 hypercall, which loops over
all DRHD
> >>>> units, and explicitly flushes all of them via
iommu_flush_all()
> >>>>
> >>>> It is at that point that it hangs up when talking to the
device that
> >>>> the IGD is plumbed up to.
> >>>>
> >>>>
> >>>> Does this point to something in the i915 driver doing
something that
> >>>> is incompatible with Xen?
> >>>
> >>> I actually separated it from the S3 hypercall, adding a new
debug key
> >>> ''F'' - to just call iommu_flush_all()
> >>> I can crash it on demand with this.
> >>>
> >>> Booting with "i915.modeset=0 single" (to prevent
both KMS, and Xorg) -
> >>> it does not occur.
> >>> So, that pretty much narrows it down to the IGD, in my mind.
> >>
> >> Indeed, I agree. Yet I can''t in any way comment on what
or why.
> >> Xiantao (perhaps some graphics person would good to be Cc-ed
> >> here too)?
> > Hi, Jan/Ben
> > Thanks for your analysis! Could you try to enable 
"snb_igd_quirk"  to have a
> try ?  thanks!
> > Xiantao
> >
> 
> 
> Thanks for your reply. I tried this param yesterday, but it did not
> change the behavior.Okay, I recalled one bug in IGD i915 driver is found recently, and it may bring
some errors  to VT-d,  and should be fixed in latest kernel.  Could you try
latest kernel 3.9.4 or 3.10-rcx ?
Xiantao

Ben Guthro

2013-Jun-06 15:17 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Jun 6, 2013, at 11:13 AM, "Zhang, Xiantao"
<xiantao.zhang@intel.com> wrote:
>
>
>> -----Original Message-----
>> From: Ben Guthro [mailto:ben.guthro@gmail.com]
>> Sent: Thursday, June 06, 2013 11:08 PM
>> To: Zhang, Xiantao
>> Cc: Jan Beulich; Ben Guthro; Andrew Cooper; xen-devel
>> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>>
>> On Jun 6, 2013, at 11:06 AM, "Zhang, Xiantao"
<xiantao.zhang@intel.com>
>> wrote:
>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>>> Sent: Thursday, June 06, 2013 2:59 PM
>>>> To: Ben Guthro
>>>> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
>>>> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation
enabled
>>>>
>>>>>>> On 06.06.13 at 01:53, Ben Guthro
<ben@guthro.net> wrote:
>>>>> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro
<ben@guthro.net> wrote:
>>>>>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich
<JBeulich@suse.com>
>> wrote:
>>>>>>>>>> On 05.06.13 at 17:25, Ben Guthro
<ben@guthro.net> wrote:
>>>>>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich
<JBeulich@suse.com>
>>>> wrote:
>>>>>>>>> Depending on whether ATS is in use, more
than one invalidation
>>>>>>>>> can be done in the processing here - could
you therefore check
>>>>>>>>> whether there''s any sign of ATS
use ("iommu=verbose" should
>>>>>>>>> make you see respective messages), and if
so see whether
>>>>>>>>> disabling it ("ats=off") makes a
difference?
>>>>>>>>
>>>>>>>> ATS does not appear to be running:
>>>>>>>>
>>>>>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address =
fed90000
>>>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address =
fed90000 iommu->reg >>>> ffff82c3ffd57000
>>>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262
ecap = f0101a
>>>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>>>> (XEN) [VT-D]dmar.c:412:   dmaru->address =
fed91000
>>>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address =
fed91000 iommu->reg >>>> ffff82c3ffd56000
>>>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262
ecap = f0105a
>>>>>>>> (XEN) [VT-D]dmar.c:354:  IOAPIC: 0000:f0:1f.0
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.0
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.1
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.2
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.3
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.4
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.5
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.6
>>>>>>>> (XEN) [VT-D]dmar.c:332:  MSI HPET: 0000:00:0f.7
>>>>>>>> (XEN) [VT-D]dmar.c:426:   flags: INCLUDE_ALL
>>>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1d.0
>>>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:1a.0
>>>>>>>> (XEN) [VT-D]dmar.c:625:   RMRR region:
base_addr ba8d5000
>>>> end_address
>>>>>>>> ba8ebfff
>>>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>>>> (XEN) [VT-D]dmar.c:338:  endpoint: 0000:00:02.0
>>>>>>>> (XEN) [VT-D]dmar.c:625:   RMRR region:
base_addr bb800000
>>>> end_address
>>>>>>>> bf9fffff
>>>>>>>>
>>>>>>>> I would expect a line with "found
ACPI_DMAR_ATSR" to be printed, if it
>>>>>>>> was found.
>>>>>>>
>>>>>>> Right. So one less variable.
>>>>>>
>>>>>> Some more info.
>>>>>> Ross Philipson provided me with a handy utility to dump
a bunch more
>>>>>> info about the DMAR tables, and with some more trace,
this appears to
>>>>>> be tied to the IGD.
>>>>>>
>>>>>> Early in the boot process, I see
queue_invalidate_wait() called for
>>>>>> DRHD unit 0, and 1
>>>>>> (unit 0 is wired up to the IGD, unit 1 is everything
else)
>>>>>>
>>>>>> Up until i915 does the following, I see that unit being
flushed with
>>>>>> queue_invalidate_wait() :
>>>>>>
>>>>>> [    0.704537] ENERGY_PERF_BIAS: Set to
''normal'', was ''performance''
>>>>>> [    0.704537] ENERGY_PERF_BIAS: View and update with
x86_energy_p
>>>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>>>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out,
falling back to
>>>>>> bit banging on pin 5
>>>>>> [    2.253551] fbcon: inteldrmfb (fb0) is primary
device
>>>>>> [    3.111838] Console: switching to colour frame
buffer device 170x48
>>>>>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame
buffer device
>>>>>> [    3.171634] i915 0000:00:02.0: registered panic
notifier
>>>>>> [    3.173339] acpi device:00: registered as
cooling_device1
>>>>>> [    3.173401] ACPI: Video Device [VID] (multi-head:
yes  rom: no  post: no)
>>>>>> [    3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>>>>> [    3.174232] [drm] Initialized i915 1.6.0 20080730
for 0000:00:02.0 on
>>>>> minor 0
>>>>>> [    3.174258] ahci 0000:00:1f.2: version 3.0
>>>>>> [    3.174270] xen: registering gsi 19 triggering 0
polarity 1
>>>>>> [    3.174274] Already setup the GSI :19
>>>>>>
>>>>>>
>>>>>> After that - the unit never seems to be flushed.
>>>>>>
>>>>>> ...until we enter into the S3 hypercall, which loops
over all DRHD
>>>>>> units, and explicitly flushes all of them via
iommu_flush_all()
>>>>>>
>>>>>> It is at that point that it hangs up when talking to
the device that
>>>>>> the IGD is plumbed up to.
>>>>>>
>>>>>>
>>>>>> Does this point to something in the i915 driver doing
something that
>>>>>> is incompatible with Xen?
>>>>>
>>>>> I actually separated it from the S3 hypercall, adding a new
debug key
>>>>> ''F'' - to just call iommu_flush_all()
>>>>> I can crash it on demand with this.
>>>>>
>>>>> Booting with "i915.modeset=0 single" (to prevent
both KMS, and Xorg) -
>>>>> it does not occur.
>>>>> So, that pretty much narrows it down to the IGD, in my
mind.
>>>>
>>>> Indeed, I agree. Yet I can''t in any way comment on
what or why.
>>>> Xiantao (perhaps some graphics person would good to be Cc-ed
>>>> here too)?
>>> Hi, Jan/Ben
>>> Thanks for your analysis! Could you try to enable 
"snb_igd_quirk"  to have a
>> try ?  thanks!
>>> Xiantao
>>
>>
>> Thanks for your reply. I tried this param yesterday, but it did not
>> change the behavior.
> Okay, I recalled one bug in IGD i915 driver is found recently, and it may
bring some errors  to VT-d,  and should be fixed in latest kernel.  Could you
try latest kernel 3.9.4 or 3.10-rcx ?
> Xiantao
It may have been dropped off of the top of this thread, but i sent out
what i have tested with, and this was one of them.

Testing 3.10 did not change this behavior.

Did you have a particular changeset in mind?

Zhang, Xiantao

2013-Jun-07 01:33 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

> >>>>> Booting with "i915.modeset=0 single" (to
prevent both KMS, and Xorg) -
> >>>>> it does not occur.
> >>>>> So, that pretty much narrows it down to the IGD, in my
mind.
> >>>>
> >>>> Indeed, I agree. Yet I can''t in any way comment
on what or why.
> >>>> Xiantao (perhaps some graphics person would good to be
Cc-ed
> >>>> here too)?
> >>> Hi, Jan/Ben
> >>> Thanks for your analysis! Could you try to enable 
"snb_igd_quirk"  to have
> a
> >> try ?  thanks!
> >>> Xiantao
> >>
> >>
> >> Thanks for your reply. I tried this param yesterday, but it did
not
> >> change the behavior.
> > Okay, I recalled one bug in IGD i915 driver is found recently, and it
may bring
> some errors  to VT-d,  and should be fixed in latest kernel.  Could you try
latest
> kernel 3.9.4 or 3.10-rcx ?
> > Xiantao
> 
> It may have been dropped off of the top of this thread, but i sent out
> what i have tested with, and this was one of them.
> 
> Testing 3.10 did not change this behavior.
> 
> Did you have a particular changeset in mind?I will try to find it out, and I am wondering whether your code base includes
this quirk ? See: http://www.gossamer-threads.com/lists/xen/devel/275623
Xiantao

Ben Guthro

2013-Jun-07 15:52 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Thu, Jun 6, 2013 at 9:33 PM, Zhang, Xiantao <xiantao.zhang@intel.com>
wrote:>> >>>>> Booting with "i915.modeset=0 single" (to
prevent both KMS, and Xorg) -
>> >>>>> it does not occur.
>> >>>>> So, that pretty much narrows it down to the IGD,
in my mind.
>> >>>>
>> >>>> Indeed, I agree. Yet I can''t in any way
comment on what or why.
>> >>>> Xiantao (perhaps some graphics person would good to be
Cc-ed
>> >>>> here too)?
>> >>> Hi, Jan/Ben
>> >>> Thanks for your analysis! Could you try to enable 
"snb_igd_quirk"  to have
>> a
>> >> try ?  thanks!
>> >>> Xiantao
>> >>
>> >>
>> >> Thanks for your reply. I tried this param yesterday, but it
did not
>> >> change the behavior.
>> > Okay, I recalled one bug in IGD i915 driver is found recently, and
it may bring
>> some errors  to VT-d,  and should be fixed in latest kernel.  Could you
try latest
>> kernel 3.9.4 or 3.10-rcx ?
>> > Xiantao
>>
>> It may have been dropped off of the top of this thread, but i sent out
>> what i have tested with, and this was one of them.
>>
>> Testing 3.10 did not change this behavior.
>>
>> Did you have a particular changeset in mind?
> I will try to find it out, and I am wondering whether your code base
includes this quirk ? See:
http://www.gossamer-threads.com/lists/xen/devel/275623
> Xiantao
Yes, it does.

I''ve worked around the issue, for now by disabling queue invalidation
for SNB systems.

Not ideal...but better than crashing

Jan Beulich

2013-Jun-14 08:38 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>> Early in the boot process, I see queue_invalidate_wait() called for
>> DRHD unit 0, and 1
>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>
>> Up until i915 does the following, I see that unit being flushed with
>> queue_invalidate_wait() :
>>
>> [    0.704537] ENERGY_PERF_BIAS: Set to ''normal'', was
''performance''
>> [    0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>> bit banging on pin 5
>> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
>> [    3.111838] Console: switching to colour frame buffer device 170x48
>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>> [    3.171634] i915 0000:00:02.0: registered panic notifier
>> [    3.173339] acpi device:00: registered as cooling_device1
>> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom: no 
post: no)
>> [    3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0
on minor 0
>> [    3.174258] ahci 0000:00:1f.2: version 3.0
>> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
>> [    3.174274] Already setup the GSI :19
>>
>>
>> After that - the unit never seems to be flushed.
With queue_invalidate_wait() having a single caller -
invalidate_sync() -, and with invalidate_sync() being called from
all interrupt setup (IO-APIC as well as MSI), that''s quite odd to be
the case. At least upon network driver load or interface-up, this
should be getting called.
>> ...until we enter into the S3 hypercall, which loops over all DRHD
>> units, and explicitly flushes all of them via iommu_flush_all()
>>
>> It is at that point that it hangs up when talking to the device that
>> the IGD is plumbed up to.
>>
>>
>> Does this point to something in the i915 driver doing something that
>> is incompatible with Xen?
> 
> I actually separated it from the S3 hypercall, adding a new debug key
> ''F'' - to just call iommu_flush_all()
> I can crash it on demand with this.
> 
> Booting with "i915.modeset=0 single" (to prevent both KMS, and
Xorg) -
> it does not occur.
> So, that pretty much narrows it down to the IGD, in my mind.
Which reminds me of a change I did several weeks back to our kernel,
but which isn''t as easily done with pv-ops: There are a number of
cases in the AGP and DRM code that qualify upon CONFIG_INTEL_IOMMU
and use intel_iommu_gfx_mapped. As you certainly know, Linux when
running on Xen doesn''t see any IOMMU, and hence the config option
being enabled or disabled is completely unrelated to whether the
driver actually runs on top of an enabled IOMMU. Similarly the setting
of intel_iommu_gfx_mapped cannot possibly happen when running on
top of Xen, as it sits in code that never gets used in this case.

A possibly simple, but rather hacky solution might be to always set
that variable when running on Xen. But that wouldn''t cover the case
of a kernel being built without CONFIG_INTEL_IOMMU, yet in that
case the driver might still run with an IOMMU enabled underneath.
(In our case I can simply always #define intel_iommu_gfx_mapped
to 1, with the INTEL_IOMMU option getting forcibly disabled for the
Xen kernel flavors anyway. Whether that''s entirely correct when
not running on an enabled IOMMU I can''t tell yet, and don''t
know
whom to ask.)

And that wouldn''t cover the IGD getting passed through to a DomU
at all - obviously Xen''s ability to properly drive all IOMMU operations
(including qinval) must not depend on the owning guest''s driver code.

I have to admit though that it entirely escapes me why a graphics
driver needs to peek into IOMMU code/state in the first place. This
very much smells of bad design.

Jan

Ben Guthro

2013-Jun-14 17:01 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Fri, Jun 14, 2013 at 4:38 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>>> Early in the boot process, I see queue_invalidate_wait() called for
>>> DRHD unit 0, and 1
>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>
>>> Up until i915 does the following, I see that unit being flushed
with
>>> queue_invalidate_wait() :
>>>
>>> [    0.704537] ENERGY_PERF_BIAS: Set to ''normal'',
was ''performance''
>>> [    0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back
to
>>> bit banging on pin 5
>>> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
>>> [    3.111838] Console: switching to colour frame buffer device
170x48
>>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer
device
>>> [    3.171634] i915 0000:00:02.0: registered panic notifier
>>> [    3.173339] acpi device:00: registered as cooling_device1
>>> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom: no 
post: no)
>>> [    3.173962] input: Video Bus as
>>>
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for
0000:00:02.0 on minor 0
>>> [    3.174258] ahci 0000:00:1f.2: version 3.0
>>> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>> [    3.174274] Already setup the GSI :19
>>>
>>>
>>> After that - the unit never seems to be flushed.
>
> With queue_invalidate_wait() having a single caller -
> invalidate_sync() -, and with invalidate_sync() being called from
> all interrupt setup (IO-APIC as well as MSI), that''s quite odd to
be
> the case. At least upon network driver load or interface-up, this
> should be getting called.
>
>>> ...until we enter into the S3 hypercall, which loops over all DRHD
>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>
>>> It is at that point that it hangs up when talking to the device
that
>>> the IGD is plumbed up to.
>>>
>>>
>>> Does this point to something in the i915 driver doing something
that
>>> is incompatible with Xen?
>>
>> I actually separated it from the S3 hypercall, adding a new debug key
>> ''F'' - to just call iommu_flush_all()
>> I can crash it on demand with this.
>>
>> Booting with "i915.modeset=0 single" (to prevent both KMS,
and Xorg) -
>> it does not occur.
>> So, that pretty much narrows it down to the IGD, in my mind.
>
> Which reminds me of a change I did several weeks back to our kernel,
> but which isn''t as easily done with pv-ops: There are a number of
> cases in the AGP and DRM code that qualify upon CONFIG_INTEL_IOMMU
> and use intel_iommu_gfx_mapped. As you certainly know, Linux when
> running on Xen doesn''t see any IOMMU, and hence the config option
> being enabled or disabled is completely unrelated to whether the
> driver actually runs on top of an enabled IOMMU. Similarly the setting
> of intel_iommu_gfx_mapped cannot possibly happen when running on
> top of Xen, as it sits in code that never gets used in this case.
>
> A possibly simple, but rather hacky solution might be to always set
> that variable when running on Xen. But that wouldn''t cover the
case
> of a kernel being built without CONFIG_INTEL_IOMMU, yet in that
> case the driver might still run with an IOMMU enabled underneath.
> (In our case I can simply always #define intel_iommu_gfx_mapped
> to 1, with the INTEL_IOMMU option getting forcibly disabled for the
> Xen kernel flavors anyway. Whether that''s entirely correct when
> not running on an enabled IOMMU I can''t tell yet, and
don''t know
> whom to ask.)
>
> And that wouldn''t cover the IGD getting passed through to a DomU
> at all - obviously Xen''s ability to properly drive all IOMMU
operations
> (including qinval) must not depend on the owning guest''s driver
code.
>
> I have to admit though that it entirely escapes me why a graphics
> driver needs to peek into IOMMU code/state in the first place. This
> very much smells of bad design.

This all makes sense, and I agree with your assessment.

Unfortunately, I went and got the machine back from our QA department,
to do some tests on this, and now I am unable to reproduce the issue,
to prove your analysis is correct.
It was 100% reproducible a week ago, and now I can''t make it happen,
using the same code base & build.

It is all very strange, and smells of a race condition, or
uninitialized variable.
I blame Alpha particles.

Ben Guthro

2013-Jun-14 18:27 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

On Fri, Jun 14, 2013 at 1:01 PM, Ben Guthro <ben@guthro.net>
wrote:> On Fri, Jun 14, 2013 at 4:38 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net>
wrote:
>>>> Early in the boot process, I see queue_invalidate_wait() called
for
>>>> DRHD unit 0, and 1
>>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>>
>>>> Up until i915 does the following, I see that unit being flushed
with
>>>> queue_invalidate_wait() :
>>>>
>>>> [    0.704537] ENERGY_PERF_BIAS: Set to
''normal'', was ''performance''
>>>> [    0.704537] ENERGY_PERF_BIAS: View and update with
x86_energy_p
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> [    1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling
back to
>>>> bit banging on pin 5
>>>> [    2.253551] fbcon: inteldrmfb (fb0) is primary device
>>>> [    3.111838] Console: switching to colour frame buffer device
170x48
>>>> [    3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer
device
>>>> [    3.171634] i915 0000:00:02.0: registered panic notifier
>>>> [    3.173339] acpi device:00: registered as cooling_device1
>>>> [    3.173401] ACPI: Video Device [VID] (multi-head: yes  rom:
no  post: no)
>>>> [    3.173962] input: Video Bus as
>>>>
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>>> [    3.174232] [drm] Initialized i915 1.6.0 20080730 for
0000:00:02.0 on minor 0
>>>> [    3.174258] ahci 0000:00:1f.2: version 3.0
>>>> [    3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>>> [    3.174274] Already setup the GSI :19
>>>>
>>>>
>>>> After that - the unit never seems to be flushed.
>>
>> With queue_invalidate_wait() having a single caller -
>> invalidate_sync() -, and with invalidate_sync() being called from
>> all interrupt setup (IO-APIC as well as MSI), that''s quite odd
to be
>> the case. At least upon network driver load or interface-up, this
>> should be getting called.
>>
>>>> ...until we enter into the S3 hypercall, which loops over all
DRHD
>>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>>
>>>> It is at that point that it hangs up when talking to the device
that
>>>> the IGD is plumbed up to.
>>>>
>>>>
>>>> Does this point to something in the i915 driver doing something
that
>>>> is incompatible with Xen?
>>>
>>> I actually separated it from the S3 hypercall, adding a new debug
key
>>> ''F'' - to just call iommu_flush_all()
>>> I can crash it on demand with this.
>>>
>>> Booting with "i915.modeset=0 single" (to prevent both
KMS, and Xorg) -
>>> it does not occur.
>>> So, that pretty much narrows it down to the IGD, in my mind.
>>
>> Which reminds me of a change I did several weeks back to our kernel,
>> but which isn''t as easily done with pv-ops: There are a number
of
>> cases in the AGP and DRM code that qualify upon CONFIG_INTEL_IOMMU
>> and use intel_iommu_gfx_mapped. As you certainly know, Linux when
>> running on Xen doesn''t see any IOMMU, and hence the config
option
>> being enabled or disabled is completely unrelated to whether the
>> driver actually runs on top of an enabled IOMMU. Similarly the setting
>> of intel_iommu_gfx_mapped cannot possibly happen when running on
>> top of Xen, as it sits in code that never gets used in this case.
>>
>> A possibly simple, but rather hacky solution might be to always set
>> that variable when running on Xen. But that wouldn''t cover the
case
>> of a kernel being built without CONFIG_INTEL_IOMMU, yet in that
>> case the driver might still run with an IOMMU enabled underneath.
>> (In our case I can simply always #define intel_iommu_gfx_mapped
>> to 1, with the INTEL_IOMMU option getting forcibly disabled for the
>> Xen kernel flavors anyway. Whether that''s entirely correct
when
>> not running on an enabled IOMMU I can''t tell yet, and
don''t know
>> whom to ask.)
>>
>> And that wouldn''t cover the IGD getting passed through to a
DomU
>> at all - obviously Xen''s ability to properly drive all IOMMU
operations
>> (including qinval) must not depend on the owning guest''s
driver code.
>>
>> I have to admit though that it entirely escapes me why a graphics
>> driver needs to peek into IOMMU code/state in the first place. This
>> very much smells of bad design.
>
>
> This all makes sense, and I agree with your assessment.
>
> Unfortunately, I went and got the machine back from our QA department,
> to do some tests on this, and now I am unable to reproduce the issue,
> to prove your analysis is correct.
> It was 100% reproducible a week ago, and now I can''t make it
happen,
> using the same code base & build.
>
> It is all very strange, and smells of a race condition, or
> uninitialized variable.
> I blame Alpha particles.
I did a little more bisecting of our builds, and it appears I was not
actually testing the version that I thought I was here, and once I did
some bisection, I found it got inadvertently fixed by another change
someone else committed to solve an unrelated problem.

The following changes

Revert:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c79c49826270b8b0061b2fca840fc3f013c8a78a

Apply:
https://lkml.org/lkml/2012/2/10/229

I don''t have a good explanation as to why re-enabling PAT would change
the behavior of this IOMMU feature...but I have a very reproducible
test case showing that it, in fact does.

Konrad - do you have any theories that would explain this one?
Or, would we like to leave this one as "Here be Dragons" and look the
other way?

Jan Beulich

2013-Jun-17 07:23 UTC

head link

Re: S3 crash with VTD Queue Invalidation enabled

>>> On 14.06.13 at 20:27, Ben Guthro <ben@guthro.net> wrote:
> I did a little more bisecting of our builds, and it appears I was not
> actually testing the version that I thought I was here, and once I did
> some bisection, I found it got inadvertently fixed by another change
> someone else committed to solve an unrelated problem.
> 
> The following changes
> 
> Revert:
>
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c7
> 9c49826270b8b0061b2fca840fc3f013c8a78a
> 
> Apply:
> https://lkml.org/lkml/2012/2/10/229 
> 
> I don''t have a good explanation as to why re-enabling PAT would
change
> the behavior of this IOMMU feature...but I have a very reproducible
> test case showing that it, in fact does.
Now, while this is good news in terms of knowing at least something
that addresses (or more likely works around) the issue, this still
leaves Xen at the mercy of the kernel running in the domain owning
the IGD. I.e. still a latent security issue. We really need to find a
solution that''s independent of the guest kernel.

Xiantao - we certainly will need your (Intel''s) help with this, and a
first step might be understanding how the above mentioned kernel
side changes can result in masking the observed problem.

Jan

Xen devel - Jun 2013 - S3 crash with VTD Queue Invalidation enabled

S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled

Re: S3 crash with VTD Queue Invalidation enabled