thr3ads.net - Xen devel - xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [Apr 2012]

If this information is useful, please help other people find it:
Share via:

Konrad Rzeszutek Wilk

2012-Apr-30 19:37 UTC

xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

I somehow thought that this has been fixed but I''ve been
getting reports that people are running into this.

What kind of fix do I need the in the kernel? I see this:
 255         xen_cpuid(&ax, &bx, &cx, &dx);
 256 
 257         xsave_mask  258                 (1 << (X86_FEATURE_XSAVE %
32)) |
 259                 (1 << (X86_FEATURE_OSXSAVE % 32));
 260 
 261         /* Xen will set CR4.OSXSAVE if supported and not disabled by force
*/
 262         if ((cx & xsave_mask) != xsave_mask)
 263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE
& OSXSAVE */
 264 }

But do I need some other one?

Jan Beulich

2012-May-02 09:00 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> I somehow thought that this has been fixed but I''ve been
> getting reports that people are running into this.
"this" being what? I too thought that all xsave related issues were
sorted out by now.

Jan
> What kind of fix do I need the in the kernel? I see this:
>  255         xen_cpuid(&ax, &bx, &cx, &dx);
>  256 
>  257         xsave_mask >  258                 (1 <<
(X86_FEATURE_XSAVE % 32)) |
>  259                 (1 << (X86_FEATURE_OSXSAVE % 32));
>  260 
>  261         /* Xen will set CR4.OSXSAVE if supported and not disabled by 
> force */
>  262         if ((cx & xsave_mask) != xsave_mask)
>  263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable
XSAVE &
> OSXSAVE */
>  264 }
> 
> But do I need some other one?

2012-May-02 18:42 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> I somehow thought that this has been fixed but I''ve been
>> getting reports that people are running into this.
>
> "this" being what? I too thought that all xsave related issues
were
> sorted out by now.
I see the following crash if I run without xsave=0 with Ubuntu 11.10
3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don''t see this with
Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.

[    7.509303] invalid opcode: 0000 [#1] SMP
[    7.509319] CPU 0
[    7.509325] Modules linked in:
[    7.509337]
[    7.509347] Pid: 0, comm: swapper Not tainted 3.0.0-17-generic
#30-Ubuntu LENOVO 4286CTO/4286CTO
[    7.509371] RIP: e030:[<ffffffff810140ec>]  [<ffffffff810140ec>]
xstate_enable+0x3c/0x50
[    7.509399] RSP: e02b:ffffffff81c01e58  EFLAGS: 00010046
[    7.509409] RAX: 0000000000000007 RBX: ffffffff81c01e94 RCX: 0000000000000000
[    7.509419] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000002660
[    7.509430] RBP: ffffffff81c01e58 R08: ffffffff81c01e90 R09: ffffffff81c01e94
[    7.509440] R10: 00000000ffffffff R11: 00000000ffffffff R12: ffffffff81c01e90
[    7.509501] R13: ffffffff81c01e8c R14: ffffffff81c01e88 R15: ffff8801e97ecd00
[    7.509519] FS:  0000000000000000(0000) GS:ffff8801e97e0000(0000)
knlGS:0000000000000000
[    7.509530] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    7.509539] CR2: 0000000000000000 CR3: 0000000001c03000 CR4: 0000000000002660
[    7.509550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    7.509561] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    7.509573] Process swapper (pid: 0, threadinfo ffffffff81c00000,
task ffffffff81c0b020)
[    7.509582] Stack:
[    7.509589]  ffffffff81c01ec8 ffffffff81cd97ac 0000000000000040
0000000000000000
[    7.509613]  ffffffff81007b4f ffffffff81004057 0000024000000007
0000000000000340
[    7.509637]  ffff8801e97eb100 0000000000000008 0000000000000004
0000000000000000
[    7.509660] Call Trace:
[    7.509679]  [<ffffffff81cd97ac>] xstate_enable_boot_cpu+0xa9/0x174
[    7.509698]  [<ffffffff81007b4f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[    7.509712]  [<ffffffff81004057>] ? xen_write_cr0+0x77/0x90
[    7.509730]  [<ffffffff815d06eb>] xsave_init+0x26/0x28
[    7.509744]  [<ffffffff815d2932>] cpu_init+0x2bb/0x2d8
[    7.509759]  [<ffffffff81cd5ff4>] trap_init+0x169/0x171
[    7.509774]  [<ffffffff81cd0a27>] start_kernel+0x1d0/0x3df
[    7.509789]  [<ffffffff81cd0388>] x86_64_start_reservations+0x132/0x136
[    7.509805]  [<ffffffff81cd3ec6>] xen_start_kernel+0x5ac/0x5b3
[    7.509814] Code: 00 04 00 ff 14 25 10 33 c1 81 48 89 c7 48 81 cf
00 00 04 00 ff 14 25 18 33 c1 81 48 8b 05 0d 15 db 00 31 c9 48 89 c2
48 c1 ea 20 <0f> 01 d1 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00
00 55
[    7.510045] RIP  [<ffffffff810140ec>] xstate_enable+0x3c/0x50
[    7.510062]  RSP <ffffffff81c01e58>
[    7.510086] ---[ end trace a7919e7f17c0a725 ]---
[    7.510097] Kernel panic - not syncing: Attempted to kill the idle task!
[    7.510109] Pid: 0, comm: swapper Tainted: G      D
3.0.0-17-generic #30-Ubuntu
[    7.510119] Call Trace:
[    7.510134]  [<ffffffff815dca66>] panic+0x91/0x194
[    7.510151]  [<ffffffff8106344b>] do_exit+0x40b/0x440
[    7.510168]  [<ffffffff815f4350>] oops_end+0xb0/0xf0
[    7.510181]  [<ffffffff8100d938>] die+0x58/0x90
[    7.510195]  [<ffffffff815f3a34>] do_trap+0xc4/0x170
[    7.510211]  [<ffffffff8100af25>] do_invalid_op+0x95/0xb0
[    7.510226]  [<ffffffff810140ec>] ? xstate_enable+0x3c/0x50
[    7.510240]  [<ffffffff81007b62>] ? check_events+0x12/0x20
[    7.510255]  [<ffffffff81008239>] ? get_phys_to_machine+0x9/0x70
[    7.510269]  [<ffffffff81005c69>] ? pte_mfn_to_pfn+0x89/0xf0
[    7.510283]  [<ffffffff8100743d>] ? xen_force_evtchn_callback+0xd/0x10
[    7.510299]  [<ffffffff815fc2db>] invalid_op+0x1b/0x20
[    7.510315]  [<ffffffff810140ec>] ? xstate_enable+0x3c/0x50
[    7.510329]  [<ffffffff810140dc>] ? xstate_enable+0x2c/0x50
[    7.510343]  [<ffffffff81cd97ac>] xstate_enable_boot_cpu+0xa9/0x174
[    7.510358]  [<ffffffff81007b4f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[    7.510371]  [<ffffffff81004057>] ? xen_write_cr0+0x77/0x90
[    7.510385]  [<ffffffff815d06eb>] xsave_init+0x26/0x28
[    7.510399]  [<ffffffff815d2932>] cpu_init+0x2bb/0x2d8
[    7.510413]  [<ffffffff81cd5ff4>] trap_init+0x169/0x171
[    7.510427]  [<ffffffff81cd0a27>] start_kernel+0x1d0/0x3df
[    7.510441]  [<ffffffff81cd0388>] x86_64_start_reservations+0x132/0x136
[    7.510457]  [<ffffffff81cd3ec6>] xen_start_kernel+0x5ac/0x5b3
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

>> What kind of fix do I need the in the kernel? I see this:
>>  255         xen_cpuid(&ax, &bx, &cx, &dx);
>>  256
>>  257         xsave_mask >>  258                 (1 <<
(X86_FEATURE_XSAVE % 32)) |
>>  259                 (1 << (X86_FEATURE_OSXSAVE % 32));
>>  260
>>  261         /* Xen will set CR4.OSXSAVE if supported and not disabled
by
>> force */
>>  262         if ((cx & xsave_mask) != xsave_mask)
>>  263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /*
disable XSAVE &
>> OSXSAVE */
>>  264 }
>>
>> But do I need some other one?
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Jan Beulich

2012-May-03 09:15 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>>> On 02.05.12 at 20:42, AP <apxeng@gmail.com> wrote:
> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>>> I somehow thought that this has been fixed but I''ve been
>>> getting reports that people are running into this.
>>
>> "this" being what? I too thought that all xsave related
issues were
>> sorted out by now.
> 
> I see the following crash if I run without xsave=0 with Ubuntu 11.10
> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don''t see this
with
> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
And in the thread starting at
http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
I gave debugging instructions that apparently no-one followed so
far. Without someone seeing the problem doing so I don''t think we
will ever get anywhere with this (unless, as also indicated there,
someone can spot something wrong with the code that non-obvious
to everyone else).

Jan

2012-May-03 18:09 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Thu, May 3, 2012 at 2:15 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 02.05.12 at 20:42, AP <apxeng@gmail.com> wrote:
>> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>>>> I somehow thought that this has been fixed but I''ve
been
>>>> getting reports that people are running into this.
>>>
>>> "this" being what? I too thought that all xsave related
issues were
>>> sorted out by now.
>>
>> I see the following crash if I run without xsave=0 with Ubuntu 11.10
>> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don''t see this
with
>> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
>
> And in the thread starting at
> http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
> I gave debugging instructions that apparently no-one followed so
> far. Without someone seeing the problem doing so I don''t think we
> will ever get anywhere with this (unless, as also indicated there,
> someone can spot something wrong with the code that non-obvious
> to everyone else).
I missed that thread. I will add some debugging to
pv_guest_cr4_fixup() and the XSETBV handling in
emulate_privileged_op() and post the output.

Thanks,
AP

2012-May-04 19:30 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Thu, May 3, 2012 at 11:09 AM, AP <apxeng@gmail.com>
wrote:> On Thu, May 3, 2012 at 2:15 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>>>> On 02.05.12 at 20:42, AP <apxeng@gmail.com> wrote:
>>> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich
<JBeulich@suse.com> wrote:
>>>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>>>>> I somehow thought that this has been fixed but
I''ve been
>>>>> getting reports that people are running into this.
>>>>
>>>> "this" being what? I too thought that all xsave
related issues were
>>>> sorted out by now.
>>>
>>> I see the following crash if I run without xsave=0 with Ubuntu
11.10
>>> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don''t see
this with
>>> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
>>
>> And in the thread starting at
>> http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
>> I gave debugging instructions that apparently no-one followed so
>> far. Without someone seeing the problem doing so I don''t think
we
>> will ever get anywhere with this (unless, as also indicated there,
>> someone can spot something wrong with the code that non-obvious
>> to everyone else).
>
> I missed that thread. I will add some debugging to
> pv_guest_cr4_fixup() and the XSETBV handling in
> emulate_privileged_op() and post the output.
I ran with the attached xsave_debug patch and saw the following output:
<snip>
(XEN) CPU: After generic identify, caps: bfebfbff 28100800 00000000
00000000 17bae3ff 00000000 00000001 00000000
(XEN) CPU: After vendor identify, caps: bfebfbff 28100800 00000000
00000000 17bae3ff 00000000 00000001 00000000
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) CPU: After all inits, caps: bfebfbff 28100800 00000000 00003f40
17bae3ff 00000000 00000001 00000000
<snip>
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[    6.791945] invalid opcode: 0000 [#1] SMP
<snip>

From the above I realized that X86_CR4_OSXSAVE was never getting set
in v->arch.pv_vcpu.ctrlreg[4]. So I tried the following patch:

diff -r 5a0d60bb536b xen/arch/x86/domain.c
--- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/arch/x86/domain.c	Fri May 04 12:23:57 2012 -0700
@@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
         hv_cr4_mask &= ~X86_CR4_DE;
     if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
         hv_cr4_mask &= ~X86_CR4_FSGSBASE;
-    if ( xsave_enabled(v) )
-        hv_cr4_mask &= ~X86_CR4_OSXSAVE;

     if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
         gdprintk(XENLOG_WARNING,
diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
--- a/xen/include/asm-x86/domain.h	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/include/asm-x86/domain.h	Fri May 04 12:23:57 2012 -0700
@@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
      & ~X86_CR4_DE)
 #define real_cr4_to_pv_guest_cr4(c)                         \
     ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
-             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
+             | X86_CR4_SMEP))

 void domain_cpuid(struct domain *d,
                   unsigned int  input,

That allowed the system to boot successfully though I did see the
following message:
(XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660

Not sure if the above patch is right fix but I hope it was at least
helpful in pointing at where the problem might be.

BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.

Thanks,
AP


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Jan Beulich

2012-May-07 07:15 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>>> On 04.05.12 at 21:30, AP <apxeng@gmail.com> wrote:
> From the above I realized that X86_CR4_OSXSAVE was never getting set
> in v->arch.pv_vcpu.ctrlreg[4].
Yes, that was the observation in the previous thread too, but the
reporter didn''t seem interested in continuing on from there.

> So I tried the following patch:
>
> diff -r 5a0d60bb536b xen/arch/x86/domain.c
> --- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
> +++ b/xen/arch/x86/domain.c	Fri May 04 12:23:57 2012 -0700
> @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
>          hv_cr4_mask &= ~X86_CR4_DE;
>      if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
>          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> -    if ( xsave_enabled(v) )
> -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> 
>      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
>          gdprintk(XENLOG_WARNING,
> diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> --- a/xen/include/asm-x86/domain.h	Fri Apr 27 21:10:59 2012 -0700
> +++ b/xen/include/asm-x86/domain.h	Fri May 04 12:23:57 2012 -0700
> @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
>       & ~X86_CR4_DE)
>  #define real_cr4_to_pv_guest_cr4(c)                         \
>      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> +             | X86_CR4_SMEP))
> 
>  void domain_cpuid(struct domain *d,
>                    unsigned int  input,
No, this is specifically the wrong thing. From what we know so far
(i.e. the outcome of the above printing you added) the problem in
in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
attempting XSETBV). What your patch efectively does is take away
control from the guest kernels to control the (virtual) CR4 flag...
> That allowed the system to boot successfully though I did see the
> following message:
> (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
... which is what this message is telling you.
> Not sure if the above patch is right fix but I hope it was at least
> helpful in pointing at where the problem might be.
> 
> BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.
Sure, as it''s a kernel problem. It''s the kernel that needs
logging added,
to find out why the CR4 write supposedly happening immediately
prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn''t actually
happen, or doesn''t set the flag. Perhaps something fishy going on
with the paravirt ops patching, since the disassembly of the opcode
bytes shown with the oops message are indicating that the right
thing is being attempted:

ff 14 25 10 33 c1 81 	callq  *0xffffffff81c13310
48 89 c7             	mov    %rax,%rdi
48 81 cf 00 00 04 00 	or     $0x40000,%rdi
                     	       ^^^^^^^^
ff 14 25 18 33 c1 81 	callq  *0xffffffff81c13318
48 8b 05 0d 15 db 00 	mov    0xdb150d(%rip),%rax
31 c9                	xor    %ecx,%ecx
48 89 c2             	mov    %rax,%rdx
48 c1 ea 20          	shr    $0x20,%rdx
0f 01 d1             	xsetbv 
5d                   	pop    %rbp
c3                   	retq   

The primary thing that strikes me as odd is that both calls are still
indirect ones, even though I thought that they should get replaced
by direct ones (or even the actual instruction, namely in the
read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

And the dumped %rdi value indicates that bit 18 did _not_ get set.

Jan

Konrad Rzeszutek Wilk

2012-May-07 16:07 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Mon, May 07, 2012 at 08:15:44AM +0100, Jan Beulich
wrote:> >>> On 04.05.12 at 21:30, AP <apxeng@gmail.com> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
> 
> Yes, that was the observation in the previous thread too, but the
> reporter didn''t seem interested in continuing on from there.
> 
> 
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c	Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> >          hv_cr4_mask &= ~X86_CR4_DE;
> >      if ( cpu_has_fsgsbase &&
!is_pv_32bit_domain(v->domain) )
> >          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > -    if ( xsave_enabled(v) )
> > -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> > 
> >      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask)
)
> >          gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h	Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h	Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> >       & ~X86_CR4_DE)
> >  #define real_cr4_to_pv_guest_cr4(c)                         \
> >      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> > -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > +             | X86_CR4_SMEP))
> > 
> >  void domain_cpuid(struct domain *d,
> >                    unsigned int  input,
> 
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
> 
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
00002660
> 
> ... which is what this message is telling you.
> 
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> > 
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
xsave=1.
> 
> Sure, as it''s a kernel problem. It''s the kernel that
needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn''t actually
> happen, or doesn''t set the flag. Perhaps something fishy going on
> with the paravirt ops patching, since the disassembly of the opcode
> bytes shown with the oops message are indicating that the right
> thing is being attempted:
> 
> ff 14 25 10 33 c1 81 	callq  *0xffffffff81c13310 [so xen_read_cr4 which is
native_read_cr4]
> 48 89 c7             	mov    %rax,%rdi
> 48 81 cf 00 00 04 00 	or     $0x40000,%rdi
>                      	       ^^^^^^^^
> ff 14 25 18 33 c1 81 	callq  *0xffffffff81c13318 [so xen_write_cr4] - which
is filtering X86_CR4_PGE and X86_CR4_PSE]
> 48 8b 05 0d 15 db 00 	mov    0xdb150d(%rip),%rax
> 31 c9                	xor    %ecx,%ecx
> 48 89 c2             	mov    %rax,%rdx
> 48 c1 ea 20          	shr    $0x20,%rdx
> 0f 01 d1             	xsetbv 
> 5d                   	pop    %rbp
> c3                   	retq   
> 
> The primary thing that strikes me as odd is that both calls are still
> indirect ones, even though I thought that they should get replaced
> by direct ones (or even the actual instruction, namely in the
> read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?
They do get replaced (during runtime - and this is done by the
alternative_instructions). Is this output from objdump or straight
from the memory (so using the Xen debugger?).

> 
> And the dumped %rdi value indicates that bit 18 did _not_ get set.
That would imply that xen_write_cr4, which is just mov to cr4
is getting trapped but somehow the hypervisor isn''t setting the
rdi value? Or maybe the the native_write_cr4 ends up
filtering in the wrong order?

   0xffffffff8102e650 <xen_write_cr4>:  push   %rbp
   0xffffffff8102e651 <xen_write_cr4+1>:        and    $0x6f,%dil
   0xffffffff8102e655 <xen_write_cr4+5>:        mov    %rsp,%rbp
   0xffffffff8102e658 <xen_write_cr4+8>:        mov    %rdi,%cr4
   0xffffffff8102e65b <xen_write_cr4+11>:       leaveq
   0xffffffff8102e65c <xen_write_cr4+12>:       retq

> 
> Jan

Jeremy Fitzhardinge

2012-May-07 22:55 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On 05/07/2012 12:15 AM, Jan Beulich wrote:> The primary thing that strikes me as odd is that both calls are still
> indirect ones, even though I thought that they should get replaced
> by direct ones (or even the actual instruction, namely in the
> read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?
Patching happens as an explicit step during (moderately late) boot, not
as a side-effect of the first call, so it doesn''t surprise me too much
that they haven''t been updated at this point.

    J

2012-May-07 23:57 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich@suse.com>
wrote:>
> >>> On 04.05.12 at 21:30, AP <apxeng@gmail.com> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
>
> Yes, that was the observation in the previous thread too, but the
> reporter didn''t seem interested in continuing on from there.
>
>
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c   Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c   Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> >          hv_cr4_mask &= ~X86_CR4_DE;
> >      if ( cpu_has_fsgsbase &&
!is_pv_32bit_domain(v->domain) )
> >          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > -    if ( xsave_enabled(v) )
> > -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> >
> >      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask)
)
> >          gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h    Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h    Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> >       & ~X86_CR4_DE)
> >  #define real_cr4_to_pv_guest_cr4(c)                         \
> >      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> > -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > +             | X86_CR4_SMEP))
> >
> >  void domain_cpuid(struct domain *d,
> >                    unsigned int  input,
>
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
>
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
00002660
>
> ... which is what this message is telling you.
>
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> >
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > xsave=1.
>
> Sure, as it''s a kernel problem. It''s the kernel that
needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn''t actually
> happen, or doesn''t set the flag. Perhaps something fishy going on
xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

static void xen_write_cr4(unsigned long cr4)
{
    cr4 &= ~X86_CR4_PGE;
    cr4 &= ~X86_CR4_PSE;
    cr4 &= ~X86_CR4_OSXSAVE;

    native_write_cr4(cr4);
}

I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
is what I see:

<snip>
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
[    6.838041] set_in_cr4: cr4: 0x42660
[    6.841743] xen_write_cr4: cr4: 0x2660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
[    6.860546] xstate_enable: Exec xsetbv
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[    6.870509] invalid opcode: 0000 [#1] SMP
<snip>

Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.

(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    7.554262] Setting X86_CR4_OSXSAVE
[    7.557869] set_in_cr4: cr4: 0x42660
[    7.561551] xen_write_cr4: cr4: 0x42660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x42660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
[    7.580522] Exec xsetbv
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340

Thanks,
AP

Konrad Rzeszutek Wilk

2012-May-08 00:08 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

> > No, this is specifically the wrong thing. From what we know so far
> > (i.e. the outcome of the above printing you added) the problem in
> > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > attempting XSETBV). What your patch efectively does is take away
> > control from the guest kernels to control the (virtual) CR4 flag...
> >
> > > That allowed the system to boot successfully though I did see the
> > > following message:
> > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
00002660
> >
> > ... which is what this message is telling you.
> >
> > > Not sure if the above patch is right fix but I hope it was at
least
> > > helpful in pointing at where the problem might be.
> > >
> > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot
with
> > > xsave=1.
> >
> > Sure, as it''s a kernel problem. It''s the kernel that
needs logging added,
> > to find out why the CR4 write supposedly happening immediately
> > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn''t
actually
> > happen, or doesn''t set the flag. Perhaps something fishy
going on
> 
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
Where did you see that code? Looking at the Linus''s tree this is what I
see
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD

So who added that code? I am not seeing it in v3.0
either?> 
> static void xen_write_cr4(unsigned long cr4)
> {
>     cr4 &= ~X86_CR4_PGE;
>     cr4 &= ~X86_CR4_PSE;
>     cr4 &= ~X86_CR4_OSXSAVE;
> 
>     native_write_cr4(cr4);
> }
> 
> I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
> is what I see:
> 
> <snip>
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
> [    6.838041] set_in_cr4: cr4: 0x42660
> [    6.841743] xen_write_cr4: cr4: 0x2660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x2660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
> [    6.860546] xstate_enable: Exec xsetbv
> (XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4:
0x2660
> [    6.870509] invalid opcode: 0000 [#1] SMP
> <snip>
> 
> Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to
boot.
> 
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    7.554262] Setting X86_CR4_OSXSAVE
> [    7.557869] set_in_cr4: cr4: 0x42660
> [    7.561551] xen_write_cr4: cr4: 0x42660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x42660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
> [    7.580522] Exec xsetbv
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
> 
> Thanks,
> AP

2012-May-08 00:41 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:>
> > > No, this is specifically the wrong thing. From what we know so
far
> > > (i.e. the outcome of the above printing you added) the problem in
> > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > attempting XSETBV). What your patch efectively does is take away
> > > control from the guest kernels to control the (virtual) CR4
flag...
> > >
> > > > That allowed the system to boot successfully though I did
see the
> > > > following message:
> > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660
->
> > > > 00002660
> > >
> > > ... which is what this message is telling you.
> > >
> > > > Not sure if the above patch is right fix but I hope it was
at least
> > > > helpful in pointing at where the problem might be.
> > > >
> > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I
boot with
> > > > xsave=1.
> > >
> > > Sure, as it''s a kernel problem. It''s the kernel
that needs logging
> > > added,
> > > to find out why the CR4 write supposedly happening immediately
> > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE))
doesn''t actually
> > > happen, or doesn''t set the flag. Perhaps something fishy
going on
> >
> > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
>
> Where did you see that code? Looking at the Linus''s tree this is
what I
> see
>
>
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD>
> So who added that code? I am not seeing it in v3.0 either?
This is in the Ubuntu 11.10 kernel:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812

After some digging around, it looks like this is an Ubuntu 11.10 only patch:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b

Thanks,
AP


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Jan Beulich

2012-May-08 06:25 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>>> On 08.05.12 at 01:57, AP <apxeng@gmail.com> wrote:
> On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>> Sure, as it''s a kernel problem. It''s the kernel that
needs logging added,
>> to find out why the CR4 write supposedly happening immediately
>> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn''t
actually
>> happen, or doesn''t set the flag. Perhaps something fishy going
on
> 
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> 
> static void xen_write_cr4(unsigned long cr4)
> {
>     cr4 &= ~X86_CR4_PGE;
>     cr4 &= ~X86_CR4_PSE;
>     cr4 &= ~X86_CR4_OSXSAVE;
> 
>     native_write_cr4(cr4);
> }
That''s definitely not the case in _any_ upstream Linux release.
Which means that this must be introduced by a distro-specific
(and broken) patch.

Jan

Konrad Rzeszutek Wilk

2012-May-08 16:39 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:> On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> >
> > > > No, this is specifically the wrong thing. From what we know
so far
> > > > (i.e. the outcome of the above printing you added) the
problem in
> > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > attempting XSETBV). What your patch efectively does is take
away
> > > > control from the guest kernels to control the (virtual) CR4
flag...
> > > >
> > > > > That allowed the system to boot successfully though I
did see the
> > > > > following message:
> > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags
00042660 ->
> > > > > 00002660
> > > >
> > > > ... which is what this message is telling you.
> > > >
> > > > > Not sure if the above patch is right fix but I hope it
was at least
> > > > > helpful in pointing at where the problem might be.
> > > > >
> > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if
I boot with
> > > > > xsave=1.
> > > >
> > > > Sure, as it''s a kernel problem. It''s the
kernel that needs logging
> > > > added,
> > > > to find out why the CR4 write supposedly happening
immediately
> > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE))
doesn''t actually
> > > > happen, or doesn''t set the flag. Perhaps something
fishy going on
> > >
> > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> >
> > Where did you see that code? Looking at the Linus''s tree this
is what I
> > see
> >
> >
>
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> >
> > So who added that code? I am not seeing it in v3.0 either?
> 
> This is in the Ubuntu 11.10 kernel:
>
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> 
> After some digging around, it looks like this is an Ubuntu 11.10 only
patch:
>
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
Which seems to say that Amazon''s HV is advertising the OXSAVE bit?

Lets ping them and see if they have some recomendations.

Matt Wilson

2012-May-08 17:02 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Tue, May 08, 2012 at 09:39:57AM -0700, Konrad Rzeszutek Wilk
wrote:> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> > 
> > This is in the Ubuntu 11.10 kernel:
> >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > 
> > After some digging around, it looks like this is an Ubuntu 11.10 only
patch:
> >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> 
> Which seems to say that Amazon''s HV is advertising the OXSAVE bit?
> 
> Lets ping them and see if they have some recomendations.
Hi,

The kernel requirements for EC2 are documented in our User Guide here:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html#PVGRUB_compatible_kernels

Specifically, there''s a callout regarding XSAVE: 
  "Kernels that disable the pv-ops XSAVE hypercall are known to work
   on all instance types, whereas those that enable this hypercall
   will fail to launch in some cases. Similarly, non-pv-ops kernels
   that do not adhere to the Xen 3.0.2 interface might fail to launch
   in some cases."

Apologies for the miswording of the note, it should say something like
"kernels that disable the XSAVE capability" instead.

Cheers,

Matt

Konrad Rzeszutek Wilk

2012-May-09 00:35 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:> On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> >
> > > > No, this is specifically the wrong thing. From what we know
so far
> > > > (i.e. the outcome of the above printing you added) the
problem in
> > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > attempting XSETBV). What your patch efectively does is take
away
> > > > control from the guest kernels to control the (virtual) CR4
flag...
> > > >
> > > > > That allowed the system to boot successfully though I
did see the
> > > > > following message:
> > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags
00042660 ->
> > > > > 00002660
> > > >
> > > > ... which is what this message is telling you.
> > > >
> > > > > Not sure if the above patch is right fix but I hope it
was at least
> > > > > helpful in pointing at where the problem might be.
> > > > >
> > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if
I boot with
> > > > > xsave=1.
> > > >
> > > > Sure, as it''s a kernel problem. It''s the
kernel that needs logging
> > > > added,
> > > > to find out why the CR4 write supposedly happening
immediately
> > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE))
doesn''t actually
> > > > happen, or doesn''t set the flag. Perhaps something
fishy going on
> > >
> > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> >
> > Where did you see that code? Looking at the Linus''s tree this
is what I
> > see
> >
> >
>
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> >
> > So who added that code? I am not seeing it in v3.0 either?
> 
> This is in the Ubuntu 11.10 kernel:
>
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> 
> After some digging around, it looks like this is an Ubuntu 11.10 only
patch:
>
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b

Ugh. There are looks to be a bug in Fedora as well:
https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.

Konrad Rzeszutek Wilk

2012-May-09 13:11 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk
wrote:> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> > >
> > > > > No, this is specifically the wrong thing. From what we
know so far
> > > > > (i.e. the outcome of the above printing you added) the
problem in
> > > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE
prior to
> > > > > attempting XSETBV). What your patch efectively does is
take away
> > > > > control from the guest kernels to control the (virtual)
CR4 flag...
> > > > >
> > > > > > That allowed the system to boot successfully
though I did see the
> > > > > > following message:
> > > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags
00042660 ->
> > > > > > 00002660
> > > > >
> > > > > ... which is what this message is telling you.
> > > > >
> > > > > > Not sure if the above patch is right fix but I
hope it was at least
> > > > > > helpful in pointing at where the problem might be.
> > > > > >
> > > > > > BTW, I see the same invalid op issue with Xen
4.1.2 if I boot with
> > > > > > xsave=1.
> > > > >
> > > > > Sure, as it''s a kernel problem. It''s
the kernel that needs logging
> > > > > added,
> > > > > to find out why the CR4 write supposedly happening
immediately
> > > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE))
doesn''t actually
> > > > > happen, or doesn''t set the flag. Perhaps
something fishy going on
> > > >
> > > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> > >
> > > Where did you see that code? Looking at the Linus''s tree
this is what I
> > > see
> > >
> > >
> >
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> > >
> > > So who added that code? I am not seeing it in v3.0 either?
> > 
> > This is in the Ubuntu 11.10 kernel:
> >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > 
> > After some digging around, it looks like this is an Ubuntu 11.10 only
patch:
> >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> 
> 
> Ugh. There are looks to be a bug in Fedora as well:
https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> 
CC-ing Intel. It seems that the userspace programs are crashingon
Sandybridge machines even if ''xsave=0'' is provided on the
command line.
Any advice?> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Ian Campbell

2012-May-09 13:30 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Wed, 2012-05-09 at 14:11 +0100, Konrad Rzeszutek Wilk
wrote:> On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > > konrad.wilk@oracle.com> wrote:
> > > >
> > > > > > No, this is specifically the wrong thing. From
what we know so far
> > > > > > (i.e. the outcome of the above printing you added)
the problem in
> > > > > > in the Dom0 kernel (in it never setting
CR4.OSXSAVE prior to
> > > > > > attempting XSETBV). What your patch efectively
does is take away
> > > > > > control from the guest kernels to control the
(virtual) CR4 flag...
> > > > > >
> > > > > > > That allowed the system to boot successfully
though I did see the
> > > > > > > following message:
> > > > > > > (XEN) domain.c:698:d0 Attempt to change CR4
flags 00042660 ->
> > > > > > > 00002660
> > > > > >
> > > > > > ... which is what this message is telling you.
> > > > > >
> > > > > > > Not sure if the above patch is right fix but
I hope it was at least
> > > > > > > helpful in pointing at where the problem
might be.
> > > > > > >
> > > > > > > BTW, I see the same invalid op issue with Xen
4.1.2 if I boot with
> > > > > > > xsave=1.
> > > > > >
> > > > > > Sure, as it''s a kernel problem.
It''s the kernel that needs logging
> > > > > > added,
> > > > > > to find out why the CR4 write supposedly happening
immediately
> > > > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE))
doesn''t actually
> > > > > > happen, or doesn''t set the flag. Perhaps
something fishy going on
> > > > >
> > > > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> > > >
> > > > Where did you see that code? Looking at the Linus''s
tree this is what I
> > > > see
> > > >
> > > >
> > >
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> > > >
> > > > So who added that code? I am not seeing it in v3.0 either?
> > > 
> > > This is in the Ubuntu 11.10 kernel:
> > >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > > 
> > > After some digging around, it looks like this is an Ubuntu 11.10
only patch:
> > >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> > 
> > 
> > Ugh. There are looks to be a bug in Fedora as well:
https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> > 
> 
> CC-ing Intel. It seems that the userspace programs are crashingon
> Sandybridge machines even if ''xsave=0'' is provided on the
command line.
Is this related to the glibc bug which uses xsave/avx without properly
following the defined procedure to detect if it is available, as
described in http://marc.info/?l=xen-devel&m=133612371602480 ?

That bug has come up a couple of times recently, I''m not sure if this
is
the same one though.

Ian.
> Any advice?
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Jan Beulich

2012-May-09 13:34 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk
<konrad@darnok.org> wrote:
> On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
>> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
>> > After some digging around, it looks like this is an Ubuntu 11.10
only
> patch:
>> > 
>
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb
>
a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> 0b
>> 
>> 
>> Ugh. There are looks to be a bug in Fedora as well: 
> https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
>> 
> 
> CC-ing Intel. It seems that the userspace programs are crashingon
> Sandybridge machines even if ''xsave=0'' is provided on the
command line.
> Any advice?
Going through the bug, all I can see are bogus attempts to pass
"xsave=" (with value 0 or 1) to the kernel. That''s a
hypervisor
option though, and the only kernel option that''s relevant here
is "noxsave" afaik.

Further, when the hypervisor has xsave support enabled, there''s
(in the pv case) nothing the kernel can do to hide the functionality
from applications, as the hardware''s CR4.OSXSAVE will be set, and
hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
command line when "xsave=1" (or xsave enabled by default as in
4.2) just doesn''t make any sense.

And finally one always has to keep in mind that there is this nice
glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
when trying to determine whether AVX or FMA is available.

Jan

Jan

Konrad Rzeszutek Wilk

2012-May-09 17:38 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On Wed, May 09, 2012 at 02:34:46PM +0100, Jan Beulich
wrote:> >>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk
<konrad@darnok.org> wrote:
> > On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> >> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> >> > After some digging around, it looks like this is an Ubuntu
11.10 only
> > patch:
> >> > 
> >
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb
> >
a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> > 0b
> >> 
> >> 
> >> Ugh. There are looks to be a bug in Fedora as well: 
> > https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> >> 
> > 
> > CC-ing Intel. It seems that the userspace programs are crashingon
> > Sandybridge machines even if ''xsave=0'' is provided
on the command line.
> > Any advice?
> 
> Going through the bug, all I can see are bogus attempts to pass
> "xsave=" (with value 0 or 1) to the kernel. That''s a
hypervisor
> option though, and the only kernel option that''s relevant here
> is "noxsave" afaik.
> 
> Further, when the hypervisor has xsave support enabled, there''s
> (in the pv case) nothing the kernel can do to hide the functionality
> from applications, as the hardware''s CR4.OSXSAVE will be set, and
> hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
> command line when "xsave=1" (or xsave enabled by default as in
> 4.2) just doesn''t make any sense.
> 
> And finally one always has to keep in mind that there is this nice
> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
> when trying to determine whether AVX or FMA is available.
It has this:
146 
147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
148     {
149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE)
!= 0
151           && ({ unsigned int xcrlow;
152                 unsigned int xcrhigh;
153                 asm ("xgetbv"
154                      : "=a" (xcrlow), "=d" (xcrhigh) :
"c" (0));
155                 (xcrlow & 6) == 6; }))
156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
157     }


And when I ran a little silly C program (attached) to probe the CPUID flags, I
got:
 /root/test-xsave 
SSE3 CMOV
AVX  XSAVE
Trying XGETBV
Illegal instruction (core dumped)

Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV
guest
still crashes on that machine) - which means that in multi-lib the
bit_YMM_Usable
is _not_ set. But then looking at the sources I see:

# ifdef USE_AS_STRCASECMP_L
102 ENTRY(__strcasecmp)
103         .type   __strcasecmp, @gnu_indirect_function
104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
105         jne     1f
106         call    __init_cpu_features
107 1:
108 #  ifdef HAVE_AVX_SUPPORT
109         leaq    __strcasecmp_avx(%rip), %rax
110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
111         jnz     2f
112 #  endif

which would imply that the AVX bit is sampled here instead of the
YMM one.

Perhaps this is needed:

--- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig	2012-05-09
13:32:10.832000122 -0400
+++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c	2012-05-09
13:33:31.043000138 -0400
@@ -154,6 +154,8 @@ __init_cpu_features (void)
 		     : "=a" (xcrlow), "=d" (xcrhigh) : "c"
(0));
 		(xcrlow & 6) == 6; }))
 	__cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
+	else
+	__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
     }
 
   __cpu_features.family = family;

?

Jeff Law

2012-May-10 19:39 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:>>
>> And finally one always has to keep in mind that there is this nice
>> glibc bug in that in some versions it is failing to look at
CPUID.OSXSAVE
>> when trying to determine whether AVX or FMA is available.
>
> It has this:
> 146
> 147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
> 148     {
> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
> 150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& 
bit_OSXSAVE) != 0
> 151&&  ({ unsigned int xcrlow;
> 152                 unsigned int xcrhigh;
> 153                 asm ("xgetbv"
> 154                      : "=a" (xcrlow), "=d"
(xcrhigh) : "c" (0));
> 155                 (xcrlow&  6) == 6; }))
> 156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
> 157     }
>
>
> And when I ran a little silly C program (attached) to probe the CPUID
flags, I got:
>   /root/test-xsave
> SSE3 CMOV
> AVX  XSAVE
> Trying XGETBV
> Illegal instruction (core dumped)
>
> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV
guest
> still crashes on that machine) - which means that in multi-lib the
bit_YMM_Usable
> is _not_ set. But then looking at the sources I see:
>
> # ifdef USE_AS_STRCASECMP_L
> 102 ENTRY(__strcasecmp)
> 103         .type   __strcasecmp, @gnu_indirect_function
> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
> 105         jne     1f
> 106         call    __init_cpu_features
> 107 1:
> 108 #  ifdef HAVE_AVX_SUPPORT
> 109         leaq    __strcasecmp_avx(%rip), %rax
> 110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
> 111         jnz     2f
> 112 #  endif
>
> which would imply that the AVX bit is sampled here instead of the
> YMM one.[ ... ]
I think Uli''s position is that this code only uses AVX encodings, but 
not the YMM registers and thus the right check is for AVX.

That doesn''t make sense to me given the text under availability and 
support here:

http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/

According to my reading AVX can only be used if the hardware supports 
AVX *and* the OS supports XSAVE.    The only weasel language is "To use 
the Intel AVX extensions reliably in most settings ..."  Which Uli might 
be relying upon for his position.

Ironically, the code in init-arch used to look like:

  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
     {
       /* Reset the AVX bit in case OSXSAVE is disabled.  */
       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & 
bit_OSXSAVE) == 0
           || ({ unsigned int xcrlow;
               unsigned int xcrhigh;
               asm ("xgetbv"
                    : "=a" (xcrlow), "=d" (xcrhigh) :
"c" (0));
               (xcrlow & 6) != 6; }))
         __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
     }


Which I think would have done the right thing.   Uli changed it to the 
form you quoted just 2 hours after installing the version I quoted.

If i''m going to make the claim Uli is wrong, some clarification from 
Intel would be appreciated.


jeff

Jeff Law

2012-May-10 20:15 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>
> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV
guest
> still crashes on that machine) - which means that in multi-lib the
bit_YMM_Usable
> is _not_ set. But then looking at the sources I see:What''s even more amusing?  Just after installing the incorrect feature 
check and introducing bit_YMM_Usable, Uli reverted all the tests which 
had been checking for usable YMM regs and made them check AVX again.

  one.>
> Perhaps this is needed:
>
> --- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig	2012-05-09
13:32:10.832000122 -0400
> +++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c	2012-05-09
13:33:31.043000138 -0400
> @@ -154,6 +154,8 @@ __init_cpu_features (void)
>   		     : "=a" (xcrlow), "=d" (xcrhigh) :
"c" (0));
>   		(xcrlow&  6) == 6; }))
>   	__cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
> +	else
> +	__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&= ~bit_AVX;
>       }
>
>     __cpu_features.family = family;I certainly agree.  The big problem here is testing...  I still can''t 
test it :(  Against my better judgment I may have to throw a glibc with 
that change over the wall and hope.  There''s been far too much of that 
already.

jeff

Konrad Rzeszutek Wilk

2012-May-10 20:57 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On May 10, 2012 3:40 PM, "Jeff Law" <law@redhat.com>
wrote:>
> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>>
>>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at
CPUID.OSXSAVE>>> when trying to determine whether AVX or FMA is available.
>>
>>
>> It has this:
>> 146
>> 147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
>>
>> 148     {
>> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>> 150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&
 bit_OSXSAVE) != 0>> 151&&  ({ unsigned int xcrlow;
>>
>> 152                 unsigned int xcrhigh;
>> 153                 asm ("xgetbv"
>> 154                      : "=a" (xcrlow), "=d"
(xcrhigh) : "c" (0));
>> 155                 (xcrlow&  6) == 6; }))
>>
>> 156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
>> 157     }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
flags, I got:>>  /root/test-xsave
>> SSE3 CMOV
>> AVX  XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17
64-bit
PV guest>> still crashes on that machine) - which means that in multi-lib the
bit_YMM_Usable>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103         .type   __strcasecmp, @gnu_indirect_function
>> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
>> 105         jne     1f
>> 106         call    __init_cpu_features
>> 107 1:
>> 108 #  ifdef HAVE_AVX_SUPPORT
>> 109         leaq    __strcasecmp_avx(%rip), %rax
>> 110         testl   $bit_AVX,
__cpu_features+CPUID_OFFSET+index_AVX(%rip)
>> 111         jnz     2f
>> 112 #  endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
>
> [ ... ]
> I think Uli''s position is that this code only uses AVX encodings,
but not
the YMM registers and thus the right check is for AVX.>
> That doesn''t make sense to me given the text under availability
and
support here:>
>
http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/>
> According to my reading AVX can only be used if the hardware supports AVX*and* the OS supports XSAVE.    The only weasel language is "To use the
Intel AVX extensions reliably in most settings ..."  Which Uli might be
relying upon for his position.>
> Ironically, the code in init-arch used to look like:
>
>
>  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
>    {
>
>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>      if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE)
== 0>          || ({ unsigned int xcrlow;
>              unsigned int xcrhigh;
>              asm ("xgetbv"
>
>                   : "=a" (xcrlow), "=d" (xcrhigh) :
"c" (0));
>              (xcrlow & 6) != 6; }))
>
>        __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
>    }
>
>
> Which I think would have done the right thing.   Uli changed it to theform you quoted just 2 hours after installing the version I quoted.

Sadly no as it would have executed the xgetv instruction. Since the first
part of the boolean logic returns false.>
> If i''m going to make the claim Uli is wrong, some clarification
from
Intel would be appreciated.>
>
> jeff
> On May 10, 2012 3:40 PM, "Jeff Law" <law@redhat.com> wrote:
> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>
>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at
CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>>
>>
>> It has this:
>> 146
>> 147   if (__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx& 
bit_AVX)
>> 148     {
>> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>> 150       if ((__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx&
>>  bit_OSXSAVE) != 0
>> 151&&  ({ unsigned int xcrlow;
>> 152                 unsigned int xcrhigh;
>> 153                 asm ("xgetbv"
>> 154                      : "=a" (xcrlow), "=d"
(xcrhigh) : "c" (0));
>> 155                 (xcrlow&  6) == 6; }))
>> 156         __cpu_features.feature[index_**YMM_Usable] |=
bit_YMM_Usable;
>> 157     }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
>> flags, I got:
>>  /root/test-xsave
>> SSE3 CMOV
>> AVX  XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17
64-bit
>> PV guest
>> still crashes on that machine) - which means that in multi-lib the
>> bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103         .type   __strcasecmp, @gnu_indirect_function
>> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%**rip)
>> 105         jne     1f
>> 106         call    __init_cpu_features
>> 107 1:
>> 108 #  ifdef HAVE_AVX_SUPPORT
>> 109         leaq    __strcasecmp_avx(%rip), %rax
>> 110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+**
>> index_AVX(%rip)
>> 111         jnz     2f
>> 112 #  endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
>>
> [ ... ]
> I think Uli''s position is that this code only uses AVX encodings,
but not
> the YMM registers and thus the right check is for AVX.
>
> That doesn''t make sense to me given the text under availability
and
> support here:
>
> http://software.intel.com/en-**us/articles/introduction-to-**
>
intel-advanced-vector-**extensions/<http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/>
>
> According to my reading AVX can only be used if the hardware supports AVX
> *and* the OS supports XSAVE.    The only weasel language is "To use
the
> Intel AVX extensions reliably in most settings ..."  Which Uli might
be
> relying upon for his position.
>
> Ironically, the code in init-arch used to look like:
>
>  if (__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx & bit_AVX)
>    {
>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>      if ((__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx &
bit_OSXSAVE)
> == 0
>          || ({ unsigned int xcrlow;
>              unsigned int xcrhigh;
>              asm ("xgetbv"
>                   : "=a" (xcrlow), "=d" (xcrhigh) :
"c" (0));
>              (xcrlow & 6) != 6; }))
>        __cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx &= ~bit_AVX;
>    }
>
>
> Which I think would have done the right thing.   Uli changed it to the
> form you quoted just 2 hours after installing the version I quoted.
>
> If i''m going to make the claim Uli is wrong, some clarification
from Intel
> would be appreciated.
>
>
> jeff
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk

2012-May-11 00:58 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>> Ironically, the code in init-arch used to look like:
>>
>>
>>  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
>>    {
>>
>>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>>      if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &
bit_OSXSAVE) =>> 0
>>          || ({ unsigned int xcrlow;
>>              unsigned int xcrhigh;
>>              asm ("xgetbv"
>>
>>                   : "=a" (xcrlow), "=d" (xcrhigh) :
"c" (0));
>>              (xcrlow & 6) != 6; }))
>>
>>        __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
>>    }
>>
>>
>> Which I think would have done the right thing.   Uli changed it to the
>> form you quoted just 2 hours after installing the version I quoted.
>
> Sadly no as it would have executed the xgetv instruction. Since the first
> part of the boolean logic returns false.
<sigh> And that is what I get from typing this while stopping at
lights and being in a hurry and doing this on a cellphone.
Please ignore what I said above - the earlier version would have worked correct.

Jeff Law

2012-May-11 02:27 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

On 05/10/2012 06:58 PM, Konrad Rzeszutek Wilk wrote:>>> Ironically, the code in init-arch used to look like:
>>>
>>>
>>>   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
>>>     {
>>>
>>>       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>>>       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& 
bit_OSXSAVE) =>>> 0
>>>           || ({ unsigned int xcrlow;
>>>               unsigned int xcrhigh;
>>>               asm ("xgetbv"
>>>
>>>                    : "=a" (xcrlow), "=d"
(xcrhigh) : "c" (0));
>>>               (xcrlow&  6) != 6; }))
>>>
>>>         __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&=
~bit_AVX;
>>>     }
>>>
>>>
>>> Which I think would have done the right thing.   Uli changed it to
the
>>> form you quoted just 2 hours after installing the version I quoted.
>>
>> Sadly no as it would have executed the xgetv instruction. Since the
first
>> part of the boolean logic returns false.
>
> <sigh>  And that is what I get from typing this while stopping at
> lights and being in a hurry and doing this on a cellphone.
> Please ignore what I said above - the earlier version would have worked
correct.No worries :-)

Had you not gotten involved, I never would have seen the xen-devel 
thread which then referred me to the appropriate Intel doc to confirm 
that your patch should fix the problem.  Your input on this issue has 
been greatly appreciated.

jeff

Jan Beulich

2012-May-11 08:23 UTC

head link

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

>>> On 10.05.12 at 21:39, Jeff Law <law@redhat.com> wrote:
> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at
CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>
>> It has this:
>> 146
>> 147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
>> 148     {
>> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>> 150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& 
bit_OSXSAVE)
> != 0
>> 151&&  ({ unsigned int xcrlow;
>> 152                 unsigned int xcrhigh;
>> 153                 asm ("xgetbv"
>> 154                      : "=a" (xcrlow), "=d"
(xcrhigh) : "c" (0));
>> 155                 (xcrlow&  6) == 6; }))
>> 156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
>> 157     }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
flags,
> I got:
>>   /root/test-xsave
>> SSE3 CMOV
>> AVX  XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17
64-bit PV
> guest
>> still crashes on that machine) - which means that in multi-lib the 
> bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103         .type   __strcasecmp, @gnu_indirect_function
>> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
>> 105         jne     1f
>> 106         call    __init_cpu_features
>> 107 1:
>> 108 #  ifdef HAVE_AVX_SUPPORT
>> 109         leaq    __strcasecmp_avx(%rip), %rax
>> 110         testl   $bit_AVX,
__cpu_features+CPUID_OFFSET+index_AVX(%rip)
>> 111         jnz     2f
>> 112 #  endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
> [ ... ]
> I think Uli''s position is that this code only uses AVX encodings,
but
> not the YMM registers and thus the right check is for AVX.
Which is wrong: (Almost?) all VEX encoded operations act on the full
YMM registers, even if they operate only on 128 bits - the upper 128
bits get zeroed in that case.

Plus all exception type descriptions in the SDM are clearly indicating
that VEX encoded AVX instructions require CR4.OSXSAVE (mirrored
in CPUID.OSXSAVE) to be set along with the corresponding specific
CPUID feature flag(s).

Jan

Xen devel - Apr 2012 - xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.