Xen devel - May 2013 - Intel: GPF from lret to load CS with weird error code

Hi Intel folks,

Please help with this, the error code seems an anamoly to me:

Guest (PVH) is running in vmx in 64bit mode,  it loads CS:

ffffffff810034d2: 2:load_cs+12                   push $0x10              
ffffffff810034d4: 2:load_cs+14                   lea 0x2(%rip), %rax     
ffffffff810034db: 2:load_cs+1b                   push %rax               
ffffffff810034dc: 2:load_cs+1c                   lret                    

The lret causes a GP. But the error code is strange (0xfffc):

VMExit: intr_info=80000b0d errcode=0000fffc ilen=00000000
        reason=00000000 qualification=00000000


I can''t figure the root cause of the GP.  Reading the SDMs over and 
over, I expect either a 0 or the selector value in the errcode field. 

The GDT is properly loaded too:
(XEN) GDTR:                           limit=0x0000007f, base=0xffffffff818c2000

ffffffff818c2000:  0000000000000000 00cf9b000000ffff
ffffffff818c2010:  00af9b000000ffff 00cf93000000ffff
ffffffff818c2020:  00cffb000000ffff 00cff3000000ffff
ffffffff818c2030:  00affb000000ffff 0000000000000000

Parsing:0xaf9b000000ffff
        Type:0xb(1011) => Code segment  (C:0 R:1 A:1)
        DPL:0  P:1  AVL:0  L:1  D:0 G:1
        Base:00000000  Limit:fffff

Parsing:0xcf93000000ffff
        Type:0x3(0011) => Data segment  (E:0 W:1 A:1)
        DPL:0  P:1  AVL:0  L:0  B:1 G:1
        Base:00000000  Limit:fffff

The DS and SS selectors are also properly loaded:

(XEN) CS: sel=0x0000, attr=0x0a09b, limit=0xffffffff, base=0x0000000000000000
(XEN) DS: sel=0x0018, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) SS: sel=0x0018, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000

I understand the base/limit are ignored for the CS in VM_ENTRY_IA32E_MODE.

Running in protected mode with paging, with LMA:

EntryControls = 000053ff   <=== VM_ENTRY_IA32E_MODE
CR0: 0x0000000080010039    <===  PE TS ET NE WP PG
cr4: 0x2660  <=== PAE mode
eflags: 0x0000000000010202 <===  eflags.VM == 0

the guest EFER:
Guest EFER = 0x0000000000000000

According to the SDM, 23.3.2.1, if VM_ENTRY_LOAD_GUEST_EFER is 0, then
LMA is loaded with setting of VM_ENTRY_IA32E_MODE, which is 1 here. So I
expect to see EFER.LMA set for the guest?  Is that the problem 
here? 

Thanks,
Mukesh

>>> Mukesh Rathor <mukesh.rathor@oracle.com> 05/30/13 4:02 AM
>>>
>Guest (PVH) is running in vmx in 64bit mode,  it loads CS:
>
>ffffffff810034d2: 2:load_cs+12                   push $0x10              
>ffffffff810034d4: 2:load_cs+14                   lea 0x2(%rip), %rax     
>ffffffff810034db: 2:load_cs+1b                   push %rax               
>ffffffff810034dc: 2:load_cs+1c                   lret                    
>
>The lret causes a GP. But the error code is strange (0xfffc):
This is a strong hint at the lret lacking a REX64 override, and hence the high
32 bits
of the intended RIP being taken as target CS. lret, other than ret,
doesn''t default to
64 bit operand size.

Jan

On Thu, 30 May 2013 07:03:24 +0100
"Jan Beulich" <jbeulich@suse.com> wrote:
> >>> Mukesh Rathor <mukesh.rathor@oracle.com> 05/30/13 4:02
AM >>>
> >Guest (PVH) is running in vmx in 64bit mode,  it loads CS:
> >
> >ffffffff810034d2: 2:load_cs+12                   push
> >$0x10 ffffffff810034d4: 2:load_cs+14                   lea
> >0x2(%rip), %rax ffffffff810034db: 2:load_cs+1b
> >push %rax ffffffff810034dc: 2:load_cs+1c
> >lret                    
> >
> >The lret causes a GP. But the error code is strange (0xfffc):
> 
> This is a strong hint at the lret lacking a REX64 override, and hence
> the high 32 bits of the intended RIP being taken as target CS. lret,
> other than ret, doesn''t default to 64 bit operand size.
Grrrrr.... rats! Thats all that was and I spent so much time on it! I''d
have thought along those lines if the err code was (0xffff) which is what
it should be if it''s loading high 32bits as CS, so is still a hardware
bug IMO, or may be the ''ret'' man page should say, #GP(0),
#GP(selector),
or #GP(somewhere in between).

As always, thanks for the help. You are a great asset to xen!

Mukesh

>>> On 31.05.13 at 02:16, Mukesh Rathor
<mukesh.rathor@oracle.com> wrote:
> On Thu, 30 May 2013 07:03:24 +0100
> "Jan Beulich" <jbeulich@suse.com> wrote:
> 
>> >>> Mukesh Rathor <mukesh.rathor@oracle.com> 05/30/13
4:02 AM >>>
>> >Guest (PVH) is running in vmx in 64bit mode,  it loads CS:
>> >
>> >ffffffff810034d2: 2:load_cs+12                   push
>> >$0x10 ffffffff810034d4: 2:load_cs+14                   lea
>> >0x2(%rip), %rax ffffffff810034db: 2:load_cs+1b
>> >push %rax ffffffff810034dc: 2:load_cs+1c
>> >lret                    
>> >
>> >The lret causes a GP. But the error code is strange (0xfffc):
>> 
>> This is a strong hint at the lret lacking a REX64 override, and hence
>> the high 32 bits of the intended RIP being taken as target CS. lret,
>> other than ret, doesn''t default to 64 bit operand size.
> 
> Grrrrr.... rats! Thats all that was and I spent so much time on it!
I''d
> have thought along those lines if the err code was (0xffff) which is what
> it should be if it''s loading high 32bits as CS, so is still a
hardware
> bug IMO, or may be the ''ret'' man page should say, #GP(0),
#GP(selector),
> or #GP(somewhere in between).
That''s absolutely documented behavior - what are the RPL bits in
the selector registers have a different meaning in error codes.

Jan