Xen devel - Mar 2008 - 32-on-64 sysenter for pvops

I implemented sysenter for 32-on-64, since it seemed straightforward 
enough.  It mostly works, but every now and again I get vcpus just 
hanging in blocked state, as if events are being lost or ignored.  Its 
very similar to the symptoms that other people have reported against the 
pvops kernel, which I have not managed to reproduce.  Perhaps using 
sysenter is exacerbating an existing bug...

Anyway, a couple of questions.  It seems that the stack frame that
Xen''s
sysenter generates is not exactly the same as the one the kernel 
expects, so the direct access to the threadinfo structure doesn''t work 
properly.  What''s the difference in the frames?

I guess the other reason for the separate PV Xen sysenter entrypoint is 
to deal with sysexit not working.  I addressed this by implementing a 
sysexit pvop using iret, though I think I could just set the TIF_IRET 
flag in threadinfo.

Anyway, could you look at these changes and see if anything problematic 
leaps out.

I''m also having debugging it, since xenctx and gdbserver-xen
don''t work
on 32-bit compat domains, and the console itself seems to locked up.  
I''m not sure how I can get any state out of the vcpus; even an eip
would
help.

Thanks,
    J




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>Anyway, a couple of questions.  It seems that the stack frame that
Xen''s
>sysenter generates is not exactly the same as the one the kernel 
>expects, so the direct access to the threadinfo structure doesn''t
work
>properly.  What''s the difference in the frames?
The frame is a normal interrupt frame (but not completely/properly filled
in - the implication of course is that the stack has been switched, other
than native sysenter would do), which is why the code in our kernels just
is a special preamble to system_call:

...
ENDPROC(ia32_sysenter_target)

	# pv sysenter call handler stub
ENTRY(ia32pv_sysenter_target)
	RING0_INT_FRAME
	movl $__USER_DS,16(%esp)
	movl %ebp,12(%esp)
	movl $__USER_CS,4(%esp)
	addl $4,%esp
	CFI_ADJUST_CFA_OFFSET -4
	/* +5*4 is SS:ESP,EFLAGS,CS:EIP. +8 is esp0 setting. */
	pushl (TI_sysenter_return-THREAD_SIZE+8+4*4)(%esp)
	CFI_ADJUST_CFA_OFFSET 4
/*
 * Load the potential sixth argument from user stack.
 * Careful about security.
 */
	cmpl $__PAGE_OFFSET-3,%ebp
	jae syscall_fault
1:	movl (%ebp),%ebp
.section __ex_table,"a"
	.align 4
	.long 1b,syscall_fault
.previous
	/* fall through */
	CFI_ENDPROC
ENDPROC(ia32pv_sysenter_target)

	# system call handler stub
ENTRY(system_call)
...
>I guess the other reason for the separate PV Xen sysenter entrypoint is 
>to deal with sysexit not working.  I addressed this by implementing a 
>sysexit pvop using iret, though I think I could just set the TIF_IRET 
>flag in threadinfo.
Either should work, but as pointed out above letting it just fall through
to system_call seems even easier.
>Anyway, could you look at these changes and see if anything problematic 
>leaps out.
This description
>The sysenter path tries to enable interrupts immediately.  Unfortunately
>this doesn''t work in a paravirt environment, because not enough
kernel
>state has been set up at that point (namely, pointing %fs to the kernel
>percpu data segment).  To fix this, defer ENABLE_INTERRUPTS until after
>the kernel state has been set up.
seems bogus: The sysenter handler in our kernels gets called with
interrupts enabled, which is as safe as int $80 going through a trap gate
(i.e. the rest of the kernel needs to be prepared to deal with interrupts
being enabled here anyway).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 4/3/08 01:06, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
> Anyway, could you look at these changes and see if anything problematic
> leaps out.
Looks okay to me. The probe for SYSENTER looks good since we promise not do
advertise CPUID.SEP to compat guests on AMD systems.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jan Beulich wrote:
>> Anyway, a couple of questions.  It seems that the stack frame that
Xen''s
>> sysenter generates is not exactly the same as the one the kernel 
>> expects, so the direct access to the threadinfo structure
doesn''t work
>> properly.  What''s the difference in the frames?
>>     
>
> The frame is a normal interrupt frame (but not completely/properly filled
> in - the implication of course is that the stack has been switched, other
> than native sysenter would do), which is why the code in our kernels just
> is a special preamble to system_call:
>   
Yes, I copied that code more or less unchanged.
> ...
> ENDPROC(ia32_sysenter_target)
>
> 	# pv sysenter call handler stub
> ENTRY(ia32pv_sysenter_target)
> 	RING0_INT_FRAME
> 	movl $__USER_DS,16(%esp)
> 	movl %ebp,12(%esp)
> 	movl $__USER_CS,4(%esp)
> 	addl $4,%esp
>   
>> I guess the other reason for the separate PV Xen sysenter entrypoint is
>> to deal with sysexit not working.  I addressed this by implementing a 
>> sysexit pvop using iret, though I think I could just set the TIF_IRET 
>> flag in threadinfo.
>>     
>
> Either should work, but as pointed out above letting it just fall through
> to system_call seems even easier.
>   
It means you need to duplicate more code.  My variant just has the 
Xen-specific stack setup on entry, but then it can just fall back to the 
normal path.
>> The sysenter path tries to enable interrupts immediately. 
Unfortunately
>> this doesn''t work in a paravirt environment, because not
enough kernel
>> state has been set up at that point (namely, pointing %fs to the kernel
>> percpu data segment).  To fix this, defer ENABLE_INTERRUPTS until after
>> the kernel state has been set up.
>>     
>
> seems bogus: The sysenter handler in our kernels gets called with
> interrupts enabled, which is as safe as int $80 going through a trap gate
> (i.e. the rest of the kernel needs to be prepared to deal with interrupts
> being enabled here anyway).
It''s a principled fix.  It''s true that there''s only a
visible problem
when making the Xen sysenter address point to the normal sysenter target 
- which doesn''t work because of the different calling convention.  But 
if it did work (ie, Xen - or another hypervisor - produced the same 
frame as the normal sysenter instruction), then ENABLE_INTERRUPTS would 
fail because it''s being called before the kernel''s percpu
segment has
been set up.

So given that ENABLE_INTERRUPTS needs to happen later, I set up 
xen_sysenter_target to enter with events masked, so that it''s as
similar
to the hardware instruction as possible, and interrupts enabled are in 
the same place in both cases.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel