Xen devel - Sep 2009 - Re: 32bit binaries on x86_64/Xen segfaults in syscall-vdso

On 08/30/09 11:16, Bastian Blank wrote:
> Hi folks
>
> I upgraded one of my 32bit chroots on a x86-64 machine runing under Xen
> lately. All binaries started to segfault. Some extensive checks later
> show the vdso as the culprit. Later I found
<gpe0vg$j67$1@ger.gmane.org>
> with the same problem. The full story can be found in the Debian bug
> 544145[1].
>
> It happens with Linux 2.6.30 and 2.6.31-rc8 on Xen 3.2 and 3.4.
>
> For the tests I set the vdso to compat mode to have it loaded on a fixed
> location.
>
> The following program is a minimal test case for the vdso in compat
> mode, it can be compiled against dietlibc to minimize other effects.
>   
Is this an AMD machine?  Does booting with vdso32=0 on the kernel
command line work around the problem?

     J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/03/09 15:02, Bastian Blank wrote:
> AFAIK only AMD support the syscall instruction, so yes it is an AMD
> machine. And yes, disabling the only thing that make the glibc call this
> instruction works around it.
>   
The bug actually appears to be in xen_sysret32, ie the crash happens on
the way out of the kernel.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/03/09 15:36, Bastian Blank wrote:
> This function looks weird. It tries to restores the user code segment.
> But the documentation from AMD explicitely stat that the CS and SS are
> restored from the STAR register.
And STAR is always set with:

    wrmsrl(MSR_STAR,  ((u64)__USER32_CS)<<48  |
((u64)__KERNEL_CS)<<32);

so when using sysret to return to 32-bit, it:

    The CS selector value is set to MSR IA32_STAR[63:48]. The SS is set
    to IA32_STAR[63:48] + 8.

so CS is __USER32_CS and SS is __USER32_DS.

The code for xen_sysret32 is:

ENTRY(xen_sysret32)
	/*
	 * We''re already on the usermode stack at this point, but
	 * still with the kernel gs, so we can easily switch back
	 */
	movq %rsp, PER_CPU_VAR(old_rsp)
	movq PER_CPU_VAR(kernel_stack), %rsp

	pushq $__USER32_DS
	pushq PER_CPU_VAR(old_rsp)
	pushq %r11
	pushq $__USER32_CS
	pushq %rcx

	pushq $VGCF_in_syscall
1:	jmp hypercall_iret

The iret frame is:

 	ss
 	rsp
 	rflags
 	cs
 	rip		<-- rsp

so this constructs a frame of:

	__USER32_DS
	user_esp
	user_eflags
	__USER32_CS
	user_eip	<-- kernel rsp

and then it does the iret hypercall.

But for some reason that''s triggering a failsafe callback, which
invokes
a GP.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/04/09 09:20, Bastian Blank wrote:
> On Fri, Sep 04, 2009 at 09:07:39AM -0700, Jeremy Fitzhardinge wrote:
>   
>> But for some reason that''s triggering a failsafe callback,
which invokes
>> a GP.
>>     
> Hmm, not in my tests. It always returned to userspace correctly and died
> some operations later, usually the "ret". This then produced
either a
> segfault (unreadable address), sigill (if it managed to reach the ELF
> header of the ld.so) or a GPF.
Hm, I may have misdiagnosed it then.  Your symptoms are odd; either its
landing back in userspace in the right place but then stumbles on for a
while before crashing (wrong processor mode?) or the eip is wrong and
its just landing in the wrong place and crashing immediately. 

How non-deterministic is it?  Does it differ every time, or from boot to
boot, build to build?

    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel