Xen devel - May 2005 - [PATCH] Fixing stack alignment in x86-64 Xen

Long mode needs to align the stack on a 16-byte boundary. Recent changes
to Xen broke the requirement, and x86-64 XenLinux stopped booting. The
attached fixes the problem. 

Signed-off-by: Jun Nakajima <jun.nakajima@intel.com>

Jun


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, May 17, 2005 at 07:25:46PM -0700, Nakajima, Jun wrote:
> Long mode needs to align the stack on a 16-byte boundary. Recent changes
> to Xen broke the requirement, and x86-64 XenLinux stopped booting. The
> attached fixes the problem. 
Awesome!

I also needed the attached patch, which between the two of these,
makes one wonder how stuff like this is ending up in the tree. Is
code getting checked in without even being compile tested?

sRp

-- 
Scott Parish
Signed-off-by: srparish@us.ibm.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 18 May 2005, at 03:25, Nakajima, Jun wrote:
> Long mode needs to align the stack on a 16-byte boundary. Recent 
> changes
> to Xen broke the requirement, and x86-64 XenLinux stopped booting. The
> attached fixes the problem.
>
> Signed-off-by: Jun Nakajima <jun.nakajima@intel.com>
Thanks Jun, that was a nasty bug to have to find. I keep forgetting 
about that annoying automatic 16-byte alignment...

  -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Long mode needs to align the stack on a 16-byte boundary. Recent 
> > changes to Xen broke the requirement, and x86-64 XenLinux stopped 
> > booting. The attached fixes the problem.
> >
> > Signed-off-by: Jun Nakajima <jun.nakajima@intel.com>
> 
> Thanks Jun, that was a nasty bug to have to find. I keep 
> forgetting about that annoying automatic 16-byte alignment...
Let''s put an assert in there.

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

"Nakajima, Jun" <jun.nakajima@intel.com> writes:
> Long mode needs to align the stack on a 16-byte boundary. Recent changes
> to Xen broke the requirement, and x86-64 XenLinux stopped booting. The
> attached fixes the problem. 
Normally it should be only needed in user space for saving FP registers.
The kernel and Xen which should not do this probably don''t need it.

Unless you save FP registers on the stack somewhere. Then I would
rather fix that place only.

At least the main linux kernel does not try to keep the stack
always 16byte aligned. There is even a gcc option to turn it off
and it saves some code (and probably stack) size.

-Andi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Andi Kleen wrote:
> "Nakajima, Jun" <jun.nakajima@intel.com> writes:
> 
>> Long mode needs to align the stack on a 16-byte boundary. Recent
>> changes to Xen broke the requirement, and x86-64 XenLinux stopped
>> booting. The attached fixes the problem.
> 
> Normally it should be only needed in user space for saving FP
> registers. The kernel and Xen which should not do this probably
don''t
> need it. 
Normally they don''t care. The problem was that Xen set rsp0 in TSS on a
8-byte boundary and upon interrupts in Ring3 (XenLinux or user
processes) the processor started pushing registers (ss, rsp, rflags, ..,
rip) on the 16-byte boundary in Xen.  

The recent optimization reset_stack_and_jump() code needs to know the
exact address of the interrupt stack (because it resets %rsp), and
calculates it based on the value that Xen set (i.e. 8-byte boundary).
Since the processor forces the rsp0 on a 16-byte boundary (i.e. moves it
down by 8 bytes), Xen sees a wrong stack when returning from the
interrupt.

Jun
> 
> Unless you save FP registers on the stack somewhere. Then I would
> rather fix that place only.
> 
> At least the main linux kernel does not try to keep the stack
> always 16byte aligned. There is even a gcc option to turn it off
> and it saves some code (and probably stack) size.
> 
> -Andi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

"Nakajima, Jun" <jun.nakajima@intel.com>
writes:
>
> The recent optimization reset_stack_and_jump() code needs to know the
> exact address of the interrupt stack (because it resets %rsp), and
> calculates it based on the value that Xen set (i.e. 8-byte boundary).
> Since the processor forces the rsp0 on a 16-byte boundary (i.e. moves it
> down by 8 bytes), Xen sees a wrong stack when returning from the
> interrupt.
I would rather fix reset_stack_and_jump then to do the necessary
rounding or better look at the original RSP the processor stored into the
stack frame. Otherwise the 16 byte alignment will probably bite you
later again.

-Andi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Andi Kleen wrote:
> "Nakajima, Jun" <jun.nakajima@intel.com> writes:
>> 
>> The recent optimization reset_stack_and_jump() code needs to know the
>> exact address of the interrupt stack (because it resets %rsp), and
>> calculates it based on the value that Xen set (i.e. 8-byte boundary).
>> Since the processor forces the rsp0 on a 16-byte boundary (i.e.
>> moves it down by 8 bytes), Xen sees a wrong stack when returning
>> from the interrupt.
> 
> I would rather fix reset_stack_and_jump then to do the necessary
> rounding or better look at the original RSP the processor stored into
> the stack frame. Otherwise the 16 byte alignment will probably bite
> you later again.
> 
> -Andi
I think the right thing is to get rsp0 in TSS on a 16-byte boundary by
getting get_stack_bottom() and get_cpu_user_regs() see the correct
stack. That will fix the reset_stack_and_jump() as well. It''s basically
what my patch does. 

Jun

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I think the right thing is to get rsp0 in TSS on a 16-byte boundary by
> getting get_stack_bottom() and get_cpu_user_regs() see the correct
> stack. That will fix the reset_stack_and_jump() as well. It''s
basically
> what my patch does. 
This means you cannot disable the 16 byte stack alignment in gcc.
Probably does not matter too much today (I guess Xen is not that
bad a stack pig), but in the far future it might come in useful.
Also it would generate smaller code.

-Andi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Andi Kleen wrote:
>> I think the right thing is to get rsp0 in TSS on a 16-byte boundary
>> by getting get_stack_bottom() and get_cpu_user_regs() see the correct
>> stack. That will fix the reset_stack_and_jump() as well. It''s
>> basically what my patch does.
> 
> This means you cannot disable the 16 byte stack alignment in gcc.
> Probably does not matter too much today (I guess Xen is not that
> bad a stack pig), but in the far future it might come in useful.
> Also it would generate smaller code.
> 
> -Andi
I don''t think that''s correct. If you look at how they
calculate the
stack pointer, fortunately they depend only on STACK_RESERVED and the
magic number 48 (see below). It does not matter if gcc used 16 byte
stack alignment or not because the current RSP will be rounded down to
the 8KB boundary when calculate the stack pointer.
  
#define STACK_RESERVED \
    (sizeof(struct cpu_user_regs) + sizeof(struct domain *))

static inline struct cpu_user_regs *get_cpu_user_regs(void)
{
    struct cpu_user_regs *cpu_user_regs;
    __asm__( "andq %%rsp,%0; addq %2,%0"
	    : "=r" (cpu_user_regs)
	    : "0" (~(STACK_SIZE-1)), "i"
(STACK_SIZE-STACK_RESERVED) );
    return cpu_user_regs;
}

/*
 * Get the bottom-of-stack, as stored in the per-CPU TSS. This is
actually
 * 48 bytes before the real bottom of the stack to allow space for:
 * domain pointer, padding, DS, ES, FS, GS. The padding is required to
 * have the stack pointer 16-byte aligned.
 */
static inline unsigned long get_stack_bottom(void)
{
    unsigned long p;
    __asm__( "andq %%rsp,%0; addq %2,%0"
	    : "=r" (p)
	    : "0" (~(STACK_SIZE-1)), "i" (STACK_SIZE-48) );
    return p;
}

Jun

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel