search for: __kernel_cs

...-decompression code loads its first GDT in startup_64, it is > still running on the CS value of the previous GDT. In the case of SEV-ES > this is the EFI GDT. > > To make exception handling work (especially IRET) the CPU needs to > switch to a CS value in the current GDT, so jump to __KERNEL_CS after > the first GDT is loaded. > > Signed-off-by: Joerg Roedel <jroedel at suse.de> > --- > arch/x86/boot/compressed/head_64.S | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S...

...suse.de> When the pre-decompression code loads its first GDT in startup_64, it is still running on the CS value of the previous GDT. In the case of SEV-ES this is the EFI GDT. To make exception handling work (especially IRET) the CPU needs to switch to a CS value in the current GDT, so jump to __KERNEL_CS after the first GDT is loaded. Signed-off-by: Joerg Roedel <jroedel at suse.de> --- arch/x86/boot/compressed/head_64.S | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 4f7e6b84be07..6b11060c3a0f 100644...

...y(int vector, void (*handler)(void)) > +{ > + unsigned long address = (unsigned long)handler; > + gate_desc entry; > + > + memset(&entry, 0, sizeof(entry)); > + > + entry.offset_low = (u16)(address & 0xffff); > + entry.segment = __KERNEL_CS; > + entry.bits.type = GATE_TRAP; ^^^ I realize we're not running a real kernel here, but GATE_TRAP is madness. Please use GATE_INTERRUPT. > + entry.bits.p = 1; > + entry.offset_middle = (u16)((address >> 16) & 0xffff); > + entry.offs...

...NEL_SEL Instead of SELECTOR == (%seg), where %seg is the live register value. This is because moves from segment registers are more costly than a single ALU instruction, and both cost a temporary register. Still, pushing the raw %cs value in the NMI after sysenter debug trap fixup code instead of __KERNEL_CS | RPL is just as efficient, since fetching RPL would require a %cs load anyway. The switch from __KERNEL_DS to __USER_DS at one point is for convenience, since they are both equivalent for %ds and %es segments; __KERNEL_DS, contrary to the name, is only useful for the %ss segment. Signed-off-by:...

...NEL_SEL Instead of SELECTOR == (%seg), where %seg is the live register value. This is because moves from segment registers are more costly than a single ALU instruction, and both cost a temporary register. Still, pushing the raw %cs value in the NMI after sysenter debug trap fixup code instead of __KERNEL_CS | RPL is just as efficient, since fetching RPL would require a %cs load anyway. The switch from __KERNEL_DS to __USER_DS at one point is for convenience, since they are both equivalent for %ds and %es segments; __KERNEL_DS, contrary to the name, is only useful for the %ss segment. Signed-off-by:...

...leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp > > + /* Setup boot GDT descriptor and load boot GDT */ > + leaq boot_gdt(%rip), %rax > + movq %rax, boot_gdt_base(%rip) > + lgdt boot_gdt_descr(%rip) > + > + /* GDT loaded - switch to __KERNEL_CS so IRET works reliably */ > + pushq $__KERNEL_CS > + leaq .Lon_kernel_cs(%rip), %rax > + pushq %rax > + lretq > + > +.Lon_kernel_cs: > + UNWIND_HINT_EMPTY I would suggest fixing at least SS as well.

Joerg Roedel <joro at 8bytes.org> writes: > + addq $8, %rsp > + > + /* > + * Make sure we return to __KERNEL_CS - the CS selector on > + * the IRET frame might still be from an old BIOS GDT > + */ > + movq $__KERNEL_CS, 8(%rsp) This doesn't make sense. Either it's running on the correct CS before the exception or not. Likely there's some other problem here that you patched over with t...

..." ((seg) << 16)); \ -} while (0) - - /* * This needs to use 'idt_table' rather than 'idt', and * thus use the _nonmapped_ version of the IDT, as the @@ -1065,7 +1051,7 @@ do { \ */ void set_intr_gate(unsigned int n, void *addr) { - _set_gate(idt_table+n,14,0,addr,__KERNEL_CS); + _set_gate(n, DESCTYPE_INT, addr, __KERNEL_CS); } /* @@ -1073,22 +1059,22 @@ void set_intr_gate(unsigned int n, void */ static inline void set_system_intr_gate(unsigned int n, void *addr) { - _set_gate(idt_table+n, 14, 3, addr, __KERNEL_CS); + _set_gate(n, DESCTYPE_INT | DESCTYPE_DPL3,...

..." ((seg) << 16)); \ -} while (0) - - /* * This needs to use 'idt_table' rather than 'idt', and * thus use the _nonmapped_ version of the IDT, as the @@ -1065,7 +1051,7 @@ do { \ */ void set_intr_gate(unsigned int n, void *addr) { - _set_gate(idt_table+n,14,0,addr,__KERNEL_CS); + _set_gate(n, DESCTYPE_INT, addr, __KERNEL_CS); } /* @@ -1073,22 +1059,22 @@ void set_intr_gate(unsigned int n, void */ static inline void set_system_intr_gate(unsigned int n, void *addr) { - _set_gate(idt_table+n, 14, 3, addr, __KERNEL_CS); + _set_gate(n, DESCTYPE_INT | DESCTYPE_DPL3,...

...sed when kernel addresses are not mapped yet */ > +SYM_DATA_LOCAL(boot_gdt_descr, .word boot_gdt_end - boot_gdt) > +SYM_DATA_LOCAL(boot_gdt_base, .quad 0) > +SYM_DATA_START(boot_gdt) > + .quad 0 > + .quad 0x00cf9a000000ffff /* __KERNEL32_CS */ > + .quad 0x00af9a000000ffff /* __KERNEL_CS */ > + .quad 0x00cf92000000ffff /* __KERNEL_DS */ > + .quad 0 /* __USER32_CS - unused */ > + .quad 0 /* __USER_DS - unused */ > + .quad 0 /* __USER_CS - unused */ > + .quad 0 /* unused */ > + .quad 0x0080890000000000 /* TSS descriptor */ > + .quad 0x0000000000...

...sed when kernel addresses are not mapped yet */ > +SYM_DATA_LOCAL(boot_gdt_descr, .word boot_gdt_end - boot_gdt) > +SYM_DATA_LOCAL(boot_gdt_base, .quad 0) > +SYM_DATA_START(boot_gdt) > + .quad 0 > + .quad 0x00cf9a000000ffff /* __KERNEL32_CS */ > + .quad 0x00af9a000000ffff /* __KERNEL_CS */ > + .quad 0x00cf92000000ffff /* __KERNEL_DS */ > + .quad 0 /* __USER32_CS - unused */ > + .quad 0 /* __USER_DS - unused */ > + .quad 0 /* __USER_CS - unused */ > + .quad 0 /* unused */ > + .quad 0x0080890000000000 /* TSS descriptor */ > + .quad 0x0000000000...

...p), %eax movl ES(%esp), %edi # get the function address movl ORIG_EAX(%esp), %edx # get the error code movl %eax, ORIG_EAX(%esp) @@ -505,12 +505,12 @@ device_not_available_emulate: * the instruction that would have done it for sysenter. */ #define FIX_STACK(offset, ok, label) \ - cmpw $__KERNEL_CS,4(%esp); \ + COMPARE_SEGMENT_STACK(__KERNEL_CS, 4); \ jne ok; \ label: \ movl TSS_sysenter_esp0+offset(%esp),%esp; \ pushfl; \ - pushl $__KERNEL_CS; \ + push %cs; \ pushl $sysenter_past_esp KPROBE_ENTRY(debug) @@ -534,10 +534,7 @@ debug_stack_correct: * fault happe...

...p), %eax movl ES(%esp), %edi # get the function address movl ORIG_EAX(%esp), %edx # get the error code movl %eax, ORIG_EAX(%esp) @@ -505,12 +505,12 @@ device_not_available_emulate: * the instruction that would have done it for sysenter. */ #define FIX_STACK(offset, ok, label) \ - cmpw $__KERNEL_CS,4(%esp); \ + COMPARE_SEGMENT_STACK(__KERNEL_CS, 4); \ jne ok; \ label: \ movl TSS_sysenter_esp0+offset(%esp),%esp; \ pushfl; \ - pushl $__KERNEL_CS; \ + push %cs; \ pushl $sysenter_past_esp KPROBE_ENTRY(debug) @@ -534,10 +534,7 @@ debug_stack_correct: * fault happe...

...RY(nmi) * resume the outer NMI. */ - movq $repeat_nmi, %rdx + leaq repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja 1f - movq $end_repeat_nmi, %rdx + leaq end_repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja nested_nmi_out 1: @@ -1440,7 +1445,8 @@ nested_nmi: pushq %rdx pushfq pushq $__KERNEL_CS - pushq $repeat_nmi + leaq repeat_nmi(%rip), %rdx + pushq %rdx /* Put stack back */ addq $(6*8), %rsp @@ -1479,7 +1485,9 @@ first_nmi: addq $8, (%rsp) /* Fix up RSP */ pushfq /* RFLAGS */ pushq $__KERNEL_CS /* CS */ - pushq $1f /* RIP */ + pushq %rax /* Support Position Independent...

.../* Set up the stack for verify_cpu(), similar to initial_stack below */ leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp + /* Setup boot GDT descriptor and load boot GDT */ + leaq boot_gdt(%rip), %rax + movq %rax, boot_gdt_base(%rip) + lgdt boot_gdt_descr(%rip) + + /* GDT loaded - switch to __KERNEL_CS so IRET works reliably */ + pushq $__KERNEL_CS + leaq .Lon_kernel_cs(%rip), %rax + pushq %rax + lretq + +.Lon_kernel_cs: + UNWIND_HINT_EMPTY + /* Sanitize CPU configuration */ call verify_cpu @@ -480,6 +494,18 @@ SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) SYM_DATA...

...+ /* Setup boot GDT descriptor and load boot GDT */ + leaq boot_gdt(%rip), %rax + movq %rax, boot_gdt_base(%rip) + lgdt boot_gdt_descr(%rip) + + /* New GDT is live - reload data segment registers */ + movl $__KERNEL_DS, %eax + movl %eax, %ds + movl %eax, %ss + movl %eax, %es + + /* Now switch to __KERNEL_CS so IRET works reliably */ + pushq $__KERNEL_CS + leaq .Lon_kernel_cs(%rip), %rax + pushq %rax + lretq + +.Lon_kernel_cs: + UNWIND_HINT_EMPTY + /* Sanitize CPU configuration */ call verify_cpu @@ -480,6 +500,22 @@ SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) SYM_DATA...

...not mapped yet */ > > +SYM_DATA_LOCAL(boot_gdt_descr, .word boot_gdt_end - boot_gdt) > > +SYM_DATA_LOCAL(boot_gdt_base, .quad 0) > > +SYM_DATA_START(boot_gdt) > > + .quad 0 > > + .quad 0x00cf9a000000ffff /* __KERNEL32_CS */ > > + .quad 0x00af9a000000ffff /* __KERNEL_CS */ > > + .quad 0x00cf92000000ffff /* __KERNEL_DS */ > > + .quad 0 /* __USER32_CS - unused */ > > + .quad 0 /* __USER_DS - unused */ > > + .quad 0 /* __USER_CS - unused */ > > + .quad 0 /* unused */ > > + .quad 0x0080890000000000 /* TSS descriptor...

...ace with LDT SS restore_nocheck: --- 2.6.18-rc3-32.orig/arch/i386/kernel/process.c +++ 2.6.18-rc3-32/arch/i386/kernel/process.c @@ -346,7 +346,7 @@ int kernel_thread(int (*fn)(void *), voi regs.xes = __USER_DS; regs.orig_eax = -1; regs.eip = (unsigned long) kernel_thread_helper; - regs.xcs = __KERNEL_CS; + regs.xcs = __KERNEL_CS | get_kernel_rpl(); regs.eflags = X86_EFLAGS_IF | X86_EFLAGS_SF | X86_EFLAGS_PF | 0x2; /* Ok, create the new process.. */ --- 2.6.18-rc3-32.orig/arch/i386/mm/extable.c +++ 2.6.18-rc3-32/arch/i386/mm/extable.c @@ -11,7 +11,7 @@ int fixup_exception(struct pt_regs *regs...

...ace with LDT SS restore_nocheck: --- 2.6.18-rc3-32.orig/arch/i386/kernel/process.c +++ 2.6.18-rc3-32/arch/i386/kernel/process.c @@ -346,7 +346,7 @@ int kernel_thread(int (*fn)(void *), voi regs.xes = __USER_DS; regs.orig_eax = -1; regs.eip = (unsigned long) kernel_thread_helper; - regs.xcs = __KERNEL_CS; + regs.xcs = __KERNEL_CS | get_kernel_rpl(); regs.eflags = X86_EFLAGS_IF | X86_EFLAGS_SF | X86_EFLAGS_PF | 0x2; /* Ok, create the new process.. */ --- 2.6.18-rc3-32.orig/arch/i386/mm/extable.c +++ 2.6.18-rc3-32/arch/i386/mm/extable.c @@ -11,7 +11,7 @@ int fixup_exception(struct pt_regs *regs...

(Andrew, please sit these in the -mm tree for cooking) Create a paravirt.h header for (almost) all the critical operations which need to be replaced with hypervisor calls. For the moment, this simply includes no_paravirt.h, where all the native implementations now live. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Signed-off-by: Zachary Amsden <zach@vmware.com>