search for: unwind_hint_iret_regs

...addq $8, (%rsp) /* Fix up RSP */ pushfq /* RFLAGS */ pushq $__KERNEL_CS /* CS */ - pushq $1f /* RIP */ + pushq %rax /* Support Position Independent Code */ + leaq 1f(%rip), %rax /* RIP */ + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ INTERRUPT_RETURN /* continues at repeat_nmi below */ UNWIND_HINT_IRET_REGS 1: -- 2.15.0.rc0.271.g36b669edcc-goog

...st, jump to the SEV-ES NMI IRET + * path. + */ + SEV_ES_NMI_COMPLETE + /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%r15 save_reg=%r14 @@ -1715,6 +1732,9 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ +nmi_return: + UNWIND_HINT_IRET_REGS + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1724,6 +1744,34 @@ nmi_restore: iretq SYM_CODE_END(nmi) +#ifdef CONFIG_AMD_MEM_ENCRYPT +SYM_CODE_START(sev_es_iret_user) + UNWIND_HINT_IRET_REG...

...pushfq /* RFLAGS */ > pushq $__KERNEL_CS /* CS */ > - pushq $1f /* RIP */ > + pushq %rax /* Support Position Independent Code */ > + leaq 1f(%rip), %rax /* RIP */ > + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ > INTERRUPT_RETURN /* continues at repeat_nmi below */ > UNWIND_HINT_IRET_REGS This patch seems to add extra overhead to the syscall fast-path even when PIE is disabled, right? Thanks, Ingo

...t_nmi: addq $8, (%rsp) /* Fix up RSP */ pushfq /* RFLAGS */ pushq $__KERNEL_CS /* CS */ - pushq $1f /* RIP */ + pushq %rax /* Support Position Independent Code */ + leaq 1f(%rip), %rax /* RIP */ + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ iretq /* continues at repeat_nmi below */ UNWIND_HINT_IRET_REGS 1: diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index a7227dfe1a2b..0c0fc259a4e2 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -208,11 +208,9 @@ identity_mapped: movq %rax, %cr3 lea PAGE_SIZE(%r8), %r...

...; > > + pushq %rax /* Support Position Independent Code */ > > > + leaq 1f(%rip), %rax /* RIP */ > > > + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ > > > iretq /* continues at repeat_nmi below */ > > > UNWIND_HINT_IRET_REGS > > > 1: > > > > Urgh, xchg with a memop has an implicit LOCK prefix. > this_cpu_xchg uses no lock cmpxchg as a replacement to reduce latency. Great, I will update my implementation. Thanks Peter and Christoph. > From linux/arch/x86/include/asm/percpu.h > /* &g...

On 14/03/2018 16:54, Christopher Lameter wrote: >>> + pushq %rax /* Support Position Independent Code */ >>> + leaq 1f(%rip), %rax /* RIP */ >>> + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ >>> iretq /* continues at repeat_nmi below */ >>> UNWIND_HINT_IRET_REGS >>> 1: >> Urgh, xchg with a memop has an implicit LOCK prefix. > this_cpu_xchg uses no lock cmpxchg as a replacement to reduce latency. That requires using a second register, since %rax is used as the comparison source. At this point it's easier to just push %rax twice: p...

...ndler_common) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during very early boot. The + * early_idt_handler_array can't be used because it returns via the + * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. + */ +SYM_CODE_START_NOALIGN(vc_no_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + call do_vc_no_ghcb + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN...

...ndler_common) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during very early boot. The + * early_idt_handler_array can't be used because it returns via the + * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. + */ +SYM_CODE_START_NOALIGN(vc_no_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + call do_vc_no_ghcb + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN...

.../* RIP */ >> + pushq %rax /* Support Position Independent Code */ >> + leaq 1f(%rip), %rax /* RIP */ >> + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ >> INTERRUPT_RETURN /* continues at repeat_nmi below */ >> UNWIND_HINT_IRET_REGS > > This patch seems to add extra overhead to the syscall fast-path even when PIE is > disabled, right? It does add extra instructions when one is not possible, I preferred that over ifdefing but I can change it. > > Thanks, > > Ingo -- Thomas

.../* RIP */ >> + pushq %rax /* Support Position Independent Code */ >> + leaq 1f(%rip), %rax /* RIP */ >> + xchgq %rax, (%rsp) /* Restore RAX, put 1f */ >> INTERRUPT_RETURN /* continues at repeat_nmi below */ >> UNWIND_HINT_IRET_REGS > > This patch seems to add extra overhead to the syscall fast-path even when PIE is > disabled, right? It does add extra instructions when one is not possible, I preferred that over ifdefing but I can change it. > > Thanks, > > Ingo -- Thomas

...entry stack) the #VC handler + * will switch to a fall-back stack (VC2) and call a special handler function. + * + * The macro is only used for one vector, but it is planned to extend it in the + * future for the #HV exception. + */ +.macro idtentry_vc vector asmsym cfunc +SYM_CODE_START(\asmsym) + UNWIND_HINT_IRET_REGS + ASM_CLAC + + /* + * If the entry is from userspace, switch stacks and treat it as + * a normal entry. + */ + testb $3, CS-ORIG_RAX(%rsp) + jnz .Lfrom_usermode_switch_stack_\@ + + /* + * paranoid_entry returns SWAPGS flag for paranoid_exit in EBX. + * EBX == 0 -> SWAPGS, EBX == 1 -> no...

...t when running on kernel + * addresses, but before the switch to the idt_table can be mase. The + * early_idt_handler_array can't be used here because it calls into a lot of + * __init code and this handler is also used during CPU offlining/onlining. + */ +SYM_CODE_START_NOALIGN(vc_boot_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + movq initial_vc_handler(%rip), %rax + ANNOTATE_RETPOLINE_SAFE + call *%rax + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pur...

Changes: - patch v2: - Adapt patch to work post KPTI and compiler changes - Redo all performance testing with latest configs and compilers - Simplify mov macro on PIE (MOVABS now) - Reduce GOT footprint - patch v1: - Simplify ftrace implementation. - Use gcc mstack-protector-guard-reg=%gs with PIE when possible. - rfc v3: - Use --emit-relocs instead of -pie to reduce

Changes: - patch v2: - Adapt patch to work post KPTI and compiler changes - Redo all performance testing with latest configs and compilers - Simplify mov macro on PIE (MOVABS now) - Reduce GOT footprint - patch v1: - Simplify ftrace implementation. - Use gcc mstack-protector-guard-reg=%gs with PIE when possible. - rfc v3: - Use --emit-relocs instead of -pie to reduce

From: Joerg Roedel <jroedel at suse.de> Hi, here is a new version of the SEV-ES Guest Support patches for x86. The previous versions can be found as a linked list starting here: https://lore.kernel.org/lkml/20200824085511.7553-1-joro at 8bytes.org/ I updated the patch-set based on ther review comments I got and the discussions around it. Another important change is that the early IDT

From: Joerg Roedel <jroedel at suse.de> Hi, here is a new version of the SEV-ES Guest Support patches for x86. The previous versions can be found as a linked list starting here: https://lore.kernel.org/lkml/20200824085511.7553-1-joro at 8bytes.org/ I updated the patch-set based on ther review comments I got and the discussions around it. Another important change is that the early IDT

Hi, here is the first public post of the patch-set to enable Linux to run under SEV-ES enabled hypervisors. The code is mostly feature-complete, but there are still a couple of bugs to fix. Nevertheless, given the size of the patch-set, I think it is about time to ask for initial feedback of the changes that come with it. To better understand the code here is a quick explanation of SEV-ES first.

Hi, here is the first public post of the patch-set to enable Linux to run under SEV-ES enabled hypervisors. The code is mostly feature-complete, but there are still a couple of bugs to fix. Nevertheless, given the size of the patch-set, I think it is about time to ask for initial feedback of the changes that come with it. To better understand the code here is a quick explanation of SEV-ES first.

Changes: - patch v1: - Simplify ftrace implementation. - Use gcc mstack-protector-guard-reg=%gs with PIE when possible. - rfc v3: - Use --emit-relocs instead of -pie to reduce dynamic relocation space on mapped memory. It also simplifies the relocation process. - Move the start the module section next to the kernel. Remove the need for -mcmodel=large on modules. Extends

Changes: - patch v1: - Simplify ftrace implementation. - Use gcc mstack-protector-guard-reg=%gs with PIE when possible. - rfc v3: - Use --emit-relocs instead of -pie to reduce dynamic relocation space on mapped memory. It also simplifies the relocation process. - Move the start the module section next to the kernel. Remove the need for -mcmodel=large on modules. Extends