Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 0/8] RFC: ia64/xen TAKE 2: paravirtualization of hand written assembly code
Hi. I rewrote the patch according to the comments. I adopted generating
in-place code because it looks the quickest way.
The point Eddie wanted to discuss is how to generate code and its ABI.
i.e. in-place generating v.s. direct jump v.s. indirect function call
Indirect function call doesn't make sense because ivt.S is compiled
multi times. And it is up to pv instances to choose in-place generating
or direct jump. So what we should define is its clobbered registers.
The ABI which this patch adopted is xen biased. Presumably we want to
twist those macros to make more hypervisor neutral. (i.e. add more
clobbered registers) Please comment.
TODO:
- more clean ups
some macro needs clean up to be aware register usage r8, r9.
- Free xen_ivt ares somehow. No waste kernel space
From Keith Owens idea.
Probably after defining ABI because this is just optimization.
Changes from TAKE 1:
Single IVT source code. compile multitimes using assembler macros.
Diffstat:
arch/ia64/kernel/Makefile | 32 +-
arch/ia64/kernel/entry.S | 568 +------------------
arch/ia64/kernel/inst_native.h | 153 +++++
arch/ia64/kernel/ivt.S | 159 +++---
arch/ia64/kernel/minstate.h | 2 +
arch/ia64/kernel/switch_leave.S | 609 ++++++++++++++++++++
arch/ia64/xen/inst_xen.h | 307 ++++++++++
arch/ia64/{kernel/minstate.h => xen/xenminstate.h} | 95 +---
include/asm-ia64/privop.h | 26 +
9 files changed, 1245 insertions(+), 706 deletions(-)
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 1/8] ia64/pv_ops: split out ia64_swtich_to(), ia64_leave_syscall() and ia64_leave_kernel from entry.S to switch_leave.S for paravirtualization.
note:
This patch may decrease performance because it eliminates two fall through and
one mov breg hint.
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp>
---
arch/ia64/kernel/Makefile | 2 +-
arch/ia64/kernel/entry.S | 564 +------------------------------------
arch/ia64/kernel/switch_leave.S | 594 +++++++++++++++++++++++++++++++++++++++
3 files changed, 603 insertions(+), 557 deletions(-)
create mode 100644 arch/ia64/kernel/switch_leave.S
diff --git a/arch/ia64/kernel/Makefile b/arch/ia64/kernel/Makefile
index 33e5a59..f9bc3c4 100644
--- a/arch/ia64/kernel/Makefile
+++ b/arch/ia64/kernel/Makefile
@@ -4,7 +4,7 @@
extra-y := head.o init_task.o vmlinux.lds
-obj-y := acpi.o entry.o efi.o efi_stub.o gate-data.o fsys.o ia64_ksyms.o irq.o
irq_ia64.o \
+obj-y := acpi.o entry.o switch_leave.o efi.o efi_stub.o gate-data.o fsys.o
ia64_ksyms.o irq.o irq_ia64.o \
irq_lsapic.o ivt.o machvec.o pal.o patch.o process.o perfmon.o ptrace.o sal.o
\
salinfo.o semaphore.o setup.o signal.o sys_ia64.o time.o traps.o unaligned.o
\
unwind.o mca.o mca_asm.o topology.o
diff --git a/arch/ia64/kernel/entry.S b/arch/ia64/kernel/entry.S
index 3c331c4..df8dcc9 100644
--- a/arch/ia64/kernel/entry.S
+++ b/arch/ia64/kernel/entry.S
@@ -14,15 +14,6 @@
* Copyright (C) 1999 Walt Drummond <drummond at valinux.com>
*/
/*
- * ia64_switch_to now places correct virtual mapping in in TR2 for
- * kernel stack. This allows us to handle interrupts without changing
- * to physical mode.
- *
- * Jonathan Nicklin <nicklin at missioncriticallinux.com>
- * Patrick O'Rourke <orourke at missioncriticallinux.com>
- * 11/07/2000
- */
-/*
* Global (preserved) predicate usage on syscall entry/exit path:
*
* pKStk: See entry.h.
@@ -175,68 +166,6 @@ GLOBAL_ENTRY(sys_clone)
END(sys_clone)
/*
- * prev_task <- ia64_switch_to(struct task_struct *next)
- * With Ingo's new scheduler, interrupts are disabled when this routine
gets
- * called. The code starting at .map relies on this. The rest of the code
- * doesn't care about the interrupt masking status.
- */
-GLOBAL_ENTRY(ia64_switch_to)
- .prologue
- alloc r16=ar.pfs,1,0,0,0
- DO_SAVE_SWITCH_STACK
- .body
-
- adds r22=IA64_TASK_THREAD_KSP_OFFSET,r13
- movl r25=init_task
- mov r27=IA64_KR(CURRENT_STACK)
- adds r21=IA64_TASK_THREAD_KSP_OFFSET,in0
- dep r20=0,in0,61,3 // physical address of "next"
- ;;
- st8 [r22]=sp // save kernel stack pointer of old task
- shr.u r26=r20,IA64_GRANULE_SHIFT
- cmp.eq p7,p6=r25,in0
- ;;
- /*
- * If we've already mapped this task's page, we can skip doing it
again.
- */
-(p6) cmp.eq p7,p6=r26,r27
-(p6) br.cond.dpnt .map
- ;;
-.done:
- ld8 sp=[r21] // load kernel stack pointer of new task
- mov IA64_KR(CURRENT)=in0 // update "current" application register
- mov r8=r13 // return pointer to previously running task
- mov r13=in0 // set "current" pointer
- ;;
- DO_LOAD_SWITCH_STACK
-
-#ifdef CONFIG_SMP
- sync.i // ensure "fc"s done by this CPU are visible on other CPUs
-#endif
- br.ret.sptk.many rp // boogie on out in new context
-
-.map:
- rsm psr.ic // interrupts (psr.i) are already disabled here
- movl r25=PAGE_KERNEL
- ;;
- srlz.d
- or r23=r25,r20 // construct PA | page properties
- mov r25=IA64_GRANULE_SHIFT<<2
- ;;
- mov cr.itir=r25
- mov cr.ifa=in0 // VA of next task...
- ;;
- mov r25=IA64_TR_CURRENT_STACK
- mov IA64_KR(CURRENT_STACK)=r26 // remember last page we mapped...
- ;;
- itr.d dtr[r25]=r23 // wire in new mapping...
- ssm psr.ic // reenable the psr.ic bit
- ;;
- srlz.d
- br.cond.sptk .done
-END(ia64_switch_to)
-
-/*
* Note that interrupts are enabled during save_switch_stack and
load_switch_stack. This
* means that we may get an interrupt with "sp" pointing to the new
kernel stack while
* ar.bspstore is still pointing to the old kernel backing store area. Since
ar.rsc,
@@ -570,7 +499,7 @@ GLOBAL_ENTRY(ia64_trace_syscall)
br.call.sptk.many rp=syscall_trace_leave // give parent a chance to catch
return value
.ret3:
(pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk
- br.cond.sptk .work_pending_syscall_end
+ br.cond.sptk ia64_work_pending_syscall_end
strace_error:
ld8 r3=[r2] // load pt_regs.r8
@@ -635,160 +564,10 @@ GLOBAL_ENTRY(ia64_ret_from_syscall)
adds r2=PT(R8)+16,sp // r2 = &pt_regs.r8
mov r10=r0 // clear error indication in r10
(p7) br.cond.spnt handle_syscall_error // handle potential syscall failure
-END(ia64_ret_from_syscall)
- // fall through
-/*
- * ia64_leave_syscall(): Same as ia64_leave_kernel, except that it doesn't
- * need to switch to bank 0 and doesn't restore the scratch registers.
- * To avoid leaking kernel bits, the scratch registers are set to
- * the following known-to-be-safe values:
- *
- * r1: restored (global pointer)
- * r2: cleared
- * r3: 1 (when returning to user-level)
- * r8-r11: restored (syscall return value(s))
- * r12: restored (user-level stack pointer)
- * r13: restored (user-level thread pointer)
- * r14: set to __kernel_syscall_via_epc
- * r15: restored (syscall #)
- * r16-r17: cleared
- * r18: user-level b6
- * r19: cleared
- * r20: user-level ar.fpsr
- * r21: user-level b0
- * r22: cleared
- * r23: user-level ar.bspstore
- * r24: user-level ar.rnat
- * r25: user-level ar.unat
- * r26: user-level ar.pfs
- * r27: user-level ar.rsc
- * r28: user-level ip
- * r29: user-level psr
- * r30: user-level cfm
- * r31: user-level pr
- * f6-f11: cleared
- * pr: restored (user-level pr)
- * b0: restored (user-level rp)
- * b6: restored
- * b7: set to __kernel_syscall_via_epc
- * ar.unat: restored (user-level ar.unat)
- * ar.pfs: restored (user-level ar.pfs)
- * ar.rsc: restored (user-level ar.rsc)
- * ar.rnat: restored (user-level ar.rnat)
- * ar.bspstore: restored (user-level ar.bspstore)
- * ar.fpsr: restored (user-level ar.fpsr)
- * ar.ccv: cleared
- * ar.csd: cleared
- * ar.ssd: cleared
- */
-ENTRY(ia64_leave_syscall)
- PT_REGS_UNWIND_INFO(0)
- /*
- * work.need_resched etc. mustn't get changed by this CPU before it
returns to
- * user- or fsys-mode, hence we disable interrupts early on.
- *
- * p6 controls whether current_thread_info()->flags needs to be check for
- * extra work. We always check for extra work when returning to user-level.
- * With CONFIG_PREEMPT, we also check for extra work when the preempt_count
- * is 0. After extra work processing has been completed, execution
- * resumes at .work_processed_syscall with p6 set to 1 if the extra-work-check
- * needs to be redone.
- */
-#ifdef CONFIG_PREEMPT
- rsm psr.i // disable interrupts
- cmp.eq pLvSys,p0=r0,r0 // pLvSys=1: leave from syscall
-(pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13
- ;;
- .pred.rel.mutex pUStk,pKStk
-(pKStk) ld4 r21=[r20] // r21 <- preempt_count
-(pUStk) mov r21=0 // r21 <- 0
- ;;
- cmp.eq p6,p0=r21,r0 // p6 <- pUStk || (preempt_count == 0)
-#else /* !CONFIG_PREEMPT */
-(pUStk) rsm psr.i
- cmp.eq pLvSys,p0=r0,r0 // pLvSys=1: leave from syscall
-(pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk
-#endif
-.work_processed_syscall:
- adds r2=PT(LOADRS)+16,r12
- adds r3=PT(AR_BSPSTORE)+16,r12
- adds r18=TI_FLAGS+IA64_TASK_SIZE,r13
;;
-(p6) ld4 r31=[r18] // load current_thread_info()->flags
- ld8 r19=[r2],PT(B6)-PT(LOADRS) // load ar.rsc value for "loadrs"
- nop.i 0
- ;;
- mov r16=ar.bsp // M2 get existing backing store pointer
- ld8 r18=[r2],PT(R9)-PT(B6) // load b6
-(p6) and r15=TIF_WORK_MASK,r31 // any work other than TIF_SYSCALL_TRACE?
- ;;
- ld8 r23=[r3],PT(R11)-PT(AR_BSPSTORE) // load ar.bspstore (may be garbage)
-(p6) cmp4.ne.unc p6,p0=r15, r0 // any special work pending?
-(p6) br.cond.spnt .work_pending_syscall
- ;;
- // start restoring the state saved on the kernel stack (struct pt_regs):
- ld8 r9=[r2],PT(CR_IPSR)-PT(R9)
- ld8 r11=[r3],PT(CR_IIP)-PT(R11)
-(pNonSys) break 0 // bug check: we shouldn't be here if pNonSys is
TRUE!
- ;;
- invala // M0|1 invalidate ALAT
- rsm psr.i | psr.ic // M2 turn off interrupts and interruption collection
- cmp.eq p9,p0=r0,r0 // A set p9 to indicate that we should restore cr.ifs
-
- ld8 r29=[r2],16 // M0|1 load cr.ipsr
- ld8 r28=[r3],16 // M0|1 load cr.iip
- mov r22=r0 // A clear r22
- ;;
- ld8 r30=[r2],16 // M0|1 load cr.ifs
- ld8 r25=[r3],16 // M0|1 load ar.unat
-(pUStk) add r14=IA64_TASK_THREAD_ON_USTACK_OFFSET,r13
- ;;
- ld8 r26=[r2],PT(B0)-PT(AR_PFS) // M0|1 load ar.pfs
-(pKStk) mov r22=psr // M2 read PSR now that interrupts are disabled
- nop 0
- ;;
- ld8 r21=[r2],PT(AR_RNAT)-PT(B0) // M0|1 load b0
- ld8 r27=[r3],PT(PR)-PT(AR_RSC) // M0|1 load ar.rsc
- mov f6=f0 // F clear f6
+ br.cond.sptk.few ia64_leave_syscall
;;
- ld8 r24=[r2],PT(AR_FPSR)-PT(AR_RNAT) // M0|1 load ar.rnat (may be garbage)
- ld8 r31=[r3],PT(R1)-PT(PR) // M0|1 load predicates
- mov f7=f0 // F clear f7
- ;;
- ld8 r20=[r2],PT(R12)-PT(AR_FPSR) // M0|1 load ar.fpsr
- ld8.fill r1=[r3],16 // M0|1 load r1
-(pUStk) mov r17=1 // A
- ;;
-(pUStk) st1 [r14]=r17 // M2|3
- ld8.fill r13=[r3],16 // M0|1
- mov f8=f0 // F clear f8
- ;;
- ld8.fill r12=[r2] // M0|1 restore r12 (sp)
- ld8.fill r15=[r3] // M0|1 restore r15
- mov b6=r18 // I0 restore b6
-
- LOAD_PHYS_STACK_REG_SIZE(r17)
- mov f9=f0 // F clear f9
-(pKStk) br.cond.dpnt.many skip_rbs_switch // B
-
- srlz.d // M0 ensure interruption collection is off (for cover)
- shr.u r18=r19,16 // I0|1 get byte size of existing "dirty"
partition
- cover // B add current frame into dirty partition & set cr.ifs
- ;;
- mov r19=ar.bsp // M2 get new backing store pointer
- mov f10=f0 // F clear f10
-
- nop.m 0
- movl r14=__kernel_syscall_via_epc // X
- ;;
- mov.m ar.csd=r0 // M2 clear ar.csd
- mov.m ar.ccv=r0 // M2 clear ar.ccv
- mov b7=r14 // I0 clear b7 (hint with __kernel_syscall_via_epc)
-
- mov.m ar.ssd=r0 // M2 clear ar.ssd
- mov f11=f0 // F clear f11
- br.cond.sptk.many rbs_switch // B
-END(ia64_leave_syscall)
+END(ia64_ret_from_syscall)
#ifdef CONFIG_IA32_SUPPORT
GLOBAL_ENTRY(ia64_ret_from_ia32_execve)
@@ -800,339 +579,12 @@ GLOBAL_ENTRY(ia64_ret_from_ia32_execve)
st8.spill [r2]=r8 // store return value in slot for r8 and set unat bit
.mem.offset 8,0
st8.spill [r3]=r0 // clear error indication in slot for r10 and set unat bit
-END(ia64_ret_from_ia32_execve)
- // fall through
-#endif /* CONFIG_IA32_SUPPORT */
-GLOBAL_ENTRY(ia64_leave_kernel)
- PT_REGS_UNWIND_INFO(0)
- /*
- * work.need_resched etc. mustn't get changed by this CPU before it
returns to
- * user- or fsys-mode, hence we disable interrupts early on.
- *
- * p6 controls whether current_thread_info()->flags needs to be check for
- * extra work. We always check for extra work when returning to user-level.
- * With CONFIG_PREEMPT, we also check for extra work when the preempt_count
- * is 0. After extra work processing has been completed, execution
- * resumes at .work_processed_syscall with p6 set to 1 if the extra-work-check
- * needs to be redone.
- */
-#ifdef CONFIG_PREEMPT
- rsm psr.i // disable interrupts
- cmp.eq p0,pLvSys=r0,r0 // pLvSys=0: leave from kernel
-(pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13
- ;;
- .pred.rel.mutex pUStk,pKStk
-(pKStk) ld4 r21=[r20] // r21 <- preempt_count
-(pUStk) mov r21=0 // r21 <- 0
- ;;
- cmp.eq p6,p0=r21,r0 // p6 <- pUStk || (preempt_count == 0)
-#else
-(pUStk) rsm psr.i
- cmp.eq p0,pLvSys=r0,r0 // pLvSys=0: leave from kernel
-(pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk
-#endif
-.work_processed_kernel:
- adds r17=TI_FLAGS+IA64_TASK_SIZE,r13
- ;;
-(p6) ld4 r31=[r17] // load current_thread_info()->flags
- adds r21=PT(PR)+16,r12
- ;;
-
- lfetch [r21],PT(CR_IPSR)-PT(PR)
- adds r2=PT(B6)+16,r12
- adds r3=PT(R16)+16,r12
;;
- lfetch [r21]
- ld8 r28=[r2],8 // load b6
- adds r29=PT(R24)+16,r12
-
- ld8.fill r16=[r3],PT(AR_CSD)-PT(R16)
- adds r30=PT(AR_CCV)+16,r12
-(p6) and r19=TIF_WORK_MASK,r31 // any work other than TIF_SYSCALL_TRACE?
- ;;
- ld8.fill r24=[r29]
- ld8 r15=[r30] // load ar.ccv
-(p6) cmp4.ne.unc p6,p0=r19, r0 // any special work pending?
- ;;
- ld8 r29=[r2],16 // load b7
- ld8 r30=[r3],16 // load ar.csd
-(p6) br.cond.spnt .work_pending
- ;;
- ld8 r31=[r2],16 // load ar.ssd
- ld8.fill r8=[r3],16
- ;;
- ld8.fill r9=[r2],16
- ld8.fill r10=[r3],PT(R17)-PT(R10)
- ;;
- ld8.fill r11=[r2],PT(R18)-PT(R11)
- ld8.fill r17=[r3],16
- ;;
- ld8.fill r18=[r2],16
- ld8.fill r19=[r3],16
- ;;
- ld8.fill r20=[r2],16
- ld8.fill r21=[r3],16
- mov ar.csd=r30
- mov ar.ssd=r31
- ;;
- rsm psr.i | psr.ic // initiate turning off of interrupt and interruption
collection
- invala // invalidate ALAT
- ;;
- ld8.fill r22=[r2],24
- ld8.fill r23=[r3],24
- mov b6=r28
- ;;
- ld8.fill r25=[r2],16
- ld8.fill r26=[r3],16
- mov b7=r29
- ;;
- ld8.fill r27=[r2],16
- ld8.fill r28=[r3],16
- ;;
- ld8.fill r29=[r2],16
- ld8.fill r30=[r3],24
- ;;
- ld8.fill r31=[r2],PT(F9)-PT(R31)
- adds r3=PT(F10)-PT(F6),r3
- ;;
- ldf.fill f9=[r2],PT(F6)-PT(F9)
- ldf.fill f10=[r3],PT(F8)-PT(F10)
- ;;
- ldf.fill f6=[r2],PT(F7)-PT(F6)
- ;;
- ldf.fill f7=[r2],PT(F11)-PT(F7)
- ldf.fill f8=[r3],32
+ // don't fall through, ia64_leave_kernel may be #define'd
+ br.cond.sptk.few ia64_leave_kernel
;;
- srlz.d // ensure that inter. collection is off (VHPT is don't care, since
text is pinned)
- mov ar.ccv=r15
- ;;
- ldf.fill f11=[r2]
- bsw.0 // switch back to bank 0 (no stop bit required beforehand...)
- ;;
-(pUStk) mov r18=IA64_KR(CURRENT)// M2 (12 cycle read latency)
- adds r16=PT(CR_IPSR)+16,r12
- adds r17=PT(CR_IIP)+16,r12
-
-(pKStk) mov r22=psr // M2 read PSR now that interrupts are disabled
- nop.i 0
- nop.i 0
- ;;
- ld8 r29=[r16],16 // load cr.ipsr
- ld8 r28=[r17],16 // load cr.iip
- ;;
- ld8 r30=[r16],16 // load cr.ifs
- ld8 r25=[r17],16 // load ar.unat
- ;;
- ld8 r26=[r16],16 // load ar.pfs
- ld8 r27=[r17],16 // load ar.rsc
- cmp.eq p9,p0=r0,r0 // set p9 to indicate that we should restore cr.ifs
- ;;
- ld8 r24=[r16],16 // load ar.rnat (may be garbage)
- ld8 r23=[r17],16 // load ar.bspstore (may be garbage)
- ;;
- ld8 r31=[r16],16 // load predicates
- ld8 r21=[r17],16 // load b0
- ;;
- ld8 r19=[r16],16 // load ar.rsc value for "loadrs"
- ld8.fill r1=[r17],16 // load r1
- ;;
- ld8.fill r12=[r16],16
- ld8.fill r13=[r17],16
-(pUStk) adds r18=IA64_TASK_THREAD_ON_USTACK_OFFSET,r18
- ;;
- ld8 r20=[r16],16 // ar.fpsr
- ld8.fill r15=[r17],16
- ;;
- ld8.fill r14=[r16],16
- ld8.fill r2=[r17]
-(pUStk) mov r17=1
- ;;
- ld8.fill r3=[r16]
-(pUStk) st1 [r18]=r17 // restore current->thread.on_ustack
- shr.u r18=r19,16 // get byte size of existing "dirty" partition
- ;;
- mov r16=ar.bsp // get existing backing store pointer
- LOAD_PHYS_STACK_REG_SIZE(r17)
-(pKStk) br.cond.dpnt skip_rbs_switch
-
- /*
- * Restore user backing store.
- *
- * NOTE: alloc, loadrs, and cover can't be predicated.
- */
-(pNonSys) br.cond.dpnt dont_preserve_current_frame
- cover // add current frame into dirty partition and set cr.ifs
- ;;
- mov r19=ar.bsp // get new backing store pointer
-rbs_switch:
- sub r16=r16,r18 // krbs = old bsp - size of dirty partition
- cmp.ne p9,p0=r0,r0 // clear p9 to skip restore of cr.ifs
- ;;
- sub r19=r19,r16 // calculate total byte size of dirty partition
- add r18=64,r18 // don't force in0-in7 into memory...
- ;;
- shl r19=r19,16 // shift size of dirty partition into loadrs position
- ;;
-dont_preserve_current_frame:
- /*
- * To prevent leaking bits between the kernel and user-space,
- * we must clear the stacked registers in the "invalid" partition
here.
- * Not pretty, but at least it's fast (3.34 registers/cycle on Itanium,
- * 5 registers/cycle on McKinley).
- */
-# define pRecurse p6
-# define pReturn p7
-#ifdef CONFIG_ITANIUM
-# define Nregs 10
-#else
-# define Nregs 14
-#endif
- alloc loc0=ar.pfs,2,Nregs-2,2,0
- shr.u loc1=r18,9 // RNaTslots <= floor(dirtySize / (64*8))
- sub r17=r17,r18 // r17 = (physStackedSize + 8) - dirtySize
- ;;
- mov ar.rsc=r19 // load ar.rsc to be used for "loadrs"
- shladd in0=loc1,3,r17
- mov in1=0
- ;;
- TEXT_ALIGN(32)
-rse_clear_invalid:
-#ifdef CONFIG_ITANIUM
- // cycle 0
- { .mii
- alloc loc0=ar.pfs,2,Nregs-2,2,0
- cmp.lt pRecurse,p0=Nregs*8,in0 // if more than Nregs regs left to clear,
(re)curse
- add out0=-Nregs*8,in0
-}{ .mfb
- add out1=1,in1 // increment recursion count
- nop.f 0
- nop.b 0 // can't do br.call here because of alloc (WAW on CFM)
- ;;
-}{ .mfi // cycle 1
- mov loc1=0
- nop.f 0
- mov loc2=0
-}{ .mib
- mov loc3=0
- mov loc4=0
-(pRecurse) br.call.sptk.many b0=rse_clear_invalid
-
-}{ .mfi // cycle 2
- mov loc5=0
- nop.f 0
- cmp.ne pReturn,p0=r0,in1 // if recursion count != 0, we need to do a br.ret
-}{ .mib
- mov loc6=0
- mov loc7=0
-(pReturn) br.ret.sptk.many b0
-}
-#else /* !CONFIG_ITANIUM */
- alloc loc0=ar.pfs,2,Nregs-2,2,0
- cmp.lt pRecurse,p0=Nregs*8,in0 // if more than Nregs regs left to clear,
(re)curse
- add out0=-Nregs*8,in0
- add out1=1,in1 // increment recursion count
- mov loc1=0
- mov loc2=0
- ;;
- mov loc3=0
- mov loc4=0
- mov loc5=0
- mov loc6=0
- mov loc7=0
-(pRecurse) br.call.dptk.few b0=rse_clear_invalid
- ;;
- mov loc8=0
- mov loc9=0
- cmp.ne pReturn,p0=r0,in1 // if recursion count != 0, we need to do a br.ret
- mov loc10=0
- mov loc11=0
-(pReturn) br.ret.dptk.many b0
-#endif /* !CONFIG_ITANIUM */
-# undef pRecurse
-# undef pReturn
- ;;
- alloc r17=ar.pfs,0,0,0,0 // drop current register frame
- ;;
- loadrs
- ;;
-skip_rbs_switch:
- mov ar.unat=r25 // M2
-(pKStk) extr.u r22=r22,21,1 // I0 extract current value of psr.pp from r22
-(pLvSys)mov r19=r0 // A clear r19 for leave_syscall, no-op otherwise
- ;;
-(pUStk) mov ar.bspstore=r23 // M2
-(pKStk) dep r29=r22,r29,21,1 // I0 update ipsr.pp with psr.pp
-(pLvSys)mov r16=r0 // A clear r16 for leave_syscall, no-op otherwise
- ;;
- mov cr.ipsr=r29 // M2
- mov ar.pfs=r26 // I0
-(pLvSys)mov r17=r0 // A clear r17 for leave_syscall, no-op otherwise
-
-(p9) mov cr.ifs=r30 // M2
- mov b0=r21 // I0
-(pLvSys)mov r18=r0 // A clear r18 for leave_syscall, no-op otherwise
-
- mov ar.fpsr=r20 // M2
- mov cr.iip=r28 // M2
- nop 0
- ;;
-(pUStk) mov ar.rnat=r24 // M2 must happen with RSE in lazy mode
- nop 0
-(pLvSys)mov r2=r0
-
- mov ar.rsc=r27 // M2
- mov pr=r31,-1 // I0
- rfi // B
-
- /*
- * On entry:
- * r20 = ¤t->thread_info->pre_count (if CONFIG_PREEMPT)
- * r31 = current->thread_info->flags
- * On exit:
- * p6 = TRUE if work-pending-check needs to be redone
- */
-.work_pending_syscall:
- add r2=-8,r2
- add r3=-8,r3
- ;;
- st8 [r2]=r8
- st8 [r3]=r10
-.work_pending:
- tbit.z p6,p0=r31,TIF_NEED_RESCHED //
current_thread_info()->need_resched==0?
-(p6) br.cond.sptk.few .notify
-#ifdef CONFIG_PREEMPT
-(pKStk) dep r21=-1,r0,PREEMPT_ACTIVE_BIT,1
- ;;
-(pKStk) st4 [r20]=r21
- ssm psr.i // enable interrupts
-#endif
- br.call.spnt.many rp=schedule
-.ret9: cmp.eq p6,p0=r0,r0 // p6 <- 1
- rsm psr.i // disable interrupts
- ;;
-#ifdef CONFIG_PREEMPT
-(pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13
- ;;
-(pKStk) st4 [r20]=r0 // preempt_count() <- 0
-#endif
-(pLvSys)br.cond.sptk.few .work_pending_syscall_end
- br.cond.sptk.many .work_processed_kernel // re-check
-
-.notify:
-(pUStk) br.call.spnt.many rp=notify_resume_user
-.ret10: cmp.ne p6,p0=r0,r0 // p6 <- 0
-(pLvSys)br.cond.sptk.few .work_pending_syscall_end
- br.cond.sptk.many .work_processed_kernel // don't re-check
-
-.work_pending_syscall_end:
- adds r2=PT(R8)+16,r12
- adds r3=PT(R10)+16,r12
- ;;
- ld8 r8=[r2]
- ld8 r10=[r3]
- br.cond.sptk.many .work_processed_syscall // re-check
-
-END(ia64_leave_kernel)
+END(ia64_ret_from_ia32_execve)
+#endif /* CONFIG_IA32_SUPPORT */
ENTRY(handle_syscall_error)
/*
@@ -1234,7 +686,7 @@ ENTRY(sys_rt_sigreturn)
adds sp=16,sp
;;
ld8 r9=[sp] // load new ar.unat
- mov.sptk b7=r8,ia64_leave_kernel
+ mov b7=r8
;;
mov ar.unat=r9
br.many b7
diff --git a/arch/ia64/kernel/switch_leave.S b/arch/ia64/kernel/switch_leave.S
new file mode 100644
index 0000000..5ca5b84
--- /dev/null
+++ b/arch/ia64/kernel/switch_leave.S
@@ -0,0 +1,594 @@
+/*
+ * arch/ia64/kernel/switch_leave.S
+ * Kernel entry points.
+ * ia64_switch_to(), ia64_leave_syscall() and ia64_leave_kernel()
+ * split from arch/ia64/kernel/entry.S for paravirtualization
+ *
+ * Copyright (C) 1998-2003, 2005 Hewlett-Packard Co
+ * David Mosberger-Tang <davidm at hpl.hp.com>
+ * Copyright (C) 1999, 2002-2003
+ * Asit Mallick <Asit.K.Mallick at intel.com>
+ * Don Dugger <Don.Dugger at intel.com>
+ * Suresh Siddha <suresh.b.siddha at intel.com>
+ * Fenghua Yu <fenghua.yu at intel.com>
+ * Copyright (C) 1999 VA Linux Systems
+ * Copyright (C) 1999 Walt Drummond <drummond at valinux.com>
+ */
+/*
+ * ia64_switch_to now places correct virtual mapping in in TR2 for
+ * kernel stack. This allows us to handle interrupts without changing
+ * to physical mode.
+ *
+ * Jonathan Nicklin <nicklin at missioncriticallinux.com>
+ * Patrick O'Rourke <orourke at missioncriticallinux.com>
+ * 11/07/2000
+ */
+/*
+ * Copyright (c) 2008 Isaku Yamahata <yamahata at valinux co jp>
+ * VA Linux Systems Japan K.K.
+ * pv_ops.
+ */
+/*
+ * Global (preserved) predicate usage on syscall entry/exit path:
+ *
+ * pKStk: See entry.h.
+ * pUStk: See entry.h.
+ * pSys: See entry.h.
+ * pNonSys: !pSys
+ */
+
+
+#include <asm/asmmacro.h>
+#include <asm/kregs.h>
+#include <asm/asm-offsets.h>
+#include <asm/pgtable.h>
+#include <asm/thread_info.h>
+
+#include "minstate.h"
+
+
+/*
+ * prev_task <- ia64_switch_to(struct task_struct *next)
+ * With Ingo's new scheduler, interrupts are disabled when this routine
gets
+ * called. The code starting at .map relies on this. The rest of the code
+ * doesn't care about the interrupt masking status.
+ */
+GLOBAL_ENTRY(ia64_switch_to)
+ .prologue
+ alloc r16=ar.pfs,1,0,0,0
+ DO_SAVE_SWITCH_STACK
+ .body
+
+ adds r22=IA64_TASK_THREAD_KSP_OFFSET,r13
+ movl r25=init_task
+ mov r27=IA64_KR(CURRENT_STACK)
+ adds r21=IA64_TASK_THREAD_KSP_OFFSET,in0
+ dep r20=0,in0,61,3 // physical address of "next"
+ ;;
+ st8 [r22]=sp // save kernel stack pointer of old task
+ shr.u r26=r20,IA64_GRANULE_SHIFT
+ cmp.eq p7,p6=r25,in0
+ ;;
+ /*
+ * If we've already mapped this task's page, we can skip doing it
again.
+ */
+(p6) cmp.eq p7,p6=r26,r27
+(p6) br.cond.dpnt .map
+ ;;
+.done:
+ ld8 sp=[r21] // load kernel stack pointer of new task
+ mov IA64_KR(CURRENT)=in0 // update "current" application register
+ mov r8=r13 // return pointer to previously running task
+ mov r13=in0 // set "current" pointer
+ ;;
+ DO_LOAD_SWITCH_STACK
+
+#ifdef CONFIG_SMP
+ sync.i // ensure "fc"s done by this CPU are visible on other CPUs
+#endif
+ br.ret.sptk.many rp // boogie on out in new context
+
+.map:
+ rsm psr.ic // interrupts (psr.i) are already disabled here
+ movl r25=PAGE_KERNEL
+ ;;
+ srlz.d
+ or r23=r25,r20 // construct PA | page properties
+ mov r25=IA64_GRANULE_SHIFT<<2
+ ;;
+ mov cr.itir=r25
+ mov cr.ifa=in0 // VA of next task...
+ ;;
+ mov r25=IA64_TR_CURRENT_STACK
+ mov IA64_KR(CURRENT_STACK)=r26 // remember last page we mapped...
+ ;;
+ itr.d dtr[r25]=r23 // wire in new mapping...
+ ssm psr.ic // reenable the psr.ic bit
+ ;;
+ srlz.d
+ br.cond.sptk .done
+END(ia64_switch_to)
+
+/*
+ * ia64_leave_syscall(): Same as ia64_leave_kernel, except that it doesn't
+ * need to switch to bank 0 and doesn't restore the scratch registers.
+ * To avoid leaking kernel bits, the scratch registers are set to
+ * the following known-to-be-safe values:
+ *
+ * r1: restored (global pointer)
+ * r2: cleared
+ * r3: 1 (when returning to user-level)
+ * r8-r11: restored (syscall return value(s))
+ * r12: restored (user-level stack pointer)
+ * r13: restored (user-level thread pointer)
+ * r14: set to __kernel_syscall_via_epc
+ * r15: restored (syscall #)
+ * r16-r17: cleared
+ * r18: user-level b6
+ * r19: cleared
+ * r20: user-level ar.fpsr
+ * r21: user-level b0
+ * r22: cleared
+ * r23: user-level ar.bspstore
+ * r24: user-level ar.rnat
+ * r25: user-level ar.unat
+ * r26: user-level ar.pfs
+ * r27: user-level ar.rsc
+ * r28: user-level ip
+ * r29: user-level psr
+ * r30: user-level cfm
+ * r31: user-level pr
+ * f6-f11: cleared
+ * pr: restored (user-level pr)
+ * b0: restored (user-level rp)
+ * b6: restored
+ * b7: set to __kernel_syscall_via_epc
+ * ar.unat: restored (user-level ar.unat)
+ * ar.pfs: restored (user-level ar.pfs)
+ * ar.rsc: restored (user-level ar.rsc)
+ * ar.rnat: restored (user-level ar.rnat)
+ * ar.bspstore: restored (user-level ar.bspstore)
+ * ar.fpsr: restored (user-level ar.fpsr)
+ * ar.ccv: cleared
+ * ar.csd: cleared
+ * ar.ssd: cleared
+ */
+ENTRY(ia64_leave_syscall)
+ PT_REGS_UNWIND_INFO(0)
+ /*
+ * work.need_resched etc. mustn't get changed by this CPU before it
returns to
+ * user- or fsys-mode, hence we disable interrupts early on.
+ *
+ * p6 controls whether current_thread_info()->flags needs to be check for
+ * extra work. We always check for extra work when returning to user-level.
+ * With CONFIG_PREEMPT, we also check for extra work when the preempt_count
+ * is 0. After extra work processing has been completed, execution
+ * resumes at .work_processed_syscall with p6 set to 1 if the extra-work-check
+ * needs to be redone.
+ */
+#ifdef CONFIG_PREEMPT
+ rsm psr.i // disable interrupts
+ cmp.eq pLvSys,p0=r0,r0 // pLvSys=1: leave from syscall
+(pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13
+ ;;
+ .pred.rel.mutex pUStk,pKStk
+(pKStk) ld4 r21=[r20] // r21 <- preempt_count
+(pUStk) mov r21=0 // r21 <- 0
+ ;;
+ cmp.eq p6,p0=r21,r0 // p6 <- pUStk || (preempt_count == 0)
+#else /* !CONFIG_PREEMPT */
+(pUStk) rsm psr.i
+ cmp.eq pLvSys,p0=r0,r0 // pLvSys=1: leave from syscall
+(pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk
+#endif
+.work_processed_syscall:
+ adds r2=PT(LOADRS)+16,r12
+ adds r3=PT(AR_BSPSTORE)+16,r12
+ adds r18=TI_FLAGS+IA64_TASK_SIZE,r13
+ ;;
+(p6) ld4 r31=[r18] // load current_thread_info()->flags
+ ld8 r19=[r2],PT(B6)-PT(LOADRS) // load ar.rsc value for "loadrs"
+ nop.i 0
+ ;;
+ mov r16=ar.bsp // M2 get existing backing store pointer
+ ld8 r18=[r2],PT(R9)-PT(B6) // load b6
+(p6) and r15=TIF_WORK_MASK,r31 // any work other than TIF_SYSCALL_TRACE?
+ ;;
+ ld8 r23=[r3],PT(R11)-PT(AR_BSPSTORE) // load ar.bspstore (may be garbage)
+(p6) cmp4.ne.unc p6,p0=r15, r0 // any special work pending?
+(p6) br.cond.spnt .work_pending_syscall
+ ;;
+ // start restoring the state saved on the kernel stack (struct pt_regs):
+ ld8 r9=[r2],PT(CR_IPSR)-PT(R9)
+ ld8 r11=[r3],PT(CR_IIP)-PT(R11)
+(pNonSys) break 0 // bug check: we shouldn't be here if pNonSys is
TRUE!
+ ;;
+ invala // M0|1 invalidate ALAT
+ rsm psr.i | psr.ic // M2 turn off interrupts and interruption collection
+ cmp.eq p9,p0=r0,r0 // A set p9 to indicate that we should restore cr.ifs
+
+ ld8 r29=[r2],16 // M0|1 load cr.ipsr
+ ld8 r28=[r3],16 // M0|1 load cr.iip
+ mov r22=r0 // A clear r22
+ ;;
+ ld8 r30=[r2],16 // M0|1 load cr.ifs
+ ld8 r25=[r3],16 // M0|1 load ar.unat
+(pUStk) add r14=IA64_TASK_THREAD_ON_USTACK_OFFSET,r13
+ ;;
+ ld8 r26=[r2],PT(B0)-PT(AR_PFS) // M0|1 load ar.pfs
+(pKStk) mov r22=psr // M2 read PSR now that interrupts are disabled
+ nop 0
+ ;;
+ ld8 r21=[r2],PT(AR_RNAT)-PT(B0) // M0|1 load b0
+ ld8 r27=[r3],PT(PR)-PT(AR_RSC) // M0|1 load ar.rsc
+ mov f6=f0 // F clear f6
+ ;;
+ ld8 r24=[r2],PT(AR_FPSR)-PT(AR_RNAT) // M0|1 load ar.rnat (may be garbage)
+ ld8 r31=[r3],PT(R1)-PT(PR) // M0|1 load predicates
+ mov f7=f0 // F clear f7
+ ;;
+ ld8 r20=[r2],PT(R12)-PT(AR_FPSR) // M0|1 load ar.fpsr
+ ld8.fill r1=[r3],16 // M0|1 load r1
+(pUStk) mov r17=1 // A
+ ;;
+(pUStk) st1 [r14]=r17 // M2|3
+ ld8.fill r13=[r3],16 // M0|1
+ mov f8=f0 // F clear f8
+ ;;
+ ld8.fill r12=[r2] // M0|1 restore r12 (sp)
+ ld8.fill r15=[r3] // M0|1 restore r15
+ mov b6=r18 // I0 restore b6
+
+ LOAD_PHYS_STACK_REG_SIZE(r17)
+ mov f9=f0 // F clear f9
+(pKStk) br.cond.dpnt.many skip_rbs_switch // B
+
+ srlz.d // M0 ensure interruption collection is off (for cover)
+ shr.u r18=r19,16 // I0|1 get byte size of existing "dirty"
partition
+ cover // B add current frame into dirty partition & set cr.ifs
+ ;;
+ mov r19=ar.bsp // M2 get new backing store pointer
+ mov f10=f0 // F clear f10
+
+ nop.m 0
+ movl r14=__kernel_syscall_via_epc // X
+ ;;
+ mov.m ar.csd=r0 // M2 clear ar.csd
+ mov.m ar.ccv=r0 // M2 clear ar.ccv
+ mov b7=r14 // I0 clear b7 (hint with __kernel_syscall_via_epc)
+
+ mov.m ar.ssd=r0 // M2 clear ar.ssd
+ mov f11=f0 // F clear f11
+ br.cond.sptk.many rbs_switch // B
+END(ia64_leave_syscall)
+
+GLOBAL_ENTRY(ia64_leave_kernel)
+ PT_REGS_UNWIND_INFO(0)
+ /*
+ * work.need_resched etc. mustn't get changed by this CPU before it
returns to
+ * user- or fsys-mode, hence we disable interrupts early on.
+ *
+ * p6 controls whether current_thread_info()->flags needs to be check for
+ * extra work. We always check for extra work when returning to user-level.
+ * With CONFIG_PREEMPT, we also check for extra work when the preempt_count
+ * is 0. After extra work processing has been completed, execution
+ * resumes at .work_processed_syscall with p6 set to 1 if the extra-work-check
+ * needs to be redone.
+ */
+#ifdef CONFIG_PREEMPT
+ rsm psr.i // disable interrupts
+ cmp.eq p0,pLvSys=r0,r0 // pLvSys=0: leave from kernel
+(pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13
+ ;;
+ .pred.rel.mutex pUStk,pKStk
+(pKStk) ld4 r21=[r20] // r21 <- preempt_count
+(pUStk) mov r21=0 // r21 <- 0
+ ;;
+ cmp.eq p6,p0=r21,r0 // p6 <- pUStk || (preempt_count == 0)
+#else
+(pUStk) rsm psr.i
+ cmp.eq p0,pLvSys=r0,r0 // pLvSys=0: leave from kernel
+(pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk
+#endif
+.work_processed_kernel:
+ adds r17=TI_FLAGS+IA64_TASK_SIZE,r13
+ ;;
+(p6) ld4 r31=[r17] // load current_thread_info()->flags
+ adds r21=PT(PR)+16,r12
+ ;;
+
+ lfetch [r21],PT(CR_IPSR)-PT(PR)
+ adds r2=PT(B6)+16,r12
+ adds r3=PT(R16)+16,r12
+ ;;
+ lfetch [r21]
+ ld8 r28=[r2],8 // load b6
+ adds r29=PT(R24)+16,r12
+
+ ld8.fill r16=[r3],PT(AR_CSD)-PT(R16)
+ adds r30=PT(AR_CCV)+16,r12
+(p6) and r19=TIF_WORK_MASK,r31 // any work other than TIF_SYSCALL_TRACE?
+ ;;
+ ld8.fill r24=[r29]
+ ld8 r15=[r30] // load ar.ccv
+(p6) cmp4.ne.unc p6,p0=r19, r0 // any special work pending?
+ ;;
+ ld8 r29=[r2],16 // load b7
+ ld8 r30=[r3],16 // load ar.csd
+(p6) br.cond.spnt .work_pending
+ ;;
+ ld8 r31=[r2],16 // load ar.ssd
+ ld8.fill r8=[r3],16
+ ;;
+ ld8.fill r9=[r2],16
+ ld8.fill r10=[r3],PT(R17)-PT(R10)
+ ;;
+ ld8.fill r11=[r2],PT(R18)-PT(R11)
+ ld8.fill r17=[r3],16
+ ;;
+ ld8.fill r18=[r2],16
+ ld8.fill r19=[r3],16
+ ;;
+ ld8.fill r20=[r2],16
+ ld8.fill r21=[r3],16
+ mov ar.csd=r30
+ mov ar.ssd=r31
+ ;;
+ rsm psr.i | psr.ic // initiate turning off of interrupt and interruption
collection
+ invala // invalidate ALAT
+ ;;
+ ld8.fill r22=[r2],24
+ ld8.fill r23=[r3],24
+ mov b6=r28
+ ;;
+ ld8.fill r25=[r2],16
+ ld8.fill r26=[r3],16
+ mov b7=r29
+ ;;
+ ld8.fill r27=[r2],16
+ ld8.fill r28=[r3],16
+ ;;
+ ld8.fill r29=[r2],16
+ ld8.fill r30=[r3],24
+ ;;
+ ld8.fill r31=[r2],PT(F9)-PT(R31)
+ adds r3=PT(F10)-PT(F6),r3
+ ;;
+ ldf.fill f9=[r2],PT(F6)-PT(F9)
+ ldf.fill f10=[r3],PT(F8)-PT(F10)
+ ;;
+ ldf.fill f6=[r2],PT(F7)-PT(F6)
+ ;;
+ ldf.fill f7=[r2],PT(F11)-PT(F7)
+ ldf.fill f8=[r3],32
+ ;;
+ srlz.d // ensure that inter. collection is off (VHPT is don't care, since
text is pinned)
+ mov ar.ccv=r15
+ ;;
+ ldf.fill f11=[r2]
+ bsw.0 // switch back to bank 0 (no stop bit required beforehand...)
+ ;;
+(pUStk) mov r18=IA64_KR(CURRENT)// M2 (12 cycle read latency)
+ adds r16=PT(CR_IPSR)+16,r12
+ adds r17=PT(CR_IIP)+16,r12
+
+(pKStk) mov r22=psr // M2 read PSR now that interrupts are disabled
+ nop.i 0
+ nop.i 0
+ ;;
+ ld8 r29=[r16],16 // load cr.ipsr
+ ld8 r28=[r17],16 // load cr.iip
+ ;;
+ ld8 r30=[r16],16 // load cr.ifs
+ ld8 r25=[r17],16 // load ar.unat
+ ;;
+ ld8 r26=[r16],16 // load ar.pfs
+ ld8 r27=[r17],16 // load ar.rsc
+ cmp.eq p9,p0=r0,r0 // set p9 to indicate that we should restore cr.ifs
+ ;;
+ ld8 r24=[r16],16 // load ar.rnat (may be garbage)
+ ld8 r23=[r17],16 // load ar.bspstore (may be garbage)
+ ;;
+ ld8 r31=[r16],16 // load predicates
+ ld8 r21=[r17],16 // load b0
+ ;;
+ ld8 r19=[r16],16 // load ar.rsc value for "loadrs"
+ ld8.fill r1=[r17],16 // load r1
+ ;;
+ ld8.fill r12=[r16],16
+ ld8.fill r13=[r17],16
+(pUStk) adds r18=IA64_TASK_THREAD_ON_USTACK_OFFSET,r18
+ ;;
+ ld8 r20=[r16],16 // ar.fpsr
+ ld8.fill r15=[r17],16
+ ;;
+ ld8.fill r14=[r16],16
+ ld8.fill r2=[r17]
+(pUStk) mov r17=1
+ ;;
+ ld8.fill r3=[r16]
+(pUStk) st1 [r18]=r17 // restore current->thread.on_ustack
+ shr.u r18=r19,16 // get byte size of existing "dirty" partition
+ ;;
+ mov r16=ar.bsp // get existing backing store pointer
+ LOAD_PHYS_STACK_REG_SIZE(r17)
+(pKStk) br.cond.dpnt skip_rbs_switch
+
+ /*
+ * Restore user backing store.
+ *
+ * NOTE: alloc, loadrs, and cover can't be predicated.
+ */
+(pNonSys) br.cond.dpnt dont_preserve_current_frame
+ cover // add current frame into dirty partition and set cr.ifs
+ ;;
+ mov r19=ar.bsp // get new backing store pointer
+rbs_switch:
+ sub r16=r16,r18 // krbs = old bsp - size of dirty partition
+ cmp.ne p9,p0=r0,r0 // clear p9 to skip restore of cr.ifs
+ ;;
+ sub r19=r19,r16 // calculate total byte size of dirty partition
+ add r18=64,r18 // don't force in0-in7 into memory...
+ ;;
+ shl r19=r19,16 // shift size of dirty partition into loadrs position
+ ;;
+dont_preserve_current_frame:
+ /*
+ * To prevent leaking bits between the kernel and user-space,
+ * we must clear the stacked registers in the "invalid" partition
here.
+ * Not pretty, but at least it's fast (3.34 registers/cycle on Itanium,
+ * 5 registers/cycle on McKinley).
+ */
+# define pRecurse p6
+# define pReturn p7
+#ifdef CONFIG_ITANIUM
+# define Nregs 10
+#else
+# define Nregs 14
+#endif
+ alloc loc0=ar.pfs,2,Nregs-2,2,0
+ shr.u loc1=r18,9 // RNaTslots <= floor(dirtySize / (64*8))
+ sub r17=r17,r18 // r17 = (physStackedSize + 8) - dirtySize
+ ;;
+ mov ar.rsc=r19 // load ar.rsc to be used for "loadrs"
+ shladd in0=loc1,3,r17
+ mov in1=0
+ ;;
+ TEXT_ALIGN(32)
+rse_clear_invalid:
+#ifdef CONFIG_ITANIUM
+ // cycle 0
+ { .mii
+ alloc loc0=ar.pfs,2,Nregs-2,2,0
+ cmp.lt pRecurse,p0=Nregs*8,in0 // if more than Nregs regs left to clear,
(re)curse
+ add out0=-Nregs*8,in0
+}{ .mfb
+ add out1=1,in1 // increment recursion count
+ nop.f 0
+ nop.b 0 // can't do br.call here because of alloc (WAW on CFM)
+ ;;
+}{ .mfi // cycle 1
+ mov loc1=0
+ nop.f 0
+ mov loc2=0
+}{ .mib
+ mov loc3=0
+ mov loc4=0
+(pRecurse) br.call.sptk.many b0=rse_clear_invalid
+
+}{ .mfi // cycle 2
+ mov loc5=0
+ nop.f 0
+ cmp.ne pReturn,p0=r0,in1 // if recursion count != 0, we need to do a br.ret
+}{ .mib
+ mov loc6=0
+ mov loc7=0
+(pReturn) br.ret.sptk.many b0
+}
+#else /* !CONFIG_ITANIUM */
+ alloc loc0=ar.pfs,2,Nregs-2,2,0
+ cmp.lt pRecurse,p0=Nregs*8,in0 // if more than Nregs regs left to clear,
(re)curse
+ add out0=-Nregs*8,in0
+ add out1=1,in1 // increment recursion count
+ mov loc1=0
+ mov loc2=0
+ ;;
+ mov loc3=0
+ mov loc4=0
+ mov loc5=0
+ mov loc6=0
+ mov loc7=0
+(pRecurse) br.call.dptk.few b0=rse_clear_invalid
+ ;;
+ mov loc8=0
+ mov loc9=0
+ cmp.ne pReturn,p0=r0,in1 // if recursion count != 0, we need to do a br.ret
+ mov loc10=0
+ mov loc11=0
+(pReturn) br.ret.dptk.many b0
+#endif /* !CONFIG_ITANIUM */
+# undef pRecurse
+# undef pReturn
+ ;;
+ alloc r17=ar.pfs,0,0,0,0 // drop current register frame
+ ;;
+ loadrs
+ ;;
+skip_rbs_switch:
+ mov ar.unat=r25 // M2
+(pKStk) extr.u r22=r22,21,1 // I0 extract current value of psr.pp from r22
+(pLvSys)mov r19=r0 // A clear r19 for leave_syscall, no-op otherwise
+ ;;
+(pUStk) mov ar.bspstore=r23 // M2
+(pKStk) dep r29=r22,r29,21,1 // I0 update ipsr.pp with psr.pp
+(pLvSys)mov r16=r0 // A clear r16 for leave_syscall, no-op otherwise
+ ;;
+ mov cr.ipsr=r29 // M2
+ mov ar.pfs=r26 // I0
+(pLvSys)mov r17=r0 // A clear r17 for leave_syscall, no-op otherwise
+
+(p9) mov cr.ifs=r30 // M2
+ mov b0=r21 // I0
+(pLvSys)mov r18=r0 // A clear r18 for leave_syscall, no-op otherwise
+
+ mov ar.fpsr=r20 // M2
+ mov cr.iip=r28 // M2
+ nop 0
+ ;;
+(pUStk) mov ar.rnat=r24 // M2 must happen with RSE in lazy mode
+ nop 0
+(pLvSys)mov r2=r0
+
+ mov ar.rsc=r27 // M2
+ mov pr=r31,-1 // I0
+ rfi // B
+
+ /*
+ * On entry:
+ * r20 = ¤t->thread_info->pre_count (if CONFIG_PREEMPT)
+ * r31 = current->thread_info->flags
+ * On exit:
+ * p6 = TRUE if work-pending-check needs to be redone
+ */
+.work_pending_syscall:
+ add r2=-8,r2
+ add r3=-8,r3
+ ;;
+ st8 [r2]=r8
+ st8 [r3]=r10
+.work_pending:
+ tbit.z p6,p0=r31,TIF_NEED_RESCHED //
current_thread_info()->need_resched==0?
+(p6) br.cond.sptk.few .notify
+#ifdef CONFIG_PREEMPT
+(pKStk) dep r21=-1,r0,PREEMPT_ACTIVE_BIT,1
+ ;;
+(pKStk) st4 [r20]=r21
+ ssm psr.i // enable interrupts
+#endif
+ br.call.spnt.many rp=schedule
+.ret9: cmp.eq p6,p0=r0,r0 // p6 <- 1
+ rsm psr.i // disable interrupts
+ ;;
+#ifdef CONFIG_PREEMPT
+(pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13
+ ;;
+(pKStk) st4 [r20]=r0 // preempt_count() <- 0
+#endif
+(pLvSys)br.cond.sptk.few ia64_work_pending_syscall_end
+ br.cond.sptk.many .work_processed_kernel // re-check
+
+.notify:
+(pUStk) br.call.spnt.many rp=notify_resume_user
+.ret10: cmp.ne p6,p0=r0,r0 // p6 <- 0
+(pLvSys)br.cond.sptk.few ia64_work_pending_syscall_end
+ br.cond.sptk.many .work_processed_kernel // don't re-check
+
+.global ia64_work_pending_syscall_end;
+ia64_work_pending_syscall_end:
+ adds r2=PT(R8)+16,r12
+ adds r3=PT(R10)+16,r12
+ ;;
+ ld8 r8=[r2]
+ ld8 r10=[r3]
+ br.cond.sptk.many .work_processed_syscall // re-check
+END(ia64_leave_kernel)
--
1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 2/8] ia64/pv_ops: preparation for paravirtualizatin of switch_leave.S and ivt.S
make some symbol global and add some hooks. define __IA64_ASM_PARAVIRTUALIZED_NATIVE to tell its native compilation when compiling ivt.S and switch_leave.S Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/kernel/Makefile | 6 ++++++ arch/ia64/kernel/entry.S | 4 ++-- arch/ia64/kernel/minstate.h | 2 ++ arch/ia64/kernel/switch_leave.S | 19 ++++++++++--------- include/asm-ia64/privop.h | 26 ++++++++++++++++++++++++++ 5 files changed, 46 insertions(+), 11 deletions(-) create mode 100644 include/asm-ia64/privop.h diff --git a/arch/ia64/kernel/Makefile b/arch/ia64/kernel/Makefile index f9bc3c4..9281bf6 100644 --- a/arch/ia64/kernel/Makefile +++ b/arch/ia64/kernel/Makefile @@ -70,3 +70,9 @@ $(obj)/gate-syms.o: $(obj)/gate.lds $(obj)/gate.o FORCE # We must build gate.so before we can assemble it. # Note: kbuild does not track this dependency due to usage of .incbin $(obj)/gate-data.o: $(obj)/gate.so + +# +# native ivt.S and switch_leave.S +# +AFLAGS_ivt.o += -D__IA64_ASM_PARAVIRTUALIZED_NATIVE +AFLAGS_switch_leave.o += -D__IA64_ASM_PARAVIRTUALIZED_NATIVE diff --git a/arch/ia64/kernel/entry.S b/arch/ia64/kernel/entry.S index df8dcc9..de91f61 100644 --- a/arch/ia64/kernel/entry.S +++ b/arch/ia64/kernel/entry.S @@ -304,7 +304,7 @@ END(save_switch_stack) * - b7 holds address to return to * - must not touch r8-r11 */ -ENTRY(load_switch_stack) +GLOBAL_ENTRY(load_switch_stack) .prologue .altrp b7 @@ -624,7 +624,7 @@ END(ia64_invoke_schedule_tail) * be set up by the caller. We declare 8 input registers so the system call * args get preserved, in case we need to restart a system call. */ -ENTRY(notify_resume_user) +GLOBAL_ENTRY(notify_resume_user) .prologue ASM_UNW_PRLG_RP|ASM_UNW_PRLG_PFS, ASM_UNW_PRLG_GRSAVE(8) alloc loc1=ar.pfs,8,2,3,0 // preserve all eight input regs in case of syscall restart! mov r9=ar.unat diff --git a/arch/ia64/kernel/minstate.h b/arch/ia64/kernel/minstate.h index c9ac8ba..3dfd36c 100644 --- a/arch/ia64/kernel/minstate.h +++ b/arch/ia64/kernel/minstate.h @@ -3,6 +3,7 @@ #include "entry.h" +#ifdef __IA64_ASM_PARAVIRTUALIZED_NATIVE /* * DO_SAVE_MIN switches to the kernel stacks (if necessary) and saves * the minimum state necessary that allows us to turn psr.ic back @@ -129,6 +130,7 @@ ;; \ bsw.1; /* switch back to bank 1 (must be last in insn group) */ \ ;; +#endif /* __IA64_ASM_PARAVIRTUALIZED_NATIVE */ /* * SAVE_REST saves the remainder of pt_regs (with psr.ic on). diff --git a/arch/ia64/kernel/switch_leave.S b/arch/ia64/kernel/switch_leave.S index 5ca5b84..b235917 100644 --- a/arch/ia64/kernel/switch_leave.S +++ b/arch/ia64/kernel/switch_leave.S @@ -53,7 +53,7 @@ * called. The code starting at .map relies on this. The rest of the code * doesn't care about the interrupt masking status. */ -GLOBAL_ENTRY(ia64_switch_to) +GLOBAL_ENTRY(__ia64_switch_to) .prologue alloc r16=ar.pfs,1,0,0,0 DO_SAVE_SWITCH_STACK @@ -107,7 +107,7 @@ GLOBAL_ENTRY(ia64_switch_to) ;; srlz.d br.cond.sptk .done -END(ia64_switch_to) +END(__ia64_switch_to) /* * ia64_leave_syscall(): Same as ia64_leave_kernel, except that it doesn't @@ -153,7 +153,7 @@ END(ia64_switch_to) * ar.csd: cleared * ar.ssd: cleared */ -ENTRY(ia64_leave_syscall) +GLOBAL_ENTRY(__ia64_leave_syscall) PT_REGS_UNWIND_INFO(0) /* * work.need_resched etc. mustn't get changed by this CPU before it returns to @@ -163,7 +163,7 @@ ENTRY(ia64_leave_syscall) * extra work. We always check for extra work when returning to user-level. * With CONFIG_PREEMPT, we also check for extra work when the preempt_count * is 0. After extra work processing has been completed, execution - * resumes at .work_processed_syscall with p6 set to 1 if the extra-work-check + * resumes at ia64_work_processed_syscall with p6 set to 1 if the extra-work-check * needs to be redone. */ #ifdef CONFIG_PREEMPT @@ -181,7 +181,8 @@ ENTRY(ia64_leave_syscall) cmp.eq pLvSys,p0=r0,r0 // pLvSys=1: leave from syscall (pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk #endif -.work_processed_syscall: +.global __ia64_work_processed_syscall; +__ia64_work_processed_syscall: adds r2=PT(LOADRS)+16,r12 adds r3=PT(AR_BSPSTORE)+16,r12 adds r18=TI_FLAGS+IA64_TASK_SIZE,r13 @@ -260,9 +261,9 @@ ENTRY(ia64_leave_syscall) mov.m ar.ssd=r0 // M2 clear ar.ssd mov f11=f0 // F clear f11 br.cond.sptk.many rbs_switch // B -END(ia64_leave_syscall) +END(__ia64_leave_syscall) -GLOBAL_ENTRY(ia64_leave_kernel) +GLOBAL_ENTRY(__ia64_leave_kernel) PT_REGS_UNWIND_INFO(0) /* * work.need_resched etc. mustn't get changed by this CPU before it returns to @@ -590,5 +591,5 @@ ia64_work_pending_syscall_end: ;; ld8 r8=[r2] ld8 r10=[r3] - br.cond.sptk.many .work_processed_syscall // re-check -END(ia64_leave_kernel) + br.cond.sptk.many ia64_work_processed_syscall // re-check +END(__ia64_leave_kernel) diff --git a/include/asm-ia64/privop.h b/include/asm-ia64/privop.h new file mode 100644 index 0000000..b8dce79 --- /dev/null +++ b/include/asm-ia64/privop.h @@ -0,0 +1,26 @@ +#ifndef _ASM_IA64_PRIVOP_H +#define _ASM_IA64_PRIVOP_H + +#ifndef _ASM_IA64_INTRINSICS_H +#error "don't include privop.h directly. instead include intrinsics.h" +#endif +/* + * Copyright (C) 2005 Hewlett-Packard Co + * Dan Magenheimer <dan.magenheimer at hp.com> + * + */ + +#ifdef CONFIG_XEN +#include <asm/xen/privop.h> +#endif + +/* fallback for native case */ + +#ifndef IA64_PARAVIRTUALIZED_ENTRY +#define ia64_switch_to __ia64_switch_to +#define ia64_leave_syscall __ia64_leave_syscall +#define ia64_work_processed_syscall __ia64_work_processed_syscall +#define ia64_leave_kernel __ia64_leave_kernel +#endif /* !IA64_PARAVIRTUALIZED_ENTRY */ + +#endif /* _ASM_IA64_PRIVOP_H */ -- 1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 3/8] ia64/pv_ops: paravirtualization hand written assnbly code. inst_native.h
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/kernel/inst_native.h | 153 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 153 insertions(+), 0 deletions(-) create mode 100644 arch/ia64/kernel/inst_native.h diff --git a/arch/ia64/kernel/inst_native.h b/arch/ia64/kernel/inst_native.h new file mode 100644 index 0000000..3ba88f4 --- /dev/null +++ b/arch/ia64/kernel/inst_native.h @@ -0,0 +1,153 @@ +/****************************************************************************** + * inst_native.h + * + * Copyright (c) 2008 Isaku Yamahata <yamahata at valinux co jp> + * VA Linux Systems Japan K.K. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + */ + +#define IA64_ASM_PARAVIRTUALIZED_NATIVE + +#undef BR_IF_NATIVE +#define BR_IF_NATIVE(targ, reg, pred) /* nothing */ + +#define __paravirt_switch_to __ia64_switch_to +#define __paravirt_leave_syscall __ia64_leave_syscall +#define __paravirt_work_processed_syscall __ia64_work_processed_syscall +#define __paravirt_leave_kernel __ia64_leave_kernel +#define __paravirt_pending_syscall_end ia64_work_pending_syscall_end +#define __paravirt_work_processed_syscall_target \ + ia64_work_processed_syscall + +#define MOV_FROM_IFA(reg) \ + mov reg = cr.ifa + +#define MOV_FROM_ITIR(reg) \ + mov reg = cr.itir + +#define MOV_FROM_ISR(reg) \ + mov reg = cr.isr + +#define MOV_FROM_IHA(reg) \ + mov reg = cr.iha + +#define MOV_FROM_IPSR(reg) \ + mov reg = cr.ipsr + +#define MOV_FROM_IIM(reg) \ + mov reg = cr.iim + +#define MOV_FROM_IIP(reg) \ + mov reg = cr.iip + +#if 0 +#define MOV_FROM_IVR(reg, clob) \ + mov reg = cr.ivr +#else +#define MOV_FROM_IVR(reg) \ + mov reg = cr.ivr +#endif + +#define MOV_FROM_PSR(pred, reg, clob) \ + (pred) mov reg = psr + +#define MOV_TO_IFA(reg, clob) \ + mov cr.ifa = reg + +#define MOV_TO_ITIR(pred, reg, clob) \ + (pred) mov cr.itir = reg + +#define MOV_TO_IHA(pred, reg, clob) \ + (pred) mov cr.iha = reg + +#define MOV_TO_IPSR(reg, clob) \ + mov cr.ipsr = reg + +#define MOV_TO_IFS(pred, reg, clob) \ + (pred) mov cr.ifs = reg + +#define MOV_TO_IIP(reg, clob) \ + mov cr.iip = reg + +#if 0 +#define MOV_TO_KR(kr, reg, clob0, clob1) \ + mov IA64_KR(kr) = reg +#else +#define MOV_TO_KR(kr, reg) \ + mov IA64_KR(kr) = reg +#endif + +#define ITC_I(pred, reg, clob) \ + (pred) itc.i reg + +#define ITC_D(pred, reg, clob) \ + (pred) itc.d reg + +#define ITC_I_AND_D(pred_i, pred_d, reg, clob) \ + (pred_i) itc.i reg; \ + (pred_d) itc.d reg + +#define THASH(pred, reg0, reg1, clob) \ + (pred) thash reg0 = reg1 + +#define SSM_PSR_IC_AND_DEFAULT_BITS(clob0, clob1) \ + ssm psr.ic | PSR_DEFAULT_BITS \ + ;; \ + srlz.i /* guarantee that interruption collectin is on */ \ + ;; + +#define SSM_PSR_IC_AND_SRLZ_D(clob0, clob1) \ + ssm psr.ic; \ + ;; \ + srlz.d + +#define RSM_PSR_IC(clob) \ + rsm psr.ic + +#define SSM_PSR_I(pred, clob) \ + (pred) ssm psr.i + +#define RSM_PSR_I(pred, clob0, clob1) \ + (pred) rsm psr.i + +#define RSM_PSR_I_IC(clob0, clob1, clob2) \ + rsm psr.i | psr.ic + +#define RSM_PSR_DT \ + rsm psr.dt \ + +#define RSM_PSR_DT_AND_SRLZ_I \ + rsm psr.dt \ + ;; \ + srlz.i + +#define SSM_PSR_DT_AND_SRLZ_I \ + ssm psr.dt \ + ;; \ + srlz.i + +#define BSW_0(clob0, clob1, clob2) \ + bsw.0 + +#define BSW_1(clob0, clob1) \ + bsw.1 + +#define CONVER \ + cover + +#define RFI \ + rfi -- 1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 4/8] ia64/pv_ops: paravirtualize arch/ia64/kernel/switch_leave.S
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/kernel/switch_leave.S | 86 ++++++++++++++++++++++---------------- 1 files changed, 50 insertions(+), 36 deletions(-) diff --git a/arch/ia64/kernel/switch_leave.S b/arch/ia64/kernel/switch_leave.S index b235917..c016556 100644 --- a/arch/ia64/kernel/switch_leave.S +++ b/arch/ia64/kernel/switch_leave.S @@ -44,8 +44,14 @@ #include <asm/pgtable.h> #include <asm/thread_info.h> -#include "minstate.h" +#ifdef __IA64_ASM_PARAVIRTUALIZED_XEN +#include "../xen/inst_xen.h" +#include "../xen/xenminstate.h" +#else +#include "inst_native.h" +#endif +#include "minstate.h" /* * prev_task <- ia64_switch_to(struct task_struct *next) @@ -53,7 +59,8 @@ * called. The code starting at .map relies on this. The rest of the code * doesn't care about the interrupt masking status. */ -GLOBAL_ENTRY(__ia64_switch_to) +GLOBAL_ENTRY(__paravirt_switch_to) + BR_IF_NATIVE(__ia64_switch_to, r22, p7) .prologue alloc r16=ar.pfs,1,0,0,0 DO_SAVE_SWITCH_STACK @@ -77,7 +84,7 @@ GLOBAL_ENTRY(__ia64_switch_to) ;; .done: ld8 sp=[r21] // load kernel stack pointer of new task - mov IA64_KR(CURRENT)=in0 // update "current" application register + MOV_TO_KR(CURRENT, in0) // update "current" application register mov r8=r13 // return pointer to previously running task mov r13=in0 // set "current" pointer ;; @@ -89,25 +96,30 @@ GLOBAL_ENTRY(__ia64_switch_to) br.ret.sptk.many rp // boogie on out in new context .map: - rsm psr.ic // interrupts (psr.i) are already disabled here + RSM_PSR_IC(r25) // interrupts (psr.i) are already disabled here movl r25=PAGE_KERNEL ;; srlz.d or r23=r25,r20 // construct PA | page properties mov r25=IA64_GRANULE_SHIFT<<2 ;; - mov cr.itir=r25 - mov cr.ifa=in0 // VA of next task... + MOV_TO_ITIR(p0, r25, r8) + MOV_TO_IFA(in0, r8) // VA of next task... ;; mov r25=IA64_TR_CURRENT_STACK - mov IA64_KR(CURRENT_STACK)=r26 // remember last page we mapped... + MOV_TO_KR(CURRENT_STACK, r26) // remember last page we mapped... ;; itr.d dtr[r25]=r23 // wire in new mapping... - ssm psr.ic // reenable the psr.ic bit - ;; - srlz.d + SSM_PSR_IC_AND_SRLZ_D(r8, r9) // reenable the psr.ic bit br.cond.sptk .done -END(__ia64_switch_to) +END(__paravirt_switch_to) + +#ifdef IA64_ASM_PARAVIRTUALIZED_XEN +GLOBAL_ENTRY(xen_work_processed_syscall_with_check) + BR_IF_NATIVE(__ia64_work_processed_syscall, r2, p7) + br.cond.sptk xen_work_processed_syscall +END(xen_work_processed_syscall_with_check) +#endif /* IA64_ASM_PARAVIRTUALIZED_XEN */ /* * ia64_leave_syscall(): Same as ia64_leave_kernel, except that it doesn't @@ -153,7 +165,8 @@ END(__ia64_switch_to) * ar.csd: cleared * ar.ssd: cleared */ -GLOBAL_ENTRY(__ia64_leave_syscall) +GLOBAL_ENTRY(__paravirt_leave_syscall) + BR_IF_NATIVE(__ia64_leave_syscall, r22, p7) PT_REGS_UNWIND_INFO(0) /* * work.need_resched etc. mustn't get changed by this CPU before it returns to @@ -177,12 +190,12 @@ GLOBAL_ENTRY(__ia64_leave_syscall) ;; cmp.eq p6,p0=r21,r0 // p6 <- pUStk || (preempt_count == 0) #else /* !CONFIG_PREEMPT */ -(pUStk) rsm psr.i + RSM_PSR_I(pUStk, r2, r18) cmp.eq pLvSys,p0=r0,r0 // pLvSys=1: leave from syscall (pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk #endif -.global __ia64_work_processed_syscall; -__ia64_work_processed_syscall: +.global __paravirt_work_processed_syscall; +__paravirt_work_processed_syscall: adds r2=PT(LOADRS)+16,r12 adds r3=PT(AR_BSPSTORE)+16,r12 adds r18=TI_FLAGS+IA64_TASK_SIZE,r13 @@ -205,7 +218,7 @@ __ia64_work_processed_syscall: (pNonSys) break 0 // bug check: we shouldn't be here if pNonSys is TRUE! ;; invala // M0|1 invalidate ALAT - rsm psr.i | psr.ic // M2 turn off interrupts and interruption collection + RSM_PSR_I_IC(r28, r29, r30) // M2 turn off interrupts and interruption collection cmp.eq p9,p0=r0,r0 // A set p9 to indicate that we should restore cr.ifs ld8 r29=[r2],16 // M0|1 load cr.ipsr @@ -217,7 +230,7 @@ __ia64_work_processed_syscall: (pUStk) add r14=IA64_TASK_THREAD_ON_USTACK_OFFSET,r13 ;; ld8 r26=[r2],PT(B0)-PT(AR_PFS) // M0|1 load ar.pfs -(pKStk) mov r22=psr // M2 read PSR now that interrupts are disabled + MOV_FROM_PSR(pKStk, r22, r21) // M2 read PSR now that interrupts are disabled nop 0 ;; ld8 r21=[r2],PT(AR_RNAT)-PT(B0) // M0|1 load b0 @@ -246,7 +259,7 @@ __ia64_work_processed_syscall: srlz.d // M0 ensure interruption collection is off (for cover) shr.u r18=r19,16 // I0|1 get byte size of existing "dirty" partition - cover // B add current frame into dirty partition & set cr.ifs + COVER // B add current frame into dirty partition & set cr.ifs ;; mov r19=ar.bsp // M2 get new backing store pointer mov f10=f0 // F clear f10 @@ -261,9 +274,10 @@ __ia64_work_processed_syscall: mov.m ar.ssd=r0 // M2 clear ar.ssd mov f11=f0 // F clear f11 br.cond.sptk.many rbs_switch // B -END(__ia64_leave_syscall) +END(__paravirt_leave_syscall) -GLOBAL_ENTRY(__ia64_leave_kernel) +GLOBAL_ENTRY(__paravirt_leave_kernel) + BR_IF_NATIVE(__ia64_leave_kernel, r22, p7) PT_REGS_UNWIND_INFO(0) /* * work.need_resched etc. mustn't get changed by this CPU before it returns to @@ -287,7 +301,7 @@ GLOBAL_ENTRY(__ia64_leave_kernel) ;; cmp.eq p6,p0=r21,r0 // p6 <- pUStk || (preempt_count == 0) #else -(pUStk) rsm psr.i + RSM_PSR_I(pUStk, r17, r31) cmp.eq p0,pLvSys=r0,r0 // pLvSys=0: leave from kernel (pUStk) cmp.eq.unc p6,p0=r0,r0 // p6 <- pUStk #endif @@ -335,7 +349,7 @@ GLOBAL_ENTRY(__ia64_leave_kernel) mov ar.csd=r30 mov ar.ssd=r31 ;; - rsm psr.i | psr.ic // initiate turning off of interrupt and interruption collection + RSM_PSR_I_IC(r23, r22, r25) // initiate turning off of interrupt and interruption collection invala // invalidate ALAT ;; ld8.fill r22=[r2],24 @@ -367,13 +381,13 @@ GLOBAL_ENTRY(__ia64_leave_kernel) mov ar.ccv=r15 ;; ldf.fill f11=[r2] - bsw.0 // switch back to bank 0 (no stop bit required beforehand...) + BSW_0(r2, r3, r15) // switch back to bank 0 (no stop bit required beforehand...) ;; (pUStk) mov r18=IA64_KR(CURRENT)// M2 (12 cycle read latency) adds r16=PT(CR_IPSR)+16,r12 adds r17=PT(CR_IIP)+16,r12 -(pKStk) mov r22=psr // M2 read PSR now that interrupts are disabled + MOV_FROM_PSR(pKStk, r22, r29) // M2 read PSR now that interrupts are disabled nop.i 0 nop.i 0 ;; @@ -421,7 +435,7 @@ GLOBAL_ENTRY(__ia64_leave_kernel) * NOTE: alloc, loadrs, and cover can't be predicated. */ (pNonSys) br.cond.dpnt dont_preserve_current_frame - cover // add current frame into dirty partition and set cr.ifs + COVER // add current frame into dirty partition and set cr.ifs ;; mov r19=ar.bsp // get new backing store pointer rbs_switch: @@ -524,16 +538,16 @@ skip_rbs_switch: (pKStk) dep r29=r22,r29,21,1 // I0 update ipsr.pp with psr.pp (pLvSys)mov r16=r0 // A clear r16 for leave_syscall, no-op otherwise ;; - mov cr.ipsr=r29 // M2 + MOV_TO_IPSR(r29, r25) // M2 mov ar.pfs=r26 // I0 (pLvSys)mov r17=r0 // A clear r17 for leave_syscall, no-op otherwise -(p9) mov cr.ifs=r30 // M2 + MOV_TO_IFS(p9, r30, r25)// M2 mov b0=r21 // I0 (pLvSys)mov r18=r0 // A clear r18 for leave_syscall, no-op otherwise mov ar.fpsr=r20 // M2 - mov cr.iip=r28 // M2 + MOV_TO_IIP(r28, r25) // M2 nop 0 ;; (pUStk) mov ar.rnat=r24 // M2 must happen with RSE in lazy mode @@ -542,7 +556,7 @@ skip_rbs_switch: mov ar.rsc=r27 // M2 mov pr=r31,-1 // I0 - rfi // B + RFI // B /* * On entry: @@ -568,28 +582,28 @@ skip_rbs_switch: #endif br.call.spnt.many rp=schedule .ret9: cmp.eq p6,p0=r0,r0 // p6 <- 1 - rsm psr.i // disable interrupts + RSM_PSR_I(p0, r2, r20) // disable interrupts ;; #ifdef CONFIG_PREEMPT (pKStk) adds r20=TI_PRE_COUNT+IA64_TASK_SIZE,r13 ;; (pKStk) st4 [r20]=r0 // preempt_count() <- 0 #endif -(pLvSys)br.cond.sptk.few ia64_work_pending_syscall_end +(pLvSys)br.cond.sptk.few __paravirt_pending_syscall_end br.cond.sptk.many .work_processed_kernel // re-check .notify: (pUStk) br.call.spnt.many rp=notify_resume_user .ret10: cmp.ne p6,p0=r0,r0 // p6 <- 0 -(pLvSys)br.cond.sptk.few ia64_work_pending_syscall_end +(pLvSys)br.cond.sptk.few __paravirt_pending_syscall_end br.cond.sptk.many .work_processed_kernel // don't re-check -.global ia64_work_pending_syscall_end; -ia64_work_pending_syscall_end: +.global __paravirt_pending_syscall_end; +__paravirt_pending_syscall_end: adds r2=PT(R8)+16,r12 adds r3=PT(R10)+16,r12 ;; ld8 r8=[r2] ld8 r10=[r3] - br.cond.sptk.many ia64_work_processed_syscall // re-check -END(__ia64_leave_kernel) + br.cond.sptk.many __paravirt_work_processed_syscall_target // re-check +END(__paravirt_leave_kernel) -- 1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 5/8] ia64/pv_ops: paravirtualize arch/ia64/kernel/ivt.S.
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/kernel/ivt.S | 159 +++++++++++++++++++++++++----------------------- 1 files changed, 84 insertions(+), 75 deletions(-) diff --git a/arch/ia64/kernel/ivt.S b/arch/ia64/kernel/ivt.S index 34f44d8..1347cda 100644 --- a/arch/ia64/kernel/ivt.S +++ b/arch/ia64/kernel/ivt.S @@ -12,6 +12,13 @@ * * 00/08/23 Asit Mallick <asit.k.mallick at intel.com> TLB handling for SMP * 00/12/20 David Mosberger-Tang <davidm at hpl.hp.com> DTLB/ITLB handler now uses virtual PT. + * + * Copyright (C) 2005 Hewlett-Packard Co + * Dan Magenheimer <dan.magenheimer at hp.com> + * Xen paravirtualization + * Copyright (c) 2008 Isaku Yamahata <yamahata at valinux co jp> + * VA Linux Systems Japan K.K. + * pv_ops. */ /* * This file defines the interruption vector table used by the CPU. @@ -68,6 +75,13 @@ # define DBG_FAULT(i) #endif +#ifdef __IA64_ASM_PARAVIRTUALIZED_XEN +#include "../xen/inst_xen.h" +#include "../xen/xenminstate.h" +#else +#include "inst_native.h" +#endif + #include "minstate.h" #define FAULT(n) \ @@ -102,13 +116,13 @@ ENTRY(vhpt_miss) * - the faulting virtual address uses unimplemented address bits * - the faulting virtual address has no valid page table mapping */ - mov r16=cr.ifa // get address that caused the TLB miss + MOV_FROM_IFA(r16) // get address that caused the TLB miss #ifdef CONFIG_HUGETLB_PAGE movl r18=PAGE_SHIFT - mov r25=cr.itir + MOV_FROM_ITIR(r25) #endif ;; - rsm psr.dt // use physical addressing for data + RSM_PSR_DT // use physical addressing for data mov r31=pr // save the predicate registers mov r19=IA64_KR(PT_BASE) // get page table base address shl r21=r16,3 // shift bit 60 into sign bit @@ -168,21 +182,20 @@ ENTRY(vhpt_miss) dep r21=r19,r20,3,(PAGE_SHIFT-3) // r21=pte_offset(pmd,addr) ;; (p7) ld8 r18=[r21] // read *pte - mov r19=cr.isr // cr.isr bit 32 tells us if this is an insn miss + MOV_FROM_ISR(r19) // cr.isr bit 32 tells us if this is an insn miss ;; (p7) tbit.z p6,p7=r18,_PAGE_P_BIT // page present bit cleared? - mov r22=cr.iha // get the VHPT address that caused the TLB miss + MOV_FROM_IHA(r22) // get the VHPT address that caused the TLB miss ;; // avoid RAW on p7 (p7) tbit.nz.unc p10,p11=r19,32 // is it an instruction TLB miss? dep r23=0,r20,0,PAGE_SHIFT // clear low bits to get page address ;; -(p10) itc.i r18 // insert the instruction TLB entry -(p11) itc.d r18 // insert the data TLB entry + ITC_I_AND_D(p10, p11, r18, r24) // insert the instruction TLB entry and + // insert the data TLB entry (p6) br.cond.spnt.many page_fault // handle bad address/page not present (page fault) - mov cr.ifa=r22 - + MOV_TO_IFA(r22, r24) #ifdef CONFIG_HUGETLB_PAGE -(p8) mov cr.itir=r25 // change to default page-size for VHPT + MOV_TO_ITIR(p8, r25, r24) // change to default page-size for VHPT #endif /* @@ -192,7 +205,7 @@ ENTRY(vhpt_miss) */ adds r24=__DIRTY_BITS_NO_ED|_PAGE_PL_0|_PAGE_AR_RW,r23 ;; -(p7) itc.d r24 + ITC_D(p7, r24, r25) ;; #ifdef CONFIG_SMP /* @@ -234,7 +247,7 @@ ENTRY(vhpt_miss) #endif mov pr=r31,-1 // restore predicate registers - rfi + RFI END(vhpt_miss) .org ia64_ivt+0x400 @@ -248,11 +261,11 @@ ENTRY(itlb_miss) * mode, walk the page table, and then re-execute the PTE read and * go on normally after that. */ - mov r16=cr.ifa // get virtual address + MOV_FROM_IFA(r16) // get virtual address mov r29=b0 // save b0 mov r31=pr // save predicates .itlb_fault: - mov r17=cr.iha // get virtual address of PTE + MOV_FROM_IHA(r17) // get virtual address of PTE movl r30=1f // load nested fault continuation point ;; 1: ld8 r18=[r17] // read *pte @@ -261,7 +274,7 @@ ENTRY(itlb_miss) tbit.z p6,p0=r18,_PAGE_P_BIT // page present bit cleared? (p6) br.cond.spnt page_fault ;; - itc.i r18 + ITC_I(p0, r18, r19) ;; #ifdef CONFIG_SMP /* @@ -278,7 +291,7 @@ ENTRY(itlb_miss) (p7) ptc.l r16,r20 #endif mov pr=r31,-1 - rfi + RFI END(itlb_miss) .org ia64_ivt+0x0800 @@ -292,11 +305,11 @@ ENTRY(dtlb_miss) * mode, walk the page table, and then re-execute the PTE read and * go on normally after that. */ - mov r16=cr.ifa // get virtual address + MOV_FROM_IFA(r16) // get virtual address mov r29=b0 // save b0 mov r31=pr // save predicates dtlb_fault: - mov r17=cr.iha // get virtual address of PTE + MOV_FROM_IHA(r17) // get virtual address of PTE movl r30=1f // load nested fault continuation point ;; 1: ld8 r18=[r17] // read *pte @@ -305,7 +318,7 @@ dtlb_fault: tbit.z p6,p0=r18,_PAGE_P_BIT // page present bit cleared? (p6) br.cond.spnt page_fault ;; - itc.d r18 + ITC_D(p0, r18, r19) ;; #ifdef CONFIG_SMP /* @@ -322,7 +335,7 @@ dtlb_fault: (p7) ptc.l r16,r20 #endif mov pr=r31,-1 - rfi + RFI END(dtlb_miss) .org ia64_ivt+0x0c00 @@ -330,9 +343,9 @@ END(dtlb_miss) // 0x0c00 Entry 3 (size 64 bundles) Alt ITLB (19) ENTRY(alt_itlb_miss) DBG_FAULT(3) - mov r16=cr.ifa // get address that caused the TLB miss + MOV_FROM_IFA(r16) // get address that caused the TLB miss movl r17=PAGE_KERNEL - mov r21=cr.ipsr + MOV_FROM_IPSR(r21) movl r19=(((1 << IA64_MAX_PHYS_BITS) - 1) & ~0xfff) mov r31=pr ;; @@ -341,9 +354,9 @@ ENTRY(alt_itlb_miss) ;; cmp.gt p8,p0=6,r22 // user mode ;; -(p8) thash r17=r16 + THASH(p8, r17, r16, r23) ;; -(p8) mov cr.iha=r17 + MOV_TO_IHA(p8, r17, r23) (p8) mov r29=b0 // save b0 (p8) br.cond.dptk .itlb_fault #endif @@ -358,9 +371,9 @@ ENTRY(alt_itlb_miss) or r19=r19,r18 // set bit 4 (uncached) if the access was to region 6 (p8) br.cond.spnt page_fault ;; - itc.i r19 // insert the TLB entry + ITC_I(p0, r19, r18) // insert the TLB entry mov pr=r31,-1 - rfi + RFI END(alt_itlb_miss) .org ia64_ivt+0x1000 @@ -368,11 +381,11 @@ END(alt_itlb_miss) // 0x1000 Entry 4 (size 64 bundles) Alt DTLB (7,46) ENTRY(alt_dtlb_miss) DBG_FAULT(4) - mov r16=cr.ifa // get address that caused the TLB miss + MOV_FROM_IFA(r16) // get address that caused the TLB miss movl r17=PAGE_KERNEL - mov r20=cr.isr + MOV_FROM_ISR(r20) movl r19=(((1 << IA64_MAX_PHYS_BITS) - 1) & ~0xfff) - mov r21=cr.ipsr + MOV_FROM_IPSR(r21) mov r31=pr mov r24=PERCPU_ADDR ;; @@ -381,9 +394,9 @@ ENTRY(alt_dtlb_miss) ;; cmp.gt p8,p0=6,r22 // access to region 0-5 ;; -(p8) thash r17=r16 + THASH(p8, r17, r16, r25) ;; -(p8) mov cr.iha=r17 + MOV_TO_IHA(r17, r25) (p8) mov r29=b0 // save b0 (p8) br.cond.dptk dtlb_fault #endif @@ -402,7 +415,7 @@ ENTRY(alt_dtlb_miss) tbit.nz p9,p0=r20,IA64_ISR_NA_BIT // is non-access bit on? ;; (p10) sub r19=r19,r26 -(p10) mov cr.itir=r25 + MOV_TO_ITIR(p10, r25, r24) cmp.ne p8,p0=r0,r23 (p9) cmp.eq.or.andcm p6,p7=IA64_ISR_CODE_LFETCH,r22 // check isr.code field (p12) dep r17=-1,r17,4,1 // set ma=UC for region 6 addr @@ -413,9 +426,9 @@ ENTRY(alt_dtlb_miss) or r19=r19,r17 // insert PTE control bits into r19 (p6) mov cr.ipsr=r21 ;; -(p7) itc.d r19 // insert the TLB entry + ITC_D(p7, r19, r18) // insert the TLB entry mov pr=r31,-1 - rfi + RFI END(alt_dtlb_miss) .org ia64_ivt+0x1400 @@ -444,10 +457,10 @@ ENTRY(nested_dtlb_miss) * * Clobbered: b0, r18, r19, r21, r22, psr.dt (cleared) */ - rsm psr.dt // switch to using physical data addressing + RSM_PSR_DT_AND_SRLZ_I // switch to using physical data addressing mov r19=IA64_KR(PT_BASE) // get the page table base address shl r21=r16,3 // shift bit 60 into sign bit - mov r18=cr.itir + MOV_FROM_ITIR(r18) ;; shr.u r17=r16,61 // get the region number into r17 extr.u r18=r18,2,6 // get the faulting page size @@ -510,21 +523,15 @@ END(ikey_miss) //----------------------------------------------------------------------------------- // call do_page_fault (predicates are in r31, psr.dt may be off, r16 is faulting address) ENTRY(page_fault) - ssm psr.dt - ;; - srlz.i + SSM_PSR_DT_AND_SRLZ_I ;; SAVE_MIN_WITH_COVER alloc r15=ar.pfs,0,0,3,0 - mov out0=cr.ifa - mov out1=cr.isr + MOV_FROM_IFA(out0) + MOV_FROM_ISR(out1) + SSM_PSR_IC_AND_DEFAULT_BITS(r14, r3) adds r3=8,r2 // set up second base pointer - ;; - ssm psr.ic | PSR_DEFAULT_BITS - ;; - srlz.i // guarantee that interruption collectin is on - ;; -(p15) ssm psr.i // restore psr.i + SSM_PSR_I(p15, r14) // restore psr.i movl r14=ia64_leave_kernel ;; SAVE_REST @@ -556,10 +563,10 @@ ENTRY(dirty_bit) * page table TLB entry isn't present, we take a nested TLB miss hit where we look * up the physical address of the L3 PTE and then continue at label 1 below. */ - mov r16=cr.ifa // get the address that caused the fault + MOV_FROM_IFA(r16) // get the address that caused the fault movl r30=1f // load continuation point in case of nested fault ;; - thash r17=r16 // compute virtual address of L3 PTE + THASH(p0, r17, r16, r18) // compute virtual address of L3 PTE mov r29=b0 // save b0 in case of nested fault mov r31=pr // save pr #ifdef CONFIG_SMP @@ -576,7 +583,7 @@ ENTRY(dirty_bit) ;; (p6) cmp.eq p6,p7=r26,r18 // Only compare if page is present ;; -(p6) itc.d r25 // install updated PTE + ITC_D(p6, r25, r18) // install updated PTE ;; /* * Tell the assemblers dependency-violation checker that the above "itc" instructions @@ -602,7 +609,7 @@ ENTRY(dirty_bit) itc.d r18 // install updated PTE #endif mov pr=r31,-1 // restore pr - rfi + RFI END(dirty_bit) .org ia64_ivt+0x2400 @@ -611,7 +618,7 @@ END(dirty_bit) ENTRY(iaccess_bit) DBG_FAULT(9) // Like Entry 8, except for instruction access - mov r16=cr.ifa // get the address that caused the fault + MOV_FROM_IFA(r16) // get the address that caused the fault movl r30=1f // load continuation point in case of nested fault mov r31=pr // save predicates #ifdef CONFIG_ITANIUM @@ -626,7 +633,7 @@ ENTRY(iaccess_bit) (p6) mov r16=r18 // if so, use cr.iip instead of cr.ifa #endif /* CONFIG_ITANIUM */ ;; - thash r17=r16 // compute virtual address of L3 PTE + THASH(p0, r17, r16, r18) // compute virtual address of L3 PTE mov r29=b0 // save b0 in case of nested fault) #ifdef CONFIG_SMP mov r28=ar.ccv // save ar.ccv @@ -642,7 +649,7 @@ ENTRY(iaccess_bit) ;; (p6) cmp.eq p6,p7=r26,r18 // Only if page present ;; -(p6) itc.i r25 // install updated PTE + ITC_I(p6, r25, r26) // install updated PTE ;; /* * Tell the assemblers dependency-violation checker that the above "itc" instructions @@ -668,7 +675,7 @@ ENTRY(iaccess_bit) itc.i r18 // install updated PTE #endif /* !CONFIG_SMP */ mov pr=r31,-1 - rfi + RFI END(iaccess_bit) .org ia64_ivt+0x2800 @@ -677,10 +684,10 @@ END(iaccess_bit) ENTRY(daccess_bit) DBG_FAULT(10) // Like Entry 8, except for data access - mov r16=cr.ifa // get the address that caused the fault + MOV_FROM_IFA(r16) // get the address that caused the fault movl r30=1f // load continuation point in case of nested fault ;; - thash r17=r16 // compute virtual address of L3 PTE + THASH(p0, r17, r16, r18) // compute virtual address of L3 PTE mov r31=pr mov r29=b0 // save b0 in case of nested fault) #ifdef CONFIG_SMP @@ -697,7 +704,7 @@ ENTRY(daccess_bit) ;; (p6) cmp.eq p6,p7=r26,r18 // Only if page is present ;; -(p6) itc.d r25 // install updated PTE + ITC_D(p6, r25, r26) // install updated PTE /* * Tell the assemblers dependency-violation checker that the above "itc" instructions * cannot possibly affect the following loads: @@ -721,7 +728,7 @@ ENTRY(daccess_bit) #endif mov b0=r29 // restore b0 mov pr=r31,-1 - rfi + RFI END(daccess_bit) .org ia64_ivt+0x2c00 @@ -745,10 +752,10 @@ ENTRY(break_fault) */ DBG_FAULT(11) mov.m r16=IA64_KR(CURRENT) // M2 r16 <- current task (12 cyc) - mov r29=cr.ipsr // M2 (12 cyc) + MOV_FROM_IPSR(r29) // M2 (12 cyc) mov r31=pr // I0 (2 cyc) - mov r17=cr.iim // M2 (2 cyc) + MOV_FROM_IIM(r17) // M2 (2 cyc) mov.m r27=ar.rsc // M2 (12 cyc) mov r18=__IA64_BREAK_SYSCALL // A @@ -767,7 +774,7 @@ ENTRY(break_fault) nop.m 0 movl r30=sys_call_table // X - mov r28=cr.iip // M2 (2 cyc) + MOV_FROM_IIP(r28) // M2 (2 cyc) cmp.eq p0,p7=r18,r17 // I0 is this a system call? (p7) br.cond.spnt non_syscall // B no -> // @@ -831,10 +838,10 @@ ENTRY(break_fault) 1: mov ar.rsc=0x3 // M2 set eager mode, pl 0, LE, loadrs=0 nop 0 - bsw.1 // B (6 cyc) regs are saved, switch to bank 1 + BSW_1(r2, r14) // B (6 cyc) regs are saved, switch to bank 1 ;; - ssm psr.ic | PSR_DEFAULT_BITS // M2 now it's safe to re-enable intr.-collection + SSM_PSR_IC_AND_DEFAULT_BITS(r3, r16) // M2 now it's safe to re-enable intr.-collection movl r3=ia64_ret_from_syscall // X ;; @@ -842,7 +849,7 @@ ENTRY(break_fault) mov rp=r3 // I0 set the real return addr (p10) br.cond.spnt.many ia64_ret_from_syscall // B return if bad call-frame or r15 is a NaT -(p15) ssm psr.i // M2 restore psr.i + SSM_PSR_I(p15, r16) // M2 restore psr.i (p14) br.call.sptk.many b6=b6 // B invoke syscall-handker (ignore return addr) br.cond.spnt.many ia64_trace_syscall // B do syscall-tracing thingamagic // NOT REACHED @@ -866,7 +873,7 @@ ENTRY(interrupt) mov r31=pr // prepare to save predicates ;; SAVE_MIN_WITH_COVER // uses r31; defines r2 and r3 - ssm psr.ic | PSR_DEFAULT_BITS + SSM_PSR_IC_AND_DEFAULT_BITS(r3, r14) ;; adds r3=8,r2 // set up second base pointer for SAVE_REST srlz.i // ensure everybody knows psr.ic is back on @@ -875,7 +882,7 @@ ENTRY(interrupt) ;; MCA_RECOVER_RANGE(interrupt) alloc r14=ar.pfs,0,0,2,0 // must be first in an insn group - mov out0=cr.ivr // pass cr.ivr as first arg + MOV_FROM_IVR(out0) // pass cr.ivr as first arg add out1=16,sp // pass pointer to pt_regs as second arg ;; srlz.d // make sure we see the effect of cr.ivr @@ -944,6 +951,7 @@ END(interrupt) * - ar.fpsr: set to kernel settings * - b6: preserved (same as on entry) */ +#ifdef __IA64_ASM_PARAVIRTUALIZED_NATIVE GLOBAL_ENTRY(ia64_syscall_setup) #if PT(B6) != 0 # error This code assumes that b6 is the first field in pt_regs. @@ -1035,6 +1043,7 @@ GLOBAL_ENTRY(ia64_syscall_setup) (p10) mov r8=-EINVAL br.ret.sptk.many b7 END(ia64_syscall_setup) +#endif /* __IA64_ASM_PARAVIRTUALIZED_NATIVE */ .org ia64_ivt+0x3c00 ///////////////////////////////////////////////////////////////////////////////////////// @@ -1181,10 +1190,10 @@ ENTRY(dispatch_to_fault_handler) SAVE_MIN_WITH_COVER_R19 alloc r14=ar.pfs,0,0,5,0 mov out0=r15 - mov out1=cr.isr - mov out2=cr.ifa - mov out3=cr.iim - mov out4=cr.itir + MOV_FROM_ISR(out1) + MOV_FROM_IFA(out2) + MOV_FROM_IIM(out3) + MOV_FROM_ITIR(out4) ;; ssm psr.ic | PSR_DEFAULT_BITS ;; @@ -1255,8 +1264,8 @@ END(iaccess_rights) // 0x5300 Entry 23 (size 16 bundles) Data Access Rights (14,53) ENTRY(daccess_rights) DBG_FAULT(23) - mov r16=cr.ifa - rsm psr.dt + MOV_FROM_IFA(r16) + RSM_PSR_DT mov r31=pr ;; srlz.d @@ -1352,7 +1361,7 @@ ENTRY(speculation_vector) mov cr.ipsr=r16 ;; - rfi // and go back + RFI END(speculation_vector) .org ia64_ivt+0x5800 @@ -1506,7 +1515,7 @@ ENTRY(ia32_intercept) (p6) br.cond.spnt 1f // eflags.ac bit didn't change ;; mov pr=r31,-1 // restore predicate registers - rfi + RFI 1: #endif // CONFIG_IA32_SUPPORT -- 1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 6/8] ia64/pv_ops xen: xen paravirtualization of hand written assembly code. inst_xen.h
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/xen/inst_xen.h | 307 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 307 insertions(+), 0 deletions(-) create mode 100644 arch/ia64/xen/inst_xen.h diff --git a/arch/ia64/xen/inst_xen.h b/arch/ia64/xen/inst_xen.h new file mode 100644 index 0000000..c0bd736 --- /dev/null +++ b/arch/ia64/xen/inst_xen.h @@ -0,0 +1,307 @@ +/****************************************************************************** + * inst_xen.h + * + * Copyright (c) 2008 Isaku Yamahata <yamahata at valinux co jp> + * VA Linux Systems Japan K.K. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + */ + +#define IA64_ASM_PARAVIRTUALIZED_XEN + +#define ia64_ivt xen_ivt + +#define __paravirt_switch_to xen_switch_to +#define __paravirt_leave_syscall xen_leave_syscall +#define __paravirt_work_processed_syscall xen_work_processed_syscall +#define __paravirt_leave_kernel xen_leave_kernel +#define __paravirt_pending_syscall_end xen_work_pending_syscall_end +#define __paravirt_work_processed_syscall_target \ + xen_work_processed_syscall + +#define MOV_FROM_IFA(reg) \ + movl reg = XSI_IFA; \ + ;; \ + ld8 reg = [reg] + +#define MOV_FROM_ITIR(reg) \ + movl reg = XSI_ITIR; \ + ;; \ + ld8 reg = [reg] + +#define MOV_FROM_ISR(reg) \ + movl reg = XSI_ISR; \ + ;; \ + ld8 reg = [reg] + +#define MOV_FROM_IHA(reg) \ + movl reg = XSI_IHA; \ + ;; \ + ld8 reg = [reg] + +#define MOV_FROM_IPSR(reg) \ + movl reg = XSI_IPSR; \ + ;; \ + ld8 reg = [reg] + +#define MOV_FROM_IIM(reg) \ + movl reg = XSI_IIM; \ + ;; \ + ld8 reg = [reg] + +#define MOV_FROM_IIP(reg) \ + movl reg = XSI_IIP; \ + ;; \ + ld8 reg = [reg] + +#if 0 +#define MOV_FROM_IVR(reg, clob) \ + clob = r8; \ + ;; \ + XEN_HYPER_GET_IVR \ + ;; \ + mov reg = r8; \ + ;; \ + mov r8 = clob +#else +#define MOV_FROM_IVR(reg) \ + ;; \ + XEN_HYPER_GET_IVR \ + ;; \ + mov reg = r8 +#endif + +#define MOV_FROM_PSR(pred, reg, clob) \ + (pred) mov clob = r8; \ + (pred) XEN_HYPER_GET_PSR; \ + ;; \ + (pred) mov reg = r8; \ + (pred) mov r8 = clob; \ + ;; + + +#define MOV_TO_IFA(reg, clob) \ + movl clob = XSI_IFA; \ + ;; \ + st8 [clob] = reg \ + +#define MOV_TO_ITIR(pred, reg, clob) \ + (pred) movl clob = XSI_ITIR; \ + ;; \ + (pred) st8 [clob] = reg + +#define MOV_TO_IHA(pred, reg, clob) \ + (pred) movl clob = XSI_IHA; \ + ;; \ + (pred) st8 [clob] = reg + +#define MOV_TO_IPSR(reg, clob) \ + movl clob = XSI_IPSR; \ + ;; \ + st8 [clob] = r29; \ + ;; + +#define MOV_TO_IFS(pred, reg, clob) \ + (pred) movl clob = XSI_IFS; \ + ;; \ + (pred) st8 [clob] = reg; \ + ;; + +#define MOV_TO_IIP(reg, clob) \ + movl clob = XSI_IIP; \ + ;; \ + st8 [clob] = reg + +#if 0 +#define MOV_TO_KR(kr, reg, clob0, clob1) \ + mov clob0 = r8; \ + mov clob1 = r9; \ + ;; \ + mov r8 = IA64_KR_ ## kr ; \ + mov r9 = reg; \ + ;; \ + XEN_HYPER_SET_KR; \ + mov r8 = clob0; \ + mov r9 = clob1 +#else +#define MOV_TO_KR(kr, reg) \ + mov r8 = IA64_KR_ ## kr; \ + mov r9 = reg; \ + ;; \ + XEN_HYPER_SET_KR +#endif + + +#define ITC_I(pred, reg, clob) \ + (pred) mov clob = r8; \ + (pred) mov r8 = reg; \ + ;; \ + (pred) XEN_HYPER_ITC_I; \ + ;; \ + (pred) mov r8 = clob; \ + ;; + +#define ITC_D(pred, reg, clob) \ + (pred) mov clob = r8; \ + (pred) mov r8 = reg; \ + ;; \ + (pred) XEN_HYPER_ITC_D; \ + ;; \ + (pred) mov r8 = clob; \ + ;; + +#define ITC_I_AND_D(pred_i, pred_d, reg, clob) \ + mov clob = r8; \ + mov r8 = reg; \ + ;; \ + (pred_i) XEN_HYPER_ITC_I; \ + ;; \ + (pred_d) XEN_HYPER_ITC_D; \ + ;; \ + mov r8 = clob; \ + ;; + +#define THASH(pred, reg0, reg1, clob) \ + (pred) mov clob = r8; \ + (pred) mov r8 = reg1; \ + (pred) XEN_HYPER_THASH; \ + ;; \ + (pred) mov reg0 = r8; \ + (pred) mov r8 = clob; \ + ;; + +#define SSM_PSR_IC_AND_DEFAULT_BITS(clob0, clob1) \ + mov clob0 = 1; \ + movl clob1 = XSI_PSR_IC; \ + ;; \ + st4 [clob1] = clob0 \ + ;; + +#define SSM_PSR_IC_AND_SRLZ_D(clob0, clob1) \ + ;; \ + srlz.d; \ + mov clob1 = 1; \ + movl clob0=XSI_PSR_IC; \ + ;; \ + st4 [clob0] = clob1 + +#define RSM_PSR_IC(clob) \ + movl clob = XSI_PSR_IC; \ + ;; \ + st4 [clob] = r0; \ + ;; + +/* pred will be clobbered */ +#define MASK_TO_PEND_OFS (-1) +#define SSM_PSR_I(pred, clob) \ + (pred) movl clob = XSI_PSR_I_ADDR \ + ;; \ + (pred) ld8 clob = [clob] \ + ;; \ + /* if (pred) vpsr.i = 1 */ \ + /* if (pred) (vcpu->vcpu_info->evtchn_upcall_mask)=0 */ \ + (pred) st1 [clob] = r0, MASK_TO_PEND_OFS \ + ;; \ + /* if (vcpu->vcpu_info->evtchn_upcall_pending) */ \ + (pred) ld1 clob = [clob] \ + ;; \ + (pred) cmp.ne pred, p0 = clob, r0 \ + ;; \ + (pred) XEN_HYPER_SSM_I /* do areal ssm psr.i */ + +#define RSM_PSR_I(pred, clob0, clob1) \ + movl clob0 = XSI_PSR_I_ADDR; \ + mov clob1 = 1; \ + ;; \ + ld8 clob0 = [clob0]; \ + ;; \ + (pred) st1 [clob0] = clob1 + +#define RSM_PSR_I_IC(clob0, clob1, clob2) \ + movl clob0 = XSI_PSR_I_ADDR; \ + movl clob1 = XSI_PSR_IC; \ + ;; \ + ld8 clob0 = [clob0]; \ + mov clob2 = 1; \ + ;; \ + /* note: clears both vpsr.i and vpsr.ic! */ \ + st1 [clob0] = clob2; \ + st4 [clob1] = r0; \ + ;; + +#define RSM_PSR_DT \ + XEN_HYPER_RSM_PSR_DT + +#define RSM_PSR_DT_AND_SRLZ_I \ + XEN_HYPER_RSM_PSR_DT + +#define SSM_PSR_DT_AND_SRLZ_I \ + XEN_HYPER_SSM_PSR_DT + +#define BSW_0(clob0, clob1, clob2) \ + ;; \ + /* r16-r31 all now hold bank1 values */ \ + mov clob2 = ar.unat; \ + movl clob0 = XSI_BANK1_R16; \ + movl clob1 = XSI_BANK1_R16 + 8; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r16, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r17, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r18, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r19, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r20, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r21, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r22, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r23, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r24, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r25, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r26, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r27, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r28, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r29, 16; \ + ;; \ + .mem.offset 0, 0; st8.spill [clob0] = r30, 16; \ + .mem.offset 8, 0; st8.spill [clob1] = r31, 16; \ + ;; \ + mov clob1 = ar.unat; \ + movl clob0 = XSI_B1NAT; \ + ;; \ + st8 [clob0] = clob1; \ + mov ar.unat = clob2; \ + movl clob0 = XSI_BANKNUM; \ + ;; \ + st4 [clob0] = r0 + + /* xen_bsw1 clobbers clob1 */ +#define BSW_1(clob0, clob1) \ + mov clob0 = b0; \ + br.call.sptk b0 = xen_bsw1; \ + ;; \ + mov b0=clob0; \ + ;; + +#define COVER \ + XEN_HYPER_COVER + +#define RFI \ + XEN_HYPER_RFI; \ + dv_serialize_data -- 1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 7/8] ia64/xen: paravirtualize minstate.h, DO_SAVE_MIN.
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/xen/xenminstate.h | 148 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 148 insertions(+), 0 deletions(-) create mode 100644 arch/ia64/xen/xenminstate.h diff --git a/arch/ia64/xen/xenminstate.h b/arch/ia64/xen/xenminstate.h new file mode 100644 index 0000000..cdaf910 --- /dev/null +++ b/arch/ia64/xen/xenminstate.h @@ -0,0 +1,148 @@ +#include <asm/cache.h> + +#ifdef CONFIG_XEN +#include "../kernel/entry.h" +#else +#include "entry.h" +#endif + +#ifdef __IA64_ASM_PARAVIRTUALIZED_XEN +/* + * DO_SAVE_MIN switches to the kernel stacks (if necessary) and saves + * the minimum state necessary that allows us to turn psr.ic back + * on. + * + * Assumed state upon entry: + * psr.ic: off + * r31: contains saved predicates (pr) + * + * Upon exit, the state is as follows: + * psr.ic: off + * r2 = points to &pt_regs.r16 + * r8 = contents of ar.ccv + * r9 = contents of ar.csd + * r10 = contents of ar.ssd + * r11 = FPSR_DEFAULT + * r12 = kernel sp (kernel virtual address) + * r13 = points to current task_struct (kernel virtual address) + * p15 = TRUE if psr.i is set in cr.ipsr + * predicate registers (other than p2, p3, and p15), b6, r3, r14, r15: + * preserved + * CONFIG_XEN note: p6/p7 are not preserved + * + * Note that psr.ic is NOT turned on by this macro. This is so that + * we can pass interruption state as arguments to a handler. + */ +#define DO_SAVE_MIN(COVER,SAVE_IFS,EXTRA) \ + mov r16=IA64_KR(CURRENT); /* M */ \ + mov r27=ar.rsc; /* M */ \ + mov r20=r1; /* A */ \ + mov r25=ar.unat; /* M */ \ + /* mov r29=cr.ipsr; /* M */ \ + movl r29=XSI_IPSR;; \ + ld8 r29=[r29];; \ + mov r26=ar.pfs; /* I */ \ + /* mov r28=cr.iip; /* M */ \ + movl r28=XSI_IIP;; \ + ld8 r28=[r28];; \ + mov r21=ar.fpsr; /* M */ \ + COVER; /* B;; (or nothing) */ \ + ;; \ + adds r16=IA64_TASK_THREAD_ON_USTACK_OFFSET,r16; \ + ;; \ + ld1 r17=[r16]; /* load current->thread.on_ustack flag */ \ + st1 [r16]=r0; /* clear current->thread.on_ustack flag */ \ + adds r1=-IA64_TASK_THREAD_ON_USTACK_OFFSET,r16 \ + /* switch from user to kernel RBS: */ \ + ;; \ + invala; /* M */ \ + /* SAVE_IFS; /* see xen special handling below */ \ + cmp.eq pKStk,pUStk=r0,r17; /* are we in kernel mode already? */ \ + ;; \ +(pUStk) mov ar.rsc=0; /* set enforced lazy mode, pl 0, little-endian, loadrs=0 */ \ + ;; \ +(pUStk) mov.m r24=ar.rnat; \ +(pUStk) addl r22=IA64_RBS_OFFSET,r1; /* compute base of RBS */ \ +(pKStk) mov r1=sp; /* get sp */ \ + ;; \ +(pUStk) lfetch.fault.excl.nt1 [r22]; \ +(pUStk) addl r1=IA64_STK_OFFSET-IA64_PT_REGS_SIZE,r1; /* compute base of memory stack */ \ +(pUStk) mov r23=ar.bspstore; /* save ar.bspstore */ \ + ;; \ +(pUStk) mov ar.bspstore=r22; /* switch to kernel RBS */ \ +(pKStk) addl r1=-IA64_PT_REGS_SIZE,r1; /* if in kernel mode, use sp (r12) */ \ + ;; \ +(pUStk) mov r18=ar.bsp; \ +(pUStk) mov ar.rsc=0x3; /* set eager mode, pl 0, little-endian, loadrs=0 */ \ + adds r17=2*L1_CACHE_BYTES,r1; /* really: biggest cache-line size */ \ + adds r16=PT(CR_IPSR),r1; \ + ;; \ + lfetch.fault.excl.nt1 [r17],L1_CACHE_BYTES; \ + st8 [r16]=r29; /* save cr.ipsr */ \ + ;; \ + lfetch.fault.excl.nt1 [r17]; \ + tbit.nz p15,p0=r29,IA64_PSR_I_BIT; \ + mov r29=b0 \ + ;; \ + adds r16=PT(R8),r1; /* initialize first base pointer */ \ + adds r17=PT(R9),r1; /* initialize second base pointer */ \ +(pKStk) mov r18=r0; /* make sure r18 isn't NaT */ \ + ;; \ +.mem.offset 0,0; st8.spill [r16]=r8,16; \ +.mem.offset 8,0; st8.spill [r17]=r9,16; \ + ;; \ +.mem.offset 0,0; st8.spill [r16]=r10,24; \ +.mem.offset 8,0; st8.spill [r17]=r11,24; \ + ;; \ + /* xen special handling for possibly lazy cover */ \ + movl r8=XSI_PRECOVER_IFS; \ + ;; \ + ld8 r30=[r8]; \ + ;; \ + st8 [r16]=r28,16; /* save cr.iip */ \ + st8 [r17]=r30,16; /* save cr.ifs */ \ +(pUStk) sub r18=r18,r22; /* r18=RSE.ndirty*8 */ \ + mov r8=ar.ccv; \ + mov r9=ar.csd; \ + mov r10=ar.ssd; \ + movl r11=FPSR_DEFAULT; /* L-unit */ \ + ;; \ + st8 [r16]=r25,16; /* save ar.unat */ \ + st8 [r17]=r26,16; /* save ar.pfs */ \ + shl r18=r18,16; /* compute ar.rsc to be used for "loadrs" */ \ + ;; \ + st8 [r16]=r27,16; /* save ar.rsc */ \ +(pUStk) st8 [r17]=r24,16; /* save ar.rnat */ \ +(pKStk) adds r17=16,r17; /* skip over ar_rnat field */ \ + ;; /* avoid RAW on r16 & r17 */ \ +(pUStk) st8 [r16]=r23,16; /* save ar.bspstore */ \ + st8 [r17]=r31,16; /* save predicates */ \ +(pKStk) adds r16=16,r16; /* skip over ar_bspstore field */ \ + ;; \ + st8 [r16]=r29,16; /* save b0 */ \ + st8 [r17]=r18,16; /* save ar.rsc value for "loadrs" */ \ + cmp.eq pNonSys,pSys=r0,r0 /* initialize pSys=0, pNonSys=1 */ \ + ;; \ +.mem.offset 0,0; st8.spill [r16]=r20,16; /* save original r1 */ \ +.mem.offset 8,0; st8.spill [r17]=r12,16; \ + adds r12=-16,r1; /* switch to kernel memory stack (with 16 bytes of scratch) */ \ + ;; \ +.mem.offset 0,0; st8.spill [r16]=r13,16; \ +.mem.offset 8,0; st8.spill [r17]=r21,16; /* save ar.fpsr */ \ + mov r13=IA64_KR(CURRENT); /* establish `current' */ \ + ;; \ +.mem.offset 0,0; st8.spill [r16]=r15,16; \ +.mem.offset 8,0; st8.spill [r17]=r14,16; \ + ;; \ +.mem.offset 0,0; st8.spill [r16]=r2,16; \ +.mem.offset 8,0; st8.spill [r17]=r3,16; \ + ;; \ + EXTRA; \ + mov r2=b0; br.call.sptk b0=xen_bsw1;; mov b0=r2; \ + adds r2=IA64_PT_REGS_R16_OFFSET,r1; \ + ;; \ + movl r1=__gp; /* establish kernel global pointer */ \ + ;; \ + /*bsw.1;*/ /* switch back to bank 1 (must be last in insn group) */ \ + ;; +#endif /* __IA64_ASM_PARAVIRTUALIZED_XEN */ -- 1.5.3
Isaku Yamahata
2008-Feb-26 13:58 UTC
[PATCH 8/8] ia64/pv_ops/xen: multi compiling switch_leave.S and ivt.S for xen.
Signed-off-by: Isaku Yamahata <yamahata at valinux.co.jp> --- arch/ia64/kernel/Makefile | 24 ++++++++++++++++++++++++ 1 files changed, 24 insertions(+), 0 deletions(-) diff --git a/arch/ia64/kernel/Makefile b/arch/ia64/kernel/Makefile index 9281bf6..05ea5f8 100644 --- a/arch/ia64/kernel/Makefile +++ b/arch/ia64/kernel/Makefile @@ -76,3 +76,27 @@ $(obj)/gate-data.o: $(obj)/gate.so # AFLAGS_ivt.o += -D__IA64_ASM_PARAVIRTUALIZED_NATIVE AFLAGS_switch_leave.o += -D__IA64_ASM_PARAVIRTUALIZED_NATIVE + +# +# xenivt.o +# +obj-$(CONFIG_XEN) += xen_ivt.o +ifeq ($(CONFIG_XEN), y) +targets += xen_ivt.o +$(obj)/build-in.o: xen_ivt.o +endif +AFLAGS_xen_ivt.o += -D__IA64_ASM_PARAVIRTUALIZED_XEN +$(obj)/xen_ivt.o: $(src)/ivt.S FORCE + $(call if_changed_dep,as_o_S) + +# +# xen_switch_leave.o +# +obj-$(CONFIG_XEN) += xen_switch_leave.o +ifeq ($(CONFIG_XEN), y) +targets += xen_switch_leave.o +$(obj)/build-in.o: xen_switch_leave.o +endif +AFLAGS_xen_switch_leave.o += -D__IA64_ASM_PARAVIRTUALIZED_XEN +$(obj)/xen_switch_leave.o: $(src)/switch_leave.S FORCE + $(call if_changed_dep,as_o_S) -- 1.5.3
Possibly Parallel Threads
- [PATCH 0/8] RFC: ia64/xen TAKE 2: paravirtualization of hand written assembly code
- [PATCH 0/4] ia64/xen: paravirtualization of hand written assembly code
- [PATCH 0/4] ia64/xen: paravirtualization of hand written assembly code
- [PATCH 00/15] ia64/pv_ops take 5
- [PATCH 00/15] ia64/pv_ops take 5