I''m having trouble switching into user mode with my domU kernel -- everytime I call __HYPERVISOR_iret, the hypervisor calls into my kernel via the registered callback handler. None of the bits in the event channel pending arrays are set. If I return via the iret call, the hypervisor immediately calls back into the handler. The user RIP never changes, so it is clearly not making forward progress. The stackframe should be correctly setup with the flags, rip, cs, rflags, rsp, and ss values and the currently installed LDT has slots for the code and data segments that the user will use. Am I not jumping into user space correctly? Is there something else that my code should do to make the transition? On a documentation question, is __HYPERVISOR_iret documented anywhere? The Xen API* doesn''t seem to include any details on it. I deduced the stack arguments from looking at the code in the Linux xen architecture and the xen-x86_64.h header. This comment in the header, however, doesn''t seem to match the behaviour that my code seems to experience: * If flags contains VGCF_in_syscall: * Restore RAX, RIP, RFLAGS, RSP. * Discard R11, RCX, CS, SS. * Otherwise: * Restore RAX, R11, RCX, CS:RIP, RFLAGS, SS:RSP. I am passing flags==0 and only push the values for flags, CS:RIP, RFLAGS, and SS:RSP. If I push values for RAX, R11, and RCX on the stack the kernel ends up in all sorts of weird places rather than my intended RIP and RSP. Thanks! -- Trammell *: http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/interface/interface.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net> wrote:> Am I not jumping into user space correctly? Is there something > else that my code should do to make the transition?It sounds like event delivery is masked before the iret, and then you have IF set in the RFLAGS value in the iret frame, which causes event delivery to be unmasked during iret.> I am passing flags==0 and only push the values for flags, CS:RIP, > RFLAGS, and SS:RSP. If I push values for RAX, R11, and RCX on the > stack the kernel ends up in all sorts of weird places rather than > my intended RIP and RSP.That doesn''t make sense. The iret implementation (for an x86/64 guest) always expects RAX/R11/RCX on the stack. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Trammell Hudson
2007-Oct-01 18:43 UTC
Re: [Xen-devel] Switching to user mode from domU kernel
On Mon, Oct 01, 2007 at 06:24:07AM +0100, Keir Fraser wrote:> On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net> wrote: > > Am I not jumping into user space correctly? Is there something > > else that my code should do to make the transition? > > It sounds like event delivery is masked before the iret, and then you have > IF set in the RFLAGS value in the iret frame, which causes event delivery to > be unmasked during iret.That makes sense, although the event channel pending array is all zero and the shared_info->vcpu_info[0].evtchn_upcall_pending value is also zero. If I do not set IF in RFLAGS, Xen crashes when it tries to handle a GPF: (XEN) traps.c:1587: GPF (001c): ffff830000165555 -> ffff83000016556b (XEN) ----[ Xen-3.0.4-1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e010:[<ffff83000016364b>] restore_all_xen+0x1b/0x20 (XEN) RFLAGS: 0000000000010282 CONTEXT: hypervisor (XEN) rax: 000000400001c480 rbx: 000000600053dde8 rcx: 0000000000000017 (XEN) rdx: 00000000decafbad rsi: 0000000000012345 rdi: 000000000badbabe (XEN) rbp: 00000040005fffc0 rsp: ffff83000074ffa8 r8: 000000000000001f (XEN) r9: 0000000000000017 r10: 00000040005fffc0 r11: 0000000000000000 (XEN) r12: 0000006000017000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: ffff830000163726 cr0: 000000008005003b cr4: 00000000000006f0 (XEN) cr3: 000000002734d000 cr2: ffff820000001000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e018 cs: e010 (XEN) Xen stack trace from rsp=ffff83000074ffa8: (XEN) 000001000001c480 0000006000101000 000000000000e033 000000400058bec0 (XEN) 0000006000102fa8 000000000000e02b 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffff830000744080 (XEN) Xen call trace: (XEN) [<ffff83000016364b>] restore_all_xen+0x1b/0x20 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) GENERAL PROTECTION FAULT (XEN) [error_code=1000] (XEN) **************************************** (XEN) (XEN) Reboot in five seconds...> > I am passing flags==0 and only push the values for flags, CS:RIP, > > RFLAGS, and SS:RSP. If I push values for RAX, R11, and RCX on the > > stack the kernel ends up in all sorts of weird places rather than > > my intended RIP and RSP. > > That doesn''t make sense. The iret implementation (for an x86/64 guest) > always expects RAX/R11/RCX on the stack.That was my expectation as well, but if I push the three extra values onto the stack then it jumps to the wrong place: context.rax = 0x12345; context.r11 = 0xdecafbad; context.rcx = 0xbadbabe; context.rflags = 0x200; context.rip = start_addr; // 01f:000000400001c480 context.cs = USER_CS; context.flags = 0; context.rsp = (uintptr_t) stack; // 017:00000040005fffc0 context.ss = USER_DS; __asm__ __volatile__ ( "push %0\n" "push %1\n" "push %2\n" "push %3\n" "push %4\n" "push %5\n" "push %6\n" "push %7\n" "push %8\n" "jmp hypercall_page + __HYPERVISOR_iret*32\n" : : "r" (context.ss), "r" (context.rsp), "r" (context.rflags), "r" (context.cs), "r" (context.rip), "r" (context.flags), "r" (context.rcx), "r" (context.r11), "r" (context.rax), "r" (USER_DS) ); Ends up with RIP e033:00000000decafbad and RSP e02b:000000400001c480: (XEN) domain_crash_sync called from entry.S (XEN) Domain 147 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-3.0.4-1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<00000000decafbad>] (XEN) RFLAGS: 0000000000010202 CONTEXT: guest (XEN) rax: 0000000000000017 rbx: 0000000000000200 rcx: 00000000decafbad (XEN) rdx: 0000000000012345 rsi: 000000000badbabe rdi: 0000000000000000 (XEN) rbp: 000000600053dde8 rsp: 000000400001c480 r8: 000000400001c480 (XEN) r9: 000000000000001f r10: 0000000000000017 r11: 0000000000000200 (XEN) r12: 00000040005fffc0 r13: 0000006000017000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000006f0 (XEN) cr3: 0000000026ae5000 cr2: 00000000decafbad (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 Looking in hypercall_page_initialise_ring3_kernel() it appears that the hypervisor call does the rcx, r11 and rax pushing, not the domU kernel calling the hypercall: /* * HYPERVISOR_iret is special because it doesn''t return and expects a * special stack frame. Guests jump at this transfer point instead of * calling it. */ p = (char *)(hypercall_page + (__HYPERVISOR_iret * 32)); *(u8 *)(p+ 0) = 0x51; /* push %rcx */ *(u16 *)(p+ 1) = 0x5341; /* push %r11 */ *(u8 *)(p+ 3) = 0x50; /* push %rax */ *(u8 *)(p+ 4) = 0xb8; /* mov $__HYPERVISOR_iret,%eax */ *(u32 *)(p+ 5) = __HYPERVISOR_iret; *(u16 *)(p+ 9) = 0x050f; /* syscall */ -- Trammell _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 1/10/07 19:43, "Trammell Hudson" <hudson@osresearch.net> wrote:> If I do not set IF in RFLAGS, Xen crashes when it tries to handle > a GPF: > > (XEN) traps.c:1587: GPF (001c): ffff830000165555 -> ffff83000016556b > (XEN) ----[ Xen-3.0.4-1 x86_64 debug=n Not tainted ]---- > (XEN) CPU: 0 > (XEN) RIP: e010:[<ffff83000016364b>] restore_all_xen+0x1b/0x20Did you make this happen with an entirely unmodified Xen? Do you still have the xen-syms file that relates to this crash? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Trammell Hudson
2007-Oct-22 19:52 UTC
Re: [Xen-devel] Switching to user mode from domU kernel
On Mon, Oct 01, 2007 at 06:24:07AM +0100, Keir Fraser wrote:> On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net> wrote: > > > Am I not jumping into user space correctly? Is there something > > else that my code should do to make the transition? > > It sounds like event delivery is masked before the iret, and then you have > IF set in the RFLAGS value in the iret frame, which causes event delivery to > be unmasked during iret.If I disable the domU event channel console and use the HYPERVISOR_console_io hypercall instead, the user process is run as expected. It deliberately causes a GPF by executing HLT after a few instructions so that I know that it began executing user code. That let''s me know that my GDT and LDT are setup correctly and that the user page tables are installed for at least the code segment. If I have done any console output via EVTCHNOP_send my user app never makes progress -- the domU kernel event callback is called a few million times per second and the registers on the stack indicate that the app is still on its first instruction (not the HLT). The console output code is nearly a direct copy of the mini-os code and uses the xencons_interface in the same way (with the addition of newline mapping): struct xencons_interface *intf = xen_console; XENCONS_RING_IDX cons = intf->out_cons; XENCONS_RING_IDX prod = intf->out_prod; mb(); while ((sent < len) && ((prod - cons) < sizeof(intf->out)-1)) { char c = data[ sent++ ]; if( c == ''\n'' ) intf->out[MASK_XENCONS_IDX(prod++, intf->out)] = ''\r''; intf->out[MASK_XENCONS_IDX(prod++, intf->out)] = c; } wmb(); intf->out_prod = prod; struct evtchn_send send = { .port = xen_start_info->console.domU.evtchn, }; HYPERVISOR_event_channel_op( EVTCHNOP_send, &send ); I''ve masked all events and the pending field in the shared_info structure is zero. There are no waiting characters in the incoming xen_console->in_prod array. So, any ideas as to what I''m missing and why Xen keeps calling the callback? Thanks! -- Trammell _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 22/10/07 20:52, "Trammell Hudson" <hudson@osresearch.net> wrote:> I''ve masked all events and the pending field in the shared_info > structure is zero. There are no waiting characters in the incoming > xen_console->in_prod array. > > So, any ideas as to what I''m missing and why Xen keeps calling > the callback?Perhaps the page you think is shared_info isn''t really? If Xen is calling you back repeatedly then it thinks events are pending. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Trammell Hudson
2007-Oct-22 22:00 UTC
Re: [Xen-devel] Switching to user mode from domU kernel
On Mon, Oct 22, 2007 at 09:12:05PM +0100, Keir Fraser wrote:> On 22/10/07 20:52, "Trammell Hudson" <hudson@osresearch.net> wrote: > > I''ve masked all events and the pending field in the shared_info > > structure is zero. There are no waiting characters in the incoming > > xen_console->in_prod array. > > Perhaps the page you think is shared_info isn''t really? If Xen is calling > you back repeatedly then it thinks events are pending.That was it! I had misunderstood the comment in the start_info structure in xen.h: unsigned long shared_info; /* MACHINE address of shared info struct. */ xen_pfn_t store_mfn; /* MACHINE page number of shared page. */ I thought that the store_mfn was the machine page number for the shared info structure and had used that as my xen_shared_info pointer. Mapping the address pointed to by shared_info instead makes it work perfectly. Now my kernel transfers into the user code and faults as expected. Thank you! -- Trammell _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel