thr3ads.net - Xen devel - [Xen-devel] Switching to user mode from domU kernel [Sep 2007]

If this information is useful, please help other people find it:
Share via:

Trammell Hudson

2007-Sep-30 20:11 UTC

[Xen-devel] Switching to user mode from domU kernel

I''m having trouble switching into user mode with my domU kernel --
everytime I call __HYPERVISOR_iret, the hypervisor calls into my
kernel via the registered callback handler.  None of the bits in
the event channel pending arrays are set.  If I return via the
iret call, the hypervisor immediately calls back into the handler.
The user RIP never changes, so it is clearly not making forward
progress.

The stackframe should be correctly setup with the flags, rip, cs,
rflags, rsp, and ss values and the currently installed LDT has
slots for the code and data segments that the user will use.

Am I not jumping into user space correctly?  Is there something
else that my code should do to make the transition?

On a documentation question, is __HYPERVISOR_iret documented
anywhere?  The Xen API* doesn''t seem to include any details on it.
I deduced the stack arguments from looking at the code in the Linux
xen architecture and the xen-x86_64.h header.

This comment in the header, however, doesn''t seem to match the
behaviour that my code seems to experience:

 * If flags contains VGCF_in_syscall:
 *   Restore RAX, RIP, RFLAGS, RSP.
 *   Discard R11, RCX, CS, SS.
 * Otherwise:
 *   Restore RAX, R11, RCX, CS:RIP, RFLAGS, SS:RSP.

I am passing flags==0 and only push the values for flags, CS:RIP,
RFLAGS, and SS:RSP.  If I push values for RAX, R11, and RCX on the
stack the kernel ends up in all sorts of weird places rather than
my intended RIP and RSP.


Thanks!
-- Trammell

*:
http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/interface/interface.html

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2007-Oct-01 05:24 UTC

head link

Re: [Xen-devel] Switching to user mode from domU kernel

On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net>
wrote:
> Am I not jumping into user space correctly?  Is there something
> else that my code should do to make the transition?
It sounds like event delivery is masked before the iret, and then you have
IF set in the RFLAGS value in the iret frame, which causes event delivery to
be unmasked during iret.
> I am passing flags==0 and only push the values for flags, CS:RIP,
> RFLAGS, and SS:RSP.  If I push values for RAX, R11, and RCX on the
> stack the kernel ends up in all sorts of weird places rather than
> my intended RIP and RSP.
That doesn''t make sense. The iret implementation (for an x86/64 guest)
always expects RAX/R11/RCX on the stack.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Trammell Hudson

2007-Oct-01 18:43 UTC

head link

Re: [Xen-devel] Switching to user mode from domU kernel

On Mon, Oct 01, 2007 at 06:24:07AM +0100, Keir Fraser
wrote:> On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net>
wrote:
> > Am I not jumping into user space correctly?  Is there something
> > else that my code should do to make the transition?
> 
> It sounds like event delivery is masked before the iret, and then you have
> IF set in the RFLAGS value in the iret frame, which causes event delivery
to
> be unmasked during iret.
That makes sense, although the event channel pending array is
all zero and the shared_info->vcpu_info[0].evtchn_upcall_pending
value is also zero.

If I do not set IF in RFLAGS, Xen crashes when it tries to handle
a GPF:

(XEN) traps.c:1587: GPF (001c): ffff830000165555 -> ffff83000016556b
(XEN) ----[ Xen-3.0.4-1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e010:[<ffff83000016364b>] restore_all_xen+0x1b/0x20
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 000000400001c480   rbx: 000000600053dde8   rcx: 0000000000000017
(XEN) rdx: 00000000decafbad   rsi: 0000000000012345   rdi: 000000000badbabe
(XEN) rbp: 00000040005fffc0   rsp: ffff83000074ffa8   r8:  000000000000001f
(XEN) r9:  0000000000000017   r10: 00000040005fffc0   r11: 0000000000000000
(XEN) r12: 0000006000017000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: ffff830000163726   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 000000002734d000   cr2: ffff820000001000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff83000074ffa8:
(XEN)    000001000001c480 0000006000101000 000000000000e033 000000400058bec0
(XEN)    0000006000102fa8 000000000000e02b 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 ffff830000744080
(XEN) Xen call trace:
(XEN)    [<ffff83000016364b>] restore_all_xen+0x1b/0x20
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) GENERAL PROTECTION FAULT
(XEN) [error_code=1000]
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...
> > I am passing flags==0 and only push the values for flags, CS:RIP,
> > RFLAGS, and SS:RSP.  If I push values for RAX, R11, and RCX on the
> > stack the kernel ends up in all sorts of weird places rather than
> > my intended RIP and RSP.
> 
> That doesn''t make sense. The iret implementation (for an x86/64
guest)
> always expects RAX/R11/RCX on the stack.
That was my expectation as well, but if I push the three extra values
onto the stack then it jumps to the wrong place:

        context.rax     = 0x12345;
        context.r11     = 0xdecafbad;
        context.rcx     = 0xbadbabe;
        context.rflags  = 0x200;
        context.rip     = start_addr; // 01f:000000400001c480
        context.cs      = USER_CS;
        context.flags   = 0;
        context.rsp     = (uintptr_t) stack; // 017:00000040005fffc0
        context.ss      = USER_DS;

        __asm__ __volatile__ (
                "push   %0\n"
                "push   %1\n"
                "push   %2\n"
                "push   %3\n"
                "push   %4\n"
                "push   %5\n"
                "push   %6\n"
                "push   %7\n"
                "push   %8\n"
                "jmp hypercall_page + __HYPERVISOR_iret*32\n"
                :
                : "r" (context.ss),
                "r" (context.rsp),
                "r" (context.rflags),
                "r" (context.cs),
                "r" (context.rip),
                "r" (context.flags),
                "r" (context.rcx),
                "r" (context.r11),
                "r" (context.rax),
                "r" (USER_DS)
        );

Ends up with RIP e033:00000000decafbad and RSP e02b:000000400001c480:

(XEN) domain_crash_sync called from entry.S
(XEN) Domain 147 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.0.4-1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<00000000decafbad>]
(XEN) RFLAGS: 0000000000010202   CONTEXT: guest
(XEN) rax: 0000000000000017   rbx: 0000000000000200   rcx: 00000000decafbad
(XEN) rdx: 0000000000012345   rsi: 000000000badbabe   rdi: 0000000000000000
(XEN) rbp: 000000600053dde8   rsp: 000000400001c480   r8:  000000400001c480
(XEN) r9:  000000000000001f   r10: 0000000000000017   r11: 0000000000000200
(XEN) r12: 00000040005fffc0   r13: 0000006000017000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 0000000026ae5000   cr2: 00000000decafbad
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033

Looking in hypercall_page_initialise_ring3_kernel() it appears that
the hypervisor call does the rcx, r11 and rax pushing, not the
domU kernel calling the hypercall:

    /*
     * HYPERVISOR_iret is special because it doesn''t return and expects
a
     * special stack frame. Guests jump at this transfer point instead of
     * calling it.
     */
    p = (char *)(hypercall_page + (__HYPERVISOR_iret * 32));
    *(u8  *)(p+ 0) = 0x51;    /* push %rcx */
    *(u16 *)(p+ 1) = 0x5341;  /* push %r11 */
    *(u8  *)(p+ 3) = 0x50;    /* push %rax */
    *(u8  *)(p+ 4) = 0xb8;    /* mov  $__HYPERVISOR_iret,%eax */
    *(u32 *)(p+ 5) = __HYPERVISOR_iret;
    *(u16 *)(p+ 9) = 0x050f;  /* syscall */

-- Trammell

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2007-Oct-02 05:33 UTC

head link

Re: [Xen-devel] Switching to user mode from domU kernel

On 1/10/07 19:43, "Trammell Hudson" <hudson@osresearch.net>
wrote:
> If I do not set IF in RFLAGS, Xen crashes when it tries to handle
> a GPF:
> 
> (XEN) traps.c:1587: GPF (001c): ffff830000165555 -> ffff83000016556b
> (XEN) ----[ Xen-3.0.4-1  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e010:[<ffff83000016364b>] restore_all_xen+0x1b/0x20
Did you make this happen with an entirely unmodified Xen? Do you still have
the xen-syms file that relates to this crash?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Trammell Hudson

2007-Oct-22 19:52 UTC

head link

Re: [Xen-devel] Switching to user mode from domU kernel

On Mon, Oct 01, 2007 at 06:24:07AM +0100, Keir Fraser
wrote:> On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net>
wrote:
> 
> > Am I not jumping into user space correctly?  Is there something
> > else that my code should do to make the transition?
> 
> It sounds like event delivery is masked before the iret, and then you have
> IF set in the RFLAGS value in the iret frame, which causes event delivery
to
> be unmasked during iret.
If I disable the domU event channel console and use the
HYPERVISOR_console_io hypercall instead, the user process is run
as expected.  It deliberately causes a GPF by executing HLT after
a few instructions so that I know that it began executing user code.
That let''s me know that my GDT and LDT are setup correctly and
that the user page tables are installed for at least the code segment.
If I have done any console output via EVTCHNOP_send my user app
never makes progress -- the domU kernel event callback is called
a few million times per second and the registers on the stack indicate
that the app is still on its first instruction (not the HLT).

The console output code is nearly a direct copy of the mini-os code
and uses the xencons_interface in the same way (with the addition
of newline mapping):

        struct xencons_interface *intf = xen_console;
        XENCONS_RING_IDX cons = intf->out_cons;
        XENCONS_RING_IDX prod = intf->out_prod;

        mb();
        while ((sent < len) && ((prod - cons) <
sizeof(intf->out)-1))
        {
                char c = data[ sent++ ];
                if( c == ''\n'' )
                        intf->out[MASK_XENCONS_IDX(prod++, intf->out)] =
''\r'';
                intf->out[MASK_XENCONS_IDX(prod++, intf->out)] = c;
        }

        wmb();
        intf->out_prod = prod;

        struct evtchn_send send = {
                .port = xen_start_info->console.domU.evtchn,
        };

        HYPERVISOR_event_channel_op( EVTCHNOP_send, &send );

I''ve masked all events and the pending field in the shared_info
structure is zero.  There are no waiting characters in the incoming
xen_console->in_prod array.

So, any ideas as to what I''m missing and why Xen keeps calling
the callback?

Thanks!
-- Trammell

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2007-Oct-22 20:12 UTC

head link

Re: [Xen-devel] Switching to user mode from domU kernel

On 22/10/07 20:52, "Trammell Hudson" <hudson@osresearch.net>
wrote:
> I''ve masked all events and the pending field in the shared_info
> structure is zero.  There are no waiting characters in the incoming
> xen_console->in_prod array.
> 
> So, any ideas as to what I''m missing and why Xen keeps calling
> the callback?
Perhaps the page you think is shared_info isn''t really? If Xen is
calling
you back repeatedly then it thinks events are pending.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Trammell Hudson

2007-Oct-22 22:00 UTC

head link

Re: [Xen-devel] Switching to user mode from domU kernel

On Mon, Oct 22, 2007 at 09:12:05PM +0100, Keir Fraser
wrote:> On 22/10/07 20:52, "Trammell Hudson"
<hudson@osresearch.net> wrote:
> > I''ve masked all events and the pending field in the
shared_info
> > structure is zero.  There are no waiting characters in the incoming
> > xen_console->in_prod array.
> 
> Perhaps the page you think is shared_info isn''t really? If Xen is
calling
> you back repeatedly then it thinks events are pending.
That was it!  I had misunderstood the comment in the start_info
structure in xen.h:

    unsigned long shared_info;  /* MACHINE address of shared info struct. */
    xen_pfn_t store_mfn;        /* MACHINE page number of shared page.    */

I thought that the store_mfn was the machine page number for the
shared info structure and had used that as my xen_shared_info
pointer.  Mapping the address pointed to by shared_info instead
makes it work perfectly.

Now my kernel transfers into the user code and faults as expected.

Thank you!

-- Trammell

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Maybe Matching Threads

Search for more maybe matching threads

Xen devel - Sep 2007 - Switching to user mode from domU kernel

[Xen-devel] Switching to user mode from domU kernel

Re: [Xen-devel] Switching to user mode from domU kernel

Re: [Xen-devel] Switching to user mode from domU kernel

Re: [Xen-devel] Switching to user mode from domU kernel

Re: [Xen-devel] Switching to user mode from domU kernel

Re: [Xen-devel] Switching to user mode from domU kernel

Re: [Xen-devel] Switching to user mode from domU kernel

Maybe Matching Threads