I''m trying to trace the execution path taken by the system when I make a hypercall from a domU kernel. What I''m seeing in the hypercall entry function in Xen is that the domU eip value points to a location in the domU hypercall page for the __HYPERCALL_sched_op hypercall, even though this is *not* the hypercall that I just called. Could someone help clarify this for me? Why would the vcpu for the guest domain have this seemingly incorrect eip value? Thanks, bryan - Bryan D. Payne Graduate Student, Computer Science Georgia Tech Information Security Center http://www.bryanpayne.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2007-02-22 at 19:16 +0100, Daniel Stodden wrote:> On Thu, 2007-02-22 at 12:17 -0500, Bryan D. Payne wrote: > > I''m trying to trace the execution path taken by the system when I make a > > hypercall from a domU kernel. What I''m seeing in the hypercall entry > > function in Xen is that the domU eip value points to a location in the > > domU hypercall page for the __HYPERCALL_sched_op hypercall, even though > > this is *not* the hypercall that I just called. > > > > Could someone help clarify this for me? Why would the vcpu for the > > guest domain have this seemingly incorrect eip value? > > hard to imagine :} > > apart from emulating execution, the single place i''m aware of playing > with eip are continuations. > > so: what does it point at? what''s the page address, which offset? > > after looking into traps.c: take good care with the layout: hypercall > pages on x86_32 are 32 (!) bytes per entry (not 8 as the loop body might > suggest). are your calculations correct?(i keep forgetting cc''s) -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> so: what does it point at? what''s the page address, which offset?On my machine (xen 3.0.4_1, linux 2.6.16.33-xen), the EIP register in the guest''s vcpu holds 0xc01013a7. For reference, the hypercall page is 0xc0101000, this is at offset 0x3a7 on that page. Looking at the disassembly from that page at this location, I see the following: mov 0x1d, %eax int $0x82 ret The EIP value is pointing at the ''ret'' instruction. Looking in the hypercall table, this (0x1d) is the sched_op hypercall (which is not the hypercall that I called).> after looking into traps.c: take good care with the layout: hypercall > pages on x86_32 are 32 (!) bytes per entry (not 8 as the loop body might > suggest). are your calculations correct?I believe so, but you can verify with the information that I provided above. 0x1d * 32 = 0x3a0... this matches the offset I''m seeing. Hopefully someone can help me make sense of this :-) Thanks, bryan - Bryan D. Payne Graduate Student, Computer Science Georgia Tech Information Security Center http://www.bryanpayne.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2007-02-22 at 13:30 -0500, Bryan D. Payne wrote:> > so: what does it point at? what''s the page address, which offset? > > On my machine (xen 3.0.4_1, linux 2.6.16.33-xen), the EIP register in > the guest''s vcpu holds 0xc01013a7. For reference, the hypercall page is > 0xc0101000, this is at offset 0x3a7 on that page. Looking at the > disassembly from that page at this location, I see the following: > > mov 0x1d, %eax > int $0x82 > ret > > The EIP value is pointing at the ''ret'' instruction. Looking in the > hypercall table, this (0x1d) is the sched_op hypercall (which is not the > hypercall that I called). > > > > after looking into traps.c: take good care with the layout: hypercall > > pages on x86_32 are 32 (!) bytes per entry (not 8 as the loop body might > > suggest). are your calculations correct? > > I believe so, but you can verify with the information that I provided > above. 0x1d * 32 = 0x3a0... this matches the offset I''m seeing. > > Hopefully someone can help me make sense of this :-)The most common reason for sched_op being called is that the vcpu is idle (SCHEDOP_block). Are you sure you are looking at the correct vcpu? How are you looking at the EIP register? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Are you sure you are looking at the correct vcpu?There is only one vcpu assigned to this domain. Based on this, I''m assuming that my code (shown below) is correct.> How are you looking at the EIP register?uint32_t eip = domain->vcpu[0]->arch.guest_context.user_regs.eip; Thanks, -bryan - Bryan D. Payne Graduate Student, Computer Science Georgia Tech Information Security Center http://www.bryanpayne.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2007-02-22 at 14:11 -0500, Bryan D. Payne wrote:> > Are you sure you are looking at the correct vcpu? > > There is only one vcpu assigned to this domain. Based on this, I''m > assuming that my code (shown below) is correct. > > > > How are you looking at the EIP register? > > uint32_t eip = domain->vcpu[0]->arch.guest_context.user_regs.eip;That seems reasonable. Where have you put this tracing? How you are arranging for the hypercall you are expecting to be called and how you are matching that up with where your tracing is placed? Perhaps you could post your code so we can see what you are actually doing? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Where have you put this tracing? > > How you are arranging for the hypercall you are expecting to be called > and how you are matching that up with where your tracing is placed? > > Perhaps you could post your code so we can see what you are actually > doing?This all started with me playing with grant tables. I have a simple example setup where I am passing a string from a domU kernel module to a dom0 kernel module. Dom0 allocates the page and sets permissions in the grant table using gnttab_grant_foreign_access. In domU, the kernel module makes a GNTTABOP_copy hypercall to pass the string. The domU code looks like this: static int send_string (grant_ref_t gref) { struct gnttab_copy op; char str[MAX_STR_LENGTH]; /* hard code string for testing */ memset(str, 0, MAX_STR_LENGTH); memcpy(str, "This is a test\n", 16); op.source.domid = DOMID_SELF; op.source.offset = (PAGE_SIZE-1) & (uint32_t)str; op.source.u.gmfn = virt_to_mfn(str); op.dest.domid = 0; op.dest.offset = 0; op.dest.u.ref = gref; op.len = strnlen(str, MAX_LOG_LENGTH); op.flags = GNTCOPY_dest_gref; HYPERVISOR_grant_table_op(GNTTABOP_priv_write, &op, 1); /* make sure that the hypercall succeeded */ if (op.status){ printk("Grant table operation failure\n"); return 1; } return 0; } This code setup seems to work just fine. I can pass the string to dom0 without any problems. Next, I wanted to study the execution path between the two kernel modules. As part of this, I placed code in the hypervisor to print out the EIP value from domU while servicing the hypercall (the one shown in the code above). I put this code in the __gnttab_copy function of common/grant_table.c, because this is the function that does most of the work for this hypercall. Here I simply print out the EIP value, as indicated in my prior email: uint32_t eip = sd->vcpu[0]->arch.guest_context.user_regs.eip; gdprintk(XENLOG_WARNING, "eip=0x%x", eip); Note that I''m using ''sd'' for the domain since I''m interested in the domain that invoked the hypercall. This is where I see the eip value that I indicated in my previous email. Also note that I am doing this after ''sd'' is initialized, of course :-) My assumptions are as follows: * When I make a hypercall from domU, the execution switches to the hypervisor immediately (as soon as the ''int'' instruction is issued). * The hypervisor services this hypercall and then returns to domU where it left off. However, this is now what I''m seeing... and this is why I''m confused. Hopefully this helps explain the problem. Let me know if you have any other questions about my setup. Thanks, bryan - Bryan D. Payne Graduate Student, Computer Science Georgia Tech Information Security Center http://www.bryanpayne.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2007-02-22 at 15:28 -0500, Bryan D. Payne wrote:> uint32_t eip = sd->vcpu[0]->arch.guest_context.user_regs.eip; > gdprintk(XENLOG_WARNING, "eip=0x%x", eip);sd is current->domain here so the guest context is saved on the stack rather than in the domain structure. Therefore you need to use guest_cpu_user_regs()->eip instead. I should have thought of that before, sorry. The context in the vcpu''s arch.guest_context will be a stale left over from the last time the domain was rescheduled, which is why it is doing a sched_op since it was presumably just waking up from idle. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> sd is current->domain here so the guest context is saved on the stack > rather than in the domain structure. Therefore you need to use > guest_cpu_user_regs()->eip instead. I should have thought of that > before, sorry.Thanks! That gives a much more reasonable value :-) -bryan - Bryan D. Payne Graduate Student, Computer Science Georgia Tech Information Security Center http://www.bryanpayne.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel