Hi there,
I am using Xen-3.2-testing (non-PAE and x86-32) on Intel''s Core 2
Quad processor. My HVM domain is Fedora i386 with Linux kernel 2.6.16
(with 8K of stack and CONFIG_FRAME_POINTER = y).
In my project, I am trying to perform stack walk on the HVM guest''s
kernel stack from the Xen hypervisor. In order to do that, whenever
a VMEXIT happens, I get the ebp value from cpu_user_regs and try
to use this address as a starting point for the walk. I add 4 bytes
in this address (for return address ) and try to get the value at
this location using Xen''s function hvm_copy_from_guest_virt(), which
reads 4 bytes of return address. I, then, do this process recursively
by reading the value at the location pointed by the ebp to get the
previous frame and so on.
During the recursive process, sometimes when I try to get the
return address or next frame address, hvm_copy_from_guest_virt() function
crashes as it is not able to read that memory.
I saw a function named "show_guest_stack" in xen/arch/x86/traps.c,
which does the stack walk incase of PV domain. And, returns if
it finds HVM domain. Is there any function available for doing
stack walk in HVM domain?
If not, could you please let me know whether my stack walk
procedure is correct or not? Or, do I need to do something different
incase of an HVM domain? I tried using different methods of putting
exit condition in my stack walk code such as frame-pointer = 0,
return address = 0, Stack walk depth = 24, decreasing frame pointer and
ebp should be less than guest''s current thread_info + 8912.
As of now, I am clueless as how to do this. Any help would be highly
appreciated.
Thanks,
Abhinav
Add more friends to your messenger and enjoy! Go to
http://messenger.yahoo.com/invite/
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Abhinav Srivastava wrote:> Hi there, > > I am using Xen-3.2-testing (non-PAE and x86-32) on Intel''s Core 2 > Quad processor. My HVM domain is Fedora i386 with Linux kernel 2.6.16 > (with 8K of stack and CONFIG_FRAME_POINTER = y). > > In my project, I am trying to perform stack walk on the HVM guest''s > kernel stack from the Xen hypervisor. In order to do that, whenever > a VMEXIT happens, I get the ebp value from cpu_user_regs and try > to use this address as a starting point for the walk. I add 4 bytes > in this address (for return address ) and try to get the value at > this location using Xen''s function hvm_copy_from_guest_virt(), which > reads 4 bytes of return address. I, then, do this process recursively > by reading the value at the location pointed by the ebp to get the > previous frame and so on. > > During the recursive process, sometimes when I try to get the > return address or next frame address, hvm_copy_from_guest_virt() function > crashes as it is not able to read that memory.There are several reasons why you can run into trouble. The simplest reason is that the guest may be running in user mode. Since you can''t predict that state of the stack in user mode, you should first check for guest kernel mode before you try to trace the stack. The second issue is that for hvm guests, VMEXITs may occur at arbitrary points in the guest kernel execution. This means that you may find the guest kernel in the middle of handling an exception or interrupt. In these situations, the stack layout will be different than you expect. To avoid this, you may want to ignore stacks when interrupts are disabled in the guest kernel. Note here that disabling interrupts in a guest kernel doesn''t prevent a VMEXIT.> I saw a function named "show_guest_stack" in xen/arch/x86/traps.c, > which does the stack walk incase of PV domain. And, returns if > it finds HVM domain. Is there any function available for doing > stack walk in HVM domain?You may want to look at the crash utility (http://people.redhat.com/anderson/). It knows how to dump kernel stacks from live systems or dump files. It is also smart enough to properly recognize exception frames and properly trace past them.> If not, could you please let me know whether my stack walk > procedure is correct or not? Or, do I need to do something different > incase of an HVM domain? I tried using different methods of putting > exit condition in my stack walk code such as frame-pointer = 0, > return address = 0, Stack walk depth = 24, decreasing frame pointer and > ebp should be less than guest''s current thread_info + 8912.Your starting and ending conditions are too simplistic to work in the case of an arbitrary VMEXIT from an hvm domain. You need to handle every variation possible with a linux kernel based stack tracer, and in addition, deal with partially constructed/deconstructed exception frames. Steve> As of now, I am clueless as how to do this. Any help would be highly appreciated. > > Thanks, > Abhinav > > > > Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Steve, Thanks for the reply. I will look into the crash utility to see how are they doing. However, I have one question. I am doing stack walk whenever a HVM guest makes VMExit due to a page fault. The way I am doing this: I am making some kernel memory pages write protected from the hypervisor using shadow page tables. And, whenever the kernel code writes to those pages, the guest faults, VMExit happens, and control comes to the hypervisor''s page fault handler, where my code checks whether this is due my protection or not. If yes, then I do the stack walk. So, I am not doing stack walk from user-mode context. But, I am doing stack walk whenever kernel page fault is happening? Will it make my case less difficult or still I have to do all the things that you mentioned in your email. Thanks for your help. -Abhinav --- On Tue, 7/10/08, Steve Ofsthun <sofsthun@virtualiron.com> wrote:> From: Steve Ofsthun <sofsthun@virtualiron.com> > Subject: Re: [Xen-devel] Show HVM guest stack > To: abhinavs_iitkgp@yahoo.co.in > Cc: xen-devel@lists.xensource.com > Date: Tuesday, 7 October, 2008, 4:31 AM > Abhinav Srivastava wrote: > > Hi there, > > > > I am using Xen-3.2-testing (non-PAE and x86-32) on > Intel''s Core 2 > > Quad processor. My HVM domain is Fedora i386 with > Linux kernel 2.6.16 > > (with 8K of stack and CONFIG_FRAME_POINTER = y). > > > > In my project, I am trying to perform stack walk on > the HVM guest''s > > kernel stack from the Xen hypervisor. In order to do > that, whenever > > a VMEXIT happens, I get the ebp value from > cpu_user_regs and try > > to use this address as a starting point for the walk. > I add 4 bytes > > in this address (for return address ) and try to get > the value at > > this location using Xen''s function > hvm_copy_from_guest_virt(), which > > reads 4 bytes of return address. I, then, do this > process recursively > > by reading the value at the location pointed by the > ebp to get the > > previous frame and so on. > > > > During the recursive process, sometimes when I try to > get the > > return address or next frame address, > hvm_copy_from_guest_virt() function > > crashes as it is not able to read that memory. > > There are several reasons why you can run into trouble. > The simplest reason is that the guest may be running in user > mode. Since you can''t predict that state of the stack > in user mode, you should first check for guest kernel mode > before you try to trace the stack. The second issue is that > for hvm guests, VMEXITs may occur at arbitrary points in the > guest kernel execution. This means that you may find the > guest kernel in the middle of handling an exception or > interrupt. In these situations, the stack layout will be > different than you expect. To avoid this, you may want to > ignore stacks when interrupts are disabled in the guest > kernel. Note here that disabling interrupts in a guest > kernel doesn''t prevent a VMEXIT. > > > I saw a function named "show_guest_stack" in > xen/arch/x86/traps.c, > > which does the stack walk incase of PV domain. And, > returns if > > it finds HVM domain. Is there any function available > for doing > > stack walk in HVM domain? > > You may want to look at the crash utility > (http://people.redhat.com/anderson/). It knows how to dump > kernel stacks from live systems or dump files. It is also > smart enough to properly recognize exception frames and > properly trace past them. > > > If not, could you please let me know whether my stack > walk > > procedure is correct or not? Or, do I need to do > something different > > incase of an HVM domain? I tried using different > methods of putting > > exit condition in my stack walk code such as > frame-pointer = 0, > > return address = 0, Stack walk depth = 24, decreasing > frame pointer and > > ebp should be less than guest''s current > thread_info + 8912. > > Your starting and ending conditions are too simplistic to > work in the case of an arbitrary VMEXIT from an hvm domain. > You need to handle every variation possible with a linux > kernel based stack tracer, and in addition, deal with > partially constructed/deconstructed exception frames. > > Steve > > > As of now, I am clueless as how to do this. Any help > would be highly appreciated. > > > > Thanks, > > Abhinav > > > > > > > > Add more friends to your messenger and enjoy! Go > to http://messenger.yahoo.com/invite/ > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-develAdd more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Abhinav Srivastava wrote:> Hi Steve, > > Thanks for the reply. I will look into the crash utility to see how are they doing. > > However, I have one question. I am doing stack walk whenever a HVM guest makes VMExit due to a page fault. The way I am doing this: I am making some kernel memory pages write protected from the hypervisor using shadow page tables. And, whenever the kernel code writes to those pages, the guest faults, VMExit happens, and control comes to the hypervisor''s page fault handler, where my code checks whether this is due my protection or not. If yes, then I do the stack walk.OK. This may make your job simpler. What type of kernel addresses are you write protecting? Be careful when making any kernel pages read-only that are normally written in the page fault path (kernel stacks, some kernel data, per_cpu data). These may cause nested exceptions that may confuse your stack tracer (as well as crash the guest if you don''t properly hide them). If you add faults to any exception entry/exit code, you will also see confusing stack trace conditions. I guess the bottom line is, if you are only adding write faults to paths that can already write fault, any standard kernel stack trace should work (ala crash). If you are adding new write fault points to the guest kernel (faults that would crash the guest if not intercepted), you may see stack conditions that a normal kernel stack tracer would not see.> So, I am not doing stack walk from user-mode context. But, I am doing stack walk whenever kernel page fault is happening? Will it make my case less difficult or still I have to do all the things that you mentioned in your email.A user process can write to a kernel address (causing a protection fault). Normal user programs don''t do this. As long as you don''t need to worry about malicious programs doing this, then you should be OK. Steve> Thanks for your help. > > -Abhinav > > > > --- On Tue, 7/10/08, Steve Ofsthun <sofsthun@virtualiron.com> wrote: > >> From: Steve Ofsthun <sofsthun@virtualiron.com> >> Subject: Re: [Xen-devel] Show HVM guest stack >> To: abhinavs_iitkgp@yahoo.co.in >> Cc: xen-devel@lists.xensource.com >> Date: Tuesday, 7 October, 2008, 4:31 AM >> Abhinav Srivastava wrote: >>> Hi there, >>> >>> I am using Xen-3.2-testing (non-PAE and x86-32) on >> Intel''s Core 2 >>> Quad processor. My HVM domain is Fedora i386 with >> Linux kernel 2.6.16 >>> (with 8K of stack and CONFIG_FRAME_POINTER = y). >>> >>> In my project, I am trying to perform stack walk on >> the HVM guest''s >>> kernel stack from the Xen hypervisor. In order to do >> that, whenever >>> a VMEXIT happens, I get the ebp value from >> cpu_user_regs and try >>> to use this address as a starting point for the walk. >> I add 4 bytes >>> in this address (for return address ) and try to get >> the value at >>> this location using Xen''s function >> hvm_copy_from_guest_virt(), which >>> reads 4 bytes of return address. I, then, do this >> process recursively >>> by reading the value at the location pointed by the >> ebp to get the >>> previous frame and so on. >>> >>> During the recursive process, sometimes when I try to >> get the >>> return address or next frame address, >> hvm_copy_from_guest_virt() function >>> crashes as it is not able to read that memory. >> There are several reasons why you can run into trouble. >> The simplest reason is that the guest may be running in user >> mode. Since you can''t predict that state of the stack >> in user mode, you should first check for guest kernel mode >> before you try to trace the stack. The second issue is that >> for hvm guests, VMEXITs may occur at arbitrary points in the >> guest kernel execution. This means that you may find the >> guest kernel in the middle of handling an exception or >> interrupt. In these situations, the stack layout will be >> different than you expect. To avoid this, you may want to >> ignore stacks when interrupts are disabled in the guest >> kernel. Note here that disabling interrupts in a guest >> kernel doesn''t prevent a VMEXIT. >> >>> I saw a function named "show_guest_stack" in >> xen/arch/x86/traps.c, >>> which does the stack walk incase of PV domain. And, >> returns if >>> it finds HVM domain. Is there any function available >> for doing >>> stack walk in HVM domain? >> You may want to look at the crash utility >> (http://people.redhat.com/anderson/). It knows how to dump >> kernel stacks from live systems or dump files. It is also >> smart enough to properly recognize exception frames and >> properly trace past them. >> >>> If not, could you please let me know whether my stack >> walk >>> procedure is correct or not? Or, do I need to do >> something different >>> incase of an HVM domain? I tried using different >> methods of putting >>> exit condition in my stack walk code such as >> frame-pointer = 0, >>> return address = 0, Stack walk depth = 24, decreasing >> frame pointer and >>> ebp should be less than guest''s current >> thread_info + 8912. >> >> Your starting and ending conditions are too simplistic to >> work in the case of an arbitrary VMEXIT from an hvm domain. >> You need to handle every variation possible with a linux >> kernel based stack tracer, and in addition, deal with >> partially constructed/deconstructed exception frames. >> >> Steve >> >>> As of now, I am clueless as how to do this. Any help >> would be highly appreciated. >>> Thanks, >>> Abhinav >>> >>> >>> >>> Add more friends to your messenger and enjoy! Go >> to http://messenger.yahoo.com/invite/ >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel > > > Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel