thr3ads.net - Xen devel - [Xen-devel] help on debugging kernel module in Xen [Jan 2007]

If this information is useful, please help other people find it:
Share via:

Lily Huang

2007-Jan-12 18:41 UTC

[Xen-devel] help on debugging kernel module in Xen

Hi all,

I am developing a kernel module running in both Xen0 and XenU. I want to
debug it but don''t know how. Does someone has experience on this?

I know printk is an option, but not very convenient.


Thanks in advance,
lily


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Brendan Cully

2007-Jan-12 19:20 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

On Friday, 12 January 2007 at 13:41, Lily Huang wrote:> Hi all,
> 
> I am developing a kernel module running in both Xen0 and XenU. I want to
> debug it but don''t know how. Does someone has experience on this?
> 
> I know printk is an option, but not very convenient.
There''s a nice howto for using gdb to debug guest kernels in
tools/debugger/gdb/README.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 20:25 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

Thanks, Brendan.

I followed the README instruction and did follows:

1. start a domU
2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where 5 is
the
domain id of the domU
3. I got the response:
/*******************************/
Attached; pid = 5
Listening on port 9999
/*******************************/
And then, I lost connection with the domU, I tried both network and console,
neither works for me.

4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping to
reactivate domU with debug command.

the response is as follows:

/*******************************/
GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host
libthread_db
library "/lib/libthread_db.so.1".
/*******************************/

when I typed "cont" command in the gdb, I got "The program is not
being
run."

Am I doing something wrong? Any further help will be highly appreciated.

lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca>
wrote:>
> On Friday, 12 January 2007 at 13:41, Lily Huang wrote:
> > Hi all,
> >
> > I am developing a kernel module running in both Xen0 and XenU. I want
to
> > debug it but don''t know how. Does someone has experience on
this?
> >
> > I know printk is an option, but not very convenient.
>
> There''s a nice howto for using gdb to debug guest kernels in
> tools/debugger/gdb/README.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 20:35 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

Sorry. I was wrong. I forgot to run the rest of gdb command.

Now I did the following commands in gdb:

1. "directory /boot"
response:  "Source directories searched: /boot:$cdir:$cwd"

2. br filp_open
response: "Breakpoint 1 at 0xc0160246: file fs/open.c, line 885."

3.  cont

4. But no matter what file I opened in domU, the breakpoint was never
encountered.

What''s wrong?

Thanks,
lily


On 1/12/07, Lily Huang <ushuanglily@gmail.com>
wrote:>
> Thanks, Brendan.
>
> I followed the README instruction and did follows:
>
> 1. start a domU
> 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where
5 is the
> domain id of the domU
> 3. I got the response:
> /*******************************/
> Attached; pid = 5
> Listening on port 9999
> /*******************************/
> And then, I lost connection with the domU, I tried both network and
> console, neither works for me.
>
> 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping
to
> reactivate domU with debug command.
>
> the response is as follows:
>
> /*******************************/
> GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty"
for
> details.
> This GDB was configured as "i386-redhat-linux-gnu"...Using host
> libthread_db library "/lib/libthread_db.so.1".
> /*******************************/
>
> when I typed "cont" command in the gdb, I got "The program
is not being
> run."
>
> Am I doing something wrong? Any further help will be highly appreciated.
>
> lily
>
>
>
> On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
> >
> > On Friday, 12 January 2007 at 13:41, Lily Huang wrote:
> > > Hi all,
> > >
> > > I am developing a kernel module running in both Xen0 and XenU. I
want
> > to
> > > debug it but don''t know how. Does someone has experience
on this?
> > >
> > > I know printk is an option, but not very convenient.
> >
> > There''s a nice howto for using gdb to debug guest kernels in
> > tools/debugger/gdb/README.
> >
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Brendan Cully

2007-Jan-12 20:53 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

On Friday, 12 January 2007 at 15:25, Lily Huang wrote:> Thanks, Brendan.
> 
> I followed the README instruction and did follows:
> 
> 1. start a domU
> 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5", where
5 is the
> domain id of the domU
> 3. I got the response:
> /*******************************/
> Attached; pid = 5
> Listening on port 9999
> /*******************************/
> And then, I lost connection with the domU, I tried both network and
console,
> neither works for me.
when the debugger attaches, it pauses the guest.
> 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0, hoping
to
> reactivate domU with debug command.
> 
> the response is as follows:
> 
> /*******************************/
> GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty"
for details.
> This GDB was configured as "i386-redhat-linux-gnu"...Using host
libthread_db
> library "/lib/libthread_db.so.1".
> /*******************************/
> 
> when I typed "cont" command in the gdb, I got "The program
is not being
> run."
> 
> Am I doing something wrong? Any further help will be highly appreciated.
you need to attach to the domain from within gdb:

target remote localhost:9999

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 21:07 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

Many thanks for your help!

I tried the command "target remote 127.0.0.1:9999", and got the
response:

/******************************************************************************/
Remote debugging using 127.0.0.1:9999
[New Thread 0]
[Switching to Thread 0]
0xc01013a7 in hypercall_page ()
warning: shared library handler failed to enable breakpoint
<------------------ this could be a problem
/******************************************************************************/

then, I did "br filp_open", and then "cont".

But no matter what file I opened in domU, the breakpoint is never
encountered. It appears that the breakpoint is not activate.

I googled but cannot find answer so far.

Did you run into similar problems?

Thanks,
lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca>
wrote:>
> On Friday, 12 January 2007 at 15:25, Lily Huang wrote:
> > Thanks, Brendan.
> >
> > I followed the README instruction and did follows:
> >
> > 1. start a domU
> > 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach 5",
where 5 is
> the
> > domain id of the domU
> > 3. I got the response:
> > /*******************************/
> > Attached; pid = 5
> > Listening on port 9999
> > /*******************************/
> > And then, I lost connection with the domU, I tried both network and
> console,
> > neither works for me.
>
> when the debugger attaches, it pauses the guest.
>
> > 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in dom0,
hoping to
> > reactivate domU with debug command.
> >
> > the response is as follows:
> >
> > /*******************************/
> > GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and
you
> are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show
warranty" for
> details.
> > This GDB was configured as "i386-redhat-linux-gnu"...Using
host
> libthread_db
> > library "/lib/libthread_db.so.1".
> > /*******************************/
> >
> > when I typed "cont" command in the gdb, I got "The
program is not being
> > run."
> >
> > Am I doing something wrong? Any further help will be highly
appreciated.
>
> you need to attach to the domain from within gdb:
>
> target remote localhost:9999
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 21:10 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

Oh. I found that gdbserver printed out some error messages:

unsupported xc_ptrace request PTRACE_SETFPXREGS
unsupported xc_ptrace request PTRACE_SETFPREGS

Are they the reason why the breakpoint does not work?

lily

On 1/12/07, Lily Huang <ushuanglily@gmail.com>
wrote:>
> Many thanks for your help!
>
> I tried the command "target remote 127.0.0.1:9999", and got the
response:
>
>
/******************************************************************************/
>
> Remote debugging using 127.0.0.1:9999
> [New Thread 0]
> [Switching to Thread 0]
> 0xc01013a7 in hypercall_page ()
> warning: shared library handler failed to enable breakpoint
> <------------------ this could be a problem
>
>
/******************************************************************************/
>
> then, I did "br filp_open", and then "cont".
>
> But no matter what file I opened in domU, the breakpoint is never
> encountered. It appears that the breakpoint is not activate.
>
> I googled but cannot find answer so far.
>
> Did you run into similar problems?
>
> Thanks,
> lily
>
> On 1/12/07, Brendan Cully <brendan@cs.ubc.ca > wrote:
> >
> > On Friday, 12 January 2007 at 15:25, Lily Huang wrote:
> > > Thanks, Brendan.
> > >
> > > I followed the README instruction and did follows:
> > >
> > > 1. start a domU
> > > 2. in Dom0, run "gdbserver-xen 127.0.0.1:9999 --attach
5", where 5 is
> > the
> > > domain id of the domU
> > > 3. I got the response:
> > > /*******************************/
> > > Attached; pid = 5
> > > Listening on port 9999
> > > /*******************************/
> > > And then, I lost connection with the domU, I tried both network
and
> > console,
> > > neither works for me.
> >
> > when the debugger attaches, it pauses the guest.
> >
> > > 4. I run "gdb /boot/vmlinux-syms-2.6.16.33-xenU" in
dom0, hoping to
> > > reactivate domU with debug command.
> > >
> > > the response is as follows:
> > >
> > > /*******************************/
> > > GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
> > > Copyright 2004 Free Software Foundation, Inc.
> > > GDB is free software, covered by the GNU General Public License,
and
> > you are
> > > welcome to change it and/or distribute copies of it under certain
> > > conditions.
> > > Type "show copying" to see the conditions.
> > > There is absolutely no warranty for GDB.  Type "show
warranty" for
> > details.
> > > This GDB was configured as
"i386-redhat-linux-gnu"...Using host
> > libthread_db
> > > library "/lib/libthread_db.so.1".
> > > /*******************************/
> > >
> > > when I typed "cont" command in the gdb, I got "The
program is not
> > being
> > > run."
> > >
> > > Am I doing something wrong? Any further help will be highly
> > appreciated.
> >
> > you need to attach to the domain from within gdb:
> >
> > target remote localhost:9999
> >
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Brendan Cully

2007-Jan-12 21:14 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

On Friday, 12 January 2007 at 16:07, Lily Huang wrote:> Many thanks for your help!
> 
> I tried the command "target remote 127.0.0.1:9999", and got the
response:
> 
>
/******************************************************************************/
> Remote debugging using 127.0.0.1:9999
> [New Thread 0]
> [Switching to Thread 0]
> 0xc01013a7 in hypercall_page ()
> warning: shared library handler failed to enable breakpoint
> <------------------ this could be a problem
>
/******************************************************************************/
> 
> then, I did "br filp_open", and then "cont".
> 
> But no matter what file I opened in domU, the breakpoint is never
> encountered. It appears that the breakpoint is not activate.
> 
> I googled but cannot find answer so far.
> 
> Did you run into similar problems?
I think I''m out of answers at this point. Perhaps it''s
possible that
the function has been inlined or something (though that sounds like a
function pointer). You might try breaking a little higher in the call
chain. And, until you figure out what''s wrong with the debugger,
printk can help you make sure you''re at least triggering the code you
expect to hit the breakpoint :)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Brendan Cully

2007-Jan-12 21:26 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

On Friday, 12 January 2007 at 16:10, Lily Huang wrote:> Oh. I found that gdbserver printed out some error messages:
> 
> unsupported xc_ptrace request PTRACE_SETFPXREGS
> unsupported xc_ptrace request PTRACE_SETFPREGS
> 
> Are they the reason why the breakpoint does not work?
no, those are fairly harmless.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 21:32 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

Hi Brendan,

You are right. After I lift the breakpoint a little bit (to sys_open), the
breakpoint works.

But now the domU kept complaining after I did "target remote
127.0.0.1:9999"
and then "br sys_open":

/************************************************?
BUG: soft lockup detected on CPU#0!

Pid: 0, comm:              swapper
EIP: 0061:[<c01013a7>] CPU: 0
EIP is at 0xc01013a7
 EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
 [<c0105bb3>] show_trace+0x13/0x20
 [<c0103390>] show_regs+0x190/0x1e0
 [<c013cf48>] softlockup_tick+0x88/0xa0
 [<c0126d45>] do_timer+0x215/0x440
 [<c010858c>] timer_interrupt+0x1bc/0x6a0
 [<c013d09f>] handle_IRQ_event+0x3f/0xd0
 [<c013d1be>] __do_IRQ+0x8e/0xf0
 [<c0106efd>] do_IRQ+0x1d/0x30
 [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
 [<c0105309>] hypervisor_callback+0x3d/0x48
 [<c01039df>] xen_idle+0x2f/0x60
 [<c0103a82>] cpu_idle+0x72/0xc0
 [<c0102035>] rest_init+0x35/0x40
 [<c035656a>] start_kernel+0x2ea/0x3a0
 [<c010006f>] 0xc010006f

/********************************************************/

What''s wrong? Looks like I am really doomed to run into so many weird
problems. :(

lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca>
wrote:>
> On Friday, 12 January 2007 at 16:10, Lily Huang wrote:
> > Oh. I found that gdbserver printed out some error messages:
> >
> > unsupported xc_ptrace request PTRACE_SETFPXREGS
> > unsupported xc_ptrace request PTRACE_SETFPREGS
> >
> > Are they the reason why the breakpoint does not work?
>
> no, those are fairly harmless.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Brendan Cully

2007-Jan-12 21:41 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

On Friday, 12 January 2007 at 16:32, Lily Huang wrote:> Hi Brendan,
> 
> You are right. After I lift the breakpoint a little bit (to sys_open), the
> breakpoint works.
> 
> But now the domU kept complaining after I did "target remote
127.0.0.1:9999"
> and then "br sys_open":
> 
> /************************************************?
> BUG: soft lockup detected on CPU#0!
> 
> Pid: 0, comm:              swapper
> EIP: 0061:[<c01013a7>] CPU: 0
> EIP is at 0xc01013a7
> EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
> EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
> ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
> CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
> [<c0105bb3>] show_trace+0x13/0x20
> [<c0103390>] show_regs+0x190/0x1e0
> [<c013cf48>] softlockup_tick+0x88/0xa0
> [<c0126d45>] do_timer+0x215/0x440
> [<c010858c>] timer_interrupt+0x1bc/0x6a0
> [<c013d09f>] handle_IRQ_event+0x3f/0xd0
> [<c013d1be>] __do_IRQ+0x8e/0xf0
> [<c0106efd>] do_IRQ+0x1d/0x30
> [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
> [<c0105309>] hypervisor_callback+0x3d/0x48
> [<c01039df>] xen_idle+0x2f/0x60
> [<c0103a82>] cpu_idle+0x72/0xc0
> [<c0102035>] rest_init+0x35/0x40
> [<c035656a>] start_kernel+0x2ea/0x3a0
> [<c010006f>] 0xc010006f
> 
> /********************************************************/
> 
> What''s wrong? Looks like I am really doomed to run into so many
weird
> problems. :(
soft lockup gets triggered when the kernel hasn''t gotten a timer
interrupt in a while. This tends to happen frequently when the kernel
is being debugged. I think there''s a kernel config option to turn this
off, or you can just live with the messages.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 23:04 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

That''s a great point. I just disabled the "Detect Soft
Lockup" option in the
"kernel hacking" group. Now the soft lockup disappeared and I can
debug
kernel now.

Next, I tried to debug kernel module in domU. New problem occurs.

After I set "br xxx(function in a loaded kernel module" and then
"cont", I
worked in the domu to activate the breakpoint.

Unfortunately, I found domU immediately print error messages:

/**********************************************************/
Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: int3: 0000 [#1]
<----------------------------------------------------- seems to be a problem

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: SMP

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: CPU:    0

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: EIP is at nfs3_proc_readdir+0xd/0x100 [nfs]

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: eax: daa0c740   ebx: 00001000   ecx: c1353aa0   edx: db25cd14

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: esi: df02ced0   edi: daa0885c   ebp: da9c7d34   esp: da9c7c4c

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: ds: 007b   es: 007b   ss: 0069

Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
normal kernel: Process ls (pid: 1670, threadinfo=da9c6000 task=c05d7070)
/**********************************************************/

Looks like "int 3" triggered kernel error. Why just for kernel module
debugging?


BTW: I built the kernel moduel with "-g" option and use gdb command to
load
module symbols

/*************************************************/
add-symbol-file /path/to/module.ko 0xdf01f000 \
        -s .altinstr_replacement 0xdf036629 \
        -s .altinstructions 0xdf03add0 \
        -s .bss 0xdf042a60 \
        -s .data 0xdf041a20 \
        -s .exit.text 0xdf0365f0 \
        -s .gnu.linkonce.this_module 0xdf042840 \
        -s .init.text 0xdf045000 \
        -s .rodata 0xdf036680 \
        -s .rodata.str1.1 0xdf039ef4 \
        -s .rodata.str1.4 0xdf036cb8 \
        -s .strtab 0xdf03e450 \
        -s .symtab 0xdf03aec0
/*************************************************/

Thanks,
lily

On 1/12/07, Brendan Cully <brendan@cs.ubc.ca>
wrote:>
> On Friday, 12 January 2007 at 16:32, Lily Huang wrote:
> > Hi Brendan,
> >
> > You are right. After I lift the breakpoint a little bit (to sys_open),
> the
> > breakpoint works.
> >
> > But now the domU kept complaining after I did "target remote
> 127.0.0.1:9999"
> > and then "br sys_open":
> >
> > /************************************************?
> > BUG: soft lockup detected on CPU#0!
> >
> > Pid: 0, comm:              swapper
> > EIP: 0061:[<c01013a7>] CPU: 0
> > EIP is at 0xc01013a7
> > EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
> > EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
> > ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
> > CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
> > [<c0105bb3>] show_trace+0x13/0x20
> > [<c0103390>] show_regs+0x190/0x1e0
> > [<c013cf48>] softlockup_tick+0x88/0xa0
> > [<c0126d45>] do_timer+0x215/0x440
> > [<c010858c>] timer_interrupt+0x1bc/0x6a0
> > [<c013d09f>] handle_IRQ_event+0x3f/0xd0
> > [<c013d1be>] __do_IRQ+0x8e/0xf0
> > [<c0106efd>] do_IRQ+0x1d/0x30
> > [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
> > [<c0105309>] hypervisor_callback+0x3d/0x48
> > [<c01039df>] xen_idle+0x2f/0x60
> > [<c0103a82>] cpu_idle+0x72/0xc0
> > [<c0102035>] rest_init+0x35/0x40
> > [<c035656a>] start_kernel+0x2ea/0x3a0
> > [<c010006f>] 0xc010006f
> >
> > /********************************************************/
> >
> > What''s wrong? Looks like I am really doomed to run into so
many weird
> > problems. :(
>
> soft lockup gets triggered when the kernel hasn''t gotten a timer
> interrupt in a while. This tends to happen frequently when the kernel
> is being debugged. I think there''s a kernel config option to turn
this
> off, or you can just live with the messages.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lily Huang

2007-Jan-12 23:20 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

Sorry, please skip my last question. I fixed the problem myself.

The problem occurred because I failed to run "add-symbol-file"
command.
Somehow gdb failed to recognize the strtab and symtab sections.

After I removed them, it works now.

Again, thanks for your help!

lily

On 1/12/07, Lily Huang <ushuanglily@gmail.com>
wrote:>
> That''s a great point. I just disabled the "Detect Soft
Lockup" option in
> the "kernel hacking" group. Now the soft lockup disappeared and I
can debug
> kernel now.
>
> Next, I tried to debug kernel module in domU. New problem occurs.
>
> After I set "br xxx(function in a loaded kernel module" and then
"cont", I
> worked in the domu to activate the breakpoint.
>
> Unfortunately, I found domU immediately print error messages:
>
> /**********************************************************/
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: int3: 0000 [#1]
> <----------------------------------------------------- seems to be a
problem
>
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: SMP
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: CPU:    0
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: EIP is at nfs3_proc_readdir+0xd/0x100 [nfs]
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: eax: daa0c740   ebx: 00001000   ecx: c1353aa0   edx:
> db25cd14
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: esi: df02ced0   edi: daa0885c   ebp: da9c7d34   esp:
> da9c7c4c
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: ds: 007b   es: 007b   ss: 0069
>
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: Process ls (pid: 1670, threadinfo=da9c6000 task=c05d7070)
> /**********************************************************/
>
> Looks like "int 3" triggered kernel error. Why just for kernel
module
> debugging?
>
>
> BTW: I built the kernel moduel with "-g" option and use gdb
command to
> load module symbols
>
> /*************************************************/
> add-symbol-file /path/to/module.ko 0xdf01f000 \
>         -s .altinstr_replacement 0xdf036629 \
>         -s .altinstructions 0xdf03add0 \
>         -s .bss 0xdf042a60 \
>         -s .data 0xdf041a20 \
>         -s .exit.text 0xdf0365f0 \
>         -s .gnu.linkonce.this_module 0xdf042840 \
>         -s .init.text 0xdf045000 \
>         -s .rodata 0xdf036680 \
>         -s .rodata.str1.1 0xdf039ef4 \
>         -s .rodata.str1.4 0xdf036cb8 \
>         -s .strtab 0xdf03e450 \
>         -s .symtab 0xdf03aec0
> /*************************************************/
>
> Thanks,
> lily
>
> On 1/12/07, Brendan Cully <brendan@cs.ubc.ca> wrote:
> >
> > On Friday, 12 January 2007 at 16:32, Lily Huang wrote:
> > > Hi Brendan,
> > >
> > > You are right. After I lift the breakpoint a little bit (to
sys_open),
> > the
> > > breakpoint works.
> > >
> > > But now the domU kept complaining after I did "target remote
> > 127.0.0.1:9999"
> > > and then "br sys_open":
> > >
> > > /************************************************?
> > > BUG: soft lockup detected on CPU#0!
> > >
> > > Pid: 0, comm:              swapper
> > > EIP: 0061:[<c01013a7>] CPU: 0
> > > EIP is at 0xc01013a7
> > > EFLAGS: 00000346    Not tainted  (2.6.16.33-xenU #12)
> > > EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000000
> > > ESI: 00000001 EDI: c0354000 EBP: c0355f84 DS: 007b ES: 007b
> > > CR0: 8005003b CR2: b7f77000 CR3: 1d633000 CR4: 00000640
> > > [<c0105bb3>] show_trace+0x13/0x20
> > > [<c0103390>] show_regs+0x190/0x1e0
> > > [<c013cf48>] softlockup_tick+0x88/0xa0
> > > [<c0126d45>] do_timer+0x215/0x440
> > > [<c010858c>] timer_interrupt+0x1bc/0x6a0
> > > [<c013d09f>] handle_IRQ_event+0x3f/0xd0
> > > [<c013d1be>] __do_IRQ+0x8e/0xf0
> > > [<c0106efd>] do_IRQ+0x1d/0x30
> > > [<c025d2d1>] evtchn_do_upcall+0xa1/0xe0
> > > [<c0105309>] hypervisor_callback+0x3d/0x48
> > > [<c01039df>] xen_idle+0x2f/0x60
> > > [<c0103a82>] cpu_idle+0x72/0xc0
> > > [<c0102035>] rest_init+0x35/0x40
> > > [<c035656a>] start_kernel+0x2ea/0x3a0
> > > [<c010006f>] 0xc010006f
> > >
> > > /********************************************************/
> > >
> > > What''s wrong? Looks like I am really doomed to run into
so many weird
> > > problems. :(
> >
> > soft lockup gets triggered when the kernel hasn''t gotten a
timer
> > interrupt in a while. This tends to happen frequently when the kernel
> > is being debugged. I think there''s a kernel config option to
turn this
> > off, or you can just live with the messages.
> >
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Brendan Cully

2007-Jan-12 23:21 UTC

head link

Re: [Xen-devel] help on debugging kernel module in Xen

On Friday, 12 January 2007 at 18:04, Lily Huang wrote:> That''s a great point. I just disabled the "Detect Soft
Lockup" option in the
> "kernel hacking" group. Now the soft lockup disappeared and I can
debug
> kernel now.
> 
> Next, I tried to debug kernel module in domU. New problem occurs.
> 
> After I set "br xxx(function in a loaded kernel module" and then
"cont", I
> worked in the domu to activate the breakpoint.
> 
> Unfortunately, I found domU immediately print error messages:
> 
> /**********************************************************/
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: int3: 0000 [#1]
> <----------------------------------------------------- seems to be a
problem
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: SMP
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: CPU:    0
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: EIP is at nfs3_proc_readdir+0xd/0x100 [nfs]
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: eax: daa0c740   ebx: 00001000   ecx: c1353aa0   edx:
db25cd14
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: esi: df02ced0   edi: daa0885c   ebp: da9c7d34   esp:
da9c7c4c
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: ds: 007b   es: 007b   ss: 0069
> 
> Message from syslogd@normal at Fri Jan 12 17:54:41 2007 ...
> normal kernel: Process ls (pid: 1670, threadinfo=da9c6000 task=c05d7070)
> /**********************************************************/
> 
> Looks like "int 3" triggered kernel error. Why just for kernel
module
> debugging?
I''m afraid you''re beyond my knowledge at this point.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Jan 2007 - help on debugging kernel module in Xen

[Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen

Re: [Xen-devel] help on debugging kernel module in Xen