We''ve been able to verify that it is not a hardware issue, but
we''ve
managed to narrow down that the problem seems to involve osth4_1 and
osth4_2.
We disabled recovery on ost start to see if it would behave
differently to no avail. The results are attatched.
Anyone have any idea?
thanks,
-k
On Thu, Mar 13, 2008 at 03:51:22PM -0500, Fernsler; Karen
wrote:> 2.6.9-42.0.10.EL_lustre-1.4.10.1smp
>
> This is a 2.6.9-42.0.10.E kernel with lustre-1.4.10.1.
>
> This has been working ok for almost a year. We did try to
> export this filesystem to another cluster over nfs before
> we started seeing problems, but I don''t know how related if
> at all that is.
>
> We are now trying to dissect the problem by inspecting
> the switch logs these nodes are connected to.
>
> thanks,
> -k
>
> On Thu, Mar 13, 2008 at 04:50:04PM -0400, Aaron Knister wrote:
> > What version of lustre/kernel is running on the problematic server?
> >
> > On Mar 13, 2008, at 11:02 AM, Michelle Butler wrote:
> >
> > >We got past that point by e2fsck the individual partitions first.
> > >
> > >But we are still having problems.. I''m sorry to
> > >say. we have an I/O server that is fine until
> > >we start Lustre. It starts spewing lustre call traces :
> > >
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > > <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > > <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > ><ffffffff80110ebb>{child_rip+8}
> > > <ffffffffa03e0163>{:ptlrpc:ptlrpc_main+0}
> > ><ffffffff80110eb3>{child_rip+0}
> > >
> > >ll_ost_io_232 S 000001037d6bbee8 0 26764 1 26765
> > >26763 (L-TLB)
> > >000001037d6bbe58 0000000000000046 0000000100000246
0000000000000003
> > > 0000000000000016 0000000000000001 00000104100bcb20
> > >0000000300000246
> > > 00000103f5470030 000000000001d381
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > > <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > > <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > ><ffffffff80110ebb>{child_rip+8}
> > > <ffffffffa03e0163>{:ptlrpc:ptlrpc_main+0}
> > ><ffffffff80110eb3>{child_rip+0}
> > >
> > >ll_ost_io_233 S 00000103de847ee8 0 26765 1 26766
> > >26764 (L-TLB)
> > >00000103de847e58 0000000000000046 0000000100000246
0000000000000001
> > > 0000000000000016 0000000000000001 000001040f83c620
> > >0000000100000246
> > > 00000103e627e030 000000000001d487
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > > <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > > <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > ><ffffffff80110ebb>{child_rip+8}
> > > <ffffffffa03e0163>{:ptlrpc:ptlrpc_main+0}
> > ><ffffffff80110eb3>{child_rip+0}
> > >
> > >ll_ost_io_234 S 00000100c4353ee8 0 26766 1 26767
> > >26765 (L-TLB)
> > >00000100c4353e58 0000000000000046 0000000100000246
0000000000000003
> > > 0000000000000016 0000000000000001 00000104100bcc60
> > >0000000300000246
> > > 00000103de81b810 000000000001d945
> > >Call
> > >Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22}
> > ><ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408}
> > > <ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > >
> >
><ffffffffa03e0156>{:ptlrpc:ptlrpc_retr???f?????????c?????????c??????
> > >
> > >Ks[F????????????
> > ><ffffffff8013327d>{default_wake_function+0}
> > ><ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0}
> > > <ffffffffa03e0156>{:ptl
> > >
> > >It then panic''s the kernel.. ??
> > >
> > >Michelle Butler
> > >
> > >At 02:39 AM 3/13/2008, Andreas Dilger wrote:
> > >>On Mar 12, 2008 06:44 -0500, Karen M. Fernsler wrote:
> > >>>I''m running:
> > >>>
> > >>>e2fsck -y -v --mdsdb mdsdb --ostdb osth3_1
/dev/mapper/27l4
> > >>>
> > >>>and getting:
> > >>>
> > >>>Pass 6: Acquiring information for lfsck
> > >>>error getting mds_hdr (3685469441:8) in
> > >>/post/cfg/mdsdb: DB_NOTFOUND: No matching key/data pair found
> > >>>e2fsck: aborted
> > >>>
> > >>>Any ideas how to get around this?
> > >>
> > >>Does "mdsdb" actually exist? This should be created
by first
> > >>running:
> > >>
> > >>e2fsck --mdsdb mdsdb /dev/{mdsdevicename}
> > >>
> > >>before running your above command on the OST.
> > >>
> > >>Please also try specifying the absolute pathname for the mdsdb
and
> > >>ostdb
> > >>files.
> > >>
> > >>Cheers, Andreas
> > >>--
> > >>Andreas Dilger
> > >>Sr. Staff Engineer, Lustre Group
> > >>Sun Microsystems of Canada, Inc.
> > >
> > >
> > >_______________________________________________
> > >Lustre-discuss mailing list
> > >Lustre-discuss at lists.lustre.org
> > >http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> > Aaron Knister
> > Associate Systems Analyst
> > Center for Ocean-Land-Atmosphere Studies
> >
> > (301) 595-7000
> > aaron at iges.org
> >
> >
> >
>
> --
> Karen Fernsler Systems Engineer
> National Center for Supercomputing Applications
> ph: (217) 265 5249
> email: fernsler at ncsa.uiuc.edu
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
--
Karen Fernsler Systems Engineer
National Center for Supercomputing Applications
ph: (217) 265 5249
email: fernsler at ncsa.uiuc.edu
-------------- next part --------------
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 14674:0:(filter.c:730:filter_init_server_data()) RECOVERY: service
osth2_1, 3 recoverable
clients, last_rcvd 21428009
Lustre: OST osth2_1 now serving /dev/mapper/24l4
(044e7efd-496b-43cc-b794-3a9e9c48bc78), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10;
during that time new
clients will not be allowed to connect. Recovery progress can be monitored by
watching
/proc/fs/lustre/obdfilter/osth2_1/recovery_status.
Lustre: 14603:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-141.142.71.12 at o2ib portal
28 match 115715421 offset 0 length 240: 2
Lustre: 14600:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-141.142.71.12 at o2ib portal
28 match 115715558 offset 0 length 240: 2
Lustre: 14600:0:(lib-move.c:1644:lnet_parse_put()) Skipped 23 previous similar
messages
LustreError: 14872:0:(filter.c:3257:filter_iocontrol()) aborting recovery for
device osth2_1
LustreError: 14872:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_1: recovery
period over;
disconnecting unfinished clients.
LustreError: 14872:0:(genops.c:908:class_disconnect_stale_exports()) osth2_1:
disconnecting 3 stale
clients
Lustre: 14872:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_1: sending
delayed replies to
recovered clients
Lustre: 14872:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall
DEFAULT
RECOVERY_OVER osth2_1_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15907:0:(filter.c:730:filter_init_server_data()) RECOVERY: service
osth2_2, 3 recoverable
clients, last_rcvd 23278178
Lustre: OST osth2_2 now serving /dev/mapper/2al4
(a4bbdf80-f96d-4bc1-84e0-fcd42ca14eb4), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10;
during that time new
clients will not be allowed to connect. Recovery progress can be monitored by
watching
/proc/fs/lustre/obdfilter/osth2_2/recovery_status.
LustreError: 15909:0:(filter.c:3257:filter_iocontrol()) aborting recovery for
device osth2_2
LustreError: 15909:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_2: recovery
period over;
disconnecting unfinished clients.
LustreError: 15909:0:(genops.c:908:class_disconnect_stale_exports()) osth2_2:
disconnecting 3 stale
clients
Lustre: 15909:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_2: sending
delayed replies to
recovered clients
Lustre: 15909:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall
DEFAULT
RECOVERY_OVER osth2_2_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15915:0:(filter.c:730:filter_init_server_data()) RECOVERY: service
osth2_3, 3 recoverable
clients, last_rcvd 22572894
Lustre: OST osth2_3 now serving /dev/mapper/26l5
(ae63c80f-43dc-46f2-828b-ebe4fadf9ebc), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10;
during that time new
clients will not be allowed to connect. Recovery progress can be monitored by
watching
/proc/fs/lustre/obdfilter/osth2_3/recovery_status.
LustreError: 15917:0:(filter.c:3257:filter_iocontrol()) aborting recovery for
device osth2_3
LustreError: 15917:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_3: recovery
period over;
disconnecting unfinished clients.
LustreError: 15917:0:(genops.c:908:class_disconnect_stale_exports()) osth2_3:
disconnecting 3 stale
clients
Lustre: 15917:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_3: sending
delayed replies to
recovered clients
Lustre: 15917:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall
DEFAULT
RECOVERY_OVER osth2_3_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15923:0:(filter.c:730:filter_init_server_data()) RECOVERY: service
osth2_4, 3 recoverable
clients, last_rcvd 23889626
Lustre: OST osth2_4 now serving /dev/mapper/22l6
(2f906cea-8413-4821-b92c-0637230959c7), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10;
during that time new
clients will not be allowed to connect. Recovery progress can be monitored by
watching
/proc/fs/lustre/obdfilter/osth2_4/recovery_status.
LustreError: 15925:0:(filter.c:3257:filter_iocontrol()) aborting recovery for
device osth2_4
LustreError: 15925:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_4: recovery
period over;
disconnecting unfinished clients.
LustreError: 15925:0:(genops.c:908:class_disconnect_stale_exports()) osth2_4:
disconnecting 3 stale
clients
Lustre: 15925:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_4: sending
delayed replies to
recovered clients
Lustre: 15925:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall
DEFAULT
RECOVERY_OVER osth2_4_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15931:0:(filter.c:730:filter_init_server_data()) RECOVERY: service
osth2_5, 3 recoverable
clients, last_rcvd 21923408
Lustre: OST osth2_5 now serving /dev/mapper/28l6
(f237633e-b2e8-4372-998f-2d7c6f2913bd), but will be
inrecovery until 3 clients reconnect, or if no clients reconnect for 4:10;
during that time new
clients will not be allowed to connect. Recovery progress can be monitored by
watching
/proc/fs/lustre/obdfilter/osth2_5/recovery_status.
LustreError: 15933:0:(filter.c:3257:filter_iocontrol()) aborting recovery for
device osth2_5
LustreError: 15933:0:(ldlm_lib.c:969:target_abort_recovery()) osth2_5: recovery
period over;
disconnecting unfinished clients.
LustreError: 15933:0:(genops.c:908:class_disconnect_stale_exports()) osth2_5:
disconnecting 3 stale
clients
Lustre: 15933:0:(ldlm_lib.c:867:target_finish_recovery()) osth2_5: sending
delayed replies to
recovered clients
Lustre: 15933:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall
DEFAULT
RECOVERY_OVER osth2_5_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
Lustre: 15963:0:(filter.c:730:filter_init_server_data()) RECOVERY: service
osth4_1, 6 recoverable
clients, last_rcvd 41799396
Lustre: OST osth4_1 now serving /dev/mapper/26l4
(a2babbd1-a8c1-40a5-9d4d-c0139d8ecc9b), but will be
inrecovery until 6 clients reconnect, or if no clients reconnect for 4:10;
during that time new
clients will not be allowed to connect. Recovery progress can be monitored by
watching
/proc/fs/lustre/obdfilter/osth4_1/recovery_status.
LustreError: 15965:0:(filter.c:3257:filter_iocontrol()) aborting recovery for
device osth4_1
LustreError: 15965:0:(ldlm_lib.c:969:target_abort_recovery()) osth4_1: recovery
period over;
disconnecting unfinished clients.
LustreError: 15965:0:(genops.c:908:class_disconnect_stale_exports()) osth4_1:
disconnecting 6 stale
clients
Lustre: 15965:0:(ldlm_lib.c:867:target_finish_recovery()) osth4_1: sending
delayed replies to
recovered clients
Lustre: 15965:0:(recover.c:77:ptlrpc_run_recovery_over_upcall()) Invoked upcall
DEFAULT
RECOVERY_OVER osth4_1_UUID
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at extents:301
invalid operand: 0000 [1] SMP
CPU 2
Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) lquota(U) ko2iblnd(U)
rdma_cm(U) iw_cm(U)
ib_addr(U) ib_local_sa(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U)
libcfs(U) ldiskfs(U)
ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) dell_rbu(U) netconsole(U)
netdump(U) autofs4(U)
i2c_dev(U) i2c_core(U) nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ds(U)
yenta_socket(U) pcmcia_core(U)
dm_round_robin(U) dm_multipath(U) button(U) battery(U) ac(U) uhci_hcd(U)
ehci_hcd(U) hw_random(U)
ib_mthca(U) ib_ipoib(U) ib_umad(U) ib_ucm(U) ib_uverbs(U) ib_cm(U) ib_sa(U)
ib_mad(U) ib_core(U)
md5(U) ipv6(U) e1000(U) floppy(U) qla2300(U) qla2xxx(U) scsi_transport_fc(U)
sg(U) dm_snapshot(U)
dm_zero(U) dm_mirror(U) ext3(U) jbd(U) dm_mod(U) mptscsih(U) mptsas(U) mptspi(U)
mptfc(U) mptscsi(U)
mptbase(U) sd_mod(U) scsi_mod(U)
Pid: 15971, comm: lctl Tainted: GF 2.6.9-42.0.10.EL_lustre-1.4.10.1smp
RIP: 0010:[<ffffffffa030647f>]
<ffffffffa030647f>{:ldiskfs:ldiskfs_ext_find_extent+323}
RSP: 0018:0000010235dc5a88 EFLAGS: 00010246
RAX: 00000101b15cc520 RBX: 00000101b15cc520 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000080f RDI: 00000101dbb446d0
RBP: 00000101dbb44680 R08: 0000000000000000 R09: 00000101dbb44680
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000001
R13: 00000101b15cc520 R14: 0000010235dc5ae8 R15: 0000000000000000
FS: 0000002a95574360(0000) GS:ffffffff80479d00(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000032c658fae0 CR3: 000000000817e000 CR4: 00000000000006e0
Process lctl (pid: 15971, threadinfo 0000010235dc4000, task 0000010229c0e810)
Stack: 0000000000000001 00000101b15cc618 0000000000000000 00000101b15cc638
0000010235dc5b88 0000000000000000 0000000000000000 ffffffffa0308919
00000101b15cc638 00000001b15cc638
Call Trace:<ffffffffa0308919>{:ldiskfs:ldiskfs_ext_get_block+256}
<ffffffff8018578e>{may_open+88}
<ffffffffa02f8a3c>{:ldiskfs:ldiskfs_getblk+155}
<ffffffffa02f8c0a>{:ldiskfs:ldiskfs_bread+15}
<ffffffffa04bc059>{:fsfilt_ldiskfs:fsfilt_ldiskfs_read_record+466}
<ffffffffa04c95d2>{:obdfilter:filter_prep+1857}
<ffffffff8017da00>{do_kern_mount+304}
<ffffffffa04cc56c>{:obdfilter:filter_common_setup+2002}
<ffffffffa04ccfda>{:obdfilter:filter_setup+367}
<ffffffffa03b707e>{:obdclass:class_setup+1561}
<ffffffffa03bab44>{:obdclass:class_process_config+3366}
<ffffffffa03a613a>{:obdclass:class_handle_ioctl+1954}
<ffffffff80143631>{do_sigaction+490}
<ffffffff80189265>{sys_ioctl+853}
<ffffffff8011029a>{system_call+126}
Code: 0f 0b aa 25 31 a0 ff ff ff ff 2d 01 48 8d 48 0c 49 89 49 10
RIP <ffffffffa030647f>{:ldiskfs:ldiskfs_ext_find_extent+323} RSP
<0000010235dc5a88>
CPU#0 is frozen.
CPU#1 is frozen.
CPU#2 is executing netdump.
CPU#3 is frozen.
< netdump activated - performing handshake with the server. >
NETDUMP START!
< handshake completed - listening for dump requests. >
0(4296190029)\
Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) lquota(U) ko2iblnd(U)
rdma_cm(U) iw_cm(U)
ib_addr(U) ib_local_sa(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U)
libcfs(U) ldiskfs(U)
ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) dell_rbu(U) netconsole(U)
netdump(U) autofs4(U)
i2c_dev(U) i2c_core(U) nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ds(U)
yenta_socket(U) pcmcia_core(U)
dm_round_robin(U) dm_multipath(U) button(U) battery(U) ac(U) uhci_hcd(U)
ehci_hcd(U) hw_random(U)
ib_mthca(U) ib_ipoib(U) ib_umad(U) ib_ucm(U) ib_uverbs(U) ib_cm(U) ib_sa(U)
ib_mad(U) ib_core(U)
md5(U) ipv6(U) e1000(U) floppy(U) qla2300(U) qla2xxx(U) scsi_transport_fc(U)
sg(U) dm_snapshot(U)
dm_zero(U) dm_mirror(U) ext3(U) jbd(U) dm_mod(U) mptscsih(U) mptsas(U) mptspi(U)
mptfc(U) mptscsi(U)
mptbase(U) sd_mod(U) scsi_mod(U)
Pid: 15971, comm: lctl Tainted: GF 2.6.9-42.0.10.EL_lustre-1.4.10.1smp
RIP: 0010:[<ffffffffa030647f>]
<ffffffffa030647f>{:ldiskfs:ldiskfs_ext_find_extent+323}
RSP: 0018:0000010235dc5a88 EFLAGS: 00010246
RAX: 00000101b15cc520 RBX: 00000101b15cc520 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000080f RDI: 00000101dbb446d0
RBP: 00000101dbb44680 R08: 0000000000000000 R09: 00000101dbb44680
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000001
R13: 00000101b15cc520 R14: 0000010235dc5ae8 R15: 0000000000000000
FS: 0000002a95574360(0000) GS:ffffffff80479d00(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000032c658fae0 CR3: 000000000817e000 CR4: 00000000000006e0
Call Trace:<ffffffffa030640a>{:ldiskfs:ldiskfs_ext_find_extent+206}
<ffffffffa0308919>{:ldiskfs:ldiskfs_ext_get_block+256}
<ffffffff8018578e>{may_open+88}
<ffffffffa02f8a3c>{:ldiskfs:ldiskfs_getblk+155}
<ffffffffa02f8c0a>{:ldiskfs:ldiskfs_bread+15}
<ffffffffa04bc059>{:fsfilt_ldiskfs:fsfilt_ldiskfs_read_record+466}
<ffffffffa04c95d2>{:obdfilter:filter_prep+1857}
<ffffffff8017da00>{do_kern_mount+304}
<ffffffffa04cc56c>{:obdfilter:filter_common_setup+2002}
<ffffffffa04ccfda>{:obdfilter:filter_setup+367}
<ffffffffa03b707e>{:obdclass:class_setup+1561}
<ffffffffa03bab44>{:obdclass:class_process_config+3366}
<ffffffffa03a613a>{:obdclass:class_handle_ioctl+1954}
<ffffffff80143631>{do_sigaction+490}
<ffffffff80189265>{sys_ioctl+853}
<ffffffff8011029a>{system_call+126}
sibling
task PC pid father child younger older
init S 000000000000000b 0 1 0 2 (NOTLB)
00000101fffe1d78 0000000000000046 00000102ffc40d68 00000102fffe8e00
000000000000000b 0000000000000400 000000d000000001 0000000300000246
00000102fffd0810 0000000000000dd8
Call Trace:<ffffffff8013f210>{__mod_timer+293}
<ffffffff80309ea4>{schedule_timeout+367}
<ffffffff8013fc4c>{process_timeout+0}
<ffffffff80189c5f>{do_select+939}
<ffffffff801897f9>{__pollwait+0}
<ffffffff80189fde>{sys_select+820}
<ffffffff8011029a>{system_call+126}
migration/0 S 00000100010b87e0 0 2 1 3 (L-TLB)
00000102fff89ec8 0000000000000046 00000100010b87e0 0000001900000076
0000010229c0e810 0000000000000076 00000100010a1a40 0000000000000001
00000102fffd4810 000000000000018a
Call Trace:<ffffffff8013443e>{migration_thread+324}
<ffffffff801342fa>{migration_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
ksoftirqd/0 S 0000000000000000 0 3 1 4 2 (L-TLB)
00000102fff8df08 0000000000000046 00000102fffd4810 0000001900000077
0000010400fb7030 0000000000000077 00000100010a1a40 0000000000000000
00000102fffd4030 00000000000000f5
Call Trace:<ffffffff8013c1ec>{ksoftirqd+0}
<ffffffff8013c228>{ksoftirqd+60}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
migration/1 S 00000100010b87e0 0 4 1 5 3 (L-TLB)
00000102fff8fec8 0000000000000046 00000100010b87e0 0000001900000077
0000010229c0e810 0000000000000077 00000100010a9a40 0000000100000001
00000102fffd5810 00000000000001cf
Call Trace:<ffffffff8013443e>{migration_thread+324}
<ffffffff801342fa>{migration_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
ksoftirqd/1 S 0000000000000000 0 5 1 6 4 (L-TLB)
00000101fff91f08 0000000000000046 000001040c6e7810 0000001900000079
000001040f8c0810 0000000000000079 00000100010a9a40 0000000100000000
00000102fffd5030 0000000000000158
Call Trace:<ffffffff8013c1ec>{ksoftirqd+0}
<ffffffff8013c228>{ksoftirqd+60}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
migration/2 S 00000100010a87e0 0 6 1 7 5 (L-TLB)
00000101fff95ec8 0000000000000046 00000100010a87e0 0000001900000074
000001023761f030 0000000000000074 00000100010b1a40 0000000200000001
00000102fffa3810 00000000000001b0
Call Trace:<ffffffff8013443e>{migration_thread+324}
<ffffffff801342fa>{migration_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
ksoftirqd/2 R running task 0 7 1 8 6 (L-TLB)
migration/3 S 00000100010a07e0 0 8 1 9 7 (L-TLB)
00000100dff89ec8 0000000000000046 00000100010a07e0 0000001900000077
000001023761f030 0000000000000077 00000100010b9a40 0000000300000001
00000101fff9f810 00000000000001cf
Call Trace:<ffffffff8013443e>{migration_thread+324}
<ffffffff801342fa>{migration_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
ksoftirqd/3 S 0000000000000000 0 9 1 10 8 (L-TLB)
00000100dffbdf08 0000000000000046 00000101a7aaa030 0000000000000246
00000100dffbc000 0000000000000246 00000100010bbcc0 0000000300000000
00000101fff9f030 00000000000001a4
Call Trace:<ffffffff8013c1ec>{ksoftirqd+0}
<ffffffff8013c228>{ksoftirqd+60}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
events/0 S ffffffff80160fc6 0 10 1 14 11 9 (L-TLB)
00000103ffe43e68 0000000000000046 00000103ffe43da8 0000000000000246
0000000000000246 0000000000000246 000000010012a1f8 0000000000000246
00000103ffe13810 0000000000001c0a
Call Trace:<ffffffff80160fc6>{cache_reap+0}
<ffffffff80146d01>{worker_thread+226}
<ffffffff8013327d>{default_wake_function+0}
<ffffffff801332ce>{__wake_up_common+67}
<ffffffff8013327d>{default_wake_function+0}
<ffffffff80146c1f>{worker_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
events/1 S ffffffffa011d1d8 0 11 1 399 12 10 (L-TLB)
00000103ffe45e68 0000000000000046 00000000182a3c00 0000000000000000
00000104101a1738 ffffffffa012100a 00000104101a1738 0000000100000246
00000103ffe13030 00000000000025b9
Call Trace:<ffffffffa012100a>{:e1000:e1000_read_phy_reg+510}
<ffffffffa011d1d8>{:e1000:e1000_watchdog_task+0}
<ffffffff80146d01>{worker_thread+226}
<ffffffff8013327d>{default_wake_function+0}
<ffffffff801332ce>{__wake_up_common+67}
<ffffffff8013327d>{default_wake_function+0}
<ffffffff80146c1f>{worker_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8} <ffffffff8014a96f>{kthread+0}
<ffffffff80110eb3>{child_rip+0}
events/2 S ffffffff80160fc6 0 12 1 8059 13 11 (L-TLB)
00000103ffe49e68 0000000000000046 00000102fffd3810 0000001900000086
000001040bc48030 0000000000000246 0000000100129861 0000000200000246
00000103ffe14810 00000000000011b8
Call Trace:<ffffffff80160fc6>{cache_reap+0}
<ffffffff80146d01>{worker_thread+226}
<ffffffff8013327d>{default_wake_function+0}
<ffffffff801332ce>{__wake_up_common+67}
<ffffffff8013327d>{default_wake_function+0}
<ffffffff80146c1f>{worker_thread+0}
<ffffffff8014aa37>{kthread+200}
<ffffffff80110ebb>{child_rip+8}
<ffffffff8014a96f>{kthread+0} <ffffffff80110eb3>{child_rip+0}
events/3 S ffffffff80160fc6 0 13