thr3ads.net - Lustre discuss - [Lustre-discuss] Kernel oops after cat on /proc/fs/lustre/mgs/MGS/exports/*/stats [Apr 2010]

If this information is useful, please help other people find it:
Share via:

Christopher Huhn

2010-Apr-23 12:18 UTC

[Lustre-discuss] Kernel oops after cat on /proc/fs/lustre/mgs/MGS/exports/*/stats

Dear lustre wizards,

we are experiencing problems on our MDS and our Lustre expert is abroad
(he just attended LUG meeting).

One of the symptoms we observe are reproducible kernel oopses when
viewing some stats files beneath /proc/fs/lustre/mgs/MGS/exports :

    mds:~# cat /proc/fs/lustre/mgs/MGS/exports/10.12... at tcp/stats
    Killed
    mds:~#  mds kernel: Oops: 0000 [38] SMP
    Apr 23 13:23:19 mds kernel: Unable to handle kernel paging request
    at ffffffff00040024 RIP:
    Apr 23 13:23:19 mds kernel: [<ffffffff883d6680>]
    :obdclass:lprocfs_stats_seq_show+0x80/0x1e0
    Apr 23 13:23:19 mds kernel: PGD 203067 PUD 0
    Apr 23 13:23:19 mds kernel: Oops: 0000 [38] SMP
    Apr 23 13:23:20 mds kernel: CPU 7
    Apr 23 13:23:20 mds kernel: Modules linked in: mds fsfilt_ldiskfs(F)
    mgs mgc ldiskfs crc16 lustre lov mdc lquota osc ksocklnd ptlrpc
    obdclass lnet lvfs libcfs xt_tcpudp iptable_filter ip_tables
    x_tables drbd cn button ac battery bonding xfs ipmi_si ipmi_devintf
    ipmi_msghandler serio_raw psmouse joydev pcspkr i2c_i801 i2c_core
    shpchp pci_hotplug evdev parport_pc parport ext3 jbd mbcache
    dm_mirror dm_snapshot dm_mod raid10 raid456 xor raid1 raid0
    multipath linear md_mod sd_mod ide_cd cdrom ata_generic libata
    generic usbhid hid piix 3w_9xxx floppy ide_core ehci_hcd uhci_hcd
    e1000 scsi_mod thermal processor fan
    Apr 23 13:23:20 mds kernel: Pid: 7293, comm: cat Tainted: GF     
    2.6.22+lustre1.6.7.2+0.credativ.etch.1 #2
    Apr 23 13:23:20 mds kernel: RIP: 0010:[<ffffffff883d6680>] 
    [<ffffffff883d6680>] :obdclass:lprocfs_stats_seq_show+0x80/0x1e0
    Apr 23 13:23:20 mds kernel: RSP: 0018:ffff8103ba5f9e48  EFLAGS: 00010282
    Apr 23 13:23:20 mds kernel: RAX: ffffffff00040004 RBX:
    7fffffffffffffff RCX: 0000000000000006
    Apr 23 13:23:20 mds kernel: RDX: 0101010101010101 RSI:
    0000000000000000 RDI: 0000000000000000
    Apr 23 13:23:20 mds kernel: RBP: 0000000000000000 R08:
    0000000000000008 R09: 0000000000000000
    Apr 23 13:23:20 mds kernel: R10: 0000000000000000 R11:
    0000000000000000 R12: 0000000000000000
    Apr 23 13:23:20 mds kernel: R13: 0000000000000000 R14:
    0000000000000000 R15: ffff8108000a1760
    Apr 23 13:23:20 mds kernel: FS:  00002b4a366786d0(0000)
    GS:ffff81081004b840(0000) knlGS:0000000000000000
    Apr 23 13:23:20 mds kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
    000000008005003b
    Apr 23 13:23:20 mds kernel: CR2: ffffffff00040024 CR3:
    000000078f018000 CR4: 00000000000006e0
    Apr 23 13:23:20 mds kernel: Process cat (pid: 7293, threadinfo
    ffff8103ba5f8000, task ffff8107dc299530)
    Apr 23 13:23:20 mds kernel: Stack:  0000000000000202
    ffffffff00000000 ffffffff00040004 ffff81067dae2640
    Apr 23 13:23:20 mds kernel: 000000004bd18327 00000000000ca54d
    0000000000000000 ffff81067dae2640
    Apr 23 13:23:20 mds kernel: ffffffff00040004 0000000000040004
    0000000000000400 0000000000000000
    Apr 23 13:23:20 mds kernel: Call Trace:
    Apr 23 13:23:20 mds kernel: [<ffffffff8029c0ac>] seq_read+0x105/0x28d
    Apr 23 13:23:20 mds kernel: [<ffffffff80283f23>] vfs_read+0xcb/0x153
    Apr 23 13:23:20 mds kernel: [<ffffffff802842bf>] sys_read+0x45/0x6e
    Apr 23 13:23:20 mds kernel: [<ffffffff80209d8e>] system_call+0x7e/0x83
    Apr 23 13:23:20 mds kernel:
    Apr 23 13:23:20 mds kernel:
    Apr 23 13:23:20 mds kernel: Code: 48 8b 50 20 48 8b 48 28 4c 03 60
    10 4c 03 68 18 48 39 d3 48
    Apr 23 13:23:20 mds kernel: RIP  [<ffffffff883d6680>]
    :obdclass:lprocfs_stats_seq_show+0x80/0x1e0
     mds kernel: CR2: ffffffff00040024
    Apr 23 13:23:20 mds kernel: RSP <ffff8103ba5f9e48>
    Apr 23 13:23:20 mds kernel: CR2: ffffffff00040024


Server and affected client both run Lustre 1.6.7.2 on Debian Etch/x86_64
in this case. The behavior does not change after a client reboot.

All hints on how to solve this are really appreciated.

Kind regards,
    Christopher

-- 
Christopher Huhn
Linux therapist

GSI Helmholtzzentrum fuer Schwerionenforschung GmbH
Planckstr. 1
64291 Darmstadt
http://www.gsi.de/

Gesellschaft mit beschraenkter Haftung

Sitz der Gesellschaft / Registered Office:                    Darmstadt
Handelsregister       / Commercial Register: 
                                        Amtsgericht Darmstadt, HRB 1528

Geschaeftsfuehrung    / Managing Directors:  
                                 Professor Dr. Dr. h.c. Horst Stoecker,
                                                    Christiane Neumann,
                                                   Dr. Hartmut Eickhoff
Vorsitzende des Aufsichtsrates / Supervisory Board Chair:  
                                           Dr. Beatrix Vierkorn-Rudolph
Stellvertreter        / Deputy Chair:                 Dr. Rolf Bernhard

Wojciech Turek

2010-Apr-23 14:07 UTC

head link

[Lustre-discuss] Kernel oops after cat on /proc/fs/lustre/mgs/MGS/exports/*/stats

Hi,

This is a known bug that is fixed in 1.8.2

https://bugzilla.lustre.org/show_bug.cgi?id=21420

Best regards

Wojciech

On 23 April 2010 13:18, Christopher Huhn <C.Huhn at gsi.de> wrote:
> Dear lustre wizards,
>
> we are experiencing problems on our MDS and our Lustre expert is abroad
> (he just attended LUG meeting).
>
> One of the symptoms we observe are reproducible kernel oopses when
> viewing some stats files beneath /proc/fs/lustre/mgs/MGS/exports :
>
>    mds:~# cat /proc/fs/lustre/mgs/MGS/exports/10.12... at tcp/stats
>    Killed
>    mds:~#  mds kernel: Oops: 0000 [38] SMP
>    Apr 23 13:23:19 mds kernel: Unable to handle kernel paging request
>    at ffffffff00040024 RIP:
>    Apr 23 13:23:19 mds kernel: [<ffffffff883d6680>]
>    :obdclass:lprocfs_stats_seq_show+0x80/0x1e0
>    Apr 23 13:23:19 mds kernel: PGD 203067 PUD 0
>    Apr 23 13:23:19 mds kernel: Oops: 0000 [38] SMP
>    Apr 23 13:23:20 mds kernel: CPU 7
>    Apr 23 13:23:20 mds kernel: Modules linked in: mds fsfilt_ldiskfs(F)
>    mgs mgc ldiskfs crc16 lustre lov mdc lquota osc ksocklnd ptlrpc
>    obdclass lnet lvfs libcfs xt_tcpudp iptable_filter ip_tables
>    x_tables drbd cn button ac battery bonding xfs ipmi_si ipmi_devintf
>    ipmi_msghandler serio_raw psmouse joydev pcspkr i2c_i801 i2c_core
>    shpchp pci_hotplug evdev parport_pc parport ext3 jbd mbcache
>    dm_mirror dm_snapshot dm_mod raid10 raid456 xor raid1 raid0
>    multipath linear md_mod sd_mod ide_cd cdrom ata_generic libata
>    generic usbhid hid piix 3w_9xxx floppy ide_core ehci_hcd uhci_hcd
>    e1000 scsi_mod thermal processor fan
>    Apr 23 13:23:20 mds kernel: Pid: 7293, comm: cat Tainted: GF
>    2.6.22+lustre1.6.7.2+0.credativ.etch.1 #2
>    Apr 23 13:23:20 mds kernel: RIP: 0010:[<ffffffff883d6680>]
>    [<ffffffff883d6680>] :obdclass:lprocfs_stats_seq_show+0x80/0x1e0
>    Apr 23 13:23:20 mds kernel: RSP: 0018:ffff8103ba5f9e48  EFLAGS: 00010282
>    Apr 23 13:23:20 mds kernel: RAX: ffffffff00040004 RBX:
>    7fffffffffffffff RCX: 0000000000000006
>    Apr 23 13:23:20 mds kernel: RDX: 0101010101010101 RSI:
>    0000000000000000 RDI: 0000000000000000
>    Apr 23 13:23:20 mds kernel: RBP: 0000000000000000 R08:
>    0000000000000008 R09: 0000000000000000
>    Apr 23 13:23:20 mds kernel: R10: 0000000000000000 R11:
>    0000000000000000 R12: 0000000000000000
>    Apr 23 13:23:20 mds kernel: R13: 0000000000000000 R14:
>    0000000000000000 R15: ffff8108000a1760
>    Apr 23 13:23:20 mds kernel: FS:  00002b4a366786d0(0000)
>    GS:ffff81081004b840(0000) knlGS:0000000000000000
>    Apr 23 13:23:20 mds kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
>    000000008005003b
>    Apr 23 13:23:20 mds kernel: CR2: ffffffff00040024 CR3:
>    000000078f018000 CR4: 00000000000006e0
>    Apr 23 13:23:20 mds kernel: Process cat (pid: 7293, threadinfo
>    ffff8103ba5f8000, task ffff8107dc299530)
>    Apr 23 13:23:20 mds kernel: Stack:  0000000000000202
>    ffffffff00000000 ffffffff00040004 ffff81067dae2640
>    Apr 23 13:23:20 mds kernel: 000000004bd18327 00000000000ca54d
>    0000000000000000 ffff81067dae2640
>    Apr 23 13:23:20 mds kernel: ffffffff00040004 0000000000040004
>    0000000000000400 0000000000000000
>    Apr 23 13:23:20 mds kernel: Call Trace:
>    Apr 23 13:23:20 mds kernel: [<ffffffff8029c0ac>]
seq_read+0x105/0x28d
>    Apr 23 13:23:20 mds kernel: [<ffffffff80283f23>]
vfs_read+0xcb/0x153
>    Apr 23 13:23:20 mds kernel: [<ffffffff802842bf>]
sys_read+0x45/0x6e
>    Apr 23 13:23:20 mds kernel: [<ffffffff80209d8e>]
system_call+0x7e/0x83
>    Apr 23 13:23:20 mds kernel:
>    Apr 23 13:23:20 mds kernel:
>    Apr 23 13:23:20 mds kernel: Code: 48 8b 50 20 48 8b 48 28 4c 03 60
>    10 4c 03 68 18 48 39 d3 48
>    Apr 23 13:23:20 mds kernel: RIP  [<ffffffff883d6680>]
>    :obdclass:lprocfs_stats_seq_show+0x80/0x1e0
>     mds kernel: CR2: ffffffff00040024
>    Apr 23 13:23:20 mds kernel: RSP <ffff8103ba5f9e48>
>    Apr 23 13:23:20 mds kernel: CR2: ffffffff00040024
>
>
> Server and affected client both run Lustre 1.6.7.2 on Debian Etch/x86_64
> in this case. The behavior does not change after a client reboot.
>
> All hints on how to solve this are really appreciated.
>
> Kind regards,
>    Christopher
>
> --
> Christopher Huhn
> Linux therapist
>
> GSI Helmholtzzentrum fuer Schwerionenforschung GmbH
> Planckstr. 1
> 64291 Darmstadt
> http://www.gsi.de/
>
> Gesellschaft mit beschraenkter Haftung
>
> Sitz der Gesellschaft / Registered Office:                    Darmstadt
> Handelsregister       / Commercial Register:
>                                        Amtsgericht Darmstadt, HRB 1528
>
> Geschaeftsfuehrung    / Managing Directors:
>                                 Professor Dr. Dr. h.c. Horst Stoecker,
>                                                    Christiane Neumann,
>                                                   Dr. Hartmut Eickhoff
> Vorsitzende des Aufsichtsrates / Supervisory Board Chair:
>                                           Dr. Beatrix Vierkorn-Rudolph
> Stellvertreter        / Deputy Chair:                 Dr. Rolf Bernhard
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>


-- 
--
Wojciech Turek

Assistant System Manager

High Performance Computing Service
University of Cambridge
Email: wjt27 at cam.ac.uk
Tel: (+)44 1223 763517
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100423/277987e4/attachment.html

Lustre discuss - Apr 2010 - Kernel oops after cat on /proc/fs/lustre/mgs/MGS/exports/*/stats

[Lustre-discuss] Kernel oops after cat on /proc/fs/lustre/mgs/MGS/exports/*/stats

[Lustre-discuss] Kernel oops after cat on /proc/fs/lustre/mgs/MGS/exports/*/stats