Lustre discuss - Sep 2007 - Lustre 1.6.2 + 2.6.18 debian kernel OOPS

We''re poking at lustre 1.6.2 in combination with the Debian Stable 
2.6.18 kernel on Ubuntu LTS. The thing is built using a lightly hacked 
version of the Debian packaging from 
svn://alioth.debian.org/svn/pkg-lustre

This seems to work, but one of the OST''s crashed with a callback 
pointing towards mballoc during a small-file creation test.

One additional issue might be that the kernel is compiled with gcc4.0, 
and I don''t remember seeing any notes on whether gcc4-compilation of 
Lustre is supported or not nowadays.

Any ideas/suggestions?

[368924.709222] ----------- [cut here ] --------- [please bite here ] ---------
[368924.716533] Kernel BUG at
/usr/src/modules/lustre/ldiskfs/ldiskfs/mballoc.c:3351
[368924.724241] invalid opcode: 0000 [1] SMP
[368924.728519] CPU 1
[368924.730693] Modules linked in: obdfilter ost fsfilt_ldiskfs mds mgc lustre
lov lquota mdc ksocklnd ptlrpc obdclass lnet lvfs libcfs ldiskfs crc16 xt_tcpudp
iptable_filter ip_tables x_tables openafs ipv6 autofs4 ext2 ext3 jbd mbcache
md_mod psmouse serio_raw parport_pc parport tg3 floppy pcspkr e100 mii
i2c_amd8111 amd_rng i2c_amd756 i2c_core shpchp pci_hotplug sg evdev xfs dm_mod
ide_generic ohci_hcd usbcore generic amd74xx sata_sil libata sd_mod thermal
processor fan 3w_9xxx 3w_xxxx scsi_mod
[368924.776736] Pid: 168, comm: kswapd1 Tainted: PF    
2.6.18-lustre-amd64-k8-smp #1
[368924.784563] RIP: 0010:[<ffffffff883c3a87>]  [<ffffffff883c3a87>]
:ldiskfs:ldiskfs_mb_generate_from_pa+0xb7/0x130
[368924.795217] RSP: 0018:ffff8100f0e59aa0  EFLAGS: 00010297
[368924.800786] RAX: 0000000000000000 RBX: ffff810099f292d0 RCX:
ffff8100f0e59ab8
[368924.808253] RDX: 0000000000000000 RSI: 0000000000001cd1 RDI:
ffff810025b20000
[368924.815736] RBP: ffff810099f292e0 R08: ffff8100f0e59ac0 R09:
ffff8100753f1e10
[368924.823208] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000000
[368924.830710] R13: ffff8100b201f000 R14: ffff81003de614e8 R15:
0000000000001cd0
[368924.838185] FS:  00002b1f6d3ec6d0(0000) GS:ffff8100f160e340(0000)
knlGS:0000000000000000
[368924.846663] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b

[368924.852726] CR2: 00002b3316c34000 CR3: 00000000754c6000 CR4:
00000000000006e0
[368924.852726] CR2: 00002b3316c34000 CR3: 00000000754c6000 CR4:
00000000000006e0
[368924.860228] Process kswapd1 (pid: 168, threadinfo ffff8100f0e58000, task
ffff810037a5f140)
[368924.868879] Stack:  0000000000000000 ffff8100b201f000 ffff8100ecafc000
0000000000000000
[368924.877339]  0000000000001cd1 0000000000000000 0000000000001cd0
0000000000001cd0
[368924.885160]  ffff8100ecafc000 0000000000001cd0 0000000000000001
ffffffff883c5a04
[368924.892791] Call Trace:
[368924.895642]  [<ffffffff883c5a04>]
:ldiskfs:ldiskfs_mb_init_cache+0x6f4/0x8a0[368924.903064] 
[<ffffffff80266ef6>] add_to_page_cache+0x86/0xb0
[368924.909132]  [<ffffffff883c5ce5>]
:ldiskfs:ldiskfs_mb_load_buddy+0x135/0x2e0[368924.916553] 
[<ffffffff883c6ba6>]
:ldiskfs:ldiskfs_mb_discard_inode_preallocations+0x1a6/0x310
[368924.925594]  [<ffffffff80314a11>] __up_write+0x31/0x160
[368924.931032]  [<ffffffff883a82e6>]
:ldiskfs:ldiskfs_discard_reservation+0x36/0xa0
[368924.938774]  [<ffffffff883bc053>]
:ldiskfs:ldiskfs_clear_inode+0x83/0xd0
[368924.945773]  [<ffffffff802aa640>] clear_inode+0xe0/0x120
[368924.951323]  [<ffffffff802aa6e8>] dispose_list+0x68/0x110
[368924.956988]  [<ffffffff802aa99f>] shrink_icache_memory+0x20f/0x290
[368924.963470]  [<ffffffff8026fba8>] shrink_slab+0x108/0x190
[368924.969132]  [<ffffffff80271301>] try_to_free_pages+0x221/0x2c0
[368924.975340]  [<ffffffff80271736>] kswapd+0x396/0x4d0
[368924.980568]  [<ffffffff8024b500>] autoremove_wake_function+0x0/0x30
[368924.987167]  [<ffffffff802713a0>] kswapd+0x0/0x4d0
[368924.992209]  [<ffffffff8024b0e0>] keventd_create_kthread+0x0/0x80

[368924.998610]  [<ffffffff8024b33b>] kthread+0xdb/0x120
[368925.003863]  [<ffffffff8020ac4c>] child_rip+0xa/0x12
[368925.009106]  [<ffffffff8024b0e0>] keventd_create_kthread+0x0/0x80
[368925.015527]  [<ffffffff8021d430>] physflat_send_IPI_mask+0x0/0x80
[368925.021928]  [<ffffffff8024b260>] kthread+0x0/0x120
[368925.027034]  [<ffffffff8020ac42>] child_rip+0x0/0x12
[368925.032266]
[368925.033883]
[368925.033884] Code: 0f 0b 68 80 ec 3c 88 c2 17 0d 8b 54 24 18 8d 34 10 39 f2
7d
[368925.043411] RIP  [<ffffffff883c3a87>]
:ldiskfs:ldiskfs_mb_generate_from_pa+0xb7/0x130
[368925.051612]  RSP <ffff8100f0e59aa0>
[368925.055758]  <4>Lustre:
6817:0:(ldlm_lib.c:511:target_handle_reconnect()) hpfs-OST0001: hpfs-mdtlov_UUID
reconnecting
[369091.616526] Lustre: 6817:0:(ldlm_lib.c:739:target_handle_connect())
hpfs-OST0001: refuse reconnection from hpfs-mdtlov_UUID at 130.239.78.233@tcp to
0xffff810038ade000/2


/Nikke
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
  Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se     |    nikke at hpc2n.umu.se
---------------------------------------------------------------------------
  Editing is a rewording activity.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

On Mon, Sep 24, 2007 at 04:20:09PM +0200, Niklas Edmundsson
wrote:
> [368924.716533] Kernel BUG at
/usr/src/modules/lustre/ldiskfs/ldiskfs/mballoc.c:3351
> [368924.784563] RIP: 0010:[<ffffffff883c3a87>] 
[<ffffffff883c3a87>] :ldiskfs:ldiskfs_mb_generate_from_pa+0xb7/0x130
This bug has already been reported in bugzilla ticket 13438.

Johann

Niklas Edmundsson <Niklas.Edmundsson at hpc2n.umu.se> writes:
> We''re poking at lustre 1.6.2 in combination with the Debian Stable
> 2.6.18 kernel on Ubuntu LTS. The thing is built using a lightly hacked 
> version of the Debian packaging from 
> svn://alioth.debian.org/svn/pkg-lustre
What did you have to change?

MfG
        Goswin

On Tue, 25 Sep 2007, Goswin von Brederlow wrote:
> Niklas Edmundsson <Niklas.Edmundsson at hpc2n.umu.se> writes:
>
>> We''re poking at lustre 1.6.2 in combination with the Debian
Stable
>> 2.6.18 kernel on Ubuntu LTS. The thing is built using a lightly hacked
>> version of the Debian packaging from
>> svn://alioth.debian.org/svn/pkg-lustre
>
> What did you have to change?
Mostly pure packaging fluff...

trunk/debian/control:
- A few dependecies , for example texlive-latex-recommended isn''t
   present in Dapper (or Debian stable for that matter)
- The ${binary:Version} variable isn''t supported by the Dapper version
   of dpkg-gencontrol, I simply used ${Source-Version} instead.

Add trunk/debian/patches/fix-2.6.15-versioncheck.dpatch so it compiles 
on vanilla/ubuntu 2.6.15 (sent to pkg-lustre-maintainers).

trunk/debian/rules: Add kludge to move modules to the right directory 
so they don''t always end up in /lib/modules/2.6.18/

To build the thing we needed to fix a symlink so the lyx binary is 
found as "lyx" (on Dapper it''s named lyx-qt or lyx-xforms).

That''s about it I think.


/Nikke
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
  Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se     |    nikke at hpc2n.umu.se
---------------------------------------------------------------------------
  I know it all, I just can''t remember it all at once.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=