Lustre discuss - Mar 2011 - clean unmounting of OST with external journal

I notice the following behaviour on a test system while setting up init.d
scripts to allow clean shutdown and re-start of an OSS (lustre 1.8.4)

format an OST with an external journal. The external journal is located on a
raid10 array connected to one of the network interfaces (aoe).

the OST was tested in a filesystem and appears to operate normally.
I can cleanly mount and unmount the OST with mount / umount

however the following sequence leads to an unmountable filesystem on startup

sync
umount  the OST
sync (again for certainty)
stop the internal and external raid arrays
stop network interface

restart network interface
bring up the internal and external raid arrays

now an attempt to re-mount the OST fails with 
LDISKFS-fs (md14): failed to open journal device unknown-block(152,225): -6

an e2fsck fixes this external superblock 

[root at OST2 ~]# e2fsck -j /dev/etherd/e9.24p1 /dev/md14
e2fsck 1.41.10.sun2 (24-Feb-2010)
Superblock hint for external superblock should be 0x409802.  Fix<y>? yes

the subsequent OST is marked clean and can be re-mounted.

I am not sure why this is happening. The external raid partitions of relevance
are all presented under the same /dev/ device when they are stopped/re-started.

any ideas?

the workaround is maybe to put an e2fsck into the startup scripts (maybe not a
bad idea?) but seems like a kludge.



Professor Samuel Aparicio BM BCh PhD FRCPath
Nan and Lorraine Robertson Chair UBC/BC Cancer Agency

On Sun, 2011-03-06 at 17:43 -0800, Samuel Aparicio
wrote:
> I am not sure why this is happening. The external raid partitions of
> relevance are all presented under the same /dev/ device when they are
> stopped/re-started.
I don''t think I''ve seen the exact messages you''ve
produced, but I have
seen similar issues when the external journal isn''t on the same device
major/minor as it is expected to be on.

I would suggest checking /proc/partitions and/or ls -l on the device to
see if the major or minor number is changing. It looks like the AOE
major is fixed to 152, so perhaps the minor is the one changing on you.
I don''t know enough about AOE to know if there''s a way to
force a
specific minor, though.

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

Hello!

On Mar 6, 2011, at 8:43 PM, Samuel Aparicio wrote:
> now an attempt to re-mount the OST fails with 
> LDISKFS-fs (md14): failed to open journal device unknown-block(152,225): -6
> 
> an e2fsck fixes this external superblock 
> 
> [root at OST2 ~]# e2fsck -j /dev/etherd/e9.24p1 /dev/md14
> e2fsck 1.41.10.sun2 (24-Feb-2010)
> Superblock hint for external superblock should be 0x409802.  Fix<y>?
yes
Hm, to me it sounds like after you restarted the network and reimport AoE,
then it created the device with different minor/major.
I believe that the path to the journal is stored in superblock or some such, not
just the "hint" on how to find the journal.
(hint is just major:minor).
(-6 is "No such device or address", I wonder where does that come from
inside of the kernel)
> 
> the subsequent OST is marked clean and can be re-mounted.
> 
> I am not sure why this is happening. The external raid partitions of
relevance are all presented under the same /dev/ device when they are
stopped/re-started.
> any ideas?
Did you check that device major,minor pair is the same every time?

I guess you can use journal_dev mount option to point ldiskfs at the proper
journal device.

Bye,
    Oleg

i just confirmed this is what is happening: the minor number changes after a
network/driver reload. I will have to look into the mechanics to see if this can
be made fixed.

... don''t see a journal_dev option in mount.lustre although the path to
the device is hardcoded at mkfs time ...
is the hardcoded path made use of .. it looks as if it''s the superblock
hint.

Professor Samuel Aparicio BM BCh PhD FRCPath
Nan and Lorraine Robertson Chair UBC/BC Cancer Agency


On Mar 6, 2011, at 6:44 PM, Oleg Drokin wrote:
> Hello!
> 
> On Mar 6, 2011, at 8:43 PM, Samuel Aparicio wrote:
>> now an attempt to re-mount the OST fails with 
>> LDISKFS-fs (md14): failed to open journal device
unknown-block(152,225): -6
>> 
>> an e2fsck fixes this external superblock 
>> 
>> [root at OST2 ~]# e2fsck -j /dev/etherd/e9.24p1 /dev/md14
>> e2fsck 1.41.10.sun2 (24-Feb-2010)
>> Superblock hint for external superblock should be 0x409802. 
Fix<y>? yes
> 
> Hm, to me it sounds like after you restarted the network and reimport AoE,
> then it created the device with different minor/major.
> I believe that the path to the journal is stored in superblock or some
such, not just the "hint" on how to find the journal.
> (hint is just major:minor).
> (-6 is "No such device or address", I wonder where does that come
from inside of the kernel)
> 
>> 
>> the subsequent OST is marked clean and can be re-mounted.
>> 
>> I am not sure why this is happening. The external raid partitions of
relevance are all presented under the same /dev/ device when they are
stopped/re-started.
>> any ideas?
> 
> Did you check that device major,minor pair is the same every time?
> 
> I guess you can use journal_dev mount option to point ldiskfs at the proper
journal device.
> 
> Bye,
>    Oleg

On Sun, Mar 06, 2011 at 07:20:32PM -0800, Samuel Aparicio
wrote:
> ... don''t see a journal_dev option in mount.lustre although the
path to the device is hardcoded at mkfs time ...
Although the -o journal_dev=0xMMmm option is not documented in the man page, it
should still work with mount.lustre.
We used to have a bug here, but it was fixed in 1.6.7/1.8.0/2.0.0 (see bugzilla
ticket 16140 for more information).

Cheers,
Johann