Lustre discuss - Jan 2010 - Lustre claims OST is mounted when it is not

We had an OSS lockup and had to be reset. Heartbeat failed to mount one of
the OSTs and unmounted all of its local OSTs.

I''m trying to run mount on one of the OSTs (ost08) but it claims its
mounted
when it is not.

I have other OSTs mounted so I can''t remove the driver right now. Any
ideas?

Redhat 5.3

[root at oss-0-0 ~]# uname -a
Linux oss-0-0.local 2.6.18-128.7.1.el5_lustre.1.8.1.1 #1 SMP Tue Oct 6
05:48:57 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux

[root at oss-0-0 ~]# mount | grep ost
/dev/dsk/ost12 on /mnt/scratch/ost12 type lustre (rw)
/dev/dsk/ost16 on /mnt/scratch/ost16 type lustre (rw)
/dev/dsk/ost20 on /mnt/scratch/ost20 type lustre (rw)
/dev/dsk/ost00 on /mnt/scratch/ost00 type lustre (rw)
/dev/dsk/ost04 on /mnt/scratch/ost04 type lustre (rw)
/dev/dsk/ost110 on /mnt/scratch/ost110 type lustre (rw)

[root at oss-0-0 ~]# umount -f /mnt/scratch/ost08
umount2: Invalid argument
umount: /mnt/scratch/ost08: not mounted

[root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost08 | tee
/state/partition1/e2fsck-n.ost08_`date
''+%m.%d.%y-%H:%M:%S''`.log
e2fsck 1.41.6.sun1 (30-May-2009)
device /dev/sdj mounted by lustre per
/proc/fs/lustre/obdfilter/scratch-OST0018/mntdev
Warning!  /dev/dsk/ost08 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem
check.
scratch-OST0018 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

Free blocks count wrong (895594326, counted=895594327).

Fix? no

scratch-OST0018: 3363862/244015104 files (1.2% non-contiguous),
80436992/976031318 blocks
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100115/47acba4a/attachment.html

On Friday 15 January 2010, Erik Froese wrote:
> We had an OSS lockup and had to be reset. Heartbeat failed to mount one of
> the OSTs and unmounted all of its local OSTs.
> 
> I''m trying to run mount on one of the OSTs (ost08) but it claims
its
>  mounted when it is not.
> 
> I have other OSTs mounted so I can''t remove the driver right now.
Any
>  ideas?
> 
> Redhat 5.3
> 
> [root at oss-0-0 ~]# uname -a
> Linux oss-0-0.local 2.6.18-128.7.1.el5_lustre.1.8.1.1 #1 SMP Tue Oct 6
> 05:48:57 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux
> 
> [root at oss-0-0 ~]# mount | grep ost
> /dev/dsk/ost12 on /mnt/scratch/ost12 type lustre (rw)
> /dev/dsk/ost16 on /mnt/scratch/ost16 type lustre (rw)
> /dev/dsk/ost20 on /mnt/scratch/ost20 type lustre (rw)
> /dev/dsk/ost00 on /mnt/scratch/ost00 type lustre (rw)
> /dev/dsk/ost04 on /mnt/scratch/ost04 type lustre (rw)
> /dev/dsk/ost110 on /mnt/scratch/ost110 type lustre (rw)
> 
> [root at oss-0-0 ~]# umount -f /mnt/scratch/ost08
> umount2: Invalid argument
> umount: /mnt/scratch/ost08: not mounted
> 
> [root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost08 | tee
> /state/partition1/e2fsck-n.ost08_`date
''+%m.%d.%y-%H:%M:%S''`.log
> e2fsck 1.41.6.sun1 (30-May-2009)
> device /dev/sdj mounted by lustre per
> /proc/fs/lustre/obdfilter/scratch-OST0018/mntdev
> Warning!  /dev/dsk/ost08 is mounted.
> Warning: skipping journal recovery because doing a read-only filesystem
> check.
see here:

https://bugzilla.lustre.org/show_bug.cgi?id=19566
https://bugzilla.lustre.org/show_bug.cgi?id=21359

-- 
Bernd Schubert
DataDirect Networks

Thanks Bernd.
>From the bug reports it looks like the OST is actually still mounted by
lustre, unbeknownst to Linux and VFS.
Is there a mechanism to unmount it or do I need to reboot?

Erik

On Fri, Jan 15, 2010 at 3:28 PM, Bernd Schubert
<bs_lists at aakef.fastmail.fm>wrote:
> On Friday 15 January 2010, Erik Froese wrote:
> > We had an OSS lockup and had to be reset. Heartbeat failed to mount
one
> of
> > the OSTs and unmounted all of its local OSTs.
> >
> > I''m trying to run mount on one of the OSTs (ost08) but it
claims its
> >  mounted when it is not.
> >
> > I have other OSTs mounted so I can''t remove the driver right
now. Any
> >  ideas?
> >
> > Redhat 5.3
> >
> > [root at oss-0-0 ~]# uname -a
> > Linux oss-0-0.local 2.6.18-128.7.1.el5_lustre.1.8.1.1 #1 SMP Tue Oct 6
> > 05:48:57 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux
> >
> > [root at oss-0-0 ~]# mount | grep ost
> > /dev/dsk/ost12 on /mnt/scratch/ost12 type lustre (rw)
> > /dev/dsk/ost16 on /mnt/scratch/ost16 type lustre (rw)
> > /dev/dsk/ost20 on /mnt/scratch/ost20 type lustre (rw)
> > /dev/dsk/ost00 on /mnt/scratch/ost00 type lustre (rw)
> > /dev/dsk/ost04 on /mnt/scratch/ost04 type lustre (rw)
> > /dev/dsk/ost110 on /mnt/scratch/ost110 type lustre (rw)
> >
> > [root at oss-0-0 ~]# umount -f /mnt/scratch/ost08
> > umount2: Invalid argument
> > umount: /mnt/scratch/ost08: not mounted
> >
> > [root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost08 | tee
> > /state/partition1/e2fsck-n.ost08_`date
''+%m.%d.%y-%H:%M:%S''`.log
> > e2fsck 1.41.6.sun1 (30-May-2009)
> > device /dev/sdj mounted by lustre per
> > /proc/fs/lustre/obdfilter/scratch-OST0018/mntdev
> > Warning!  /dev/dsk/ost08 is mounted.
> > Warning: skipping journal recovery because doing a read-only
filesystem
> > check.
>
> see here:
>
> https://bugzilla.lustre.org/show_bug.cgi?id=19566
> https://bugzilla.lustre.org/show_bug.cgi?id=21359
>
> --
> Bernd Schubert
> DataDirect Networks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100115/74f0c48d/attachment.html

Hello Erik,

unfortunately, there is no solution than to reboot. For some unknown (yet to 
debug reasons) variable references could not given up, so in order to prevent 
NULL point dereferences, Lustre did not umount.


Cheers,
Bernd

On Friday 15 January 2010, Erik Froese wrote:
> Thanks Bernd.
> 
> From the bug reports it looks like the OST is actually still mounted by
> lustre, unbeknownst to Linux and VFS.
> Is there a mechanism to unmount it or do I need to reboot?
> 
> Erik
> 
> On Fri, Jan 15, 2010 at 3:28 PM, Bernd Schubert
> 
> <bs_lists at aakef.fastmail.fm>wrote:
> > On Friday 15 January 2010, Erik Froese wrote:
> > > We had an OSS lockup and had to be reset. Heartbeat failed to
mount one
> >
> > of
> >
> > > the OSTs and unmounted all of its local OSTs.
> > >
> > > I''m trying to run mount on one of the OSTs (ost08) but
it claims its
> > >  mounted when it is not.
> > >
> > > I have other OSTs mounted so I can''t remove the driver
right now. Any
> > >  ideas?
> > >
> > > Redhat 5.3
> > >
> > > [root at oss-0-0 ~]# uname -a
> > > Linux oss-0-0.local 2.6.18-128.7.1.el5_lustre.1.8.1.1 #1 SMP Tue
Oct 6
> > > 05:48:57 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > [root at oss-0-0 ~]# mount | grep ost
> > > /dev/dsk/ost12 on /mnt/scratch/ost12 type lustre (rw)
> > > /dev/dsk/ost16 on /mnt/scratch/ost16 type lustre (rw)
> > > /dev/dsk/ost20 on /mnt/scratch/ost20 type lustre (rw)
> > > /dev/dsk/ost00 on /mnt/scratch/ost00 type lustre (rw)
> > > /dev/dsk/ost04 on /mnt/scratch/ost04 type lustre (rw)
> > > /dev/dsk/ost110 on /mnt/scratch/ost110 type lustre (rw)
> > >
> > > [root at oss-0-0 ~]# umount -f /mnt/scratch/ost08
> > > umount2: Invalid argument
> > > umount: /mnt/scratch/ost08: not mounted
> > >
> > > [root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost08 | tee
> > > /state/partition1/e2fsck-n.ost08_`date
''+%m.%d.%y-%H:%M:%S''`.log
> > > e2fsck 1.41.6.sun1 (30-May-2009)
> > > device /dev/sdj mounted by lustre per
> > > /proc/fs/lustre/obdfilter/scratch-OST0018/mntdev
> > > Warning!  /dev/dsk/ost08 is mounted.
> > > Warning: skipping journal recovery because doing a read-only
filesystem
> > > check.
> >
> > see here:
> >
> > https://bugzilla.lustre.org/show_bug.cgi?id=19566
> > https://bugzilla.lustre.org/show_bug.cgi?id=21359
> >
> > --
> > Bernd Schubert
> > DataDirect Networks
> 

-- 
Bernd Schubert
DataDirect Networks