CentOS - Mar 2015 - Cannot remount drive after lost iSCSI connection

Hi all,

We've having an issue at the moment where an iSCSI connection was
temporarily lost on a few VMs running CentOS 6 on ESXi.

The problem is, now that the iSCSI connection has returned, we are not able
to remount the drive.

At first the drive is read-only, so I tried '*mount -o remount,rw*'
which
didn't work (still read-only), so then I tried a '*umount*' (which
worked),
but now I get the following error when trying to mount it again:

root at server [~]# mount /backup/
mount: /dev/sdb1 already mounted or /backup busy
root at server [~]# mount /dev/sdb1
mount: /dev/sdb1 already mounted or /backup busy


I have also tried '*echo "- - -" >
/sys/class/scsi_host/host1/scan*' and
checked to see if any processes were using /dev/sdb1 or /backup using '
*fuser*' and '*lsof*'.

I know a reboot will solve it, but I'm trying to avoid that as best I can,
so I'm wondering if anyone else has any other ideas?

Any help would be greatly appreciated, thanks. :)

Regards,

Kyle Thorne

-- 
*The contents of this email are strictly private and confidential unless 
otherwise noted and is intended for the marked recipients only. If you are 
not a marked recipient please disregard and delete this email.*

Any clue on dmesg? I'd remove de disk and rescan...

El jue., 5 de marzo de 2015 a las 7:40, Kyle Thorne (<
kthorne at staff.ventraip.com>) escribi?:
> Hi all,
>
> We've having an issue at the moment where an iSCSI connection was
> temporarily lost on a few VMs running CentOS 6 on ESXi.
>
> The problem is, now that the iSCSI connection has returned, we are not able
> to remount the drive.
>
> At first the drive is read-only, so I tried '*mount -o remount,rw*'
which
> didn't work (still read-only), so then I tried a '*umount*'
(which worked),
> but now I get the following error when trying to mount it again:
>
> root at server [~]# mount /backup/
> mount: /dev/sdb1 already mounted or /backup busy
> root at server [~]# mount /dev/sdb1
> mount: /dev/sdb1 already mounted or /backup busy
>
>
> I have also tried '*echo "- - -" >
/sys/class/scsi_host/host1/scan*' and
> checked to see if any processes were using /dev/sdb1 or /backup using '
> *fuser*' and '*lsof*'.
>
> I know a reboot will solve it, but I'm trying to avoid that as best I
can,
> so I'm wondering if anyone else has any other ideas?
>
> Any help would be greatly appreciated, thanks. :)
>
> Regards,
>
> Kyle Thorne
>
> --
> *The contents of this email are strictly private and confidential unless
> otherwise noted and is intended for the marked recipients only. If you are
> not a marked recipient please disregard and delete this email.*
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

The most recent message is:

[3108269.919256] sd 2:0:1:0: timing out command, waited 1080s
[3108269.919528] sd 2:0:1:0: [sdb] Unhandled error code
[3108269.919535] sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK
driverbyte=DRIVER_OK
[3108269.919540] sd 2:0:1:0: [sdb] CDB: Read(10): 28 00 00 01 21 47 00 00
08 00
[3108269.919586] EXT4-fs error (device sdb1): ext4_find_entry: reading
directory #2 offset 0

Removing the device, rescanning, and then re-adding it worked, but that
moved the device to /dev/sdc instead. Which is fine, but it would be much
better if it was /dev/sdb.

Thanks for your help. :)


On 5 March 2015 at 23:17, Marcelo Roccasalva <
marcelo-centos at irrigacion.gov.ar> wrote:
> Any clue on dmesg? I'd remove de disk and rescan...
>
> El jue., 5 de marzo de 2015 a las 7:40, Kyle Thorne (<
> kthorne at staff.ventraip.com>) escribi?:
>
> > Hi all,
> >
> > We've having an issue at the moment where an iSCSI connection was
> > temporarily lost on a few VMs running CentOS 6 on ESXi.
> >
> > The problem is, now that the iSCSI connection has returned, we are not
> able
> > to remount the drive.
> >
> > At first the drive is read-only, so I tried '*mount -o
remount,rw*' which
> > didn't work (still read-only), so then I tried a
'*umount*' (which
> worked),
> > but now I get the following error when trying to mount it again:
> >
> > root at server [~]# mount /backup/
> > mount: /dev/sdb1 already mounted or /backup busy
> > root at server [~]# mount /dev/sdb1
> > mount: /dev/sdb1 already mounted or /backup busy
> >
> >
> > I have also tried '*echo "- - -" >
/sys/class/scsi_host/host1/scan*' and
> > checked to see if any processes were using /dev/sdb1 or /backup using
'
> > *fuser*' and '*lsof*'.
> >
> > I know a reboot will solve it, but I'm trying to avoid that as
best I
> can,
> > so I'm wondering if anyone else has any other ideas?
> >
> > Any help would be greatly appreciated, thanks. :)
> >
> > Regards,
> >
> > Kyle Thorne
> >
> > --
> > *The contents of this email are strictly private and confidential
unless
> > otherwise noted and is intended for the marked recipients only. If you
> are
> > not a marked recipient please disregard and delete this email.*
> > _______________________________________________
> > CentOS mailing list
> > CentOS at centos.org
> > http://lists.centos.org/mailman/listinfo/centos
> >
> >
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
-- 
*The contents of this email are strictly private and confidential unless 
otherwise noted and is intended for the marked recipients only. If you are 
not a marked recipient please disregard and delete this email.*

Hi,

I have had the same problem and I solved it with following commands :

iscsiadm -m node -o show  (not sure this was useful)
iscsiadm -m node --login 
iscsiadm -m session -o show

after, umount -f and mount each mountpoint and now I can show disks with
command blkid.

Hope will help.
Laurent



--
View this message in context:
http://centos.1050465.n5.nabble.com/CentOS-Cannot-remount-drive-after-lost-iSCSI-connection-tp5734264p5735868.html
Sent from the CentOS mailing list archive at Nabble.com.