CentOS - Jan 2012 - Cannot remove lvs associated with deleted vm guests

At the beginning of January I encountered a problem where
several vm guests on a single host somehow managed to see
the the virtual disks assigned to other guests on the same
hosts.  I was unable to resolve this situation and
shutdown the affected guests after creating new guest
instances and moving the services and data off the
corrupted guests.

I have since removed these guests via virt-manager but all
attempts to remove from the host the logical volumes
associated with the former VirtIO disks fail.  The volumes
are considered open by lvremove and nothing I have tried
can get them to close for removal.  The --force option has
no effect on this situation.

# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
  Can't remove open logical volume "lv_vm_base"

# dmsetup info -c vg_vhost01-lv_vm_base
Name                  Maj Min Stat Open Targ Event  UUID
vg_vhost01-lv_vm_base 253   5 L--w    2    1      0
LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52

# dmsetup remove vg_vhost01-lv_vm_base
device-mapper: remove ioctl failed: Device or resource busy
Command failed


There are several bugs filed on similar issues and udev is
sometimes identified as the culprit.  If I kill the udev
daemon with  T=`pidof -x udevd`; kill $T and rerun the
lvremove -f command then I see this change in behaviour:

# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
  Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
not
/dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
  Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_basep2 not
/dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
  Can't remove open logical volume "lv_vm_base"

I need to get this system stable and return the lost disk
space to the storage pool.  Does anyone have any
suggestions as to how to proceed?

If I cannot solve this using the available system commands
then prudence dictates that I have to re-install the
server OS and rebuild all of the vm guests. As these
guests have been laboriously transferred from other hosts
during the past month this is a task I would rather not
have to do.

Any help is gratefully accepted.

-- 
***          E-Mail is NOT a SECURE channel          ***
James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

Hi,

Its ages since I came across this problem so my memory is a little hazey but
something is obviously holding on to the lv so you'll need to figure out
whats holding it and kill that. The open value returned by "dmsetup
info"? is how many things have the device file open), it looks to be 2 at
the moment and you cant "dmsetup remove" untill its zero (lvm might?
be one of them I cant remember)


A few things worth trying are


1) make sure its not mounted anywhere!
2) if multipathd is running try stopping that.

3) either lsof or fuser on the device file may be able to tell you which process
has it open
4) something vm related might not have let go properly, are there any
deamons/processes etc still running?

5) reboot, the sledgehammer aproach to killing off processes!! 

6) you could try hitting it with dmsetup again, you need to suspend the device
first using "dmsetup suspend" which *may* persuade the holding process
to let go, if it does reduce the open count you'll need to "dmsetup
resume" and then suspend again untill open reaches zero when "dmsetup
remove" should work. I'd try and avoid this option if you can,
you're messing beneath the lvm layer and it may not like that, should be ok
but...


if none of that is possible/works you could try asking on the lvm-linux list.


Reinstalling really shouldn't be necesary.


chris





----- Original Message -----
> From: James B. Byrne <byrnejb at harte-lyne.ca>
> To: centos-virt at centos.org; centos at centos.org
> Cc: 
> Sent: Friday, 27 January 2012, 17:50
> Subject: [CentOS] Cannot remove lvs associated with deleted vm guests
> 
> At the beginning of January I encountered a problem where
> several vm guests on a single host somehow managed to see
> the the virtual disks assigned to other guests on the same
> hosts.? I was unable to resolve this situation and
> shutdown the affected guests after creating new guest
> instances and moving the services and data off the
> corrupted guests.
> 
> I have since removed these guests via virt-manager but all
> attempts to remove from the host the logical volumes
> associated with the former VirtIO disks fail.? The volumes
> are considered open by lvremove and nothing I have tried
> can get them to close for removal.? The --force option has
> no effect on this situation.
> 
> # /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
> ? Can't remove open logical volume "lv_vm_base"
> 
> # dmsetup info -c vg_vhost01-lv_vm_base
> Name? ? ? ? ? ? ? ? ? Maj Min Stat Open Targ Event? UUID
> vg_vhost01-lv_vm_base 253?  5 L--w? ? 2? ? 1? ? ? 0
> LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
> 
> # dmsetup remove vg_vhost01-lv_vm_base
> device-mapper: remove ioctl failed: Device or resource busy
> Command failed
> 
> 
> There are several bugs filed on similar issues and udev is
> sometimes identified as the culprit.? If I kill the udev
> daemon with? T=`pidof -x udevd`; kill $T and rerun the
> lvremove -f command then I see this change in behaviour:
> 
> # /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
> ? Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
> using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
> not
> /dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
> ? Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
> using /dev/mapper/vg_vhost01-lv_vm_basep2 not
> /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
> ? Can't remove open logical volume "lv_vm_base"
> 
> I need to get this system stable and return the lost disk
> space to the storage pool.? Does anyone have any
> suggestions as to how to proceed?
> 
> If I cannot solve this using the available system commands
> then prudence dictates that I have to re-install the
> server OS and rebuild all of the vm guests. As these
> guests have been laboriously transferred from other hosts
> during the past month this is a task I would rather not
> have to do.
> 
> Any help is gratefully accepted.
> 
> -- 
> ***? ? ? ? ? E-Mail is NOT a SECURE channel? ? ? ? ? ***
> James B. Byrne? ? ? ? ? ? ? ? mailto:ByrneJB at Harte-Lyne.ca
> Harte & Lyne Limited? ? ? ? ? http://www.harte-lyne.ca
> 9 Brockley Drive? ? ? ? ? ? ? vox: +1 905 561 1241
> Hamilton, Ontario? ? ? ? ? ?  fax: +1 905 561 0757
> Canada? L8E 3C3
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>

On Sat, January 28, 2012 12:19, Phil Schaffner wrote:
> Pardon the top-post and cross-post, but this apparently
> should have gone
> to centos-virt at centos.org
>
> Phil
>
> chris procter wrote on 01/27/2012 03:29 PM:
>> Hi,
>>
>> Its ages since I came across this problem so my memory
>> is a little hazey but something is obviously holding on
>> to the lv so you'll need to figure out whats holding it
>> and kill that. The open value returned by "dmsetup info"
>>  is how many things have the device file open), it looks
>> to be 2 at the moment and you cant "dmsetup remove"
>> untill its zero (lvm might  be one of them I cant
>> remember)
>>
>>
>> A few things worth trying are
>>
>>
>> 1) make sure its not mounted anywhere!
[root at vhost01 ~]# umount /dev/vg_vhost01/lv_vm_base
umount: /dev/vg_vhost01/lv_vm_base: not mounted
>> 2) if multipathd is running try stopping that.
[root at vhost01 ~]# ps -ef | grep multipathd
root       465   307  0 09:40 pts/0    00:00:00 grep
multipathd
[root at vhost01 ~]# service multipathd status
multipathd: unrecognized service
[root at vhost01 ~]# find / -name multipathd
[root at vhost01 ~]#
>> 3) either lsof or fuser on the device file may be able
[root at vhost01 ~]# fuser /dev/vg_vhost01/lv_vm_base
[root at vhost01 ~]# lsof /dev/vg_vhost01/lv_vm_base
[root at vhost01 ~]#
>> 4) something vm related might not have let go properly,
>> are there any deamons/processes etc still running?
There are many vm qeum processes running, non contain any
reference to lv_vm_base

[root at vhost01 ~]# ps -ef | grep vm_base
root       549   307  0 09:44 pts/0    00:00:00 grep vm_base
[root at vhost01 ~]#
>> 5) reboot, the sledgehammer aproach to killing off
>> processes!!
>>
>> 6) you could try hitting it with dmsetup again, you need
>> to suspend the device first using "dmsetup suspend"
>> which *may* persuade the holding process to let go, if
>> it does reduce the open count you'll need to "dmsetup
>> resume" and then suspend again untill open reaches zero
>> when "dmsetup remove" should work. I'd try and avoid
>> this option if you can, you're messing beneath the lvm
>> layer and it may not like that, should be ok but...
[root at vhost01 ~]#  dmsetup info -c vg_vhost01-lv_vm_base
Name                  Maj Min Stat Open Targ Event  UUID
vg_vhost01-lv_vm_base 253   5 L--w    2    1      0
LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
[root at vhost01 ~]# dmsetup suspend vg_vhost01-lv_vm_base
[root at vhost01 ~]#  dmsetup info -c vg_vhost01-lv_vm_base
Name                  Maj Min Stat Open Targ Event  UUID
vg_vhost01-lv_vm_base 253   5 L-sw    2    1      0
LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
>> if none of that is possible/works you could try asking
>> on the lvm-linux list.
Am so doing.
>>
>>
>> Reinstalling really shouldn't be necesary.
>>
Yes.  However.  .  .


-- 
***          E-Mail is NOT a SECURE channel          ***
James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

This problem was finally resolved with the generous help
of Bryn M. Reeves on the inux-lvm at redhat.com list.

The difficulty was that the utility kpartx had the lvs
opened via their mappings.  Further, the mappings were
created with the -pp option of kpartx and that option had
to be provided to the delete action in order for that to
work:

# kpartx -d -pp /path/to/logical/volume/name

Once the mappings were removed then the lvremove also
worked and the volumes were removed.

A small point, failure to provide the -pp option to the
kpartx -d action does not raise an error.  Nor does it
remove the mapping however.


-- 
***          E-Mail is NOT a SECURE channel          ***
James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3