We are in the process of migrating several stand-alone server hosts onto a CentOS-6 kvm virtual host. We also use Webmin to administer our hosts. All of the guests, without exception, have been cloned brom a prototype guest using virt-manager. All of the additional VirtIO disks assigned to some of the guests have been added through virt-manager as well. Recently I have encountered a situation with the development copy of Webmin where I can see an additional VirtIO disk assigned to one guest instance, but not yet added to the lvg of that guest, as an unknown device on different guest. Further, I can see an lv assigned to one guest instance as an lv on a difference guest. The webmin author is aware of this problem and is assiduously trying to determine its cause. My difficulty is answering the question: How is it even possible for an application running under a httpd service on one guest to see anything at all besides the VirtIO storage assigned to that guest? Has anyone else encountered this anomaly? -- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB at Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
On Mon, January 16, 2012 13:05, James B. Byrne wrote:> How is it even possible for an > application running under a httpd service on one guest to > see anything at all besides the VirtIO storage assigned to > that guest? > > Has anyone else encountered this anomaly? >I just cloned a guest instance. The clone prototype was set up with a single VirtIO disk of 8Gbs, divided into a 500 Mb boot and a ~7.1Gb root partition. The root partition was entirely assigned to the basic vg and two lv were created, one for swap and one for the actual root partition. When the guest was cloned there was only one VirtIO disk of 8 Gb assigned to it and this was cloned and given a new name. When I look at the newly cloned guest instance with pvdisplay this is what I see: # pvdisplay Couldn't find device with uuid umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De. --- Physical volume --- PV Name /dev/vda2 VG Name vg_vm_centos_6 PV Size 7.32 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 1874 Free PE 0 Allocated PE 1874 PV UUID djM23m-6Yeb-BQ2x-gPh9-ORMt-dX2i-Ou9xBQ --- Physical volume --- PV Name unknown device VG Name vg_vm_centos_6 PV Size 31.25 GiB / not usable 3.97 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 7999 Free PE 0 Allocated PE 7999 PV UUID umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De When I look at it using vgdisplay then this is what I see: # vgdisplay Couldn't find device with uuid umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De. --- Volume group --- VG Name vg_vm_centos_6 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 10 VG Access read/write VG Status resizable MAX LV 0 Cur LV 4 Open LV 2 Max PV 0 Cur PV 2 Act PV 1 VG Size 38.57 GiB PE Size 4.00 MiB Total PE 9873 Alloc PE / Size 9873 / 38.57 GiB Free PE / Size 0 / 0 VG UUID qa6jwq-5gTp-6mMH-IWl9-OrEK-HjWc-pbaFsa What is going on and how do I fix this? The size of the ghost pv (31Gb) is showing up as the size of the clone's vg whereas the pv for the cloned instance is only 8Gb. I am only using virt-manager to manage disk storage for these guests and I have no idea why or how this mismash is happening. Any ideas? -- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB at Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
If I log on to the newly cloned guest and I try and rename
the vg used by that instance I see this:
[root at vm-centos-6 ~]# vgrename vg_vm_centos_6 vg_vm_renamed
Couldn't find device with uuid
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De.
Cannot change VG vg_vm_centos_6 while PVs are missing.
Consider vgreduce --removemissing.
[root at vm-centos-6 ~]# vgreduce vg_vm_centos_6 --removemissing
Couldn't find device with uuid
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De.
WARNING: Partial LV lv_pgsql needs to be repaired or
removed.
WARNING: Partial LV lv_backups needs to be repaired or
removed.
WARNING: There are still partial LVs in VG vg_vm_centos_6.
To remove them unconditionally use: vgreduce
--removemissing --force.
Proceeding to remove empty missing PVs.
The xmldump for this guest instance contains this:
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'
cache='none'/>
<source dev='/dev/vg_vhost01/lv_vm_test-vg-rename'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1'
unit='0'/>
</disk>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x1'/>
The /dev entries for this vm guest are these:
[root at vm-centos-6 ~]# ll /dev/vg_vm_centos_6/*
lrwxrwxrwx. 1 root root 7 Jan 16 15:36
/dev/vg_vm_centos_6/lv_root -> ../dm-0
lrwxrwxrwx. 1 root root 7 Jan 16 15:36
/dev/vg_vm_centos_6/lv_swap -> ../dm-1
Searching for the missing uuid I find these files:
# find /etc -print | xargs grep
'umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De'
/etc/lvm/archive/vg_vm_centos_6_00005-1429183950.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00002-807578735.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00004-1687335328.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00003-650133889.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00006-1907108135.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/backup/vg_vm_centos_6: id
"umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
Looking in the files of /etc/lvm/archive created by the
vgrename command shows a complete mess of lvs from
different vm guests non of which have anything in common
with the test vm guest, other than they all share the same
volume group name.
Evidently, cloning vm instances for the purpose of setting
up a new vm guest to run with other vms cloned from the
same prototype is not a good idea,
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
Dennis Jacobfeuerborn
2012-Jan-16 21:51 UTC
[CentOS-virt] VirtIO disk 'leakage' across guests?
On 01/16/2012 07:05 PM, James B. Byrne wrote:> We are in the process of migrating several stand-alone > server hosts onto a CentOS-6 kvm virtual host. We also > use Webmin to administer our hosts. All of the guests, > without exception, have been cloned brom a prototype guest > using virt-manager. All of the additional VirtIO disks > assigned to some of the guests have been added through > virt-manager as well. > > Recently I have encountered a situation with the > development copy of Webmin where I can see an additional > VirtIO disk assigned to one guest instance, but not yet > added to the lvg of that guest, as an unknown device on > different guest. Further, I can see an lv assigned to one > guest instance as an lv on a difference guest. > > The webmin author is aware of this problem and is > assiduously trying to determine its cause. My difficulty > is answering the question: How is it even possible for an > application running under a httpd service on one guest to > see anything at all besides the VirtIO storage assigned to > that guest? > > Has anyone else encountered this anomaly?Can you share the (vg|lv|pv)display on two such guests that allegedly share a disk? A dumpxml and lvdisplay of the host side of there two guests would be useful too. The only case where I have seen stuff like this was when someone was using unpartitioned virtual disks as physical volumes in the guests which resulted in the host picking up the metadata and "discovering" the vg/lv/pv data of the guests on the host side. What a mess that was. Regards, Dennis
On 01/16/2012 10:16 PM, James B. Byrne wrote:> WARNING: There are still partial LVs in VG vg_vm_centos_6. > To remove them unconditionally use: vgreduce > --removemissing --force.You have not used "--force", try it. -- Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe Google is the Mother, Google is the Father, and traceroute is your trusty Spiderman... StarOS, Mikrotik and CentOS/RHEL/Linux consultant
Possibly Parallel Threads
- VirtIO disk 'leakage' across guests?
- [CentOS] VirtIO disk 'leakage' across guests?
- "No free sectors available" while try to extend logical volumen in a virtual machine running CentOS 6.5
- Looking for a life-save LVM Guru
- How to remove LVM Physical Volume from Volume Group?