libvirt users - Aug 2019 - Starting VM fails with: "Setting different DAC user or group on /path... which is already in use" after upgrading to libvirt 5.6.0-1

Hi,

I upgraded to a Fedora 29 host using virt-preview repo to
libvirt-daemon-5.6.0-1.fc29.x86_64
The host was using plain Fedora 29 without virt-preview before that.

After the upgrade, starting some vms that were running fine fail now with
this error:

Error starting domain: internal error: child reported (status=125):
Requested operation is not valid: Setting different DAC user or group on
/home/libvirt/images/voodoo4-os.img which is already in use

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in
cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in
tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py",
line
66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line
1279,
in startup
    self._backend.create()
  File "/usr/lib64/python3.7/site-packages/libvirt.py", line 1089, in
create
    if ret == -1: raise libvirtError ('virDomainCreate() failed',
dom=self)
libvirt.libvirtError: internal error: child reported (status=125):
Requested operation is not valid: Setting different DAC user or group on
/home/libvirt/images/voodoo4-os.img which is already in use

These vms we created by creating one vm, and the cloning the vms.

I tried to delete the disks and add them back in one of the vms, but the vm
still fail with the
same error.

I hope that someone have a clue what is the issue, and how it can be fixed.

Here some details about the setup:

vm1:

<domain type='kvm'>
  <name>voodoo4</name>
  <uuid>0b3aa57a-00b6-4e99-81f9-8f216f85ccaf</uuid>
  <title>voodoo4 (fedora 29, gluster)</title>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="
http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://fedoraproject.org/fedora/29"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64'
machine='pc-q35-3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0'
target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/voodoo4-os.img'/>
      <target dev='vda' bus='virtio'/>
      <serial>os</serial>
      <address type='pci' domain='0x0000' bus='0x04'
slot='0x00'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/voodoo4-gv0.img'/>
      <target dev='vdb' bus='virtio'/>
      <serial>gv0</serial>
      <address type='pci' domain='0x0000' bus='0x05'
slot='0x00'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/voodoo4-gv1.img'/>
      <target dev='vdc' bus='virtio'/>
      <serial>gv1</serial>
      <address type='pci' domain='0x0000' bus='0x08'
slot='0x00'
function='0x0'/>
    </disk>
    <controller type='usb' index='0'
model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02'
slot='0x00'
function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x1f'
function='0x2'/>
    </controller>
    <controller type='pci' index='0'
model='pcie-root'/>
    <controller type='pci' index='1'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x1'/>
    </controller>
    <controller type='pci' index='3'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x2'/>
    </controller>
    <controller type='pci' index='4'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x3'/>
    </controller>
    <controller type='pci' index='5'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x4'/>
    </controller>
    <controller type='pci' index='6'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x5'/>
    </controller>
    <controller type='pci' index='7'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x6'/>
    </controller>
    <controller type='pci' index='8'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x7'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03'
slot='0x00'
function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:dd:a4:5c'/>
      <source bridge='ovirt'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01'
slot='0x00'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio'
name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0'
bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0'
bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x1b'
function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536'
vgamem='16384' heads='1'
primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06'
slot='0x00'
function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x07'
slot='0x00'
function='0x0'/>
    </rng>
  </devices>
</domain>


vm 2:

<domain type='kvm'>
  <name>voodoo5</name>
  <uuid>8ded8ea2-6524-4fc0-94f6-31667338a5f2</uuid>
  <title>voodoo5 (fedora 29, gluster)</title>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="
http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://fedoraproject.org/fedora/29"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64'
machine='pc-q35-3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/voodoo5-os.img'/>
      <target dev='vda' bus='virtio'/>
      <serial>os</serial>
      <address type='pci' domain='0x0000' bus='0x04'
slot='0x00'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/voodoo5-gv0.img'/>
      <target dev='vdb' bus='virtio'/>
      <serial>gv0</serial>
      <address type='pci' domain='0x0000' bus='0x05'
slot='0x00'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/voodoo5-gv1.img'/>
      <target dev='vdc' bus='virtio'/>
      <serial>gv1</serial>
      <address type='pci' domain='0x0000' bus='0x08'
slot='0x00'
function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0'
target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'
model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02'
slot='0x00'
function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x1f'
function='0x2'/>
    </controller>
    <controller type='pci' index='0'
model='pcie-root'/>
    <controller type='pci' index='1'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x1'/>
    </controller>
    <controller type='pci' index='3'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x2'/>
    </controller>
    <controller type='pci' index='4'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x3'/>
    </controller>
    <controller type='pci' index='5'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x4'/>
    </controller>
    <controller type='pci' index='6'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x5'/>
    </controller>
    <controller type='pci' index='7'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x6'/>
    </controller>
    <controller type='pci' index='8'
model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x7'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03'
slot='0x00'
function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:53:6a:b4'/>
      <source bridge='ovirt'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01'
slot='0x00'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio'
name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0'
bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0'
bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x1b'
function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536'
vgamem='16384' heads='1'
primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06'
slot='0x00'
function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x07'
slot='0x00'
function='0x0'/>
    </rng>
  </devices>
</domain>


ls -lhZ /home/libvirt/images/voodoo4*
# ls -lhZ /home/libvirt/images/voodoo4*
-rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17 03:55
/home/libvirt/images/voodoo4-gv0.img
-rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17 03:55
/home/libvirt/images/voodoo4-gv1.img
-rw-------. 1 root root system_u:object_r:virt_image_t:s0 50G Aug 17 03:52
/home/libvirt/images/voodoo4-os.img


cat /etc/libvirt/storage/images.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
  virsh pool-edit images
or other application using the libvirt API.
-->

The related pool:

<pool type='dir'>
  <name>images</name>
  <uuid>f7190095-947d-442b-b94b-4a99790795bc</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
  </source>
  <target>
    <path>/home/libvirt/images</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>

Nir

On 8/19/19 9:53 PM, Nir Soffer wrote:
> Hi,
> 
> I upgraded to a Fedora 29 host using virt-preview repo to
> libvirt-daemon-5.6.0-1.fc29.x86_64
> The host was using plain Fedora 29 without virt-preview before that.
> 
> After the upgrade, starting some vms that were running fine fail now with
> this error:
> 
> Error starting domain: internal error: child reported (status=125):
> Requested operation is not valid: Setting different DAC user or group on
> /home/libvirt/images/voodoo4-os.img which is already in use
> 
> Traceback (most recent call last):
>    File "/usr/share/virt-manager/virtManager/asyncjob.py", line
75, in
> cb_wrapper
>      callback(asyncjob, *args, **kwargs)
>    File "/usr/share/virt-manager/virtManager/asyncjob.py", line
111, in tmpcb
>      callback(*args, **kwargs)
>    File
"/usr/share/virt-manager/virtManager/object/libvirtobject.py", line
> 66, in newfn
>      ret = fn(self, *args, **kwargs)
>    File "/usr/share/virt-manager/virtManager/object/domain.py",
line 1279,
> in startup
>      self._backend.create()
>    File "/usr/lib64/python3.7/site-packages/libvirt.py", line
1089, in create
>      if ret == -1: raise libvirtError ('virDomainCreate() failed',
dom=self)
> libvirt.libvirtError: internal error: child reported (status=125):
> Requested operation is not valid: Setting different DAC user or group on
> /home/libvirt/images/voodoo4-os.img which is already in use
> 
> These vms we created by creating one vm, and the cloning the vms.
> 
> I tried to delete the disks and add them back in one of the vms, but the vm
> still fail with the
> same error.
> 
> I hope that someone have a clue what is the issue, and how it can be fixed.
How do you clone the vms? The error message suggests that the image is 
in use - is it possible that you're trying to start two domains over the 
same disk?

Michal

On Tue, Aug 20, 2019 at 1:12 PM Michal Privoznik <mprivozn@redhat.com>
wrote:
> On 8/19/19 9:53 PM, Nir Soffer wrote:
> > Hi,
> >
> > I upgraded to a Fedora 29 host using virt-preview repo to
> > libvirt-daemon-5.6.0-1.fc29.x86_64
> > The host was using plain Fedora 29 without virt-preview before that.
> >
> > After the upgrade, starting some vms that were running fine fail now
with
> > this error:
> >
> > Error starting domain: internal error: child reported (status=125):
> > Requested operation is not valid: Setting different DAC user or group
on
> > /home/libvirt/images/voodoo4-os.img which is already in use
> >
> > Traceback (most recent call last):
> >    File "/usr/share/virt-manager/virtManager/asyncjob.py",
line 75, in
> > cb_wrapper
> >      callback(asyncjob, *args, **kwargs)
> >    File "/usr/share/virt-manager/virtManager/asyncjob.py",
line 111, in
> tmpcb
> >      callback(*args, **kwargs)
> >    File
"/usr/share/virt-manager/virtManager/object/libvirtobject.py",
> line
> > 66, in newfn
> >      ret = fn(self, *args, **kwargs)
> >    File
"/usr/share/virt-manager/virtManager/object/domain.py", line
> 1279,
> > in startup
> >      self._backend.create()
> >    File "/usr/lib64/python3.7/site-packages/libvirt.py",
line 1089, in
> create
> >      if ret == -1: raise libvirtError ('virDomainCreate()
failed',
> dom=self)
> > libvirt.libvirtError: internal error: child reported (status=125):
> > Requested operation is not valid: Setting different DAC user or group
on
> > /home/libvirt/images/voodoo4-os.img which is already in use
> >
> > These vms we created by creating one vm, and the cloning the vms.
> >
> > I tried to delete the disks and add them back in one of the vms, but
the
> vm
> > still fail with the
> > same error.
> >
> > I hope that someone have a clue what is the issue, and how it can be
> fixed.
>
> How do you clone the vms?

Using virt-manager "Clone..." command.

The error message suggests that the image is
> in use - is it possible that you're trying to start two domains over
the
> same disk?
>
No, the disks are different (these vms were running for 2-3 weeks) before
the upgrade.

# ls -lhiZ /home/libvirt/images/voodoo{4,5,8}-*.img
  247 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17
03:55 /home/libvirt/images/voodoo4-gv0.img
37249 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17
03:55 /home/libvirt/images/voodoo4-gv1.img
37252 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 50G Aug 17
03:52 /home/libvirt/images/voodoo4-os.img
37250 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17
03:55 /home/libvirt/images/voodoo5-gv0.img
37281 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17
03:55 /home/libvirt/images/voodoo5-gv1.img
  223 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 50G Aug 17
03:45 /home/libvirt/images/voodoo5-os.img
37253 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17
03:55 /home/libvirt/images/voodoo8-gv0.img
37282 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 20G Aug 17
03:55 /home/libvirt/images/voodoo8-gv1.img
37251 -rw-------. 1 root root system_u:object_r:virt_image_t:s0 50G Aug 17
03:46 /home/libvirt/images/voodoo8-os.img

I suspected that the disk serial is the issue - I'm using
the same serial for for all "os" and "gvX" disks, used in
the guest
to locate the right disk (e.g. /dev/disk/by-id/virtio-gv0).

I tried to replace to use unique serials (e.g. gv0 -> voodoo4-gv0) but I
still see
the same error, so this must be something else.

>
> Michal
>