libvirt users - Jun 2019 - Re: Using qemu active blockcommit results in 'Permission denied' error

Hi Peter,

On 31.05.19 09:57, Peter Krempa wrote:
> On Thu, May 30, 2019 at 22:12:14 +0200, Marcus Hoffmann wrote:
>> Hello all,
> 
> Hi,
> 
>>
>> I tried following this guide:
>> https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
>>
>> Unfortunately when I try to do the final virsh blockcommit step I
always
>> get the following error:
>>
>> error: internal error: unable to execute QEMU command
'block-commit':
>> Could not reopen file: Permission denied
>>
>> I checked directory und image file permissions, app-armor profiles (set
>> to complain mode for now.) and libvirt logs but nothing there gives me
>> any hints what might be going wrong.
>>
>> This is on a debian buster system, using
> 
> I was doing some changes in the blockcommit code recently so I might
> have messed something up. Could you please collect debug logs:
> 
> https://wiki.libvirt.org/page/DebugLogs
> 
> when the problem reproduces and also a directory listing of the path
> where the image is stored so I can check if the permission code is
> working properly.
> 

Logfile attached.

~ # ls -al /var/lib/libvirt/images/
total 1271917588
drwxr-xr-x 2 root         root                 4096 Mai 29 14:47 .
drwxr-xr-x 7 root         root                 4096 Mai 27 15:55 ..
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 214821961728 Mai 29 14:47
drache3.qcow2
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 395671371776 Mai 29 14:47
drache_addon.qcow2
-rw-r--r-- 1 libvirt-qemu libvirt-qemu  90944765952 Mai 31 14:02
drache_overlaya.qcow2
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 115860766720 Mai 31 14:02
drache_overlayb.qcow2

Marcus

On May 31, 2019 2:03:40 PM GMT+02:00, Marcus Hoffmann <bubu@bubu1.eu>
wrote:
>Hi Peter,
>
>On 31.05.19 09:57, Peter Krempa wrote:
>> On Thu, May 30, 2019 at 22:12:14 +0200, Marcus Hoffmann wrote:
>>> Hello all,
>> 
>> Hi,
>> 
>>>
>>> I tried following this guide:
>>>
>https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
>>>
>>> Unfortunately when I try to do the final virsh blockcommit step I
>always
>>> get the following error:
>>>
>>> error: internal error: unable to execute QEMU command
>'block-commit':
>>> Could not reopen file: Permission denied
>>>
>>> I checked directory und image file permissions, app-armor profiles
>(set
>>> to complain mode for now.) and libvirt logs but nothing there gives
>me
>>> any hints what might be going wrong.
>>>
>>> This is on a debian buster system, using
>> 
>> I was doing some changes in the blockcommit code recently so I might
>> have messed something up. Could you please collect debug logs:
>> 
>> https://wiki.libvirt.org/page/DebugLogs
>> 
>> when the problem reproduces and also a directory listing of the path
>> where the image is stored so I can check if the permission code is
>> working properly.
>> 
>
>
>Logfile attached.
>
>~ # ls -al /var/lib/libvirt/images/
>total 1271917588
>drwxr-xr-x 2 root         root                 4096 Mai 29 14:47 .
>drwxr-xr-x 7 root         root                 4096 Mai 27 15:55 ..
>-rw-r--r-- 1 libvirt-qemu libvirt-qemu 214821961728 Mai 29 14:47
>drache3.qcow2
>-rw-r--r-- 1 libvirt-qemu libvirt-qemu 395671371776 Mai 29 14:47
>drache_addon.qcow2
>-rw-r--r-- 1 libvirt-qemu libvirt-qemu  90944765952 Mai 31 14:02
>drache_overlaya.qcow2
>-rw-r--r-- 1 libvirt-qemu libvirt-qemu 115860766720 Mai 31 14:02
>drache_overlayb.qcow2
>
>Marcus
Hi Peter,

did you get the logfile I sent? (Just wanting to make sure the email actually
got delivered. I get a lot of dmarc failures from the mailing list :-/)

Did you have a chance to take a look?

Best,
Marcus

On Wed, Jun 05, 2019 at 17:02:05 +0200, Marcus Hoffmann
wrote:
> On May 31, 2019 2:03:40 PM GMT+02:00, Marcus Hoffmann <bubu@bubu1.eu>
wrote:
[...]
> Hi Peter,
Hi,
> 
> did you get the logfile I sent? (Just wanting to make sure the email
actually got delivered. I get a lot of dmarc failures from the mailing list :-/)
> 
> Did you have a chance to take a look?
I did in fact get it, but I somehow forgot to look at it. I'm marking
this and I'll have a look tomorrow.

Sorry for that.

On Fri, May 31, 2019 at 14:03:40 +0200, Marcus Hoffmann
wrote:
> Hi Peter,
> 
> On 31.05.19 09:57, Peter Krempa wrote:
> > On Thu, May 30, 2019 at 22:12:14 +0200, Marcus Hoffmann wrote:
> >> Hello all,
> > 
> > Hi,
> > 
> >>
> >> I tried following this guide:
> >>
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
> >>
> >> Unfortunately when I try to do the final virsh blockcommit step I
always
> >> get the following error:
> >>
> >> error: internal error: unable to execute QEMU command
'block-commit':
> >> Could not reopen file: Permission denied
I managed to reproduce this issue but when using selinux. I'll try to
fix it with selinux and will try to assess whether it has the possiblity
to fix apparmor too. I'll cc you on a patch when I'll be able to fix it.

On Tue, Jun 11, 2019 at 14:35:46 +0200, Peter Krempa
wrote:
> On Fri, May 31, 2019 at 14:03:40 +0200, Marcus Hoffmann wrote:
> > Hi Peter,
> > 
> > On 31.05.19 09:57, Peter Krempa wrote:
> > > On Thu, May 30, 2019 at 22:12:14 +0200, Marcus Hoffmann wrote:
> > >> Hello all,
> > > 
> > > Hi,
> > > 
> > >>
> > >> I tried following this guide:
> > >>
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
> > >>
> > >> Unfortunately when I try to do the final virsh blockcommit
step I always
> > >> get the following error:
> > >>
> > >> error: internal error: unable to execute QEMU command
'block-commit':
> > >> Could not reopen file: Permission denied
> 
> I managed to reproduce this issue but when using selinux. I'll try to
> fix it with selinux and will try to assess whether it has the possiblity
> to fix apparmor too. I'll cc you on a patch when I'll be able to
fix it.

Well,

The problem I managed to fix had the same symptoms but probably was not
what you see, as you are using libvirt 5.0.0 and I broke the permissions
code in libvirt 5.4.0.

Unfortunately I can't tell what's wrong from the debug logs you've
provided. Is there a possibility to collect anything from apparmor? In
selinux world we do collect denials of the security model in a log file
which might indicate what's happening.

Also I've pushed a patch which adds more logging to the
permission-changing code executed while doing blockjobs:

commit e6635c626a252669c79a84fe0a2af11a361aa341 (HEAD -> master,
origin/master, origin/HEAD)
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Jun 12 13:49:57 2019 +0200

    qemu: domain: Log some useful data in qemuDomainStorageSourceAccessModify
    
    Log the flags passed to the function in a exploded state so that it's
    easily visible what's happening to the image.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Ján Tomko <jtomko@redhat.com>

Unfortunately that commit can't be applied to libvirt 5.0 because it
depends on a refactor which I pushed in 5.4 (which also caused the
problem I was fixing recently). If you could test the upstream version
it would be great.

Thanks for reporting the problem and I'd be grateful if you could
collect logs from the apparmor security thing.