Libguestfs - Nov 2013 - HA: Accessing iSCSI disc images from the RHEV Manager using libguestfs

On 11/21/2013 05:56 PM, Richard W.M. Jones wrote:
> [Adding Itamar: Any suggestions on where Vitaly could get help on this
> RHEV-M API problem?]
>
> On Thu, Nov 21, 2013 at 02:27:08PM +0000, Исаев Виталий Анатольевич wrote:
>> Dear libguestfs developers,
>>
>> We are developing the protected enterprise system based on several
>> Red Hat technologies (Red Hat Enterprise Virtualisation, the IdM
>> server and so on). Our current goal is to prevent the booting of the
>> virtual machine in case of integrity loss. So what we are trying to
>> do Is to intercept the event of the VM's start (using the RHEV
>> Manager hook scripts), mount the starting VM's disk image with
>> libguestfs and check the hash sums of the system files.
>>
>> When we have a local storage of disk images (manager, hypervisor and
>> storage - all in one host), this approach works just perfect.
>>
>> But even a little bit complex network (with at least one manager,
>> one hypervisor and one iSCSI/FC storage - three standalone physical
>> hosts) blows our minds. RHEV Manager API provides the list of disk
>> images which are (in this configurations) are logical volumes on the
>> hypervisor and appear in hypervisor's /dev/mapper directory. But
>> libguestfs from RHEV manager cannot access them.
>
> When you say "cannot access them" do you get an error message? 
Could
> it be an SELinux denial?
>
> I can tell you that libguestfs *should* be able to open any disk image
> or block device.  But libguestfs doesn't have a magic way to bypass
> permissions: if the block device needs (for example) root permissions
> or SELinux is stopping things, libguestfs won't be able to open the
> file, in the same way that /bin/cat or vi couldn't open it.
>
>> Examples from libguestfs and guestfish guide (for instance, over
>> ssh)
>
> Remote access over ssh isn't enabled in RHEL 6 (nor in RHEL 7, sadly).
> Is this a remote disk?  RHEL 7.0 will support NBD remote access only,
> which probably doesn't help you.
>
> Fedora of course enables every possible remote access method, but you
> get to keep the pieces if it breaks.
>
>> do not work. And even more, which host should we access - a
>> hypervisor (with a different logical volumes) or a storage (with a
>> monolith volume group)?
>
> Itamar can probably help better than me here.
>
>> Software version: RHEV 3.1, libguestfs 1.16.34, guest OS - RHEL 6.4,
arch: x86_64.
>>
>> P.S. May be I had to post this message directly to the support, but
I'd suggest that this topic would be useful for community.
>>
>> Thank you,
>> Виталий Исаев
>> Инженер-программист
>> Группа разработки и внедрения ПСЗИ
>> Департамент информационной безопасности
>> ОАО <Финтех>
>
> Rich.
>
i may be missing something, but wouldn't vdsm hooks be a better fit:
http://www.ovirt.org/VDSM-Hooks

for example, I think the "checkimages" one would cover what you are 
trying to do already?
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=tree;f=vdsm_hooks;

you can also ask such questions on users@ovirt.org.

________________________________________
От: Itamar Heim [iheim@redhat.com]
Отправлено: 21 ноября 2013 г. 20:10
Кому: Richard W.M. Jones; Исаев Виталий Анатольевич
Копия: libguestfs@redhat.com
Тема: Re: [Libguestfs] Accessing iSCSI disc images from the RHEV Manager using
libguestfs

On 11/21/2013 05:56 PM, Richard W.M. Jones wrote:
> [Adding Itamar: Any suggestions on where Vitaly could get help on this
> RHEV-M API problem?]
>
> On Thu, Nov 21, 2013 at 02:27:08PM +0000, Исаев Виталий Анатольевич wrote:
>> Dear libguestfs developers,
>>
>> We are developing the protected enterprise system based on several
>> Red Hat technologies (Red Hat Enterprise Virtualisation, the IdM
>> server and so on). Our current goal is to prevent the booting of the
>> virtual machine in case of integrity loss. So what we are trying to
>> do Is to intercept the event of the VM's start (using the RHEV
>> Manager hook scripts), mount the starting VM's disk image with
>> libguestfs and check the hash sums of the system files.
>>
>> When we have a local storage of disk images (manager, hypervisor and
>> storage - all in one host), this approach works just perfect.
>>
>> But even a little bit complex network (with at least one manager,
>> one hypervisor and one iSCSI/FC storage - three standalone physical
>> hosts) blows our minds. RHEV Manager API provides the list of disk
>> images which are (in this configurations) are logical volumes on the
>> hypervisor and appear in hypervisor's /dev/mapper directory. But
>> libguestfs from RHEV manager cannot access them.
>
> When you say "cannot access them" do you get an error message? 
Could
> it be an SELinux denial?
Yes, when I am trying to open some logical volume on the hypervisor side, for
instance:
    $ guestfish -a /dev/mapper/dm-xx
libguestfs fails with an error message kind of that:
    $ libguestfs-supermin-helper: failed to find a suitable kernel.
    I looked for kernels in /boot and modules in /lib/modules.
    If this is a Xen guest, and you only have Xen domU kernels
    installed, try installing a fullvirt kernel (only for
    libguestfs use, you shouldn't boot the Xen guest with it).
So this is correct because hypervisor's file system is minimalist and
therefore contains
kernel files neither in /boot nor in /lib/modules. 

And it is still not clear for me, how should I implement interaction between the
RHEV-M API,
which tells me what logical volume (disk image) belongs to the every VM, and the
hypervisor's, to which
all the logical volume (disk images) are mapped. 

So can I access the VM's disk images directly from the RHEV Manager in case 
if manager, hypervisor and storage are different hosts? 
>
> I can tell you that libguestfs *should* be able to open any disk image
> or block device.  But libguestfs doesn't have a magic way to bypass
> permissions: if the block device needs (for example) root permissions
> or SELinux is stopping things, libguestfs won't be able to open the
> file, in the same way that /bin/cat or vi couldn't open it.
>
Oh, that's nice idea. I'll try to turn SELinux off. 
>> Examples from libguestfs and guestfish guide (for instance, over
>> ssh)
>
> Remote access over ssh isn't enabled in RHEL 6 (nor in RHEL 7, sadly).
> Is this a remote disk?  RHEL 7.0 will support NBD remote access only,
> which probably doesn't help you.
Unfortunately, we cannot upgrade from 6.4 due to conditions beyond our
control...
> Fedora of course enables every possible remote access method, but you
> get to keep the pieces if it breaks.
>
>> do not work. And even more, which host should we access - a
>> hypervisor (with a different logical volumes) or a storage (with a
>> monolith volume group)?
>
> Itamar can probably help better than me here.
>
>> Software version: RHEV 3.1, libguestfs 1.16.34, guest OS - RHEL 6.4,
arch: x86_64.
>>
>> P.S. May be I had to post this message directly to the support, but
I'd suggest that this topic would be useful for community.
>>
>> Thank you,
>> Виталий Исаев
>> Инженер-программист
>> Группа разработки и внедрения ПСЗИ
>> Департамент информационной безопасности
>> ОАО <Финтех>
>
> Rich.
>
>i may be missing something, but wouldn't vdsm hooks be a better fit:
>http://www.ovirt.org/VDSM-Hooks
>
>for example, I think the "checkimages" one would cover what you
are
>trying to do already?
Thank you, this hook was useful, but "gemu-img check" command is
slightly different from our purposes,
because we are trying to prevent the booting from disk image with changed system
binaries.
>http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=tree;f=vdsm_hooks;
>
>you can also ask such questions on users@ovirt.org.
Thank you for your attention and advises.
Vitaly

On 11/22/2013 12:02 AM, Исаев Виталий Анатольевич wrote:
...
>> i may be missing something, but wouldn't vdsm hooks be a better
fit:
>> http://www.ovirt.org/VDSM-Hooks
>>
>> for example, I think the "checkimages" one would cover what
you are
>> trying to do already?
>
> Thank you, this hook was useful, but "gemu-img check" command is
slightly different from our purposes,
> because we are trying to prevent the booting from disk image with changed
system binaries.
I'll leave the rest to rich, but on this last one:
you can implement any logic you want in a hook, including the libguestfs 
one. it just removes the need to listen on rhev-m events iiuc.
>
>> http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=tree;f=vdsm_hooks;
>>
>> you can also ask such questions on users@ovirt.org.
>
> Thank you for your attention and advises.
> Vitaly
>

On Thu, Nov 21, 2013 at 06:09:57PM +0200, Itamar Heim
wrote:
> i may be missing something, but wouldn't vdsm hooks be a better fit:
> http://www.ovirt.org/VDSM-Hooks
> 
> for example, I think the "checkimages" one would cover what you
are
> trying to do already?
> http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=tree;f=vdsm_hooks;
It seems like this could work.  checkImage() could be replaced
with a function that called libguestfs code:

http://libguestfs.org/guestfs-python.3.html#example-2:-inspect-a-virtual-machine-disk-image

(Of course I have not tried it ...)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[CC to Fabian - can you comment on the ovirt-node problem below]

On Thu, Nov 21, 2013 at 10:02:46PM +0000, Исаев Виталий Анатольевич
wrote:
> On 11/21/2013 05:56 PM, Richard W.M. Jones wrote:
> > When you say "cannot access them" do you get an error
message?  Could
> > it be an SELinux denial?
> 
> Yes, when I am trying to open some logical volume on the hypervisor side,
for instance:
>     $ guestfish -a /dev/mapper/dm-xx
> libguestfs fails with an error message kind of that:
>     $ libguestfs-supermin-helper: failed to find a suitable kernel.
>     I looked for kernels in /boot and modules in /lib/modules.
>     If this is a Xen guest, and you only have Xen domU kernels
>     installed, try installing a fullvirt kernel (only for
>     libguestfs use, you shouldn't boot the Xen guest with it).
> So this is correct because hypervisor's file system is minimalist and
therefore contains
> kernel files neither in /boot nor in /lib/modules. 
Ah I see, this is a slightly different problem.  If this is RHEV-H /
ovirt-node then that's a bug in ovirt-node since it is supposed to
have a working libguestfs.

Can you paste the full, unedited output of:

  libguestfs-test-tool

on the hypervisor.
> And it is still not clear for me, how should I implement interaction
between the RHEV-M API,
> which tells me what logical volume (disk image) belongs to the every VM,
and the hypervisor's, to which
> all the logical volume (disk images) are mapped. 
> 
> So can I access the VM's disk images directly from the RHEV Manager in
case
> if manager, hypervisor and storage are different hosts? 
I'm not sure about this, but I guess the hooks that Itamar pointed to
before should work.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v