thr3ads.net - Libguestfs - [Libguestfs] Crafted OVAs and handling non-existing (empty) disks in OVF [Feb 2021]

If this information is useful, please help other people find it:
Share via:

Tomáš Golembiovský

2021-Feb-22 13:51 UTC

[Libguestfs] Crafted OVAs and handling non-existing (empty) disks in OVF

Hi,

from time to time we get a request [1] to import appliance with crafted
OVF (from Cisco or Gigamon) into oVirt. The common denominator is that
some disks are missing and are supposed to be created during import of
the appliance.

Doing the import properly would require not only solving of the problem
with missing disks, but also implementing multiple concepts -- notably
concept of configurations [2] (for Cisco appliance) or concept of
properties [3] (for Gigamon appliance). This would be quite complex
change to oVirt as well as to virt-v2v and at the moment we don't feel
that such effort is justifiable. But by solving the problem with disks
we can at least provide a helping hand to those requiring the crafted
appliances. 

The idea here is that virt-v2v would ignore the non-existing disks and
user would be required to add those manually after conversion. As for
OVFs using the configurations virt-v2v would pick some settings from OVF
(random from users perspective) and user would be responsible for
editing the VM's configuration after conversion (CPUs, memory, etc.) to
size the VM properly based on the expected use. We could further
constrain this to only work with -o vdsm, but this may not be needed
unless we hit some issues with implementing the feature. It is also
possible that ignoring the disks may not work for some reasons that
we have not yet discovered and we'll se once we try.

There is one more problem with the Cisco OVA that it contains .cert file
which is not specified in manifest. But the file is not needed during
conversion. So this could be possibly handled by enforcing virt-v2v to
use only files listed in manifest instead of complaining.

Before I invest any time into this I wanted to make sure that such
approach would be acceptable to the upstream. So would this be good
enough?


***

The topics for discussions are above. What follows are the technical
details for those interested in gain deeper understanding of the
problem. You may be wondering why would we want to ignore the empty
disks if we can create them for most of the output backends. The problem
is, that we cannot. Either we don't know which disks are of the interest
because not all should be used (configurations) or we have no idea how
big the disk should be (properties).

###  Configurations

The OVF can define several configurations in DeploymentOptionSection.
A short (simplified) example may look like this:

    <DeploymentOptionSection>
        <Configuration ovf:default="true"
ovf:id="Standard" />
        <Configuration ovf:id="Express" />
        ...
    </DeploymentOptionSection>

Then in the VirtualHardwareSection there can be duplicate settings
distinguished only by the ovf:configuration attribute. For example 2 different
vCPU configurations:

    <Item ovf:configuration="Express">
        ...
        <rasd:Description>Number of Virtual CPUs</rasd:Description>
        <rasd:ResourceType>3</rasd:ResourceType>
        <rasd:VirtualQuantity>4</rasd:VirtualQuantity>
    </Item>
    <Item ovf:configuration="Standard">
        ...
        <rasd:Description>Number of Virtual CPUs</rasd:Description>
        <rasd:ResourceType>3</rasd:ResourceType>
        <rasd:VirtualQuantity>16</rasd:VirtualQuantity>
    </Item>

In terms of disks this means that only some of the disks get used. Specifically
the Cisco Appliance has 4 disks listed in the DiskSection -- 1 system disk A
and 3 empty disks B,C,D. But the created VM never has all four. It has either
only A or it has A+B or A+C or A+D. Without understanding configurations we
cannot tell whether to use B, C, D or none.


###  Properties

The OVF can define various properties in ProductSection element. Like this:


    <ProductSection>
        ...
        <Property ovf:key="datadisksize"
ovf:qualifiers="MinValue(20),MaxValue(1000)" ovf:type="int"
ovf:userConfigurable="true">
            <Label>02. Size of Data Disk</Label>
            <Description>The size of the Data Disk in
gigabytes.</Description>
            <Value ovf:value="40"/>
        </Property>
        ...
    </ProductSection

Then, the ovf:key value of the property can be used as a variable on
other places in the OVF. For example like this:

    <DiskSection>
        ...
        <Disk ovf:capacity="${datadisksize}"
ovf:fileRef="" ... />
        ...
    </DiskSection>

And as before, without understanding the concept virt-v2v has no idea
how to size the disk. The Value element is optional (if the property is
userConfigurable) so relying on the default in OVF may not work. I can
imagine some OVFs may even use a property to specify vCPU count or
memory which could bring up a question how to handle those. Possibly
default to 0 or 1 (where 1 may be better default in my opinion).

    Tomas


[1] https://bugzilla.redhat.com/1705600
[2] Open Virtualization Format Specification (DSP0243) Version 2.1.1,
    Chapter 9.8 -- DeploymentOptionSection
[3] Open Virtualization Format Specification (DSP0243) Version 2.1.1,
    Chapter 9.5.1 -- Property elements

-- 
Tom?? Golembiovsk? <tgolembi at redhat.com>

Richard W.M. Jones

2021-Feb-22 16:45 UTC

head link

[Libguestfs] Crafted OVAs and handling non-existing (empty) disks in OVF

On Mon, Feb 22, 2021 at 02:51:32PM +0100, Tom?? Golembiovsk?
wrote:> Hi,
> 
> from time to time we get a request [1] to import appliance with crafted
> OVF (from Cisco or Gigamon) into oVirt. The common denominator is that
> some disks are missing and are supposed to be created during import of
> the appliance.
> 
> Doing the import properly would require not only solving of the problem
> with missing disks, but also implementing multiple concepts -- notably
> concept of configurations [2] (for Cisco appliance) or concept of
> properties [3] (for Gigamon appliance). This would be quite complex
> change to oVirt as well as to virt-v2v and at the moment we don't feel
> that such effort is justifiable. But by solving the problem with disks
> we can at least provide a helping hand to those requiring the crafted
> appliances. 
Some immediate questions: Can virt-v2v do anything useful for these
appliances?  Does libguestfs inspection find anything inside the
appliances?  Is installing virtio drivers useful?  Do they not have
virtio drivers already?  Where are they supposed to run originally?
> The idea here is that virt-v2v would ignore the non-existing disks and
> user would be required to add those manually after conversion. As for
> OVFs using the configurations virt-v2v would pick some settings from OVF
> (random from users perspective) and user would be responsible for
> editing the VM's configuration after conversion (CPUs, memory, etc.) to
> size the VM properly based on the expected use. We could further
> constrain this to only work with -o vdsm, but this may not be needed
> unless we hit some issues with implementing the feature. It is also
> possible that ignoring the disks may not work for some reasons that
> we have not yet discovered and we'll se once we try.
> 
> There is one more problem with the Cisco OVA that it contains .cert file
> which is not specified in manifest. But the file is not needed during
> conversion. So this could be possibly handled by enforcing virt-v2v to
> use only files listed in manifest instead of complaining.
> 
> Before I invest any time into this I wanted to make sure that such
> approach would be acceptable to the upstream. So would this be good
> enough?
Does the proposed virt-v2v split help here?

https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html

With this kind of split, you could set up the disks however you liked
-- for example creating dummy input disks (nbdkit-null-plugin
instances probably) -- for the missing disks.  Then run just the
virt-v2v conversion component to carry out the conversion.

Rich.
> ***
> 
> The topics for discussions are above. What follows are the technical
> details for those interested in gain deeper understanding of the
> problem. You may be wondering why would we want to ignore the empty
> disks if we can create them for most of the output backends. The problem
> is, that we cannot. Either we don't know which disks are of the
interest
> because not all should be used (configurations) or we have no idea how
> big the disk should be (properties).
> 
> ###  Configurations
> 
> The OVF can define several configurations in DeploymentOptionSection.
> A short (simplified) example may look like this:
> 
>     <DeploymentOptionSection>
>         <Configuration ovf:default="true"
ovf:id="Standard" />
>         <Configuration ovf:id="Express" />
>         ...
>     </DeploymentOptionSection>
> 
> Then in the VirtualHardwareSection there can be duplicate settings
> distinguished only by the ovf:configuration attribute. For example 2
different
> vCPU configurations:
> 
>     <Item ovf:configuration="Express">
>         ...
>         <rasd:Description>Number of Virtual
CPUs</rasd:Description>
>         <rasd:ResourceType>3</rasd:ResourceType>
>         <rasd:VirtualQuantity>4</rasd:VirtualQuantity>
>     </Item>
>     <Item ovf:configuration="Standard">
>         ...
>         <rasd:Description>Number of Virtual
CPUs</rasd:Description>
>         <rasd:ResourceType>3</rasd:ResourceType>
>         <rasd:VirtualQuantity>16</rasd:VirtualQuantity>
>     </Item>
> 
> In terms of disks this means that only some of the disks get used.
Specifically
> the Cisco Appliance has 4 disks listed in the DiskSection -- 1 system disk
A
> and 3 empty disks B,C,D. But the created VM never has all four. It has
either
> only A or it has A+B or A+C or A+D. Without understanding configurations we
> cannot tell whether to use B, C, D or none.
> 
> 
> ###  Properties
> 
> The OVF can define various properties in ProductSection element. Like this:
> 
> 
>     <ProductSection>
>         ...
>         <Property ovf:key="datadisksize"
ovf:qualifiers="MinValue(20),MaxValue(1000)" ovf:type="int"
ovf:userConfigurable="true">
>             <Label>02. Size of Data Disk</Label>
>             <Description>The size of the Data Disk in
gigabytes.</Description>
>             <Value ovf:value="40"/>
>         </Property>
>         ...
>     </ProductSection
> 
> Then, the ovf:key value of the property can be used as a variable on
> other places in the OVF. For example like this:
> 
>     <DiskSection>
>         ...
>         <Disk ovf:capacity="${datadisksize}"
ovf:fileRef="" ... />
>         ...
>     </DiskSection>
> 
> And as before, without understanding the concept virt-v2v has no idea
> how to size the disk. The Value element is optional (if the property is
> userConfigurable) so relying on the default in OVF may not work. I can
> imagine some OVFs may even use a property to specify vCPU count or
> memory which could bring up a question how to handle those. Possibly
> default to 0 or 1 (where 1 may be better default in my opinion).
> 
>     Tomas
> 
> 
> [1] https://bugzilla.redhat.com/1705600
> [2] Open Virtualization Format Specification (DSP0243) Version 2.1.1,
>     Chapter 9.8 -- DeploymentOptionSection
> [3] Open Virtualization Format Specification (DSP0243) Version 2.1.1,
>     Chapter 9.5.1 -- Property elements
> 
> -- 
> Tom?? Golembiovsk? <tgolembi at redhat.com>
> 
> _______________________________________________
> Libguestfs mailing list
> Libguestfs at redhat.com
> https://listman.redhat.com/mailman/listinfo/libguestfs
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Libguestfs - Feb 2021 - Crafted OVAs and handling non-existing (empty) disks in OVF

[Libguestfs] Crafted OVAs and handling non-existing (empty) disks in OVF

[Libguestfs] Crafted OVAs and handling non-existing (empty) disks in OVF