Libguestfs - Oct 2015 - Re: [PATCH] v2v: virtio-win: include *.dll too

On Wed, Oct 28, 2015 at 12:10:19PM +1100, Vadim Rozenfeld
wrote:
> > > > To sum up, the packaging and naming policy of the virtio-win
rpm and the
> > > > virtio-win iso therein are different and neither is clear. 
Hardcoding
> > > > the policy in v2v without actually knowing it appears risky
at best.
> 
> It's due to historical reasons mostly. The best way would be having a
set of separate
> distribution images packaged on per-platform base.
Let me try to get the libguestfs requirements straight:

given a set of Windows drivers, it should be able to identify the ones
appropriate for the particular Windows flavor, in order to

  1) tell which devices can be configured

  2) offline-"install" the storage driver and thus enable the guest
too
     boot

  3) copy over the matching drivers into the guest and allow it to pick
     them up on the first boot


Obviously virtio-win driver packaging and libguestfs must agree on how
to deal with this.

Could you please provide any guidance on how to address this problem?

Thanks,
Roman.

> On 28 Oct 2015, at 13:53, Roman Kagan <rkagan@virtuozzo.com> wrote:
> 
> On Wed, Oct 28, 2015 at 12:10:19PM +1100, Vadim Rozenfeld wrote:
>>>>> To sum up, the packaging and naming policy of the
virtio-win rpm and the
>>>>> virtio-win iso therein are different and neither is clear. 
Hardcoding
>>>>> the policy in v2v without actually knowing it appears risky
at best.
>> 
>> It's due to historical reasons mostly. The best way would be having
a set of separate
>> distribution images packaged on per-platform base.
> 
> Let me try to get the libguestfs requirements straight:
> 
> given a set of Windows drivers, it should be able to identify the ones
> appropriate for the particular Windows flavor, in order to
> 
>  1) tell which devices can be configured
> 
>  2) offline-"install" the storage driver and thus enable the
guest too
>     boot
> 
>  3) copy over the matching drivers into the guest and allow it to pick
>     them up on the first boot
> 
> 
> Obviously virtio-win driver packaging and libguestfs must agree on how
> to deal with this.
> 
> Could you please provide any guidance on how to address this problem?
As Vadim said: "The best way would be having a set of separate distribution
images packaged on per-platform base.”

Otherwise you will have to maintain the knowledge of binary compatibility
between Windows platforms, which is different according to the driver type.

> 
> Thanks,
> Roman.

On Wed, Oct 28, 2015 at 04:15:06PM +0200, Yan Vugenfirer
wrote:
> Otherwise you will have to maintain the knowledge of binary
> compatibility between Windows platforms, which is different
> according to the driver type.
What does this mean?  Could you give an example?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On Wed, Oct 28, 2015 at 04:15:06PM +0200, Yan Vugenfirer
wrote:
> 
> > On 28 Oct 2015, at 13:53, Roman Kagan <rkagan@virtuozzo.com>
wrote:
> > 
> > On Wed, Oct 28, 2015 at 12:10:19PM +1100, Vadim Rozenfeld wrote:
> >>>>> To sum up, the packaging and naming policy of the
virtio-win rpm and the
> >>>>> virtio-win iso therein are different and neither is
clear.  Hardcoding
> >>>>> the policy in v2v without actually knowing it appears
risky at best.
> >> 
> >> It's due to historical reasons mostly. The best way would be
having a set of separate
> >> distribution images packaged on per-platform base.
> > 
> > Let me try to get the libguestfs requirements straight:
> > 
> > given a set of Windows drivers, it should be able to identify the ones
> > appropriate for the particular Windows flavor, in order to
> > 
> >  1) tell which devices can be configured
> > 
> >  2) offline-"install" the storage driver and thus enable the
guest too
> >     boot
> > 
> >  3) copy over the matching drivers into the guest and allow it to pick
> >     them up on the first boot
> > 
> > 
> > Obviously virtio-win driver packaging and libguestfs must agree on how
> > to deal with this.
> > 
> > Could you please provide any guidance on how to address this problem?
> 
> As Vadim said: "The best way would be having a set of separate
> distribution images packaged on per-platform base.”
Great, but apparently this style of packaging is supposed to happen on
virtio-win side of things, and libguestfs would just need to adapt.

Presumably there supposed to be a separate iso or floppy image
containing all the drivers for each Windows flavor, and some stable
naming convention for such images making it easy to map the guest OS to
the image.

Is there any work on that being done or planned?
> Otherwise you will have to maintain the knowledge of binary
> compatibility between Windows platforms, which is different according
> to the driver type.
Indeed, this doesn't look realistic within libguestfs.

Thanks,
Roman.

Hi,

I am the guy who puts together the virtio-win package for Red Hat. I
recently took over this role so I'm still in learning mode.


On Wed, Oct 28, 2015 at 04:15:06PM +0200, Yan Vugenfirer
wrote:
>> On 28 Oct 2015, at 13:53, Roman Kagan <rkagan@virtuozzo.com>
wrote:
>> On Wed, Oct 28, 2015 at 12:10:19PM +1100, Vadim Rozenfeld wrote:
>>>>>> To sum up, the packaging and naming policy of the
virtio-win rpm and the
>>>>>> virtio-win iso therein are different and neither is
clear.  Hardcoding
>>>>>> the policy in v2v without actually knowing it appears
risky at best.
>>>
>>> It's due to historical reasons mostly. The best way would be
having a set of separate
>>> distribution images packaged on per-platform base.
Today, the virtio-win rpm contains:

* Individual driver files, installed onto the host contained in the
  directory hierarchy:

  /usr/share/virtio-win/drivers/<arch>/<os>/<file>

* an .iso file which is intended to be used within a Windows
  guest. The .iso directory hierarchy is:

  /<driver>/<os>/<arch>/<file>

* two vfd files--one for amd64 (x86_64), the other for x86 (i386)--
  that are floppy images, intended to be used as a Windows guest
  install-time alternative to the .iso. The vfd files contain
  essential drivers needed during installation: (block, scsi, net and
  qxl drivers).

>> Let me try to get the libguestfs requirements straight:
>>
>> given a set of Windows drivers, it should be able to identify the ones
>> appropriate for the particular Windows flavor, in order to
>>
>>  1) tell which devices can be configured
>>
>>  2) offline-"install" the storage driver and thus enable the
guest too
>>     boot
>>
>>  3) copy over the matching drivers into the guest and allow it to pick
>>     them up on the first boot
>>
>>
>> Obviously virtio-win driver packaging and libguestfs must agree on how
>> to deal with this.
>>
>> Could you please provide any guidance on how to address this problem?
>
>As Vadim said: "The best way would be having a set of separate
distribution images packaged on per-platform base.”
>
>Otherwise you will have to maintain the knowledge of binary compatibility
between Windows platforms, which is different according to the driver type.
BZ 1223668 suggests reorganizing the directory structure to allow for
automatic driver discovery. This would change the iso directory
structure from:

   /<driver>/<os>/<arch>/<file>

to:

   /<arch>/<os>/<file>

which is the same structure used in /usr/share/virtio-win/drivers and
in the vfd files.

Would this work?

-Jeff
>
>>
>> Thanks,
>> Roman.
>

On Wed, 2015-10-28 at 14:53 +0300, Roman Kagan wrote:
> On Wed, Oct 28, 2015 at 12:10:19PM +1100, Vadim Rozenfeld wrote:
> > > > > To sum up, the packaging and naming policy of the
virtio-win rpm and the
> > > > > virtio-win iso therein are different and neither is
clear.  Hardcoding
> > > > > the policy in v2v without actually knowing it appears
risky at best.
> > 
> > It's due to historical reasons mostly. The best way would be
having a set of separate
> > distribution images packaged on per-platform base.
> 
> Let me try to get the libguestfs requirements straight:
> 
> given a set of Windows drivers, it should be able to identify the ones
> appropriate for the particular Windows flavor, in order to
> 
>   1) tell which devices can be configured
Not sure that I fully understated your question. but if you are going to
create some sort of off-line automatic virtio-win drivers update
utility, then it shouldn't be too complicated. Firs of all you will need
to obtain the Windows kernel version by reading the following Registry
key - "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion". Let's
say it
6.3, which means that it is Win8.1 or WS2012R2, parsing "BuildLabEx"
string from the same hive will give you information about the platform
bitness. Next you need to go through inf files, and find "DriverVer"
string, like this one . taken from from vioscsi.inf

DriverVer=08/01/2015,62.72.104.10800

This string contains build time and version stamps. The version stamp
looks as follow "62.72.104.10800"
where 62 means a target Windows kernel version multiplied by 10. In this
case it is 6.2 which means Win8 or WS2012
72 - the target host platform version multiplied bu 10 (was RHEL7.2)
104 just a magic number, but it can be changed, don't make any
assumption based on this number.
10800 our internal build number (build 108) multiplied by 100
If you found an inf file with the matching minor OS (6 in our case)
version and matching or less but close minor version number (2 vs 3)
then you are in the right directory.
> 
>   2) offline-"install" the storage driver and thus enable the
guest too
>      boot
> 
This is not so easy. You can do the same thing that Rich does for v2v an
it should work fine. But in some cases, when a guest already has at
least one third party storport miniport device driver intalled, the boot
can fail  
>   3) copy over the matching drivers into the guest and allow it to pick
>      them up on the first boot.
It should be pretty the same what v2v does. Some sort of emulation of
what "software first" installation procedure does.

Cheers,
Vadim. 
> 
> 
> Obviously virtio-win driver packaging and libguestfs must agree on how
> to deal with this.
> 
> Could you please provide any guidance on how to address this problem?
> 
> Thanks,
> Roman.

On Thu, Oct 29, 2015 at 11:05:42AM +1100, Vadim Rozenfeld
wrote:
> >   1) tell which devices can be configured
> Not sure that I fully understated your question. but if you are going to
> create some sort of off-line automatic virtio-win drivers update
> utility, then it shouldn't be too complicated.
Roman is talking about what virt-v2v already does.

This is good information:
> First of all you will need
> to obtain the Windows kernel version by reading the following Registry
> key - "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion".
Let's say it
> 6.3, which means that it is Win8.1 or WS2012R2, parsing
"BuildLabEx"
> string from the same hive will give you information about the platform
> bitness. Next you need to go through inf files, and find
"DriverVer"
> string, like this one . taken from from vioscsi.inf
> 
> DriverVer=08/01/2015,62.72.104.10800
> 
> This string contains build time and version stamps. The version stamp
> looks as follow "62.72.104.10800"
> where 62 means a target Windows kernel version multiplied by 10. In this
> case it is 6.2 which means Win8 or WS2012
> 72 - the target host platform version multiplied bu 10 (was RHEL7.2)
> 104 just a magic number, but it can be changed, don't make any
> assumption based on this number.
> 10800 our internal build number (build 108) multiplied by 100
> If you found an inf file with the matching minor OS (6 in our case)
> version and matching or less but close minor version number (2 vs 3)
> then you are in the right directory.
Seems like looking at this field in the .inf file is better than our
current approach of matching path names.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

In terms of turning what Vadim wrote into code ...
> First of all you will need to obtain the Windows kernel version by
> reading the following Registry key -
> "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion". Let's
say it
> 6.3, which means that it is Win8.1 or WS2012R2,
We already access this field in the registry and present it through
inspection, so:

http://libguestfs.org/guestfs.3.html#guestfs_inspect_get_major_version

In virt-v2v this is present in the Types.inspect.i_major_version and
Types.inspect.i_minor_version fields.
> parsing "BuildLabEx"
> string from the same hive will give you information about the platform
> bitness.
Currently we can determine the bitness of a Windows VM by examining
some of the binaries found in %systemroot%.  We present this through
inspection:

http://libguestfs.org/guestfs.3.html#guestfs_inspect_get_arch

(either "i386" or "x86-64") and this is present in virt-v2v
in the
Types.inspect.i_arch field.

I looked at BuildLabEx, which is a field we've not looked at
before.
>From Windows 7:
  "BuildLabEx"="7601.18247.amd64fre.win7sp1_gdr.130828-1532"

Apparently the first four numbers are the build number, next five are
the revision number, and after that is either ".amd64" or
".x86" (for
x86-64 or i386 respectively).
> Next you need to go through inf files, and find "DriverVer"
> string, like this one . taken from from vioscsi.inf
> 
> DriverVer=08/01/2015,62.72.104.10800
>
> This string contains build time and version stamps. The version stamp
> looks as follow "62.72.104.10800"
> where 62 means a target Windows kernel version multiplied by 10. In this
> case it is 6.2 which means Win8 or WS2012
> 72 - the target host platform version multiplied bu 10 (was RHEL7.2)
> 104 just a magic number, but it can be changed, don't make any
> assumption based on this number.
> 10800 our internal build number (build 108) multiplied by 100
> If you found an inf file with the matching minor OS (6 in our case)
> version and matching or less but close minor version number (2 vs 3)
> then you are in the right directory.
This should be easy enough to parse out.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

On Thu, Oct 29, 2015 at 11:05:42AM +1100, Vadim Rozenfeld
wrote:
> On Wed, 2015-10-28 at 14:53 +0300, Roman Kagan wrote:
> >   1) tell which devices can be configured
> Not sure that I fully understated your question. but if you are going to
> create some sort of off-line automatic virtio-win drivers update
> utility, then it shouldn't be too complicated. Firs of all you will
need
> to obtain the Windows kernel version by reading the following Registry
> key - "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion".
Let's say it
> 6.3, which means that it is Win8.1 or WS2012R2, parsing
"BuildLabEx"
> string from the same hive will give you information about the platform
> bitness. Next you need to go through inf files, and find
"DriverVer"
> string, like this one . taken from from vioscsi.inf
> 
> DriverVer=08/01/2015,62.72.104.10800
> 
> This string contains build time and version stamps. The version stamp
> looks as follow "62.72.104.10800"
> where 62 means a target Windows kernel version multiplied by 10. In this
> case it is 6.2 which means Win8 or WS2012
> 72 - the target host platform version multiplied bu 10 (was RHEL7.2)
> 104 just a magic number, but it can be changed, don't make any
> assumption based on this number.
> 10800 our internal build number (build 108) multiplied by 100
Hi Vadim,

I've almost finished implementing this, but I have a couple of
technical questions:

(1) How can we tell from the .inf file what architecture (x86 or
amd64) the driver is for?  There is a [Manufacturer] section which
looks hopeful, eg:

  [Manufacturer]
  %RHEL%   = RHEL,NTAMD64

but the contents are not very consistent across all the .inf files I
have access to.

(2) If we have a directory containing multiple drivers, can we assume
that if any .inf file found matches the guest OS, then all those
drivers can be installed?  So far this appears to be true.  If not
then we have to start associating drivers with .inf files, which seems
a bit complicated.

(3) You write:
> If you found an inf file with the matching minor OS (6 in our case)
> version and matching or less but close minor version number (2 vs 3)
> then you are in the right directory.
When you say "close minor version number" is there a particular
definition you have in mind?  eg. minor or minor-1 is OK?  Or do we
have to consider all .inf files in totality and choose the nearest
one?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html