Libguestfs - Nov 2015 - packaging virtio-win (was: Re: [PATCH] v2v: virtio-win: include *.dll too)

On Wed, Nov 18, 2015 at 11:31:17PM -0500, Li Jin wrote:
> > > > In any case the *.inf files don't seem to have any
distinguishing
> > > > feature which would allow us to check this.
> > > > 
> > > > Maybe this doesn't matter?
> > > 
> > > Let me explain how it works:
> > > We don't make any difference between server or client
platforms when
> > > building our drivers. Initially, all the files, mentioned above
(inf,
> > > cat, sys, and pdb), are absolutely the same. Then the files go to
QE for
> > > WHQL testing. QE tests the same drivers on both, desktop and
server
> > > platforms, and creates two different submissions files for
sending them
> > > to WHQL dashboard togather with the original "cat"
file.
> > > As the result, we have two different cat files created from the
same
> > > original one.
> > > 
> > > As far as I remember, there shouldn't be any problem to swap
these two
> > > cat files, and use them interchangeable. But it will be better
get some
> > > conformation from QE team.
> > 
> > Li Jin, this is a discussion we have on a public mailing-list
regarding the
> > files structure
> > of our virtio-win distribution.
> > Could you confirm Vadim's statement above?
> 
> Yes, I agree with Vadim's statement. The *.cat file is a signature
file:
> Before whql testing,the driver is only signed by redhat,when install driver
in guest,windows OS will prompt a security warning to confirm if customer want
to install a driver not signed by msft.
> After whql testing and submit to msft,we will get a *.cat file signed by
microsoft,which will make driver installation smoothly(no extra warning).
> 
> The original *.cat file is same,whether the *.cat files are still same afer
whql submission depends on whether they are submit in one submission.
> for example:
> win8-64 and win2012 *.cat files are same because QE submit them in one
submission(win8-64 and win2012 whql test both run on hck,results can be merged
into one package);
> win7-64 and win2008-64 *.cat files are different because we submit them in
different submissions(win7-64 whql test run on hck,win2008-64 run wlk,hck and
wlk must submit differently according to msft's strategy).
> 
> I tried swap cat files,drivers still can be installed correctly.
I went ahead and dug into this a little, and also talked to a colleague
who is involved in WHQL certification of drivers.  Here's what I've got
(sorry if it's obvious for anyone; I failed to find any description of
it with just enough details).

Catalog files are pkcs7 containers with multi-layered nested ASN.1
structure; on Linux they are best viewed with with dumpasn1 (which,
unlike openssl asn1parse, recognizes most of the Microsoft objects).

They contain, in addition to signatures, a number of name:value pairs
with properties of the whole package, and a list of entries describing
every file in the package, with their own set of name:value pairs
representing the properties of those entries.  In particular, for every
file a filename and an embedded (for PE images: exe, sys, dll) or
detached (for other files, e.g. inf) digest is stored, to match the file
to the record in the catalog.

For every file in the catalog theres an attribute named OSAttr; its
value is a string containing comma-separted list of Windows versions,
e.g. "2:5.1,2:5.2,2:6.0,2:6.1".

Besides, there's a global attribute "OS", containing
comma-separated
list of Windows versions in a different format, e.g.

"XPX64,Server2003X64,VistaX64,Server2008X64,7X64,Server2008R2X64"

These are presumably the ones that have to be matched against the
version of the Windows the driver is installed into.

Both per-file OSAttr and global OS are populated when the catalog is
generated by inf2cat tool from Microsoft WDK
(https://msdn.microsoft.com/en-us/library/windows/hardware/ff553618(v=vs.85).aspx),
which takes the list of values in its /os command-line parameter.

When the package is submitted for WHQL certification, MS regenerates the
the catalog with its own signature and leaving only those OSes in OSAttr
and OS for which certification is granted, and sends it back to the
submitter.

So it looks like yes, there's certain data in there which can be used to
match it against the exact Windows version; not sure what to do with all
this, though.

Roman.

On Thu, Nov 19, 2015 at 09:45:36PM +0300, Roman Kagan
wrote:
> On Wed, Nov 18, 2015 at 11:31:17PM -0500, Li Jin wrote:
> > > > > In any case the *.inf files don't seem to have any
distinguishing
> > > > > feature which would allow us to check this.
>
> Catalog files are pkcs7 containers with multi-layered nested ASN.1
> structure
[...]
> So it looks like yes, there's certain data in there which can be used
to
> match it against the exact Windows version; not sure what to do with all
> this, though.
Having given it some more thought and having discussed that with
colleagues who do various things with the drivers, like generating
unattended install helpers or just manually installing drivers into
guests, I'm now of the opinion that neither libguestfs nor any other
user of the drivers should look inside the driver files in order to
match them against the guest OS, because it's hard for programs and
unrealistic for humans.

I tend to believe it should be the responsibility of the driver packages
to lay out the drivers in a directory structure that both humans and
programs could grok easily and that wouldn't change once established; in
a sense this would be both an API and a human interface.

If I were to lay it out I'd group the stuff by guest OS, so that for a
particular guest a single entity -- a directory tree or a pre-created
ISO or floppy image -- would contain all that's needed for this guest
and nothing else.  This would allow separation of concerns, so that
actors who bring the stuff into the guest wouldn't need to care about
what's inside, and actors involved with its installation wouldn't need
to know how to find the right one.

E.g. from this POV the way drivers are stored in RHEL7 virtio-win
package on the *filesystem* looks OK, the way they are stored in the
*ISO image* in the same package doesn't.

As for naming one can stick with what's being used in that virtio-win
rpm, i.e.

{amd64,i386}/Win{XP,2003,2008,2008R2,2012,2012R2,7,8,8.1}

or switch to the names passed to inf2cat in its /os parameter, i.e.

XP_X86,Server2003_X86,...,8_X64,Server8_X64,6_3_X86,6_3_X64,Server6_3_X64
(https://msdn.microsoft.com/en-us/library/windows/hardware/ff547089(v=vs.85).aspx)

Whatever convention is chosen it needs to be followed thereafter.

The main problem with this proposal is that I don't know how to
implement it: there are multiple vendors of virtio-win packages, and,
for licensing reasons, they don't even share the source code, so it's
unclear who could own this convention and enforce it on others.

I'd appreciate any thoughts or suggestions on how to proceed from that
and whether it's worth it at all.

Thanks,
Roman.

On Mon, Nov 23, 2015 at 08:26:10PM +0300, Roman Kagan
wrote:
> On Thu, Nov 19, 2015 at 09:45:36PM +0300, Roman Kagan wrote:
> > On Wed, Nov 18, 2015 at 11:31:17PM -0500, Li Jin wrote:
> > > > > > In any case the *.inf files don't seem to have
any distinguishing
> > > > > > feature which would allow us to check this.
> >
> > Catalog files are pkcs7 containers with multi-layered nested ASN.1
> > structure
> [...]
> > So it looks like yes, there's certain data in there which can be
used to
> > match it against the exact Windows version; not sure what to do with
all
> > this, though.
> 
> Having given it some more thought and having discussed that with
> colleagues who do various things with the drivers, like generating
> unattended install helpers or just manually installing drivers into
> guests, I'm now of the opinion that neither libguestfs nor any other
> user of the drivers should look inside the driver files in order to
> match them against the guest OS, because it's hard for programs and
> unrealistic for humans.
> 
> I tend to believe it should be the responsibility of the driver packages
> to lay out the drivers in a directory structure that both humans and
> programs could grok easily and that wouldn't change once established;
in
> a sense this would be both an API and a human interface.
> 
> If I were to lay it out I'd group the stuff by guest OS, so that for a
> particular guest a single entity -- a directory tree or a pre-created
> ISO or floppy image -- would contain all that's needed for this guest
> and nothing else.  This would allow separation of concerns, so that
> actors who bring the stuff into the guest wouldn't need to care about
> what's inside, and actors involved with its installation wouldn't
need
> to know how to find the right one.
The good news is that this is what we do right now.
> E.g. from this POV the way drivers are stored in RHEL7 virtio-win
> package on the *filesystem* looks OK, the way they are stored in the
> *ISO image* in the same package doesn't.
> 
> As for naming one can stick with what's being used in that virtio-win
> rpm, i.e.
> 
> {amd64,i386}/Win{XP,2003,2008,2008R2,2012,2012R2,7,8,8.1}
>
> or switch to the names passed to inf2cat in its /os parameter, i.e.
> 
> XP_X86,Server2003_X86,...,8_X64,Server8_X64,6_3_X86,6_3_X64,Server6_3_X64
>
(https://msdn.microsoft.com/en-us/library/windows/hardware/ff547089(v=vs.85).aspx)
> 
> Whatever convention is chosen it needs to be followed thereafter.
Yup - very important that once we decide what the naming convention is
going to be, we stick with it.  Moving drivers around or using ad-hoc
naming schemes causes churn in virt-v2v and probably other projects
too.
> The main problem with this proposal is that I don't know how to
> implement it: there are multiple vendors of virtio-win packages, and,
> for licensing reasons, they don't even share the source code, so
it's
> unclear who could own this convention and enforce it on others.
I guess that virt-v2v will have to deal with this with more special
rules.
> I'd appreciate any thoughts or suggestions on how to proceed from that
> and whether it's worth it at all.
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW