Libguestfs - Nov 2009 - Attempts to install a Windows driver from WinPE

One of the things we would really like to be able to do for V2V is to 
install a new driver in a Windows guest. There are a couple of reasons 
for this:

* The guest may not be bootable without the driver installed, for 
example because the underlying virtual hardware has changed from vmscsi 
to virtio.

* If the guest can boot, the alternative is to modify the guest to run a 
script on next boot. This requires making assumptions about supporting 
software being installed and working correctly on the guest. Certain 
environments, particularly heavily locked-down environments, make this 
an unsafe assumption.

The Windows PE environment looks perfect for this task. It gives you a 
very lightweight Windows OS which can be customised with additional 
tools. It is specifically for doing installations. I spent Friday trying 
to use it to install a driver in a guest. Here's what I tried and why it 
didn't work.

Installing a driver in Windows is 'driven' by a .inf file. From my 
(admittedly limited) understanding, this broadly describes:

* The files which need to be installed
* The hardware the driver is compatible with

The files, including the .inf file itself must be copied in to the 
correct places. In addition, information from the .inf file must be 
written to the registry. It is this last part which causes problems.

 From reading documentation, it appears that a driver would normally be 
installed using the SetupCopyOEMInf() library call. I wrote a simple 
wrapper round this and installed it in the Windows PE image, along with 
the VirtIO drivers. I booted into Windows PE and attempted to install 
the driver. As you might expect, the drivers were installed into the 
Windows PE image rather than the guest. I then tried setting 
%systemdrive% and %systemroot% to the guest image. This appears to have 
no effect. This is what makes me suspect that the process is primarily 
registry driven.

I started looking around for ways of using a different registry. I 
discovered the Registry Editor PE plugin to BartPE 
(https://sourceforge.net/projects/regeditpe/) which allows editing the 
registry of a guest. Looking at how it does this, it uses reg.exe to 
load the guests's hives. I confirmed that you can do this. Unfortunately 
you don't seem to be able to replace the default hives. The new hives 
are loaded in a different part of the tree, and are therefore ignored.

This is as far as I've got. Still on my list are:

* Ask on various Windows mailing lists how to do this
* Investigate if the packaged .msi containing the drivers is more flexible.
* Look for other, possibly lower level, ways of replacing making a 
process use a different registry.

Any and all suggestions are gratefully received,

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

----- "Matthew Booth" <mbooth at redhat.com> wrote:
> One of the things we would really like to be able to do for V2V is to
> 
> install a new driver in a Windows guest. There are a couple of reasons
> 
> for this:
> 
> * The guest may not be bootable without the driver installed, for 
> example because the underlying virtual hardware has changed from
> vmscsi 
> to virtio.
> 
> * If the guest can boot, the alternative is to modify the guest to run
> a 
> script on next boot. This requires making assumptions about supporting
> 
> software being installed and working correctly on the guest. Certain 
> environments, particularly heavily locked-down environments, make this
> 
> an unsafe assumption.
> 
> The Windows PE environment looks perfect for this task. It gives you a
> 
> very lightweight Windows OS which can be customised with additional 
> tools. It is specifically for doing installations. I spent Friday
> trying 
> to use it to install a driver in a guest. Here's what I tried and why
> it 
> didn't work.
> 
> Installing a driver in Windows is 'driven' by a .inf file. From my 
> (admittedly limited) understanding, this broadly describes:
> 
> * The files which need to be installed
> * The hardware the driver is compatible with
> 
> The files, including the .inf file itself must be copied in to the 
> correct places. In addition, information from the .inf file must be 
> written to the registry. It is this last part which causes problems.
> 
>  From reading documentation, it appears that a driver would normally
> be 
> installed using the SetupCopyOEMInf() library call. I wrote a simple 
> wrapper round this and installed it in the Windows PE image, along
> with 
> the VirtIO drivers. I booted into Windows PE and attempted to install
> 
> the driver. As you might expect, the drivers were installed into the 
> Windows PE image rather than the guest. I then tried setting 
> %systemdrive% and %systemroot% to the guest image. This appears to
> have 
> no effect. This is what makes me suspect that the process is primarily
> 
> registry driven.
> 
> I started looking around for ways of using a different registry. I 
> discovered the Registry Editor PE plugin to BartPE 
> (https://sourceforge.net/projects/regeditpe/) which allows editing the
> 
> registry of a guest. Looking at how it does this, it uses reg.exe to 
> load the guests's hives. I confirmed that you can do this.
> Unfortunately 
> you don't seem to be able to replace the default hives. The new hives
> 
> are loaded in a different part of the tree, and are therefore
> ignored.
This is NOT doable. The registry holds too many configurations imperative for
the proper behaviour of the OS (in this case the WinPE hive).  If this is at all
possible (I doubt it, too many open handles and things that require booting to
change as it is) Changing this would render the system unstable (if not BSOD you
entirely if you could get so far).

Now that I understand what you are trying to do (and I don't mean to
discourage) but I believe you are going about this the wrong way.  A direct
install of the driver probably does even more things which you are not aware of
and are hardwired to the system drive (not through %SYSTEMDIR% and any other
environment variable).
Just an example, Changing system environment variables require booting the
machine (http://technet.microsoft.com/en-us/library/bb726962.aspx) this is the
relevant excerpt:
Note: When you create or modify system environment variables, the changes take
effect when you restart the computer. When you create or modify user environment
variables, the changes take effect the next time the user logs on to the system.
In addition, if the device driver is installed under the "system"
user's credentials then: you have to boot the machine after you change the
environment variables (because they are loaded only once at boot and cannot be
refreshed) and changing the systemdir for the entire machine and rebooting is
probably not recommended.

In any event, in order to reverse engineer what is going on when the driver is
installed you can probably use some "recording" applications (apps
which install filter device drivers which monitor the registry and the hard
drive, record changes and filter out known things which are irrelevant).
The next phase would be to inject the relevant files directly where they should
be and make the relevant registry changes (for instance by loading the Hives as
you did and running a script that changes the relevant entries).

The other possibility is as I mentioned before offline tools which already know
how to do this (e.g.
http://technet.microsoft.com/en-us/library/cc749465%28WS.10%29.aspx).
> 
> This is as far as I've got. Still on my list are:
> 
> * Ask on various Windows mailing lists how to do this
> * Investigate if the packaged .msi containing the drivers is more
> flexible.
> * Look for other, possibly lower level, ways of replacing making a 
> process use a different registry.
> 
> Any and all suggestions are gratefully received,
> 
> Matt
> -- 
> Matthew Booth, RHCA, RHCSS
> Red Hat Engineering, Virtualisation Team
> 
> M:       +44 (0)7977 267231
> GPG ID:  D33C3490
> GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

On 30/11/09 12:27, Ayal Baron wrote:
> Now that I understand what you are trying to do (and I don't mean to
discourage) but I believe you are going about this the wrong way.  A direct
install of the driver probably does even more things which you are not aware of
and are hardwired to the system drive (not through %SYSTEMDIR% and any other
environment variable).
> Just an example, Changing system environment variables require booting the
machine (http://technet.microsoft.com/en-us/library/bb726962.aspx) this is the
relevant excerpt:
> Note: When you create or modify system environment variables, the changes
take effect when you restart the computer. When you create or modify user
environment variables, the changes take effect the next time the user logs on to
the system.
> In addition, if the device driver is installed under the "system"
user's credentials then: you have to boot the machine after you change the
environment variables (because they are loaded only once at boot and cannot be
refreshed) and changing the systemdir for the entire machine and rebooting is
probably not recommended.
>
> In any event, in order to reverse engineer what is going on when the driver
is installed you can probably use some "recording" applications (apps
which install filter device drivers which monitor the registry and the hard
drive, record changes and filter out known things which are irrelevant).
> The next phase would be to inject the relevant files directly where they
should be and make the relevant registry changes (for instance by loading the
Hives as you did and running a script that changes the relevant entries).
>
> The other possibility is as I mentioned before offline tools which already
know how to do this (e.g.
http://technet.microsoft.com/en-us/library/cc749465%28WS.10%29.aspx).
Changing environment variables/the registry are not directly what I'm 
trying to achieve. In fact, I'd prefer not to if it could be avoided. 
I'm just trying to install a driver in an offline guest.

I hadn't found pkgmgr.exe, however, I have found a post which suggests 
that peimg.exe can install drivers in an offline guest. I'm hoping to 
prove this this afternoon, although I have some other issues with VirtIO 
under windows in general to sort out first.

Reverse engineering would be an absolute last resort for me, so if 
either pkgmgr or peimg can do this, I think this is perfect. The host, 
which is probably not running Windows, boots a Windows PE appliance 
which contains pkgmgr and/or peimg, and the target drivers. It uses 
these to install them in the guest.

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

I have some knowledge with windows and might be able to help

SetupCopyOEMInf is the right API
And it should be possible to track it's whereabouts using filemon and
regmon.

I would have also investigate more into pkgmgr.exe which sounds as a
possibly possible solution (from its description)

Another possible source of info is through the administration guide. I
know it's possible to sysprep a machine with additional drivers for later
discovery by the PnP mechanism and the same mechanism used there might be
useful here as well


BTW - storage pose additional complexity if you want to boot directly from
the new driver (like in the case of moving to virtio), there are
additional required changes (other than just installing the driver, you
have to tell it to load it before other components (something to do with
the CriticalDeviceDatabse
http://technet.microsoft.com/en-us/library/cc976055.aspx)
> -----Original Message-----
> From: Itamar Heim [mailto:iheim at redhat.com]
> Sent: Monday, November 30, 2009 2:33 PM
> To: 'Yuval Kashtan'; 'Yaniv Kaul'
> Subject: FW: Attempts to install a Windows driver from WinPE
> 
> fyi
> 
> -----Original Message-----
> From: Matthew Booth [mailto:mbooth at redhat.com]
> Sent: Monday, November 30, 2009 5:52 AM
> To: libguestfs at redhat.com
> Cc: 'Ayal Baron'; Itamar Heim
> Subject: Attempts to install a Windows driver from WinPE
> 
> One of the things we would really like to be able to do for V2V is to
> install a new driver in a Windows guest. There are a couple of reasons
> for this:
> 
> * The guest may not be bootable without the driver installed, for
> example because the underlying virtual hardware has changed from vmscsi
> to virtio.
> 
> * If the guest can boot, the alternative is to modify the guest to run
> a
> script on next boot. This requires making assumptions about supporting
> software being installed and working correctly on the guest. Certain
> environments, particularly heavily locked-down environments, make this
> an unsafe assumption.
> 
> The Windows PE environment looks perfect for this task. It gives you a
> very lightweight Windows OS which can be customised with additional
> tools. It is specifically for doing installations. I spent Friday
> trying
> to use it to install a driver in a guest. Here's what I tried and why
> it
> didn't work.
> 
> Installing a driver in Windows is 'driven' by a .inf file. From my
> (admittedly limited) understanding, this broadly describes:
> 
> * The files which need to be installed
> * The hardware the driver is compatible with
> 
> The files, including the .inf file itself must be copied in to the
> correct places. In addition, information from the .inf file must be
> written to the registry. It is this last part which causes problems.
> 
>  From reading documentation, it appears that a driver would normally be
> installed using the SetupCopyOEMInf() library call. I wrote a simple
> wrapper round this and installed it in the Windows PE image, along with
> the VirtIO drivers. I booted into Windows PE and attempted to install
> the driver. As you might expect, the drivers were installed into the
> Windows PE image rather than the guest. I then tried setting
> %systemdrive% and %systemroot% to the guest image. This appears to have
> no effect. This is what makes me suspect that the process is primarily
> registry driven.
> 
> I started looking around for ways of using a different registry. I
> discovered the Registry Editor PE plugin to BartPE
> (https://sourceforge.net/projects/regeditpe/) which allows editing the
> registry of a guest. Looking at how it does this, it uses reg.exe to
> load the guests's hives. I confirmed that you can do this.
> Unfortunately
> you don't seem to be able to replace the default hives. The new hives
> are loaded in a different part of the tree, and are therefore ignored.
> 
> This is as far as I've got. Still on my list are:
> 
> * Ask on various Windows mailing lists how to do this
> * Investigate if the packaged .msi containing the drivers is more
> flexible.
> * Look for other, possibly lower level, ways of replacing making a
> process use a different registry.
> 
> Any and all suggestions are gratefully received,
> 
> Matt
> --
> Matthew Booth, RHCA, RHCSS
> Red Hat Engineering, Virtualisation Team
> 
> M:       +44 (0)7977 267231
> GPG ID:  D33C3490
> GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

On 30/11/09 10:52, Matthew Booth wrote:
> One of the things we would really like to be able to do for V2V is to
> install a new driver in a Windows guest. There are a couple of reasons
> for this:
>
> * The guest may not be bootable without the driver installed, for
> example because the underlying virtual hardware has changed from vmscsi
> to virtio.
>
> * If the guest can boot, the alternative is to modify the guest to run a
> script on next boot. This requires making assumptions about supporting
> software being installed and working correctly on the guest. Certain
> environments, particularly heavily locked-down environments, make this
> an unsafe assumption.
>
> The Windows PE environment looks perfect for this task. It gives you a
> very lightweight Windows OS which can be customised with additional
> tools. It is specifically for doing installations. I spent Friday trying
> to use it to install a driver in a guest. Here's what I tried and why
it
> didn't work.
>
> Installing a driver in Windows is 'driven' by a .inf file. From my
> (admittedly limited) understanding, this broadly describes:
>
> * The files which need to be installed
> * The hardware the driver is compatible with
>
> The files, including the .inf file itself must be copied in to the
> correct places. In addition, information from the .inf file must be
> written to the registry. It is this last part which causes problems.
>
>  From reading documentation, it appears that a driver would normally be
> installed using the SetupCopyOEMInf() library call. I wrote a simple
> wrapper round this and installed it in the Windows PE image, along with
> the VirtIO drivers. I booted into Windows PE and attempted to install
> the driver. As you might expect, the drivers were installed into the
> Windows PE image rather than the guest. I then tried setting
> %systemdrive% and %systemroot% to the guest image. This appears to have
> no effect. This is what makes me suspect that the process is primarily
> registry driven.
>
> I started looking around for ways of using a different registry. I
> discovered the Registry Editor PE plugin to BartPE
> (https://sourceforge.net/projects/regeditpe/) which allows editing the
> registry of a guest. Looking at how it does this, it uses reg.exe to
> load the guests's hives. I confirmed that you can do this.
Unfortunately
> you don't seem to be able to replace the default hives. The new hives
> are loaded in a different part of the tree, and are therefore ignored.
>
> This is as far as I've got. Still on my list are:
>
> * Ask on various Windows mailing lists how to do this
> * Investigate if the packaged .msi containing the drivers is more flexible.
> * Look for other, possibly lower level, ways of replacing making a
> process use a different registry.
>
> Any and all suggestions are gratefully received,
An update on this as promised: I have now made it work with Windows 2008 
R2! The short version:

1. Create a Windows PE image containing VirtIO drivers

* Install WAIK for Windows 7 on your build system
* Create a stock amd64 Windows PE image
* Use DISM to mount the image
* Use DISM to install the amd64 viostor driver for Win2008 in the image
* Use DISM to unmount the image
* Copy the image to sources\boot.wim in the ISO tree
* Copy the complete contents of the virtio driver floppy into ISO tree
* Create a bootable ISO image from the ISO tree

2. Use Windows PE image to install driver in guest

* Ensure guest is configured to use VirtIO for storage
* Add Windows PE image as boot image for guest
* Boot guest with Windows PE image
     Guest system drive is D:
     Mounted ISO image is E:
* Run: dism /image:d:\ /add-driver 
/driver:e:\virtio\amd64\win2008\viostor.inf
* exit
* Configure guest to boot from HD

The guest will now boot normally using VirtIO.

The biggest gotcha in the above is that if you google 'WAIK' it will 
send you to a download page for Windows Automated Installation Kit. 
Unless you're extremely eagle-eyed (I wasn't) you won't notice that
this
was released in 2007, and it doesn't support Windows 2008 R2. However, 
if you search for WAIK in the search box (not the Bing one, the useful 
one below that), you'll find it.

DISM is new in the latest WAIK. Compared to pkgmgr it's a doddle to 
install a driver as you don't need an obtuse unattend.xml.

I'm about to test the same image against 32bit Windows 2003. I'll report
back on how that goes.

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490