Xen devel - Aug 2009 - Several vtpm patches and workarounds: persistence, stability, tpm_emulator-0.5.1

We are using xen and vtpm at JHU APL for a project and ran into many
problems. I''ve had to develop several patches and workarounds and
wanted
to contribute them back to the xen community.

Here are a few patches that will make the xen vtpm system more stable
and allow you to have persistent vtpms. In other words you can reboot a
domU and it will come back up with the same vtpm instance and retain all
the keys and data you stored in it.

Also included is a patch that ports vtpm to tpm_emulator-0.5.1. The new
emulator has a lot of bug fixes over the old 0.4 and is recommended if
you want a working vtpm implementation. This port is incomplete, so
please see the details on that patch before applying it.

Some of these are actual bug fixes while others are hacks/workarounds.
Becuase of this, they have been broken into several patches to assist
the developers in choosing which they want to integrate. With these
patches we have been successfully able to use persistent vtpms for
signing certificates.

All of these patches can be applied on top of each other in any order.
$ patch -p1 < patchfile


Finally, there are also some bugs in the xen hotplug system and the
vtpm_manager. Sometimes the manager can get into a corrupted state and
it will cease to work properly. Workarounds for some of the problems are
included at the end of this email.


=============================vtpm_manager-hash_error.patch
=============================There is a bug in the vtpm_manager that has to do
with hashing and
saving the NVM memory files (vtpm_dm_%d.data). The file is not truncated
when it is written and this results in the hash becoming invalid because
of the extra bits at the end of the file.

This patch adds O_TRUNC to the flags when opening the file.

More details on this issue are in the bug report on bugzilla 
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1488

=============================vtpm-uuid.patch
=============================Right now xen will create a new vtpm instance
everytime you start up a
domU, even if you specify the instance parameter in your config file.
Each vtpm instance is then given a uuid and the vtpm.db file maps
instance numbers to uuid numbers.

This patch is a hack that lets you explicitly set the uuid of your vtpm
instance. Everytime you boot up your domU now the vtpm will get that
uuid and thus it will always get the same vtpm instance number instead
of being generated a new one.

So for example, in your config file you would do something like this
vtpm = [ ''backend=0,uuid=dcdb124b-9fed-4040-b149-dd2dfd8d094c''
]

If you are using this patch then be sure to also use the hash_error
patch, otherwise you may see checksum failed messages when booting
your domain and the vtpm tries to read the NVM file. These 2 patches
were made separate because the first is a bug fix and this one is more
of a hack.

=============================vtpm-0.4-persistence.patch
=============================This patch is only needed if you want to continue
using
tpm_emulator-0.4. It is not necessary if you are going to use the
tpm_emulator-0.5.1 patch.

This patch will add
#define TPM_STRONG_PERSISTENCE
which will make the tpm_emulator send a TPM_SaveState command after
every tpm command it executes. This is needed because some commands like
TPM_TakeOwnership do not send the TPM_SaveState command on their own.
The tpm_emulator will only request the manager to save its state when
this TPM command is sent.

So in short without this patch, if you took ownership of your vtpm and
then rebooted the domU, the the change in state would not be saved and
your vtpm would come back unowned again. 

I imagine several other tpm commands would have this problem as well.

=============================vtpm-0.5.1.patch
=============================This patch will port vtpm to use tpm_emulator-0.5.1
The newer version of the emulator contains several bug fixes, one that
we were seeing in our use of vtpm.

This patch also defines TPM_STRONG_PERSISTENCE for the new emulator.

A couple of important notes about this patch:
-This has only been tested on PVM domU''s. In theory it should work for
HVM but I have not tried it at all and can guarantee nothing.
-All the relevant changes in tools/vtpm/vtpm.patch have been ported to 
tpm_emulator-0.5.1. 
-None of the changes in tpm_emulator.patch have been ported. In
particular this means the BUILD_EMULATOR option, which as I understand
lets you use the tpm_emulator in dom0 for a machine that does
not have a real hardware TPM does not work. This functionality should be
easy to add though because the new emulator already comes with a kernel
module interface.
-No considerations were made for the VTPM_MULTI_VM feature (which is
supposedly unfinished). This patch may or may not break any progress
made on that feature.

===========================vtpm_manager and xen hotplug workarounds
===========================Here are some issues I''ve run into when
trying to use vtpm. Note that in
my test cases we were only using vtpms in PVM domains.

It might make sense to add these to a readme or something somewhere
until the hotplug issues are fixed.

1-Q) When I boot my domU with a vtpm for the first time I get 
the following error message in the vtpm_managerd output
Loading NVM.
        Sending LoadNVM command
ERROR[VTPM]: Failed to load NVM
.INFO[VTPM]: [VTPM Listener]: VTPM Listener waiting for messages.
        Reading LoadNVM header
1-A) This is ok. This message comes up when the vtpm non-volatile memory
file does not exist, which is normal when xen creates a new vtpm
instance.

2-Q) When I start vtpm_managerd it starts spamming output to the console
forever and gives the following error:
ERROR[VTPM]: [Hotplug Listener]: Hotplug Listener can''t read from ipc.
Errono = 0. Aborting... 
2-A) Sometimes the hotplug scripts and the fifos they use to
communicate 
get in a corrupted state. We need to clear all the fifos. 
1) First, stop all of the vms that have vtpms.
2) Kill the vtpm_managerd
3) Search for vtpm processes.
#ps -ef | grep vtpm
You may see processes that look like the following. If you do
not see any then skip ahead to the next step.
/bin/bash /etc/xen/scripts/vtpm add
dd skip=10 bs=1 count=4 if=/var/vtpm/fifos/to_console.fifo
/usr/bin/vtpmd
First, kill any of the dd processes, and then run ps again. Most if
not all  of the /etx/xen/scripts/vtpm processes should have quit.
Kill any of the remaining scripts/vtpm and vtpmd processes.
Note that after killing some of of the "vtpm add" processes
new "vtpm remove" processes may get spawned which you will
also need to kill.
4) Delete all of the fifos and socks
#rm /var/vtpm/fifos/*
#rm /var/vtpm/socks/*
5) Remove the lock files if they exist
# rm -rf /var/run/xen-hotplug/vtpm*
6) Now start up the vtpm_managerd again, it should start without errors.
7) Finally, you should be able to boot up the vms again without any
problems.

3-Q) When I start a domU that has a vtpm it hangs in the
pause state and will not boot. If I wait long enough it will quit and
tell me that the Hotplug scripts are not working.
3-A) This occurs when we have stale lock files that did not get removed
properly. 
Perform the same set of steps in 2-A.







_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks. Can you provide a signed-off-by line for these patches? Then at
least some of them can go into the Xen repository.

 -- Keir

On 21/08/2009 17:02, "Matt Fioravante"
<Matthew.Fioravante@jhuapl.edu>
wrote:
> We are using xen and vtpm at JHU APL for a project and ran into many
> problems. I''ve had to develop several patches and workarounds and
wanted
> to contribute them back to the xen community.
> 
> Here are a few patches that will make the xen vtpm system more stable
> and allow you to have persistent vtpms. In other words you can reboot a
> domU and it will come back up with the same vtpm instance and retain all
> the keys and data you stored in it.
> 
> Also included is a patch that ports vtpm to tpm_emulator-0.5.1. The new
> emulator has a lot of bug fixes over the old 0.4 and is recommended if
> you want a working vtpm implementation. This port is incomplete, so
> please see the details on that patch before applying it.
> 
> Some of these are actual bug fixes while others are hacks/workarounds.
> Becuase of this, they have been broken into several patches to assist
> the developers in choosing which they want to integrate. With these
> patches we have been successfully able to use persistent vtpms for
> signing certificates.
> 
> All of these patches can be applied on top of each other in any order.
> $ patch -p1 < patchfile
> 
> 
> Finally, there are also some bugs in the xen hotplug system and the
> vtpm_manager. Sometimes the manager can get into a corrupted state and
> it will cease to work properly. Workarounds for some of the problems are
> included at the end of this email.
> 
> 
> =============================> vtpm_manager-hash_error.patch
> =============================> There is a bug in the vtpm_manager that
has to do with hashing and
> saving the NVM memory files (vtpm_dm_%d.data). The file is not truncated
> when it is written and this results in the hash becoming invalid because
> of the extra bits at the end of the file.
> 
> This patch adds O_TRUNC to the flags when opening the file.
> 
> More details on this issue are in the bug report on bugzilla
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1488
> 
> =============================> vtpm-uuid.patch
> =============================> Right now xen will create a new vtpm
instance everytime you start up a
> domU, even if you specify the instance parameter in your config file.
> Each vtpm instance is then given a uuid and the vtpm.db file maps
> instance numbers to uuid numbers.
> 
> This patch is a hack that lets you explicitly set the uuid of your vtpm
> instance. Everytime you boot up your domU now the vtpm will get that
> uuid and thus it will always get the same vtpm instance number instead
> of being generated a new one.
> 
> So for example, in your config file you would do something like this
> vtpm = [
''backend=0,uuid=dcdb124b-9fed-4040-b149-dd2dfd8d094c'' ]
> 
> If you are using this patch then be sure to also use the hash_error
> patch, otherwise you may see checksum failed messages when booting
> your domain and the vtpm tries to read the NVM file. These 2 patches
> were made separate because the first is a bug fix and this one is more
> of a hack.
> 
> =============================> vtpm-0.4-persistence.patch
> =============================> This patch is only needed if you want to
continue using
> tpm_emulator-0.4. It is not necessary if you are going to use the
> tpm_emulator-0.5.1 patch.
> 
> This patch will add
> #define TPM_STRONG_PERSISTENCE
> which will make the tpm_emulator send a TPM_SaveState command after
> every tpm command it executes. This is needed because some commands like
> TPM_TakeOwnership do not send the TPM_SaveState command on their own.
> The tpm_emulator will only request the manager to save its state when
> this TPM command is sent.
> 
> So in short without this patch, if you took ownership of your vtpm and
> then rebooted the domU, the the change in state would not be saved and
> your vtpm would come back unowned again.
> 
> I imagine several other tpm commands would have this problem as well.
> 
> =============================> vtpm-0.5.1.patch
> =============================> This patch will port vtpm to use
tpm_emulator-0.5.1
> The newer version of the emulator contains several bug fixes, one that
> we were seeing in our use of vtpm.
> 
> This patch also defines TPM_STRONG_PERSISTENCE for the new emulator.
> 
> A couple of important notes about this patch:
> -This has only been tested on PVM domU''s. In theory it should work
for
> HVM but I have not tried it at all and can guarantee nothing.
> -All the relevant changes in tools/vtpm/vtpm.patch have been ported to
> tpm_emulator-0.5.1.
> -None of the changes in tpm_emulator.patch have been ported. In
> particular this means the BUILD_EMULATOR option, which as I understand
> lets you use the tpm_emulator in dom0 for a machine that does
> not have a real hardware TPM does not work. This functionality should be
> easy to add though because the new emulator already comes with a kernel
> module interface.
> -No considerations were made for the VTPM_MULTI_VM feature (which is
> supposedly unfinished). This patch may or may not break any progress
> made on that feature.
> 
> ===========================> vtpm_manager and xen hotplug workarounds
> ===========================> Here are some issues I''ve run into
when trying to use vtpm. Note that in
> my test cases we were only using vtpms in PVM domains.
> 
> It might make sense to add these to a readme or something somewhere
> until the hotplug issues are fixed.
> 
> 1-Q) When I boot my domU with a vtpm for the first time I get
> the following error message in the vtpm_managerd output
> Loading NVM.
>         Sending LoadNVM command
> ERROR[VTPM]: Failed to load NVM
> .INFO[VTPM]: [VTPM Listener]: VTPM Listener waiting for messages.
>         Reading LoadNVM header
> 1-A) This is ok. This message comes up when the vtpm non-volatile memory
> file does not exist, which is normal when xen creates a new vtpm
> instance.
> 
> 2-Q) When I start vtpm_managerd it starts spamming output to the console
> forever and gives the following error:
> ERROR[VTPM]: [Hotplug Listener]: Hotplug Listener can''t read from
ipc.
> Errono = 0. Aborting...
> 2-A) Sometimes the hotplug scripts and the fifos they use to
> communicate 
> get in a corrupted state. We need to clear all the fifos.
> 1) First, stop all of the vms that have vtpms.
> 2) Kill the vtpm_managerd
> 3) Search for vtpm processes.
> #ps -ef | grep vtpm
> You may see processes that look like the following. If you do
> not see any then skip ahead to the next step.
> /bin/bash /etc/xen/scripts/vtpm add
> dd skip=10 bs=1 count=4 if=/var/vtpm/fifos/to_console.fifo
> /usr/bin/vtpmd
> First, kill any of the dd processes, and then run ps again. Most if
> not all  of the /etx/xen/scripts/vtpm processes should have quit.
> Kill any of the remaining scripts/vtpm and vtpmd processes.
> Note that after killing some of of the "vtpm add" processes
> new "vtpm remove" processes may get spawned which you will
> also need to kill.
> 4) Delete all of the fifos and socks
> #rm /var/vtpm/fifos/*
> #rm /var/vtpm/socks/*
> 5) Remove the lock files if they exist
> # rm -rf /var/run/xen-hotplug/vtpm*
> 6) Now start up the vtpm_managerd again, it should start without errors.
> 7) Finally, you should be able to boot up the vms again without any
> problems.
> 
> 3-Q) When I start a domU that has a vtpm it hangs in the
> pause state and will not boot. If I wait long enough it will quit and
> tell me that the Hotplug scripts are not working.
> 3-A) This occurs when we have stale lock files that did not get removed
> properly. 
> Perform the same set of steps in 2-A.
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Signed-off-by: Matt Fioravante <Matthew.Fioravante@jhuapl.edu>

On Fri, 2009-08-21 at 12:33 -0400, Keir Fraser wrote:
> Thanks. Can you provide a signed-off-by line for these patches? Then at
> least some of them can go into the Xen repository.
> 
>  -- Keir
> 
> On 21/08/2009 17:02, "Matt Fioravante"
<Matthew.Fioravante@jhuapl.edu>
> wrote:
> 
> > We are using xen and vtpm at JHU APL for a project and ran into many
> > problems. I''ve had to develop several patches and workarounds
and wanted
> > to contribute them back to the xen community.
> > 
> > Here are a few patches that will make the xen vtpm system more stable
> > and allow you to have persistent vtpms. In other words you can reboot
a
> > domU and it will come back up with the same vtpm instance and retain
all
> > the keys and data you stored in it.
> > 
> > Also included is a patch that ports vtpm to tpm_emulator-0.5.1. The
new
> > emulator has a lot of bug fixes over the old 0.4 and is recommended if
> > you want a working vtpm implementation. This port is incomplete, so
> > please see the details on that patch before applying it.
> > 
> > Some of these are actual bug fixes while others are hacks/workarounds.
> > Becuase of this, they have been broken into several patches to assist
> > the developers in choosing which they want to integrate. With these
> > patches we have been successfully able to use persistent vtpms for
> > signing certificates.
> > 
> > All of these patches can be applied on top of each other in any order.
> > $ patch -p1 < patchfile
> > 
> > 
> > Finally, there are also some bugs in the xen hotplug system and the
> > vtpm_manager. Sometimes the manager can get into a corrupted state and
> > it will cease to work properly. Workarounds for some of the problems
are
> > included at the end of this email.
> > 
> > 
> > =============================> > vtpm_manager-hash_error.patch
> > =============================> > There is a bug in the
vtpm_manager that has to do with hashing and
> > saving the NVM memory files (vtpm_dm_%d.data). The file is not
truncated
> > when it is written and this results in the hash becoming invalid
because
> > of the extra bits at the end of the file.
> > 
> > This patch adds O_TRUNC to the flags when opening the file.
> > 
> > More details on this issue are in the bug report on bugzilla
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1488
> > 
> > =============================> > vtpm-uuid.patch
> > =============================> > Right now xen will create a new
vtpm instance everytime you start up a
> > domU, even if you specify the instance parameter in your config file.
> > Each vtpm instance is then given a uuid and the vtpm.db file maps
> > instance numbers to uuid numbers.
> > 
> > This patch is a hack that lets you explicitly set the uuid of your
vtpm
> > instance. Everytime you boot up your domU now the vtpm will get that
> > uuid and thus it will always get the same vtpm instance number instead
> > of being generated a new one.
> > 
> > So for example, in your config file you would do something like this
> > vtpm = [
''backend=0,uuid=dcdb124b-9fed-4040-b149-dd2dfd8d094c'' ]
> > 
> > If you are using this patch then be sure to also use the hash_error
> > patch, otherwise you may see checksum failed messages when booting
> > your domain and the vtpm tries to read the NVM file. These 2 patches
> > were made separate because the first is a bug fix and this one is more
> > of a hack.
> > 
> > =============================> > vtpm-0.4-persistence.patch
> > =============================> > This patch is only needed if
you want to continue using
> > tpm_emulator-0.4. It is not necessary if you are going to use the
> > tpm_emulator-0.5.1 patch.
> > 
> > This patch will add
> > #define TPM_STRONG_PERSISTENCE
> > which will make the tpm_emulator send a TPM_SaveState command after
> > every tpm command it executes. This is needed because some commands
like
> > TPM_TakeOwnership do not send the TPM_SaveState command on their own.
> > The tpm_emulator will only request the manager to save its state when
> > this TPM command is sent.
> > 
> > So in short without this patch, if you took ownership of your vtpm and
> > then rebooted the domU, the the change in state would not be saved and
> > your vtpm would come back unowned again.
> > 
> > I imagine several other tpm commands would have this problem as well.
> > 
> > =============================> > vtpm-0.5.1.patch
> > =============================> > This patch will port vtpm to
use tpm_emulator-0.5.1
> > The newer version of the emulator contains several bug fixes, one that
> > we were seeing in our use of vtpm.
> > 
> > This patch also defines TPM_STRONG_PERSISTENCE for the new emulator.
> > 
> > A couple of important notes about this patch:
> > -This has only been tested on PVM domU''s. In theory it should
work for
> > HVM but I have not tried it at all and can guarantee nothing.
> > -All the relevant changes in tools/vtpm/vtpm.patch have been ported to
> > tpm_emulator-0.5.1.
> > -None of the changes in tpm_emulator.patch have been ported. In
> > particular this means the BUILD_EMULATOR option, which as I understand
> > lets you use the tpm_emulator in dom0 for a machine that does
> > not have a real hardware TPM does not work. This functionality should
be
> > easy to add though because the new emulator already comes with a
kernel
> > module interface.
> > -No considerations were made for the VTPM_MULTI_VM feature (which is
> > supposedly unfinished). This patch may or may not break any progress
> > made on that feature.
> > 
> > ===========================> > vtpm_manager and xen hotplug
workarounds
> > ===========================> > Here are some issues
I''ve run into when trying to use vtpm. Note that in
> > my test cases we were only using vtpms in PVM domains.
> > 
> > It might make sense to add these to a readme or something somewhere
> > until the hotplug issues are fixed.
> > 
> > 1-Q) When I boot my domU with a vtpm for the first time I get
> > the following error message in the vtpm_managerd output
> > Loading NVM.
> >         Sending LoadNVM command
> > ERROR[VTPM]: Failed to load NVM
> > .INFO[VTPM]: [VTPM Listener]: VTPM Listener waiting for messages.
> >         Reading LoadNVM header
> > 1-A) This is ok. This message comes up when the vtpm non-volatile
memory
> > file does not exist, which is normal when xen creates a new vtpm
> > instance.
> > 
> > 2-Q) When I start vtpm_managerd it starts spamming output to the
console
> > forever and gives the following error:
> > ERROR[VTPM]: [Hotplug Listener]: Hotplug Listener can''t read
from ipc.
> > Errono = 0. Aborting...
> > 2-A) Sometimes the hotplug scripts and the fifos they use to
> > communicate 
> > get in a corrupted state. We need to clear all the fifos.
> > 1) First, stop all of the vms that have vtpms.
> > 2) Kill the vtpm_managerd
> > 3) Search for vtpm processes.
> > #ps -ef | grep vtpm
> > You may see processes that look like the following. If you do
> > not see any then skip ahead to the next step.
> > /bin/bash /etc/xen/scripts/vtpm add
> > dd skip=10 bs=1 count=4 if=/var/vtpm/fifos/to_console.fifo
> > /usr/bin/vtpmd
> > First, kill any of the dd processes, and then run ps again. Most if
> > not all  of the /etx/xen/scripts/vtpm processes should have quit.
> > Kill any of the remaining scripts/vtpm and vtpmd processes.
> > Note that after killing some of of the "vtpm add" processes
> > new "vtpm remove" processes may get spawned which you will
> > also need to kill.
> > 4) Delete all of the fifos and socks
> > #rm /var/vtpm/fifos/*
> > #rm /var/vtpm/socks/*
> > 5) Remove the lock files if they exist
> > # rm -rf /var/run/xen-hotplug/vtpm*
> > 6) Now start up the vtpm_managerd again, it should start without
errors.
> > 7) Finally, you should be able to boot up the vms again without any
> > problems.
> > 
> > 3-Q) When I start a domU that has a vtpm it hangs in the
> > pause state and will not boot. If I wait long enough it will quit and
> > tell me that the Hotplug scripts are not working.
> > 3-A) This occurs when we have stale lock files that did not get
removed
> > properly. 
> > Perform the same set of steps in 2-A.
> > 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel