Libguestfs - Oct 2013 - Re: Hivex - Trailing garbage at the end of hive file

Rich,

Our workflow is something like this:

0. Start with a fresh copy of windows server 2k8
1. We read the system hive and then write to it a bunch of times
2. Boot windows
3. Read from the system hive

Hivex reports the failure at step #3. I also noticed that the size of the
registry hive observed in step #3 is the same as step #0. Is it possible
that hivex issues write that cause a hive file to shrink in size and while
compacting the hive file, it retains the size but zeroes out the end of
the file? That would point to the trailing zeroes getting introduced in
step #1

It's also possible that windows is padding those zeroes in step #2. I'm
adding some instrumentation to confirm this

Thanks for your help!
~ Hari

On 10/4/13 11:01 AM, "Richard W.M. Jones" <rjones@redhat.com>
wrote:
>On Fri, Oct 04, 2013 at 02:12:08PM +0000, Subramanian, Hari wrote:
>> To respond to you question about "whether it fails", hivexsh
is unable
>>to
>> open the hive file and it prints this message and exits. I've
attached
>>the
>> verbose logs as requested
>
>OK, I understand it now.  It is in fact failing, setting
>errno = ENOSYS and returning an error.
>
>> This extract from the logs shows that the hivexsh complains content
>>after
>> file offset 0x77c000 is garbage
>> 
>> hivex: badsys: trailing garbage at end of file (at 0x77c000, after 1849
>> pages)
>> 
>> So, I went ahead and truncated the contents of the file after that file
>> offset and hivex was able to successfully open the new hive file
>
>I guess if the hive comes from a real guest we can change this to warn
>but not fail.
>
>Should be a pretty simple patch.
>
>Rich.
>
>-- 
>Richard Jones, Virtualization Group, Red Hat
>http://people.redhat.com/~rjones
>virt-df lists disk usage of guests without needing to install any
>software inside the virtual machine.  Supports Linux and Windows.
>http://people.redhat.com/~rjones/virt-df/

On Fri, Oct 04, 2013 at 03:13:47PM +0000, Subramanian, Hari
wrote:
> Rich,
> 
> Our workflow is something like this:
> 
> 0. Start with a fresh copy of windows server 2k8
> 1. We read the system hive and then write to it a bunch of times
> 2. Boot windows
> 3. Read from the system hive
> 
> Hivex reports the failure at step #3. I also noticed that the size of the
> registry hive observed in step #3 is the same as step #0. Is it possible
> that hivex issues write that cause a hive file to shrink in size and while
> compacting the hive file, it retains the size but zeroes out the end of
> the file?
Ah so hivex wrote the hive?  That could indicate a bug in hivex or
perhaps some synchronization issue when uploading it into the guest?

Anyway hivex itself never shrinks the hive.  In fact it only ever
appends to the end of the hive, extending it if necessary (it's not
always necessary).  When it extends the hive it's supposed to be
initializing the hbin correctly.

On the other hand it's anyone's guess what Windows does to the hive in
step 2.

BTW the source is all open ...
https://github.com/libguestfs/hivex/blob/master/lib/write.c
> That would point to the trailing zeroes getting introduced in
> step #1
> 
> It's also possible that windows is padding those zeroes in step #2.
I'm
> adding some instrumentation to confirm this
A good start would be logging the size and MD5 checksum of the hive
before and after each step, and also from within Windows (IIRC there
is an MD5 checksum program in PowerShell).  That would rule out some
sort of non-hivex-related corruption when you're downloading or
uploading the hive.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Hi Rich,

I'm still working on validating whether the trailing zeroes were
introduced by hivex or by windows (though I highly doubt it's hivex). But
since it's part of a more complex workflow which is not that easy to
modify, it's still a work in progress

In the meanwhile, I wanted to be sure that ignoring the condition when
hivex saw these trailing zeroes was the right solution. I applied the
following patch to hivex. Not sure if this is what you had in mind

[ec2-user@ip-10-66-218-126 hivex-1.3.8]$ diff handle.c handle-mod.c
222,226c222,223
< SET_ERRNO (ENOTSUP,
< "%s: trailing garbage at end of file "
< "(at 0x%zx, after %zu pages)",
< filename, off, pages);
< goto error;
---
> printf("hivex: %s: trailing garbage at end of file (at 0x%zx, after
%zu
>pages)\n", filename, off, pages);
> break;

I wanted to understand the behavior of hivex when we wrote to the end of
the hive file (added a new entry under ControlSet001\services). When I do
that I get the following error

badsys-1-win-add-reg\ControlSet001\services> ls
hivex: _hivex_get_children: returning EFAULT because: subkey_lf is not a
valid block (0x780020)
ls: Bad address
badsys-1-win-add-reg\ControlSet001\services> quit


Obviously the add/modify was not successful. But regedit parsed the hive
successfully. I'll continue to debug this but thought I should send it to
the list/you as well

Thanks
~ Hari


On 10/4/13 11:21 AM, "Richard W.M. Jones" <rjones@redhat.com>
wrote:
>On Fri, Oct 04, 2013 at 03:13:47PM +0000, Subramanian, Hari wrote:
>> Rich,
>> 
>> Our workflow is something like this:
>> 
>> 0. Start with a fresh copy of windows server 2k8
>> 1. We read the system hive and then write to it a bunch of times
>> 2. Boot windows
>> 3. Read from the system hive
>> 
>> Hivex reports the failure at step #3. I also noticed that the size of
>>the
>> registry hive observed in step #3 is the same as step #0. Is it
possible
>> that hivex issues write that cause a hive file to shrink in size and
>>while
>> compacting the hive file, it retains the size but zeroes out the end of
>> the file?
>
>Ah so hivex wrote the hive?  That could indicate a bug in hivex or
>perhaps some synchronization issue when uploading it into the guest?
>
>Anyway hivex itself never shrinks the hive.  In fact it only ever
>appends to the end of the hive, extending it if necessary (it's not
>always necessary).  When it extends the hive it's supposed to be
>initializing the hbin correctly.
>
>On the other hand it's anyone's guess what Windows does to the hive
in
>step 2.
>
>BTW the source is all open ...
>https://github.com/libguestfs/hivex/blob/master/lib/write.c
>
>> That would point to the trailing zeroes getting introduced in
>> step #1
>> 
>> It's also possible that windows is padding those zeroes in step #2.
I'm
>> adding some instrumentation to confirm this
>
>A good start would be logging the size and MD5 checksum of the hive
>before and after each step, and also from within Windows (IIRC there
>is an MD5 checksum program in PowerShell).  That would rule out some
>sort of non-hivex-related corruption when you're downloading or
>uploading the hive.
>
>Rich.
>
>-- 
>Richard Jones, Virtualization Group, Red Hat
>http://people.redhat.com/~rjones
>Fedora Windows cross-compiler. Compile Windows programs, test, and
>build Windows installers. Over 100 libraries supported.
>http://fedoraproject.org/wiki/MinGW