Libguestfs - Feb 2017 - [PATCH v4 1/5] add HIVEX_OPEN_UNSAFE flag.

The following patches address issues when dealing with hives that have
corrupted data in them but are otherwise readable/writable. Those were
found on some rather rare Windows installations that seem to work fine
but current hivex fails to even open.

Those patches change hivex to simply log and ignore such "corrupted"
regions instead of aborting because the caller might be looking at keys
that are perfectly readable/writable (e.g. to identify Windows version
from HKLM/Software/Microsoft/Windows NT/CurrentVersion) and other
"corrupted" and irrelevant keys might prevent one from doing so.

Changes in v4:
 * rebase on current master
 * add HIVEX_OPEN_UNSAFE flag to be used as a guard to enable changes
   made in these series. This is because heuristic approach is not
   guaranteed to be always accurate/safe nor is tolerating corrupted
   blocks when traversing nodes. It's better to have this behavior
   optional.
 * make the "hbin" while loop seek by 4k again and also check against
   h->size as well as h->endpages - same as the outer loop.
 * made hivesh and hivesregedit take -u and --unsafe arguments
   respectively, and also be more forgiving when errors happen that
   we can recover from - as separate patches 4 & 5

Regards,

Dawid Zamirski (5):
  add HIVEX_OPEN_UNSAFE flag.
  lib: change how hbin sections are read.
  lib: allow to walk registry with corrupted blocks
  hivexsh: add -u flag for HIVEX_OPEN_UNSAFE.
  hivexregedit: allow to pass HIVEX_OPEN_UNSAFE

 generator/generator.ml        |  8 +++++
 lib/handle.c                  | 68 +++++++++++++++++++++++++++++++++++++------
 lib/hivex-internal.h          |  1 +
 lib/node.c                    | 46 ++++++++++++++++++++++-------
 perl/lib/Win/Hivex/Regedit.pm | 59 +++++++++++++++++++++++++++++++++----
 regedit/hivexregedit          | 20 +++++++++++--
 sh/hivexsh.c                  | 26 +++++++++++++----
 7 files changed, 193 insertions(+), 35 deletions(-)

-- 
2.9.3

This flag will be used to control behavior of libhivex API functions so
that they tolerate corruption in hives by either using heuristic
recovery from unexpected situations or simply ignore bad registry
keys/values whenever possible.
---
 generator/generator.ml | 8 ++++++++
 lib/handle.c           | 1 +
 lib/hivex-internal.h   | 1 +
 3 files changed, 10 insertions(+)

diff --git a/generator/generator.ml b/generator/generator.ml
index e71a0e7..4125ae7 100755
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -113,6 +113,7 @@ let open_flags = [
   1, "VERBOSE", "Verbose messages";
   2, "DEBUG", "Debug messages";
   4, "WRITE", "Enable writes to the hive";
+  8, "UNSAFE", "Enable heuristics to allow read/write of
corrupted hives";
 ]
 
 (* The API calls. *)
@@ -145,6 +146,13 @@ Open the hive for writing.  If omitted, the hive is
read-only.
 
 See L<hivex(3)/WRITING TO HIVE FILES>.
 
+=item HIVEX_OPEN_UNSAFE
+
+Open the hive in unsafe mode that enables heuristics to handle corrupted hives.
+
+This may allow to read or write registry keys/values that appear intact in an
+otherwise corrupted hive. Use at your own risk.
+
 =back";
 
   "close", (RErrDispose, [AHive]),
diff --git a/lib/handle.c b/lib/handle.c
index d33c1d0..a4982dd 100644
--- a/lib/handle.c
+++ b/lib/handle.c
@@ -83,6 +83,7 @@ hivex_open (const char *filename, int flags)
   DEBUG (2, "created handle %p", h);
 
   h->writable = !!(flags & HIVEX_OPEN_WRITE);
+  h->unsafe = !!(flags & HIVEX_OPEN_UNSAFE);
   h->filename = strdup (filename);
   if (h->filename == NULL)
     goto error;
diff --git a/lib/hivex-internal.h b/lib/hivex-internal.h
index 26f4964..9a497ed 100644
--- a/lib/hivex-internal.h
+++ b/lib/hivex-internal.h
@@ -41,6 +41,7 @@ struct hive_h {
   size_t size;
   int msglvl;                   /* 1 = verbose, 2 or 3 = debug */
   int writable;
+  int unsafe;
 
   /* Registry file, memory mapped if read-only, or malloc'd if writing. */
   union {
-- 
2.9.3

Only when HIVEX_OPEN_UNSAFE flag is set:

* hivex_open: when looping over hbin sections (aka pages), handle a
  case where following hbin section may not begin at exactly at the end
  of previous one. If this happens, scan the page section until next
  one is found and validate it by checking declared offset with actual
  one - if they match, all is good and we can safely move on.

Rationale: there are registry hives there is some garbage data between
hbin section but the hive is still perfectly usable as long as the
offsets stated in hbin headers are correct.
---
 lib/handle.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 46 insertions(+), 5 deletions(-)

diff --git a/lib/handle.c b/lib/handle.c
index a4982dd..8db2f6c 100644
--- a/lib/handle.c
+++ b/lib/handle.c
@@ -227,11 +227,42 @@ hivex_open (const char *filename, int flags)
         page->magic[1] != 'b' ||
         page->magic[2] != 'i' ||
         page->magic[3] != 'n') {
-      SET_ERRNO (ENOTSUP,
-                 "%s: trailing garbage at end of file "
-                 "(at 0x%zx, after %zu pages)",
-                 filename, off, pages);
-      goto error;
+
+      if (!h->unsafe) {
+        SET_ERRNO (ENOTSUP,
+                   "%s: trailing garbage at end of file "
+                   "(at 0x%zx, after %zu pages)",
+                   filename, off, pages);
+        goto error;
+      }
+
+      DEBUG (2,
+             "page not found at expected offset 0x%zx, "
+             "seeking until one is found or EOF is reached",
+             off);
+
+      int found = 0;
+      while (off < h->size) {
+        off += 0x1000;
+
+        if (off >= h->endpages)
+          break;
+
+        page = (struct ntreg_hbin_page *) ((char *) h->addr + off);
+        if (page->magic[0] == 'h' &&
+            page->magic[1] == 'b' &&
+            page->magic[2] == 'i' &&
+            page->magic[3] == 'n') {
+          DEBUG (2, "found next page by seeking at 0x%zx", off);
+          found = 1;
+          break;
+        }
+      }
+
+      if (!found) {
+        DEBUG (2, "page not found and end of pages section reached");
+        break;
+      }
     }
 
     size_t page_size = le32toh (page->page_size);
@@ -255,6 +286,16 @@ hivex_open (const char *filename, int flags)
       goto error;
     }
 
+    size_t page_offset = le32toh(page->offset_first) + 0x1000;
+
+    if (page_offset != off) {
+      SET_ERRNO (ENOTSUP,
+                 "%s: declared page offset (0x%zx) does not match computed
"
+                 "offset (0x%zx), bad registry",
+                 filename, page_offset, off);
+      goto error;
+    }
+
     /* Read the blocks in this page. */
     size_t blkoff;
     struct ntreg_hbin_block *block;
-- 
2.9.3

Only when HIVEX_OPEN_UNSAFE flag is set.

There are some corrupted registry files that have invalid hbin cells
but are still readable. This patch makes the following changes:

* hivex_open - do not abort with complete failure if we run across a
  block with invalid size (unless it's the root block). Instead just
  log the event, and move on. This will allow open hives that have
  apparent invalid blocks but the ones of potential interest might be
  perfectly accessible.
* _hivex_get_children - similiarly, if the's invalid subkey, just skip
  it instead of failing so one can continue to browse other valid
  subkeys.

The above is similar to the behavior to Windows regedit where one can
load such corrupted hives with e.g. "reg load HKU\Corrupted" and
browse/change it despite some keys might be missing.
---
 lib/handle.c | 16 ++++++++++++----
 lib/node.c   | 46 +++++++++++++++++++++++++++++++++++-----------
 2 files changed, 47 insertions(+), 15 deletions(-)

diff --git a/lib/handle.c b/lib/handle.c
index 8db2f6c..05d48bb 100644
--- a/lib/handle.c
+++ b/lib/handle.c
@@ -317,10 +317,18 @@ hivex_open (const char *filename, int flags)
 #pragma GCC diagnostic ignored "-Wstrict-overflow"
       if (seg_len <= 4 || (seg_len & 3) != 0) {
 #pragma GCC diagnostic pop
-        SET_ERRNO (ENOTSUP,
-                   "%s: block size %" PRIi32 " at 0x%zx, bad
registry",
-                   filename, le32toh (block->seg_len), blkoff);
-        goto error;
+        if (is_root || !h->unsafe) {
+          SET_ERRNO (ENOTSUP,
+                     "%s, the block at 0x%zx has invalid size %"
PRIi32
+                     ", bad registry",
+                     filename, blkoff, le32toh (block->seg_len));
+          goto error;
+        } else {
+          DEBUG (2,
+                 "%s: block at 0x%zx has invalid size %" PRIi32
", skipping",
+                 filename, blkoff, le32toh (block->seg_len));
+          break;
+        }
       }
 
       if (h->msglvl >= 2) {
diff --git a/lib/node.c b/lib/node.c
index 822c250..36e61c4 100644
--- a/lib/node.c
+++ b/lib/node.c
@@ -343,11 +343,18 @@ _hivex_get_children (hive_h *h, hive_node_h node,
    */
   size_t nr_children = _hivex_get_offset_list_length (&children);
   if (nr_subkeys_in_nk != nr_children) {
-    SET_ERRNO (ENOTSUP,
-               "nr_subkeys_in_nk = %zu "
-               "is not equal to number of children read %zu",
-               nr_subkeys_in_nk, nr_children);
-    goto error;
+    if (!h->unsafe) {
+      SET_ERRNO (ENOTSUP,
+                 "nr_subkeys_in_nk = %zu "
+                 "is not equal to number of childred read %zu",
+                 nr_subkeys_in_nk, nr_children);
+      goto error;
+    } else {
+      DEBUG (2,
+             "nr_subkeys_in_nk = %zu "
+             "is not equal to number of children read %zu",
+             nr_subkeys_in_nk, nr_children);
+    }
   }
 
  out:
@@ -407,8 +414,14 @@ _get_children (hive_h *h, hive_node_h blkoff,
     for (i = 0; i < nr_subkeys_in_lf; ++i) {
       hive_node_h subkey = le32toh (lf->keys[i].offset);
       subkey += 0x1000;
-      if (check_child_is_nk_block (h, subkey, flags) == -1)
-        return -1;
+      if (check_child_is_nk_block (h, subkey, flags) == -1) {
+        if (h->unsafe) {
+          DEBUG (2, "subkey at 0x%zx is not an NK block, skipping",
subkey);
+          continue;
+        } else {
+          return -1;
+        }
+      }
       if (_hivex_add_to_offset_list (children, subkey) == -1)
         return -1;
     }
@@ -435,8 +448,14 @@ _get_children (hive_h *h, hive_node_h blkoff,
     for (i = 0; i < nr_offsets; ++i) {
       hive_node_h subkey = le32toh (ri->offset[i]);
       subkey += 0x1000;
-      if (check_child_is_nk_block (h, subkey, flags) == -1)
-        return -1;
+      if (check_child_is_nk_block (h, subkey, flags) == -1) {
+        if (h->unsafe) {
+          DEBUG (2, "subkey at 0x%zx is not an NK block, skipping",
subkey);
+          continue;
+        } else {
+          return -1;
+        }
+      }
       if (_hivex_add_to_offset_list (children, subkey) == -1)
         return -1;
     }
@@ -458,8 +477,13 @@ _get_children (hive_h *h, hive_node_h blkoff,
       hive_node_h offset = le32toh (ri->offset[i]);
       offset += 0x1000;
       if (!IS_VALID_BLOCK (h, offset)) {
-        SET_ERRNO (EFAULT, "ri-offset is not a valid block (0x%zx)",
offset);
-        return -1;
+        if (h->unsafe) {
+          DEBUG (2, "ri-offset is not a valid block (0x%zx),
skipping", offset);
+          continue;
+        } else {
+          SET_ERRNO (EFAULT, "ri-offset is not a valid block
(0x%zx)", offset);
+          return -1;
+        }
       }
 
       if (_get_children (h, offset, children, blocks, flags) == -1)
-- 
2.9.3

and pass it to hivex_open. Additionally make hivex_value_value failures
non-critical in this mode when iterating through node children/values.
---
 sh/hivexsh.c | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

diff --git a/sh/hivexsh.c b/sh/hivexsh.c
index c497fb6..b925ddb 100644
--- a/sh/hivexsh.c
+++ b/sh/hivexsh.c
@@ -67,6 +67,7 @@
 
 static int quit = 0;
 static int is_tty;
+static int unsafe = 0;
 static hive_h *h = NULL;
 static char *prompt_string = NULL; /* Normal prompt string. */
 static char *loaded = NULL;     /* Basename of loaded file, if any. */
@@ -97,7 +98,7 @@ static int cmd_setval (char *args);
 static void
 usage (void)
 {
-  fprintf (stderr, "hivexsh [-dfw] [hivefile]\n");
+  fprintf (stderr, "hivexsh [-dfwu] [hivefile]\n");
   exit (EXIT_FAILURE);
 }
 
@@ -115,7 +116,7 @@ main (int argc, char *argv[])
 
   set_prompt_string ();
 
-  while ((c = getopt (argc, argv, "df:w")) != EOF) {
+  while ((c = getopt (argc, argv, "df:wu")) != EOF) {
     switch (c) {
     case 'd':
       open_flags |= HIVEX_OPEN_DEBUG;
@@ -126,6 +127,10 @@ main (int argc, char *argv[])
     case 'w':
       open_flags |= HIVEX_OPEN_WRITE;
       break;
+    case 'u':
+      open_flags |= HIVEX_OPEN_UNSAFE;
+      unsafe = 1;
+      break;
     default:
       usage ();
     }
@@ -775,6 +780,7 @@ cmd_lsval (char *key)
 
       hive_type t;
       size_t len;
+
       if (hivex_value_type (h, values[i], &t, &len) == -1)
         goto error;
 
@@ -783,8 +789,12 @@ cmd_lsval (char *key)
       case hive_t_expand_string:
       case hive_t_link: {
         char *str = hivex_value_string (h, values[i]);
-        if (!str)
-          goto error;
+        if (!str) {
+          if (unsafe)
+            continue;
+          else
+            goto error;
+        }
 
         if (t != hive_t_string)
           printf ("str(%u):", t);
@@ -817,8 +827,12 @@ cmd_lsval (char *key)
       default: {
         unsigned char *data            (unsigned char *) hivex_value_value (h,
values[i], &t, &len);
-        if (!data)
-          goto error;
+        if (!data) {
+          if (unsafe)
+            continue;
+          else
+            goto error;
+        }
 
         printf ("hex(%u):", t);
         size_t j;
-- 
2.9.3

via new --unsafe flag. Also make --export catpure, log and skip over
errors when reading subkeys/values so that export in unsafe mode does
not abort at first sign of error.
---
 perl/lib/Win/Hivex/Regedit.pm | 59 ++++++++++++++++++++++++++++++++++++++-----
 regedit/hivexregedit          | 20 ++++++++++++---
 2 files changed, 70 insertions(+), 9 deletions(-)

diff --git a/perl/lib/Win/Hivex/Regedit.pm b/perl/lib/Win/Hivex/Regedit.pm
index 8914f9e..355699e 100644
--- a/perl/lib/Win/Hivex/Regedit.pm
+++ b/perl/lib/Win/Hivex/Regedit.pm
@@ -67,7 +67,7 @@ package Win::Hivex::Regedit;
 use strict;
 use warnings;
 
-use Carp qw(croak confess);
+use Carp qw(croak carp confess);
 use Encode qw(encode decode);
 
 require Exporter;
@@ -528,19 +528,51 @@ sub reg_export_node
     print $fh "]\n";
 
     my $unsafe_printable_strings = $params{unsafe_printable_strings};
+    my $unsafe = $params{unsafe};
+
+    my @values;
+    my @safe_values;
 
     # Get the values.
-    my @values = $h->node_values ($node);
+    if ($unsafe) {
+        my $have_vals = 0;
+        eval {
+            @values = $h->node_values ($node);
+            $have_vals = 1;
+        };
+
+        if (!$have_vals) {
+            carp "Failed to read node values at $path";
+        }
+    } else {
+        @values = $h->node_values ($node);
+    }
 
     foreach (@values) {
         use bytes;
 
         my $key = $h->value_key ($_);
-        my ($type, $data) = $h->value_value ($_);
-        $_ = { key => $key, type => $type, data => $data }
+        my ($type, $data);
+
+        if ($unsafe) {
+            my $val_ok = 0;
+            eval {
+                ($type, $data) = $h->value_value ($_);
+                $val_ok = 1;
+            };
+
+            if (!$val_ok) {
+                carp "skipping unreadable value of key: $key in
$path";
+                next;
+            }
+        } else {
+            ($type, $data) = $h->value_value ($_);
+        }
+
+        push @safe_values, { key => $key, type => $type, data => $data
};
     }
 
-    @values = sort { $a->{key} cmp $b->{key} } @values;
+    @values = sort { $a->{key} cmp $b->{key} } @safe_values;
 
     # Print the values.
     foreach (@values) {
@@ -573,7 +605,22 @@ sub reg_export_node
     }
     print $fh "\n";
 
-    my @children = $h->node_children ($node);
+    my @children;
+
+    if ($unsafe) {
+        my $have_children = 0;
+        eval {
+            @children = $h->node_children ($node);
+            $have_children = 1;
+        };
+
+        if (!$have_children) {
+            carp "Could not get children of $path";
+        }
+    } else {
+        @children = $h->node_children ($node);
+    }
+
     @children = sort { $h->node_name ($a) cmp $h->node_name ($b) }
@children;
     reg_export_node ($h, $_, $fh, @_) foreach @children;
 }
diff --git a/regedit/hivexregedit b/regedit/hivexregedit
index 6b31a3a..02c382b 100755
--- a/regedit/hivexregedit
+++ b/regedit/hivexregedit
@@ -252,6 +252,17 @@ into another program or stored in another hive.
 
 =cut
 
+my $unsafe;
+
+=item B<--unsafe>
+
+Use heuristics to tolerate certain levels of corruption within hives.
+
+This is unsafe but may allow to export/merge valid keys/values in an
+othewise corrupted hive.
+
+=cut
+
 GetOptions ("help|?" => \$help,
             "debug" => \$debug,
             "merge|import" => \$merge,
@@ -259,6 +270,7 @@ GetOptions ("help|?" => \$help,
             "prefix=s" => \$prefix,
             "encoding=s" => \$encoding,
             "unsafe-printable-strings" =>
\$unsafe_printable_strings,
+            "unsafe" => \$unsafe,
     ) or pod2usage (2);
 pod2usage (1) if $help;
 
@@ -281,7 +293,8 @@ if ($merge) {                   # --merge (reg_import)
 
     my $hivefile = shift @ARGV;
 
-    my $h = Win::Hivex->open ($hivefile, write => 1, debug => $debug);
+    my $h = Win::Hivex->open ($hivefile, write => 1, debug => $debug,
+                              unsafe => $unsafe);
 
     # Read from stdin unless other files have been specified.
     unshift (@ARGV, '-') unless @ARGV;
@@ -312,13 +325,14 @@ if ($merge) {                   # --merge (reg_import)
     my $hivefile = shift @ARGV;
     my $key = shift @ARGV;
 
-    my $h = Win::Hivex->open ($hivefile, debug => $debug);
+    my $h = Win::Hivex->open ($hivefile, debug => $debug, unsafe =>
$unsafe);
 
     print "Windows Registry Editor Version 5.00\n\n";
 
     reg_export ($h, $key, \*STDOUT,
                 prefix => $prefix,
-                unsafe_printable_strings => $unsafe_printable_strings);
+                unsafe_printable_strings => $unsafe_printable_strings,
+                unsafe => $unsafe);
 }
 
 =head1 SEE ALSO
-- 
2.9.3

Thanks - the series looks good to me and passes both the
internal tests and hivex-extra-tests.  I have pushed it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v