About ReactOS hive files and end_pages in header being possibly wrong

[Richard W.M. Jones]

Hi, I'm one of the authors of hivex, a program for reading Windows
registry hive files:

http://libguestfs.org/hivex.3.html

The library can't currently read ReactOS hive files.  I'm *very*
hestitant to say anyone is "wrong" here since the documentation for
hive files is obviously non-existent, but it appears that ReactOS
doesn't fill in the end_pages field in the header, and instead relies
on the size of the file itself.

For example, a hive downloaded from a fresh ReactOS install:

  hivex_open: header fields:
    file version             1.3
    sequence nos             3 3
      (sequences nos should match if hive was synched at shutdown)
    original file name       
      (only 32 chars are stored, name is probably truncated)
    root offset              0x70 + 0x1000
**  end of last page         0x1000 + 0x1000 (total file size 0x22000)
    checksum                 0x8dc38fdf (calculated 0x8dc38fdf)

Notice that "end of last page" field is 0x1000, versus a completely
different file size.

hivex currently refuses to read past the end_pages limit, and we are
able to read every Windows hive that has ever been thrown at us.

If I hack hivex so that it ignores end_pages and simply uses the size
of the file as a guide, then hivex can successfully read the whole of
the file.

However I'm a bit unwilling to make this change to hivex just for
ReactOS, since it works fine with real Windows already.

So I wonder if this could just be a mistake in ReactOS or if there's
some deeper reason that I'm missing.

TIA,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Aleksey Bragin]

Hi,
thanks a lot for your comment, I'm quite interested in it!

Certainly, since ReactOS's hive file format [should be] is the same  
as NT one, we need to fix ReactOS. It would be very very helpful if  
you could find other differences.

In the meanwhile, I will try to see that end of last page thing.


WBR,
Aleksey Bragin.


On May 31, 2011, at 4:47 PM, Richard W.M. Jones wrote:

>  
>  Hi, I'm one of the authors of hivex, a program for reading Windows
>  registry hive files:
>
>  http://libguestfs.org/hivex.3.html
>
>  The library can't currently read ReactOS hive files.  I'm *very*
>  hestitant to say anyone is "wrong" here since the documentation for
>  hive files is obviously non-existent, but it appears that ReactOS
>  doesn't fill in the end_pages field in the header, and instead relies
>  on the size of the file itself.
>
>  For example, a hive downloaded from a fresh ReactOS install:
>
>    hivex_open: header fields:
>      file version             1.3
>      sequence nos             3 3
>        (sequences nos should match if hive was synched at shutdown)
>      original file name
>        (only 32 chars are stored, name is probably truncated)
>      root offset              0x70 + 0x1000
>  **  end of last page         0x1000 + 0x1000 (total file size 0x22000)
>      checksum                 0x8dc38fdf (calculated 0x8dc38fdf)
>
>  Notice that "end of last page" field is 0x1000, versus a completely
>  different file size.
>
>  hivex currently refuses to read past the end_pages limit, and we are
>  able to read every Windows hive that has ever been thrown at us.
>
>  If I hack hivex so that it ignores end_pages and simply uses the size
>  of the file as a guide, then hivex can successfully read the whole of
>  the file.
>
>  However I'm a bit unwilling to make this change to hivex just for
>  ReactOS, since it works fine with real Windows already.
>
>  So I wonder if this could just be a mistake in ReactOS or if there's
>  some deeper reason that I'm missing.
>
>  TIA,
>
>  Rich.
>
>  -- 
>  Richard Jones, Virtualization Group, Red Hat http:// 
>  people.redhat.com/~rjones
>  virt-df lists disk usage of guests without needing to install any
>  software inside the virtual machine.  Supports Linux and Windows.
>  http://et.redhat.com/~rjones/virt-df/
>
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev

[Alex Ionescu]

Hi!

I'm one of the main authors of ReactOS' registry code in the kernel.

Although ReactOS uses a very compatible version of the Cm library which handles the
higher-order registry functionality, it uses a customized version of the Hive Library (Hv)
which is incompatible with the Windows one on many factors, including the one you
mentioned (as well as incorrectly sorting the keys -- you must not be relying on that
functionality).

A good GSoC project would've been for someone to rewrite the Hv code and/or fix it.

--
Best regards,
Alex Ionescu

On 2011-05-31, at 8:47 AM, Richard W.M. Jones wrote:

>   
>  Hi, I'm one of the authors of hivex, a program for reading Windows
>  registry hive files:
>  
>  http://libguestfs.org/hivex.3.html
>  
>  The library can't currently read ReactOS hive files.  I'm *very*
>  hestitant to say anyone is "wrong" here since the documentation for
>  hive files is obviously non-existent, but it appears that ReactOS
>  doesn't fill in the end_pages field in the header, and instead relies
>  on the size of the file itself.
>  
>  For example, a hive downloaded from a fresh ReactOS install:
>  
>   hivex_open: header fields:
>     file version             1.3
>     sequence nos             3 3
>       (sequences nos should match if hive was synched at shutdown)
>     original file name       
>       (only 32 chars are stored, name is probably truncated)
>     root offset              0x70 + 0x1000
>  **  end of last page         0x1000 + 0x1000 (total file size 0x22000)
>     checksum                 0x8dc38fdf (calculated 0x8dc38fdf)
>  
>  Notice that "end of last page" field is 0x1000, versus a completely
>  different file size.
>  
>  hivex currently refuses to read past the end_pages limit, and we are
>  able to read every Windows hive that has ever been thrown at us.
>  
>  If I hack hivex so that it ignores end_pages and simply uses the size
>  of the file as a guide, then hivex can successfully read the whole of
>  the file.
>  
>  However I'm a bit unwilling to make this change to hivex just for
>  ReactOS, since it works fine with real Windows already.
>  
>  So I wonder if this could just be a mistake in ReactOS or if there's
>  some deeper reason that I'm missing.
>  
>  TIA,
>  
>  Rich.
>  
>  -- 
>  Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
>  virt-df lists disk usage of guests without needing to install any
>  software inside the virtual machine.  Supports Linux and Windows.
>  http://et.redhat.com/~rjones/virt-df/
>  
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev

[Richard W.M. Jones]

On Tue, May 31, 2011 at 09:03:15AM -0400, Alex Ionescu wrote:
>   Hi!
>  
>  I'm one of the main authors of ReactOS' registry code in the kernel.
>
>  Although ReactOS uses a very compatible version of the Cm library
>  which handles the higher-order registry functionality, it uses a
>  customized version of the Hive Library (Hv) which is incompatible
>  with the Windows one on many factors, including the one you
>  mentioned (as well as incorrectly sorting the keys -- you must not
>  be relying on that functionality). 
Yes, IIRC we do rely on the sorting of keys as well .. I will have
to check that.

>   A good GSoC project would've been for someone to
> rewrite the Hv code
>  and/or fix it. 
I've checked out the code from SVN now and I'll take a look if there
are any easy fixes, particularly this issue.

hivex can read and write hives and it's got to the stage where it's
very reliable (at least for reading and small writes), so it may be a
useful reference for potential GSoC coders.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

The fix isn't very trivial.  It would have to start by fixing
the documentation for that field:

Index: hivedata.h
===================================================================
--- hivedata.h  (revision 52019)
+++ hivedata.h  (working copy)
@@ -143,7 +143,9 @@
       If the hive is volatile, this is the actual pointer to the CM_KEY_NODE */
    HCELL_INDEX RootCell;
 
-   /* Size of each hive block ? */
+   /* Length of meaningful data in the file, excluding base block.
+      The file itself can be longer than this, but Windows will ignore
+      anything after. */
    ULONG Length;
 
    /* (1?) */

----------------------------------------------------------------------

I'm still setting up a ReactOS build environment.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

[Colin Finck]

Richard W.M. Jones <rjones(a)redhat.com> wrote:
>   I'm still setting up a ReactOS build environment.
I hope you have taken a look at http://reactos.org/wiki/Build_Environment.
We have prepackaged all needed build tools in the right versions, so 
this environment guarantees comparable builds.

Best regards,

Colin