ReactOS versioning - Ros-dev

7 Mar 2015

Hi,

Here's the promised suggestion regarding how we handle versioning 
problems in reactos. It has some relationship to the tree restructure.

Since some time we now run into issues with our targeted Windows 
version. This is both wine dlls, as well as applications that refuse to 
run due to reactos being limited to Windows server 2003 SP2.

I think many of us, me included, see more in ReactOS than an academic 
research project, or a nice way for 3rd party companies to cheaply get 
insight into how the Windows kernel works. So we are interested in 
making it an actually useful operating system. To achieve this goal, it 
is obviously important to make it run modern Windows applications.

The current approach of pure Windows 2003 Server SP2 compatibility on 
user mode side is a dead end. Our target OS version is starting to 
become a fossil. With time more and more applications will simply refuse 
to work on it. Even wine DLLs start to require Vista APIs and their 
number will most likely increase.

So what can we do? It is obvious, that we cannot instantaniously switch 
all user mode to Windows 7/8/10 compatibility, due to the amount of 
required work, especially regarding missing kernel features.

The wine approach is just adding whatever is needed, creating a Windows 
version chimera. It has already been discussed here and shown to be a 
problem, since it can easily fool applications into believing they run 
on Vista or Windows 7, making them demand all the modern features, which 
we cannot provide, thus failing to run, while they would run flawlessly, 
when being provided a pure Windows 2003 environment, restricting itself 
to this functionality. So this is also not a very good way, either.

So the conclusion is, that we need a mechanism, that allows us to 
control this, providing individual applications with what they require, 
while leaving others in a more restricted environment. And at the same 
time allowing our internal/wine DLLs to make use of higher version 
functionality.

Suggested approach:

1. We need a method to specify which application should be run in which 
environment. We should probably use the same mechanism that is used on 
Windows. Compatibility information is stored in a registry key 
HKCU\Software\Microsodt\Windows NT\CurrentVersion\AppCompatFlags\... The 
trick is to make this easy / transparent for the user. A right-click -> 
properties -> compatibility approach should for now probably be the 
easiest thing, even if it requires the user to actively make this 
setting. A larger app compatibility database would be nice, but it would 
be difficult to figure out what application is being run. And it's also 
a problem to maintain such a list. Potential solutions: detect failures 
to load due to missing imports and app crashes and invoke a 
"compatibility assistent" in that case. Detect first-run of a new 
application and try to identify it, either based on a hash or based on 
PE version information.

2. We need a way to provide the application transparently with the 
environment we want to give it. In terms of DLL exports this could be 
done on the loader side, making it chose the right DLL, potentially 
adding a suffix to the DLL name or selecting a different folder other 
than system32. While this will most likely work good in the majority of 
cases, it is not 100% transparent. Therefore a mechanism in the kernel, 
using file system redirection, like it already exists on 64 bit Windows 
for WoW64, seems to be a more promising approach. The file system 
redirection would redirect system32 into merged folders, containing the 
version specific DLLs, while everything that is not existing in this 
folder will be taken from the original system32. Potential naming 
scheme: system32.601 system32.602, etc.

3. We need a method to create and maintain the required DLLs for 
different OS versions. Preferably avoiding bloat by sharing common code 
in common "parent" DLLs. But also allowing to still plug the DLLs into 
the related Windows version for testing. This can be tricky. I suggest a 
DLL import forwarding scheme. This is both to avoid bloat, i.e. avoid to 
compile and deploy all full blown DLLs for all OS versions, as well as 
creating a better organized system. So each DLL, lets say ADVAPI32, as 
located in different version specific system32 folders, would 
mainly/only consist of forwarders to a "parent" DLL. On Windows we can 
see this being developed similarly, using "api sets" and redirections 
made by the loader. Cloning this mechanism 1:1 might not be the right 
thing though, since it does not address all our requirements. So instead 
I suggest proving our own custom "parent DLLs". While these could be 
organized the same way as on Windows 2003, this is probably not optimal. 
Instead I suggest merging stuff together into 1 or few DLLs (similar to 
how stuff was combined in kernelbase.dll)
This might looks like this (note, that the names are just quickly made 
up names, I don't claim that they are good)
- user32/gdi32 -> ros-win32-core.dll
- kernel32/advapi32 -> ros-kernel-base.dll
- msvcr* -> ros-crt.dll
- ntdll -> ros-ntdll (the kernel would need to load this one)

This also allows our DLLs to make use of higher version APIs, by linking 
them to the parent DLL.
Now this obviously introduces a problem when trying to run individual 
DLLs within a Windows system. To still be able to do this for testing 
purposes, we need to make sure that they still run there.
First, if they import from a custom ros-* DLL, it won't run on Windows 
without that DLL. So we need the possibility to either statically link 
these functions, or compile a "helper DLL", that contains these 
functions, so our DLLs can be run on Windows.
If we used a common parent DLL, this also creates the problem of DLL 
initialization. e.g. the DLL parent for kernel32 and advapi32 would do 
the initialization for both of these, so this would not be suitable to 
use when replacing only one of the DLLs. Instead DLL initialization 
could be done by calling a specific initialization function in the 
parent DLL from the child DLL. So kernel32!DllMain would call 
ros-kernel-base!DllMain_kernel32, passing the original parameters as 
well as a version number, that the parent DLL can use to do version 
specific initialization. This way The parent DLL would only do the 
kernel32 initialization, when the related kernel32 child DLL was used, 
the advapi32 initialization would not be done.
There is still a problem: relocation. So we would need to make sure we 
chose base addresses that still allow us to plug the stuff into Windows 
without causing everything to relocate, which often simply doesn't work.
As an alternative, we should provide a compile time switch to compile 
specific DLLs in a self-contained way.

In terms of structure we could use the MS api-sets as a base for static 
libraries. Then we can link these either into the parent DLLs like 
ros-kernel-base.dll or - when a compile switch is given - to link 
together fully self contained DLLs.

I am really not interested in answers like "This is not what WIndows 
does!", "THIS CANNOT WORK!!!", "You are a ***** even suggesting
this", 
"What if application x parses the import table and disassembles the DLL 
to hook into internal functions, ...":

I am only interested in *constructive* comments.
Everything else: -> /dev/null

Thanks,
Timo