Porting ReactOS to the L4 micro-kernel

[Henning Schild]

Hi there,

i am thinking of porting ros to the L4 micro-kernel as topic for my
diploma thesis. The operating systems group in Dresden already ported
Linux to L4 (http://os.inf.tu-dresden.de/L4/LinuxOnL4/) and proofed
that L4 offers all that is needed to run an operating system in
userspace on top of it.

The goal of the work would be to be able to run windows applications
next to miro-kernel applications. And to compare the performance with
that of pure ros, L4Linux/wine and windows. A comparison between the
port and L4Linux could also be part of the work. (mapping processes to
L4 tasks, memory management, interrupt-handling and so on)

I am not very experienced in C or assembler programming but am
motivated to learn. But before i start working on that topic i have
some questions on the portability of ros.

In svn and on the ros website i found that there is a ppc port of ros.
And on irc someone told me that there is ongoing work on a ros usermode
port. I also read about a xen port. But i did not yet find working
copies of the any of that ports. I built ros for i386 on my
machine and was able to boot it in vmware. I also tried building the
latest svn version with ROS_ARCH=powerpc but that did not succeed.

1. Did anyone ever finish a port to another arch?
2. How portable is ros in general? Meaning how much code would have to
be changed. I hope there are abstractions and only 10-15% of the kernel
would have to be changed.
3. Do you believe that one person new to ros can finish that kind of
work in 6 months? I know that might be hard to answer but how long did
any of you need to learn ros in detail?
4. If there are working ports,  how long did it take to implement them?
If there are not, what are the reasons they where never finished? I am
thinking of lots of assembler and i386 hacks.

5. Finally, what do think of the idea itself?

I hope you take the time to answer my questions. I am still on the way
to determine whether i really want to do that and whether i can do this
in 6 months.

regards,
Henning Schild

[Magnus Olsen]

Hi

XEN was working with a early version of XEN
the main problem is XEN change the interface in each
release of XEN. so new port of XEN is need it for each
version of XEN.

The port to PPC are not finish it have taken around one year
for one devloper doing it with help with other devs. it is
allot left todo before PPC port are finish. It is the frist
port to another CPU we are doing.

ReactOS are using same kernel desgin as windows nt (windwos 2003 server)
in the bottom. But the desgin comptable are not finish.

it is not so simple as u think the whole graphic subsystem are in win32k.sys
and printer, partly audio, printers, keyborad and more. ntoskrnl is the 
kernel
that you need as well it is not a easy thing to port. it is month of work 
doing that
same with the graphic subsystem

ReactOS goal is to use windows drivers and apps direcly out of the box.
if you ask if it posible within 6 month. doing port to L4. I say no. we talk
about year works todo it. everthing in ReactOS begin be very hard bound
to the win32k and ntoskrnl and they two are bound to dirvers and the
boot process of whole OS. you need revent the wheel again for all everthing 
again


----- Original Message ----- 
From: "Henning Schild" <s2247401(a)inf.tu-dresden.de>
To: <ros-dev(a)reactos.org>
Sent: Wednesday, October 10, 2007 1:08 PM
Subject: [ros-dev] Porting ReactOS to the L4 micro-kernel


>   Hi there,
>
>  i am thinking of porting ros to the L4 micro-kernel as topic for my
>  diploma thesis. The operating systems group in Dresden already ported
>  Linux to L4 (http://os.inf.tu-dresden.de/L4/LinuxOnL4/) and proofed
>  that L4 offers all that is needed to run an operating system in
>  userspace on top of it.
>
>  The goal of the work would be to be able to run windows applications
>  next to miro-kernel applications. And to compare the performance with
>  that of pure ros, L4Linux/wine and windows. A comparison between the
>  port and L4Linux could also be part of the work. (mapping processes to
>  L4 tasks, memory management, interrupt-handling and so on)
>
>  I am not very experienced in C or assembler programming but am
>  motivated to learn. But before i start working on that topic i have
>  some questions on the portability of ros.
>
>  In svn and on the ros website i found that there is a ppc port of ros.
>  And on irc someone told me that there is ongoing work on a ros usermode
>  port. I also read about a xen port. But i did not yet find working
>  copies of the any of that ports. I built ros for i386 on my
>  machine and was able to boot it in vmware. I also tried building the
>  latest svn version with ROS_ARCH=powerpc but that did not succeed.
>
>  1. Did anyone ever finish a port to another arch?
>  2. How portable is ros in general? Meaning how much code would have to
>  be changed. I hope there are abstractions and only 10-15% of the kernel
>  would have to be changed.
>  3. Do you believe that one person new to ros can finish that kind of
>  work in 6 months? I know that might be hard to answer but how long did
>  any of you need to learn ros in detail?
>  4. If there are working ports,  how long did it take to implement them?
>  If there are not, what are the reasons they where never finished? I am
>  thinking of lots of assembler and i386 hacks.
>
>  5. Finally, what do think of the idea itself?
>
>  I hope you take the time to answer my questions. I am still on the way
>  to determine whether i really want to do that and whether i can do this
>  in 6 months.
>
>  regards,
>  Henning Schild
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev

[Henning Schild]

Hi,

thanks for your comments. I was afraid that it might be very difficult.
I am not really afraid of doing difficult work but i would like to
finish it in time. Both your comments did not make me confident, that i
can archive this goal. Especially since i don't know much about NT
kernel or ros yet.

I will go for another topic.

regards,
Henning Schild

[Marc Piulachs]

Hi,

On the reactos frontpage I see that the project aims to reimplement
WindowsXP , that is NT 5.2 , but currently most of the modules define their
own target version ranging from NT4 to NT 5.3 (2003) .On recent shell32
commits I have even seen someone introducing Vista-only APIs.

I think an effort has to be done to fix that situation ,IMHO targeting
multiple windows versions is fine but not a mix of them so I recently posted
a bug report on mozilla to have it present at least.
http://www.reactos.org/bugzilla/show_bug.cgi?id=2745 but It has been marked
as invalid and I can't understand the reason given, I may be wrong about
that but I'd like to know what I'm missing here, can anyone clarify?

/Marc

[Jerry Maine]

Yes, I would agree. We need to senq all defined windows versions.

Marc Piulachs wrote:
>   Hi,
>
>  On the reactos frontpage I see that the project aims to reimplement
>  WindowsXP , that is NT 5.2 , but currently most of the modules define their
>  own target version ranging from NT4 to NT 5.3 (2003) .On recent shell32
>  commits I have even seen someone introducing Vista-only APIs.
>
>  I think an effort has to be done to fix that situation ,IMHO targeting
>  multiple windows versions is fine but not a mix of them so I recently posted
>  a bug report on mozilla to have it present at least.
>  http://www.reactos.org/bugzilla/show_bug.cgi?id=2745 but It has been marked
>  as invalid and I can't understand the reason given, I may be wrong about
>  that but I'd like to know what I'm missing here, can anyone clarify?
>
>  /Marc
>
>
>
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev
>
>

[Colin Finck]

Hi Marc,

>   Marc Piulachs wrote:
>  On the reactos frontpage I see that the project aims to reimplement
>  WindowsXP , that is NT 5.2 
No, Windows XP is NT 5.1 and Windows Server 2003 is NT 5.2.
Our main target is to reimplement a Windows Server 2003 (SP1)-compatible
operating system (so yes, NT 5.2).

But some parts are not yet at this version. For example, our Storage Stack
in the kernel is still only NT4-compatible.
Nevertheless, when we have to decide between a Windows Server 2003 SP1
behaviour and one of another Windows version (like it has been done in
r28953), we should choose the Windows Server 2003 SP1 behaviour.


>   On recent shell32 commits I have even seen someone
> introducing
>  Vista-only APIs. 
In my opinion, this is no problem as long as these new API's don't break
compatibility with the NT 5.2 API's.
For example, the Vista comctl32.dll introduced many new flags and I think we
can also implement them as they don't break compatibility with NT 5.2's
comctl32.dll (from what I read in MSDN).

Regards,

Colin

[Marc Piulachs]

Hi,

>  In my opinion, this is no problem as long as these new
> API's don't break
> compatibility with the NT 5.2 API's.
> For example, the Vista comctl32.dll introduced many new flags and I think
> we
> can also implement them as they don't break compatibility with NT 5.2's
> comctl32.dll (from what I read in MSDN). 
Ummm I partly disagree with that, IMHO we should not mix different versions
of the WinAPI even when they do not (apparently) conflict each other.  There
are practical reasons, any *well* written application designed for WinXP
(for example) is going to use Vista constants or flags. Should we allow this
application to work on Ros when it will fail on Windows? I think this will
make reactos LESS compatible with windows. Also, supporting some vista flags
or not supporting them is an arbitrary decision IMHO we should have a strict
rule about that or will end up having a FrankensWin :D

There are of course exceptions like the current storage stack you used as an
example but I think a decision has to be made between targeting a specific
windows version and update it time to time dropping older versions like NT4
(what we currently do) or target multiple versions at the same time.

I know it means more work but I see this as a strategic decision, in the
future reactos can benefit from massive demand of companies moving away from
old unsupported Oss like NT4 or Windows 2000 seeking a modern OS that will
run their unmodified NT4 applications or drivers.

[David Hinz]

Marc Piulachs schrieb:
>   I know it means more work but I see this as a
> strategic decision, in the
>  future reactos can benefit from massive demand of companies moving away from
>  old unsupported Oss like NT4 or Windows 2000 seeking a modern OS that will
>  run their unmodified NT4 applications or drivers. 
And why does this mean we
can't support new features, when they don't 
conflict with older things?

I really don't get your point...

The problem I see with sticking to one WinAPI version for some time 
(even if the successor is already released) is, that we're always way 
behind Windows and that's not good if we want to be an alternative.

[Richard Campbell]

his may seem good, but down the road you'll run into issues.  Probably 
one of the most obvious issues is if a user develops his application on 
ROS.  If he makes use of these vista only APIs then suddenly his app 
doesn't work on Windows.  Another reason is application developers do 
funky things.  They've been known to check for certain entry points in a 
DLL.  If said DLL has the entry point they may wrongfully assume they 
are running on vista and start making use of VISTA APIs.  If this 
happens and ROS doesn't support ALL the vista APIs then the app will 
more then likely crash.

I don't even go into the issue of bloat (wasted disk space, memory, etc.)

A solution to this would be to disable the compilation of these items 
unless a flag is turned on.  That way vista compatibility could still be 
enabled at a future date when the api is mature.

David Hinz wrote:
>   Marc Piulachs schrieb:
>    
> >   I know it means more work but I see this as a
> > strategic decision, in the
> >  future reactos can benefit from massive demand of companies moving away from
> >  old unsupported Oss like NT4 or Windows 2000 seeking a modern OS that will
> >  run their unmodified NT4 applications or drivers.
> >       
>  And why does this mean we can't support new features, when they don't
>
>  conflict with older things?
>
>  I really don't get your point...
>
>  The problem I see with sticking to one WinAPI version for some time 
>  (even if the successor is already released) is, that we're always way 
>  behind Windows and that's not good if we want to be an alternative.
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev
>
>

[Richard Campbell]

his may seem good, but down the road you'll run into issues.  Probably 
one of the most obvious issues is if a user develops his application on 
ROS.  If he makes use of these vista only APIs then suddenly his app 
doesn't work on Windows.  Another reason is application developers do 
funky things.  They've been known to check for certain entry points in a 
DLL.  If said DLL has the entry point they may wrongfully assume they 
are running on vista and start making use of VISTA APIs.  If this 
happens and ROS doesn't support ALL the vista APIs then the app will 
more then likely crash.

I don't even go into the issue of bloat (wasted disk space, memory, etc.)

A solution to this would be to disable the compilation of these items 
unless a flag is turned on.  That way vista compatibility could still be 
enabled at a future date when the api is mature.

David Hinz wrote:
>   Marc Piulachs schrieb:
>    
> >   I know it means more work but I see this as a
> > strategic decision, in the
> >  future reactos can benefit from massive demand of companies moving away from
> >  old unsupported Oss like NT4 or Windows 2000 seeking a modern OS that will
> >  run their unmodified NT4 applications or drivers.
> >       
>  And why does this mean we can't support new features, when they don't
>
>  conflict with older things?
>
>  I really don't get your point...
>
>  The problem I see with sticking to one WinAPI version for some time 
>  (even if the successor is already released) is, that we're always way 
>  behind Windows and that's not good if we want to be an alternative.
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev
>
>

[Timo Kreuzer]

Richard Campbell schrieb:
>   his may seem good, but down the road you'll run
> into issues.  Probably 
>  one of the most obvious issues is if a user develops his application on 
>  ROS.  If he makes use of these vista only APIs then suddenly his app 
>  doesn't work on Windows.   
If you develop an app on Vista using Vista APIs, you
will find that it 
won't work on XP. If you use .NET you will find it won't work on a 
system without .NET installed. That's the way the cookie crumbles. A 
developer should at least take a quick look at MSDN before using a 
function and MSDN clearly says if a function or flag or whatever is 
available on nt4, 2k, xp, 2003 or vista. If he doesn't care, his problem.
Our goal is that everything that works on windows also works on ros. But 
noone ever said that everything that works on ros will also work on 
windows. Sooner or later ros will be installable on an EXT2 filesystem 
and probably others too. Does that wok with Windows?

>   Another reason is application developers do 
>  funky things.  They've been known to check for certain entry points in a 
>  DLL.  If said DLL has the entry point they may wrongfully assume they 
>  are running on vista and start making use of VISTA APIs.  If this 
>  happens and ROS doesn't support ALL the vista APIs then the app will 
>  more then likely crash.
>     
I doubt that a developer would go that way. Instead the app should check 
the windows version reported from GetVersion(Ex). And afaik we still 
report NT 5.0. If we found some app does this, than this becomes a valid 
point.

>   I don't even go into the issue of bloat (wasted
> disk space, memory, etc.)
>     
Compare the size of windows to ros. You still speak of bloat? Those few 
additional vista apis don't make much disk space.
>   A solution to this would be to disable the compilation
> of these items 
>  unless a flag is turned on.  That way vista compatibility could still be 
>  enabled at a future date when the api is mature.
>     
A variable for aimed compatibility in the config.rbuild or somewhere 
else seems to be a good idea. So if you want Vista APIs, just change 
that variable. It could also be used for other compatibility things like 
syscall numbers. If we want to be compatible to win 2003, use 2k3 
syscall ids, or if we want to be compatible to xp, use xp syscalls. Most 
things should be uneffected, but we could in fact add some conditional 
compiling here and there to achieve the maximum compatibility with one 
windows version. Might be useful for testing, especially for 
gdi32/user32 and not too much work. It would probably be the best choice 
to make our user32 and gdi32 work on different platforms (currently xp 
is the only platform with matching syscall ids) without much overhead 
and still being fully compatible (some win32k apis change the numer of 
parameters between different windows version, could mostly be solved by 
a simple stub).

Regards,
Timo

[WaxDragon]

"Targeting" a specific version of windows means we are just
reimplementing a certain code base of Microsoft's.  That's not the
direction I want this project to take.  I think we should only report
specific windows versions when there is NO other compatible option.
This project is it's own code-base.  If you want WINDOWS, use WINDOWS.

WD
-- 
22:38 <@WaxDragon> false ^ true
22:39 < false> :(
22:39 < false> dont you think you can XOR me and get away with it! I
always return!

[Klemens Friedl]

In future, ReactOS will probably need a compatibility layer similar to
NT 5.1+ shim engine.

You have probably already seen it in action when using NT 5.1+ the
Compatibility shell extension in the shell32 file-property dialog or
the weird "Compatibility Wizard". It offers some options like "Disable
visual themes", "Run application in Windows xyz" [where xyz is a
previous Win32 OS from 9x to NT 5.0), limited screen resolutions, etc.
A total of 4 shim options, one with four predefined settings.

Most people don't know that NT 5.1+ comes with a shim-database of more
than one hundred shims, plus some other database files that contain
app-names, check sum and some other relational data to match specific
applications (both MS and third party). For example Visual Studio 6
Setup wouldn't work in NT 5.1 because it checks for an outdated MS
Java version, and other system files. The shim engine automatically
checks the app while loading and 'fake' some of the requested
information.
Advanced user can also create their own shim-settings and store it in
an separate shim-database (and share them with others). The related
tool called Compatibility Administrator comes with the AppComp
toolkit.
I have successfully executed several Win9x applications that had been
marked as incompatible with NT. For example, most Win9x games of the
late nineties check for Win9x and refuse to execute on NT. One of them
are games of the Need for Speed game series. I applied some of the
shims to the game executable and player NfS 4 and NfS Porsche was not
a problem at all.

Alex has already investigated how the NT shim-engine work internally,
and 4 of 11 of his (up-to-date) unfinished blog-article series about
it, can be found on his website.

Some information can also be found on msdn website, especially at the
channel9 (videos) sub-section.

ReactOS would benefit from a similar technology as well, for example
video driver from vendors like nVidia and AMD are known for various NT
version specific hacks (as I have been told).
And probably a just-in-time patch of the executable via the system
compatibility layer would be reasonable. On the other side, NT 4 & NT5
video driver are kernel mode drivers and the shim engine of NT 5.1+
acts mainly on the user mode side.
There are two options, afaik, user mode video driver like NT 6, or
patch the driver while installing process.

I am not sure why 64-bit NT 5.2 (2003, XP-64) and NT 6 don't use the
shim-engine to redirect 32-bit executable to the 'correct' path (file
system, registry). And if those OS's use it, why it is such a mess
(System32 (64bit), and SysWOW64 for 32bit; apps dirs, etc.). Afaik,
all those redirection takes place in WOW64.
Using the shim layer, several design flaws could be fixed without
break compatibility; it's a great chance for ReactOS to learn from the
early inventors and implementers and make it better from the
beginning.


Microsoft Application Compatibility Toolkit 5.0 (for NT 5.1+, dotNet
1.1 required, no WGA):
http://www.microsoft.com/downloads/details.aspx?FamilyID=24da89e9-b581-47b0…

app compatibility msdn pages:
http://msdn2.microsoft.com/en-us/library/Aa286552.aspx
http://technet.microsoft.com/en-us/library/bb457032.aspx

Alex Ionescu's related blog entries:
http://www.alex-ionescu.com/index.php?s=Application+Compatilibity+Database



Klemens

[Zachary Gorden]

If the application was written for Windows XP, the programmers wouldn't make
use of Vista flags in the first place, unless their testing procedures
really was crappy or they didn't know what they were doing.  Also, ROS
identifies itself to applications as Server 2003.  At that point, it really
is up to the applications to not use APIs they should know isn't there.  It
ultimately is up to programmers to do their jobs.

Z98