This is just a note for the WIN32K developers.
I ran the dumppe utility to compare imported symbols in WIN32K.SYS on a real system and on ROS'. Here are the results.
ReactOS WIN32K.SYS exported symbols containg substing "Port": ZwConnectPort ZwRequestWaitReplyPort
XPSP2 WIN32K.SYS exported symbols containg substing "Port": LpcRequestWaitReplyPort PsGetProcessDebugPort LpcRequestPort
This suggests port handles are not used in regular LPC processing within WIN32K and that there is not a twin connection to \Windows\ApiPort (ntuser/csr.c).
Emanuele (just guessing)
Emanuele Aliberti wrote:
This is just a note for the WIN32K developers.
I ran the dumppe utility to compare imported symbols in WIN32K.SYS on a real system and on ROS'. Here are the results.
ReactOS WIN32K.SYS exported symbols containg substing "Port": ZwConnectPort ZwRequestWaitReplyPort
XPSP2 WIN32K.SYS exported symbols containg substing "Port": LpcRequestWaitReplyPort PsGetProcessDebugPort LpcRequestPort
This suggests port handles are not used in regular LPC processing within WIN32K and that there is not a twin connection to \Windows\ApiPort (ntuser/csr.c).
Emanuele (just guessing) _______________________________________________
Hi Emanuele,
I'm currently working/researching on the LPC implementation and I'm about to soon start reworking some of our code. Just to avoid any duplication, are you also working on such a thing, or simply exploring?
Best regards, Alex Ionescu
Hi Alex, I hope you are fine now, after the party and a restful night... ;-)
I'm currently working/researching on the LPC implementation and I'm about to soon start reworking some of our code. Just to avoid any duplication, are you also working on such a thing, or simply exploring?
David Welch wrote our initial LPC code to resemble the BSD sockets general design (an LPC port behaves exactly like a BSD socket; there are ports in listening state and connected ports to send/receive datagrams). That is, of course, not the way NT LPC works, but at the time David wrote the code, we knew almost nothing about LPC, but the incomplete information Prasad Dabak wrote in "Undocumented Windows NT" (1999). That code was good enough to start with user mode subsystem servers, but till now it has prevented us from testing real system components like the SM, CSR and friends etc. for compatibility in ROS. This is what I have done for the last three years. What I got from the researching is some better understanding of how LPC works, but not a complete picture. I am glad to see you are working or planning to work on LPC, because there is at least one API that is incompatible with NT's, and the general semantics of the path connect->request->reply is wrong. In short, it is the the listening port that receives the messages from the clients even after the connection happens and not the server side connected port). This could be deducted by studing more carefully the sample code provided by Dabak, but I found it indipendently with some simple c/s test code. I told it to David, Eric and others and they agreed that should be correct. I will share what I know and I can send my unfinished code if it may be of interest (not that I am that a skilled coder, but you could just tour it for fun). Recalling what David Welch told me in many occasions, NT LPC has many flaws, many are security related (this led Microsoft to introduce secure ports with NT 5.0, mostly to armour the LSA). The overall problem with the real LPC is that it looks like really old, not well understood in Microsoft and almost untouched for years (many unfixed LPC bugs still plague Windows). You could even imagine it was part of the Prism project at DEC, in the '80s, perhaps with the name "Channel" for what we now know as "port" (undocumented and unimplemented APIs existed in NT 3.1, NT 3.5, NT 3.51, NT 4.0 and NT 5.0, but disappeared in NT 5.1 and NT 5.2). It seems that only lately (1999+) Microsoft forced the NT kernel team to face the old LPC code to improve it. Alex, what are your plans for rewriting the LPC code?
Emanuele
Emanuele Aliberti wrote:
Hi Alex, I hope you are fine now, after the party and a restful night... ;-)
Someone reads my ROS Blogs! =)
David Welch wrote our initial LPC code to resemble the BSD sockets general design (an LPC port behaves exactly like a BSD socket; there are ports in listening state and connected ports to send/receive datagrams).
An understandble implementation, since in many ways LPC ports are like Sockets, but the internal semantics are quite different (although to the outside you have your basic listen/accept/connect commands).
That is, of course, not the way NT LPC works, but at the time David wrote the code, we knew almost nothing about LPC, but the incomplete information Prasad Dabak wrote in "Undocumented Windows NT" (1999).
Yes, back then most Windows Internals were very well kept. Dabak was the first to go into LPC territory, although unfortunately most of that info is now really outdated (such as Quick LPC).
That code was good enough to start with user mode subsystem servers, but till now it has prevented us from testing real system components like the SM, CSR and friends etc. for compatibility in ROS.
Unforunately, most of our LPC-dependent components are really broken because they depend on the not-so-correct LPC implementation.
This is what I have done for the last three years. What I got from the researching is some better understanding of how LPC works, but not a complete picture. I am glad to see you are working or planning to work on LPC, because there is at least one API that is incompatible with NT's, and the general semantics of the path connect->request->reply is wrong.
Also checking with some PDFs, some of our checks are incorrect, and a lot of our stuff is just hard-coded. It took me a day to figure out where the max data/message sizes come from, but I think it was worth it, because that was just a small step in understanding the bigger picture.
In short, it is the the listening port that receives the messages from the clients even after the connection happens and not the server side connected port). This could be deducted by studing more carefully the sample code provided by Dabak, but I found it indipendently with some simple c/s test code.
One of the first functions I re-coded was LPC Port closing, and I noticed this fact because we were doing stuff almost backwards.
I told it to David, Eric and others and they agreed that should be correct. I will share what I know and I can send my unfinished code if it may be of interest (not that I am that a skilled coder, but you could just tour it for fun).
Anytime, just send it to my personal e-mail and we can collaborate.
Recalling what David Welch told me in many occasions, NT LPC has many flaws, many are security related (this led Microsoft to introduce secure ports with NT 5.0, mostly to armour the LSA).
One of my goals is to introduce support for secure ports. The problem with LPC is that it was "Security through obscurity". Once people found out how it worked, problems started appearing.
The overall problem with the real LPC is that it looks like really old, not well understood in Microsoft and almost untouched for years (many unfixed LPC bugs still plague Windows).
It was recently re-touched with the delivery of XP. It now uses Paged Lookaside Lists instead of Zones, and a lot of buffer checks have been added.
You could even imagine it was part of the Prism project at DEC, in the '80s, perhaps with the name "Channel" for what we now know as "port" (undocumented and unimplemented APIs existed in NT 3.1, NT 3.5, NT 3.51, NT 4.0 and NT 5.0, but disappeared in NT 5.1 and NT 5.2).
From what I was told, Channels were supposed to be extra high-speed LPCs for use in Win32.
It seems that only lately (1999+) Microsoft forced the NT kernel team to face the old LPC code to improve it.
Yep, as I said above, XP has a lot of fixes.
Alex, what are your plans for rewriting the LPC code?
I'll email them privately.
Emanuele
Best regards, Alex Ionescu
Alex Ionescu wrote:
That code was good enough to start with user mode subsystem servers, but till now it has prevented us from testing real system components like the SM, CSR and friends etc. for compatibility in ROS. Unforunately, most of our LPC-dependent components are really broken because they depend on the not-so-correct LPC implementation.
It probably may be fixed easily by peeking up messages from the named port's queue and not from the connected post's queue. That is the major difference.
Also checking with some PDFs, some of our checks are incorrect, and a lot of our stuff is just hard-coded. It took me a day to figure out where the max data/message sizes come from, but I think it was worth it, because that was just a small step in understanding the bigger picture.
Where do they come from? It is still a mistery for me!
Anytime, just send it to my personal e-mail and we can collaborate.
Done.
About LPC flaws - This one is a good gallery by Razor: http://www.bindview.com/Support/RAZOR/Utilities/Windows/LPCAdvisory.cfm
Emanuele
Emanuele Aliberti wrote:
Alex Ionescu wrote:
That code was good enough to start with user mode subsystem servers, but till now it has prevented us from testing real system components like the SM, CSR and friends etc. for compatibility in ROS. Unforunately, most of our LPC-dependent components are really broken because they depend on the not-so-correct LPC implementation.
It probably may be fixed easily by peeking up messages from the named port's queue and not from the connected post's queue. That is the major difference.
Also checking with some PDFs, some of our checks are incorrect, and a lot of our stuff is just hard-coded. It took me a day to figure out where the max data/message sizes come from, but I think it was worth it, because that was just a small step in understanding the bigger picture.
Where do they come from? It is still a mistery for me!
/* Allocate the LPC Lookaside List */ LpcpMaxMessageSize = sizeof(LPCP_MESSAGE) + sizeof(LPCP_CONNECTION_MESSAGE) + PORT_MAXIMUM_MESSAGE_LENGTH;
Where #define PORT_MAXIMUM_MESSAGE_LENGTH 256.
Anytime, just send it to my personal e-mail and we can collaborate.
Done.
I'm taking a look at it.
About LPC flaws - This one is a good gallery by Razor: http://www.bindview.com/Support/RAZOR/Utilities/Windows/LPCAdvisory.cfm
Emanuele
Best regards, Alex Ionescu
Ey, rather interesting all this. Leads directly to questions: Why is it needed to maintain compatibility with NT. Either API and Semantics are used just and only in ROS. OK, some never seen sub systems exist, but will they ever run on ROS or wil new ROS-ss ever run on NT?
Emanuele Aliberti wrote:
Hi Alex, I hope you are fine now, after the party and a restful night... ;-)
I'm currently working/researching on the LPC implementation and I'm about to soon start reworking some of our code. Just to avoid any duplication, are you also working on such a thing, or simply exploring?
David Welch wrote our initial LPC code to resemble the BSD sockets general design (an LPC port behaves exactly like a BSD socket; there are ports in listening state and connected ports to send/receive datagrams). That is, of course, not the way NT LPC works, but at the time David wrote the code, we knew almost nothing about LPC, but the incomplete information Prasad Dabak wrote in "Undocumented Windows NT" (1999). That code was good enough to start with user mode subsystem servers, but till now it has prevented us from testing real system components like the SM, CSR and friends etc. for compatibility in ROS. This is what I have done for the last three years. What I got from the researching is some better understanding of how LPC works, but not a complete picture. I am glad to see you are working or planning to work on LPC, because there is at least one API that is incompatible with NT's, and the general semantics of the path connect->request->reply is wrong. In short, it is the the listening port that receives the messages from the clients even after the connection happens and not the server side connected port). This could be deducted by studing more carefully the sample code provided by Dabak, but I found it indipendently with some simple c/s test code. I told it to David, Eric and others and they agreed that should be correct. I will share what I know and I can send my unfinished code if it may be of interest (not that I am that a skilled coder, but you could just tour it for fun). Recalling what David Welch told me in many occasions, NT LPC has many flaws, many are security related (this led Microsoft to introduce secure ports with NT 5.0, mostly to armour the LSA). The overall problem with the real LPC is that it looks like really old, not well understood in Microsoft and almost untouched for years (many unfixed LPC bugs still plague Windows). You could even imagine it was part of the Prism project at DEC, in the '80s, perhaps with the name "Channel" for what we now know as "port" (undocumented and unimplemented APIs existed in NT 3.1, NT 3.5, NT 3.51, NT 4.0 and NT 5.0, but disappeared in NT 5.1 and NT 5.2). It seems that only lately (1999+) Microsoft forced the NT kernel team to face the old LPC code to improve it. Alex, what are your plans for rewriting the LPC code?
Emanuele _______________________________________________ Ros-dev mailing list Ros-dev@reactos.com http://reactos.com:8080/mailman/listinfo/ros-dev
Robert Köpferl wrote:
Ey, rather interesting all this. Leads directly to questions: Why is it needed to maintain compatibility with NT. Either API and Semantics are used just and only in ROS. OK, some never seen sub systems exist, but will they ever run on ROS or wil new ROS-ss ever run on NT?
1) The SRM, in the executive, uses LPC to talk to the LSA, which runs in user mode. Therefore the security model in NT is LPC dependent. You could find an alternative IPC mechanism, but you won't bet some subtle behaviour does not depend on LPC.
2) RPC is totally based on LPC, if the remote process is located on the same node of the calling process. And OLE/COM is based on RPC. Again some subtle behaviours may depend on LPC.
3) A side effect of having the LPC APIs and semantics compatible with NT's is that subsystems developed for ROS will run in Windows NT, Windows 2000, Windows XP, and in Windows Server 2003.
4) Another advantage is that you can pick up the smss.exe from an XP box and try running it under ROS. Or the csrss.exe, which is the Win32 emulation process: there are some subtle details of the Win32 APIs that can not be explained it you don't go down to the LPC level. As LPC is still not fully documented, it may help understanding its internals, just like running real Windows programs helps Wine coders understand where their Win32 APIs implementation will need tuning.
5) Having a compatible LPC implementation could even help MS (OK, stop bragging!)
Emanuele
-
Alex Ionescu -
Emanuele Aliberti -
Robert Köpferl