And so this year's Summer of Code is over. I am moderately optimistic about the outcome, but it could have gone much better...
What's there: * the porting! I gutted rdesktop and isolated the smallest subset possible required for the protocol and nothing else. Then I made it re-entrant, as rdesktop was affected by the "accumulate and fire" anti-pattern, setting function arguments in global variables and preventing multiple instances of the library from running (a no-no for an ActiveX control, and for any library in general. Then, again, rdesktop was no library in the first place) * the porting, part 2. I implemented all the "orders" supported by the RDP protocol in terms of Win32 GDI, meaning the display is fully functional (barring the occasional artifact) * the porting, part 3. I implemented mouse/keyboard input. You can interact with your session. That's it. Basically, the RDP client just... works. That's it * the ActiveX control itself, in its glorious hundreds of methods and properties. By far the hardest part of the project. Friends don't let friends code ActiveX controls without ATL!
What's not there: * the effort in learning as quickly as possible the most fucked up component model in the history of computing had by far the largest impact on the project, meaning in practice I was stopped halfway through integrating the *actual* code with the extensive ActiveX dressing. In other words, the code currently in the repository is non-functional. Go back a couple of revisions to skip the ugly ActiveX meatworks and see the working proof-of-concept. My biggest mistake was assuming I could tame the ActiveX beast only armed with a stick * some useless ActiveX glue that does not pertain to the kind of control the RDP client is, but has to be there nonetheless * the redirectors were not ported, and no support yet for virtual channel plug-in DLLs. Not a giant effort, though * the mstsc.exe shell
Miscellaneous issues: dependency on OpenSSL. This will be hard to eradicate. CryptoAPI isn't an option, beacuse it was designed primarily for PKI, not random arbitrary encryption protocols. In particular, the following limitations make it unfit for use in the RDP protocol: * each provider only supports a single size for RC4 keys, either 40 or 128 bits. Not a major obstacle, but it gets worse * it cannot import plaintext RC4 session keys, at least not without a convoluted workaround (see http://support.microsoft.com/kb/228786/en-us). Still not major, but a little bit worse * it cannot perform RSA encryption without padding, required for the session key exchange protocol. CryptoAPI will always pad the cipher with PKCS #1 padding, limiting the plaintext to 11 bytes less than the key size. No workaround for this Best solution: reimplement the internal cryptography library used by Microsoft (the library exported as SystemFunctionXXX from advapi32.dll), which is what the original client uses
Afterword: I'll certainly continue working on this. It felt extremely irritating to be so close to a working implementation and just burning time on a huge glue layer probably larger than the implementation itself. Watch this space for future news on my progress! (not in the immediate future though. I'm still recovering from the trauma of hand-coding an ActiveX control)