Summer of Code, status report - Ros-dev

23 Aug 2006

And so this year's Summer of Code is over. I am moderately optimistic 
about the outcome, but it could have gone much better...

What's there:
 * the porting! I gutted rdesktop and isolated the smallest subset 
possible required for the protocol and nothing else. Then I made it 
re-entrant, as rdesktop was affected by the "accumulate and fire" 
anti-pattern, setting function arguments in global variables and 
preventing multiple instances of the library from running (a no-no for 
an ActiveX control, and for any library in general. Then, again, 
rdesktop was no library in the first place)
 * the porting, part 2. I implemented all the "orders" supported by the 
RDP protocol in terms of Win32 GDI, meaning the display is fully 
functional (barring the occasional artifact)
 * the porting, part 3. I implemented mouse/keyboard input. You can 
interact with your session. That's it. Basically, the RDP client just... 
works. That's it
 * the ActiveX control itself, in its glorious hundreds of methods and 
properties. By far the hardest part of the project. Friends don't let 
friends code ActiveX controls without ATL!

What's not there:
 * the effort in learning as quickly as possible the most fucked up 
component model in the history of computing had by far the largest 
impact on the project, meaning in practice I was stopped halfway through 
integrating the *actual* code with the extensive ActiveX dressing. In 
other words, the code currently in the repository is non-functional. Go 
back a couple of revisions to skip the ugly ActiveX meatworks and see 
the working proof-of-concept. My biggest mistake was assuming I could 
tame the ActiveX beast only armed with a stick
 * some useless ActiveX glue that does not pertain to the kind of 
control the RDP client is, but has to be there nonetheless
 * the redirectors were not ported, and no support yet for virtual 
channel plug-in DLLs. Not a giant effort, though
 * the mstsc.exe shell

Miscellaneous issues: dependency on OpenSSL. This will be hard to 
eradicate. CryptoAPI isn't an option, beacuse it was designed primarily 
for PKI, not random arbitrary encryption protocols. In particular, the 
following limitations make it unfit for use in the RDP protocol:
 * each provider only supports a single size for RC4 keys, either 40 or 
128 bits. Not a major obstacle, but it gets worse
 * it cannot import plaintext RC4 session keys, at least not without a 
convoluted workaround (see 
<http://support.microsoft.com/kb/228786/en-us>). Still not major, but a 
little bit worse
 * it cannot perform RSA encryption without padding, required for the 
session key exchange protocol. CryptoAPI will always pad the cipher with 
PKCS #1 padding, limiting the plaintext to 11 bytes less than the key 
size. No workaround for this
Best solution: reimplement the internal cryptography library used by 
Microsoft (the library exported as SystemFunctionXXX from advapi32.dll), 
which is what the original client uses

Afterword: I'll certainly continue working on this. It felt extremely 
irritating to be so close to a working implementation and just burning 
time on a huge glue layer probably larger than the implementation 
itself. Watch this space for future news on my progress! (not in the 
immediate future though. I'm still recovering from the trauma of 
hand-coding an ActiveX control)