New subject: proper nt-style authentication (reactos, wine, samba tng)

2 Sep 2005


      [please note: due to its cross-discipline and cross-project nature,
 this message is going out to SEVERAL related project mailing lists.
 when replying, please take EXTRA caution not least because some of
 these lists are subscriber-only.  also, please note: i _would_
 post this also to the samba mailing lists but due to the fascist
 censorship in place since the 15th dec 2004, i am unable to do so.
 this is their decision and it is their loss.  i am not asking you
 to respect that decision i am simply making you aware of it.]
hi,
out the woodwork i pop - not necessarily ready to chew anything because
i know just how much work's involved, but what i did want to do was
say "hi, i'm still here" and do a brain-dump of how authentication
ideally needs to be implemented in reactos.
at 2,600 words and 16k, this message is quite long and so i have
placed a copy at http://lkcl.net/software/reports/reactos-auth.txt,
just in case it doesn't make it past various mailing-list limits
(i'll find out in a couple of minutes... :)
it breaks down into a number of sections:
1) i describe the timescales and ways to cut them,
       along with some warnings and stuff.
2) amonst other stuff i outline some background as to
       why i am posting this to so many lists.
3) i outline a project plan and the dependencies
       and "optional" steps,
4) i describe a recommended implementation
       roadmap starting with the "minimum" requirements,
       and expand on some technical info and references
       i found, which would help with some of the
       "nice-to-have" steps.
so.  first.
please do not be scared by how much work is involved, and how much code
there is.  it all hinges one ONE function and that one function... you
are _not_ going to believe how much code that one function drags in,
kicking and screaming, behind it.
please also please i beg you DO NOT consider going "bleedin 'ellfire,
that's so much frakkin work we can't POSSIBLY do it the way you
describe, we MUST do it our own way, starting with what we know, love
and have already started tackling, and can't possibly back out of what
we've already done".
should you choose to exercise the "NIH" option, i frakkin
guarantee you that you will waste about five to eight years
of your (collective) lives reinventing the wheel.
"The Plan" outlined here will shave that down to about 12 to 18 months,
utilising some SIX HUNDRED THOUSAND lines of pre-existing code, and
later in this message i also outline a prioritisation of the necessary
work to "cut down" the time to maybe about ...  mmm... three or so 
months, by leaving out some of the "nice-to-have" stuff.  actually...
_all_ of the nice-to-have stuff :)
that will at least "GetItWorking(tm)" and the rest of the bits
can be considered at leisure once people go "god that's awful,
we can't possibly leave it like that" and hopefully hopefully
things will actually progress from there.
remember - please: this stuff is sufficiently complicated such that
you really can't afford the niceties such as "It Must Be Perfect (tm)"
before it can be accepted.  i've seen that shit before, and it's
a baad mxxxxxxxer path to go down, especially with such complex
and heavily interdependant reverse-engineering projects as reactos,
wine, samba and freedce.
anyway.
fyi - before i begin, i should mention a couple of things:
1) this message also goes out the the apache devel mailing
list (specifically the APR one) because the NT style
authentication thing is the sort of thing that really _should_
be in a (special?) APR util library, along with "NT Named Pipes" -
see http://en.wikipedia.org/wiki/NamedPipes
one of the reasons why NT-style NamedPipes is _not_ in an APR util is
because it is believed that NT-style NamedPipes do not fit the
"least common denominator".  by having the infrastructure outlined
below, it is possible to move things "up one level" to the point
where unix _has_ that denominononmination.
also, if APR still has support for that god-awful program-running
program called Win95, it would, with not much extra effort, be
possible to port some reactos components to Win95 (!) such that
ThePlan outlined here would make Win95 have proper NT-style
authentication (!) now there's a horrible thought that will give
MS and free software people nightmares both: upgrading Win95 by
installing free software components at it.  muhahahahah ahem.
2) i have filled in a number of pages on wikipedia.org.  see
http://en.wikipedia.org/wiki/User:lkcl.  please take the time to
review these pages, PLEASE help me with my totally biased "POV"
(point-of-view) comments, by either editing the pages direct or by adding
comments on the "talk" pages as appropriate.  or alternatively dumping
me in the nearest pond if ever you meet me.
wikipedia is supposed to be encyclopedia-ic and some of my
comments are anything but that.
help!!!
3) please due to the quite broad distribution of this message across
multiple mailing lists, please do NOT ask stupid questions like
"what the hell do we need to use samba tng for, why can't we use samba
in reactos"? and "why is this guy bothering us with this insane stuff?"
and "how do i use killfiles?" or terry pratchett's - more specifically
the librarian's favourite - "where has all the pie and custard gone?"
please do your research: see if nothing else
http://en.wikipedia.org/wiki/Samba_TNG_software and then start googling.
anyway.
onward.
roughly in order of dependence, with "nice-to-have" status added as
well:
1) port FreeDCE to Win32 - more specifically add in autoconf support for an
alternative threading model - the one in reactos.  no, this is not a joke:
MSRPC is a critical part of NT infrastructure it's just that very few people
are actually aware of this (including the US DoJ and the EU commission...)
see http://www.google.co.uk/search?q=lkcl+wine+freedce for discussion
links as to why - it's a long story and it took me several dozen
messages spanning over many days to outline it in sufficient detail
for it to be understood (mostly me getting things straight in my
head...)
status: "nice to have" but you will soon wish that you _did_ have it!!!
there is also an opportunity to support Wine, here: essentially
it's the same job, for DCOM, for proper "authentication"
purposes in Wine just like in ReactOS... it's a long story.
2) remove dependence on "unix" security model from Samba TNG's services
starting with samrd (perhaps by finishing off samrtdbd?)
status: "essential".
sub-projects:
a) in Samba TNG, rework winbindd such that it is a REQUIRED service,
   and it has "modes" that, instead of "inventing" links between
   unix uids and nt sids, it reads smbpasswd and correlates unix uids
   with nt sids _just_ like is done now - but hard-coded through
   the awful code that has had to dieee since... about two weeks after
   i wrote it, back in 1999.
this one's a long story but it's not actually required for
   reactos but _might_ be required for wine...
status: "pretty much essential" but not for reactos but for making life
easy to share tng development between reactos and unix.
b) in Samba TNG, make the authentication code (smb-side and rpc server
   back-end-wise) contact winbindd BUT ONLY for resolving unix uids and
   unix gids - which is actually a VERY small task involving a few
   hundred lines of code.
this goes back to the "SURS" stuff i wrote up in 1999 - but instead
   of a library it hands off the responsibility for sid<->uid+gid
   lookups to winbindd.
it _may_ also be essential for Wine to operate correctly
   with authentication (which is a "stub" at the moment just
   like it is in reactos), to interoperate with other programs,
   to cache login credentials in order to do smb client-side
   operations, that sort of thing, under which circumstances,
   it will be necessary for Wine to have the same sort of thing.
it's no big deal :)
status: same as for 2a).
c) in Samba TNG, abandon all use of hard-coded MSRPC stuff and utilise
   FreeDCE instead, and do a rewrite of all services, one by one
   (and because samba tng is modular, this _can_ be done one-by-one,
    checking MSRPC interoperability along the way between the "old"
    services, the "new" services, and also with NT itself).
in and of itself, this critically depends on making
   DCEThreads reliable, though, or adding support (finally!) for
   POSIX threads.
status: "non-essential" but you would soon end up wishing that you had :)
3) complete LsaLogonUser, add LsaLogonUserEx, in NTOSKRNL.DLL, and
   friends.
my favourites.  the LSASS stuff.
these functions are quite simple: they are "redirectors" - a vector
   table of functions is required to be passed in, which includes things
   like "authenticate with my lovely service" and "free some memory".
there are many LSASS sub-services: one of them is Netware, one of
   them is MSV1_0.DLL, one of them is Kerberos.
there are others.
LSASS is based on _exactly_ the same technique that dlopen does e.g
   in libdvdcss, and in freedce's ntlmsspauth.so, and in... mmm...
   the freedce transport module infrastructure and _many_ other
   projects ...  except of course it's NT-based table of
   higher-order-functions not unix "dlopen()" ones.
i'll stop trying to teach people to suck eggs, now *embarrassed* :)
status: "essential".
sub-projects:
a) write an MSV1_0.DLL which registers with the LSASS service.
   this will utilise MSRPC functions that call into the Samba TNG
   "NETLOGON" service.
there already exists "basic" functions inside Samba TNG that...
   well... it's a bit messy, but they work.
the key function to be calling is cli_nt_login_interactive,
   and you will notice _very_ quickly from the arguments hey, some
   of those look... kinda... familiar!!
see http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/rpc_client/cli_n...
status: "essential."
the "porting" bit of this code to FreeDCE i would classify as
"non-essential" but again, you will soon wish that you _were_ using
FreeDCE.
that's basically it: there are other sub-projects such as
turning the "Registry" code in ReactOS (or Wine) into an MSRPC
service (yes, i did say pretty much everything that's anything
critical in NT is an MSRPC service, didn't i? :) but they aren't
"essential".
i say basically it, but that ONE stupid function, cli_nt_login_interactive,
drags in quite literally HALF A MILLION lines of code - 250,000 lines of
it in FreeDCE, alone (which, like i said, could possibly be avoided but
the hard-coded MSRPC stuff in samba tng is sufficiently awkward to make
a rewrite utilising FreeDCE very attractive).
what i would suggest is the following:
1) write an MSV1_0.DLL "test stub" in combination with completing the
LSASS functions, which supports a username "test", domain of "test" and
password of "test".
code up a hard-coded "blob" - a NET_USER_INFO_3 structure - that contains
the response / essential information of groups, gids, sids, user session
key etc.
see http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/include/rpc_netl...
(note: i _did_ say it would be nice to utilise FreeDCE didn't i?  well,
the NET_USER_INFO_3 structure in that header file is a "messy" version that
i recreated from off-the-wire back in about ... mmmm... 1997.  there
_do_ exist IDL-generated versions of this data structure, thanks to
matthew chapman - NETLOGON.idl for example).
once that works...
2) write a lovely insecure method of "outsourcing" the username,
domain and password to an external server - Samba TNG - which performs
the authentication on your behalf and gets back "real" data.
this could be done simply with a TCP connection, throw the data
in-the-clear over to a simple temporary shim service blah blah,
bob's your uncle.
3) port samba tng's netlogond, samrd and lsarpcd to ReactOS.
this is quite straightforward: about the only really essential
   "missing" bit is to tie the services in to "NT Named Pipes" rather
   than unix domain sockets.
_but_ - they only need to be plugged in to a back-end transport
   which all services - any of samba tng's MSRPC services - would
   use.  so there's a key bit of work needed - probably under
   400 lines of code - and the rest should fall into place.
this is where it would be SO much easier to be utilising FreeDCE.
all that's needed would be to write a FreeDCE "transport"
   plugin for ReactOS - ncacn_np - and then you'd be DONE.
it's a long story...
hey, has someone implemented TransactNamedPipe() and CreateNamedPipe()
   in ReactOS?
if not, it's quite straightforward to do, and it involves dropping
   opaque data blobs onto smbd (again, smbd ported to reactos...)
i _did_ say it's a long story, didn't i? :)
4) finally: track down how "LsaLogonUserEx" works.  LsaLogonUser
utilises cleartext passwords.  LsaLogonUserEx i BELIEVE utilises NT and
LM password hashes, or some brain-dead encrypted variant thereof.
once 1) is completed, then TA-DAAAAA!!! lib/secur/lsa.c's "LsaLogonUser"
function actually gets REAL DATA!
okay.
notes:
1) regarding FreeDCE.  see http://en.wikipedia.org/wiki/FreeDCE
freedce is an interoperable version of MSRPC that is derived from
EXACTLY the same source code (DCE 1.0 reference implementation) that
is present in Windows NT.
DCE/RPC was co-opted / adopted / borg-ified by microsoft to form
the basis of NT domains because paul leach, a co-founder of
apollo computers and originator of NCS which became DCE/RPC,
ended up working for microsoft back at the time when dave
cutler was doing the original NT 3.1.
using FreeDCE is non-essential for small projects of less than about
8,000 lines of code.  Samba TNG's MSRPC code comprises OVER ONE HUNDRED 
THOUSAND lines of code.
that i carried on hand-crafting MSRPC packets for three years simply
demonstrates quite how bloody stupid i am.
that you - the wine team - continue to reinvent an non-interoperable
version of MSRPC, for binary-level "DCOM" interoperabiltiy ONLY,
demonstrates quite how just as bloody stupid you are being.  that _can_
be taken as a compliment, as i genuinely i mean it with the greatest of
respect.
2) regarding a strategy to "minimise" the amount of time needed to
   "Get Things Working".
a) utilise samba tng as-it-is, porting it as-is to ReactOS (mingw32)
b) utilise cli_nt_login_interactive() as-it-is.
c) complete MSV1_0.DLL.
this will be SUFFICIENT and would only take a few months.  enhancements
come later, roughly in this order:
a) complete a port of FreeDCE to ReactOS, by beating DCEthreads to
   death and replacing it with Win32 (NT) threads - #ifdef style,
   autoconf style.
b) locate NETLOGON.idl, samr.idl and lsarpc.idl from matthew chapman's
   stuff, or from "todd sabin's" http://razor.bindview.com, or from
   the samba web site.
actually... see: http://www.bindview.com/services/razor/utilities/
compile up CLIENT-SIDE ONLY, using dceidl, and create a
   library for use inside MSV1_0.DLL.
i know todd developed interoperable versions of these idl files,
   because he was using them, compiled with MIDL.EXE, to do security
   tests against NT 4.0.  remember: these IDL files are what microsoft
   DOESN'T want anyone to have, and there are very good _legitimate_
   reasons for it: they are "behind-the-scenes" APIs which, if some
   idiot inside microsoft went and wrote a tool which became publicly
   used and relied on, they would be stuffed: they have ENOUGH apis
   which date back 15 years and they don't need any more.  that
   having been said, i hate the fact that they are forcing people
   to pay $50k up-front and then $100 per-client for frakkin
   _information_.  bastards.
c) replace cli_nt_login_interactive() with a function that utilises
   the above library that utilises netlogon, samr and lsarpc client-side
   stuff from b) above.
basically, what it boils down to is that the functions inside
   cli_nt_login_interactive i wrote BY HAND after examining MSRPC
   traffic.  those functions neeeed too dieeee and they can easily
   be replaced one-by-one with the "proper" versions from doing
   "dceidl -client lsarpc.idl".  it takes a couple of seconds and
   you get a header file lsarpc.h and a lsarpc.o and you're DONE.
question.  why did i spend so long doing hand-marshalling of the
   nt domains code.  perhaps because i would not be able to give
   microsoft absolute hell for three frakkin years?
d) consider porting, one-by-one, the Samba TNG services lsarpcd,
   netlogond and samrd (or better samrtdbd) to FreeDCE runtime
   infrastructure.
this is NOT essential but it is very worthwhile.  every project
   needs to do at least two throw-aways and with the samba tng
   code, there has been approximately one and a half throwaways
   so far... time for a major one, ESPECIALLY in light of the reactos
   project.
FINALLY:
e) consider writing that winregd service but actually picking up REAL
   registry hive files.  someone did write a "reader" of registry files,
   i do not recall who it was.
[note: YESSSS!!! yeeeeeehawww, todd i could KISS you!!]
http://www.bindview.com/Services/RAZOR/Utilities/Unix_Linux/ntreg_readme.cfm
todd wrote a registry-hive-reader as a linux filesystem driver!!!!!
todd, you are bloody mad.  *smooch*.
f) consider writing a samr service with the FreeDCE runtime
   that utilises "registry" functions RegXXXX - _properly_.
anyway, if you got this far: congratulations and welcome to a brief
nightmare history of the development behind NT Domains Authentication.
all of the infrastructure above _already_ exists - in Samba TNG.
it's just...  it works.  it's a bit creaky at the edges, but it works.
how about it?
l.
p.s. it almost goes without saying, but i will hint at it once again:
you will not find the samba team's goals and strategic direction to be
compatible with reactos.  as a team, they lack the insight, vision and
project management skills to be able to cope with such inter-dependant
insane gibberish.  this i say with much sadness because they are highly
respected individuals in the free software community,  and yet they are
taking the samba project in very different - and ultimately
time-consuming - directions from what is really really needed.  due
to pride they cannot admit this in public but they have been known
to admit it in private.  much much disappointment and sadness and there's
absolutely nothing i can do about it.
-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--