Kernel-mode COM required for Kernel Streaming (ks.sys)

[Andrew "Silver Blade" Greenwood]

This is the most ugliest beast I've come across.

Trying to implement parts of the Kernel Streaming (ks) stuff. I decided 
to start with portcls.sys.

So far, it appears ks.h (the Kernel Streaming main header) isn't present 
in our include files (or w32api) and the official MS DDK version is 
100KB + so that's a bit of a knock-out already.

Anyway I had a poke around and stubbed a lot of the main portcls.sys 
functions. But I have no idea what I'm doing right now so it's pointless 
me trying.

It appears to be built on COM - the MSDN documents describe "Kernel-mode 
COM" in the Audio and Video section.

Obviously the standard COM calls can't be used in kernel-mode. And I 
found in kcom.h some declarations for functions such as KoCreateInstance 
(kernel mode version of CoCreateInstance?) yet I cannot find any 
references to it on Google.

The sample drivers are written in C++, and appear to rely on kcom.h's 
C++ classes to get things done.

There's just not a lot I can do as the foundation isn't even there for 
me to start implementing Kernel Streaming.

And I have *no* idea how to use COM. So far it looks really really ugly.

Can someone give me a hand?

-Andrew

[KJKHyperion]

Andrew "Silver Blade" Greenwood wrote:

>   And I have *no* idea how to use COM. So far it looks
> really really ugly. 
don't let COM scare you, it's just pretty dressing around arrays of 
function pointers. Quick crash course on COM...

== Co-classes and interfaces ==
An interface is a list of methods supported by an object, easy as that. 
The binary format to embed an interface in an object is very simple: an 
interface is a pointer to a so-called virtual table (v-table), which in 
turn is simply an array (actually a structure, because they 
realistically have different prototypes) of function pointers. Here's 
the most famous COM interface, IUnknown, in C syntax:

struct IUnknown
{
    struct IUnknown_Vtbl * lpVtbl;
};

struct IUnknown_Vtbl
{
    HRESULT (* QueryInterface)(struct IUnknown * lpThis, REFIID iid, 
void ** ppvObject);
    ULONG (* AddRef)(struct IUnknown * lpThis);
    ULONG (* Release)(struct IUnknown * lpThis);
};

All interfaces are identified uniquely by an UUID, which in this context 
is referred to as an IID

A co-class is an implementation (literally, a bunch of code) of a 
certain set of interfaces. Co-classes are identified uniquely, too, and 
their identifiers are known as CLSIDs

== How to implement co-classes in C ==
First of all, let's remind ourselves that an "object" is really just 
allocated data pointed to by a typed pointer (or even an opaque 
non-pointer, like a handle). To make a practical example, I'll show you 
a way to implement ISequentialStream, probably the simplest interface 
that actually serves a purpose on its own. First, we declare the co-class:

struct MyStream
{
    // COM-visible
    union
    {
       IUnknown IUnknown;
       ISequentialStream ISequentialStream;
    };

    // private
    LONG refCount;
    HANDLE hFile;
};

I could have used C-style struct inheritance, which is cleaner than 
using an union but hides too many details. Why the union? It will be 
clearer when we'll implement IUnknown::QueryInterface. For know, it 
probably helps to know that ISequentialStream contains all the function 
pointers that IUnknown does (all COM interfaces are required to), plus 
two more on its own, so the use of an union is somewhat safe

How are objects of type struct MyStream created? COM doesn't really 
care, and it doesn't specify an unified way to allocate, instantiate and 
initialize an object. In many cases ("push" model) you can simply pass 
around interfaces, and nobody will ask where do they come from. In the 
frequent cases in which instead you have to *ask* for an object ("pull" 
model), a class factory is used. A class factory (in general) is an 
object that does nothing else than creating objects of a certain class. 
The general outline is this:
 - client needs an implementation of ISomething. The configuration says 
to use the implementation provided by the co-class with id {such-and-such}
 - client calls COM asking for an instance of co-class {such-and-such} 
and a pointer to its ISomething
 - the COM database says {such-and-such}is implemented in blah.dll. COM 
loads blah.dll, and asks it (doesn't matter how for now, it's documented 
anyway) to create a class factory for {such-and-such} (the class 
factory, in turn, is an implementation of IClassFactory)
 - COM tells the class factory to create an object of type {such-and-such}
 - with the object in hand, COM asks the object for its ISomething
 - if everything is ok, COM returns the ISomething pointer to the client
(note: "instance of co-class {such-and-such}" and "object of type 
{such-and-such}" are just two ways of saying the same thing)

For the sake of simplicity, let's assume a "push" model, where we create 
the object ourselves in an unspecified way and pass pointers to its 
interfaces around. Here's how we allocate, instantiate and initialize an 
object of type struct MyStream:

// allocation
struct MyStream * obj = malloc(sizeof(struct MyStream));

// instantiation
obj->ISequentialStream.lpVtbl = MyStream_ISequentialStream_Vtbl;
obj->refCount = 1;

// initialization
obj->hFile = CreateFile(...);

As for MyStream_ISequentialStream_Vtbl, it sounds menacing but it's 
quite easy:

static const ISequentialStream MyStream_ISequentialStream_Vtbl =
{
    // IUnknown methods
    MyStream_QueryInterface,
    MyStream_AddRef,
    MyStream_Release,

    // ISequentialStream methods
    MyStream_Read,
    MyStream_Write
};

QueryInterface, AddRef and Release are merely tedious, but they are 
implemented with boilerplate code. Here's a quite typical implementation:

// This is how our clients can ask if we implement an interface
HRESULT MyStream_QueryInterface(struct ISequentialStream * lpThis, 
REFIID iid, void ** ppvObject)
{
    // access our private data. Note that thanks to CONTAINING_RECORD 
interfaces could be anywhere
    // in an object, not necessarily at the beginning
    struct MyStream * This = CONTAINING_RECORD(lpThis, struct MyStream, 
ISequentialStream);

    *ppvObject = NULL;

    if(IsEqualIID(iid, &IID_IUnknown))
        *ppvObject = &This->IUnknown;
    else if(IsEqualIID(iid, &IID_ISequentialStream))
        *ppvObject = &This->ISequentialStream;
    else
        return E_NOINTERFACE;

    return S_OK:
}

// This is how we're told the object is in use
ULONG AddRef(struct ISequentialStream * lpThis)
{
    struct MyStream * This = CONTAINING_RECORD(lpThis, struct MyStream, 
ISequentialStream);
    return InterlockedIncrement(&This->refCount);
}

// And this is how we're told our services are no longer required
ULONG Release(struct ISequentialStream * lpThis)
{
    struct MyStream * This = CONTAINING_RECORD(lpThis, struct MyStream, 
ISequentialStream);
    ULONG ul = InterlockedDecrement(&This->refCount);

    // no pointers to this object exist anywhere anymore: we can destroy 
and free it
    if(ul == 0)
    {
        // destroy
        CloseHandle(This->hFile);

        // free
        free(This);

        // decrement the global count of objects
        InterlockedDecrement(&numObjects);
    }

    return ul;
}

No, seriously, just that. That's it. Note we aren't required to actually 
keep a reference count, since nothing in COM forbids to implement 
objects as singletons (i.e. a single instance for all clients), and 
indeed in many common cases we can simplify things bynot using 
malloc/free and instead passing around every time the same 
statically-allocated interface.

(as for the other functions, well, they don't really matter. 
MyStream_Read calls ReadFile on This->hFile and MyStream_Write calls 
WriteFile)

And this is all you absolutely need to know about COM! if you need more, 
just ask

[Andrew "Silver Blade" Greenwood]

KJKHyperion wrote:

>   [A long and useful tutorial on the usage of COM]
>  
Thanks for that! I'll have a read of that later when I'm a bit more awake
;)

My main concern right now is support of COM in the kernel. It seems to 
me like a pretty evil idea (the concept of what it aims to achieve is 
good, but it's still darn ugly!)

I don't know how much of this ReactOS is capable of already. It seems 
the MS DDK has files such as "punknown.h" which define IUnknown, yet we 
don't have that. When I included the MS file, I got compiler errors 
about duplicate definitions of IUnknown.

It seems that I may need to create some additional framework for 
kernel-mode COM.

And I'm wondering, of course, how you work with these classes in 
kernel-mode, where there's no CoCreateInstance (seems to be a 
KoCreateInstance but it's part of ks.sys and not documented?)

I got stuck with this when stubbing portcls.sys and got errors regarding 
parameters with types that weren't declared. I looked them up and found 
them to be COM objects with a seemingly endless structure!

But it seems a little more logical: PcCreatePort is a factory for IPort 
(and, presumably, "sub-classes"?)

Finally, the MS DDK has most of it's WDM samples in C++, with no 
indication of how the calls get translated into the Kernel Streaming 
functions. What's the deal here? Do I use C++ or is it not necessary?

-Andrew

[Maarten Bosma]

Andrew "Silver Blade" Greenwood wrote:

>   KJKHyperion wrote:
>
> >   [A long and useful tutorial on the usage of COM]
>  
I don't know which wiki it comes from (but I guess it come from one 
because of the ==Wiki
Headlines==). But it looks usefull so it would be nice if you could copy 
it to our one.

BTW: On the new homepage we will have an article posting system where 
things like that can
be posted.

Maarten Bosma

[KJKHyperion]

Maarten Bosma wrote:

> > >   [A long and useful tutorial on the usage of COM]
> >  
>  I don't know which wiki it comes from 
no wiki, I wrote it myself