Re: [ros-svn] [hbirr] 15821: - Changed the calculation of the base address of an image section.

[Alex Ionescu]

[e-mail] wrote:
> Changed the calculation of the base address of an image section.
> Removed some unnecessary members from section object.
> Changed the allocation of a section object back to paged pool.
>
> Updated files:
> trunk/reactos/ntoskrnl/include/internal/mm.h
> trunk/reactos/ntoskrnl/mm/section.c
>
> Ros-svn mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-svn

Thanks Hartmut.

If you're up for it, here are the public NT structures used...maybe it 
won't be a bad idea if we use them if it's not too much code change:

lkd> dt _SECTION_OBJECT -b
   +0x000 StartingVa       : Ptr32
   +0x004 EndingVa         : Ptr32
   +0x008 Parent           : Ptr32
   +0x00c LeftChild        : Ptr32
   +0x010 RightChild       : Ptr32
   +0x014 Segment          : Ptr32
lkd> dt _SEGMENT_OBJECT
   +0x000 BaseAddress      : Ptr32 Void
   +0x004 TotalNumberOfPtes : Uint4B
   +0x008 SizeOfSegment    : _LARGE_INTEGER
   +0x010 NonExtendedPtes  : Uint4B
   +0x014 ImageCommitment  : Uint4B
   +0x018 ControlArea      : Ptr32 _CONTROL_AREA
   +0x01c Subsection       : Ptr32 _SUBSECTION
   +0x020 LargeControlArea : Ptr32 _LARGE_CONTROL_AREA
   +0x024 MmSectionFlags   : Ptr32 _MMSECTION_FLAGS
   +0x028 MmSubSectionFlags : Ptr32 _MMSUBSECTION_FLAGS
lkd> DT _CONTROL_AREA
   +0x000 Segment          : Ptr32 _SEGMENT
   +0x004 DereferenceList  : _LIST_ENTRY
   +0x00c NumberOfSectionReferences : Uint4B
   +0x010 NumberOfPfnReferences : Uint4B
   +0x014 NumberOfMappedViews : Uint4B
   +0x018 NumberOfSubsections : Uint2B
   +0x01a FlushInProgressCount : Uint2B
   +0x01c NumberOfUserReferences : Uint4B
   +0x020 u                : __unnamed
   +0x024 FilePointer      : Ptr32 _FILE_OBJECT
   +0x028 WaitingForDeletion : Ptr32 _EVENT_COUNTER
   +0x02c ModifiedWriteCount : Uint2B
   +0x02e NumberOfSystemCacheViews : Uint2B
lkd> dt _SEGMENT
   +0x000 ControlArea      : Ptr32 _CONTROL_AREA
   +0x004 TotalNumberOfPtes : Uint4B
   +0x008 NonExtendedPtes  : Uint4B
   +0x00c WritableUserReferences : Uint4B
   +0x010 SizeOfSegment    : Uint8B
   +0x018 SegmentPteTemplate : _MMPTE
   +0x01c NumberOfCommittedPages : Uint4B
   +0x020 ExtendInfo       : Ptr32 _MMEXTEND_INFO
   +0x024 SystemImageBase  : Ptr32 Void
   +0x028 BasedAddress     : Ptr32 Void
   +0x02c u1               : __unnamed
   +0x030 u2               : __unnamed
   +0x034 PrototypePte     : Ptr32 _MMPTE
   +0x038 ThePtes          : [1] _MMPTE
lkd> dt _SUBSECTION
   +0x000 ControlArea      : Ptr32 _CONTROL_AREA
   +0x004 u                : __unnamed
   +0x008 StartingSector   : Uint4B
   +0x00c NumberOfFullSectors : Uint4B
   +0x010 SubsectionBase   : Ptr32 _MMPTE
   +0x014 UnusedPtes       : Uint4B
   +0x018 PtesInSubsection : Uint4B
   +0x01c NextSubsection   : Ptr32 _SUBSECTION

I think it's important because many of the Section implementation 
details are documented in books (including driver-related books) and 
known to driver developers who might be using the structures in their 
code... I can name NT Insider, Windows Internals 4th Edition and Windows 
NT File System Internals as some examples...but anyways it's just an idea.

There's even a nice diagram in one of the books:

As a sidenote, do we have a PFN Database? It seems so much of our Mm is 
messy and old code that was written in a hurry to get ReactOS to work. I 
dislike the abusive use of macros and the difficult to understand code 
(imo).

Best regards,
Alex Ionescu

[Hartmut Birr]

Alex Ionescu wrote:
> If you're up for it, here are the public NT structures used...maybe it
> won't be a bad idea if we use them if it's not too much code change:
>
> I think it's important because many of the Section implementation
> details are documented in books (including driver-related books) and
> known to driver developers who might be using the structures in their
> code... I can name NT Insider, Windows Internals 4th Edition and
> Windows NT File System Internals as some examples...but anyways it's
> just an idea.

I think that the structures are not public. No driver can use this
structures. M$ can change the structures between service packs. IMO it
exsits no reason to use exactly the same structures. In the near future,
I will do some changes. It is necessary to split the segment structure
in one large 'page' table and some short structures for each segment. It
is nearly the same like the control area, the subsections and the
segment. It is also necessary to use only one memory area for a mapped
image section. I need this to implement the executing of low alignment
executables (like device drivers).
> There's even a nice diagram in one of the books:

I know this diagram because I own some of the books.
> As a sidenote, do we have a PFN Database?

We have a 'PFN' database. It is in freelist.c.

- Hartmut

[Alex Ionescu]

Hartmut Birr wrote:
> Alex Ionescu wrote:
> > If you're up for it, here are the public NT structures used...maybe it
> > won't be a bad idea if we use them if it's not too much code change:
> >
> > I think it's important because many of the Section implementation
> > details are documented in books (including driver-related books) and
> > known to driver developers who might be using the structures in their
> > code... I can name NT Insider, Windows Internals 4th Edition and
> > Windows NT File System Internals as some examples...but anyways it's
> > just an idea.
>
> I think that the structures are not public.

Where did I paste them from...?
> No driver can use this
> structures.

You live in a dream world. I can get many drivers use this. I will keep 
referring to one example of horrible kernel code: the NVIDIA driver 
reads a static, hard-coded pointer inside kernel32 (from kernel-mode!). 
If a well known company does this, how much do you want to bet other 
drivers certaintly poke around in structures. It's a complete illusion 
to think that every driver doesn't. NT Insider recently talked about a 
WHQLed driver that instantly ASSERTS on a debug build; driver quality is 
truly horrible. We have the option to support more drivers by imitating 
public structures; it's in our best interest to do so.
> M$ can change the structures between service packs.

Yes, and they can also rename "Windows" to "Alexows" and declare Paged 
Pool memory deprecated. The fact is they won't. MS takes great care to 
ensure that structures they know drivers are "illegaly" using don't 
change to the point that they will break the drivers.
> IMO it
> exsits no reason to use exactly the same structures.

Well I've just pointed them out. However, if it takes considerable 
effort to do so, then we can forget about it for now.
> In the near future,
> I will do some changes. It is necessary to split the segment structure
> in one large 'page' table and some short structures for each segment.

At least that will match the general implementation details.
> It
> is nearly the same like the control area, the subsections and the
> segment.

Yes..
> It is also necessary to use only one memory area for a mapped
> image section. I need this to implement the executing of low alignment
> executables (like device drivers).

Oh...so this is why they don't work! I'm glad to find this out! :)
> > There's even a nice diagram in one of the books:
>
> I know this diagram because I own some of the books.

So therefore the implementation details are public and as I pointed out 
the structures too. What's to stop a driver from using them?
> > As a sidenote, do we have a PFN Database?
>
> We have a 'PFN' database. It is in freelist.c.

Thanks...but I'll suppose it's nothing like the NT implementation :)
> Hartmut

Best regards,
Alex Ionescu
> Ros-dev mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-dev

[Robert Köpferl]

From a realistic point of view, i have to agree with Alex.

What can be done will be done.. That's the world.

Alex Ionescu wrote:
> Hartmut Birr wrote:
> > Alex Ionescu wrote:
> > > If you're up for it, here are the public NT structures used...maybe it
> > > won't be a bad idea if we use them if it's not too much code change:
> > >
> > > I think it's important because many of the Section implementation
> > > details are documented in books (including driver-related books) and
> > > known to driver developers who might be using the structures in their
> > > code... I can name NT Insider, Windows Internals 4th Edition and
> > > Windows NT File System Internals as some examples...but anyways it's
> > > just an idea.
> >
> > I think that the structures are not public.
>
> Where did I paste them from...?
> > No driver can use this
> > structures.
>
> You live in a dream world. I can get many drivers use this. I will keep 
> referring to one example of horrible kernel code: the NVIDIA driver 
> reads a static, hard-coded pointer inside kernel32 (from kernel-mode!). 
> If a well known company does this, how much do you want to bet other 
> drivers certaintly poke around in structures. It's a complete illusion 
> to think that every driver doesn't. NT Insider recently talked about a 
> WHQLed driver that instantly ASSERTS on a debug build; driver quality is 
> truly horrible. We have the option to support more drivers by imitating 
> public structures; it's in our best interest to do so.
> > M$ can change the structures between service packs.
>
> Yes, and they can also rename "Windows" to "Alexows" and declare Paged 
> Pool memory deprecated. The fact is they won't. MS takes great care to 
> ensure that structures they know drivers are "illegaly" using don't 
> change to the point that they will break the drivers.
> > IMO it
> > exsits no reason to use exactly the same structures.
>
> Well I've just pointed them out. However, if it takes considerable 
> effort to do so, then we can forget about it for now.
> > In the near future,
> > I will do some changes. It is necessary to split the segment structure
> > in one large 'page' table and some short structures for each segment.
>
> At least that will match the general implementation details.
> > It
> > is nearly the same like the control area, the subsections and the
> > segment.
>
> Yes..
> > It is also necessary to use only one memory area for a mapped
> > image section. I need this to implement the executing of low alignment
> > executables (like device drivers).
>
> Oh...so this is why they don't work! I'm glad to find this out! :)
> > > There's even a nice diagram in one of the books:
> >
> > I know this diagram because I own some of the books.
>
> So therefore the implementation details are public and as I pointed out 
> the structures too. What's to stop a driver from using them?
> > > As a sidenote, do we have a PFN Database?
> >
> > We have a 'PFN' database. It is in freelist.c.
>
> Thanks...but I'll suppose it's nothing like the NT implementation :)
> > Hartmut
>
> Best regards,
> Alex Ionescu
> > Ros-dev mailing list
> > [e-mail]
> > http://reactos.com:8080/mailman/listinfo/ros-dev
>
> Ros-dev mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-dev