Re: [ros-svn] [ion] 15492: Nonpaged Pool Liberation Day: Allow PagedPool to be used earlier, allow fast mutex to be used earlier on debug builds. Allocate all Se stuff from PagedPool, set the right object types to use paged pool, allocate all strings from paged pool, allocate PE sections from paged pool, and a bunch of other things which should, imo, be in paged pool. If anyone has any contradicting proof, let me know...until then, enjoy ~4-6MB more NonPagedPool

[Hartmut Birr]

ion(a)svn.reactos.com wrote:

>  Nonpaged Pool Liberation Day: Allow PagedPool to be
> used earlier, allow fast mutex to be used earlier on debug builds. Allocate all Se stuff
> from PagedPool, set the right object types to use paged pool, allocate all strings from
> paged pool, allocate PE sections from paged pool, and a bunch of other things which
> should, imo, be in paged pool. If anyone has any contradicting proof, let me know...until
> then, enjoy ~4-6MB more NonPagedPool
>
>
>   
>  
Hi,

I don't like the exchange from NonPagedPool to PagePool at the current
state of ros. There are two reasons: 

A) The allocation of paged pool is very slowly compared with the non
paged pool allocation.

B) Since the change to the new object type structures, ros has a big
memory leakage in the memory (pool) allocation/deallocation of objects.
For the non paged pool, there exist some debug functions to find the
problem. The paged pool doesn't implement anything to find such problems.

- Hartmut

[Alex Ionescu]

Hartmut Birr wrote:

>  ion(a)svn.reactos.com wrote:
>
>   
>
> >  Nonpaged Pool Liberation Day: Allow PagedPool to
> > be used earlier, allow fast mutex to be used earlier on debug builds. Allocate all Se
> > stuff from PagedPool, set the right object types to use paged pool, allocate all strings
> > from paged pool, allocate PE sections from paged pool, and a bunch of other things which
> > should, imo, be in paged pool. If anyone has any contradicting proof, let me know...until
> > then, enjoy ~4-6MB more NonPagedPool
> >
> >
> >  
> >
> >     
> >  
> Hi,
>
> I don't like the exchange from NonPagedPool to PagePool at the current
> state of ros. There are two reasons: 
>
> A) The allocation of paged pool is very slowly compared with the non
> paged pool allocation.
>   
>  
I am aware it's a bit slower (it shouldn't be noticeable), but do we 
really need to store the Bootup SID, ACL and SDs in *nonpaged* memory? 
Since when is security a top-priority for speed? Same thing for strings. 
Windows doesn't even let you RtlCreate strings in non-paged...it's a 
hack we created because our paged pool memory didn't work early enough.

The only reasons those allocations were ever done in nonpaged pool is 
because it was impossible to do them in paged pool earlier, but I've 
fixed that.

>  B) Since the change to the new object type structures,
> ros has a big
> memory leakage in the memory (pool) allocation/deallocation of objects.
> For the non paged pool, there exist some debug functions to find the
> problem. The paged pool doesn't implement anything to find such problems.
>   
>  
I can try to quickly hack up a new debug function to dump the Paged Pool 
(doesn't one exist?).
I disagree with hacking the *whole tree* for *everyone* just so that 
some developers can do debugging. I still have them set to nonpaged 
locally, just like I have some stuff //#define NDEBUG locally to help me 
debug problems..but that doesn't mean that everyone should be penalized.

After my changes, I can now boot ROS with only 16MB of Physical Memory. 
I think it's more important then allocating a security ACL 0.0001 
seconds faster... if you think otherwise, I'm open to discussion.

>  - Hartmut
>   
>  
Best regards,
Alex Ionescu

[Alex Ionescu]

>  B) Since the change to the new object type structures,
> ros has a big
> memory leakage in the memory (pool) allocation/deallocation of objects.
> For the non paged pool, there exist some debug functions to find the
> problem. The paged pool doesn't implement anything to find such problems.
>
> - Hartmut
> _
>  
By the way:

1) Object Types are still allocaged in NP Pool. So are 99% of objects 
(except Sections...)
2) I've now implemented tagging for the Object Types so it should be 
easier to debug.

Best regards,
Alex Ionescu

[Hartmut Birr]

>   By the way:
>
>  1) Object Types are still allocaged in NP Pool. So are 99% of objects
>  (except Sections...)
>  2) I've now implemented tagging for the Object Types so it should be
>  easier to debug.
>  
I can see the following lines frome the non paged pool statistics at the
end of compiling ros on ros:

...
Tag 656c6946 (File) Blocks 3172 Total Size 549816 Average Size 173
...
Tag 6e657645 (Even) Blocks 12938 Total Size 793144 Average Size 61
...
Tag 6d4e624f (ObNm) Blocks 28070 Total Size 747432 Average Size 26
Tag 4943624f (ObCI) Blocks 26378 Total Size 1405552 Average Size 53

Now I know that something is wrong with events. But I have no feeling
which are the other 13,000 objects.

- Hartmut

[Alex Ionescu]

Hartmut Birr wrote:

> >  By the way:
> >
> > 1) Object Types are still allocaged in NP Pool. So are 99% of objects
> > (except Sections...)
> > 2) I've now implemented tagging for the Object Types so it should be
> > easier to debug.
> >
> >     
> >  
> I can see the following lines frome the non paged pool statistics at the
> end of compiling ros on ros:
>
> ...
> Tag 656c6946 (File) Blocks 3172 Total Size 549816 Average Size 173
> ...
> Tag 6e657645 (Even) Blocks 12938 Total Size 793144 Average Size 61
> ...
> Tag 6d4e624f (ObNm) Blocks 28070 Total Size 747432 Average Size 26
>   
>  
Name Information.

>  Tag 4943624f (ObCI) Blocks 26378 Total Size 1405552
> Average Size 53
>   
>  
Create Information.

>  Now I know that something is wrong with events. But I
> have no feeling
> which are the other 13,000 objects.
>   
>  
Looks like the objects aren't being dereferenced properly. Autochk.exe 
is still alive after boot as well.

Best regards,
Alex Ionescu

>  - Hartmut
>
>
> _______________________________________________
> Ros-dev mailing list
> Ros-dev(a)reactos.com
> http://reactos.com:8080/mailman/listinfo/ros-dev
>
>   
>

[Alex Ionescu]

Alex Ionescu wrote:

>   Looks like the objects aren't being dereferenced
> properly. Autochk.exe 
>  is still alive after boot as well. 
I've some some testing:

1) Create ExEvent with Nt Function
2) Reference by handle
3) Print New NP Allocs + RefCount
4) Deref
5) Print RefCount
6) Close Handle
7) Print RefCount + Messages in ObpDeleteObject:

(ex/init.c:561) Creating event
******* Dumping non paging pool contents ******
Size 0x28 Tag 0x656e6f4e (None) Allocator 0x80048a45 => The 'hack' for 
creating the name. My next patch in my WC does this properly, I will 
probably commit it in the next 2-3 hours or so.
Size 0x50 Tag 0x6e657645 (Even) Allocator 0x80049982 => The actual Event
(ex/init.c:584) Refcount of New Event: 2 Header: 8039bd40 => After the 
referencing by handle
(ex/init.c:588) Refcount of New Event: 1 => After the ObDeRef
(ob/object.c:877) ObpDeleteObject(Header 8039bd58) => During Ntclose
(ob/object.c:899) Clearing name => The 0x28 block is freed.
(ob/object.c:925) Freeing header at :8039bd40 => The 0x50 block is freed
(ex/init.c:592) Refcount of New Event: cccccccc => After the NT Close. 
The header has been de-allocated

So no leaks on this path =(.

>  
>  Best regards,
>  Alex Ionescu

[Alex Ionescu]

Hartmut Birr wrote:

>  Now I know that something is wrong with events. But I
> have no feeling
> which are the other 13,000 objects.
>   
>  
Sections...try editing mm/section.c to be sure.

>  - Hartmut
>   
>  
Best regards,
Alex Ionescu

[Hartmut Birr]

Hartmut Birr wrote:

> >  By the way:
> >
> > 1) Object Types are still allocaged in NP Pool. So are 99% of objects
> > (except Sections...)
> > 2) I've now implemented tagging for the Object Types so it should be
> > easier to debug.
> >
> >     
> >  
> I can see the following lines frome the non paged pool statistics at the
> end of compiling ros on ros:
>
> ...
> Tag 656c6946 (File) Blocks 3172 Total Size 549816 Average Size 173
> ...
> Tag 6e657645 (Even) Blocks 12938 Total Size 793144 Average Size 61
> ...
> Tag 6d4e624f (ObNm) Blocks 28070 Total Size 747432 Average Size 26
> Tag 4943624f (ObCI) Blocks 26378 Total Size 1405552 Average Size 53
>
> Now I know that something is wrong with events. 
>  
One third of the events comes from the 'master' critical section from
ntdll. It is only one event per process and not all processes don't
remove this event.

- Hartmut