Re: [ros-dev] DLL Base Addresses (For Marc)

[Aleksey Bragin]

On Dec 16, 2007, at 4:10 PM, Marc Piulachs wrote:

>   Hi Alex,
>
>  Let me see if I understood this correctly. The problem is not  
>  really on
>  manually choosing the base address, the problem is that module's
>  "baseaddress" attribute is optional, even when the module is of type
>  "win32dll", if this attribute is empty rbuild sets a common default  
>  base
>  address (0x10000000 for win32 dll's). 
No, the problem is exactly choosing the address: modules overlap with  
each other, and manually calculating baseaddresses often doesn't  
work. Alex proposed to automate this process and do a  
baseaddress.rbuild file generator.

There are not many dlls with baseaddress missing, and if it's  
missing, it should be added to those (in trunk).


>   AFAIK we are using the same base address that the
> equivalent  
>  component has
>  on windows so auto generating it using any sort of tool is not  
>  really an
>  option. 
Not really, in most cases (might be with a few exceptions), it's not  
possible to use exactly Windows' base address (because size of our  
modules and windows modules often differs). The problem is to  
distribute the dlls in the memory address space so that they don't  
overlap.


As for baseaddress in xml form - baseaddress is a module's property,  
it's not a standalone object which is part of the module. I don't see  
why it should be given a standalone tag (maybe I'm missing something).

WBR,
Aleksey Bragin.

[Marc Piulachs]

Hi,

>   No, the problem is exactly choosing the address:
> modules overlap with
>  each other, and manually calculating baseaddresses often doesn't
>  work. 
OK, I see the problem.

>   There are not many dlls with baseaddress missing, and
> if it's
>  missing, it should be added to those (in trunk). 
You are correct, only a few have missing preferred load address; I created a
report http://www.codexchange.net/rosdoc/dllbaseaddresses.htm to help to
correct them.

>   The problem is to
>  distribute the dlls in the memory address space so that they don't
>  overlap. 
As alex pointed there are two ways to accomplish it (I will add a third one)

A) create a new tool to investigate the build dll images , use it's file
size to distribute dlls in the memory space and generate baseaddress.xml .
run it from time to time.

	PROS: Dll's will have always a predictable base address,
performance: optimal. Integrated with rbuild.
	CONS: Dll's build with msvc will probably have different file size,
a valid calculated base address for a gcc build dll may become invalid when
build with msvc.

B) Make rbuild relocate base addresses on every build:

	PROS: It will work with images generated no matter what compiler
because addresses are dynamically calculated.
	CONS: Very inefficient, this process will be executed always. Even
when recompiling a single .c, Not integrated with rbuild.

C) Add a new report to rbuild (my approach): Inset of auto generating base
address and having the problems listed above we could add a new target "make
baseadressreport" which analyze the build dll images found on \obj-* (no
matter who compiled them) and use it's file size + module "baseaddress"
property to detect possible overlaps. By comparing reports with binaries
build by gcc and msvc we should be able to reconfigure base address to not
to conflict.

	PROS: same advantages as case A. will work with images build by any
compiler but some fixes may be latter required based on report.easy to
implement.
	CONS: baseaddress are still calculated by hand (but validated
automatically on request).

>   As for baseaddress in xml form - baseaddress is a
> module's property,
>  it's not a standalone object which is part of the module. I don't see
>  why it should be given a standalone tag (maybe I'm missing something). 
"property" IS actually an object, which is basically a generic key-value
container. The information it contains is *meaningless* to rbuild as it just
simply returns its value when someone reference it by his key. It can
contain anything from an empty string to Boolean or int ... .Rbuild don't
care about it as there is no way to validate the information they contain
because we don't know what to expect. "baseaddres" by contrast is a
specialized property (the C++ object is actually a property subclass so
technically still a property) which allows us to apply any particular logic
we desire to it , for example , generic property's value can be empty
baseaddres propery's value not. Generic property's value is not required to
be unique but baseaddres property must be. and so on...

Don't confuse rbuild xml syntax with objects as it's just a way to express
and persist the information. For example the "include" element <include
base="cacls"/>.</include> is also using an attribute to reference an
object
in this case a module, once it's resolved it only uses the module's path.
Same case as above.

Regards,
/Marc

[Steven Edwards]

On Dec 16, 2007 12:40 PM, Marc Piulachs <marc.piulachs(a)codexchange.net> wrote:
>           CONS: baseaddress are still calculated by hand
> (but validated
>  automatically on request).
>
> >   As for baseaddress in xml form - baseaddress is a
> > module's property,
> >  it's not a standalone object which is part of the module. I don't see
> >  why it should be given a standalone tag (maybe I'm missing something). 
>  "property" IS actually an object, which is basically a generic key-value
>  container. The information it contains is *meaningless* to rbuild as it just
>  simply returns its value when someone reference it by his key. It can
>  contain anything from an empty string to Boolean or int ... .Rbuild don't
>  care about it as there is no way to validate the information they contain
>  because we don't know what to expect. "baseaddres" by contrast is a
>  specialized property (the C++ object is actually a property subclass so
>  technically still a property) which allows us to apply any particular logic
>  we desire to it , for example , generic property's value can be empty
>  baseaddres propery's value not. Generic property's value is not required to
>  be unique but baseaddres property must be. and so on... 
I don't understand why if it can be validated, it can't be
autogenerated and validated by the same process. It seems like the
validation logic is going to be the same as autogeneration logic so
you might as well have it spit out the address and save having to
write them out by hand in each module, or in the root configuration
for bassaddresses.

-- 
Steven Edwards

"There is one thing stronger than all the armies in the world, and
that is an idea whose time has come." - Victor Hugo

[Marc Piulachs]

Hi,

>   I don't understand why if it can be validated, it
> can't be
>  autogenerated and validated by the same process. It seems like the
>  validation logic is going to be the same as autogeneration logic so
>  you might as well have it spit out the address and save having to
>  write them out by hand in each module, or in the root configuration
>  for bassaddresses. 
Validating is really a two step process, in this case I was referring to the
first part when rbuild reads the build files and starts processing object's
properties. When it finds a property representing a base address the
information it contains must at least *look like* a base address otherwise
it may be a potential problem that has to be reported to the user. At this
point the source code is not yet build so rbuild can't perform any
additional validations. The second step is when the build has ended, now
rbuild can use (((module base address) + file size + ¿headers size?) rounded
to nearest 64K multiple) to see if any dlls are *in theory* overlapping
other's address space. IMO having to run this second process at every build
is really inefficient, a waste of resources and presents other disadvantages
I mentioned in my previous mail.

Summarizing, to auto generate and/or fully validate base address we need to
have the source code compiled, the thing is, if we want to auto generate
them at every build all the dlls will have to be re-linked to the new
address. If we generate them by hand we need a way to ensure they will be
valid under different conditions (compiler used).

Regards,
/Marc

[Steven Edwards]

On Dec 17, 2007 4:13 AM, Marc Piulachs <marc.piulachs(a)codexchange.net> wrote:
>   Validating is really a two step process, in this case
> I was referring to the
>  first part when rbuild reads the build files and starts processing object's
>  properties. When it finds a property representing a base address the
>  information it contains must at least *look like* a base address otherwise
>  it may be a potential problem that has to be reported to the user. At this
>  point the source code is not yet build so rbuild can't perform any
>  additional validations. The second step is when the build has ended, now
>  rbuild can use (((module base address) + file size + ¿headers size?) rounded
>  to nearest 64K multiple) to see if any dlls are *in theory* overlapping
>  other's address space. IMO having to run this second process at every build
>  is really inefficient, a waste of resources and presents other disadvantages
>  I mentioned in my previous mail.
>
>  Summarizing, to auto generate and/or fully validate base address we need to
>  have the source code compiled, the thing is, if we want to auto generate
>  them at every build all the dlls will have to be re-linked to the new
>  address. If we generate them by hand we need a way to ensure they will be
>  valid under different conditions (compiler used). 
Yes but are you talking about extra minutes of compile time or longer?
The amount of hours++ spent managing manually re-basing the dlls over
the years has got to justify spreading the burden to the people doing
the compiling. The extra few minutes Relinking/rebasing seems more
than justifiable compared to the maintenance of fighting with
reservation amounts across various revisions of the compiler.

-- 
Steven Edwards

"There is one thing stronger than all the armies in the world, and
that is an idea whose time has come." - Victor Hugo

[ShadowFlare]

On Dec 16, 2007 10:40 AM, Marc Piulachs <marc.piulachs(a)codexchange.net>
wrote:

>   A) create a new tool to investigate the build dll
> images , use it's file
>  size to distribute dlls in the memory space and generate baseaddress.xml .
>  run it from time to time.
>  
The file size can be different than the size it will be when loaded into
memory, so that must be taken into account.  The headers in the file would
need to be checked to find the size it will be when loaded.

-ShadowFlare

[Magnus Olsen]

Hi
swedish version have diffent base address that english version of xp/2000

that why some nivida drv that work in english version does not work with swedish version
nivida forget this time to time

[KJK::Hyperion]

Magnus Olsen ha scritto:
>   Hi
>  swedish version have diffent base address that english version of xp/2000 
And that's [one of the trillion reasons] why we should have english-only
binaries, and satellite DLLs for localization. Resource-only DLLs need
no relocation, so their base address doesn't matter