Re: [ros-dev] Autogenerating compiler specific .def files

[Aleksey Bragin]

Theoretically extending this logic to infinity will lead to creation  
of a new high level language, stored in xml documents, parsing which  
rbuild generates source code in C or other language needed. :-).

The thing is, .def is rather simple, .spec is a bit more advanced to  
include more stuff ("stub", what else does reactos use from it?).  
There is absolutely no need to invent yet another format for storing  
export functions. It was invented once by MSVC, it was invented by  
GCC/MinGW, both are incompatible, it was invented by Wine to have  
ability of easy functions stubbing. Should there be another invention  
of the same wheel by ReactOS now?

If GCC/MinGW wouldn't be that *strange*, it could stdcall-fixup, and  
then properly link, this way .def could be created in the lowest  
common denominator format.


With the best regards,
Aleksey Bragin.


On Nov 24, 2007, at 4:36 PM, Marc Piulachs wrote:

>   I wouldn't use .spec files for several reasons,
> first of all is  
>  that it
>  violates the way how rbuild currently works. Rbuild reads xml  
>  documents in a
>  custom format and produce different outputs depending on various  
>  conditions.
>  IMO rbuild shouldn't be extended to parse anything else but  
>  just .rbuild
>  files.
>
>  Also, winebuild from the rbuild point of view is not different from  
>  any
>  other compiler , so I think the problem is just the opposite ,  
>  rbuild should
>  use the information stored in it's rbuild file to produce the spec  
>  file in
>  addition to the specific def file (it shouldn't be produced by  
>  winebuild
>  anymore) as both files are required to compile wine module XYZ.
>
>  Having the information in rbuild files has also other advantages,  
>  the most
>  important is that it would allow us to use the conditional logic to  
>  export
>  functions based on a given condition, for example hacks like
>  /lib/debugsup.rbuild won't be necessary anymore.
>
>  I wouldn't fork winebuild.
>
>  Regards,
>  Marc

[Marc Piulachs]

>  There is absolutely no need to invent yet another
> format for storing  
> export functions. It was invented once by MSVC, it was invented by  
> GCC/MinGW, both are incompatible, it was invented by Wine to have  
> ability of easy functions stubbing. Should there be another invention  
> of the same wheel by ReactOS now? 
>  If GCC/MinGW wouldn't be that *strange*, it could
> stdcall-fixup, and  
> then properly link, this way .def could be created in the lowest  
> common denominator format. 
That's the problem I'm trying to solve..:) The point on this change is not
really store them in xml inset of plain text, is not that important, for me
there are two reasons, first declare this information so rbuild can
manipulate it as objects and apply checking , logic , whatever ... and the
most important store the information on a neutral , compiler agnostic and
rich format that allow us to generate (export to) ANY format we desire out
of it. Currently def files are tided to mingw which is bad.

At the end the information must be there so what's the difference between
having it in a file with extension .def or .spec and having it on a file
with .rbuild extension?  IMO It's not reinventing the wheel

/Marc

On Nov 24, 2007, at 4:36 PM, Marc Piulachs wrote:

>   I wouldn't use .spec files for several reasons,
> first of all is  
>  that it
>  violates the way how rbuild currently works. Rbuild reads xml  
>  documents in a
>  custom format and produce different outputs depending on various  
>  conditions.
>  IMO rbuild shouldn't be extended to parse anything else but  
>  just .rbuild
>  files.
>
>  Also, winebuild from the rbuild point of view is not different from  
>  any
>  other compiler , so I think the problem is just the opposite ,  
>  rbuild should
>  use the information stored in it's rbuild file to produce the spec  
>  file in
>  addition to the specific def file (it shouldn't be produced by  
>  winebuild
>  anymore) as both files are required to compile wine module XYZ.
>
>  Having the information in rbuild files has also other advantages,  
>  the most
>  important is that it would allow us to use the conditional logic to  
>  export
>  functions based on a given condition, for example hacks like
>  /lib/debugsup.rbuild won't be necessary anymore.
>
>  I wouldn't fork winebuild.
>
>  Regards,
>  Marc 
_______________________________________________
Ros-dev mailing list
Ros-dev(a)reactos.org
http://www.reactos.org/mailman/listinfo/ros-dev

[Klemens Friedl]

2007/11/24, Marc Piulachs <marc.piulachs(a)codexchange.net>et>:
>   That's the problem I'm trying to solve..:) The
> point on this change is not
>  really store them in xml inset of plain text, is not that important, for me
>  there are two reasons, first declare this information so rbuild can
>  manipulate it as objects and apply checking , logic , whatever ... and the
>  most important store the information on a neutral , compiler agnostic and
>  rich format that allow us to generate (export to) ANY format we desire out
>  of it. Currently def files are tided to mingw which is bad. 

The "plain text" files (aka *.def, *.spec) can be described with
(E)BNF notion. And the (E)BNF could be of coursed used to feed a
lexer/parser generator, etc. (in theory) ;-)

I think it's not the best idea to re-invent the wheel by constructing
another new high level language with xml alike syntax.

as Aleksey mentioned:
>   Theoretically extending this logic to infinity will
> lead to creation
>  of a new high level language, stored in xml documents, parsing which
>  rbuild generates source code in C or other language needed. :-). 


btw. it would be very useful to have a documentation of the current
(trunk) rbuild syntax, for example in EBNF (Extended Backus–Naur form)
notion.

With the EBNF syntax on a single sheet, you could construct a rbuild
file from scratch without forking an existing rbuild file or reading
the rbuild source code.


Klemens

[Klemens Friedl]

>   btw. it would be very useful to have a documentation
> of the current
>  (trunk) rbuild syntax, for example in EBNF (Extended Backus–Naur form)
>  notion. 
just for record: I haven't forgot about the rbuild\doc\rbuild.txt
file; a metasyntax notation like BNF would be probably more readable
and useful.

[Klemens Friedl]

Our rbuild comes with project.dtd syntax file that is using the DTD
(Document Type Definition) format, but comes without header info.

DTD is sadly a bit hard to read (custom mixture or SGML and XML
syntax), and lacks certain expression-support. DTD got a 'successor'
with "XML Schema" (W3C standard) in 2001.

[Marc Piulachs]

>  There is absolutely no need to invent yet another
> format for storing  
> export functions. It was invented once by MSVC, it was invented by  
> GCC/MinGW, both are incompatible, it was invented by Wine to have  
> ability of easy functions stubbing. Should there be another invention  
> of the same wheel by ReactOS now? 
Just to clarify one thing, if we were going to support only one compiler
which speaks only one dialect of those 3 there wouldn't be any problem. But
it's a fact that we want to support at least 3 compilers (for now) msvc ,
gcc and winebuild which use incompatible syntax between them. Expressing
this information in a structured format like xml makes the parsing stage
easier and we have all the power builtin in rbuild to process that
information and generate back the msvc , gcc and winebuild versions with
ease. All the pieces fit together and we don't have to modify winebuild not
even create any other tool or parser from scratch.

I think is the easier solution to archive real compiler independent .def
files.

/marc


On Nov 24, 2007, at 4:36 PM, Marc Piulachs wrote:

>   I wouldn't use .spec files for several reasons,
> first of all is  
>  that it
>  violates the way how rbuild currently works. Rbuild reads xml  
>  documents in a
>  custom format and produce different outputs depending on various  
>  conditions.
>  IMO rbuild shouldn't be extended to parse anything else but  
>  just .rbuild
>  files.
>
>  Also, winebuild from the rbuild point of view is not different from  
>  any
>  other compiler , so I think the problem is just the opposite ,  
>  rbuild should
>  use the information stored in it's rbuild file to produce the spec  
>  file in
>  addition to the specific def file (it shouldn't be produced by  
>  winebuild
>  anymore) as both files are required to compile wine module XYZ.
>
>  Having the information in rbuild files has also other advantages,  
>  the most
>  important is that it would allow us to use the conditional logic to  
>  export
>  functions based on a given condition, for example hacks like
>  /lib/debugsup.rbuild won't be necessary anymore.
>
>  I wouldn't fork winebuild.
>
>  Regards,
>  Marc 
_______________________________________________
Ros-dev mailing list
Ros-dev(a)reactos.org
http://www.reactos.org/mailman/listinfo/ros-dev

[Colin Finck]

Hi Marc!

Marc Piulachs wrote:
>   But it's a fact that we want to support at least 3
> compilers (for 
>  now) msvc , gcc and winebuild 
Don't see winebuild as a compiler!
Wine also needs to create export definitions for many different compilers
and actually winebuild is their solution to this problem.
It processes .spec files, which also just contain information about each
export and it can convert them to different output formats (which are then
processed by a real compiler).

Currently, we only make use of the .def file output of winebuild, but the
generated .def files are MinGW-only, so if we want to use .spec files, we
would need to implement MSVC .def file support in winebuild.
But as winebuild supports many more options and output formats we don't use
and will probably never use, I proposed a rewrite of it for our purposes,
which should be a relatively easy task and result in a more lightweight
tool.

So, if we would use .spec files for export definitions
* we just need to add MSVC .def file output to winebuild (or create an own
tool with MinGW .def and MSVC .def output)
* we need to convert all existing .def files, but can leave the already
existing .spec files untouched
* we use the same format Wine also uses for their export definitions.
Therefore Wine modules can just be imported and don't require further
changes in the export definition files.

If we would use the .rbuild method you proposed
* we need to work out a completely new format for export definitions in
rbuild's XML files and add MinGW .def and MSVC .def file generation to
rbuild.
* we need to convert all existing .def *and* .spec files to the new XML
format
* we invent yet another format for defining exports. Wine modules cannot
also just be imported, because we need to convert their .spec files to the
XML format everytime then.

An advantage of your proposed .rbuild solution might be that we can use the
rbuild conditional logic, but I don't know how important this really is.
Your example lib/debugsup/debugsup.rbuild just uses the same export
definition file for two modules, I don't consider this a hack.

Just my two cents.

Regards,

Colin

[Marc Piulachs]

Hi Colin,

Your opinion makes sense, you almost convinced me :D but let me post a few
open questions/comments:

The modified winebuild or equivalent tool you proposed will work as the
current version, completely disconnected from rbuild , it only knows that
this tool should be called when a .spec file is found inside a module , then
it expects to find a def file with the format modulename.spec.def that he
doesn't even know was generated by winebuild , it's simply there. Now if we
are going to generate two .def files f.e: modulename.msvc.spec.def and
modulename.gcc.spec.def how we should declare it on the modulename.rbuild?
we have to add some logic there to map the appropriate generated def file or
even worse hardcode the filename in rbuild backends. How does exactly fit on
the current rbuild format/backends?

* With my proposed method the def files are created by backends on request
so on a msvc build only msvc compatible def files would be generated , also
the process takes places inside rbuild so a part from error/checking and the
other advantages the generated def file can be automatically added to the
module so would render the importlibrary element obsolete. It would be all
automatic.

>   Don't see winebuild as a compiler! 
Actually it is :) . A compiler is blackbox which produces something (in this
case a def file + C files) out of a human-readable text file.

>  Your example lib/debugsup/debugsup.rbuild just uses the
> same export
> definition file for two modules, I don't consider this a hack. 
Of course it works but I think is hack for two reasons: 

- debugsup_ntoskrnl and debugsup_ntdll are not 'real' modules , they don't
represent or produce anything they are there only because there isn't a
better solution.
- If you take a look at /tools/rbuild/modulehandler.cpp:2506 you will be
horrorified to see that both module names are hardcoded on rbuild which is
not good.

Last but not least, from my point of view integration is important, We have
a fairly complex build tool, all the information we can give to it about the
codebase will worth it because more automatic checking we will be able to do
and more things we will be able to autogenerate , we should let rbuild
itself generate as much things as possible.

/marc

[Colin Finck]

Hi Marc,

Marc Piulachs wrote: 
>   Now if we are going to generate two .def files f.e:
modulename.msvc.spec.def
>   and modulename.gcc.spec.def how we should declare it
> on the 
>  modulename.rbuild? 
I think we should change the <importlibrary> tag to support .spec files
directly.

For example in the module you have: <importlibrary definition="module.spec"
/>
When you compile using the MinGW backend, you will get a file
"module_gcc.def" in the intermediate directory. As rbuild builds the
filename itself, it can simply use it for makefile.auto and expect that the
.spec to .def converter created this file.
When the MSVC backend is used (by calling "make msvc*"), it will create a
file "module_msvc.def" in the intermediate directory.


>   * With my proposed method the def files are created by
>
>  backends on request so on a msvc build only msvc
>  compatible def files would be generated 
Is also possible with the .spec file method (see above).


>   - If you take a look at
> /tools/rbuild/modulehandler.cpp:2506 
>  you will be horrorified to see that both module names are 
>  hardcoded on rbuild which is not good. 
Agreed, this should be handled better, but this is something we can care
about later :-P


>   Last but not least, from my point of view integration
> is 
>  important, We have a fairly complex build tool, all the information
>  we can give to it about the codebase will worth it because more
>  automatic checking we will be able to do and more things we will
>  be able to autogenerate , we should let rbuild itself generate as much
>  things as possible. 
When doing such changes, always remember that rbuild is mainly just a tool
that processes makefile-like files and doesn't compile code files itself.
Also processing XML files is slower compared to processing plain text files.
In this case, there also exists a format that is already used to solve this
problem, so we should really adopt it.

Regards,

Colin