Faster compilation

[Casper Hornstrup]

Hi.

I've been working on getting ReactOS to build faster. By compiling fewer,
but larger source files (compilation units), we can build ReactOS much faster.
I estimate that we can build ReactOS 200-300% faster if we do this. See
the attached patch for kernel32 for an example. On my slow box (Windows),
I get the following numbers:

kernel32 (1 compilation unit per file):
build time: 32 seconds
space consumed by object files: 52MB

kernel32 (fewer compilation units):
build time: 11 seconds
space consumed by object files: 14MB

Another advantage is the less disk space requirements. I estimate ~700-800MB
will be needed for object files after this change. Compare that to the 2GB
that is needed today.

Gé did some testing on Linux:

kernel32 (1 compilation unit per file):
build time: 11.3 seconds

kernel32 (fewer compilation units):
build time: 4.4 seconds

Some issues:
* You can't mix source code files with different file name extensions in the
  same compilation unit (currently only *.c is supported)
* There must not be duplicate definitions in the same compilation unit even
  though it is spread over several source code files (duplicating code is bad
  practice anyway).
* Like when you compile a big source file, memory usage during compilation
  is higher than when compiling several smaller source code files, one at a
  time. This puts a practical limit on how large the compilation unit can be.
* Every source code file in the compilation unit is recompiled after a change
  to one or more source code files within the compilation unit. The time
  needed for linking may however be significantly less (due to the fewer, much
  smaller object files) so it may be faster in the end.
* Pre-compiled headers are not easy to support, while at the same time
  maintaining compatibility with MSVC and one source code file per compilation
  unit mode. Currently PCHs are disabled for a module if the module has
  compilation units with more than one source code file in it.

While compiling kernel32 as one big compilation unit, GCC required 60MB of
virtual memory. What is an acceptable memory usage? It determines how large
we can make the compilation units and affects how fast we can build ReactOS.

Now we have 300+ modules that need to be split into compilation units. Are there
any volunteers that wish to help with these tasks?

Thoughts?

Casper

[Nathan Woods]

Casper Hornstrup writes: 

>   [snip]
>  Now we have 300+ modules that need to be split into compilation units. Are 
>  there any volunteers that wish to help with these tasks? 
>  
>  Thoughts? 
 

Maybe I do not understand enough about how compilation units work so this 
might be a dumb question, but why cant rbuild automatically split projects 
up into compilation units automatically? 


rbuild is supposed to insulate projects from the details of the underlying 
build system, and it could decide how many compilation units it would use 
for any given project based on how many lines of code are involved, or just 
the file sizes for that matter.  Of course, if some projects have duplicate 
code in such a way that compilation units can't be used, then thats a 
problem.  But I would imagine that those would be easily fixed. 


 -N

[Casper Hornstrup]

>   Maybe I do not understand enough about how compilation
> units work so this
>  might be a dumb question, but why cant rbuild automatically split projects
>  up into compilation units automatically?
>  
>  
>  rbuild is supposed to insulate projects from the details of the underlying
>  build system, and it could decide how many compilation units it would use
>  for any given project based on how many lines of code are involved, or just
>  the file sizes for that matter.  Of course, if some projects have duplicate
>  code in such a way that compilation units can't be used, then thats a
>  problem.  But I would imagine that those would be easily fixed. 
If all files for all modules can be built as one compilation unit, then yes,
rbuild can just do this automatically. Either make the module one big
compilation unit or split it based on available memory. However, it leaves no
room for exceptions.

Casper