Hi.
I've been working on getting ReactOS to build faster. By compiling fewer, but larger source files (compilation units), we can build ReactOS much faster. I estimate that we can build ReactOS 200-300% faster if we do this. See the attached patch for kernel32 for an example. On my slow box (Windows), I get the following numbers:
kernel32 (1 compilation unit per file): build time: 32 seconds space consumed by object files: 52MB
kernel32 (fewer compilation units): build time: 11 seconds space consumed by object files: 14MB
Another advantage is the less disk space requirements. I estimate ~700-800MB will be needed for object files after this change. Compare that to the 2GB that is needed today.
Gé did some testing on Linux:
kernel32 (1 compilation unit per file): build time: 11.3 seconds
kernel32 (fewer compilation units): build time: 4.4 seconds
Some issues: * You can't mix source code files with different file name extensions in the same compilation unit (currently only *.c is supported) * There must not be duplicate definitions in the same compilation unit even though it is spread over several source code files (duplicating code is bad practice anyway). * Like when you compile a big source file, memory usage during compilation is higher than when compiling several smaller source code files, one at a time. This puts a practical limit on how large the compilation unit can be. * Every source code file in the compilation unit is recompiled after a change to one or more source code files within the compilation unit. The time needed for linking may however be significantly less (due to the fewer, much smaller object files) so it may be faster in the end. * Pre-compiled headers are not easy to support, while at the same time maintaining compatibility with MSVC and one source code file per compilation unit mode. Currently PCHs are disabled for a module if the module has compilation units with more than one source code file in it.
While compiling kernel32 as one big compilation unit, GCC required 60MB of virtual memory. What is an acceptable memory usage? It determines how large we can make the compilation units and affects how fast we can build ReactOS.
Now we have 300+ modules that need to be split into compilation units. Are there any volunteers that wish to help with these tasks?
Thoughts?
Casper
Casper Hornstrup writes:
[snip] Now we have 300+ modules that need to be split into compilation units. Are there any volunteers that wish to help with these tasks?
Thoughts?
Maybe I do not understand enough about how compilation units work so this might be a dumb question, but why cant rbuild automatically split projects up into compilation units automatically?
rbuild is supposed to insulate projects from the details of the underlying build system, and it could decide how many compilation units it would use for any given project based on how many lines of code are involved, or just the file sizes for that matter. Of course, if some projects have duplicate code in such a way that compilation units can't be used, then thats a problem. But I would imagine that those would be easily fixed.
-N
Maybe I do not understand enough about how compilation units work so this might be a dumb question, but why cant rbuild automatically split projects up into compilation units automatically?
rbuild is supposed to insulate projects from the details of the underlying build system, and it could decide how many compilation units it would use for any given project based on how many lines of code are involved, or just the file sizes for that matter. Of course, if some projects have duplicate code in such a way that compilation units can't be used, then thats a problem. But I would imagine that those would be easily fixed.
If all files for all modules can be built as one compilation unit, then yes, rbuild can just do this automatically. Either make the module one big compilation unit or split it based on available memory. However, it leaves no room for exceptions.
Casper
-
Casper Hornstrup -
Nathan Woods