Hello all,
The "Trunk_x86_GCCLin Debug" Buildslave has been migrated to RosBE-Unix 2.0-RC1. All builds starting with r53983 are now created using a CMake environment.
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
If nobody has an idea how to debug or fix this, I'm afraid I have to revert everything back to the rbuild scripts until a proper solution is found.
- Colin
Unfortunately, i cannot replicate that. Tried this iso in vbox and qemu - all works well. Any chance that KVM is screwed?
On Tuesday, October 04, 2011 2:53 AM, "Colin Finck" colin@reactos.org wrote:
Hello all,
The "Trunk_x86_GCCLin Debug" Buildslave has been migrated to RosBE-Unix 2.0-RC1. All builds starting with r53983 are now created using a CMake environment.
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
If nobody has an idea how to debug or fix this, I'm afraid I have to revert everything back to the rbuild scripts until a proper solution is found.
- Colin
Op 4-10-2011 2:53, Colin Finck schreef:
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
If nobody has an idea how to debug or fix this, I'm afraid I have to revert everything back to the rbuild scripts until a proper solution is found.
Any chance of you updating KVM's SeaBIOS? It's pretty ancient considering they're at 0.6.2 now ( http://www.seabios.org/Releases ). Maybe that would help.
Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?
På Tuesday, October 04, 2011 4:05 AM, skrev "Bernd Blaauw" bblaauw@home.nl:
Op 4-10-2011 2:53, Colin Finck schreef:
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
Any chance of you updating KVM's SeaBIOS? It's pretty ancient considering they're at 0.6.2 now ( http://www.seabios.org/Releases ). Maybe that would help.
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
I agree with Olaf. Especially since it used to work before. The problem is not on the test server, but on the CMake build. Updating BIOS and praying it works would only hide the issue.
Olaf, for your tests, did you use Qemu? KVM? Through libvirt? On Linux or Windows?
Regards, Pierre
caemyr@myopera.com wrote:
Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?
På Tuesday, October 04, 2011 4:05 AM, skrev "Bernd Blaauw" bblaauw@home.nl:
Op 4-10-2011 2:53, Colin Finck schreef:
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
Any chance of you updating KVM's SeaBIOS? It's pretty ancient considering they're at 0.6.2 now ( http://www.seabios.org/Releases ). Maybe that would help.
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Windows as host, QEMU windows build, 0.13.1 iirc.
On Tuesday, October 04, 2011 1:17 PM, "Pierre Schweitzer" pierre@reactos.org wrote:
I agree with Olaf. Especially since it used to work before. The problem is not on the test server, but on the CMake build. Updating BIOS and praying it works would only hide the issue.
Olaf, for your tests, did you use Qemu? KVM? Through libvirt? On Linux or Windows?
Regards, Pierre
On Oct 4, 2011, at 7:04 AM, caemyr@myopera.com wrote:
Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?
Does anyone NOT see the irony here? It's CMake that brings regressions (for seemingly no reason). Off the top of my head, I can think of this freeloader thing (broken only on CMake), the freeloader bug that Timo ran into (only worked on CMake), and the AC97 hang (broken only on CMake). Let's fix our CMake build before we go rushing into it and introduce regressions from something as trivial and stupid as a build system. Hold off on RosBE 2.0! We're clearly not ready for it!
Regards, Cameron
I could also add intl.cpl not working in 2nd stage setup. Still, those regressions, apart from the KVM one are meaningless, compared to the profits CMake brings. Changing build system/buildchain is ALWAYS dramatic and troublesome. We cannot stop it, as supporting CMake and rbuild at once is purely a waste of resources. There is no going back to rbuild, Cameron.
On Tuesday, October 04, 2011 9:28 AM, "Cameron Gutman" aicommander@gmail.com wrote:
On Oct 4, 2011, at 7:04 AM, caemyr@myopera.com wrote:
Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?
Does anyone NOT see the irony here? It's CMake that brings regressions (for seemingly no reason). Off the top of my head, I can think of this freeloader thing (broken only on CMake), the freeloader bug that Timo ran into (only worked on CMake), and the AC97 hang (broken only on CMake). Let's fix our CMake build before we go rushing into it and introduce regressions from something as trivial and stupid as a build system. Hold off on RosBE 2.0! We're clearly not ready for it!
Regards, Cameron
On Oct 4, 2011, at 10:06 AM, caemyr@myopera.com wrote:
I could also add intl.cpl not working in 2nd stage setup. Still, those regressions, apart from the KVM one are meaningless, compared to the profits CMake brings. Changing build system/buildchain is ALWAYS dramatic and troublesome. We cannot stop it, as supporting CMake and rbuild at once is purely a waste of resources. There is no going back to rbuild, Cameron.
But why is changing a build system "ALWAYS dramatic"? Why do we have issues like this? We're not changing the compiler, linker, assembler, or any source (in theory, but not really). The real build tools (GCC, LD, GAS, etc) should be receiving the same parameters for the build in the end whether via the CMake frontend or the rbuild frontend. So either the new debugging symbol stuff is exposing/creating bugs in the code, or CMake is not passing the same parameters to the build chain as rbuild is. The reason we're having trouble is because we tried to do too much with this change at one time. It's the same reason Intel does the tick-tock release cycle. We've changed our build system, changed the type of symbols in our binaries, and made CMake-specific modifications to our code (to read the symbols). This should have been done at least in 2 steps, not in one giant mess as we have done. We absolutely cannot have bugs introduced due to our build system. All the build system provides is a nicer way to talk to the core build tools. That shouldn't be causing a problem unless something is wrong in the CMake configuration files. In which case, that must be corrected before CMake is adopted.
The mere fact that we have to deal with this only detracts from the development of real code. I took a good hour trying figure out why the hell I couldn't boot trunk when I was trying to test my ACPI fixes. The the Vbox testbot was doing fine, yet I couldn't boot my builds on Vbox. Only after reverting commits on rbuild and downloading the latest CMake build did I realize that only CMake builds would actually boot HEAD source code. I know I'm going to get "Well you should have used CMake, then it would've been fine," but CMake can't even properly do multithreaded compilation. My CPU usage using CMake is about 45%, while CPU usage while using rbuild is close to 100% for the majority of the build, and the build times reflect this. I look forward to adopting CMake but only when it's ready, and from the looks of the slowness and regressions, it's definitely not ready.
Regards, Cameron
Op 4-10-2011 18:08, Cameron Gutman schreef:
On Oct 4, 2011, at 10:06 AM, caemyr@myopera.com wrote:
The mere fact that we have to deal with this only detracts from the development of real code.
As you're (most likely correctly) excluding the possibility of SeaBIOS (which is in heavy development also, people might run into bios bugs due to some newly encountered corner cases) :
Did that test machine also choke on a ISO generated by Windows Cmake, or only on the Linux Cmake generated ISO?
Hi,
Le mardi 04 octobre 2011 à 18:49 +0200, Bernd Blaauw a écrit :
Did that test machine also choke on a ISO generated by Windows Cmake, or only on the Linux Cmake generated ISO?
This test is already planned and will be done tonight.
FYI, Linux buildslave will be taken offline during the test.
Regards, Pierre
I've just run the said test. As you can see, no real difference: http://img155.imageshack.us/img155/3831/debugdiablo.png
Le mardi 04 octobre 2011 à 18:53 +0200, Pierre Schweitzer a écrit :
Hi,
Le mardi 04 octobre 2011 à 18:49 +0200, Bernd Blaauw a écrit :
Did that test machine also choke on a ISO generated by Windows Cmake, or only on the Linux Cmake generated ISO?
This test is already planned and will be done tonight.
FYI, Linux buildslave will be taken offline during the test.
Regards, Pierre
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Op 4-10-2011 22:14, Pierre Schweitzer schreef:
I've just run the said test. As you can see, no real difference: http://img155.imageshack.us/img155/3831/debugdiablo.png
Did that test machine also choke on a ISO generated by Windows Cmake, or only on the Linux Cmake generated ISO?
This test is already planned and will be done tonight.
Could you provide any instructions to reproduce this issue please? Software versions, etc.
What I've tried so far: * download http://iso.reactos.org/bootcd/bootcd-53990-dbgwin.7z * download http://downloads.sourceforge.net/reactos/ReactOS-0.3.13-REL-qemu.zip * unpack this preloaded VM * verify stuff works by booting this QEMU with default settings * unpack bootcd archive and place it in QEMU dir * modify boot.bat ( http://www.reactos.org/wiki/QEMU ) to add cdrom.
End result for me was a working configuration with following batchfile. As you can see that BIOS is more recent than your KVM BIOS. If it was KVM used instead of QEMU, your screenshot doesn't show appear to show that.
I've got no real conclusion yet, could only recommend to try with above reference QEMU download file as provided on the ReactOS website.
@echo off title QEMU 013 (SeaBIOS 0.6.1pre-20100713-085324-titi ) qemu -L . -m 128 -hda ReactOS.vmdk -cdrom bootcd-53990-dbgwin.iso -boot d -net nic,model=ne2k_pci -net user -serial file:CON
Bernd Blaauw bblaauw@home.nl wrote:
Could you provide any instructions to reproduce this issue please? Software versions, etc.
Our buildslave uses KVM under Ubuntu 10.04. The underlying QEMU version is given as 0.12.3 with SeaBIOS 0.5.1.
- Colin
Then it looks like a specific libvirt issue with all cmake builds.
On Wednesday, October 05, 2011 12:30 AM, "Colin Finck" colin@reactos.org wrote:
Bernd Blaauw bblaauw@home.nl wrote:
Could you provide any instructions to reproduce this issue please? Software versions, etc.
Our buildslave uses KVM under Ubuntu 10.04. The underlying QEMU version is given as 0.12.3 with SeaBIOS 0.5.1.
- Colin
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
"Cameron Gutman" aicommander@gmail.com писал(а) в своём письме Tuesday, October 04, 2011 12:08 PM:
But why is changing a build system "ALWAYS dramatic"? Why do we have issues like this?
Its because Linux bot conversion was not done they way it should be.
This should have been done at least in 2 steps, not in one giant mess as we have done.
On my bot we had cmake builder added on-demand the moment CMake builds were mature enough to produce ISO. The switch to default CMake happenend here before the summer iirc and new BE is being tested since September. I think its enough steps.
We absolutely cannot have bugs introduced due to our build system.
Good to know we can have weeks-lasting breakages from other components. Only build system must be perfect!
The mere fact that we have to deal with this only detracts from the development of real code. I took a good hour trying figure out why the hell I couldn't boot trunk when I was trying to test my ACPI fixes. The the Vbox testbot was doing fine, yet I couldn't boot my builds on Vbox. Only after reverting commits on rbuild and downloading the latest CMake build did I realize that only CMake builds would actually boot HEAD source code.
First of all, ACPI is not the best example to bring out in this particular case. Secondly, we have two buildbots, and VBox one was running CMake builds when you tested ACPI. Thirdly, it only proves what i said before, running rbuild and cmake side by side is the thing responsible for distractions from ROS development. It uses our resources and time, in your example i would HAVE to be running twice the builds, for both cmake and rbuild, something i simply have no hardware to perform. And what you opt for IS running them side by side for much longer than I would want to.
I know I'm going to get "Well you should have used CMake, then it would've been fine," but CMake can't even properly do multithreaded compilation. My CPU usage using CMake is about 45%, while CPU usage while using rbuild is close to 100% for the majority of the build, and the build times reflect this.
I am sorry to say that, but this problem is entirely your fault. Had you come out and tell someone about it, we would have a solution for you long ago. We had this issue in windows cmake and it was fixed long ago... you cannot expect 4 minute ReactOS build with only one core?
I look forward to adopting CMake but only when it's ready, and from the looks of the slowness and regressions, it's definitely not ready.
We cannot fix any issues that are not reported and i dont recall you reporting anything in particular, besides some occasional comments that CMake is bad and is not working for you, only when the transition was nearing. In my opinion, some steps will have to be forced, and this switch is one of them. The longer we run rbuild and cmake side by side, the more regressions we are likely to create! The longer we go like that, more time and effort is needed to actually complete the goal, simply because you and others are reluctant to switch. It is you who should know best that drastic steps are sometimes necessary to finish up stuff.
Regards, Cameron
Am 04.10.2011 02:53, schrieb Colin Finck:
Hello all,
The "Trunk_x86_GCCLin Debug" Buildslave has been migrated to RosBE-Unix 2.0-RC1. All builds starting with r53983 are now created using a CMake environment.
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
If nobody has an idea how to debug or fix this, I'm afraid I have to revert everything back to the rbuild scripts until a proper solution is found.
Can't we have both buildbots and let rbuild one be the default one and have cmake only build on demand. This way we have a better chance to really fix the issue and see if it worked.
I have now reverted everything back to the RBuild scripts. Will set up a side-by-side solution with manual CMake builds as soon as possible - unless somebody solves all problems in the meantime :)
Note that we don't just have a testing problem now, but also issues with PCH files (see http://build.reactos.org/builders/Trunk_x86_GCCLin%20Debug/builds/9903/steps...). The old ccache 2.4 has already been replaced by a fresh version 3.1.6 today, with apparently no effect on our problems though.
- Colin
Op 5-10-2011 1:29, Colin Finck schreef:
I have now reverted everything back to the RBuild scripts. Will set up a side-by-side solution with manual CMake builds as soon as possible - unless somebody solves all problems in the meantime :)
If I recall correctly Timo recently managed to fix a ReactOS CD bootsector or something so KVM would work again. If that's all that remained it might be worthwile trying again.
As ReactOS now supports ACPI, is it possible to optionally let each stage shutdown the (virtual) machine ? Then you could use individual QEMU commands to launch each phase of ReactOS with different settings. Perhaps something for the unattend.inf thing. Benefit might be some form of automated regression testing.
pseudo syntax: @echo off echo Starting trunk install from virtual CD: start /w qemu L . -m 32 -hda roshdd.img -boot d -cdrom trunk.iso echo Starting phase 2 configuration: start /w qemu L . -m 80 -hda roshdd.img -boot c echo Starting ReactOS start /w qemu L . -m 64 -hda roshdd.img -boot c
Colin Finck colin@reactos.org wrote:
Unfortunately, the CMake builds don't boot under KVM on the same machine, but apparently for everybody else. To give you an idea what I mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
Okay, I fixed the side-by-side building and started a new test run with r54062 yesterday. Unfortunately, we still have the same result (http://build.reactos.org/builders/Linux_AMD64_1%20KVM-CMake-Test/builds/6/st...), even after Amine's revert back to RSYM symbols.
So, what else could be wrong now? :-)
To force a build yourself on the CMake slave, don't forget to specify the revision for now. Will fix this later.
- Colin