CMake migration of Trunk_x86_GCCLin Debug Buildslave

[Colin Finck]

Hello all,

The "Trunk_x86_GCCLin Debug" Buildslave has been migrated to RosBE-Unix 
2.0-RC1. All builds starting with r53983 are now created using a CMake 
environment.

Unfortunately, the CMake builds don't boot under KVM on the same 
machine, but apparently for everybody else. To give you an idea what I 
mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png

If nobody has an idea how to debug or fix this, I'm afraid I have to 
revert everything back to the rbuild scripts until a proper solution is 
found.

- Colin

Unfortunately, i cannot replicate that. Tried this iso in vbox and qemu - all works well. Any chance that KVM is screwed?

On Tuesday, October 04, 2011 2:53 AM, "Colin Finck" [e-mail] wrote:
> Hello all,
>
> The "Trunk_x86_GCCLin Debug" Buildslave has been migrated to RosBE-Unix 
> 2.0-RC1. All builds starting with r53983 are now created using a CMake 
> environment.
>
> Unfortunately, the CMake builds don't boot under KVM on the same 
> machine, but apparently for everybody else. To give you an idea what I 
> mean by "don't boot":
> http://img830.imageshack.us/img830/5145/image1iz.png
>
> If nobody has an idea how to debug or fix this, I'm afraid I have to 
> revert everything back to the rbuild scripts until a proper solution is 
> found.
> Colin
-- 
With best regards
Caemyr

[Bernd Blaauw]

Op 4-10-2011 2:53, Colin Finck schreef:
> Unfortunately, the CMake builds don't boot under KVM on the same
> machine, but apparently for everybody else. To give you an idea what I
> mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
>
> If nobody has an idea how to debug or fix this, I'm afraid I have to
> revert everything back to the rbuild scripts until a proper solution is
> found.

Any chance of you updating KVM's SeaBIOS? It's pretty ancient 
considering they're at 0.6.2 now ( http://www.seabios.org/Releases ).
Maybe that would help.

Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?

På Tuesday, October 04, 2011 4:05 AM, skrev "Bernd Blaauw" [e-mail]:
> Op 4-10-2011 2:53, Colin Finck schreef:
> > Unfortunately, the CMake builds don't boot under KVM on the same
> > machine, but apparently for everybody else. To give you an idea what I
> > mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
>
> Any chance of you updating KVM's SeaBIOS? It's pretty ancient 
> considering they're at 0.6.2 now ( http://www.seabios.org/Releases ).
> Maybe that would help.
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev
-- 
With best regards
Caemyr

[Pierre Schweitzer]

I agree with Olaf. Especially since it used to work before. The problem 
is not on the test server, but on the CMake build. Updating BIOS and 
praying it works would only hide the issue.

Olaf, for your tests, did you use Qemu? KVM? Through libvirt? On Linux 
or Windows?

Regards,
Pierre

[e-mail] wrote:
> Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?
>
> På Tuesday, October 04, 2011 4:05 AM, skrev "Bernd Blaauw" [e-mail]:
> > Op 4-10-2011 2:53, Colin Finck schreef:
> > > Unfortunately, the CMake builds don't boot under KVM on the same
> > > machine, but apparently for everybody else. To give you an idea what I
> > > mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png
> >
> > Any chance of you updating KVM's SeaBIOS? It's pretty ancient 
> > considering they're at 0.6.2 now ( http://www.seabios.org/Releases ).
> > Maybe that would help.
> >
> > Ros-dev mailing list
> > [e-mail]
> > http://www.reactos.org/mailman/listinfo/ros-dev

Windows as host, QEMU windows build, 0.13.1 iirc.

On Tuesday, October 04, 2011 1:17 PM, "Pierre Schweitzer" [e-mail] wrote:
> I agree with Olaf. Especially since it used to work before. The problem 
> is not on the test server, but on the CMake build. Updating BIOS and 
> praying it works would only hide the issue.
>
> Olaf, for your tests, did you use Qemu? KVM? Through libvirt? On Linux 
> or Windows?
>
> Regards,
> Pierre
-- 
With best regards
Caemyr

[Cameron Gutman]

On Oct 4, 2011, at 7:04 AM, [e-mail] wrote:
> Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?

Does anyone NOT see the irony here? It's CMake that brings regressions (for seemingly no reason). Off the top of my head, I can think of this freeloader thing (broken only on CMake), the freeloader bug that Timo ran into (only worked on CMake), and the AC97 hang (broken only on CMake). Let's fix our CMake build before we go rushing into it and introduce regressions from something as trivial and stupid as a build system. Hold off on RosBE 2.0! We're clearly not ready for it!

Regards,
Cameron

I could also add intl.cpl not working in 2nd stage setup. Still, those regressions, apart from the KVM one are meaningless, compared to the profits CMake brings. Changing build system/buildchain is ALWAYS dramatic and troublesome. We cannot stop it, as supporting CMake and rbuild at once is purely a waste of resources. There is no going back to rbuild, Cameron.

On Tuesday, October 04, 2011 9:28 AM, "Cameron Gutman" [e-mail] wrote:
> On Oct 4, 2011, at 7:04 AM, [e-mail] wrote:
> > Without knowing the cause such upgrade is dangerous as new regressions might occur. Did you Colin try to enable freeldr debugging?
>
> Does anyone NOT see the irony here? It's CMake that brings regressions
> (for seemingly no reason). Off the top of my head, I can think of this
> freeloader thing (broken only on CMake), the freeloader bug that Timo ran
> into (only worked on CMake), and the AC97 hang (broken only on CMake).
> Let's fix our CMake build before we go rushing into it and introduce
> regressions from something as trivial and stupid as a build system. Hold
> off on RosBE 2.0! We're clearly not ready for it!
>
> Regards,
> Cameron
-- 
With best regards
Caemyr

[Cameron Gutman]

On Oct 4, 2011, at 10:06 AM, [e-mail] wrote:
> I could also add intl.cpl not working in 2nd stage setup. Still, those regressions, apart from the KVM one are meaningless, compared to the profits CMake brings. Changing build system/buildchain is ALWAYS dramatic and troublesome. We cannot stop it, as supporting CMake and rbuild at once is purely a waste of resources. There is no going back to rbuild, Cameron.

But why is changing a build system "ALWAYS dramatic"? Why do we have issues like this? We're not changing the compiler, linker, assembler, or any source (in theory, but not really). The real build tools (GCC, LD, GAS, etc) should be receiving the same parameters for the build in the end whether via the CMake frontend or the rbuild frontend. So either the new debugging symbol stuff is exposing/creating bugs in the code, or CMake is not passing the same parameters to the build chain as rbuild is. The reason we're having trouble is because we tried to do too much with this change at one time. It's the same reason Intel does the tick-tock release cycle. We've changed our build system, changed the type of symbols in our binaries, and made CMake-specific modifications to our code (to read the symbols). This should have been done at least in 2 steps, not in one giant mess as we have done. We absolutely cannot have bugs introduced due to our build system. All the build system provides is a nicer way to talk to the core build tools. That shouldn't be causing a problem unless something is wrong in the CMake configuration files. In which case, that must be corrected before CMake is adopted.

The mere fact that we have to deal with this only detracts from the development of real code. I took a good hour trying figure out why the hell I couldn't boot trunk when I was trying to test my ACPI fixes. The the Vbox testbot was doing fine, yet I couldn't boot my builds on Vbox. Only after reverting commits on rbuild and downloading the latest CMake build did I realize that only CMake builds would actually boot HEAD source code. I know I'm going to get "Well you should have used CMake, then it would've been fine," but CMake can't even properly do multithreaded compilation. My CPU usage using CMake is about 45%, while CPU usage while using rbuild is close to 100% for the majority of the build, and the build times reflect this. I look forward to adopting CMake but only when it's ready, and from the looks of the slowness and regressions, it's definitely not ready.

Regards,
Cameron

[Bernd Blaauw]

Op 4-10-2011 18:08, Cameron Gutman schreef:
> On Oct 4, 2011, at 10:06 AM, [e-mail] wrote:
> The mere fact that we have to deal with this only detracts from the development of real code.

As you're (most likely correctly) excluding the possibility of SeaBIOS 
(which is in heavy development also, people might run into bios bugs due 
to some newly encountered corner cases) :

Did that test machine also choke on a ISO generated by Windows Cmake, or 
only on the Linux Cmake generated ISO?

[Pierre Schweitzer]

Hi,

Le mardi 04 octobre 2011 à 18:49 +0200, Bernd Blaauw a écrit :
> Did that test machine also choke on a ISO generated by Windows Cmake, or 
> only on the Linux Cmake generated ISO?

This test is already planned and will be done tonight.

FYI, Linux buildslave will be taken offline during the test.

Regards,
Pierre

[Pierre Schweitzer]

I've just run the said test. As you can see, no real difference:
http://img155.imageshack.us/img155/3831/debugdiablo.png

Le mardi 04 octobre 2011 à 18:53 +0200, Pierre Schweitzer a écrit :
> Hi,
>
> Le mardi 04 octobre 2011 à 18:49 +0200, Bernd Blaauw a écrit :
> > Did that test machine also choke on a ISO generated by Windows Cmake, or 
> > only on the Linux Cmake generated ISO?
>
> This test is already planned and will be done tonight.
>
> FYI, Linux buildslave will be taken offline during the test.
>
> Regards,
> Pierre
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev

[Bernd Blaauw]

Op 4-10-2011 22:14, Pierre Schweitzer schreef:
> I've just run the said test. As you can see, no real difference:
> http://img155.imageshack.us/img155/3831/debugdiablo.png
> > > Did that test machine also choke on a ISO generated by Windows Cmake, or
> > > only on the Linux Cmake generated ISO?
> >
> > This test is already planned and will be done tonight.

Could you provide any instructions to reproduce this issue please? 
Software versions, etc.

What I've tried so far:
* download http://iso.reactos.org/bootcd/bootcd-53990-dbgwin.7z
* download 
http://downloads.sourceforge.net/reactos/ReactOS-0.3.13-REL-qemu.zip
* unpack this preloaded VM
* verify stuff works by booting this QEMU with default settings
* unpack bootcd archive and place it in QEMU dir
* modify boot.bat ( http://www.reactos.org/wiki/QEMU ) to add cdrom.

End result for me was a working configuration with following batchfile.
As you can see that BIOS is more recent than your KVM BIOS.
If it was KVM used instead of QEMU, your screenshot doesn't show appear 
to show that.

I've got no real conclusion yet, could only recommend to try with above 
reference QEMU download file as provided on the ReactOS website.

@echo off
title QEMU 013 (SeaBIOS 0.6.1pre-20100713-085324-titi )
qemu -L . -m 128 -hda ReactOS.vmdk -cdrom bootcd-53990-dbgwin.iso -boot 
d -net nic,model=ne2k_pci -net user -serial file:CON

[Colin Finck]

Bernd Blaauw [e-mail] wrote:
> Could you provide any instructions to reproduce this issue please?
> Software versions, etc.

Our buildslave uses KVM under Ubuntu 10.04. The underlying QEMU version 
is given as 0.12.3 with SeaBIOS 0.5.1.

- Colin

Then it looks like a specific libvirt issue with all cmake builds.

On Wednesday, October 05, 2011 12:30 AM, "Colin Finck" [e-mail] wrote:
> Bernd Blaauw [e-mail] wrote:
> > Could you provide any instructions to reproduce this issue please?
> > Software versions, etc.
>
> Our buildslave uses KVM under Ubuntu 10.04. The underlying QEMU version 
> is given as 0.12.3 with SeaBIOS 0.5.1.
> Colin
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev
-- 
With best regards
Caemyr

"Cameron Gutman" [e-mail] писал(а) в своём письме Tuesday, October 04, 2011 12:08 PM:
> But why is changing a build system "ALWAYS dramatic"? Why do we have
> issues like this?

Its because Linux bot conversion was not done they way it should be.
> This should have been done at least in 2 steps, not in one giant mess as we have done.

On my bot we had cmake builder added on-demand the moment CMake builds were mature enough to produce ISO. The switch to default CMake happenend here before the summer iirc and new BE is being tested since September. I think its enough steps.
> We absolutely cannot have bugs introduced due to our build system.

Good to know we can have weeks-lasting breakages from other components. Only build system must be perfect!
> The mere fact that we have to deal with this only detracts from the
> development of real code. I took a good hour trying figure out why the
> hell I couldn't boot trunk when I was trying to test my ACPI fixes. The
> the Vbox testbot was doing fine, yet I couldn't boot my builds on Vbox.
> Only after reverting commits on rbuild and downloading the latest CMake
> build did I realize that only CMake builds would actually boot HEAD
> source code.

First of all, ACPI is not the best example to bring out in this particular case. Secondly, we have two buildbots, and VBox one was running CMake builds when you tested ACPI. Thirdly, it only proves what i said before, running rbuild and cmake side by side is the thing responsible for distractions from ROS development. It uses our resources and time, in your example i would HAVE to be running twice the builds, for both cmake and rbuild, something i simply have no hardware to perform. And what you opt for IS running them side by side for much longer than I would want to.
> I know I'm going to get "Well you should have used CMake,
> then it would've been fine," but CMake can't even properly do
> multithreaded compilation. My CPU usage using CMake is about 45%, while
> CPU usage while using rbuild is close to 100% for the majority of the
> build, and the build times reflect this.

I am sorry to say that, but this problem is entirely your fault. Had you come out and tell someone about it, we would have a solution for you long ago. We had this issue in windows cmake and it was fixed long ago... you cannot expect 4 minute ReactOS build with only one core?
> I look forward to adopting CMake but only when it's ready, and from the looks of the slowness and
> regressions, it's definitely not ready.

We cannot fix any issues that are not reported and i dont recall you reporting anything in particular, besides some occasional comments that CMake is bad and is not working for you, only when the transition was nearing. In my opinion, some steps will have to be forced, and this switch is one of them. The longer we run rbuild and cmake side by side, the more regressions we are likely to create! The longer we go like that, more time and effort is needed to actually complete the goal, simply because you and others are reluctant to switch. It is you who should know best that drastic steps are sometimes necessary to finish up stuff.
> Regards,
> Cameron
-- 
With best regards
Caemyr

[Timo Kreuzer]

Am 04.10.2011 02:53, schrieb Colin Finck:
> Hello all,
>
> The "Trunk_x86_GCCLin Debug" Buildslave has been migrated to
> RosBE-Unix 2.0-RC1. All builds starting with r53983 are now created
> using a CMake environment.
>
> Unfortunately, the CMake builds don't boot under KVM on the same
> machine, but apparently for everybody else. To give you an idea what I
> mean by "don't boot":
> http://img830.imageshack.us/img830/5145/image1iz.png
>
> If nobody has an idea how to debug or fix this, I'm afraid I have to
> revert everything back to the rbuild scripts until a proper solution
> is found.

Can't we have both buildbots and let rbuild one be the default one and 
have cmake only build on demand.
This way we have a better chance to really fix the issue and see if it 
worked.

[Colin Finck]

I have now reverted everything back to the RBuild scripts.
Will set up a side-by-side solution with manual CMake builds as soon as 
possible - unless somebody solves all problems in the meantime :)

Note that we don't just have a testing problem now, but also issues with 
PCH files (see 
http://build.reactos.org/builders/Trunk_x86_GCCLin%20Debug/builds/9903/steps...).
The old ccache 2.4 has already been replaced by a fresh version 3.1.6 
today, with apparently no effect on our problems though.

- Colin

[Bernd Blaauw]

Op 5-10-2011 1:29, Colin Finck schreef:
> I have now reverted everything back to the RBuild scripts.
> Will set up a side-by-side solution with manual CMake builds as soon as
> possible - unless somebody solves all problems in the meantime :)

If I recall correctly Timo recently managed to fix a ReactOS CD 
bootsector or something so KVM would work again. If that's all that 
remained it might be worthwile trying again.

As ReactOS now supports ACPI, is it possible to optionally let each 
stage shutdown the (virtual) machine ?
Then you could use individual QEMU commands to launch each phase of 
ReactOS with different settings. Perhaps something for the unattend.inf 
thing. Benefit might be some form of automated regression testing.

pseudo syntax:
@echo off
echo Starting trunk install from virtual CD:
start /w qemu L . -m 32 -hda roshdd.img -boot d -cdrom trunk.iso
echo Starting phase 2 configuration:
start /w qemu L . -m 80 -hda roshdd.img -boot c
echo Starting ReactOS
start /w qemu L . -m 64 -hda roshdd.img -boot c

[Colin Finck]

Colin Finck [e-mail] wrote:
> Unfortunately, the CMake builds don't boot under KVM on the same
> machine, but apparently for everybody else. To give you an idea what I
> mean by "don't boot": http://img830.imageshack.us/img830/5145/image1iz.png

Okay, I fixed the side-by-side building and started a new test run with 
r54062 yesterday. Unfortunately, we still have the same result 
(http://build.reactos.org/builders/Linux_AMD64_1%20KVM-CMake-Test/builds/6/st...), 
even after Amine's revert back to RSYM symbols.

So, what else could be wrong now? :-)

To force a build yourself on the CMake slave, don't forget to specify 
the revision for now. Will fix this later.

- Colin