Fwd: buildbot failure in ReactOS on Windows_AMD64_1 VBox-Test

It seems that any crash in APITESTS on vbox testbot is causing
rosautotest to redo from start. We can see this in stdio log:
http://build.reactos.org/builders/Windows_AMD64_1%20VBox-Test/builds/6323/s…
(just search for "restarting" word). APITESTS are repated and such log
as a whole is messing up testman, which is treating it like two
submissions for the same revision. It accepts the first one (incomplete
due to crash) and rejects the second one, reporting a duplicate: "ERROR!
submit(13111, 518, ...) - We already have a result for this test suite
in this test run! "Windows_AMD64_1 VBox-Test""

As we all know, we should expect tests to fail and even crash (this is
what they are designed for), hence the error is in rosautotest and
testman.

Rosautotest, strangely, is affected only on vbox, on kvm the apitest
crash is not causing a restart of test run. Winetests, if i recall
correctly, were never affected in such way.

Testman should not reject a submission after its saved up and revision -
blocked.

I am aware that some will be tempted to comment out the test in
question, but this is not a proper solution to this situation. Other
apitest will crash in the future, bringing the whole thing back. Same as
with regressions, lack of tests due to such issue, will bite us back in
the future, when least expected and most needed. This is why I humbly
request this issue to be solved properly, once and for all.

With best regards

----- Original message -----
From: buildbot(a)reactos.org
To: caemyr(a)myopera.com
Subject: buildbot failure in ReactOS on Windows_AMD64_1 VBox-Test
Date: Tue, 14 Aug 2012 00:23:24 +0000

The Buildbot has finished a build on builder Windows_AMD64_1 VBox-Test
while building ReactOS.
Full details are available at:
 http://build.reactos.org/builders/Windows_AMD64_1%20VBox-Test/builds/6324

Buildbot URL: http://build.reactos.org/

Buildslave for this Build: Windows_AMD64_2

Build Reason: Triggerable(Windows_AMD64_1 VBox-Test Trigger)
Build Source Stamp: 57075
Blamelist: akhaldi

BUILD FAILED: failed Cleanup

sincerely,
 -The Buildbot



-- 
With best regards
Caemyr

[Pierre Schweitzer]

Hi,

Le mardi 14 août 2012 à 02:48 +0200, caemyr(a)myopera.com a écrit :
>   Rosautotest, strangely, is affected only on vbox, on
> kvm the apitest
>  crash is not causing a restart of test run. Winetests, if i recall
>  correctly, were never affected in such way. 
Nope. KVM exposes the same issue. Well, at least, randomly.

>   Testman should not reject a submission after its saved
> up and revision -
>  blocked. 
It has. This prevents for storing partial runs, or not consistant runs.
Furthermore, I'm against hacking testman because rosautotest is failing.
That's non-sense.

With my best regards,
-- 
Pierre Schweitzer <pierre(a)reactos.org>
Systems Administrator
ReactOS Foundation

On Tue, Aug 14, 2012, at 08:02 AM, Pierre Schweitzer wrote:
>   Nope. KVM exposes the same issue. Well, at least,
> randomly. 
Are you sure? I never seen KVM failing on test log upload.

>   It has. This prevents for storing partial runs, or not
> consistant runs.
>  Furthermore, I'm against hacking testman because rosautotest is failing.
>  That's non-sense. 
But the whole problem with testman now is that it actually is storing
partial/incomplete runs:
http://www.reactos.org/testman/compare.php?ids=13111
http://www.reactos.org/testman/compare.php?ids=13114

By that its preventing the full test run to be uploaded. Hence its not
just rosautotest, there is partial blame on testman as well.

-- 
With best regards
Caemyr

[Colin Finck]

caemyr(a)myopera.com wrote:
>   It seems that any crash in APITESTS on vbox testbot is
> causing
>  rosautotest to redo from start. 
Apparently, we're dealing with lost cache contents or even filesystem 
corruption here. If you look for "Writing initial journal file", you'll 
see that it appears multiple times in the log while this should only 
happen once at the very beginning.

If somebody has a few minutes to spend, please take a look at 
CJournaledTestList.cpp in rosautotest. Maybe we can add a manual flush 
command here to ensure that the journal is written back to the disk 
before we start testing.

Cheers,

Colin

[Colin Finck]

Colin Finck <colin(a)reactos.org> wrote:
>   If somebody has a few minutes to spend, please take a
> look at
>  CJournaledTestList.cpp in rosautotest. 
I've added flags to bypass caches in r57077, but this doesn't fix the 
problem unfortunately. I'm not sure whether any other (proper) fix is 
even doable in rosautotest itself.
We're terminating the VM right after a crash, so what assumptions can we 
make about its filesystem contents? So far, the journal has been 
properly written back to disk for most VMs, but can we blame the VM if 
this doesn't happen?

This also seems to be a recent issue if I'm not mistaken. Older test 
runs on VirtualBox don't suffer from this problem when looking at BuildBot.

Cheers,

Colin

This issue comes and goes, from what i noticed its triggered by any
crash in APITESTS. Crashes in WINETESTS are not triggereing it.

On Tue, Aug 14, 2012, at 11:28 PM, Colin Finck wrote:
 
>   I've added flags to bypass caches in r57077, but
> this doesn't fix the 
>  problem unfortunately. I'm not sure whether any other (proper) fix is 
>  even doable in rosautotest itself.
>  We're terminating the VM right after a crash, so what assumptions can we 
>  make about its filesystem contents? So far, the journal has been 
>  properly written back to disk for most VMs, but can we blame the VM if
>  this doesn't happen?
>  
>  This also seems to be a recent issue if I'm not mistaken. Older test 
>  runs on VirtualBox don't suffer from this problem when looking at
>  BuildBot.
>  
>  Cheers,
>  
>  Colin
>  
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev 

-- 
With best regards
Caemyr

Guys, this is serious, crash is still present. Without test coverage we
risk slipping regressions in and KVM testbot does not cover ahk app
tests.

-- 
With best regards
Caemyr

[Giannis Adamopoulos]

Hi,
I'm really sorry about the inconvenience. When I committed the test case I didn't
expect it would cause such great problems and thought that testbot would continue
gratefully. So I deactivated the tests till either it doesn't crash in the kernel or
the test process is more solid. 

Now that we hit this problem we should try fixing it before we forget it again.
Perhaps a possible solution would be for rosautotest to read a text file that contains
tests that may cause a crash and run these tests at the end of the test process. Another
solution would be to cause the vm to reboot right after creating the journal. I think that
both solutions could give the vm enough time to write the journal to the disk.

ReactOS Development List <ros-dev(a)reactos.org> wrote on Thu, August 16th, 2012,
10:24 AM:
>   Guys, this is serious, crash is still present. Without
> test coverage we
>  risk slipping regressions in and KVM testbot does not cover ahk app
>  tests.