Re: [ros-dev] [jira] (ROSTESTS-153) gdi32_winetest:bitmap test_mono_bitmap skipped because it takes too long and spams the debug log with too many failures

[Thomas Faber]

On 2015-02-16 10:27, zefklop (JIRA) wrote:
>   zefklop commented on ROSTESTS-153:
>  ----------------------------------
>  
>  This and the advapi32:service disabled test call for (optional) per-test timeout
> parameter in rostests or testman.We can't afford skipping tests, that's an open
> door to all kind of regressions. 
I agree that skipping tests is a really bad idea. It should be used as
a last resort only. And we need a proper process for it** -- I've been
spending way too much time hunting down skipped tests recently, and way
too many of them had their underlying bugs (and sometimes even Jira
issue) fixed without re-enabling the test.

However I don't see a practical way to implement your solution. For one,
the timeout seems like it would have to cause a machine reboot, which
significantly increases test time. I guess rosautotest could keep track
of the time and kill the child process, but I don't know how reliable
that would be.
Secondly and more importantly, killing the test on timeout seems worse
than skipping the offending part, because then any tests running
_after_ the code causing the timeout would never be executed. That means
instead of skipping a test that's known to be problematic, we skip
completely unrelated and innocent tests.

I'm not sure if there's a way to win this. Let me know if you have any
ideas. As it stands I think skipping the offenders is an okay solution
-- we just need to stay very aware of them and make sure to re-test more
regularly than we do now.

Thanks!
-Thomas


** For the record, here's an example process that I try to keep to:
- Create a ROSTESTS bug to indicate that a test was skipped
- Link the ROSTESTS bug to ROSTESTS-125, the Skipped Tests Epic
- Do not #if out the test, instead check for winetest_interactive, and
  skip the test only if it's not set
- Make sure to call skip() with a message that references the ROSTESTS
  bug
- Optionally create a CORE issue for the underlying problem if enough
  information is available; make sure it blocks the ROSTESTS issue
- Do not under any circumstances close the ROSTESTS bug before the test
  code is actually restored and no longer skips the test in question
This allows the issue to be actually found, by checking the Epic, or
grepping for "ROSTESTS-" or "winetest_interactive". Skipped tests
should be re-tested regularly, at the very least every time the test is
Wine-synced.

>  > gdi32_winetest:bitmap test_mono_bitmap skipped
> because it takes too long and spams the debug log with too many failures
> >
> -----------------------------------------------------------------------------------------------------------------------
> >
> >                 Key: ROSTESTS-153
> >                 URL: https://jira.reactos.org/browse/ROSTESTS-153
> >             Project: ReactOS Test Suite
> >          Issue Type: Bug
> >          Components: Wine Tests
> >            Reporter: Thomas Faber
> >            Assignee: Bug Zilla
> >              Labels: HACK
> >
> > If you close this task, that means the winetest_interactive check and skip() in
> test_mono_bitmap must be gone!