[ros-diffs] [akhaldi] 60074: [WINE/TEST] * Introduce a way to properly mark the todos instead of counting them as failures. Brought to you by Thomas Faber.

[James Tabor]

Hi.

This should be removed, we are not wine and I personally work "hard on"
getting wine todos fixed in ReactOS! This also breaks the DCE test.

Background, GvG and then the old guard back in the day thought the same way
as I do about this issue. We should count the todo's as failures and not
bypass these failues. Why? It may point to a critical bug that could break
functionality somewhere else. That was the reasoning then and I agree with
it. Cheating the tests is just plain cheating......

James

[Jérôme Gardou]

Hi James,

I agree that this doesn't make sense for our in-house modules (WIN32 
subsystem, kernel32, ntdll, you name it...), but this should be used for 
modules that we import from wine, thus permitting us to clearly 
differentiate between features that wine doesn't implement and failed 
tests that fail because of a bug in our underlying modules.

This is not cheating, this is pinpointing issues are ours, and where 
they are wine's.

Regards
Jérôme

James Tabor a écrit :
> Hi.
>
> This should be removed, we are not wine and I personally work "hard on"
> getting wine todos fixed in ReactOS! This also breaks the DCE test.
>
> Background, GvG and then the old guard back in the day thought the same
> way as I do about this issue. We should count the todo's as failures and
> not bypass these failues. Why? It may point to a critical bug that could
> break functionality somewhere else. That was the reasoning then and I
> agree with it. Cheating the tests is just plain cheating......
>
> James
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev

[Timo Kreuzer]

Hi,

I disagree here. The major purpose of these tests is to let us detect
regressions. But when there is all red anyway, noone notices anything.
It's hard to spot what went wrong, when there is 2275 failures instead
of 2272. Also the results are not lost, they are just moved out of the
way of failing tests. You can still see all the todos.
Being able to see where ros fails, but wine succeeds is a valuable
feature, but it's pretty pointless, when it comes at the cost of being
able to quickly and properly detect regressions. Maybe we can have both
and provide a way to disable this, if someone wants to compare against
wine. Patchbot is our friend.

My wish would be that we only had all green in testbot and whenever
something gets red, Alekseys red telephone starts ringing and he
mobilizes the red army... ;-)

Just my 50 cents,
Timo

Am 14.11.2013 19:18, schrieb James Tabor:
> Hi.
>
> This should be removed, we are not wine and I personally work "hard
> on" getting wine todos fixed in ReactOS! This also breaks the DCE test.
>
> Background, GvG and then the old guard back in the day thought the
> same way as I do about this issue. We should count the todo's as
> failures and not bypass these failues. Why? It may point to a critical
> bug that could break functionality somewhere else. That was the
> reasoning then and I agree with it. Cheating the tests is just plain
> cheating......
>
> James
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev

[Aleksey Bragin]

I agree to every Timo's line here, so my 50 cents add up to make a buck 
(or a euro) there.

Regards,
Aleksey

On 14.11.2013 23:10, Timo Kreuzer wrote:
> Hi,
>
> I disagree here. The major purpose of these tests is to let us detect 
> regressions. But when there is all red anyway, noone notices anything. 
> It's hard to spot what went wrong, when there is 2275 failures instead 
> of 2272. Also the results are not lost, they are just moved out of the 
> way of failing tests. You can still see all the todos.
> Being able to see where ros fails, but wine succeeds is a valuable 
> feature, but it's pretty pointless, when it comes at the cost of being 
> able to quickly and properly detect regressions. Maybe we can have 
> both and provide a way to disable this, if someone wants to compare 
> against wine. Patchbot is our friend.
>
> My wish would be that we only had all green in testbot and whenever 
> something gets red, Alekseys red telephone starts ringing and he 
> mobilizes the red army... ;-)
>
> Just my 50 cents,
> Timo
>
> Am 14.11.2013 19:18, schrieb James Tabor:
> > Hi.
> >
> > This should be removed, we are not wine and I personally work "hard 
> > on" getting wine todos fixed in ReactOS! This also breaks the DCE test.
> >
> > Background, GvG and then the old guard back in the day thought the 
> > same way as I do about this issue. We should count the todo's as 
> > failures and not bypass these failues. Why? It may point to a 
> > critical bug that could break functionality somewhere else. That was 
> > the reasoning then and I agree with it. Cheating the tests is just 
> > plain cheating......
> >
> > James

[James Tabor]

Hiding test results make you feel happy, well I'm happy for you.

But,,,,, I can not be held to the false test result from this point
on....... I will close bug reports based on false results that anyone
points to me....... I do not get paid for chasing ghost bugs, wow I do
not get paid at all!!!

That's my two cents......

On 11/14/13, Aleksey Bragin [e-mail] wrote:
> I agree to every Timo's line here, so my 50 cents add up to make a buck
> (or a euro) there.
>
> Regards,
> Aleksey
>
> On 14.11.2013 23:10, Timo Kreuzer wrote:
> > Hi,
> >
> > I disagree here. The major purpose of these tests is to let us detect
> > regressions. But when there is all red anyway, noone notices anything.
> > It's hard to spot what went wrong, when there is 2275 failures instead
> > of 2272. Also the results are not lost, they are just moved out of the
> > way of failing tests. You can still see all the todos.
> > Being able to see where ros fails, but wine succeeds is a valuable
> > feature, but it's pretty pointless, when it comes at the cost of being
> > able to quickly and properly detect regressions. Maybe we can have
> > both and provide a way to disable this, if someone wants to compare
> > against wine. Patchbot is our friend.
> >
> > My wish would be that we only had all green in testbot and whenever
> > something gets red, Alekseys red telephone starts ringing and he
> > mobilizes the red army... ;-)
> >
> > Just my 50 cents,
> > Timo
> >
> > Am 14.11.2013 19:18, schrieb James Tabor:
> > > Hi.
> > >
> > > This should be removed, we are not wine and I personally work "hard
> > > on" getting wine todos fixed in ReactOS! This also breaks the DCE test.
> > >
> > > Background, GvG and then the old guard back in the day thought the
> > > same way as I do about this issue. We should count the todo's as
> > > failures and not bypass these failues. Why? It may point to a
> > > critical bug that could break functionality somewhere else. That was
> > > the reasoning then and I agree with it. Cheating the tests is just
> > > plain cheating......
> > >
> > > James

[James Tabor]

As usual the real issue get sides stepped into oblivion.

The reason for this discussion was why did the wine user32 dce test
change after r60074? How does this relate to test bot strangeness and
mixed results. Good example is CORE-7574 , how did a change in one
test file msg.c effect tests in sysparams.c?

On 11/14/13, James Tabor [e-mail] wrote:
> Hiding test results make you feel happy, well I'm happy for you.
>
> But,,,,, I can not be held to the false test result from this point
> on....... I will close bug reports based on false results that anyone
> points to me....... I do not get paid for chasing ghost bugs, wow I do
> not get paid at all!!!
>
> That's my two cents......
>
> On 11/14/13, Aleksey Bragin [e-mail] wrote:
> > I agree to every Timo's line here, so my 50 cents add up to make a buck
> > (or a euro) there.
> >
> > Regards,
> > Aleksey
> >
> > On 14.11.2013 23:10, Timo Kreuzer wrote:
> > > Hi,
> > >
> > > I disagree here. The major purpose of these tests is to let us detect
> > > regressions. But when there is all red anyway, noone notices anything.
> > > It's hard to spot what went wrong, when there is 2275 failures instead
> > > of 2272. Also the results are not lost, they are just moved out of the
> > > way of failing tests. You can still see all the todos.
> > > Being able to see where ros fails, but wine succeeds is a valuable
> > > feature, but it's pretty pointless, when it comes at the cost of being
> > > able to quickly and properly detect regressions. Maybe we can have
> > > both and provide a way to disable this, if someone wants to compare
> > > against wine. Patchbot is our friend.
> > >
> > > My wish would be that we only had all green in testbot and whenever
> > > something gets red, Alekseys red telephone starts ringing and he
> > > mobilizes the red army... ;-)
> > >
> > > Just my 50 cents,
> > > Timo
> > >
> > > Am 14.11.2013 19:18, schrieb James Tabor:
> > > > Hi.
> > > >
> > > > This should be removed, we are not wine and I personally work "hard
> > > > on" getting wine todos fixed in ReactOS! This also breaks the DCE test.
> > > >
> > > > Background, GvG and then the old guard back in the day thought the
> > > > same way as I do about this issue. We should count the todo's as
> > > > failures and not bypass these failues. Why? It may point to a
> > > > critical bug that could break functionality somewhere else. That was
> > > > the reasoning then and I agree with it. Cheating the tests is just
> > > > plain cheating......
> > > >
> > > > James