Caution, Wine Tests maybe Wrong!

[James Tabor]

Hi all,
When running ntdll_test or ntdll_crosstest with rtl selected, it will fail at
the last test, test_HandleTables. Right at first I checked the wine headers and
noticed the handle table structures are not the same as ros.

B^| Wow, -what- -a- -sur- -prise-.

Convert tests to Ros? 8^>
James

[Alex Ionescu]

James Tabor wrote:
> Hi all,
> When running ntdll_test or ntdll_crosstest with rtl selected, it will 
> fail at
> the last test, test_HandleTables. Right at first I checked the wine 
> headers and
> noticed the handle table structures are not the same as ros.
>
> B^| Wow, -what- -a- -sur- -prise-.
>
> Convert tests to Ros? 8^>
> James

Hi,

When I asked Thomas to re-write the handle table implementation I 
remember that the WINE code was wrong and using incorrect structures, so 
I can confirm that their tests will incorrectly fail.

Best regards,
Alex Ionescu

[Filip Navara]

Alex Ionescu wrote:
> James Tabor wrote:
> > Hi all,
> > When running ntdll_test or ntdll_crosstest with rtl selected, it will 
> > fail at
> > the last test, test_HandleTables. Right at first I checked the wine 
> > headers and
> > noticed the handle table structures are not the same as ros.
> >
> > B^| Wow, -what- -a- -sur- -prise-.
> >
> > Convert tests to Ros? 8^>
> > James
>
> Hi,
>
> When I asked Thomas to re-write the handle table implementation I 
> remember that the WINE code was wrong and using incorrect structures, 
> so I can confirm that their tests will incorrectly fail.
>
> Best regards,
> Alex Ionescu

The test passes correctly on Windows, so it can't be wrong:
rtl: 600111 tests executed, 0 marked as todo, 0 failures.

- Filip

[Alex Ionescu]

Filip Navara wrote:
> The test passes correctly on Windows, so it can't be wrong:
> rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> Filip

The handle structure inconsitencies which I mentionned are true but not 
exposed to the outside APIs, therefore it is entirely normal that the 
WINE test would've worked in Windows, since it seems the problem is 
located in some of our actual exported implementation. My original reply 
was due to my misunderstanding that the test failed due to the structure 
differences. My apoligies.

Best regards,
Alex Ionescu

[James Tabor]

Hi,
Filip Navara wrote:
> Alex Ionescu wrote:
> > James Tabor wrote:
> > > Hi all,
> > > When running ntdll_test or ntdll_crosstest with rtl selected, it will 
> > > fail at
> > > the last test, test_HandleTables. Right at first I checked the wine 
> > > headers and
> > > noticed the handle table structures are not the same as ros.
> > >
> > > B^| Wow, -what- -a- -sur- -prise-.
> > >
> > > Convert tests to Ros? 8^>
> > > James
> >
> > Hi,
> >
> > When I asked Thomas to re-write the handle table implementation I 
> > remember that the WINE code was wrong and using incorrect structures, 
> > so I can confirm that their tests will incorrectly fail.
> >
> > Best regards,
> > Alex Ionescu
>
> The test passes correctly on Windows, so it can't be wrong:
> rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> Filip

I know, that is a problem.
Thanks,
James

[Andrew Murphy]

On 07/08/05, James Tabor [e-mail] 
wrote:
> Hi,
> Filip Navara wrote:
> > Alex Ionescu wrote:
> > > James Tabor wrote:
> > > > Hi all,
> > > > When running ntdll_test or ntdll_crosstest with rtl selected, it will
> > > > fail at
> > > > the last test, test_HandleTables. Right at first I checked the wine
> > > > headers and
> > > > noticed the handle table structures are not the same as ros.
> > > >
> > > > B^| Wow, -what- -a- -sur- -prise-.
> > > >
> > > > Convert tests to Ros? 8^>
> > > > James
> > >
> > > Hi,
> > >
> > > When I asked Thomas to re-write the handle table implementation I
> > > remember that the WINE code was wrong and using incorrect structures,
> > > so I can confirm that their tests will incorrectly fail.
> > >
> > > Best regards,
> > > Alex Ionescu
> >
> > The test passes correctly on Windows, so it can't be wrong:
> > rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> > Filip
>
> I know, that is a problem.
> Thanks,
> James
> _______________________________________________
> Ros-dev mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-dev

[Rob Shearman]

James Tabor wrote:
> Filip Navara wrote:
> > Alex Ionescu wrote:
> > > James Tabor wrote:
> > > > Hi all,
> > > > When running ntdll_test or ntdll_crosstest with rtl selected, it 
> > > > will fail at
> > > > the last test, test_HandleTables. Right at first I checked the wine 
> > > > headers and
> > > > noticed the handle table structures are not the same as ros.
> > > >
> > > > B^| Wow, -what- -a- -sur- -prise-.
> > > >
> > > > Convert tests to Ros? 8^>
> > >
> > > When I asked Thomas to re-write the handle table implementation I 
> > > remember that the WINE code was wrong and using incorrect 
> > > structures, so I can confirm that their tests will incorrectly fail.
> >
> > The test passes correctly on Windows, so it can't be wrong:
> > rtl: 600111 tests executed, 0 marked as todo, 0 failures.
>
> I know, that is a problem.

Is this an acknowledgement that the test is indeed correct? If not, what 
do you want changed in the test? The RtlpMakeHandleAllocated part is 
deliberate and not merely an implementation detail since I have observed 
Windows components outside of NtDll needing this.

Thanks,
Rob

P.S. As the original author of both Wine's handle table code and the 
corresponding test code, I find your sarcastic comment somewhat insulting.

[Alex Ionescu]

Rob Shearman wrote:
> James Tabor wrote:
> > > The test passes correctly on Windows, so it can't be wrong:
> > > rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> >
> > I know, that is a problem.
>
> P.S. As the original author of both Wine's handle table code and the 
> corresponding test code, I find your sarcastic comment somewhat 
> insulting.
> _______________________________________________

Pardon me, but how does James' comment stating that the WINE test 
working on Windows but not on ReactOS is a problem become sarcastic and 
insulting? If anything, it was commending WINE on the quality of its test.

Best regards,
Alex Ionescu

[James Hawkins]

On 8/8/05, Alex Ionescu [e-mail] wrote:
> Rob Shearman wrote:
> > P.S. As the original author of both Wine's handle table code and the
> > corresponding test code, I find your sarcastic comment somewhat
> > insulting.
>
> Pardon me, but how does James' comment stating that the WINE test
> working on Windows but not on ReactOS is a problem become sarcastic and
> insulting? If anything, it was commending WINE on the quality of its test.
> > > > B^| Wow, -what- -a- -sur- -prise-.

I believe this is the comment Rob was referring to.
-- 
James Hawkins

[Rob Shearman]

Alex Ionescu wrote:
> Rob Shearman wrote:
> > James Tabor wrote:
> > > > The test passes correctly on Windows, so it can't be wrong:
> > > > rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> > >
> > > I know, that is a problem.
> >
> > P.S. As the original author of both Wine's handle table code and the 
> > corresponding test code, I find your sarcastic comment somewhat 
> > insulting.
>
> Pardon me, but how does James' comment stating that the WINE test 
> working on Windows but not on ReactOS is a problem become sarcastic 
> and insulting? If anything, it was commending WINE on the quality of 
> its test.

The title of this thread is "Caution, Wine Tests maybe Wrong!". I 
therefore took the line "B^| Wow, -what- -a- -sur- -prise-." to mean 
that James thinks that Wine has gets things like this wrong on a regular 
basis (and incidentally, I don't dispute this, just the manner in which 
it was said). I apologise if I got the wrong end of the stick on this one.

Also, I would like to add that it may be of help to either email the 
author of the code or wine-devel if any ReactOS developers suspect they 
find bugs in Wine code. One further thing to note is that we have a 
framework set up to run tests on a variety of different Windows 
platforms and the results are published on a web site by date tested: 
http://test.winehq.org/data/

It may be helpful for someone to start performing automated tests on 
ReactOS too. Steven, you have been doing some work on fixing tests in 
the past few days: would this be useful, or are you working through the 
tests on a case-by-case basis for the moment?

Thanks,
Rob

[Alex Ionescu]

Rob Shearman wrote:
> Alex Ionescu wrote:
> > Rob Shearman wrote:
> > > James Tabor wrote:
> > > > > The test passes correctly on Windows, so it can't be wrong:
> > > > > rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> > > >
> > > > I know, that is a problem.
> > >
> > > P.S. As the original author of both Wine's handle table code and the 
> > > corresponding test code, I find your sarcastic comment somewhat 
> > > insulting.
> >
> > Pardon me, but how does James' comment stating that the WINE test 
> > working on Windows but not on ReactOS is a problem become sarcastic 
> > and insulting? If anything, it was commending WINE on the quality of 
> > its test.
>
> The title of this thread is "Caution, Wine Tests maybe Wrong!". I 
> therefore took the line "B^| Wow, -what- -a- -sur- -prise-." to mean 
> that James thinks that Wine has gets things like this wrong on a 
> regular basis (and incidentally, I don't dispute this, just the manner 
> in which it was said). I apologise if I got the wrong end of the stick 
> on this one.

Ah, sorry, I didn't even see that comment. I don't even agree that WINE 
gets things wrong on a regular basis, at least not in this context. Your 
structure is "wrong" in a way that doesn't influence external behaviour 
at all. I guess the nice way of saying it would've been "Wow, what a 
surprise, WINE doesn't need to clone internal structures!" which, 
although sarcastic, is a lot more factual and is more a pun towards 
ReactOS ;)
> Also, I would like to add that it may be of help to either email the 
> author of the code or wine-devel if any ReactOS developers suspect 
> they find bugs in Wine code.

We usually send patches, although it's true that kernel32/ntdll sharing 
has been very hard due to the large architectural differences. Mike and 
other developers (afaik) are working on making this more NT/ROS-like, 
and I can't wait until we can share more code. From what I've seen in 
WINE and the WINETEST failures we're getting, our kernel32/ntdll 
user-mode code is a lot worse then WINEs... our original sync was what, 
5 years ago? ;)
> One further thing to note is that we have a framework set up to run 
> tests on a variety of different Windows platforms and the results are 
> published on a web site by date tested: http://test.winehq.org/data/

Nice, Steven never told me about this...thanks a lot!
> It may be helpful for someone to start performing automated tests on 
> ReactOS too. Steven, you have been doing some work on fixing tests in 
> the past few days: would this be useful, or are you working through 
> the tests on a case-by-case basis for the moment?

Actually, Steven has been doing work on forwarding me the bugs, and I've 
been fixing them ;) I'm handling this on a case-by-case basis, but we 
are currently working on a system called SIN, which is a Continous 
Integration System. Casper is leading the project, but it allows us to 
automatically compile builds to ensure that the latest commit doesn't 
break it (in which case it will be automatically removed). A large 
number of us (especially Steven and I) want to make SIN also actually 
boot the image in an emulator, and then run automated winetests. This is 
currently being disputed, since we cannot possibly do this for every 
build, it would slow down the process too much. One of the suggestions 
was to do this only X builds, but I'm not sure if that comprise has been 
approved yet. Defintely, I think you don't have this problem in WINE, 
since you can easily execute WINE in the compiler environment without 
requiring booting from an image in an emulator. But we're working on it! 
We've even thrown the idea of User-Mode ReactOS on the table.
> Thanks,
> Rob

Best regards,
Alex Ionescu

[Steven Edwards]

--- Alex Ionescu [e-mail] wrote:
> Actually, Steven has been doing work on forwarding me the bugs, and I've 
> been fixing them ;) I'm handling this on a case-by-case basis, but we 
> are currently working on a system called SIN, which is a Continous 
> Integration System. Casper is leading the project, but it allows us to 
> automatically compile builds to ensure that the latest commit doesn't 
> break it (in which case it will be automatically removed). A large 
> number of us (especially Steven and I) want to make SIN also actually 
> boot the image in an emulator, and then run automated winetests. This is 
> currently being disputed, since we cannot possibly do this for every 
> build, it would slow down the process too much. One of the suggestions 
> was to do this only X builds, but I'm not sure if that comprise has been 
> approved yet. Defintely, I think you don't have this problem in WINE, 
> since you can easily execute WINE in the compiler environment without 
> requiring booting from an image in an emulator. But we're working on it! 
> We've even thrown the idea of User-Mode ReactOS on the table.

I think it might be possible to do it with QEMU if we accept that we can't run all of the tests to
start with. User32,Gdi32,kernel32,ntdll and advapi32 are the most important ones to start with. I
expect Winetest can run tests for those dlls in less than 2 mins. We can embrace and extend the
Winehq winetest data page a bit and host a page on ReactOS.org to track builds by revision rather
than just date.

Thanks
Steven

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

[Alex Ionescu]

Steven Edwards wrote:
> I think it might be possible to do it with QEMU if we accept that we can't run all of the tests to
> start with. User32,Gdi32,kernel32,ntdll and advapi32 are the most important ones to start with. I
> expect Winetest can run tests for those dlls in less than 2 mins. We can embrace and extend the
> Winehq winetest data page a bit and host a page on ReactOS.org to track builds by revision rather
> than just date.

I would be willing to create a special Test-Version of ReactOS which 
would boot much rapidly. I am thinking in the order of reducing boot 
time to less then 5 seconds. But this isn't a promise and if I ever want 
to do this, it might be in a long time and mostly for my own fun, so 
please don't take this as a thing I'm promising any time soon.
> Thanks
> Steven

Best regards,
Alex Ionescu

[James Tabor]

Hi!
Alex Ionescu wrote:
> Rob Shearman wrote:
> > Alex Ionescu wrote:
> > > Rob Shearman wrote:
> > > > James Tabor wrote:
> > > > > > The test passes correctly on Windows, so it can't be wrong:
> > > > > > rtl: 600111 tests executed, 0 marked as todo, 0 failures.
> > > > >
> > > > > I know, that is a problem.
> > > >
> > > > P.S. As the original author of both Wine's handle table code and the 
> > > > corresponding test code, I find your sarcastic comment somewhat 
> > > > insulting.

Did not know that.
> > > Pardon me, but how does James' comment stating that the WINE test 
> > > working on Windows but not on ReactOS is a problem become sarcastic 
> > > and insulting? If anything, it was commending WINE on the quality of 
> > > its test.
> >
> > The title of this thread is "Caution, Wine Tests maybe Wrong!". I 
> > therefore took the line "B^| Wow, -what- -a- -sur- -prise-." to mean 
> > that James thinks that Wine has gets things like this wrong on a 
> > regular basis (and incidentally, I don't dispute this, just the manner 
> > in which it was said). I apologise if I got the wrong end of the stick 
> > on this one.

No you did not, end of the Stick, no. Don't forget I said "maybe wrong".
I work for Ros, not Wine so I don't know.

Yes, Caution is needed! I'm not just testing one thing at a time. I test
multiple applications at the same time. Including wine tests. Most of the
time they cause bug checks, and kill the system. So "Caution" is the word.

You did answer one question, there are no structures compatible to work
from.
> Ah, sorry, I didn't even see that comment. I don't even agree that WINE 
> gets things wrong on a regular basis, at least not in this context. Your 
> structure is "wrong" in a way that doesn't influence external behaviour 
> at all. I guess the nice way of saying it would've been "Wow, what a 
> surprise, WINE doesn't need to clone internal structures!" which, 
> although sarcastic, is a lot more factual and is more a pun towards 
> ReactOS ;)

Yeah, is there a standard structure we can use together? This was one of
my grips about using Wine Dll's. What if these structures propagated beyond
kernel32 and are used in comdlg or comctrl? This could fu everything!

Blank,
James

[Jakob Eriksson]

Alex Ionescu wrote:
> Actually, Steven has been doing work on forwarding me the bugs, and 
> I've been fixing them ;) I'm handling this on a case-by-case basis, 
> but we are currently working on a system called SIN, which is a 
> Continous Integration System.

I am looking (not actively at the moment) at a similar system, but only 
for winetest.exe.  It would ensure
no patch would add to failed tests in winetest.exe

//Jakob

[Casper Hornstrup]

Hi. Please join the project at http://sin.tigris.org if you want to
discuss the implementation of such a system.

Casper
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of Jakob Eriksson
> Sent: 13. august 2005 23:43
> To: ReactOS Development List
> Subject: Re: [ros-dev] Caution, Wine Tests maybe Wrong!
>
> Alex Ionescu wrote:
> > Actually, Steven has been doing work on forwarding me the bugs, and
> > I've been fixing them ;) I'm handling this on a case-by-case basis,
> > but we are currently working on a system called SIN, which is a
> > Continous Integration System.
>
> I am looking (not actively at the moment) at a similar system, but only
> for winetest.exe.  It would ensure
> no patch would add to failed tests in winetest.exe
>
> //Jakob
> _______________________________________________
> Ros-dev mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-dev

> Hi. Please join the project at http://sin.tigris.org if you want to
> discuss the implementation of such a system.

At the moment I get a HTTP 500 at the address above.

//Jakob

[Casper Hornstrup]

And now?
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of
> [e-mail]
> Sent: 15. august 2005 09:11
> To: ReactOS Development List
> Cc: 'ReactOS Development List'
> Subject: RE: [ros-dev] Caution, Wine Tests maybe Wrong!
> > Hi. Please join the project at http://sin.tigris.org if you want to
> > discuss the implementation of such a system.
>
> At the moment I get a HTTP 500 at the address above.
>
> //Jakob
>
> Ros-dev mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-dev

[Andrew Murphy]

Works here

On 15/08/05, Casper Hornstrup [e-mail] wrote:
> And now?
> > -----Original Message-----
> > From: [e-mail] [mailto:[e-mail]]
>
> On Behalf Of
> > [e-mail]
> > Sent: 15. august 2005 09:11
> > To: ReactOS Development List
> > Cc: 'ReactOS Development List'
> > Subject: RE: [ros-dev] Caution, Wine Tests maybe Wrong!
> > > Hi. Please join the project at http://sin.tigris.org if you want to
> > > discuss the implementation of such a system.
> >
> > At the moment I get a HTTP 500 at the address above.
> >
> > //Jakob
> >
> > Ros-dev mailing list
> > [e-mail]
> > http://reactos.com:8080/mailman/listinfo/ros-dev
>
> Ros-dev mailing list
> [e-mail]
> http://reactos.com:8080/mailman/listinfo/ros-dev

[Steven Edwards]

Hi Rob,

--- Rob Shearman [e-mail] wrote:
> Also, I would like to add that it may be of help to either email the 
> author of the code or wine-devel if any ReactOS developers suspect they 
> find bugs in Wine code. One further thing to note is that we have a 
> framework set up to run tests on a variety of different Windows 
> platforms and the results are published on a web site by date tested: 
> http://test.winehq.org/data/
>
> It may be helpful for someone to start performing automated tests on 
> ReactOS too. Steven, you have been doing some work on fixing tests in 
> the past few days: would this be useful, or are you working through the 
> tests on a case-by-case basis for the moment?

I need to make some changes to winetest and dig in and understand the framework a little better
but long term yes. Right now I am just trying to address the issue of tests causing bugchecks.
There are some apis Winetest uses that are not implemented at this time so I have to stub and or
find some way around.

Thanks
Steven

____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs

[Steven Edwards]

Hi,

--- James Tabor [e-mail] wrote:
> When running ntdll_test or ntdll_crosstest with rtl selected, it will fail at
> the last test, test_HandleTables. Right at first I checked the wine headers and
> noticed the handle table structures are not the same as ros.

Yes the tests might not be 100% correct yet. The structures will need to be checked etc. Working
with the Wine tests will need to be done in stages.

Step One: No wine regression test should cause a bug check no matter how wrong it is. If it does
there is a need of SEH and return value checking

Step Two: Check all the tests on Windows and make sure they pass

Step Three: Make ros pass everything.

Thanks
Steven

____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs