Hello everyone.
I think there is alot of time and energy being wasted regarding the Testing Coordinator role, and it's responsibilities. First the responsibilites need to be solidified. When I started acting in the TC role, this was the outline I was shown by Alex, and agreed to perform:
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator
I am pretty pleased with the description, and have managed to balace performing these responsibilities with having a real life, wife and child included. Seems everyone agrees I am doing acceptably.
I've tried to review the stability of ReactOS about once a week, since it takes about 4 to 5 solid hours to build debug and release builds, install them, load test applications and check out regressions, bugs, and newly working features.
I've tried to keep the bugzilla database up to date, and filled with "Good Bugs", and I even wrote up a wiki article on how to do so. http://www.reactos.org/wiki/index.php/File_Bugs. I have tried to steer people away from reporting bugs in the IRC channel and the forums. I am usually available on IRC, and occationally surf the forums for bug reports and reminders.
The last part of the obligations of the TC is the working applicaitons list, and when Alex and I discussed this, he came up with the example listed, but it was understood that the format of list was flexible. This is sort of what the "State of the Repository" emails have been about.
I am collecting notes on a testing method than can be applied by our more inclined users, and a way to integrate a "testing database" that can show the status of apps using reports submitted by users or a testing team. I am also working on updating the parts of the wiki that involves testing in some manner, including building the source (and setting up the environment), using virtual machines to test, debugging, filing bugs and so on..
Please review and discuss. I would like to have the vote as soon as possible.
Thanks for everyone's support, WD
-- <arty> don't question it ... it's clearly an optimization
First the responsibilites need to be solidified. When I started acting in the TC role, this was the outline I was shown by Alex, and agreed to perform:
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator
The only thing that needs changing in that description is remove the sentences
"The TC can decide as his will which bugs to promote to Blocker status (thereby blocking a new release), or to demote bugs below that status."
and
"However, in the rare case where it is clear to a majority of developers that the decision to block a release is unwarranted, a group of no less then three developers, with the approval from a board member, may override the TC's decision. Such actions should never have to happen however, and communicaton is highly recommended in order to reach a moderated debate."
In my opinion, the developers should determine if a release should be blocked, not the TC. This is different from giving the developers the right to override the TC. I'm not sure why "a board member" is mentioned here, the project is primarily run by the developers, not the foundation board.
It would be nice if the TC could provide a list (shortly after feature freeze) of the biggest problems the testing team found. I, as a developer, would certainly use that list to prioritize what bugs I'm going to work on during feature freeze. I do reserve the right however to skip certain bugs, because frankly I don't always agree with the classification (example: bug 880 which is currently classified as "blocker", but in my view it should be "minor" since there is a very easy work around available).
Gé van Geldorp.
On 10/16/05, Ge van Geldorp gvg@reactos.org wrote:
The only thing that needs changing in that description is remove the sentences
"The TC can decide as his will which bugs to promote to Blocker status (thereby blocking a new release), or to demote bugs below that status."
Complete control over Bugzilla was the *only* power specifically granted to the TC, and it's virtually useless since I'm pretty sure all Bugzilla users are allowed to chage the severity of a bug. So this was just a policy change saying the TC gets the final say on bugs. Removing this power simply makes the TC a normal user again.
and
"However, in the rare case where it is clear to a majority of developers that the decision to block a release is unwarranted, a group of no less then three developers, with the approval from a board member, may override the TC's decision. Such actions should never have to happen however, and communicaton is highly recommended in order to reach a moderated debate."
This was just added as a safety, in case the TC gets disagreeable. I have always tried to communicate the status of bugs and get developer input, such as I was doing when all of these issues came up.
In my opinion, the developers should determine if a release should be blocked, not the TC. This is different from giving the developers the right to override the TC. I'm not sure why "a board member" is mentioned here, the project is primarily run by the developers, not the foundation board.
These would be the same developers that break the build without knowing it half the time, and wouldn't fix some bugs unless harrassed.
It would be nice if the TC could provide a list (shortly after feature freeze) of the biggest problems the testing team found. I, as a developer, would certainly use that list to prioritize what bugs I'm going to work on during feature freeze. I do reserve the right however to skip certain bugs, because frankly I don't always agree with the classification (example: bug 880 which is currently classified as "blocker", but in my view it should be "minor" since there is a very easy work around available).
So you would like the TC to toil away testing applications on builds, making lists of what works and what doesn't, just to have no power to get anything actually fixed? Frankly I tire of sitting on IRC begging people to fix bugs.
880 is not a blocker against 0.2.8, and I was going to reclassify it as such. It's not a minor bug to me, Gé, since I boot to cmd and this breaks ReactOS rather badly for me. I know cmd as a shell is not really supported, so I wasn't going to press it.
Does anyone agree with Gé?
WD
-----Original Message----- From: ros-dev-bounces@reactos.org [mailto:ros-dev-bounces@reactos.org] On Behalf Of WaxDragon Sent: 16. oktober 2005 15:22 To: ReactOS Development List Subject: Re: [ros-dev] Prelude to voting for the Testing Coordinator Roles andResponsibilities.
It would be nice if the TC could provide a list (shortly after feature freeze) of the biggest problems the testing team found. I, as a developer, would certainly use that list to prioritize what bugs I'm going to work on during feature freeze. I do reserve the right however to skip certain bugs, because frankly I don't always agree with the classification (example: bug 880 which is currently classified as "blocker", but in my view it should be "minor" since there is a very easy work around available).
So you would like the TC to toil away testing applications on builds, making lists of what works and what doesn't, just to have no power to get anything actually fixed? Frankly I tire of sitting on IRC begging people to fix bugs.
IMHO the TC shouldn't be able to tell people what to do. Nobody, not even our Project Coordinator should be able to do that. The project is still made up of volunteer people and nobody is getting paid to work on ReactOS. If you can convince someone to fix a problem then fine. Get a date for when it is expected to be fixed and ask here if we should delay the release for a few days. If not then just go ahead with the release as there might not be anyone that will fix it for several months. Just like everyone else, you have at least the following options to get something done: 1. Implement it yourself 2. Convince someone else to do it 3. Pay someone else to do it
Casper
On 10/16/05, Casper Hornstrup ch@csh-consult.dk wrote:
IMHO the TC shouldn't be able to tell people what to do. Nobody, not even our Project Coordinator should be able to do that. The project is still made up of volunteer people and nobody is getting paid to work on ReactOS. If you can
I agree.
Jason
From: WaxDragon
So you would like the TC to toil away testing applications on builds, making lists of what works and what doesn't, just to have no power to get anything actually fixed?
No, that's not what I said. I said I would carefully look at your list and basically follow it. However, I do want an escape route. After all, it's my time I invest. And of course, I'm more than willing to talk about priorities with the TC.
Frankly I tire of sitting on IRC begging people to fix bugs.
I certainly hope you won't have to beg. When weird_w asked me last week to look at the SBITs font problem I hope I didn't make him feel like a begger. He just explained to me why it was important to him that this bug got fixed and I put it on the top of my list. The same with bug 880. To me it seems a minor annoyance that you have to press Alt-Tab when returning from graphics mode to restore the text screen properly, but I understand that it's more than annoying to you. Ok, I dig that and will fix it this week. Please understand that sometimes you need to provide a little background to make me grasp exactly why it's such a big problem. As far as I know, I've always treated requests from fellow developers (and I include you in the developers) with higher priority than requests from Joe Random User. But you need to give me some leeway to have fun of my own too. Now, I like debugging and fixing stuff (I see a bug as a puzzle to be solved), but picking up a bug from Bugzilla has this distinct "fixing other peoples problems" feel to it. Usually I have a pet project going on (right now it's FireFox) and often I find it even more fun to work on that than on fixing other peoples bugs. I need to have some fun so I won't burn out.
From: Casper Hornstrup
IMHO the TC shouldn't be able to tell people what to do. Nobody, not even our Project Coordinator should be able to do that. The project is still made up of volunteer people and nobody is getting paid to work on ReactOS. If you can convince someone to fix a problem then fine. Get a date for when it is expected to be fixed and ask here if we should delay the release for a few days. If not then just go ahead with the release as there might not be anyone that will fix it for several months.
Wow, I couldn't have said it better. That is exactly what I meant.
Ge van Geldorp.
On 10/16/05, Ge van Geldorp gvg@reactos.org wrote:
From: WaxDragon
So you would like the TC to toil away testing applications on builds, making lists of what works and what doesn't, just to have no power to get anything actually fixed?
No, that's not what I said. I said I would carefully look at your list and basically follow it. However, I do want an escape route. After all, it's my time I invest. And of course, I'm more than willing to talk about priorities with the TC.
I understand it's your time, I've always tried to keep that in mind when talking with the developers about bugs. That is also why I spend some of my own time trying to comprehend and document the bug beforehand.
Frankly I tire of sitting on IRC begging people to fix bugs.
I certainly hope you won't have to beg. When weird_w asked me last week to look at the SBITs font problem I hope I didn't make him feel like a begger. He just explained to me why it was important to him that this bug got fixed and I put it on the top of my list. The same with bug 880. To me it seems a minor annoyance that you have to press Alt-Tab when returning from graphics mode to restore the text screen properly, but I understand that it's more than annoying to you. Ok, I dig that and will fix it this week. Please understand that sometimes you need to provide a little background to make me grasp exactly why it's such a big problem. As far as I know, I've always treated requests from fellow developers (and I include you in the developers) with higher priority than requests from Joe Random User. But you need to give me some leeway to have fun of my own too. Now, I like debugging and fixing stuff (I see a bug as a puzzle to be solved), but picking up a bug from Bugzilla has this distinct "fixing other peoples problems" feel to it. Usually I have a pet project going on (right now it's FireFox) and often I find it even more fun to work on that than on fixing other peoples bugs. I need to have some fun so I won't burn out.
Again, I understand. I take breaks from bugzilla and work on other pet projects too. The SBIT problem is an old one, and wierd_w had been on the channel several times asking about it, without much response, but he pressed on and kept coming back. After hearing his findings several times, I pressed him to file a specific bug, and you were quick to pick it up and resolve it. I was were happy, as was wierd_w. I've appreciated the work you have done for me, Ge, and I have tried to not hassle you.
From: Casper Hornstrup
IMHO the TC shouldn't be able to tell people what to do. Nobody, not even our Project Coordinator should be able to do that. The project is still made up of volunteer people and nobody is getting paid to work on ReactOS. If you can convince someone to fix a problem then fine. Get a date for when it is expected to be fixed and ask here if we should delay the release for a few days. If not then just go ahead with the release as there might not be anyone that will fix it for several months.
Wow, I couldn't have said it better. That is exactly what I meant.
We let two specific blockers ship in the last release after discussing with with the devs on IRC. There are "big" bugs still in the tree, and I under stand that some of them will take a long time to get fixed. I understand we cannot force people to work on anything. I sent out a *tentative* blocker list for 0.2.8 and all I got was attacks. I was just trying to prompt discussion about the next release, but here we are.
The main issue from my standpoint is that the Release Coordinator should work with the Testing Coordinator. Brandon and I were trying to work with Robert to train another on doing releases, and when I got back that afternoon from shopping, RC1 was built and up on sourceforge with a rather large bug. Had we simply waited a day, we would have been able to ship an RC1 without that single bug. I just wanted us to look good.
I think the TC is an important position that ReactOS needs, due to it's complexity. However, I have decided to no longer continue as TC, if I was even TC to start with.
It's arguments like these that hurt ReactOS more than anything.
WD
-----Original Message----- From: ros-dev-bounces@reactos.org [mailto:ros-dev-bounces@reactos.org] On Behalf Of WaxDragon Sent: 17. oktober 2005 02:22 To: ReactOS Development List Subject: Re: [ros-dev] Prelude to voting for the Testing Coordinator Roles andResponsibilities.
The main issue from my standpoint is that the Release Coordinator should work with the Testing Coordinator. Brandon and I were trying to work with Robert to train another on doing releases, and when I got back that afternoon from shopping, RC1 was built and up on sourceforge with a rather large bug. Had we simply waited a day, we would have been able to ship an RC1 without that single bug. I just wanted us to look good.
I think the TC is an important position that ReactOS needs, due to it's complexity. However, I have decided to no longer continue as TC, if I was even TC to start with.
It's arguments like these that hurt ReactOS more than anything.
WD
[CSH] So, unless you get the right to block a release on your own, you won't be TC?
TC and RC can work together, but blocking a release indefinitely is not acceptable IMHO. Get a new release date, tell everyone about the new release date (and why the release is delayed). If people disagree, they can call for a vote on the subject to settle the matter.
Casper
RC working without TC is a nonsense.
RC: "We release" TC: "There are lots of bugs" RC: "I don't care, release date is already set".
Does this look serious? I don't think so. Indefinite delaying of the release due to bugs also shouldn't be allowed, if bugs aren't blockers of course.
Before, I thought that RC and TC position could be merged, but now I don't think it's good option - project is growing quickly.
Bytheway, can someone please point me how to find a thread with elections of Release Coordinator? Our email archive doesn't support search, I tried google, but didn't find it. We usually had Release Engineer? Or is it just rename?
WBR, Aleksey Bragin.
----- Original Message ----- From: "WaxDragon" waxdragon@gmail.com To: "ReactOS Development List" ros-dev@reactos.org Sent: Monday, October 17, 2005 4:21 AM Subject: Re: [ros-dev] Prelude to voting for the Testing Coordinator Roles andResponsibilities.
The main issue from my standpoint is that the Release Coordinator should work with the Testing Coordinator. Brandon and I were trying to work with Robert to train another on doing releases, and when I got back that afternoon from shopping, RC1 was built and up on sourceforge with a rather large bug. Had we simply waited a day, we would have been able to ship an RC1 without that single bug. I just wanted us to look good.
I think the TC is an important position that ReactOS needs, due to it's complexity. However, I have decided to no longer continue as TC, if I was even TC to start with.
It's arguments like these that hurt ReactOS more than anything.
WD
From: WaxDragon
I understand it's your time, I've always tried to keep that in mind when talking with the developers about bugs. That is also why I spend some of my own time trying to comprehend and document the bug beforehand.
And I understand and appreciate that. After all, that's YOUR time :-)
Again, I understand. I take breaks from bugzilla and work on other pet projects too. The SBIT problem is an old one, and wierd_w had been on the channel several times asking about it, without much response, but he pressed on and kept coming back. After hearing his findings several times, I pressed him to file a specific bug
Yeah, that's somewhat of a problem. People notify us of bugs all over the place (IRC, mailing list, forum), it's difficult to keep track of them that way. You have been doing a superb job of getting them into Bugzilla where we can keep track of them.
I've appreciated the work you have done for me
No no, please don't think of it as work I did for YOU, implying you should be grateful or something like that. I do the work for the project.
I have tried to not hassle you.
Feel free to hassle me every now and then. I need a kick under the butt as much as anyone else sometimes. I just wanted to explain that sometimes I won't react (hmm, that's funny, I actually typed "reactos" there and had to go back and change it <g>) to requests immediately, I just might be working on something else at that moment.
We let two specific blockers ship in the last release after discussing with with the devs on IRC. There are "big" bugs still in the tree, and I under stand that some of them will take a long time to get fixed. I understand we cannot force people to work on anything. I sent out a *tentative* blocker list for 0.2.8 and all I got was attacks.
Not entirely true. You listed 5 bugs:
805 - Shutdown issues Hartmutt has provided a fix which I think takes the "blocker" status away from it (it doesn't solve the root cause, but I think it fixes the stack overflow which made Qemu hang so hard). I think I found a final solution, it's attached to the bug, just waiting for comments by you and/or Gunnar.
493 - excessive repainting in tree-view controls I don't agree this is a major/critical bug, but I do intend to commit the fixes attached to the bug.
688 - AllocConsole fails on real hardware due to i8042prt I've tried very hard to reproduce this, and I understand at least Martin tried that too. It is obviously a real bug, multiple people have reported it, but when you can't reproduce it it's very hard to even try to fix it (especially since this is in a part of the code I haven't worked on before).
703 - ReactOS is a fat big. Minimum mem reqs exceeded. At least some work has been done on this.
880 - Some dlls stop display output when starting in CLI I'll fix it this week
I'd say your list did spur a lot of action. It just might not be immediately visible.
I was just trying to prompt discussion about the next release, but here we are.
From my point of view, entering feature freeze for 0.2.8 made me switch from
"pet project" mode to "bug fixing" mode. I argue that starting a release is the kick under the butt we sometimes need to get long-standing bugs squashed.
The main issue from my standpoint is that the Release Coordinator should work with the Testing Coordinator. Brandon and I were trying to work with Robert to train another on doing releases, and when I got back that afternoon from shopping, RC1 was built and up on sourceforge with a rather large bug.
Yes, that is what RC1 is for. It is supposed to be made immediately after branching, when entering feature freeze. It's status is basically no more than that of a random svn version. Perhaps "RC1" is a misnomer. The problem we had when deciding on the name is that calling it "Beta" would create great confusion too: "What, has the ReactOS project switched from Alpha status to Beta now? Wow, they must be almost done". Perhaps we should call it "FF" (Feature Freeze) instead and rename RC2 to "CF" (Code Freeze)?
Had we simply waited a day, we would have been able to ship an RC1 without that single bug. I just wanted us to look good.
The point is to make the final release look as good as possible. RC1 is just a throw-away snapshot. I'm not sure which "single bug" you are referring to, AFAIK none of the bugs listed above was closed yesterday (one day after RC1 shipment).
I think the TC is an important position that ReactOS needs, due to it's complexity.
You have certainly convinced me of that, by setting the example what a TC is capable of.
However, I have decided to no longer continue as TC, if I was even TC to start with.
I am really sorry to hear this and hope you will reconsider.
It's arguments like these that hurt ReactOS more than anything.
I'd say not having arguments like these is going to hurt far more in the long term. I expect the outcome of this discussion to be that everyone involved is clear about the roles of TC, RC and developers during a release cycle and exactly how our release process works. If we don't discuss it, we will have mismatched expectations on every release to come. Discussions tend to get heated sometimes, that just shows how much the people involved care about the project. The one thing that does hurt ReactOS is discussions not being conducted in a civilized, rational way, instead being a flame war of personal attacks. And yes, I realize that I have been guilty of those personal attacks too. That's why I started my original mail "I'm trying to keep this as non-personal and technical as possible." Obviously I failed in that (again) and I very much regret that.
Gé van Geldorp.
Why not call it a "Preview Release" instead of a "Release Candidate"? The Firefox team have often said Preview Releases (Such as the first few 1.4 releases) were nothing more than blessed Nightly Builds.
Or even, "Community ReactOS Preview".
On 10/17/05, Ge van Geldorp gvg@reactos.org wrote:
From: WaxDragon
I understand it's your time, I've always tried to keep that in mind when talking with the developers about bugs. That is also why I spend some of my own time trying to comprehend and document the bug beforehand.
And I understand and appreciate that. After all, that's YOUR time :-)
Again, I understand. I take breaks from bugzilla and work on other pet projects too. The SBIT problem is an old one, and wierd_w had been on the channel several times asking about it, without much response, but he pressed on and kept coming back. After hearing his findings several times, I pressed him to file a specific bug
Yeah, that's somewhat of a problem. People notify us of bugs all over the place (IRC, mailing list, forum), it's difficult to keep track of them that way. You have been doing a superb job of getting them into Bugzilla where we can keep track of them.
I've appreciated the work you have done for me
No no, please don't think of it as work I did for YOU, implying you should be grateful or something like that. I do the work for the project.
I have tried to not hassle you.
Feel free to hassle me every now and then. I need a kick under the butt as much as anyone else sometimes. I just wanted to explain that sometimes I won't react (hmm, that's funny, I actually typed "reactos" there and had to go back and change it <g>) to requests immediately, I just might be working on something else at that moment.
We let two specific blockers ship in the last release after discussing with with the devs on IRC. There are "big" bugs still in the tree, and I under stand that some of them will take a long time to get fixed. I understand we cannot force people to work on anything. I sent out a *tentative* blocker list for 0.2.8 and all I got was attacks.
Not entirely true. You listed 5 bugs:
805 - Shutdown issues Hartmutt has provided a fix which I think takes the "blocker" status away from it (it doesn't solve the root cause, but I think it fixes the stack overflow which made Qemu hang so hard). I think I found a final solution, it's attached to the bug, just waiting for comments by you and/or Gunnar.
493 - excessive repainting in tree-view controls I don't agree this is a major/critical bug, but I do intend to commit the fixes attached to the bug.
688 - AllocConsole fails on real hardware due to i8042prt I've tried very hard to reproduce this, and I understand at least Martin tried that too. It is obviously a real bug, multiple people have reported it, but when you can't reproduce it it's very hard to even try to fix it (especially since this is in a part of the code I haven't worked on before).
703 - ReactOS is a fat big. Minimum mem reqs exceeded. At least some work has been done on this.
880 - Some dlls stop display output when starting in CLI I'll fix it this week
I'd say your list did spur a lot of action. It just might not be immediately visible.
I was just trying to prompt discussion about the next release, but here we are.
From my point of view, entering feature freeze for 0.2.8 made me switch from
"pet project" mode to "bug fixing" mode. I argue that starting a release is the kick under the butt we sometimes need to get long-standing bugs squashed.
The main issue from my standpoint is that the Release Coordinator should work with the Testing Coordinator. Brandon and I were trying to work with Robert to train another on doing releases, and when I got back that afternoon from shopping, RC1 was built and up on sourceforge with a rather large bug.
Yes, that is what RC1 is for. It is supposed to be made immediately after branching, when entering feature freeze. It's status is basically no more than that of a random svn version. Perhaps "RC1" is a misnomer. The problem we had when deciding on the name is that calling it "Beta" would create great confusion too: "What, has the ReactOS project switched from Alpha status to Beta now? Wow, they must be almost done". Perhaps we should call it "FF" (Feature Freeze) instead and rename RC2 to "CF" (Code Freeze)?
Had we simply waited a day, we would have been able to ship an RC1 without that single bug. I just wanted us to look good.
The point is to make the final release look as good as possible. RC1 is just a throw-away snapshot. I'm not sure which "single bug" you are referring to, AFAIK none of the bugs listed above was closed yesterday (one day after RC1 shipment).
I think the TC is an important position that ReactOS needs, due to it's complexity.
You have certainly convinced me of that, by setting the example what a TC is capable of.
However, I have decided to no longer continue as TC, if I was even TC to start with.
I am really sorry to hear this and hope you will reconsider.
It's arguments like these that hurt ReactOS more than anything.
I'd say not having arguments like these is going to hurt far more in the long term. I expect the outcome of this discussion to be that everyone involved is clear about the roles of TC, RC and developers during a release cycle and exactly how our release process works. If we don't discuss it, we will have mismatched expectations on every release to come. Discussions tend to get heated sometimes, that just shows how much the people involved care about the project. The one thing that does hurt ReactOS is discussions not being conducted in a civilized, rational way, instead being a flame war of personal attacks. And yes, I realize that I have been guilty of those personal attacks too. That's why I started my original mail "I'm trying to keep this as non-personal and technical as possible." Obviously I failed in that (again) and I very much regret that.
Gé van Geldorp.
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
-- "I had a handle on life, but then it broke"
Ge van Geldorp wrote:
Perhaps we should call it "FF" (Feature Freeze) instead and rename RC2 to "CF" (Code Freeze)?
I really like this nomenclature a lot better, as it more accurately describes what's going on here.
-
Aleksey Bragin -
Casper Hornstrup -
Ge van Geldorp -
Jason Filby -
Royce Mitchell III -
TwoTailedFox -
WaxDragon