TC function description: opening discussion prior to vote

[Ge van Geldorp]

There was some confusion and disagreement about the Testing Coordinator
function description recently. The best way to solve it seems to be a vote.
With the new voting procedure established and 0.2.8 (almost) out the door,
this seems a good time to start the voting process. Our (draft) constitution
calls for a 5 day discussion period, followed by a 5 day voting period. This
email starts the 5 day discussion period. Preferred location for the
discussion is the ros-dev mailing list, since all developers have easy
access to that. At the end of the discussion, I'll post a call for votes to
the mailing list and create the vote on the forum.

We have two proposals for the TC function description, A and B. A is the
original function description. Proposal B is almost the same, the only
difference is that the TC does not have the authority to block a release.

This is the motion I would like to vote on:

"Do you want to:
A) set the function description for the TC role as described in
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator or
B) set the function description for the TC role as described in
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator_Alternate…

Gé van Geldorp.

[Royce Mitchell III]

Ge van Geldorp wrote:

>  There was some confusion and disagreement about the
> Testing Coordinator
> function description recently. The best way to solve it seems to be a vote.
> With the new voting procedure established and 0.2.8 (almost) out the door,
> this seems a good time to start the voting process. Our (draft) constitution
> calls for a 5 day discussion period, followed by a 5 day voting period. This
> email starts the 5 day discussion period. Preferred location for the
> discussion is the ros-dev mailing list, since all developers have easy
> access to that. At the end of the discussion, I'll post a call for votes to
> the mailing list and create the vote on the forum.
>
> We have two proposals for the TC function description, A and B. A is the
> original function description. Proposal B is almost the same, the only
> difference is that the TC does not have the authority to block a release.
>
> This is the motion I would like to vote on:
>
> "Do you want to:
> A) set the function description for the TC role as described in
> http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator or
> B) set the function description for the TC role as described in
> http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator_Alternate…
>
> Gé van Geldorp.
>   
>  
Under "Alternate" URL, Section "Powers", 2nd paragraph, you
have: 
"...hinder any legitimate efforts...". I think the word legitimate is 
too vague here, and the sentence should be changed to be more precise.

I'm having a difficult time coming up with an alternate suggestion, so I 
figured I'd think aloud here... All powers granted by voting people into 
positions are temporary. We are not setting anybody up as god. So in 
either scenario, there has to be the ability to overrule the TC. 
Furthermore, the only time the TC's "prioritization" of a bug is during 
a release process. For those reasons, based on the two proposals as they 
are, I think the voting should be between:

A) 3 devs have the power to overrule the TC regarding a bug's priority 
for the purposes of a release.

B) it takes a majority vote to overrule the TC regarding a bug's 
priority for the purposes of a release.

Moreover, I'm thinking maybe there's another way to skin this cat...

Instead of the TC having to fight to prevent a release he thinks is 
necessary, we could have a slightly different procedure:

1) RC is ready for FF. Asks TC for revision to branch. If TC feels there 
are too many blockers for FF, he presents a list of blockers to the 
list. ( You can't really set up a forum vote for this - forum votes only 
allow you to pick one item from a list. ). We'd have to decide here 
exactly what kind of a "vote" this would be - should 3 devs in agreement 
on a single blocker be able to enforce the TC's block, or what?

2) The CF process would work the same as FF, except without the 
branching, as the code is already branched.

The advantage of doing it this way is we already have a list of "known 
bugs" we can apply to the release if we decide to go forward.

Anyways, just some random musings...

Peace

[Ge van Geldorp]

>   From: Royce Mitchell III
>  
>  Furthermore, the only time the TC's "prioritization" of a bug 
>  is during a release process. For those reasons, based on the 
>  two proposals as they are, I think the voting should be between:
>  
>  A) 3 devs have the power to overrule the TC regarding a bug's 
>  priority for the purposes of a release.
>  
>  B) it takes a majority vote to overrule the TC regarding a 
>  bug's priority for the purposes of a release. 
The whole point of proposal B is to not give the TC special powers with
respect to a release. In proposal A, the TC decides whether to publish a
release or not (with a possibility to overrule that decision). Personally, I
think that's fundamentally wrong. In my view, the developers are responsible
for the quality of the code, not the TC. If a dev introduces a bug which is
not found during testing, it's still the devs "fault", not the
"fault" of
the testing team. With the devs responsible for the quality, they should be
the primary decision making group on whether to release or not. 
Of course, you don't have to agree with my point of view, that's why there
are two proposals up for vote.

>   Instead of the TC having to fight to prevent a release
> he 
>  thinks is necessary, we could have a slightly different procedure:
>  
>  1) RC is ready for FF. Asks TC for revision to branch. If TC 
>  feels there are too many blockers for FF, he presents a list 
>  of blockers to the list. ( You can't really set up a forum 
>  vote for this - forum votes only allow you to pick one item 
>  from a list. ). We'd have to decide here exactly what kind of 
>  a "vote" this would be - should 3 devs in agreement on a 
>  single blocker be able to enforce the TC's block, or what?
>  
>  2) The CF process would work the same as FF, except without 
>  the branching, as the code is already branched. 
(For the benefit of those wondering about the acronyms: FF = Feature Freeze,
point where we make the release branch, no new features go in the branch,
only bug fixes. CF = Code Freeze, only major bugs are fixed during this
stage, every commit needs to be discussed on the mailinglist first).
The procedure you suggest has the danger of postponing releases indefinetly.
There will always be a feature just around the corner (networking, anyone?)
or a serious bug. I believe we should go back to time-based releases. Every
two months, the release coordinator publishes a release schedule: "we're
going to branch for 0.x.x and enter feature freeze on xx/xx, with code
freeze expected on xx/xx and final release on xx/xx". So, it's possible that
we end up branching with a serious bug present. That bug will be in the FF
(or Preview or whatever you want to call it, the thing we currently call
RC1) which is published on SourceForge. The point of the FF is not to have a
perfect .iso out there, but to get extra testers to find other bugs.
The TC is very important especially during the feature freeze phase. It
would be good to have a list of serious problems (hopefully that list is
empty, by the way) when we enter feature freeze and be able to cross the
problems away as we fix them. It's just that I believe we shouldn't make the
decision to branch based on the problem list, it should be time-based
instead.
Now, entering code freeze is another matter. That's basically the point
where we say "ok, what we have in the release branch is good enough for an
official release". That's why I said "code freeze expected on xx/xx"
above,
the date of a code freeze can and should shift if serious problems are still
present.

>   Peace 
And hapiness to all.

Gé.

[Casper Hornstrup]

>   -----Original Message-----
>  From: ros-dev-bounces(a)reactos.org [mailto:[e-mail]] On Behalf Of Ge
> van Geldorp
>  Sent: 25. oktober 2005 23:17
>  To: 'ReactOS Development List'
>  Subject: [ros-dev] TC function description: opening discussion prior to vote
>  
>  There was some confusion and disagreement about the Testing Coordinator
>  function description recently. The best way to solve it seems to be a vote.
>  With the new voting procedure established and 0.2.8 (almost) out the door,
>  this seems a good time to start the voting process. Our (draft) constitution
>  calls for a 5 day discussion period, followed by a 5 day voting period. This
>  email starts the 5 day discussion period. Preferred location for the
>  discussion is the ros-dev mailing list, since all developers have easy
>  access to that. At the end of the discussion, I'll post a call for votes to
>  the mailing list and create the vote on the forum. 
[CSH] 
7 days actually, unless specified otherwise.

>   
>  We have two proposals for the TC function description, A and B. A is the
>  original function description. Proposal B is almost the same, the only
>  difference is that the TC does not have the authority to block a release.
>  
>  This is the motion I would like to vote on:
>  
>  "Do you want to:
>  A) set the function description for the TC role as described in
>  http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator or
>  B) set the function description for the TC role as described in
>  http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator_Alternate…
>  
>  Gé van Geldorp. 
[CSH] Both A and B would violate the constitution if we agree to it.
It says a vote can override any decision or action made by any project member.

This paragraph violates it:
"The TC has final say over the priority of a bug. By his election, the developers and
board members have agreed to trust in his
judgement related to bug priorities, and must not hinder any legitimate efforts that the
TC is attempting in order to prioritize
bugs. "

I believe the whole paragraph is void. The TC (and any other project member) can
do almost anything (including changing priority of a bug and attempting to block
a release). All the draft constitution asks for, is that every registered project
member is given an opportunity to object (and call for a vote on the subject if needed).

I'm not sure what this sentence implies:
"The TC's tools include complete access over Bugzilla, as well as a team of
testers which respond directly to the TC."

Casper

[Ge van Geldorp]

>   From: Casper Hornstrup
>  
> >   Our (draft) constitution calls for a 5 day
> > discussion period,
> >  followed by a 5 day voting period. 
>  
>  [CSH]
>  7 days actually, unless specified otherwise. 
Damn, I should have checked instead of relying on my memory. Thanks for the
correction, we now have 6 days to go in the discussion period which will be
followed by a 7 day voting period.
 
>   [CSH] Both A and B would violate the constitution if
> we agree to it.
>  It says a vote can override any decision or action made by 
>  any project member.
>  
>  This paragraph violates it:
>  "The TC has final say over the priority of a bug. By his 
>  election, the developers and board members have agreed to 
>  trust in his judgement related to bug priorities, and must 
>  not hinder any legitimate efforts that the TC is attempting 
>  in order to prioritize bugs. "
>  
>  I believe the whole paragraph is void. The TC (and any other 
>  project member) can do almost anything (including changing 
>  priority of a bug and attempting to block a release). All the 
>  draft constitution asks for, is that every registered project 
>  member is given an opportunity to object (and call for a vote 
>  on the subject if needed). 
Royce also pointed out that that paragraph is not very clear. We should
probably drop it in proposal B. I don't feel very comfortable making changes
to proposal A, I didn't write and don't feel responsible for it. I hope
someone else will pick up this issue with respect to A.

Gé van Geldorp.

[Ge van Geldorp]

One day late (I still like coding better than these administrative tasks
:-)) I call for a vote on the TC function description. The vote will run for
7 days and ends 8 Nov. I will create the vote on the forum.

I'm slightly amending the proposals to vote on, after Casper pointed out
both of the proposals would violate the (draft) constitution. In proposal A
(which gives the TC power to block a release) "the approval from a board
member" is no longer required to override the TC's decision to block a
release. In proposal B (no blocking power) the sentence "The TC has final
say over the priority of a bug. By his election, the developers and board
members have agreed to trust in his judgement related to bug priorities, and
must not hinder any legitimate efforts that the TC is attempting in order to
prioritize bugs." was dropped.

In order to preserve the full text of the proposals for our children and
grandchildren, I'm including them below so they will get archived.

This is the resolution to vote on:

Do you want to:
A) set the function description for the TC role as described below under
"Proposal A" or
B) set the function description for the TC role as described below under
"Proposal B"

Happy voting, Gé van Geldorp.

========= Proposal A =========

Mandate

The Testing Coordinator's (TC) job is to ensure continous quality releases
of ReactOS builds and to periodically verify the current status of the SVN
"HEAD" Branch in order to quickly identify regressions. The TC's job
becomes
even more critical as we approach an official release, as every application
known to work in a previous release must continue working, and the fixing of
older bugs needs to be assured in order to ensure a steady stream of quality
increase between releases. The TC's tools include complete access over
Bugzilla, as well as a team of testers which respond directly to the TC.

Term

1 year renewable.

Powers

Since one of the TC's primary and most important tools is the Bugzilla
Database, the TC is granted complete administrative control over the
Database, with the power to create, delete, modify any bugs he chooses, as
well as the ability to perform administrative maintenance tools such as
adding a new revision to the databases's list, handling developer emails,
assigning bugs, etc. Due to the sensitive nature of such control, the TC
must be properly trained in the usage of the administrative interface, in
order to avoid accidental damage to the DB.

The TC has final say over the priority of a bug. The TC can decide as his
will which bugs to promote to Blocker status (thereby blocking a new
release), or to demote bugs below that status. Although the TC is
recommended to specify a reason for his judgement, his decisions should not
be the cause of endless flame wars and challenges. By his election, the
developers and board members have agreed to trust in his judgement related
to bug priorities, and must not hinder any legitimate efforts that the TC is
attempting in order to prioritize bugs. However, in the rare case where it
is clear to a majority of developers that the decision to block a release is
unwarranted, a group of no less then three developers may override the TC's
decision. Such actions should never have to happen however, and communicaton
is highly recommended in order to reach a moderated debate.

Obligations

Since the TC's job is critical to the success of ReactOS on the public
scene, as well as a smooth development environment internally, unhindered by
the constant search of 3000-revision old regressions, there are some
obligations which the TC has towards the ReactOS Team.

One of the key obligations of the TC is constant contact with the developer
team. The TC must be present in daily development activities as much as
possible, to ensure that he is constantly aware of new bugs, difficulties in
fixing some bugs, and other design or architectural decisions which might
impact the severity and/or presence of a bug or regression. This is to avoid
cases where the entire developer team is aware of a voluntary temporary
regression, but the TC knows nothign about it and blocks the release.

In order to fulfill his actual mandate, the TC is required to test the
current SVN at an acceptable pace. It is hard to give an exact number of how
many revisions can pass beteween a new test, as sometimes there can be 40
consecutive revisions that merely fix some naming, while other times, a
single revision can change everything. Therefore, it is up to the TC to be
in contact with the dev team, and also his personal judgement to determine
when a certain treshold of potentially regressive changes have been done.

Finally, the TC should have an active list of applications, tools, and
testing environments used to test the builds. He should provide this list
publically so the other members of the team can collaborate, comment on, and
potentially verify and more important duplicate results. This will be key to
allowing developers to easily fix regressions or bugs. The TC should set up
a page or alternative method of visual real-time communication showing the
current applications tested/in testing and wether any flaws have been noted.
Preferably, specific features of each application should be tested. An
example is listed below:

                                      CURRENT REV: 25000
 - Abiword
   - Menu System                                               V
   - File/Open System                                          V
   - Toolbars, docking, and other graphical elements           F -
Regression. Docking a toolbar makes it disappear
 - Regedit
   - Deleting a key                                            V
   - Creating a key                                            F - Known
bug. Creating a key cannot be done.

========= Proposal B =========

Mandate

The Testing Coordinator's (TC) job is to ensure continous quality releases
of ReactOS builds and to periodically verify the current status of the SVN
"HEAD" Branch in order to quickly identify regressions. The TC's job
becomes
even more critical as we approach an official release, as every application
known to work in a previous release must continue working, and the fixing of
older bugs needs to be assured in order to ensure a steady stream of quality
increase between releases. The TC's tools include complete access over
Bugzilla, as well as a team of testers which respond directly to the TC.

Term

1 year renewable.

Powers

Since one of the TC's primary and most important tools is the Bugzilla
Database, the TC is granted complete administrative control over the
Database, with the power to create, delete, modify any bugs he chooses, as
well as the ability to perform administrative maintenance tools such as
adding a new revision to the databases's list, handling developer emails,
assigning bugs, etc. Due to the sensitive nature of such control, the TC
must be properly trained in the usage of the administrative interface, in
order to avoid accidental damage to the DB.

Obligations

Since the TC's job is critical to the success of ReactOS on the public
scene, as well as a smooth development environment internally, unhindered by
the constant search of 3000-revision old regressions, there are some
obligations which the TC has towards the ReactOS Team.

One of the key obligations of the TC is constant contact with the developer
team. The TC must be present in daily development activities as much as
possible, to ensure that he is constantly aware of new bugs, difficulties in
fixing some bugs, and other design or architectural decisions which might
impact the severity and/or presence of a bug or regression. This is to avoid
cases where the entire developer team is aware of a voluntary temporary
regression, but the TC knows nothign about it and blocks the release.

In order to fulfill his actual mandate, the TC is required to test the
current SVN at an acceptable pace. It is hard to give an exact number of how
many revisions can pass beteween a new test, as sometimes there can be 40
consecutive revisions that merely fix some naming, while other times, a
single revision can change everything. Therefore, it is up to the TC to be
in contact with the dev team, and also his personal judgement to determine
when a certain treshold of potentially regressive changes have been done.

Finally, the TC should have an active list of applications, tools, and
testing environments used to test the builds. He should provide this list
publically so the other members of the team can collaborate, comment on, and
potentially verify and more important duplicate results. This will be key to
allowing developers to easily fix regressions or bugs. The TC should set up
a page or alternative method of visual real-time communication showing the
current applications tested/in testing and wether any flaws have been noted.
Preferably, specific features of each application should be tested. An
example is listed below:

                                      CURRENT REV: 25000
 - Abiword
   - Menu System                                               V
   - File/Open System                                          V
   - Toolbars, docking, and other graphical elements           F -
Regression. Docking a toolbar makes it disappear
 - Regedit
   - Deleting a key                                            V
   - Creating a key                                            F - Known
bug. Creating a key cannot be done.

[Alex Ionescu]

Ge van Geldorp wrote:

>  In proposal B (no blocking power) the sentence
> "The TC has final
> say over the priority of a bug. By his election, the developers and board
> members have agreed to trust in his judgement related to bug priorities, and
> must not hinder any legitimate efforts that the TC is attempting in order to
> prioritize bugs." was dropped.
>  
I agree with everything, including removing "The TC has final say over 
the priority of a bug." which caused all this, but is it really right to 
remove:

"By his election, the developers have agreed to trust in his judgement 
related to bug priorities"? It would seem to me like that's a given. It 
doesn't represent any grant of power, just states the obvious. We're 
voting for Foo as TC because we trust he'll be a good TC. Addtionally: 
"must not hinder any legitimate efforts that the TC is attempting in 
order to prioritize bugs." is also not an adtional grant of power, but 
simply specifies that if the TC thinks Bug XXX is low priority, a 
developer (i'm talking about one, not a majority) shouldn't constantly 
raise it back to BLOCKER just because he feels this is very important to 
him. I think this is also normal curteous behaviour... once again, I'm 
not saying that the TC's decision should be absolute, just that if the 
TC and ONE developer shouldn't have to go through priority wars (unless 
there is a majority). I hope this makes sense.. I'm waiting for your 
comments.

Best regards,
Alex Ionescu

[Ge van Geldorp]

>   From: Alex Ionescu
>  
>  I agree with everything, including removing "The TC has final 
>  say over the priority of a bug." which caused all this, but 
>  is it really right to remove:
>  
>  "By his election, the developers have agreed to trust in his 
>  judgement related to bug priorities"? It would seem to me 
>  like that's a given. It doesn't represent any grant of power, 
>  just states the obvious. We're voting for Foo as TC because 
>  we trust he'll be a good TC. 
If it's a given and obvious, there's no reason to put it in? The problem
with the sentence is that it can be read to imply that the TCs decisions in
this regard cannot be overridden by a developers vote. Since it didn't seem
to add anything but could be misinterpreted, the simplest solution was to
just drop it.

>   Addtionally: 
>  "must not hinder any legitimate efforts that the TC is 
>  attempting in order to prioritize bugs." is also not an 
>  adtional grant of power, but simply specifies that if the TC 
>  thinks Bug XXX is low priority, a developer (i'm talking 
>  about one, not a majority) shouldn't constantly raise it back 
>  to BLOCKER just because he feels this is very important to 
>  him. I think this is also normal curteous behaviour... 
Royce pointed out
(http://www.reactos.org/archives/public/ros-dev/2005-October/005779.html)
that "legitimate" isn't very precise. Again, since it doesn't add
anything
(the TC is given administrative control over the Bugzilla installation in
the previous paragraph) I dropped it.

GvG

[Brian]

Im not a member of the team, i know, but it seems that there is
starting to become an aulful lot of politics involved with the
project. I didn't think that ReactOS was that big already...

On 11/1/05, Ge van Geldorp <gvg(a)reactos.org> wrote:
> >   From: Alex
> > Ionescu
> >
> >  I agree with everything, including removing "The TC has final
> >  say over the priority of a bug." which caused all this, but
> >  is it really right to remove:
> >
> >  "By his election, the developers have agreed to trust in his
> >  judgement related to bug priorities"? It would seem to me
> >  like that's a given. It doesn't represent any grant of power,
> >  just states the obvious. We're voting for Foo as TC because
> >  we trust he'll be a good TC. 
>  If it's a given and obvious, there's no reason to put it in? The problem
>  with the sentence is that it can be read to imply that the TCs decisions in
>  this regard cannot be overridden by a developers vote. Since it didn't seem
>  to add anything but could be misinterpreted, the simplest solution was to
>  just drop it.
>
> >   Addtionally:
> >  "must not hinder any legitimate efforts that the TC is
> >  attempting in order to prioritize bugs." is also not an
> >  adtional grant of power, but simply specifies that if the TC
> >  thinks Bug XXX is low priority, a developer (i'm talking
> >  about one, not a majority) shouldn't constantly raise it back
> >  to BLOCKER just because he feels this is very important to
> >  him. I think this is also normal curteous behaviour... 
>  Royce pointed out
>  (http://www.reactos.org/archives/public/ros-dev/2005-October/005779.html)
>  that "legitimate" isn't very precise. Again, since it doesn't add
> anything
>  (the TC is given administrative control over the Bugzilla installation in
>  the previous paragraph) I dropped it.
>
>  GvG
>
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev
>

[Casper Hornstrup]

We used to beat each other until everyone, but one gave up.
It wasn't very constructive.

Casper

>   -----Original Message-----
>  From: ros-dev-bounces(a)reactos.org [mailto:[e-mail]] On Behalf Of
> Brian
>  Sent: 2. november 2005 18:35
>  To: ReactOS Development List
>  Subject: [ros-dev] Re: TC function description: Call for vote
>  
>  Im not a member of the team, i know, but it seems that there is
>  starting to become an aulful lot of politics involved with the
>  project. I didn't think that ReactOS was that big already...

[Ge van Geldorp]

The the vote on the TC function description is now closed. Final results:

Proposal A 11 votes
Proposal B 5 votes

which means proposal A is accepted. I will update the website accordingly
(probably tomorrow).
This outcome means we also have to change/update the release process
document. It is also time to elect a TC now. I'll start two separate email
threads for these.

Gé van Geldorp.