Hello, I have thought long time before writing this email, and finally it's time to write it... N.B. I don't intend to hurt or offense anyone in this email. And there is not so much use from a project leader who only says "yes", "good", "agreed".
We all see ReactOS became quite a big thing. Big everywhere - source code, goals, development time... I think one would wish every FOSS project to gain that level and speed of development.
If someone of you played Civ during youth, you certainly remember that your cities can't grow, grow and grow without any efforts - people become unhappy, city is ovecrowded, demanding sanitation and special facilities in order to continue growth. If you don't provide those needed facilities, the city is going to at least stop its development, and usually goes into disorder due to unhappines.
Similar thing happens in software development too. Once project reached certain size (even in terms of SLOC), developers' "tools" should be upgraded and enhanced.
That's it for theory. Let's have a look at ReactOS, and more specifically its trunk. I'm getting more and more complaints that trunk is unstable, remains unstable, and most of the time doesn't even boot, and the time came to actually solve this problem, once and for ever.
As an example, I will show only one of common scenario: Some developer commits his code, which works for him on his, say, qemu but doesn't work for another dev on his real hardware. That developer continues to commit code, and in some time encounters that dev whose machine doesn't boot reactos and he says: "hey, you ...ng ...rd, you broke trunk!". I don't even dare to cite which reply follows, because it would take a few pages to quote. The developer who broke trunk starts to regress test, reverting revision by revision, spending time to identify the regressed revision (this may take a few days, and we don't have fulltime paid developers), and then finally finds the bug, fixes and commits. By that time, another developer commits code with a regression in other place, and trunk is still unbootable. And blaming continues, flamewars start, who broke what, instead of actually enjoying the development process.
Pretty unproductive, yeah? And you, the reader, are definately wondering - what is the solution? I answer: There are a few solutions, but as always I will list only a few
1) "Wine"-method. There is one leader who decides what, how and when to commit. He maintains the tree, he makes sure tree is in good shape, he spends all of his time to do testing, merging, reverting, remerging. It works really good for Wine. Will it work good for us? I am in doubts, really. 2) Improving our current development system. That's the direction I would like to use.
Major and the first improvement needed is testing. Testing often, testing early, automating testing, getting more people to test, regress test and feed results back to developers. Buildbot was a step in this direction, but more steps are needed. If someone broke booting, a proper recognized complain would be "Revision NNXXY regresses 3rd boot with a bugcheck code NN stack trace attached". The sooner this information is available and developer is notified - the faster this bug will be solved. If developer is inaccessible within a long period of time, a decision to revert this revision might be taken (that should be a really rare case).
This shouldn't come to absurd of course, I can tolerate having trunk broken for a few hours certainly, when developer is working on a certain feature, and, well, of course he may mistake, not fully commit something and so forth. But having it half-broken for 2 weeks AND noone took time to regress test and find the guilty revision.
Or yet another example, fortunately last for this email. I implemented an "alpha" version of usb mouse driver, along with that recently incorporated NT4 compatible usb driver. And assumed, and even asked in irc to test it - it's not that hard, but gives me a chance to fix not-so-obvious issues before it'll be enabled by default in trunk and will drive people crazy with some regress which I didn't see on my machine, and, at last, even hearing "yay, your usb mouse driver works!" just simply motivates me to finish e.g. a usb keyboard driver, find and fix bugs, etc.
Conclusions. 1) To improve development quality we must improve testing. Automate, gather bigger testing team, etc. 2) Developers must be way more careful during commit. Just remember a simple rule: Trunk is not a wastebin where you dump code to. It's quite the opposite.
Feedback. Your feedback is greatly encouraged and awaited. Feel free to discuss this in this ML.
With the best regards, Aleksey Bragin ReactOS Project Coordinator
I'm going to lose a friend over this, but whatever.
I only have one comment:
Get a TC that has the time and will to do his job as outlined in the TC manifesto, and that finds a proper Testing Team as outlined in that same manifesto, which we all voted on and agreed as part of our constitution.
Maybe one more:
Don't bitch at the most active developer when he used to have his own branch that "it's not fair his ReactOS hare more features!!!" and then hold a vote that bans "Misc branches", then complain about why he doesn't use branches. It seems Art is the only one that remembers this :)
Aleksey Bragin wrote:
Hello,I have thought long time before writing this email, and finally it's time to write it... N.B. I don't intend to hurt or offense anyone in this email. And there is not so much use from a project leader who only says "yes", "good", "agreed".
We all see ReactOS became quite a big thing. Big everywhere - source code, goals, development time... I think one would wish every FOSS project to gain that level and speed of development.
If someone of you played Civ during youth, you certainly remember that your cities can't grow, grow and grow without any efforts - people become unhappy, city is ovecrowded, demanding sanitation and special facilities in order to continue growth. If you don't provide those needed facilities, the city is going to at least stop its development, and usually goes into disorder due to unhappines.
Similar thing happens in software development too. Once project reached certain size (even in terms of SLOC), developers' "tools" should be upgraded and enhanced.
That's it for theory. Let's have a look at ReactOS, and more specifically its trunk. I'm getting more and more complaints that trunk is unstable, remains unstable, and most of the time doesn't even boot, and the time came to actually solve this problem, once and for ever.
As an example, I will show only one of common scenario: Some developer commits his code, which works for him on his, say, qemu but doesn't work for another dev on his real hardware. That developer continues to commit code, and in some time encounters that dev whose machine doesn't boot reactos and he says: "hey, you ...ng ...rd, you broke trunk!". I don't even dare to cite which reply follows, because it would take a few pages to quote. The developer who broke trunk starts to regress test, reverting revision by revision, spending time to identify the regressed revision (this may take a few days, and we don't have fulltime paid developers), and then finally finds the bug, fixes and commits. By that time, another developer commits code with a regression in other place, and trunk is still unbootable. And blaming continues, flamewars start, who broke what, instead of actually enjoying the development process.
Pretty unproductive, yeah? And you, the reader, are definately wondering - what is the solution? I answer: There are a few solutions, but as always I will list only a few
- "Wine"-method. There is one leader who decides what, how and when
to commit. He maintains the tree, he makes sure tree is in good shape, he spends all of his time to do testing, merging, reverting, remerging. It works really good for Wine. Will it work good for us? I am in doubts, really. 2) Improving our current development system. That's the direction I would like to use.
Major and the first improvement needed is testing. Testing often, testing early, automating testing, getting more people to test, regress test and feed results back to developers. Buildbot was a step in this direction, but more steps are needed. If someone broke booting, a proper recognized complain would be "Revision NNXXY regresses 3rd boot with a bugcheck code NN stack trace attached". The sooner this information is available and developer is notified - the faster this bug will be solved. If developer is inaccessible within a long period of time, a decision to revert this revision might be taken (that should be a really rare case).
This shouldn't come to absurd of course, I can tolerate having trunk broken for a few hours certainly, when developer is working on a certain feature, and, well, of course he may mistake, not fully commit something and so forth. But having it half-broken for 2 weeks AND noone took time to regress test and find the guilty revision.
Or yet another example, fortunately last for this email. I implemented an "alpha" version of usb mouse driver, along with that recently incorporated NT4 compatible usb driver. And assumed, and even asked in irc to test it - it's not that hard, but gives me a chance to fix not-so-obvious issues before it'll be enabled by default in trunk and will drive people crazy with some regress which I didn't see on my machine, and, at last, even hearing "yay, your usb mouse driver works!" just simply motivates me to finish e.g. a usb keyboard driver, find and fix bugs, etc.
Conclusions.
- To improve development quality we must improve testing. Automate,
gather bigger testing team, etc. 2) Developers must be way more careful during commit. Just remember a simple rule: Trunk is not a wastebin where you dump code to. It's quite the opposite.
Feedback. Your feedback is greatly encouraged and awaited. Feel free to discuss this in this ML.
With the best regards, Aleksey Bragin ReactOS Project Coordinator
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
On Mon, 16 Oct 2006 16:53:09 -0400 Alex Ionescu ionucu@videotron.ca wrote:
I'm going to lose a friend over this, but whatever.
I only have one comment:
Get a TC that has the time and will to do his job as outlined in the TC manifesto, and that finds a proper Testing Team as outlined in that same manifesto, which we all voted on and agreed as part of our constitution.
Maybe one more:
Don't bitch at the most active developer when he used to have his own branch that "it's not fair his ReactOS hare more features!!!" and then hold a vote that bans "Misc branches", then complain about why he doesn't use branches. It seems Art is the only one that remembers this :)
Alex: I agree but I certainly don't hold it against waxdragon. He's been working hard and doing a good job with what he had. I think you're being as unfair to him as casper ever was to you.
I've been pretty guilty myself, but reactos hasn't been an easy project to work on recently; between recrimination and brokenness I've felt pretty alienated from working on trunk. We need to increase hacker friendliness here otherwise we're really sunk. If nobody can even run reactos just to turn on debug messages, then I think we'll have trouble getting new devs. Most of the devs we've had until recently joined when the GUI got going and reactos had some successes running some apps.
We need a stable branch, or we need to branch dangerous edits. There's no other choice imo. We can live with broken branches. We can't live with a 100% nonworking operating system in trunk.
As to who's fault it is, it should not be up to one person or one group to solve it whenever things are broken. Putting it all on Wax really is a despicable thing to do. We each need to take responsibility for our stuff when it's keeping ReactOS from booting or installing and allowing others to work.
art yerkes wrote:
We need a stable branch, or we need to branch dangerous edits. There's no other choice imo. We can live with broken branches. We can't live with a 100% nonworking operating system in trunk.
I can only second that. Recently I haven't had a lot of time for ROS, I have only a few hours I can use each week, but whenever I want to work something (yes, I'd like to test it on a running system) there's almost always only two scenarios:
1. ROS doesn't build (ok, I use gcc4, my fault) 2. ROS doesn't boot
I've been fixing compiling issues because I'm one of the few people who use a more recent version of GCC. This sometimes requires quite some time, but I can live with it. But when I have to fix or debug broken code first, most of my time is gone, or I don't even have the motivation to fix it in the first place because I'd not get to work on things I'd like to, and then next time I get a chance to work on my things I'd have to go through the same procedure again. I like my tree to be up to date, but it's been impossible to work on it recently. It's a bit better now than it used to be a couple of months ago, but I remember the days where trunk was generally in a rather stable state. Of course it's sometimes broken, but we need to make sure things actually compile and that changes are tested sufficiently (Yes, many of my recent changes haven't been tested in ROS because I haven't been able to most of the time). I'm not perfect myself, neither I blame just one person. We all need to work harder on keeping trunk a little bit more stable.
- Thomas
Thomas Weidenmueller wrote:
art yerkes wrote:
We need a stable branch, or we need to branch dangerous edits. There's no other choice imo. We can live with broken branches. We can't live with a 100% nonworking operating system in trunk.
I can only second that. Recently I haven't had a lot of time for ROS, I have only a few hours I can use each week, but whenever I want to work something (yes, I'd like to test it on a running system) there's almost always only two scenarios:
- ROS doesn't build (ok, I use gcc4, my fault)
- ROS doesn't boot
I've been fixing compiling issues because I'm one of the few people who use a more recent version of GCC. This sometimes requires quite some time, but I can live with it. But when I have to fix or debug broken
I know this is a big thing to ask,,, how hard is it to update to gcc 4.x? So we can help.
code first, most of my time is gone, or I don't even have the motivation to fix it in the first place because I'd not get to work on things I'd like to, and then next time I get a chance to work on my things I'd have to go through the same procedure again. I like my tree to be up to date, but it's been impossible to work on it recently. It's a bit better now than it used to be a couple of months ago, but I remember the days where trunk was generally in a rather stable state. Of course it's sometimes broken, but we need to make sure things actually compile and that changes are tested sufficiently (Yes, many of my recent changes haven't been tested in ROS because I haven't been able to most of the time). I'm not perfect myself, neither I blame just one person. We all need to work harder on keeping trunk a little bit more stable.
- Thomas
James Tabor wrote:
I know this is a big thing to ask,,, how hard is it to update to gcc 4.x? So we can help.
I'm not asking anyone to upgrade. I don't mind having to fix compiling issues, although it's sometimes a bit annoying.
I think it'd be best if we ported GCC ourselves, so we don't have to depend on mingw. They have been behind for a long time. Porting GCC to windows should be a lot easier since 4.x because most mingw patches are already included in the main gcc source base.
- Thomas
art yerkes wrote:
On Mon, 16 Oct 2006 16:53:09 -0400 Alex Ionescu ionucu@videotron.ca wrote:
I'm going to lose a friend over this, but whatever.
I only have one comment:
Get a TC that has the time and will to do his job as outlined in the TC manifesto, and that finds a proper Testing Team as outlined in that same manifesto, which we all voted on and agreed as part of our constitution.
Maybe one more:
Don't bitch at the most active developer when he used to have his own branch that "it's not fair his ReactOS hare more features!!!" and then hold a vote that bans "Misc branches", then complain about why he doesn't use branches. It seems Art is the only one that remembers this :)
Alex: I agree but I certainly don't hold it against waxdragon. He's been working hard and doing a good job with what he had. I think you're being as unfair to him as casper ever was to you.
I've been pretty guilty myself, but reactos hasn't been an easy project to work on recently; between recrimination and brokenness I've felt pretty alienated from working on trunk. We need to increase hacker friendliness here otherwise we're really sunk. If nobody can even run reactos just to turn on debug messages, then I think we'll have trouble getting new devs. Most of the devs we've had until recently joined when the GUI got going and reactos had some successes running some apps.
We need a stable branch, or we need to branch dangerous edits. There's no other choice imo. We can live with broken branches. We can't live with a 100% nonworking operating system in trunk.
As to who's fault it is, it should not be up to one person or one group to solve it whenever things are broken. Putting it all on Wax really is a despicable thing to do. We each need to take responsibility for our stuff when it's keeping ReactOS from booting or installing and allowing others to work.
I'm not putting it all on Wax. I didn't even mention Wax's name. I simply said that we should get a TC that is able to do the job we ALL VOTED ON.
It's sad that this project has become one where being honest and truthful is now "despicable".
Anyways, I'm officially announcing that I decide not to respect the IP Policy that we voted on anymore. Anyone that complains will be called "despicable". Because hey, I'll be "trying my best".
Seriously, this isn't kindergarden. This is a serious OS Project with rules and consitution that we voted and agreed upon. When I drafted the TC document it was to make sure that such breakages would not happen again. It doesn't mean it's the TC's fault, it just means he didn't do the right job. You can have the best development team in the world, if they won't have a QA/Software Testing team you'll get code that's buggy and doesn't work on everyone's computer.
There's an important distinction between saying "something in our process is broken" versus "our process is broken because of...". I thought I clearly said the former.
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator
Where is the testing team? Where are the weekly status reports? Where is the webpage with the current "active list of applications, tools, and testing environments used to test the builds. The TC should set up a page or alternative method of visual real-time communication showing the current applications tested/in testing and wether any flaws have been noted. Preferably, specific features of each application should be tested."
I'm sorry, but these things were NOT done and the current TC breached our voted rules and obligations. When I tried to tell him about it, he told me "Well, I'll just quit then". Is this the responsible person you want to have running a testing team? Maybe we should all resort to blackmail and abandonning when someone criticizes our work.
I think it's despicable that you tried to tell me what an "evil" person I am for bringing this up. In real life, you don't get cookies for trying.
I'm not putting it all on Wax. I didn't even mention Wax's name. I simply said that we should get a TC that is able to do the job we ALL VOTED ON.
It's sad that this project has become one where being honest and truthful is now "despicable".
I don't deny that I have failed as TC, but I won't say that any other unpaid contributor would have done better.
Anyways, I'm officially announcing that I decide not to respect the IP Policy that we voted on anymore. Anyone that complains will be called "despicable". Because hey, I'll be "trying my best".
Seriously, this isn't kindergarden. This is a serious OS Project with rules and consitution that we voted and agreed upon. When I drafted the TC document it was to make sure that such breakages would not happen again. It doesn't mean it's the TC's fault, it just means he didn't do the right job. You can have the best development team in the world, if they won't have a QA/Software Testing team you'll get code that's buggy and doesn't work on everyone's computer.
For being a "serious OS Project", some of us sure act like it's "kindergarden" in our public IRC channel.
There's an important distinction between saying "something in our process is broken" versus "our process is broken because of...". I thought I clearly said the former.
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator
Where is the testing team? Where are the weekly status reports? Where is the webpage with the current "active list of applications, tools, and testing environments used to test the builds. The TC should set up a page or alternative method of visual real-time communication showing the current applications tested/in testing and wether any flaws have been noted. Preferably, specific features of each application should be tested."
I have notes on all of this, but I can't even get the "web team" to fix the Deskzilla login, let alone implement a new "testing application" on the site.
I'm sorry, but these things were NOT done and the current TC breached our voted rules and obligations. When I tried to tell him about it, he told me "Well, I'll just quit then". Is this the responsible person you want to have running a testing team? Maybe we should all resort to blackmail and abandonning when someone criticizes our work.
I said I didn't have the time anymore, and maybe I should quit to let someone else try.
I think it's despicable that you tried to tell me what an "evil" person I am for bringing this up. In real life, you don't get cookies for trying.
You are not an evil person, and sometimes your honesty is refreshing.
I've spent time teaching others my testing debugging methods on IRC, pretty much anytime anyone asked. Alas, just like developers, testers get bored, find other things to work on, stop showing up. I wrote a rather clear document on how to file bugs, but it's a fight to get people to read it. There are a few I would like to think are good testers, Usurp and Gge in particular.
WD
I have notes on all of this, but I can't even get the "web team" to fix the Deskzilla login, let alone implement a new "testing application" on the site.
I want to say sorry any inconvenience (related to deskzilla login), that we haven't managed it fix the last outstanding porblem. Afaik, in June (or May) 2006, I heard the first time about the problem. Some weeks later I spent some time testing it myself. The DeskZilla error output was really contra-productive and searching things in bugzilla code mess is a nightmare. After logging the raw data transfer with wireshark & co. wax and I found the real problem, it's bugzilla multilogon-plugin system plus roscms auto-redirect bugzilla plugin. Micheal Wirth tried in sommer to fix it, appearently he is busy with real live and had no time to finish it. I was busy with other parts of ReactOS too and eventually even stopped coding on it to fix and improve several part of the website. This process is still ongoing and I hope that all get back to normal in 3-4 weeks, after the website major update has been finished. I want to mention that there is no "web team" by definition, it's basicly (as of now) aleksey (hardware, server software, svn, etc.), aleksey's friend (helping with server transfer and related tasks) and me (web software). ReactOS has no real "teams" for years, even if the wiki may stats otherwise.
So what does this mean, as soon as I find time, I will either find someone who know how to hack the bugzilla the right way, or do that task myself.
The svn server is unrelated with the ReactOS webserver. In near future, the buildbot/master web interface will be integrated with the reactos.org website though.
Klemens
On Tue, 17 Oct 2006 09:27:22 -0400 Alex Ionescu ionucu@videotron.ca wrote:
art yerkes wrote:
On Mon, 16 Oct 2006 16:53:09 -0400 Alex Ionescu ionucu@videotron.ca wrote:
I'm going to lose a friend over this, but whatever.
I only have one comment:
Get a TC that has the time and will to do his job as outlined in the TC manifesto, and that finds a proper Testing Team as outlined in that same manifesto, which we all voted on and agreed as part of our constitution.
Maybe one more:
Don't bitch at the most active developer when he used to have his own branch that "it's not fair his ReactOS hare more features!!!" and then hold a vote that bans "Misc branches", then complain about why he doesn't use branches. It seems Art is the only one that remembers this :)
Alex: I agree but I certainly don't hold it against waxdragon. He's been working hard and doing a good job with what he had. I think you're being as unfair to him as casper ever was to you.
I've been pretty guilty myself, but reactos hasn't been an easy project to work on recently; between recrimination and brokenness I've felt pretty alienated from working on trunk. We need to increase hacker friendliness here otherwise we're really sunk. If nobody can even run reactos just to turn on debug messages, then I think we'll have trouble getting new devs. Most of the devs we've had until recently joined when the GUI got going and reactos had some successes running some apps.
We need a stable branch, or we need to branch dangerous edits. There's no other choice imo. We can live with broken branches. We can't live with a 100% nonworking operating system in trunk.
As to who's fault it is, it should not be up to one person or one group to solve it whenever things are broken. Putting it all on Wax really is a despicable thing to do. We each need to take responsibility for our stuff when it's keeping ReactOS from booting or installing and allowing others to work.
I'm not putting it all on Wax. I didn't even mention Wax's name. I simply said that we should get a TC that is able to do the job we ALL VOTED ON.
It's sad that this project has become one where being honest and truthful is now "despicable".
Anyways, I'm officially announcing that I decide not to respect the IP Policy that we voted on anymore. Anyone that complains will be called "despicable". Because hey, I'll be "trying my best".
Seriously, this isn't kindergarden. This is a serious OS Project with rules and consitution that we voted and agreed upon. When I drafted the TC document it was to make sure that such breakages would not happen again. It doesn't mean it's the TC's fault, it just means he didn't do the right job. You can have the best development team in the world, if they won't have a QA/Software Testing team you'll get code that's buggy and doesn't work on everyone's computer.
Creating a position nobody can fill didn't work. No cookies.
There's an important distinction between saying "something in our process is broken" versus "our process is broken because of...". I thought I clearly said the former.
http://www.reactos.org/wiki/index.php/ReactOS_Testing_Coordinator
Where is the testing team? Where are the weekly status reports? Where is the webpage with the current "active list of applications, tools, and testing environments used to test the builds. The TC should set up a page or alternative method of visual real-time communication showing the current applications tested/in testing and wether any flaws have been noted. Preferably, specific features of each application should be tested."
The honest truth is that I've seldom worked at any company, no matter what the resources that had testers this good, even with a QA army. Waxdragon has been as good as anybody I've worked with in a paid position. Perhaps the person who was so hopped up on getting all these votes to take place should look in the mirror.
I'm sorry, but these things were NOT done and the current TC breached our voted rules and obligations. When I tried to tell him about it, he told me "Well, I'll just quit then". Is this the responsible person you want to have running a testing team? Maybe we should all resort to blackmail and abandonning when someone criticizes our work.
I don't blame him. It is a responsible choice to leave a job you don't feel right doing anymore.
I think it's despicable that you tried to tell me what an "evil" person I am for bringing this up. In real life, you don't get cookies for trying.
I didn't say you were evil, but you were blaming "the TC" for our failure. QA is there to tell you about obscure bugs, not obvious ones. If your software already fails before QA gets it, it has no business getting there in the first place and you're just wasting their time. I don't blame our TC for feeling this way.
If you want to get into how to run this like a successful software project then we haven't even started yet.
My Thoughts:
If code could possibly break trunk, even for a few hours, it should be branched. The developer should work on that code, test it thoroughly, and ask others to test, once it's confirmed working, merge it back to trunk. Simple enough. The problem is many developers only halfway test that a given change works. Or they switch tasks in the middle of implementing something, leaving it half unimplemented.
The issue with branching to begin with was the fact that we had 38924384238 branches with XYZ feature that were incomplete, because developers left, got bored and lost interest, or otherwise. Branching should be done on the short term. Instead of doing an entire win32k rewrite in a branch, rewrite a given function at a time. If you must rewrite several functions, do so in your local tree and only commit when things are working. Better yet: don't do large rewrites at all.
FYI the ROS isos on svn.reactos.org STILL fail to install properly on my vmware machine. Formatting the virtual drive and attempting to install crashes setup, installing without formatting results in locking at the boot screen with just the little bar going, and booting the livecd also results in a lock on the boot screen. These types of regressions should not exist.
Regards, Richard Campbell
Aleksey Bragin wrote:
Hello,I have thought long time before writing this email, and finally it's time to write it... N.B. I don't intend to hurt or offense anyone in this email. And there is not so much use from a project leader who only says "yes", "good", "agreed".
We all see ReactOS became quite a big thing. Big everywhere - source code, goals, development time... I think one would wish every FOSS project to gain that level and speed of development.
If someone of you played Civ during youth, you certainly remember that your cities can't grow, grow and grow without any efforts - people become unhappy, city is ovecrowded, demanding sanitation and special facilities in order to continue growth. If you don't provide those needed facilities, the city is going to at least stop its development, and usually goes into disorder due to unhappines.
Similar thing happens in software development too. Once project reached certain size (even in terms of SLOC), developers' "tools" should be upgraded and enhanced.
That's it for theory. Let's have a look at ReactOS, and more specifically its trunk. I'm getting more and more complaints that trunk is unstable, remains unstable, and most of the time doesn't even boot, and the time came to actually solve this problem, once and for ever.
As an example, I will show only one of common scenario: Some developer commits his code, which works for him on his, say, qemu but doesn't work for another dev on his real hardware. That developer continues to commit code, and in some time encounters that dev whose machine doesn't boot reactos and he says: "hey, you ...ng ...rd, you broke trunk!". I don't even dare to cite which reply follows, because it would take a few pages to quote. The developer who broke trunk starts to regress test, reverting revision by revision, spending time to identify the regressed revision (this may take a few days, and we don't have fulltime paid developers), and then finally finds the bug, fixes and commits. By that time, another developer commits code with a regression in other place, and trunk is still unbootable. And blaming continues, flamewars start, who broke what, instead of actually enjoying the development process.
Pretty unproductive, yeah? And you, the reader, are definately wondering - what is the solution? I answer: There are a few solutions, but as always I will list only a few
- "Wine"-method. There is one leader who decides what, how and when
to commit. He maintains the tree, he makes sure tree is in good shape, he spends all of his time to do testing, merging, reverting, remerging. It works really good for Wine. Will it work good for us? I am in doubts, really. 2) Improving our current development system. That's the direction I would like to use.
Major and the first improvement needed is testing. Testing often, testing early, automating testing, getting more people to test, regress test and feed results back to developers. Buildbot was a step in this direction, but more steps are needed. If someone broke booting, a proper recognized complain would be "Revision NNXXY regresses 3rd boot with a bugcheck code NN stack trace attached". The sooner this information is available and developer is notified - the faster this bug will be solved. If developer is inaccessible within a long period of time, a decision to revert this revision might be taken (that should be a really rare case).
This shouldn't come to absurd of course, I can tolerate having trunk broken for a few hours certainly, when developer is working on a certain feature, and, well, of course he may mistake, not fully commit something and so forth. But having it half-broken for 2 weeks AND noone took time to regress test and find the guilty revision.
Or yet another example, fortunately last for this email. I implemented an "alpha" version of usb mouse driver, along with that recently incorporated NT4 compatible usb driver. And assumed, and even asked in irc to test it - it's not that hard, but gives me a chance to fix not-so-obvious issues before it'll be enabled by default in trunk and will drive people crazy with some regress which I didn't see on my machine, and, at last, even hearing "yay, your usb mouse driver works!" just simply motivates me to finish e.g. a usb keyboard driver, find and fix bugs, etc.
Conclusions.
- To improve development quality we must improve testing. Automate,
gather bigger testing team, etc. 2) Developers must be way more careful during commit. Just remember a simple rule: Trunk is not a wastebin where you dump code to. It's quite the opposite.
Feedback. Your feedback is greatly encouraged and awaited. Feel free to discuss this in this ML.
With the best regards, Aleksey Bragin ReactOS Project Coordinator
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Hi Richard! Long time!
Richard wrote:
My Thoughts:
If code could possibly break trunk, even for a few hours, it should be branched. The developer should work on that code, test it thoroughly, and ask others to test, once it's confirmed working, merge it back to trunk. Simple enough. The problem is many developers only halfway test that a given change works. Or they switch tasks in the middle of implementing something, leaving it half unimplemented.
That is true, some of our devs do not have enough hardware to test. Some just use emulators.
The issue with branching to begin with was the fact that we had 38924384238 branches with XYZ feature that were incomplete, because developers left, got bored and lost interest, or otherwise. Branching should be done on the short term. Instead of doing an entire win32k rewrite in a branch, rewrite a given function at a time. If you must rewrite several functions, do so in your local tree and only commit when things are working. Better yet: don't do large rewrites at all.
I'm currently using concurrent functionality. So it is seamless and unintrusive.
FYI the ROS isos on svn.reactos.org STILL fail to install properly on my vmware machine. Formatting the virtual drive and attempting to install crashes setup, installing without formatting results in locking at the boot screen with just the little bar going, and booting the livecd also results in a lock on the boot screen. These types of regressions should not exist.
Regards, Richard Campbell
James Tabor wrote:
Hi Richard! Long time!
Hi James!
Richard wrote:
My Thoughts:
If code could possibly break trunk, even for a few hours, it should be branched. The developer should work on that code, test it thoroughly, and ask others to test, once it's confirmed working, merge it back to trunk. Simple enough. The problem is many developers only halfway test that a given change works. Or they switch tasks in the middle of implementing something, leaving it half unimplemented.
That is true, some of our devs do not have enough hardware to test. Some just use emulators.
That is where other devs come in. Code that can't be thoroughly tested can be branched until it can be tested and declared safe.
The issue with branching to begin with was the fact that we had 38924384238 branches with XYZ feature that were incomplete, because developers left, got bored and lost interest, or otherwise. Branching should be done on the short term. Instead of doing an entire win32k rewrite in a branch, rewrite a given function at a time. If you must rewrite several functions, do so in your local tree and only commit when things are working. Better yet: don't do large rewrites at all.
I'm currently using concurrent functionality. So it is seamless and unintrusive.
Exactly.
FYI the ROS isos on svn.reactos.org STILL fail to install properly on my vmware machine. Formatting the virtual drive and attempting to install crashes setup, installing without formatting results in locking at the boot screen with just the little bar going, and booting the livecd also results in a lock on the boot screen. These types of regressions should not exist.
Regards, Richard Campbell
Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
-
Aleksey Bragin -
Alex Ionescu -
art yerkes -
James Tabor -
Klemens Friedl -
Richard -
Thomas Weidenmueller -
WaxDragon