<nettime> Open source projects need management like other projects

["nettime's roving reporter" <nettime@bbs.thing.net>]

http://www.lotus.com/developers/devbase.nsf/articles/doc2000091200

Open Source Projects Manage Themselves? Dream On
by Charles Connell

Much has been written about the open source method of software
development. By far, one of the most tantalizing statements about open
source development is these projects manage themselves. Gone are layers of
do-nothing managers with bloated bureaucracies and interminable
development schedules. In their place is a new paradigm of self-organizing
software developers with no overhead and high efficiency.

The dust jacket for Eric Raymond's open source manifesto The Cathedral and
the Bazaar makes this statement clearly. It says: ".the development of the
Linux operating system by a loose confederation of thousands of
programmers -- without central project management or control -- turns on
its head everything we thought we knew about software project management.
. It [open source] suggested a whole new way of doing business, and the
possibility of unprecedented shifts in the power structures of the
computer industry." This is not just marketing hype on a book cover,
Raymond expands the point inside: ". the Linux community seemed to
resemble a great babbling bazaar of differing agendas and approaches . out
of which a coherent and stable system could seemingly emerge only by a
succession of miracles." Other open source adherents make similar
statements in trumpeting the virtues of open source programming.

There is one problem with the statement, open source projects manage
themselves. It is not true. This article shows open source projects are
about as far as you can get from self-organizing. In fact, these projects
use strong central control, which is crucial to their success. As
evidence, I examine Raymond's fetchmail project (which is the basis of The
Cathedral and the Bazaar ) and Linus Torvalds's work with Linux. This
article describes a clearer way to understand what happens on successful
open source projects and suggests limits on the growth of the open source
method.

(Note: This article addresses issues raised in the essay titled The
Cathedral and the Bazaar . The essay also is included in a book with the
same title, which contains other essays as well.)

What Really Happened with fetchmail

The Cathedral and the Bazaar revolves around Raymond's experience in
creating a program called fetchmail by the open source method. As he
describes the software development process, he annotates the story with
lessons about open source programming and how well it worked for him. One
of Raymond's key points is that the normal functions of management are not
needed with open source development.

Raymond lists the responsibilities of traditional software managers as:
define goals and keep everybody pointed in the same direction, monitor the
project and make sure details don't get skipped, motivate people to do
boring but necessary work, organize the deployment of people for best
productivity, and marshal resources needed to sustain the project over a
long period of time. Raymond then states that none of these tasks are
needed for open source projects. Unfortunately, the majority of The
Cathedral and the Bazaar describes, in detail, how important these
management functions are and how Raymond performed them.

Eric Raymond decided what piece of software he would use as a test for
open source programming. He decided what features fetchmail would have and
would not. He generalized and simplified its design, defining the software
project. Mr. Raymond guided the project over a considerable period of
time, remaining a constant as volunteers came and went. In other words, he
marshaled resources. He surely was careful about source code control and
build procedures (or his releases would have been poor quality) so he
monitored the project. And, most significantly, Raymond heaped praise on
volunteers who helped him, which motivated those people to help some more.
(In his essay, Raymond devotes considerable space to describing how he and
Torvalds motivate their helpers.) In short, fetchmail made full use of
traditional and effective management operations, except Eric Raymond did
all of them.

Another compelling (and often-quoted) section of The Cathedral and the
Bazaar is the discussion about debugging. Raymond says: "Given enough
eyeballs, all bugs are shallow" and "Debugging is parallelizable." These
assertions simply are not true and are distortions of how the development
of fetchmail proceeded. It is true that many people, in parallel, looked
for bugs and proposed fixes. But only one person (Raymond) actually made
fixes, by incorporating the proposed changes into the official code base.
Debugging (the process of fixing the program) was performed by one person,
from suggestions made by many people. If Raymond had blindly applied all
proposed code changes, without reading them and thinking about them, the
result would have been chaos. A rare bug can be fixed completely in
isolation, with no effect on the rest of the program.

Lessons from Linux

In a similar way, on an even larger scale, Linus Torvalds pulled off a
great feat of software engineering: he coordinated the work of thousands
of people to create a high-quality operating system. Nevertheless, the
basic method was the same Raymond used for fetchmail. Torvalds was in
charge of Linux. He made all major decisions, assigned subsystems to a few
trusted people (to organize the work), resolved conflicts between
competing ideas, and inspired his followers.

Raymond provides evidence of Torvalds's control over Linux when he
describes the numbering system that Torvalds used for kernel releases.
When a significant set of new features was added to the code, the release
would be considered "major" and given a new whole number. (For example,
release 2.4 would lead to release 3.0.) When a smaller set of bug fixes
was added, the release would get just a new minor number. (For example,
release 2.4 would become 2.5.) But who made the decisions about when to
declare a major release or what fixes were minor? Torvalds. The Linux
project was (and still is) his show.

Further proof of Torvalds's key role is the fact the development of Linux
slowed to a crawl when Torvalds was distracted. The birth of his daughter
and his work at Transmeta corresponded precisely with a period of slow
progress for Linux. Why? The manager of Linux was busy with other things.
The project could not proceed efficiently without him.

Finally, there is a quote from Torvalds himself during an interview with
Bootnet.com. "Boot : You've got a full slate of global developers who are
working on Linux. Why hasn't it developed into a state of chaos? Torvalds
: It's a chaos that has some external constraints put on it. . the only
entity that can really succeed in developing Linux is the entity that is
trusted to do the right thing. And as it stands right now, I'm the only
person/entity that has that degree of trust."

Open Source Revisited

So, if the open source model is not a bazaar, what is it? To the certain
consternation of Raymond and other open source advocates, their bazaar is
really a cathedral. The fetchmail and Linux projects were built by single,
strong architects with lots of help -- just like the great cathedrals of
Europe. Beautiful cathedrals were guided by one person, over many years,
with inexpensive help from legions of workers. Just like open source
software is. And, just as with open source software, the builders of the
cathedrals were motivated by religious fervor and a divine goal. Back
then, it was celebrating the glory of God, now it is toppling Bill Gates.
(Some people think these goals are not so different.)

Consider three diagrams showing different ways of organizing a software
development project.

Figure #1 -- Traditional

Figure #2 -- Cathedral / Open Source

Figure #3 -- Bazaar

The first method (traditional) shows a Vice President of Development at
the top, with several Directors of Engineering reporting to the VP. Below
the Directors are Engineering Managers, and finally the engineers who
write the code. Many organizations use this model, and everyone agrees it
is sometimes grossly inefficient. The second method (cathedral or open
source) uses a single designer/architect at the top, with many engineers
reporting directly to the architect. The third method (bazaar) is a
peer-to-peer network of many engineers, all reporting to and coordinating
with each other, without central control. In The Cathedral and the Bazaar,
Raymond claims open source projects are run in the third style. In fact,
they are run in the style shown by the second diagram.

For more information about the parallel between software construction and
cathedral building, see the classic work of software engineering The
Mythical Man-Month by Fred Brooks. In this book , now 25 years old, Brooks
describes the chief-programmer (or surgeon) method of software
development. It is remarkably similar, in its basic philosophy, to the
open source method that Raymond and Torvalds used. Interestingly, the book
even contains a picture of a cathedral and relates it to this style of
development.

A Real Bazaar

If fetchmail and Linux were not run as bazaar projects, what would a true
bazaar project look like? A real bazaar software development method would
proceed as follows. Someone creates a minimal working version of the
software. (This follows Raymond's advice to start with a "plausible
promise.") The originator releases the working program, a description of
how to use it, and all the source code to an appropriate forum such as a
newsgroup or public web site. From this point forward, the originator
becomes just another member of the user community, with no special status.
Anyone may download the program and source, try it out, look for bugs, and
suggest fixes and enhancements. These ideas are communicated to the entire
user community through the forum. Anyone at any time, or multiple people
at the same time, may decide to create a new version of the program. They
do so by using ideas and code from the user community, along with their
own contributions. They post the new version to the user forum. Most
likely, the code "forks" as several people create new releases at the same
time. This is part of the bazaar process. The user community attempts to
settle on the best fork to follow, by trying all available versions and
focusing their attention on the best version. No single person or small
committee manages this process. Perhaps the best fork is widely recognized
and quickly selected, perhaps not. Several forks may live in parallel for
quite a while. If so, it is the decision of the user community no one fork
is the clear winner. When the community tires of parallel forks, they will
select one to follow. Development continues in this dynamic, organic
method. Leaders emerge briefly, as they create a new release or argue for
one fork over another, but they then become equal community members again.
All decisions about features, design, bug fixes, etc. are made in this
way.

Would this development method work well? While I don't know this for fact,
I suspect it would not. I believe significant human endeavors (such as
software projects) need some type of unified control in order to create
high-quality results.

It would be an interesting experiment to run an open source project in the
method I describe above. (If you know of such a project, or want to start
one, please let me know.) And it would be very exciting if it actually
worked. This would indeed fulfill the hype on the dust jacket of The
Cathedral and the Bazaar of a "whole new way of doing business, and the
possibility of unprecedented shifts in the power structures of the
computer industry."

Conclusion

I am sure many readers got out their flamethrowers at the beginning of
this article, and are now resetting their weapons from stun to vaporize.
So let me make myself clear. Open source projects are an important
development in the computer world. The open source programming method is
an exciting innovation in software engineering. But these projects do not
manage themselves. They are not run by groupthink or any self-organizing
dynamism. Successful open source projects are run by smart, effective
project leaders. In other words, these projects have good managers.

To be fair, Raymond does address the issue of project leader control in
his essay. He quickly dismisses the great importance of this control
however, by claiming he and Torvalds did not have a crucial role in the
design or creation of their software projects. He states he and Torvalds
did not design anything new, but merely recognized good ideas from others.
Raymond is not seeing clearly his and Torvalds's contributions. It is very
hard to shift through thousands of suggestions from swarms of users, find
the good ones, synthesize them together, and incorporate them into an
existing code base. This constitutes strong management and central
control.

The need for good management suggests the scalability of the open source
method may be limited. How many people have the technical sophistication
to make good software design decisions, the people skills to motivate
hundreds of contributors, and the time to dedicate to a complex project?
We should be wary about assuming that the open source method can solve the
world's software problems. It is possible only a small number of humans,
such as Raymond and Torvalds, have the requisite skill set to run an
effective open source project. If this is the case, as I suspect it is,
the number of true success stories in open source development will be
small. The open source method will run into the same wall as traditional
software development. Good technical managers are few and far between.

-------------------------------------------
Charles Connell is a Domino/Notes consultant with 10 years of experience. He
also teaches software engineering at Boston University and writes frequently
on computer topics. Charles can be reached at www.chc-3.com.
___________________________________________






#  distributed via <nettime>: no commercial use without permission
#  <nettime> is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: majordomo@bbs.thing.net and "info nettime-l" in the msg body
#  archive: http://www.nettime.org contact: nettime@bbs.thing.net