SCRUM Hyperproductive Software Development
Method
28
Jan 2000SCRUM
is based on complexity theory
Zimmerman, Brenda; Lindberg, Curt;
PlsekEdgeware, Paul. Edgeware:
insights from complexity science for health care leaders. VHA Inc., 1998.
The SCRUM
development process takes advantage of these basic principles from complexity
theory. The book shows how to apply them to healthcare organizations (or any
other organization):
- Self organization
- No single point of control
- Interdisciplinary teams
- Emergent behavior
- Outcomes emerge with high dependence
on relationship and context
- Performance far greater than
sum of individuals on the team
The latest thinking in rapid application development
involves building testing methods as part of developing every class in the software
system, a great augmentation of the SCRUM development process. We were using
this at Easel and VMARK back in 1993 in Smalltalk while inventing SCRUM. Kent
Beck provides the latest and greatest view of this approach, now christended
"Extreme Programming." I am currently using this technique
for developing enterprise, federated, heterogeneous, distributed workflow systems
in Java using XML messaging in a native Web environment.
Beck,
Kent. Smalltalk
Best Practice Patterns. Prentice Hall, 1996.
Beedle, Mike; Devos, Martine; Sharon, Yonat; Schwaber,
Ken; Sutherland, Jeff. SCRUM: An extension pattern
language for hyperproductive software development. In Harrison, Neil; Foote,
Brian; Rohnert, Hans (Eds.) Pattern Languages of Program Design 4. Addison-Wesley
Software Patterns Series, 1999 (in press).
Abstract: The patterns of the SCRUM development
method are presented as an extension pattern language to the existing organizational
pattern languages. In the last few years, the SCRUM development method has rapidly
gained recognition as an effective tool to hyper-productive software development.
However, when SCRUM patterns are combined with other existing organizational
patterns, they lead to highly adaptive, yet well-structured software development
organizations. Also, decomposing SCRUM into patterns can guide adoption of only
those parts of SCRUM that are applicable to a specific situation.
-
This book republishes the original
SCRUM paper as Chapter 2: Takeuchi, Hirotaka/Nonaka, Ikujiro [1986] The
new new product development game. Harvard Business Review 86116:137-146.
The essence of SCRUM is
daily meetings where the whole team is focused on the Goal. Bottlenecks
(constraints) are a top priority. They are identified and evaluated daily.
Small batch sizes (increments) are the rule, which shortens lead time to
market. This is the essence of Goldratt's revolutionary strategy for reengineering
manufacturing.
SCRUM is a development process for managing
object-oriented projects using an iterative/incremental delivery cycle.
It is designed to empower small teams to rapidly
evolve new and legacy software systems while solving "wicked" problems:
- Ziv's Uncertainty Principle in Software Engineering
- uncertainty is inherent and inevitable in software development processes
and products [Ziv, 1996].
-
Humphrey's Requirements Uncertainty Principle
- for a new software system, the requirements will not be completely known
until after the users have used it.
- Wegner's Lemma - it is not possible to completely
specify an interactive system [Wegner, 1995]
-
Short, fixed delivery schedules under severe
time to market pressures.
It capitalizes on opportunities for rapid evolutionary
development of production prototypes:
-
Brooks pointed out that there is no "silver
bullet." Only good people and evolving a production prototype can reduce
development cycles from years to months.
-
It is well understood in biological evolution
that change occurs sharply at intervals separated by long periods of apparent
stagnation, leading to the concept of punctuated equilibrium [Dennett,
1995]. Computer simulations of this phenomenon suggest that periods
of equilibrium are actually periods of ongoing genetic change of an organism.
The effects of that change are not apparent until several subsystems evolve
in parallel to the point where they can work together to produce a dramatic
external effect [Levy,
1993].
SCRUM Patterns
Recent Publications
Sutherland, Jeff. COMDEX/Object World '98 -
Objects, Databases, and the Web: SCRUM
High Octane Development
SCRUM empowers teams and enables virtuoso performance:
Historically, the most productive software development
groups have been "surgical teams." In the future, they will be SCRUMs.
"A proposal by Harlan Mills offers a fresh
and creative solution. Mills proposes that each segment of a large job
be tackled by a team, but that the team be organized like a surgical team
rather than a hog-butchering team. That is, instead of each member cutting
away on the problem, one does the cutting and the others give him every
support that will enhance his effectiveness and productivity." Fred Brooks,
1975.
Brooks,
Frederick P. Jr. The
Mythical Man-Month : Essays on Software Engineering. Addison-Wesley,
1995.
The problem with the surgical team approach
is that only the best companies have good surgeons, and even then there
are only one or two of them. A major benefit of the SCRUM Development Process
is that it empowers the entire team as a collective surgeon. Mere mortals
may then achieve superprogrammer performance. The team becomes the virtuoso.
"A characteristic of the cybercorp is
that virtuoso employees have to be empowered
because their managers cannot understand what they are doing." James
Martin. The
Great Transition: Using the Seven Disciplines of Enterprise Engineering
to Align People, Technology, and Strategy. American Management Association,
1995. ISBN 0-8144-0315-8
The essence of SCRUM is daily meetings where
the whole team is focused on the Goal. Bottlenecks are a major priority.
They are identified and evaluated daily. Small batch sizes (increments)
are the rule, which shortens lead time to market. This is the essence of
Goldratt's revolutionary strategy for reengineering manufacturing.
SCRUM is designed for managing a complex
adaptive system, i.e. the development process. Check out: Jim
Coplien on the Complexity of the Development Process
Why SCRUM?
A Harness Constraint for an Interactive System.
On very rare occasions, a flash of insight is
such an obvious explanation of our current problems that it is immediately
adopted by the scientific community. Such was the theory of relativity
in Einsteins era, and new findings based on complexity theory are having
the same effect in our time.
There is a scientific (mathematically provable)
reason that the Waterfall Methodology is fatally flawed and doomed to failure.
The process of software development is not a fully constrained system.
It is an Interactive System as described by Peter Wegner in his OOPSLA'95
Tutorial. A short summary of Interactive Systems has been published as:
Our attempts to use the Waterfall Methodology,
an approach which assumes a fully constrained development environment,
has been a fools errand. We have been embarked on a mission to do the mathematically
impossible! It is not surprising that a third of our development projects
are canceled before completion and that those which are completed are,
on the average, 189% over budget. Peter Wegner's observations on object-based
systems apply equally to software development methodologies in the real
world.
"The development of a conceptual framework and
formal theoretical foundation for object-based programming has proved so
elusive because the observable behavior of objects cannot be modeled by
algorithms. The irreducibility of object behavior to that of algorithms
has radical consequences for both the theory and the practice of computing.
"Interaction machines, defined by extending
Turing machines with input actions (read statements), are shown to be more
expressive than computable functions, providing a counterexample to the
hypothesis of Church and Turing that the intuitive notion of computation
corresponds to formal computability by Turing machines. The negative result
that interaction cannot be modeled by algorithms leads to positive principles
of interactive modeling by interface constraints that support partial descriptions
of interactive systems whose complete behavior is inherently unspecifiable.
The unspecifiability of complete behavior for interactive systems is a
computational analog of Godel incompleteness for the integers.
"Incompleteness is a key to expressing richer
behavior shared by empirical models of physics and the natural sciences.
Interaction machines have the behavioral power of empirical systems, providing
a precise characterization of empirical computer science. They also provide
a precise framework for object-based software engineering and agent-oriented
AI models that is more expressive than algorithmic models.
"Fortunately the complete behavior of interaction
machines is not needed to harness their behavior for practical purposes.
Interface descriptions are the primary mechanism used by software designers
and application programmers for partially describing systems for the purpose
of designing, controlling, predicting, and understanding them. Interface
descriptions are an example of "harness constraints" that constrain interactive
systems so their behavior can be harnessed for useful purposes." Peter
Wegner, OOPSLA'95 Tutorial
SCRUM was first used to describe a product development
process by:
SCRUM type processes implemented at HP.
- Regarding Scrum, overall I liked the process
and it is very similar to a process HP uses called EVO. I became involved
in this type of software project management in 1991 when I began to worry
that the drive for schedule, features, and budget predictability was being
pushed too far. I believed that it was important to be "within the ballpark"
in order to make effective business decisions, but once you were there it
was better to shift to a mentality centered on understanding and managing
the remaining uncertainty rather than continuing with efforts to try to eliminate
all uncertainty a priori. (I later confirmed this when I looked at 22
projects and found NO correlation between schedule predictability and market
success! This led me to view decision quality as a key product development
metric).
IDX Systems Corporation,
Individual, Inc., VMARK Software, and Easel Corporation have partnered with
Advanced Development
Methodologies (ADM) to formalize SCRUM and provide documentation, training,
and tools to support the process.
-
Ken Schwaber, President of ADM, presented a
paper on The Scrum Development Process
at the OOPSLA'95
Workshop on Business Object Design and Implementation.
-
Ken recently met with R&D scientists at
Dupont who specialize in process engineering. When he explained the traditional
Waterfall Development Process for building software, they pointed out that
this was an approach designed to support a "controlled process" and software
development was inherently an "uncontrolled process." Using a "controlled
process" methodology to monitor an "uncontrolled process" in a chemical
plant can lead to severe accidents and even explosions.
-
According to Dupont scientists, SCRUM is an
"uncontrolled process" methodology that is appropriate for software development.
Uncontrolled processes in a chemical plant are encapsulated as a black
box and control mechanisms are developed to keep uncontrolled activities
within the black box within carefully defined parameters. This is exactly
what SCRUM does.
Development methodologies ordered from lowest
productivity to highest productivity:
-
Waterfall methodology - Requirements, analysis,
design, development, and implementation is executed as a single linear
process. This is no longer recommended even for conventional projects because
of the typical poor results caused by unexpected events during the development
cycle.
-
Spiral methodology - The linear waterfall approach
is executed repeatedly in several phases to deliver the end product. This
is commonly used in conventional projects and on many object-oriented projects
with inexperienced leadership. Productivity gains are typically less than
30% using this approach with object technology.
-
Iterative methodology - Similar to the spiral
methodology, but each iteration delivers a complete, testable, subsystem
with clean interfaces. This approach has been used on most successful object-oriented
projects. Productivity gains are typically at least 30% on object-oriented
projects. However, many development groups have complained that the dramatic
productivity gains promised by object technology have not been achieved
using this method.
-
SCRUM methodology - Similar to the iterative
methodology, but assumes that all requirements are not known in advance,
and that the fastest path to surfacing and implementing all requirements
will be discovered empirically during the development process. Careful
control mechanisms are used to assure on-time delivery of a high quality
product, while allowing maximum flexibility of small, tightly coupled,
development teams. Requires a well motivated team and good leadership to
implement effectively. Productivity gains of 600% have been seen repeatedly
in well executed projects.
SCRUM is based on complexity theory and artificial
life experiments at Thinking Machines using highly parallel simulation
of systems evolution. It induces the phenomenon known as "punctuated equilibrium"
seen in the evolution of biological species.
SCRUM messages from comp.lang.smalltalk and
comp.object
References
Aberdeen Group. Upgrading
To ISV Methodology For Enterprise Application Development. Product
Viewpoint 8:17, December 7, 1995.
Bach, James. Process
Evolution in a Mad World. Software Testing Laboratories, Inc.
Bach, James. The
Challenge of "Good Enough" Software, American Programmer, October 1995.
Boehm, B.W. 1985. "A Spiral Model of Software
Development and Enhancement," from Proceedings of an International Workshop
on Software Process and Software Environments, Coto de Caza, Trabuco Canyon,
California, March 27-29, 1985.
Booch, Grady. Object
Oriented Analysis and Design with Applications. The Benjamin/Cummings
Publishing Company, Inc., 1994, p. 8
Booch, Grady. Object
Solutions: Managing the Object-Oriented Project. Addison-Wesley,
1995.
Brynjolfsson, Erik, Amy Austin Renshaw, Marshall
van Alstyne. The Matrix of Change:
A Tool for Business Process Reengineering. MIT Sloan School of Management,
1996.
Coplien, J. "Borland Software Craftsmanship:
A New Look at Process, Quality and Productivity." Proceedings of the 5th
Annual Borland International Conference, June 5, 1994. Orlando, Florida.
DeGrace, P. and Hulet Stahl, L. 1990. Wicked
Problems, Righteous Solutions. Yourdon Press
Dennett, Daniel. Darwin's
Dangerous Idea: Evolution and the Meanings of Life. Simon and Schuster,
1995 (some say this is the best book ever written on evolution - and it
all applies to software systems).
Gartner, Lisa. The
Rookie Primer. Radcliffe Rugby Football Club, 1996
Gleick, J. 1987. Chaos, Making A New Science.
Penguin Books.
Graham, Ian. Migrating to Object Technology.
Addison-Wesley, 1994.
Kahn, D. and Sutherland, J. March-April 1994.
"Object Insider: Let’s start under-promising and over-delivering on OT."
Object Magazine, Mar/Apr 1994.
Langton, Christopher. Artificial Life.
In Artificial Life, Volume VI: SFI Studies in the Sciences of Complexity
(Ed. C. Langton) Addison-Wesley, 1988.
Levy, Steven. Artificial
Life: A Report from the Frontier Where Computers Meet Biology.
Vintage Books, 1993.
Nonaka, Ikujiro and Takeuchi, Hirotaka. 1995.
The
Knowledge Creating Company: How Japanese Companies Create the Dynamics
of Innovation. Oxford University Press.
Ogunnaike, B. 1994. Process Dynamics,
Modeling, and Control. Oxford University Press.
Pittman, Matthew. Lessons Learned in Managing
Object-Oriented Development. IEEE Software, January, 1993, pp. 43-53.
Rumbaugh, October 1995, "What Is a Method".
Journal of Object Oriented Programming.
Schwaber, Ken. "Controlled
Chaos: Living on the Edge." American Programmer, April 1996.
Takeuchi, Hirotaka and Nonaka, Ikujiro. January-February
1986. "The New New Product Development Game." Harvard Business Review.
Wegner, Peter. Interactive
Foundations of Object-Based Programming. IEEE Computer 28:10:70-72,
Oct 95.
Wegner, Peter. Models
and Paridigms of Interaction. OOPSLA'95 Tutorial Notes.
Ziv, Hadar and Debra J. Richardson. The
Uncertainty Principle in Software Engineering. Submitted to ICSE'97,
19th International Conference on Software Engineering. 23 August 1996.
© Jeff Sutherland 1995-2000