SCRUM Hyperproductive Software Development Method

28 Jan 2000SCRUM is based on complexity theory

Zimmerman, Brenda; Lindberg, Curt; PlsekEdgeware, Paul. Edgeware: insights from complexity science for health care leaders. VHA Inc., 1998.

The SCRUM development process takes advantage of these basic principles from complexity theory. The book shows how to apply them to healthcare organizations (or any other organization):

• Self organization
• No single point of control
• Interdisciplinary teams
• Emergent behavior
• Outcomes emerge with high dependence on relationship and context
• Performance far greater than sum of individuals on the team

24 Jul 1999Extreme Programming

The latest thinking in rapid application development involves building testing methods as part of developing every class in the software system, a great augmentation of the SCRUM development process. We were using this at Easel and VMARK back in 1993 in Smalltalk while inventing SCRUM. Kent Beck provides the latest and greatest view of this approach, now christended "Extreme Programming." I am currently using this technique for developing enterprise, federated, heterogeneous, distributed workflow systems in Java using XML messaging in a native Web environment.

Beck, Kent. Smalltalk Best Practice Patterns. Prentice Hall, 1996.

11 Mar 1999SCRUM Pattern accepted for publication

Beedle, Mike; Devos, Martine; Sharon, Yonat; Schwaber, Ken; Sutherland, Jeff. SCRUM: An extension pattern language for hyperproductive software development. In Harrison, Neil; Foote, Brian; Rohnert, Hans (Eds.) Pattern Languages of Program Design 4. Addison-Wesley Software Patterns Series, 1999 (in press).

Abstract: The patterns of the SCRUM development method are presented as an extension pattern language to the existing organizational pattern languages. In the last few years, the SCRUM development method has rapidly gained recognition as an effective tool to hyper-productive software development. However, when SCRUM patterns are combined with other existing organizational patterns, they lead to highly adaptive, yet well-structured software development organizations. Also, decomposing SCRUM into patterns can guide adoption of only those parts of SCRUM that are applicable to a specific situation.

25 August 1998 - Clark, Kim B. and Steven C. Wheelwright (Eds.) The Product Development Challenge : Competing Through Speed, Quality, and Creativity. Harvard Business Review Press, 1995.

• This book republishes the original SCRUM paper as Chapter 2: Takeuchi, Hirotaka/Nonaka, Ikujiro [1986] The new new product development game. Harvard Business Review 86116:137-146.

Goldratt. Eliyahu M. The Goal : A Process of Ongoing Improvement. North River Press, 1994.

SCRUM is a development process for managing object-oriented projects using an iterative/incremental delivery cycle.

It is designed to empower small teams to rapidly evolve new and legacy software systems while solving "wicked" problems:

• Ziv's Uncertainty Principle in Software Engineering - uncertainty is inherent and inevitable in software development processes and products [Ziv, 1996].
• Humphrey's Requirements Uncertainty Principle - for a new software system, the requirements will not be completely known until after the users have used it.
• Wegner's Lemma - it is not possible to completely specify an interactive system [Wegner, 1995]
• Short, fixed delivery schedules under severe time to market pressures.

It capitalizes on opportunities for rapid evolutionary development of production prototypes:

• Brooks pointed out that there is no "silver bullet." Only good people and evolving a production prototype can reduce development cycles from years to months.
• It is well understood in biological evolution that change occurs sharply at intervals separated by long periods of apparent stagnation, leading to the concept of punctuated equilibrium [Dennett, 1995]. Computer simulations of this phenomenon suggest that periods of equilibrium are actually periods of ongoing genetic change of an organism. The effects of that change are not apparent until several subsystems evolve in parallel to the point where they can work together to produce a dramatic external effect [Levy, 1993].

SCRUM Patterns

• Bob Haugen's PullSystem Pattern
• SCRUM Pattern

Recent Publications

Aberdeen Group. Upgrading To ISV Methodology For Enterprise Application Development. Product Viewpoint 8:17, December 7, 1995.

Ken Schwaber. Controlled Chaos: Living on the Edge. American Programmer, April 1996.

Sharon, Yonat. Organizational Patterns. Kinetica, 1998.

Sutherland, Jeff. COMDEX/Object World '98 - Objects, Databases, and the Web: SCRUM High Octane Development

SCRUM empowers teams and enables virtuoso performance:

"A proposal by Harlan Mills offers a fresh and creative solution. Mills proposes that each segment of a large job be tackled by a team, but that the team be organized like a surgical team rather than a hog-butchering team. That is, instead of each member cutting away on the problem, one does the cutting and the others give him every support that will enhance his effectiveness and productivity." Fred Brooks, 1975.

Brooks, Frederick P. Jr. The Mythical Man-Month : Essays on Software Engineering. Addison-Wesley, 1995.

The problem with the surgical team approach is that only the best companies have good surgeons, and even then there are only one or two of them. A major benefit of the SCRUM Development Process is that it empowers the entire team as a collective surgeon. Mere mortals may then achieve superprogrammer performance. The team becomes the virtuoso.

"A characteristic of the cybercorp is that virtuoso employees have to be empowered because their managers cannot understand what they are doing." James Martin. The Great Transition: Using the Seven Disciplines of Enterprise Engineering to Align People, Technology, and Strategy. American Management Association, 1995. ISBN 0-8144-0315-8

The essence of SCRUM is daily meetings where the whole team is focused on the Goal. Bottlenecks are a major priority. They are identified and evaluated daily. Small batch sizes (increments) are the rule, which shortens lead time to market. This is the essence of Goldratt's revolutionary strategy for reengineering manufacturing.

SCRUM is designed for managing a complex adaptive system, i.e. the development process. Check out: Jim Coplien on the Complexity of the Development Process

Why SCRUM? A Harness Constraint for an Interactive System.

On very rare occasions, a flash of insight is such an obvious explanation of our current problems that it is immediately adopted by the scientific community. Such was the theory of relativity in Einsteins era, and new findings based on complexity theory are having the same effect in our time.

There is a scientific (mathematically provable) reason that the Waterfall Methodology is fatally flawed and doomed to failure. The process of software development is not a fully constrained system. It is an Interactive System as described by Peter Wegner in his OOPSLA'95 Tutorial. A short summary of Interactive Systems has been published as:

• Wegner, Peter. Interactive Foundations of Object-Based Programming. IEEE Computer 28:10:70-72, Oct 95.

Our attempts to use the Waterfall Methodology, an approach which assumes a fully constrained development environment, has been a fools errand. We have been embarked on a mission to do the mathematically impossible! It is not surprising that a third of our development projects are canceled before completion and that those which are completed are, on the average, 189% over budget. Peter Wegner's observations on object-based systems apply equally to software development methodologies in the real world.

"Interaction machines, defined by extending Turing machines with input actions (read statements), are shown to be more expressive than computable functions, providing a counterexample to the hypothesis of Church and Turing that the intuitive notion of computation corresponds to formal computability by Turing machines. The negative result that interaction cannot be modeled by algorithms leads to positive principles of interactive modeling by interface constraints that support partial descriptions of interactive systems whose complete behavior is inherently unspecifiable. The unspecifiability of complete behavior for interactive systems is a computational analog of Godel incompleteness for the integers.

"Incompleteness is a key to expressing richer behavior shared by empirical models of physics and the natural sciences. Interaction machines have the behavioral power of empirical systems, providing a precise characterization of empirical computer science. They also provide a precise framework for object-based software engineering and agent-oriented AI models that is more expressive than algorithmic models.

"Fortunately the complete behavior of interaction machines is not needed to harness their behavior for practical purposes. Interface descriptions are the primary mechanism used by software designers and application programmers for partially describing systems for the purpose of designing, controlling, predicting, and understanding them. Interface descriptions are an example of "harness constraints" that constrain interactive systems so their behavior can be harnessed for useful purposes." Peter Wegner, OOPSLA'95 Tutorial 

SCRUM was first used to describe a product development process by:

• Takeuchi, Hirotaka/Nonaka, Ikujiro [1986] The new new product development game. Harvard Business Review 86116:137-146.
• Nonaka, Ikujiro/Takeuchi, Hirotaka [1995] The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation . Oxford University Press.

SCRUM type processes implemented at HP.

• Regarding Scrum, overall I liked the process and it is very similar to a process HP uses called EVO.  I became involved in this type of software project management in 1991 when I began to worry that the drive for schedule, features, and budget predictability was being pushed too far.  I believed that it was important to be "within the ballpark" in order to make effective business decisions, but once you were there it was better to shift to a mentality centered on understanding and managing the remaining uncertainty rather than continuing with efforts to try to eliminate all uncertainty a priori.  (I later confirmed this when I looked at 22 projects and found NO correlation between schedule predictability and market success! This led me to view decision quality as a key product development metric).

IDX Systems Corporation, Individual, Inc., VMARK Software, and Easel Corporation have partnered with Advanced Development Methodologies (ADM) to formalize SCRUM and provide documentation, training, and tools to support the process.

• Ken Schwaber, President of ADM, presented a paper on The Scrum Development Process at the OOPSLA'95 Workshop on Business Object Design and Implementation.
• Ken recently met with R&D scientists at Dupont who specialize in process engineering. When he explained the traditional Waterfall Development Process for building software, they pointed out that this was an approach designed to support a "controlled process" and software development was inherently an "uncontrolled process." Using a "controlled process" methodology to monitor an "uncontrolled process" in a chemical plant can lead to severe accidents and even explosions.
• According to Dupont scientists, SCRUM is an "uncontrolled process" methodology that is appropriate for software development. Uncontrolled processes in a chemical plant are encapsulated as a black box and control mechanisms are developed to keep uncontrolled activities within the black box within carefully defined parameters. This is exactly what SCRUM does.

Development methodologies ordered from lowest productivity to highest productivity:

• Waterfall methodology - Requirements, analysis, design, development, and implementation is executed as a single linear process. This is no longer recommended even for conventional projects because of the typical poor results caused by unexpected events during the development cycle.
• Spiral methodology - The linear waterfall approach is executed repeatedly in several phases to deliver the end product. This is commonly used in conventional projects and on many object-oriented projects with inexperienced leadership. Productivity gains are typically less than 30% using this approach with object technology.
• Iterative methodology - Similar to the spiral methodology, but each iteration delivers a complete, testable, subsystem with clean interfaces. This approach has been used on most successful object-oriented projects. Productivity gains are typically at least 30% on object-oriented projects. However, many development groups have complained that the dramatic productivity gains promised by object technology have not been achieved using this method.
• SCRUM methodology - Similar to the iterative methodology, but assumes that all requirements are not known in advance, and that the fastest path to surfacing and implementing all requirements will be discovered empirically during the development process. Careful control mechanisms are used to assure on-time delivery of a high quality product, while allowing maximum flexibility of small, tightly coupled, development teams. Requires a well motivated team and good leadership to implement effectively. Productivity gains of 600% have been seen repeatedly in well executed projects.

SCRUM is based on complexity theory and artificial life experiments at Thinking Machines using highly parallel simulation of systems evolution. It induces the phenomenon known as "punctuated equilibrium" seen in the evolution of biological species.

• Levy, Steven [1993] Artificial Life: A Report from the Frontier Where Computers Meet Biology. Vintage Books. ISBN: 0679743898.
• Lewin, Roger [1994] Complexity: Life at the Edge of Chaos. Collier Books, ISBN: 0020147953.

SCRUM messages from comp.lang.smalltalk and comp.object

• Early Scrum newsgroup dialogue

References

Bach, James. Process Evolution in a Mad World. Software Testing Laboratories, Inc.

Bach, James. The Challenge of "Good Enough" Software, American Programmer, October 1995.

Boehm, B.W. 1985. "A Spiral Model of Software Development and Enhancement," from Proceedings of an International Workshop on Software Process and Software Environments, Coto de Caza, Trabuco Canyon, California, March 27-29, 1985.

Booch, Grady. Object Oriented Analysis and Design with Applications. The Benjamin/Cummings Publishing Company, Inc., 1994, p. 8

Booch, Grady. Object Solutions: Managing the Object-Oriented Project. Addison-Wesley, 1995.

Brynjolfsson, Erik, Amy Austin Renshaw, Marshall van Alstyne. The Matrix of Change: A Tool for Business Process Reengineering. MIT Sloan School of Management, 1996.

Coplien, J. "Borland Software Craftsmanship: A New Look at Process, Quality and Productivity." Proceedings of the 5th Annual Borland International Conference, June 5, 1994. Orlando, Florida.

DeGrace, P. and Hulet Stahl, L. 1990. Wicked Problems, Righteous Solutions. Yourdon Press

Dennett, Daniel. Darwin's Dangerous Idea: Evolution and the Meanings of Life. Simon and Schuster, 1995 (some say this is the best book ever written on evolution - and it all applies to software systems).

Gartner, Lisa. The Rookie Primer. Radcliffe Rugby Football Club, 1996

Gleick, J. 1987. Chaos, Making A New Science. Penguin Books.

Graham, Ian. Migrating to Object Technology. Addison-Wesley, 1994.

Kahn, D. and Sutherland, J. March-April 1994. "Object Insider: Let’s start under-promising and over-delivering on OT." Object Magazine, Mar/Apr 1994.

Langton, Christopher. Artificial Life. In Artificial Life, Volume VI: SFI Studies in the Sciences of Complexity (Ed. C. Langton) Addison-Wesley, 1988.

Levy, Steven. Artificial Life: A Report from the Frontier Where Computers Meet Biology. Vintage Books, 1993.

Nonaka, Ikujiro and Takeuchi, Hirotaka. 1995. The Knowledge Creating Company: How Japanese Companies Create the Dynamics of Innovation. Oxford University Press.

Ogunnaike, B. 1994. Process Dynamics, Modeling, and Control. Oxford University Press.

Pittman, Matthew. Lessons Learned in Managing Object-Oriented Development. IEEE Software, January, 1993, pp. 43-53.

Rumbaugh, October 1995, "What Is a Method". Journal of Object Oriented Programming.

Schwaber, Ken. "Controlled Chaos: Living on the Edge." American Programmer, April 1996.

Takeuchi, Hirotaka and Nonaka, Ikujiro. January-February 1986. "The New New Product Development Game." Harvard Business Review.

Wegner, Peter. Interactive Foundations of Object-Based Programming. IEEE Computer 28:10:70-72, Oct 95.

Wegner, Peter. Models and Paridigms of Interaction. OOPSLA'95 Tutorial Notes.

Ziv, Hadar and Debra J. Richardson. The Uncertainty Principle in Software Engineering. Submitted to ICSE'97, 19th International Conference on Software Engineering. 23 August 1996.