Slide 1
The Zen of SCRUM
|
|
|
So simple, anyone can implement it! |
|
So easy, all can benefit! |
|
So subtle, few achieve transcendent
performance … |
|
How is it that a project manager does
nothing, and achieves everything? |
|
Interlocking principles emerge product
like jigsaw puzzle. |
|
Novice removes one piece -- engine
never fires on all cylinders … |
|
Who can know why? |
|
ScrumMaster must understand deeply and
practice rigorously. |
|
Only then will team members say, “This
experience changed my life!” |
Complex Adaptive Systems
(cas)
|
|
|
Interacting agents respond to stimuli. |
|
Stimulus-response behavior is defined
in terms of rules. |
|
Agents adapt by changing rules as
experience accumulates. |
|
Agents aggregate into meta-agents whose
behavior is emergent. |
|
How can a collection of dumb things
emerge smart system behavior? |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Maamar, Zakaria and Sutherland,
Jeff (2000) Toward Intelligent Business Objects: Focusing on Techniques to
Enhance Business Objects that Exhibit Goal-Oriented Behaviors. Communications
of the ACM 40:10:99-101. |
Enterprise Systems are cas
|
|
|
Business entities are examples of
complex adaptive systems. |
|
Modification time is on the order of
months or years, roughly time required to change software. |
|
Automating business processes renders
parts of the business in software. |
|
Business systems have severely
constrained rule sets, making ideal test bed for cas concepts. |
|
|
|
Sutherland, Jeff and van den
Heuvel, Willem-Jan (2002) Developing and integrating enterprise components
and services: Enterprise application integration and complex adaptive
systems. Communications of the ACM 45:10:59-64. |
Objects Meet Requirements
for Evolution
|
|
|
Variation: there is a continuing
abundance of different elements (class libraries). |
|
Heredity or replication: the elements
have the capacity to create copies or replicas of themselves (inheritance). |
|
Differential "fitness": the
number of copies of an element that are created in a given time varies,
depending on interactions between the features of that element and the
features of the environment in which it persists (reuse). |
|
|
|
Daniel Dennett. Darwin's Dangerous
Idea: Evolution and the Meanings of Life. Simon and Schuster, 1995, p. 343. |
|
|
Do Programmers Meet
Evolutionary Requirements?
|
|
|
|
Algorithms create movement through
design space |
|
Simple minded repetitive procedure |
|
Incremental change to adjacent designs |
|
"Cranes" accelerate movement
through design space |
|
Sex and genetic engineers |
|
Evolutionary prototyping and smart
people |
|
Emergent architecture |
|
|
|
Brooks, Fred. No Silver Bullet:
Essence and Accidents of Software Engineering. IEEE Computer, Vol. 20, No. 4,
April 1987, pp. 10-19. |
Change is
Imperative:
Wasserman's 7 Factors Driving Change
"Why Are Systems
Late, Over Budget, Wrong?"
"The Waterfall Methodology!" (Paul
Bassett)
|
|
|
|
|
|
Analysis Paralysis |
|
static modeling overused |
|
specs are stale baked |
|
Design-from-Scratch |
|
no generic models |
|
no standard architectures |
|
Large Project Teams |
|
User Intermediaries |
|
No Early Warning Signals |
|
|
|
Bassett, Paul G. Framing Software
Reuse: Lessons from the Real World. Yourdon Press Computing Series, 1997. |
Wicked Problems:
Righteous Solutions
Out of a total cost of $37B for the sample set, 75% of
[DOD] projects failed or were never used, and only 2% were used without
extensive modification. Jarzombek. The 5th Annual JAWS S3
Proceedings, 1999.
|
|
|
Wicked problems have no definitive
formulation. Each attempt at creating a solution changes your understanding
of the problem. |
|
Wicked problems have no stopping rule.
The problem-solving process ends when resources are depleted, stakeholders
lose interest or political realities change. |
|
Solutions to wicked problems are not
true-or-false, but good-or-bad … getting all stakeholders to agree that a
resolution is "good enough" can be a challenge. |
|
There is no immediate or ultimate test
of a solution to a wicked problem. |
|
Every implemented solution to a wicked
problem has consequences. |
|
Wicked problems don't have a
well-described set of potential solutions. Various stakeholders have
differing views of acceptable solutions. |
|
Each wicked problem is essentially
unique. Part of the art of dealing with wicked problems is the art of not
knowing too early what type of solution to apply. |
|
Each wicked problem can be considered a
symptom of another problem. A wicked problem is a set of interlocking issues
and constraints that change over time, embedded in a dynamic social context. |
|
The causes of a wicked problem can be
explained in numerous ways. |
|
The planner (designer) has no right to
be wrong. |
|
|
|
Rittel, H and Webber M. Dilemmas in
a General Theory of Planning. Policy Sciences, Vol. 4. Elsevier, 1973. |
|
Degrace and Hulet's book, Wicked
Problems, Righteous Solutions, Prentice Hall, 1990 |
Software Development is
an Empirical Process
|
|
|
Ziv's Uncertainty Principle in Software
Engineering - uncertainty is inherent and inevitable in software development
processes and products [Ziv, 1996]. |
|
Humphrey's Requirements Uncertainty
Principle - for a new software system, the requirements will not be
completely known until after the users have used it. |
|
Wegner's Lemma - it is not possible to
completely specify an interactive system [Wegner, 1995]. |
Productivity: All at Once
Models
|
|
|
Single Super-Programmer |
|
Handcuffing two programmers together |
|
Brook’s Surgical Team |
|
Borland Quattro project |
|
Goldratt’s “The Goal” |
|
Senge’s systems thinking |
|
Holland’s complex adaptive systems |
Team Size: Development
Effort in Months
|
|
|
|
|
The smaller the better. |
|
491 medium sized projects with
35,000-95,000 SLOC (source lines of code) |
|
Putnam, Lawrence H. and Myers,
Ware. Familiar Metrics Management: Small is Beautiful--Once Again. IT Metrics
Strategis IV:8:12-16, Cutter Information Corp., August 1998. |
|
|
Team Size: Development
Time in Months
|
|
|
Sweet spot is 5-7 people |
|
491 medium sized projects with
35,000-95,000 SLOC (source lines of code) |
|
Putnam, Lawrence H. and Myers,
Ware. Familiar Metrics Management: Small is Beautiful--Once Again. IT Metrics
Strategis IV:8:12-16, Cutter Information Corp., August 1998. |
|
|
Bell Labs Report on most
productive project ever: Borland Quattro for Windows
James Coplien. Borland
Software Craftsmanship: A New Look at Process, Quality, and Productivity.
Proceedings of the 5th Annual Borland International Conference,
Orlando, 1994.
|
|
|
One of most remarkable organizations,
processes, and development cultures seen in AT&T Bell Laboratories
Pasteur process research project |
|
Project management, product management,
QA integral to team, all making technical contributions |
|
Higher communication saturation than
89% of projects |
|
More even distribution of workload |
|
“Anti-schismogenetic” – no cliques |
|
Highly iterative development |
|
Strong architectural interaction with
implementation |
|
More time spent in project team
meetings than anything else – several hours a day |
|
Gerry Weinberg notes that CMM Level 1
and 2 teams need strong managerial direction. Level 3 paradigm shift is
self-directing team. Borland team was clearly in this category, although not
by commonly accepted criteria. |
|
|
Team comments on Quattro
project
|
|
|
“We are satisfied by doing real work.” |
|
“Software is like a plant that grows.
You can’t predict its exact shape, or how big it will grow.” |
|
“There are no rules for this kind of
thing—it’s never been done before.” |
|
|
|
“Evolutionary development is best
technically, and it saves time and money.” |
|
Report of the Defense Science Board
Task Force on Military Software. Oct 1987. |
History of Iterative and
Incremental Development (IID)
|
|
|
|
1956 – Benington’s stagewise model –
USAF SAGE System |
|
1957 – IBM Service Bureau Corp, Project
Mercury, IBM Federal Systems Devision – Gerry Weinberg |
|
1960 – Weinberg teaching IID at IBM
Systems Research Institute |
|
1969 - Earliest published reference to
IID: |
|
Robert Glass. Elementary Level
Discussion of Compiler/Interpreter Writing. ACM Computing Surveys, Mar 1969 |
|
Larman, Craig and Basili, Vic. A
History of Iterative and Incremental Development. IEEE Computer, June 2003
(in press) |
History of Iterative and
Incremental Development (IID)
|
|
|
|
1971 – IBM Federal Systems Division |
|
Mills, Harlan. Top-down programming in
Large Systems. In Debugging Techniques in Large Systems. Prentice Hall, 1971 |
|
1972 – TRW uses IID on $100M Army Site
Defense software |
|
1975 – First original paper devoted to
IID |
|
Gasili, Vic and Turner, Albert.
Iterative Enhancement: A Practical Technique for Software Development. IEEE
Transactions on Software Engineering. Dec 1975. |
|
1977-1980 – IBM FSD builds NASA Space
Shuttle software in 17 iterations over 31 months, averaging 8 weeks per
iteration |
|
Madden and Rone. Design, Development,
Integration: Space Shuttle Flight Software. Communications of the ACM, Sept
1984. |
|
|
|
Larman, Craig and Basili, Vic. A
History of Iterative and Incremental Development. IEEE Computer, June 2003
(in press) |
History of Iterative and
Incremental Development (IID)
|
|
|
|
1985 – Barry Boehm’s Spiral Model |
|
Boehm, Barry. A Spiral Model of
Software Development and Enhancement. Proceedings of an International
Workshop on Software Process and Software Environments. March, 1985 |
|
1986 – Brooks, Fred. No Silver Bullet.
IEEE Computer, April 1987 |
|
Nothing … has so radically changed my
own practice, or its effectiveness [as incremental development]. |
|
1993 – First SCRUM at Easel Corporation |
|
|
|
1994 – DOD must manage programs using
iterative development |
|
Report of the Defense Science Board
Task Force on Acquiring Defense Software Commercially. June 1994. |
|
1995 – Microsoft IID published |
|
McCarthy, Jim. Dynamics of Software
Development. Microsoft Press, 1995. |
|
1996 – Kruchten. A Rational Development
Process. Crosstalk. July. |
|
Origins of RUP |
|
Larman, Craig and Basili, Vic. A
History of Iterative and Incremental Development. IEEE Computer, June 2003
(in press) |
History of Iterative and
Incremental Development (IID)
|
|
|
|
1996 – Kent Beck Chrysler Project |
|
Origin of XP |
|
1996 – Larman meets with principal
author of DD-STD-2167 |
|
David Maibor expressed regret for the
creation of the waterfall-based standard. He had not learned of incremental
development at the time and based his advice on textbooks and consultants
advocating the waterfall method. With the hindsight of iterative experience,
he would recommend IID. |
|
1997 – Coad and DeLuca rescue Singapore
project |
|
Origin of Feature-Driven Development |
|
1998 – Standish Group CHAOS Project |
|
Top reason for massive project failures
was waterfall methods. “Research also indicates that smaller time frames,
with delivery of software components early and often, will increase success
rate. |
|
1999 – Publication of extensive DOD
failures |
|
Out of a total cost of $37B for the
sample set, 75% of projects failed or were never used, and only 2% were used
without extensive modification. Jarzombek. The 5th Annual JAWS S3
Proceedings, 1999. |
|
Larman, Craig and Basili, Vic. A
History of Iterative and Incremental Development. IEEE Computer, June 2003
(in press) |
History of Iterative and
Incremental Development (IID)
|
|
|
|
2001 – 17 process expert “anarchists”
meet at Snow Bird |
|
Agile Manifesto initiated 100s of books
and papers on agile development |
|
2001 – MacCormack’s study of key
success factors |
|
MacCormack, Alan. Product-Develoment
Practices that Work. MIT Sloan Management Review 42:2, 2001. |
|
Larman, Craig and Basili, Vic. A
History of Iterative and Incremental Development. IEEE Computer, June 2003
(in press) |
Manifesto for Agile
Software Development
MacCormack Process
Evolution
|
|
|
Waterfall model – sequential process
maintains a document trail |
|
Rapid-Prototyping Model – disposable
prototype helps establish customer preference |
|
Spiral Model – series of prototypes
identifies major risks |
|
Incremental or Staged Delivery Model –
system is delivered to customer in chunks |
|
Evolutionary Delivery Model – iterative
approach in which customers test an actual version of the software |
|
MacCormack, Alan.
Product-Development Practices That Work: How Internet Companies Build
Software. MIT Sloan Management Review 42:2:75-84, Winter 2001. |
MacCormack Success
Factors
|
|
|
Early release of evolving product
design to customers. |
|
Daily incorporation of new software
code and rapid feedback on design changes |
|
A team with broad-based experience in
shipping multiple projects |
|
Major investment in design of product
architecture |
|
|
|
MacCormack, Alan.
Product-Development Practices That Work: How Internet Companies Build
Software. MIT Sloan Management Review 42:2:75-84, Winter 2001. |
SCRUM Origins:
Takeuchi and Nonaka Lessons from
Fuji-Xerox, Canon, Honda, NEC, Epson, Brother, 3M, Xerox, HP
|
|
|
|
Old model – Relay Race |
|
Speed and flexibility not adequate in
today’s market |
|
New model - Rugby |
Moving the SCRUM
downfield
Takeuchi and Nonaka Success Factors
|
|
|
Built-in instability |
|
Self-organizing project teams |
|
Overlapping development phases |
|
“Multilearning” |
|
Subtle control |
|
Organizational transfer of learning |
|
“These characteristics are like
pieces of a jigsaw puzzle. Each element, by itself, does not bring about
speed and flexibility. But taken as a whole, the characteristics can product
a powerful new set of dynamics that will make a difference.” |
Factor 1: Built-in
instability
|
|
|
|
Top management kicks off development
process by signaling broad goal. |
|
Project team is offered extremely
challenging goals with wide measure of freedom. |
|
Example: Fuji-Xerox gave FX-3500
project team two years to come up with a copier that cut costs in half |
|
Top management creates an element of
tension in the project team through challenging requirements with wide
freedom to achieve strategic objective. |
|
Honda Executive: “It’s like putting the
team members on the second floor, removing the ladder, and telling them to
jump or else. I believe creativity is born by pushing people against the wall
and pressuring them almost to the extreme.” |
Factor 2: Self-organizing
project teams
|
|
|
|
A project team takes on a
self-organizing character as it is driven to a state of “zero information” –
where prior knowledge does not apply. |
|
Left to stew, the process begins to
create its own dynamic order. |
|
The project team begins to operate like
a start-up company. |
|
A group possesses a self-organizing
capability when it exhibits three conditions. |
|
Autonomy |
|
Self-transcendence |
|
Cross-fertilization |
|
At some point, the team begins |
|
to create its own concept. |
Autonomy
|
|
|
|
Headquarters involvement is limited to
providing guidance, money, and moral support at the outset. |
|
On a day to day basis, management
seldom intervenes and the team is free to set its own direction. |
|
In a way, top management acts as a
venture capitalist |
|
“We open our purse and keep our mouth
closed.” |
|
Example: IBM development of personal
computer |
|
Example: Honda City project team,
average age 27, “Develop the kind of car that the youth segment would like to
drive.” |
Self-transcendence
|
|
|
The project teams appear to be absorbed
in the never-ending quest for “the limit.” |
|
They elevate their goals through the
development process. |
|
By pursuing what appear to be
contradictory goals, they devise ways to |
|
override the status quo and |
|
make the big discovery. |
|
Example: Canon AE-1 team |
Cross-fertilization
|
|
|
Team with wide variety of
specializations, thought processes, and behavior patterns carries out new
product development. |
|
Working in one large room is best
(Fuji-Xerox). |
|
“When all team members |
|
are in one room, others |
|
information becomes yours |
|
without even trying.” |
Factor 3: Overlapping
Development Phases
|
|
|
|
Self-organizing character of the team
produces unique dynamic or rhythm |
|
Sashimi system – Fuji Xerox Rugby system – Honda |
|
Hard merits (demerits) |
|
Speed and flexibility (watch out for
muck and mall) |
|
Soft merits |
|
Share responsibility and cooperation |
|
Stimulates involvement and commitment |
|
Sharpens a problem-solving focus |
|
Develops initiative and diversified
skills |
|
Heightens sensitivity to market
conditions |
|
|
Factor 4: Multilearning
|
|
|
|
Learning by doing in two dimensions |
|
Across organization |
|
Across specialty |
|
Enhanced learning opportunities |
|
15% of time devoted to “dreams” – 3M |
|
Peer pressure to study |
|
Send team to Europe to look around –
Honda |
|
Bring in top academics and consultants
– HP |
|
Everyone learns multiple skills |
Factor 5: Subtle Control
|
|
|
|
Management establishes checkpoints |
|
Prevents instability, ambiguity, and
tension from turning into chaos |
|
Emphasis on self-control, control by
peer pressure, control by love = “subtle control” |
|
Management responsible for: |
|
Selecting team members for balanced
team |
|
Creating an open working environment |
|
Encouraging engineers to go out in the
field |
|
Establishing rewards based on group
performance |
|
Tolerating and anticipating mistakes |
|
Encouraging suppliers to become
self-organizing |
Factor 6: Organizational
Transfer of Learning
|
|
|
|
Transfer knowledge outside group |
|
Scatter successful team to new projects |
|
Institutionalize practice (monthly
demos at IDX) |
|
Consciously pursue unlearning |
|
Next generation must be 40% better |
|
Cut product cycle by 80% |
|
Scrap old parts, processes, tools |
Challenges and
Opportunities
|
|
|
|
Winding the Rubber Band Principle:
Broad mandate and demanding goals create tension. |
|
Anti-Waterfall Principle: Operational
decisions are made incrementally. Strategic decisions delayed to last moment. |
|
Push/Pull Principle: Differentiation in
concept phase, integration dominates in implementation phase |
|
Spread the Wealth Principle:
Non-experts take on new tasks. |
|
Cuckoo Principle: Successful SCRUMs
become company models (or they can get crushed because they are different). |
|
Control Anti-Pattern: Seniority based
companies have difficult time. |
|
But in times of desperation, SCRUMs are
easily created. |
Spiral Methodology
|
|
|
Barry Boehm introduced the Spiral
Methodology to "fix" problems with the Waterfall Methodology. This
is the most commonly used variant of the Waterfall today. |
|
The Spiral methodology "peels the
onion", progressing through "layers" of the development
process. A prototype lets users determine if the project is on track, should
be sent back to prior phases, or should be ended. However, the phases and
phase processes are still linear. |
|
|
|
Boehm, B.W. A Spiral Model of
Software Development and Enhancement. Proceedings of an International
Workshop on Software Process and Software Environments, Coto de Caza, Trabuco
Canyon, California, March 27-29, 1985. |
|
Boehm, Barry. A Spiral Model of
Software Development and Enhancement. ACM SIGSOFT Software Engineering
Notes, August 1986.
Boehm, Barry. A Spiral Model of Software Development and Enhancement.
IEEE Computer, vol.21, #5, May 1988, pp 61-72. |
|
|
|
|
Iterative Methology
|
|
|
The Iterative methodology improves on
the Spiral methodology. |
|
Each iteration consists of all of the
standard Waterfall phases, but each iteration only addresses one subsystem.
Further iterations can add resources to the project while ramping up the
speed of delivery. |
|
Improves cost control, reduces risk,
ensures delivery of (sub)systems, and improves overall flexibility. |
|
Still assumes that the underlying
development processes are defined and linear. |
|
|
SCRUM Methodology
|
|
|
The first and last phases (Planning and
Closure) consist of defined processes. |
|
The Sprint phase is an empirical
process. It is treated as a black box that requires external controls. |
|
Sprints are nonlinear and flexible.
Sprints are used to evolve the final product. |
|
The project is open to the environment
until the Closure phase. The deliverable can be changed at any time. |
|
The deliverable is determined during
the project based on the environment. |
Methodology Comparison
Risk with Current
Methodologies
|
|
|
Any methodology is better than nothing. |
|
Current approaches rests on the fallacy
that the development processes are defined, predictable processes. |
|
They lack flexibility needed to cope
with the unpredictable results and respond to a complex environment. |
|
|
|
Schwaber, Ken. SCRUM Development
Process. Business Object Design and Implementation (Eds. Jeff Sutherland et
al.). London: Springer-Verlag, 1997. |
|
|
SCRUM Lowers Risk
|
|
|
Development teams need to operate
adaptively within a complex environment using imprecise processes. |
|
SCRUM can accelerate closure by
inducing the phenomenon known as "punctuated equilibrium" seen in
the evolution of biological species. |
|
Levy, Steven. Artificial Life: A
Report from the Frontier Where Computers Meet Biology. Vintage Books, 1993. |
|
Lewin, Roger. Complexity: Life at
the Edge of Chaos. Collier Books, 1994. |
Slide 44