Is it Scrum or Lean?

Today I got a question in email about Scrum and Lean, fallout from the recent workshop I did with Mary Poppendieck at MIT. Is Scrum related to Lean? Are we finding out new things about Scrum or Lean? Are customers asking for Lean?

After thinking about the early days of Scrum I think the root of both Scrum and Lean is complex adaptive systems theory. When we created Scrum we did not talk about Lean, we talked about complex adaptive systems. I think Scrum and Lean are complementary implementations of ways to deal with physical reality where things are often not linear, not simple, and not predictable.

After the first Scrum started at Easel in 1993, I began talking about the process in the internet newsgroups in 1994, particularly comp.object, and by 1995 we had a well documented thread of my discussions on Scrum on comp.object, interlaced with comments from Bob Martin, Kent Beck, and others.
http://jeffsutherland.com/scrum/scrum.txt

So the origins of Scrum are well documented and can be researched in Google groups by searching in the archives. My original Scrum web page started in 1994 and was widely viewed on the web throughout 1995 and updated regularly until the Agile Manifesto meeting in 2001.
http://jeffsutherland.com/scrum/history.html

During 1995, I invited Ken Schwaber to take a look at the first Scrum team which led to collaborating on the first formal paper on Scrum presented in the fall of that year at the OOPSLA. We did not discuss Lean at all but did discuss complex adaptive systems.

Mike Beedle was influenced by the online description of Scrum, implemented the process in his own company, and led the effort to drive Scrum through the Pattern Languages of Programming Design conferences. This made Scrum the first (and only) formal organizational pattern that describes a complete Agile process. Recent work by Jim Coplien shows that Scrum is deceptively simply while compressing a complex array of organizational patterns in his book, "Organizational Patterns in Agile Software Development."

Looking at "scrum.txt" you will see that Scrum derives directly from best practices in new product development documented by Takeuchi and Nonaka in their 1986 HBR paper. The leading example in the paper was Honda (Toyota was not so well know at the time). Takeuchi does not specifically address Lean as we know it today.

In 1995, I commented in the comp.object newsgroup:

Usability studies ... have convinced us that only one paper in the literature describes the rapid application development environment achieved with Object Studio (the combination of Enfin Smalltalk and Synchronicity Business Object Modeling and Persistent Object Mapping).

Takeuchi, Hirotaka and Nonaka, Ikujiro. The New Product Development Game.
Harvard Business Review, Jan/Feb 1986, pp. 137-146.

Takeuchi and Nonaka describe the Scrum methodology (a Rugby term) which we use internally. It is significantly different than anything you have seen because there is no project management schedule, only a committed delivery time for a release. The Japanese auto and electronic companies are using it to clean our clocks in the global economy. Did you notice that U.S. auto companies had a banner year and still lost
market share in 1994?

Small teams are given objectives to meet in a fixed time frame. They take the existing software (you never build anything from scratch) and small projects (2 days to 2 weeks per person) are assigned as SynchSteps. SynchSteps are dynamic pulses generated against the existing code structure that causes mutations (as in a biologic organism).

A project team is typically 3 developers, 3 QA people, 3 doc people, and one or two users. They meet daily and all agree on steps completed and next steps. For large projects, small teams of this size build components and a SCRUM of SCRUMs meets less frequently to work out interfaces between components. Developers must be outnumbered on the team by QA and documentation people or they generate too much code too fast (malignant functionality).

The code evolves like a biological system via punctuated equilibrium. You can read about this process as modeled by Denny Hillis on a Connection Machine in the book "Artificial Life." When enough mutations occur in multiple parts of the organism, the system shifts to a

higher plateau of functionality. We say at this time that a "package" is emitted. This will be a new piece of the software system we are building. For example, we are adding Use Cases to the Synchronicity product at this time and when the package appears, an objective for the release cycle is met.

A release cycle, typically six months, is built of packages which are loosely defined as objectives at the start of the release cycle.

Management must release control of the team and let it function as a self-organizing entity that grows a system like a plant. We have found that we get dramatically more functionality in less time and a much cooler product with this approach. This approach rationalizes the comments that Fred Brooks made in 1987 in his paper "There is No Silver Bullet". There, he stated that the only way to speed up development was to grow a prototype like a plant.

We track the process by the velocity of SynchStep (Sprint Backlog item) assignments versus the velocity of SynchStep completion. This gives estimate of package time completion, kind of like launching a rocket, watching its trajectory, and predicting impact point. We can adjust the impact point by lowing the arc (less functionality) or adding rocket fuel (more resources) to hit the time frame committed to management.

We monitor progress regularly via demonstrations. Nothing counts with management except delivered code that works. Packages are regularly shipped out as alpha components that people can drop into the current release of Synchronicity. Synchronicity dynamically changes, menus and all, so that people can try out the new functionality and see if they like it.

With SCRUM schedules are obsolete (developers love it) and delivery dates never slip (management loves it). If this sounds too good to be true, I suppose it is, but management will agree there is more functionality and less slippage with this process than anything they have ever seen. Our users are asking us to lengthen the product delivery schedule because we are delivering too many new releases with major upgrades
too fast. (This still doesn't stop them from complaining about not getting everything on their wish list.)

We find that the paridigm shift required for optimum utilization of the SCRUM process is significant. The object-oriented paridigm requires a shift in thinking, but Scrum requires a shift in the organization and the way people work and relate to one another.
------------------------------------------

Scrum was strongly influenced by complex adaptive systems theory - the work of Peter Senge at MIT, Christopher Langdon at the Sante Fe Institute, and many others. Also by Goldratt's constraint theory and by Rodney Brooks subsumption architecture, the basis of the company iRobot which I helped to get started shortly before Scrum began. Starting at the bottom of my original Scrum web page I commented:

"SCRUM is based on complexity theory and artificial life experiments at Thinking Machines using highly parallel simulation of systems evolution. It induces the phenomenon known as "punctuated equilibrium" seen in the evolution of biological species."

The reason the first Scrum team called Sprint Backlog items SyncSteps was that developers executed the Sprint Backlog in a carefully chosen order - that order which produced the fastest path to appearance of the next feature from the users point of view. Just as proper ordering of the Product Backlog optimizes revenue, proper ordering of the Sprint Backlog optimizes production of value. It accelerates software evolution and can produce effects seen in biological systems.

The "punctuated equilibrium" effect is achieved at Toyota using set-based concurrent engineering. As an example, Toyota does not build one radiator for a new car. They build six, and wait to the last possible moment to choose the best radiator to deploy in production. This is similar to competition between biological species in an ecosystem and the species that adapts best to the environment wins. The Scrum
community has yet to implement set based concurrent engineering strategies that were used by the first Scrum team.

However, test first development drives the software creation process in the right direction as early testing shows immediately what does not work and allows rapid deployment of alternative strategies. The key is evolution of software like a biological system based on competing implementation constructs and survival of the fittest.

A thoughtful analysis of the relationship between complex adaptive systems and Lean will show that both Scrum and Lean are instantiations of complex adaptive systems theory. A subset of complex adaptive systems theory is artificial life, a term invented by Christopher Langdon at the Sante Fe Institute in the 1980's. His seminal paper showing that organisms similated on a computer evolve faster as they approach chaotic regions strongly influence Scrum. In Scrum, we introduce chaos into the development process and then use an empirical control harness to inspect and adapt a rapidly involving system.

In trying to describe Scrum at JAOO in 2005, I used Lean to show why Scrum works. Scrum is a way to implement lean in building software. In fact, it has the advantage that if you follow it closely and implement well, you will be doing Lean as articulated by Mary and Tom Poppendieck sithout even understanding Lean. It is hard for people to understand complex systems theory so it is difficult to use that as a motivator. It is hard for people to understand Lean as well, but Toyota has become such a dominant force in industry that fear, uncertainty, and doubt wakes people up from their sleep.

In 1995, I worked with Hubert Smits and Jean Tabaka at Rally to develop a course for people who want to take their Scrum implementation to the next level all based on Lean as the easiest way to articulate what you need to focus on to optimize Scrum.
Ken Schwaber argues that by simply

focusing on the Scrum framework and the impediment list you will achieve this. This is absolutely correct. Lean is just an extra teaching tool.

There is also my paper on Scrum and CMMI Level 5 on the implementation of Scrum by Systematic Software Engineering, an implementation entirely driven by Lean. In Denmark, it is national policy to lean out all industries. As a result, it is easy to sell Scrum as a way to implement Lean in software.

Scrum is an inspect and adapt framework which is extremely simple by design to allow the average developer at Ford Motor company to get started in a couple of days. However, it is hard to implement. Less than 10% of the Scrum teams worldwide can pass the Nokia test, primarily because they cannot deliver potentially shippable (fully tested) software at the end of a Sprint. Talking to people about complex adaptive systems doesn't seem to help them as well as talking to them about Toyota to show them how their implementation of Scrum is broken, how it is riddled with disruptions in flow (muri), stressed everywhere (mura), and waste (mudah). Using Toyota as an example of a successful company who systematically elimiates muri, mura, and mudah shows them how they might best go about fixing their process.

So Lean is helpful as a teaching tool because it derives from the basic laws of the universe which are articulated at a theoretical level by complex adaptive systems research. Scrum derives from the same fundamental principles. Lean has practical ways to deal with process mapping, the cost of delays, queuing problems, reset times, and so forth. These are all useful in dealing with constraints, global system throughput, and other factors which were a primary concern for Scrum in dealing with the complex adaptive system called software development.

Lean derives from Edward Deming and the work of others teaching the Japanese that local optimization caused by traditional management by objectives was self-destructive. Deming, after consulting with the leadership in many American corporations, concluded that management is totally broken in the U.S. causing incalculable and inconceivable damage (such as the destruction of the auto industry). The global optimization promoted by Deming and his statistical quality control practices are rooted in complex adaptive systems theory, constraint theory being a specialized area that focuses on global optimization. Professor Senge's work on systems theory at MIT is the most readable resource for these early influences Scrum.

So the bottom line is that:

1. Scrum is not a direct descendant of Lean. It derives from complex adaptive systems with an indirect linkage to Lean through Takeuchi and Nonaka.

2. Management is interested in Lean, partcularly in Europe. Building on that interest and any knowledge they have about Toyota makes it easier to describe why Scrum works.

3. Lean is a good teaching tool to show Scrum teams why their implementations are broken. For example, if you do not have fully tested code and cannot create potentially shippable code in a Sprint, you have 100% work in progress going into the next Sprint. This would be viewed as a horrible and intolerable blunder in a Lean operation, yet software development teams and management seem to have difficulty understanding why this doubles your defects and cuts your velocity in half. And even when they understand it they often lack the political will to fix it. Teaching Kaizen mind and stopping the line is useful medicine for this disfunction.