Scrum: Where Did Rapid Application Development Come From?

IEEE Computer published an issue on agile development in 2003. Of particular interest is an article on the history of iterative development, highly recommended for anyone interested in the background Agile methods.

Most of the "Rapid Application Development (RAD)" methodologies have taken advantage of iterative development combined with incremental delivery. Craig Larman documents the history of iterative and incremental development.

Larman, Craig and Basili, Victor R. Iterative and Incremental Development: A Brief History. IEEE Computer 36:6:47-56, June 2003.

Iterative and incremental development dates back to the mid-1950's and prominent software-engineering though leaders from each decade supported and used IID practices. Many large projects at NASA, IBM, and elsewhere used them. The DOD standard on waterfall development was misconceived according to its author who relied mainly on textbooks and consultants. It took over a decade for a DOD committee led by Fred Books (Mythical Manmonth) to remedy the problem. Meanwhile there were at least $75B of failed DOD software projects documented and these projects are only the tip of a very large iceberg of failed efforts.

The first published paper that Larman could find on iterative development was a 1968 report from Biran Randell and F.W. Zurcher at the IBM T.J. Watson Research Center. Interestingly, it lays out the basic development model used by the first software Scrum at Easel Corporation in 1993.

"The basic approach recognizes the futility of separating design, evaluation, and documentation processes in software-system design. The design process is structured by an expanding model seeded by a formal definition of the system, which provides a first, executable, functional model. It is tested and further expanded through a sequence of models, that develop an increasing amount of function and an increasing amount of detail as to how that function is to be executed. Ultimately, the model becomes the system."

Recent research has provided more detailed information on the productivity effects of incremental delivery. MacCormack’s multivariate analysis of Hewlett Packard software projects showed three primary factors that lowered defect rate (early prototype, design reviews, and integration or regression testing at code checkin) and two primary factors that increased productivity (early prototype and daily builds). Releasing a prototype to customers that is only 40% functionally complete increases productivity by 36% and adopting the practice of daily builds increases productivity by 93%.

MacCormack, A., et al., Exploring Tradeoffs Between Productivity and Quality in the Selection of Software Development Practices. IEEE Software, 2003(Sep/Oct): p. 78-85.

In his book, Agile and Iterative Development, Larman has well documented the history of the many disasters introduced by accident when the Department of Defense standardized on a non-iterative method that was unproven on large projects. It was essentially, a blunder by a consultant who had little experience with real software development.

The DOD has long since abandoned the waterfall method, and the consultant has recanted, but the waterfall approach persists as an urban myth in many software development organizations. Craig Larman details and documents this historical tragedy in Chapter Six of his book which compares the two leading agile methods, SCRUM and XP, along with RUP and Evo, an early iterative method.