I don't think you have to look very hard to see that the development process is nothing short of formally chaotic. We informally came to that conclusion in our research here over the past 5 years. That's why we're looking at patterns (elements of social structure) rather than process (elements of time-ordered tasks) as the process perspective that drives process improvement.
Do the following experiment: try mapping your major development activities--such as design, implementation, and testing--onto empirical behaviors. You'll find implementation (coding and testing) rampant before the design phase (but people won't publicize it); you'll find lots of design changes in coding (but they never get back into the design process and never are captured in the design documents). The problem generalizes to other phases. At any stage of the process, you can't predict what will happen next. You can predict the grossest patterns at the product cycle level (sometimes), but you can't forecast their timing in the long term, and can forecast very little about them in the short term.
Note that the patterns of weather and climate serve as a good metaphor for what goes on in development. We have the same ability or inability to predict; we know the recurring long-term patterns of project cycles; within those cycles, prediction is hard. Software development obeys a process no more than the weather does.
There are exceptions, but I'm not particularly interested in them, because they're not too numerous and their problems are easily fixed with Deming's techniques.
There is a group of us, called the Villeneuve Group, who have been looking at human organizations in general, and management settings in particular, for how well they might succumb to attacks from complexity theory. It is a diverse group comprising military generals, psychiatrists, prominent authors, philosophers, business people, educators, writers, scientists and artists, all of whom have a fundamental interest in complexity. Complexity is largely a metaphor for us. Within that metaphor, there is a hunch that patterns serve as the strange attractors.
One member of the group, Mike McMaster, has written a book ("The Intelligence Advantage: Organizing for Complexity", 1995) that describes the complexity theory perpsective in quite a bit of detail. It's a very credible book--and should be: McMaster has used its findings to reengineer several well-known corporations. It rings true in the organizations I've seen go beyond statistically insignificant or small incremental changes.
This link between complexity theory and patterns is intriguing to me. If you look at Alexander's earliest work, there is clear influence from the experiments at Santa Fe Institute. McMaster is affiliated with SFI. There are other managerial pockets exploring similar theories. I think patterns can give that perspective a voice. In fact, I think that's where patterns fit best, and it's for lack of something like pattern languages that such worldviews have fallen short of acceptance in the past.
-- j.o.coplien cope@bell-labs.com ILL650 1G341 Phone: 630/713-5384 FAX: 630/713-4982 Home Page: http://www.bell-labs.com/~cope/