Evolutionary and Iterative Software Development

Evolutionary software techniques, including Extreme Programming, Scrum, Evo, Agile, and even the Rational Unified Process, have features in common:
- Development is seen as exploration of both the problem and solution domains, accepting that requirements will change as development proceeds.
- Regular and frequent iterations, i.e. releases of working code integrating contributions from everyone on the development team. Some techniques acccept internal releases, others go further and require that the customer see each iteration.
- Customer interaction with the development team.
- Workspace and work habits designed to facilitate interaction between programmers and between programmers and other stakeholders.
- Automated testing of code, with tests designed and implemented by the development team.
- Empowerment of front-line programmers to make design decisions.
- Ownership of the code base is shared among all members of the development team, which is made possible by the adoption of uniform coding standards.

Promoting Evolutionary Development by Educating the Customer

On November 25th I attended a lecture at XPToronto by Craig Larman about convincing the management of a software development organization to choose evolutionary techniques over waterfall software development. Mr. Larman's strategy is to present management with evidence that the customer gets a more useful product in less time for less money if evolutionary techniques are employed.
This approach is unlikely to convince management because:

The organization doing the development receives less money and resources. Less people are required, and the required people are more junior. Because the software does what the customer wanted the first time, the development organization doesn't get to bill for fixing it.

Evolutionary techniques devolve design and decision-making from management to frontline programmers. Evolutionary methods are antithetical to a deep organizational command hierarchy, and must be seen as a threat by individuals whose role in the hierarchy is threatened.

Iterative development with frequent customer consultation transfers power from producer to customer, threatening the producer's ability to extort cash from the customer.

Evolutionary development is hard to quote and bill for because deliverables and times are not defined up front. A fixed-price contract is seductive from the point of view of management at both customer and seller.

Management has an incentive to choose evolutionary techniques only if there are products competing on price, timeliness and quality. Software is still changing rapidly so this often isn't the case.
  As we look over the objections above, all but the last make agile processes more attractive to a project's customer. This suggests that if one wants to develop software in an evolutionary manner, the first thing to do is educate the customer.
  There is a need for book on how to be an effective software customer, describing how to compare contractors' development techniques and how to participate as a customer in agile processes.

Waterfall Software Development

If management had everything its own way, this is how software would be written. The development process is divided into a series of discrete stages, each of which must be completed before the next begins. These stages are often carried out by different groups of people. The earlier the stage, the more senior the people involved.
  First, management recognizes the need for the project to exist, perhaps by selling the project to a customer, who agrees to pay for development.
  A contract or project plan is drawn up with estimates of the amount of time and number of people who will be required to complete the project.
  A detailed design is created by a design team, who then go off and design other products, while an implementation team actually builds the project.
  Once development is well advanced, a quality assurance team begins to look for bugs.
  Only once the product is relatively bug free does the customer get to see the result.
  Of course, this doesn't actually work. See Craig Larman's book Agile and Iterative Development: A Manager's Guide for a discussion of why this is.