Agile Process Showcased at IT Architect Event

• By Kurt Mackie
• 10/25/07

Last week, IT architects put their profession squarely on the map at the first annual IT Architect Regional Conference for Southern California. This local San Diego-based event, part of the International Association of Software Architects (IASA), wasn't just a typical forum for punditry on the sometimes obscure topic of service-oriented architecture (SOA). It was more a meeting of colleagues. The keynotes and sessions were frequently paused to answer spontaneous questions and remarks from attendees.

The first keynote speech, by Scott Ambler, IASA Fellow and IBM Rational practice leader for agile development, wasn't about SOA at all. It was about the agile software development process, which Ambler predicted would probably become the norm by 2010. Ambler spoke on "Agile Strategies for Enterprise Architects."

The agile process is usually defined as the use of small teams to create frequent software builds and releases. Agile teams typically create solutions without slowing down to document the process. In that sense, agile development is often contrasted with the traditional waterfall software development approach, which is heavily documented and prescribed. While that's a common way to characterize agile development, Ambler said no formal definition for the term really exists.

The majority of companies appear to be engaging in the agile software development process. Ambler cited a March 2007 Dr. Dobbs Journal survey of agile development where 69 percent of respondents said that their organizations were doing one or more agile projects.

Ambler called the concept of following repeatable processes in software development a "stupid idea." He said that teams work together differently and that the development process should stop catering to constructs formed by bureaucrats. People are not going to follow repeatable processes for the sheer joy of it, he said. What is really wanted is repeatable results.

In practice, you can't make decisions based on your documents (a traditional waterfall approach). There's nothing wrong with documentation, but agile developers are smart about it and don't try to write speculative documentation, Ambler said.

Writing the requirements first is not practical, for a number of reasons that Ambler described.

"We build software to meet the changing needs of our stakeholders," he said. "The stakeholders will change their minds. Trying to write a requirements specification -- trying to set up a design document early in the lifecycle -- is absolutely crazy. People are not good at defining up front what they want. We've known this for a couple decades now."

One reason to go with agile software development is that 45 percent of software functionality is not used on successful software development projects, per data from the Standish Group that Ambler didn't specify. Under traditional methods, developers may spend nearly half of their time creating functionality that their customers won't use and don't really want.

To emphasize the point, Ambler said that 80 percent of business stakeholders don't want the software that is written to spec (without citing the source).

Still, something isn't quite right with the performance estimates generally expressed about IT software development. For years, research firms like the Standish Group have found that software development teams aren't doing so well. Only 31 percent of software development projects are deemed successful. Ambler said that such stats don't sound right, and so Dr. Dobb's Journal conducted a study in August of this year to allow people to define project success on their own terms. When you do that, you get significantly better success rates, he said.

So what do business stakeholders want? Respondents to the August Dr. Dobbs Journal survey defined a successful software development project as follows:

• Software was delivered when ready as opposed to being on schedule -- 61.3 percent;
• Software was produced with ROI and under budget -- 79.6 percent;
• Software quality was produced that meets needs, regardless of time and budget constraints -- 87.3 percent; and
• Software met the needs of business stakeholders -- 87.3 percent.

Having a quality product at the end of the process appears to be the key yardstick by which business stakeholders measure the success of any particular software development project.

The least likely people to agree on a software development project are the business stakeholders and the project managers, Ambler said. Business stakeholders are also the least likely to put credence into sending the software development process offshore, he added.

Agile development entails a few model strategies. First, you need to look at the whole picture, plus ensure that the architecture is attractive to your customers. There has to be active business stakeholder participation in making applications as opposed to just IT. After all, business stakeholders have made applications using Excel and Word for years. The use of just-in-time modeling is a good approach in combination with testing. Finally, Ambler said that the "good enough" basis is the most efficient model that you can have. If something is good enough, which is the definition of an agile model, then it is done. The good-enough approach becomes possible when the culture embraces change, he added.

A change requirement late in the lifecycle should be considered to be a competitive advantage, as long as you can act on it. Traditional change management, where you try to prevent a requirement from changing, is fundamentally flawed. And that's why business stakeholders don't like IT, Ambler said. He noted that a recent newspaper survey of "the most despised professions" pegged IT at the No. 3 position, just above used car salesmen, and below politicians and lawyers.

Governance is critical to success in agile development, but Ambler compared it to herding cats. The key is in providing the proper motivation for developers (the cats). He cited some successes under the agile process, particularly with regard to getting developers to follow the voting process and do testing.

"We're getting developers to test," he said. "Can you imagine ten years ago predicting that developers would demand to test -- they'd think you're crazy. Yet that is the state today in the agile community."

If you have more and more IT projects, you will need to have an IT governance process in place. However, if you let the bureaucrats define the governance process, which is what has been happening for a couple of decades, then you'll have a bureaucratic governance process. Ambler said you can get an effective governance process if you let the people who are good at software development, who are good at IT, define it.

One agile development tenet is to have self-organizing teams, but that doesn't mean you don't plan. The agile process is planned, but it's done on a just-in-time basis.

"Never let anyone tell you that agile developers don't plan," Ambler said, adding that agile developers simply don't waste time up front with modeling.

You don't want to be doing detailed models anymore. Tests are a significantly better way to specify details than models. Still, the majority (93 percent) of agile developers is doing whiteboard sketching -- they're doing a little bit of modeling, Ambler said.

Finally, developers just aren't good at estimating projects, Ambler said. However, one effective agile management technique is something called "planning poker." Essentially, planning poker is a method of estimating project length by team members where they literally pick a card. It helps motivate developers and get them to work out the reasons for their varying estimates.