Lessons from a Yahoo Scrum Rollout

Yahoo's top coach in Agile practices describes the process, and how it speeds up Web application delivery

• By Kurt Mackie
• 02/05/08

Yahoo has grown from its initial dotcom roots. However, to stay competitive with all of that growth, the company began experimenting with Agile development techniques about three years ago. Now, Yahoo has more than 200 teams using agile development processes to create software for the highly volatile general-public Web application market.

A key organizer of those teams is Gabrielle Benefield, Yahoo's senior director of methods and practices. I spoke with her this month, and she answered a few questions about the Agile development process and Scrum, and how these methods have improved software development at Yahoo.

Kurt Mackie: Would you describe Scrum as a variant of Agile development?

Gabrielle Benefield: Think of Agile as the umbrella. So what happens is that there are different methods and practices around [it]. Scrum is extreme programming. In Europe, they have something called BSDM that will help with a plethora of things. The Agile Alliance was founded by a group of people, 17 of them, that said, "Hey, we're doing good works; we have all of these common things--what are they?" They came out with a set of values that drive the whole movement. The easiest way to describe it is that [software is developed in] small groups, which means small cycles and incremental [changes], which means that you release pieces of software rather than the typical big bang. Agile is the web describing the commonality between these things, which is very values driven, and Scrum is one of the methods. Scrum is a very lightweight framework in which you can adapt--and it's not just confined to software.

KM: Why settle on Scrum?

Benefield: Scrum is great etiquette for an organization. What happens is that if you want something that's very lightweight that can make a huge impact quickly, and be really simple to implement in its simplest form, Scrum is the way to go. It's really successful when you add on the engineering practices, things like some of the extreme programming techniques where they do a lot of testing, where you continuously build your software--things that improve the quality a lot. So Scrum tends to focus on a lot of collaboration, getting the team working, getting prioritization clearly defined, and then you add the engineering practices. And there are things called Lean that came out of the local processes from companies like Toyota, when they manufacture cars--ways of ridding waste through an organization, looking at [ways of] developing things called "just in time." So a lot of these influences are very strong in the Agile community.

We actually are very pragmatic. You'll find that there are some people who are very much "by the book," but we tend to want to adapt [scrum] to your own environment and even in the context of individual teams. You need to be really aware about people's needs.

KM: It sounds like Scrum has some variation per project, but there is a set of practices to Scrum, right?

Benefield: Yes, that's correct. There are some things you don't want to mess with too much, especially when a team starts out. For example, we have meetings every day for 15 minutes, called "the daily Scrum." And that's where the whole team talks about what they've achieved in the last meeting or what are we about to work on today, and is there anything that's blocking me. So people ask, "Why do we have to do this every day?" and "Can't we just do it a couple of times a week?" But there are critical benefits. One is that you always know where a project is, because we care about visibility. We…know on a day if something's wrong--the build's broken or the machines are down--[and] if that happens, we need to resolve it immediately. So it's a risk mitigation strategy. [Scrum includes] the notion of constant communication and visibility and the values we try to drive. It's not a market management technique--it's just a way for them to communicate and resolve problems together.

KM: I see you worked for CollabNet at one point. Is that sort of collaboration method used in Scrum?

Benefield: I do come from a big open source background and the idea of collaboration, [where] you could distribute to people all around the globe. The [CollabNet] tool really helped to get these people talking. My team worked very collaboratively and that's how we came to adapt the Agile concepts. It was the only way we could start software. We had to continuously know what state our software was in. We were continuously building it and releasing it in small chunks. So a lot of those things very much influenced me in the Agile movement.

KM: A lot of developers may want to work solitary, but that seems to be part of the problem with these software projects. Is Scrum just a way to keep people engaged in meeting the project's goals?

Benefield: Well, I think one of the key points of why this works is that we are dealing with people. We tend to boil the software development process down to the tools we are using, complicated tracking methods, check lists and task allocation--saying, "You work on this; you work on that." Things that are very demotivating. But people will actually self-organize. For example, we let the team do the planning. The business will come to you and say, "These are our priorities. This is what's important. This is what we want to do." The team will decide how they want to do it. So instead of having the traditional project manager say, "Here's your task, do this--how long is it going to take?" We say, "You work out the tasks," and people volunteer for them, and they help each other. You know "Scrum" comes from rugby. The team can only score a goal if the team works together.

It is also interesting that software developers often can be sort of loners; they're a little more introverted. So these [collaboration] tendencies come from engineering saying, "How can we do a lightweight way of communicating that people feel comfortable with." At first it's a little different. And when I introduce teams, I try to get them to sit together, and that's sometimes hard. They will be resistant, but after a while, they'll see the benefits--to talk and get feedback and work together.

KM: How do you define the project owners and the stake owners?

Benefield: The stakeholders are the people with a stake in the project. The ones who write the checks ultimately are the ones defining what they need. Obviously, being Yahoo, our consumers are our biggest focus. We care about the people using software.

Agile works a lot with techniques and requirements called user stories. In it, we talk about the types of users we have. We try to release small parts of our software--as well as during lab testing. We release small amounts to all of our customers, and of course we've got millions of them, and get them to test and ask, "Is this what you want?" And then we try a little bit more. It's like this taste-testing with Agile, which is really effective, because if it's not the right product, we know very quickly. We want to release it every month, a few features at a time. So we can say, "Are we meeting the needs," and as the market changes, that lets us be a little more flexible.

KM: Who are the chickens and who are the pigs in Agile development?

Benefield: If you are talking to Agile people and you want to start a flame war, talk about chicken and pigs. There's a big group called Scrum Development, and every six months someone will talk about chicken and pigs and it will start this huge flame war. Some people used it, especially in the early days, to talk about the idea that chickens are people who flap their wings and squawk, and make a lot of noise, and the pigs are the ones really doing the work in the trenches. It was [a story] to empower people. I had to use that in one company, however, at Yahoo, we are one big culture, and I want to break down those walls --so we cannot talk about chickens and pigs, except for referencing it in history. We have some guiding rules that we tend to be very good about-- including the business and sharing knowledge--so that we are not creating conflicts in the team.

KM: So who enforces that?

Benefield: "Enforcement" is an interesting word. There's a certain amount of control in it, saying, "You must do this." I'm more about people trying to understand the benefits [of meeting]. We're dealing with dotcom, very creative, very smart people. It's worse than an insult to tell them to just do it. So we tend to want to explain the reason and value behind it. Generally, we coach the teams. We work with them and show them the difference, so they get to experience the difference.

KM: Where is the decision-making coming from?

Benefield: The way it works is that the Scrummaster is the person who understands the process mostly and coaches the team to follow the processes. We have an international coaching team, which is the one I run. So we work with the teams to kick them off; then drop into some team meetings and check up with them. But the team will be very self-adjusting. We have something called "the retrospective" at the end of each iteration, and that's the time for the team to look at their process and say what's working, what's not and do we need to change. When they have failures, we usually get involved to help them out.

KM: What was the experience that turned Yahoo toward Agile and Scrum development?

Benefield: Some companies come from very heavy bureaucratic processes. Yahoo started as a small startup and then they grew very quickly. What they were doing initially was very Agile, so the founders would be in the same room working on the software. Then, as they grew, they got to the point where they needed to communicate more. At that point, they came up with a traditional waterfall process. Waterfall tends to be put in place where people want something more formalized. It looks good on paper and it's very easy to write up and distribute, so it looks wonderful. But it was hard to implement that because we had a startup culture. We had very smart, creative people [and] they are used to just getting their ideas out. This was very unnatural to them. No one liked it. It was slowing them down. An engineer on the team started doing some evangelism, saying "Hey, we should be more Agile." We acquired a search company and they were into Scrum. It played on people's needs for a more lightweight process. So in February of 2005, we kicked off the first pilot teams [of Agile development].

KM: How do you become certified?

Benefield: The Scrum Alliance organization. There are about 50 certified Scrum trainers globally. It's a very small community. I've worked a lot with Ken Schwaber and Jeff Sutherland, who were the co-creators of Scrum. I was probably trainer No. 14 at the time.

KM: Is it true that applications get out the door quicker with Agile development?

Benefield: Just the fact that [we are] building software very early on [and] we don't take months to do planning--that tends to get software out faster in the first place. Productivity-wise, Agile teams that I've worked with, especially when I've done a lot of coaching with them, I can easily get 200 to 300 percent productivity improvements. Some teams do extremely well because they've really adopted Agile. They follow all of the practices extremely well. Other teams that may not have as much training and coaching or there are systemic blocks--then you might not see as much [improvement]. We see a range, and overall--looking at the worst teams--we're seeing around 35 percent to 36 percent productivity improvement. All things, considered, that's a fairly conservative estimate.

KM: Is change a good thing in the Agile/Scrum world?

Benefield: I think it's very human to change our minds. As soon as you start building a product and you get to see it, you want to start making changes. Instead of saying, "You can't change it," we really embrace [change]. In the Internet space, you have to be able to change really quickly. An example is that we were building the backend of mass storage for Yahoo Mail. Partway through, a competitor offered a huge amount of storage for mail. So we had to react incredibly quickly, and if we had been doing traditional waterfall, there's no way we could have [done it], and we were able to compete and meet that competitive threat very quickly. We've seen that [happen] time and time again with products.

Traditionally, companies do change management, and they add all of these change requests, which is the way that companies make money. It looks like they are on time and budget, but the projects can take twice as long because of all of these change requests that get added on. It's like putting a bill through Congress--people lobby and add on things.

KM: If you develop with the idea of changing quickly, does that pull you back toward a more prescribed method?

Benefield: Agile is very disciplined. We are very disciplined about things like having tests around all of your code, so you have the courage to make a change and understand what effect that has. It's not chaos where everyone is trying out ideas. We allow the teams to self-organize and the ideas flow out of the team. The management and the product person have ultimate control over which ideas we go with. So they prioritize from the business side; they are very much in touch with that. They have ultimate control on when we release product, what we are releasing and knowing about the market. The team is making decisions about how they do the work itself. You want to be thinking modular. You don't want to architect solutions that keep you along particular pathways where you can't make change.

KM: How frequently do you test?

Benefield: [Agile teams] call themselves "test infected." The developer or the product person will work to explain their requirements in terms of something called acceptance tests. For example, some Visa card will pass that test, but an American Express card will fail. They are very clear about the customer value and acceptance criteria tests. The developer will write a test first and they will execute the test, and we know it will fail because there's no code yet. They will write the code and run the test. If the code passes, great; if not, they fix the code immediately. We move testing from the end up to the front, because it's more expensive to find bugs at the end.

KM: Where is this all going? Will Agile be the method?

Benefield: At Yahoo, we would never want to mandate this process. I think the process is one that stands on its own and people choose to use it. Over time, it's become very dominant. One of the cofounders at Yahoo said that Agile has been one of the most positive things to happen to the company. Over time, to truly compete, I believe that Agile is the best thing for the company.

Editor's note: Readers interested in the effectiveness of Agile development processes at Yahoo can find further details in Gabrielle Benefield's paper, "Rolling out Agile in a Large Enterprise," which is available here.