Disciplined Agile Delivery Read online

  Scott and Mark not only made me think, but they also reminded me of lots of things that I had forgotten—things that the agile fashion police have made uncool to talk about. This book is not about fashionable agile; it is about serious change, and it should be required reading for any change leader.

  Dave West @davidjwest

  Chief Product Officer, Tasktop, and former VP and Research Director, Forrester Research

  Preface

  The information technology (IT) industry has an embarrassing reputation from the perspective of our customers. For decades we have squandered scarce budgets and resources, reneged on our promises, and delivered functionality that is not actually needed by the client. An outsider looking at our profession must be truly baffled. We have so many process frameworks and various bodies of knowledge such that we ourselves have difficulty keeping up with just the acronyms, let alone the wealth of material behind them. Consider: PMBOK, SWEBOK, BABOK, ITIL®, COBIT, RUP, CMMI, TOGAF, DODAF, EUP, UML, and BPMN, to name a few. Even within the narrow confines of the agile community, we have Scrum, XP, CI, CD, FDD, AMDD, TDD, and BDD, and many others. There is considerable overlap between these strategies but also considerable differences. We really need to get our act together.

  Why Agile?

  On traditional/classical projects, and sadly even on “heavy RUP” projects, basic business and system requirements often end up written in multiple documents in different fashions to suit the standards of the various standards bodies. Although in some regulatory environments this proves to be good practice, in many situations it proves to be a huge waste of time and effort that often provides little ultimate value—you must tailor your approach to meet the needs of your situation.

  Fortunately, agile methods have surfaced over the past decade so that we can save ourselves from this madness. The beauty of agile methods is that they focus us on delivering working software of high business value to our customers early and often. We are free to adjust the project objectives at any time as the business needs change. We are encouraged to minimize documentation, to minimize if not eliminate the bureaucracy in general. Who doesn’t like that?

  More importantly, agile strategies seem to be working in practice. Scott has run surveys1 within the IT industry for several years now, and he has consistently found that the agile and iterative strategies to software development have consistently outperformed both traditional and ad-hoc strategies. There’s still room for improvement, and this book makes many suggestions for such improvements, but it seems clear that agile is a step in the right direction. For example, the 2011 IT Project Success Survey revealed that respondents felt that 67% of agile projects were considered successful (they met all of their success criteria), 27% were considered challenged (they delivered but didn’t meet all success criteria), and only 6% were considered failures. The same survey showed that 50% of traditional projects were considered successful, 36% challenged, and 14% failures. The 2008 IT Project Success survey found that agile project teams were much more adept at delivering quality solutions, good return on investment (ROI), and solutions that stakeholders wanted to work with and did so faster than traditional teams. Granted, these are averages and your success at agile may vary, but they are compelling results. We’re sharing these numbers with you now to motivate you to take agile seriously but, more importantly, to illustrate a common theme throughout this book: We do our best to shy away from the overly zealous “religious” discussions found in many software process books and instead strive to have fact-based discussions backed up by both experiential and research-based evidence. There are still some holes in the evidence because research is ongoing, but we’re far past the “my process can beat up your process” arguments we see elsewhere.

  Alistair Cockburn, one of the original drafters of the Agile Manifesto, has argued that there are three primary aspects of agile methodologies:

  • Self-discipline, with Extreme Programming (XP) being the exemplar methodology

  • Self-organization, with Scrum being the exemplar methodology

  • Self-awareness, with Crystal being the exemplar methodology

  As you’ll see in this book, Disciplined Agile Delivery (DAD) addresses Cockburn’s three aspects.

  Why Disciplined Agile Delivery?

  Although agile strategies appear to work better than traditional strategies, it has become clear to us that the pendulum has swung too far the other way. We have gone from overly bureaucratic and document-centric processes to almost nothing but code. To be fair, agile teams do invest in planning, although they are unlikely to create detailed plans; they do invest in modeling, although are unlikely to create detailed models; they do create deliverable documentation (such as operations manuals and system overview documents), although are unlikely to create detailed specifications. However, agile teams have barely improved upon the results of iterative approaches. The 2011 IT Project Success survey found that 69% of iterative projects were considered successful, 25% challenged, and 6% failures, statistically identical results as agile projects. Similarly, the 2008 IT Project Success survey found that agile and iterative teams were doing statistically the same when it came to quality, ability to deliver desired functionality, and timeliness of delivery and that agile was only slightly better than iterative when it came to ROI. The reality of agile hasn’t lived up to the rhetoric, at least when we compare agile strategies with iterative strategies. The good news is that it is possible to do better.

  Our experience is that “core” agile methods such as Scrum work wonderfully for small project teams addressing straightforward problems in which there is little risk or consequence of failure. However, “out of the box,” these methods do not give adequate consideration to the risks associated with delivering solutions on larger enterprise projects, and as a result we’re seeing organizations investing a lot of effort creating hybrid methodologies combining techniques from many sources. The Disciplined Agile Delivery (DAD) process framework, as described in this book, is a hybrid approach which extends Scrum with proven strategies from Agile Modeling (AM), Extreme Programming (XP), and Unified Process (UP), amongst other methods. DAD extends the construction-focused lifecycle of Scrum to address the full, end-to-end delivery lifecycle2 from project initiation all the way to delivering the solution to its end users. The DAD process framework includes advice about the technical practices purposely missing from Scrum as well as the modeling, documentation, and governance strategies missing from both Scrum and XP. More importantly, in many cases DAD provides advice regarding viable alternatives and their trade-offs, enabling you to tailor DAD to effectively address the situation in which you find yourself. By describing what works, what doesn’t work, and more importantly why, DAD helps you to increase your chance of adopting strategies that will work for you.

  Indeed there are an increasing number of high-profile project failures associated with agile strategies that are coming to light. If we don’t start supplementing core agile practices with a more disciplined approach to agile projects at scale, we risk losing the hard-earned momentum that the agile pioneers have generated.

  This book does not attempt to rehash existing agile ideas that are described in many other books, examples of which can be found in the references sections. Rather, this book is intended to be a practical guide to getting started today with agile practices that are structured within a disciplined approach consistent with the needs of enterprise-scale, mission-critical projects.

  What Is the History?

  The Disciplined Agile Delivery (DAD) process framework began as a concept in 2007 that Scott worked on in his role as chief methodologist for agile and lean at IBM® Rational®. He was working with customers around the world to understand and apply agile techniques at scale, and he noticed time and again that organizations were struggling to adopt mainstream agile methods such as Extreme Programming (XP) and Scrum. At the same time Mark, also working with organizations to adopt and apply agile techniques in practice, observed the same
problems. In many cases, the organization’s existing command-and-control culture hampered its adoption of these more chaordic techniques. Furthermore, although many organizations were successful at agile pilot projects, they struggled to roll out agile strategies beyond these pilot teams. A common root cause was that the methods did not address the broader range of issues faced by IT departments, let alone the broader organization. Something wasn’t quite right.

  Separately we began work on addressing these problems, with Scott taking a broad approach by observing and working with dozens of organizations and Mark taking a deep approach through long-term mentoring of agile teams at several organizations. In 2009 Scott led the development of the DAD process framework within IBM Rational, an effort that continues to this day. This work included the development of DAD courseware, whitepapers, and many blog postings on IBM developerWorks®.3

  What About Lean?

  There are several reasons why lean strategies are crucial for DAD:

  • Lean provides insights for streamlining the way that DAD teams work.

  • Lean provides a solid foundation for scaling DAD to address complex situations, a topic we touch on throughout the book but intend to address in greater detail in a future book.

  • Lean principles explain why agile practices work, a common theme throughout this book.

  • Lean strategies, particularly those encapsulated by Kanban, provide an advanced adoption strategy for DAD.

  So why aren’t we writing about Disciplined Lean Development (DLD) instead? Our experience is that lean strategies, as attractive and effective as they are, are likely beyond all but a small percentage of teams at this time. Perhaps this “small” percentage is 10% to 15%—it’s certainly under 20%—but only time will tell. We’ve found that most development teams are better served with a lightweight, end-to-end process framework that provides coherent and integrated high-level advice for how to get the job done without getting bogged down in procedural details. Having said that, many of the options that we describe for addressing the goals of the DAD process framework are very clearly lean in nature, and we expect that many teams will evolve their process from a mostly agile one to a mostly lean one over time.

  DAD is the happy medium between the extremes of Scrum, a lightweight process framework that focuses on only a small part of the delivery process, and RUP, a comprehensive process framework that covers the full delivery spectrum. DAD addresses the fundamentals of agile delivery while remaining flexible enough for you to tailor it to your own environment. In many ways, Scrum taught agilists how to crawl, DAD hopes to teach agilists how to walk, and agility@scale and lean approaches such as Kanban will teach us how to run.

  How Does This Book Help?

  We believe that there are several ways that you’ll benefit from reading this book:

  • It describes an end-to-end agile delivery lifecycle.

  • It describes common agile practices, how they fit into the lifecycle, and how they work together.

  • It describes how agile teams work effectively within your overall organization in an “enterprise aware” manner, without assuming everyone else is going to be agile, too.

  • It uses consistent, sensible terminology but also provides a map to terminology used by other methods.

  • It explains the trade-offs being made and in many cases gives you options for alternative strategies.

  • It provides a foundation from which to scale your agile strategy to meet the real-world situations faced by your delivery teams.

  • It goes beyond anecdotes to give fact-based reasons for why these techniques work.

  • It really does answer the question “how do all these agile techniques fit together?”

  Where Are We Coming From?

  Both of us have seen organizations adopt Scrum and extend it with practices from XP, Agile Modeling, and other sources into something very similar to DAD or to tailor down the Unified Process into something similar to DAD. With either strategy, the organizations invested a lot of effort that could have been easily avoided. With DAD, we hope to help teams and organizations avoid the expense of a lengthy trial-and-error while still enabling teams to tailor the approach to meet their unique situation.

  Scott led the development of DAD within IBM Rational and still leads its evolution, leveraging his experiences helping organizations understand and adopt agile strategies. This book also reflects lessons learned from within IBM Software Group, a diverse organization of 27,000 developers worldwide, and IBM’s Agile with Discipline (AwD) methodology followed by professionals in IBM Global Service’s Accelerated Solution Delivery (ASD) practice. In the autumn of 2009 DAD was captured in IBM Rational’s three-day “Introduction to Disciplined Agile Delivery” workshop. This workshop was rolled out in the first quarter of 2010 to IBM business partners, including UPMentors, and Mark became one of the first non-IBMers to be qualified to deliver the workshop. Since then, Mark has made significant contributions to DAD, bringing his insights and experiences to bear.

  What’s The Best Way to Read this Book?

  Most people will want to read this cover to cover. However, there are three exceptions:

  • Experienced agile practitioners can start with Chapter 1, “Disciplined Agile Delivery in a Nutshell,” which overviews DAD. Next, read Chapter 4, “Roles, Rights, and Responsibilities,” to understand the team roles. Then, read Chapters 6 through 19 to understand in detail how DAD works.

  • Senior IT managers should read Chapter 1 to understand how DAD works at a high level and then skip to Chapter 20, “Governing Disciplined Agile Teams,” which focuses on governing4 agile teams.

  • People who prefer to work through an example of DAD in practice should read the case study chapters first. These are: Chapter 12, “Initiating a Disciplined Agile Delivery Project—Case Study”; Chapter 17, “Case Study: Construction Phase”; and Chapter 19, “Case Study: Transition Phase.”

  We hope that you embrace the core agile practices popularized by leading agile methods but choose to supplement them with some necessary rigor and tooling appropriate for your organization and project realities.

  Incidentally, a portion of the proceeds from the sale of this book are going to the Cystic Fibrosis Foundation and Toronto Sick Kid’s Hospital, so thank you for supporting these worthy causes.

  The Disciplined Agile Delivery Web Site

  www.DisciplinedAgileDelivery.com is the community Web site for anything related to DAD. Mark and Scott are the moderators. You will also find other resources such as information on DAD-related education, service providers, and supporting collateral that can be downloaded. We invite anyone who would like to contribute to DAD to participate as a blogger. Join the discussion!

  Abbreviations Used in This Book

  Acknowledgments

  We’d like to thank the following people for their feedback regarding this book: Kevin Aguanno, Brad Appleton, Ned Bader, Joshua Barnes, Peter Bauwens, Robert Boyle, Alan L. Brown, David L. Brown, Murray Cantor, Nick Clare, Steven Crago, Diana Dehm, Jim Densmore, Paul Gorans, Leslie R. Gornig, Tony Grout, Carson Holmes, Julian Holmes, Mark Kennaley, Richard Knaster, Per Kroll, Cherifa Liamani, Christophe Lucas, Bruce MacIsaac, Trevor O. McCarthy, M.K. McHugh, Jean-Louise Marechaux, Evangelos Mavrogiannakis, Brian Merzbach, Berne C. Miller, Mike Perrow, Andy Pittaway, Emily J. Ratliff, Oliver Roehrsheim, Walker Royce, Chris Sibbald, Lauren Schaefer, Paul Sims, Paula Stack, Alban Tsui, Karthikeswari Vijayapandian, Lewis J. White, Elizabeth Woodward, and Ming Zhi Xie.

  We’d also like to thank the following people for their ideas shared with us in online forums, which were incorporated into this book: Eric Jan Malotaux, Bob Marshall, Valentin Tudor Mocanu, Allan Shalloway, Steven Shaw, Horia Slusanschi, and Marvin Toll.

  About the Authors

  Scott W. Ambler is Chief Methodologist for IT with IBM Rational, working with IBM customers around the world to help them to improve their software processes. In addition to Disciplined Agile Delivery (DAD), he is th
e founder of the Agile Modeling (AM), Agile Data (AD), Agile Unified Process (AUP), and Enterprise Unified Process (EUP) methodologies and creator of the Agile Scaling Model (ASM). Scott is the (co-)author of 20 books, including Refactoring Databases, Agile Modeling, Agile Database Techniques, The Object Primer, 3rd Edition, and The Enterprise Unified Process. Scott is a senior contributing editor with Dr. Dobb’s Journal. His personal home page is www.ambysoft.com.

  Mark Lines co-founded UPMentors in 2007. He is a disciplined agile coach and mentors organizations on all aspects of software development. He is passionate about reducing the huge waste in most IT organizations and demonstrates hands-on approaches to speeding execution and improving quality with agile and lean techniques. Mark provides IT assessments and executes course corrections to turn around troubled projects. He writes for many publications and is a frequent speaker at industry conferences. Mark is also an instructor of IBM Rational and UPMentors courses on all aspects of software development. His Web site is www.UPMentors.com. Mark can be reached at [email protected].

  Part 1: Introduction to Disciplined Agile Delivery (DAD)

  Chapter 1. Disciplined Agile Delivery in a Nutshell

  For every complex problem there is a solution that is simple, neat, and wrong.