I'm involved in a project that uses an online Agile project management system. It's not a big project in terms of number of participants, but the personnel are distributed across four institutions in three different states, none closer than a hundred miles to any of the others. When it becomes manifest, it will be a major achievement in the world of biomedical research, and it may result in technologies with commercial potential, even though they will be delivered under an Open Source license.
As stated in the opening sentence, the project employs the Agile approach. It's the first time I've worked on a project branded as Agile. I'm capitalizing the word to distinguish it from the adjective "agile", because the methodology may be Agile, but that does not guarantee Agile projects are agile in practice.
I should say up front, in case any reader knows the project of which I'm speaking (and I'm sure some will), the project is moving along nicely. It has its ups and downs as with any project, but it is on track to be successful.
My critique here is directed at how the Agile approach as we are practicing it is getting in the way of the project approaching its full potential, which potential is well beyond the contractual scope as originally defined. The limitations I'm describing here could easily cause a different project to fail, but that isn't happening on my project. I'm happy with it, but I could be happier.
The problems I'm seeing with the Agile approach may relate to subtle misapplications of the Agile approach in projects involving distributed teams, and I suspect these are widespread in the industry. These problems stem from tacit cultural biases and preferences.
I say 'misapplications' because people I respect and admire are ardent Agile enthusiasts: for example, long-time friend and colleague from my commercial product development days, Rich Sheridan of Menlo Innovations. If Rich says it works for projects like mine, and he does, I know I have some learning to do. I'm still at odds with his viewpoint at present, but I know I'll probably be convinced.
An example of the kind of tacit cultural bias to which I'm referring is the tendency of software engineers to want someone else to talk to the end user. This is not at all due to laziness or indifference. Often shy and introspective by nature, most productive when left alone in silence, software designers and programmers often believe that the process works best when there is someone who can tell them what needs to be done, and then leave them alone to do it. There are often organizational policies that proscribe developer conversations with end users, and these factor into the equation as well.
This in turn is related to a more pervasive cultural bias, the assumption that intellectual constructs like the requirements or design for a product can have persistent, stable existence independent of the minds of the stakeholders (most importantly, the customers, sponsors, and implementers).If they did have that sort of stability, requirements could be written down and "thrown over the wall" as the saying goes.
Some projects must force this level of stability, for systems engineering reasons -- the software for the NASA Shuttle is an example -- but for most projects, requirements are fluid and dynamic, and worse, are never the same from one stakeholder's mind to another even at any specific point in time. Moreover, this fluidity is not limited to requirements analysis: the same can be said for detailed product design.
I'm not against creating requirements and design artifacts, whether in Agile user stories, in IEEE-standard formal documents, or UML models, as long as everyone who matters in the process understands that the documented version is always incomplete, always wrong, and never incomplete or wrong in the same way twice, i.e. at any two points in time during the life of the project.
In project management, the saying goes, Plans are nothing, but planning is everything"; I would paraphrase that here as "Requirements documents are nothing, but the requirements analysis process is everything."
To see the fallacy in the assumption of stable, documentable requirements, recall the cliched party game "Telephone". One person starts with a somewhat lengthy and usually gossipy anecdote, and whispers it into the next person's ear. That person whispers it into the next person's ear, and it proceeds in this sub-rosa serial fashion until the story has been told to everyone at the party. At the end, the last person recites out loud what the story has become in the series of translations, usually to the merriment of all present.
The telephone game metaphor in software development produces something like the ubiquitous "What the Customer Wanted" cartoons, an example of which is shown here, stolen from Allan Jardine's blog (click to open full-size in a new window):
The lesson of the Telephone game is that communication often happens without comprehension. When done well, projects using the Agile approach should be characterized by lots of communication.It is the ongoing conversation, not the artifacts, that ensure a successful project.
For that reason, when I listen to any software development methodology, I look for the points where meaningful conversations will occur. A 'meaningful conversation' in this context is a timely and thorough conversation about product and process that builds and continually refreshes a shared mental model of the vision, mission, and goals of the project -- a mental model that all participants acknowledge will evolve over time.
If these conversations occur throughout, involve all relevant stakeholders at their points of occurrence, and form the real basis for the collaborative work, the methodology will result in a successful project, barring the occurrence of any of the political, financial, and other disasters that can kill even the most promising efforts.
Rich Sheridan agrees with my premise that we were doing Agile development back in 1986 -- in fact, all the successful commercial projects we knew about used some methodology that would fit into the Agile paradigm. What we lacked was a trademarked brand wrapped around a coherent description of some incarnation of our "methodology". Rich learned that lesson well, and his group has registered a trademark for the term High-Tech Anthropology®. Menlo peddles a methodology, to put it crassly, but it's no con game -- what they teach you is eminently workable in practice, and I highly recommend their services.
That said, I think Rich would agree that their flavor of the Agile approach works as well as it does because it is an incarnation of what I will now name the Conversational Meta-Methodology, rather than as a result of any rote technique that they teach you. In fact, one of the things they will teach you is that all the rote techniques are intended to do is to work toward a shared mental model of the project as it evolves over time. It's the process, not the artifacts, that are the key to success.
To disclose my personal interest in this matter, I wrote a book back in 2002-2003 called Design Conversations: Envisioning Content-Rich Enterprise Systems. It's as much about requirements analysis as it is about design. I've gotten great feedback from many of the several dozen people who have read it (including a few who downloaded it to their Kindles), but it hasn't started a movement. Maybe it's time to trademark the Conversational Meta-MethodologyTM, write a manifesto, work up some curricula and a YouTube video or three, and found an institute of my own. :-)
I'll come back to this in a soon-to-be-written post and show how it relates to my current project.