Software Development as a Creative Process
“How big is this feature? We need an estimate.” The project manager waits for the developer to play his predictable card. “It depends. It could be 2 hours, but I might have to refactor, which could take 2 months.” Not a very useful exchange.
In software, it’s no secret that meetings between project managers and developers can get painful for both sides. With all the tools and processes we have at our disposal, projects still consistently blow budgets and trample deadlines.
Why does this happen? Why can’t developers ever give a straight answer? Why do project managers always seem to be asking the wrong questions? The root cause is a mismatch between the type of problems development presents and the metaphors we use to model them.
Our industry has always managed software development using metaphors from manufacturing and construction. We use language like “groundwork”, “build”, and “defects”, but the metaphors run much deeper. They shape the way project managers, product owners, and the general public think about development and consequently the processes and expectations developers work under.
Sure, there are some stages of some projects that resemble construction, but more often than not, especially for new applications, the metaphor is completely misapplied.
Take the Healthcare.gov meltdown for example. The failure and the subsequent “Tech Surge” that followed is a classic example of construction metaphor applied to software. The implicit assumption is that the work is linearly cumulative, and that more workers equal more units of output. We all know this isn’t true, but is it because developers can’t work together or is there something wrong with the concept of “output” as it relates to development work?
Another example is the relentless requests project managers make for estimates. The assumption is that the work follows a known path that can be measured and estimated. We all know developers are notoriously terrible at estimates. Why is that? Are they all just stupid and lazy, or is there something wrong with the framing of question?
There’s even been writing directly equating developers with factory workers, destined to be automated out of a job. The assumption is that the work is well defined and repetitious. If developers are the factory workers, then whose job is it to create the software that automates them and why has demand for them only grown?
The examples go on, and as the questions pile up, we start to get a nagging sense that there’s a mismatch between who we are and who we’re supposed to be.
Software isn’t Built, it’s Invented
Let’s untether ourselves for a minute from the world of physical goods and try to see software development as a creative problem. Creative problems like designing an interface, writing a novel, inventing a machine, or composing a symphony have much more in common with the challenges of software development than does building a house or manufacturing a car.
Creative problems have an undiscovered solution. Edison’s creative problem was to create a practical electric light. His solution was the light bulb. Leonardo da Vinci’s problem was to create a portrait of Lisa del Giocondo. His solution was the Mona Lisa. These problems have infinite solutions. They’re worked on and refined until a satisfactory one is discovered.
Some might argue that software has specifications: design documents, business requirements, visual mockups and so on, which specify a single solution, but they don’t.
From a development perspective, those documents are the problem definition, not the solution. Software can only be specified in code, in the same way that a book can only be specified through its writing. Code itself is nothing more than an unambiguous and complete specification. If business requirements were complete specifications, you’d be able to compile and run them, but you can’t.
Coding is a creative problem. Every time a developer writes code, he is taking a non-deterministic path towards an unknown solution — a unique solution that has never been created before. If the work weren’t unique, he wouldn’t need to write it, he’d simply copy and paste it.
Looking at development through this lens, it’s clear that there’s a mismatch between the type of problems we’re facing and the processes we use to solve them today. Even Agile methodologies are still rooted in the flawed construction metaphor. The mismatch between the type of problems we face and the way we choose to model them is responsible for much of the pain, frustration, and absurdity we’ve come to accept as the norm in our industry today.
The Creative Process
If software development is truly a creative problem, maybe we should look for inspiration from other fields solving creative problems.
We can start with our closest neighbors, the friendly UX and design folks we work alongside. There is a lot written about design thinking and the creative process, and design shops like Zurb have been kind enough to publish the exact processes they use to design products. The steps are basically:
-
Ideate — Experiment with divergent solutions and distill them.
-
Test & Learn — Use the prototype to test, learn and improve.
How can we apply these ideas to a development process?
Developers already do some of these creative tasks, but they’re never officially part of the process and almost always under the radar. No project manager wants to hear her tech team is “ideating.” It’s seen as an emergency measure — like “down time” for a factory line, to use our manufacturing metaphor. If software is a creative problem, then we should build exploration into the process.
Imagine what a time-boxed development sprint modeled after the design process might look like.
It could start with bringing the development team together to identify and document key problems. Developers could then individually break out and research/test diverse solutions to those problems. Without the pressure of shipping features, they could validate new methods and patterns for solving the given problems.
In a convergence meeting, the team could share and distill the diverse ideas into consensus recommendations. Developers receive feedback from other team members who may have envisioned solutions in completely different ways. Armed with a consensus direction, the team then builds out the set of features using the patterns they’ve agreed on.
As a final step, the team should instrument and measure the work product in an objective way against success criteria like performance, organization, and complexity to inform the challenges of the next sprint.
Would this exact process actually work? It’s hard to say. The example just serves to illustrate that we don’t have to look at development purely through the rigid lens of linear, incremental construction. Even if the process we described doesn’t work as a whole, nobody can deny that it includes obviously valuable development activities that are completely overlooked today.
Now What?
Acceptance is the first step. Writing software is a creative problem and today’s development processes are still in denial about that. We need to move beyond 19th century assembly line thinking and change the way we fundamentally think about the art of software development.
We should take a close look at creative processes from adjacent industries for inspiration. How do inventors design revolutionary new devices? How do engineers design a unique skyscraper? How are factories designed rather than operated? What can we learn from these processes? Let’s start drawing new metaphors and tweaking our processes a little bit at a time and see where it goes.
We’ve been peddling the Agile process for nearly 15 years now. That’s a long time for an industry that moves as fast as ours. If we can replace the broken metaphors in our heads and be honest with ourselves about the challenges we face, we can move our industry forward and open doors to entirely new ways of working.