Innovation in Software

[Up-front Disclaimer: I didn't do Computer Science at University. I did "Software Engineering". It involved formal methods and quite a lot of Z notation. I have some pretty strong views on Software Engineering as a discipline as a result. Most of my CompSci colleagues generally do not share my beliefs.]

Johannes Ernst posted yesterday a pointer to an article from IEEE Software by Tom DeMarco, pronouncing Software Engineering as “an idea whose time has come and gone”. Tom’s argument can basically be summed up as:

[...] you need to distinguish between two drastically different kinds of projects:

• Project A will eventually cost about a million dollars and produce value of around $1.1 million
• Project B will eventually cost about a million dollars and produce value of more than $50 million

What’s immediately apparent is that control is really important for Project A but almost not at all important for Project B. This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.

To my mind, the question that’s much more important than how to control a software project is, why on earth are we doing so many projects that deliver such marginal value?

Can I really be saying that it’s OK to run projects without control or with relatively little control? Almost. I’m suggesting that first we need to select projects where precise control won’t matter so much. Then we need to reduce our expectations for exactly how much we’re going to be able to control them, no matter how assiduously we apply ourselves to control.”

Frankly, I can’t believe such a narrow and misguided argument has come from the pen of Tom DeMarco.

Imagine if you will the next time I go into tender on a project.

Customer: So how much will this project cost to deliver, and how long will it take?
Me: Well, how much value is it going to add to your business?
Customer: Pardon?
Me: Is it going to add a little bit of value to your business, or a lot?
Customer: A lot, I expect.
Me: In that case, I have no idea. Let’s just roll with it and see where it goes, eh?

I’m not convinced that’ll go down well.

There are a few logical mistakes in Tom’s argument (and I can’t believe I’m having to point them out), that basically means he’s talking a great big pile of crap:

1. We can not assess the true value of software until it is shipped.

   We might be able to sense the potential value, but it’s a guess.

   I have one project on my books now with a mid-sized build budget, potential value in the hundreds of millions. Another project for a smaller build budget with a potential value of £200k-£300k. Another project with a tiny build budget that if it goes viral could be worth billions five years from now.

   Or maybe the project with the tiny build budget sinks, the medium project gets on NASDAQ in 5 years and the biggest project eeks out a living but never goes stellar.

   We simply don’t know. We can’t know. We’re guessing. There are too many variables outside of the software engineering process to be able to assess the potential value accurately.

   In essence, we have to treat all projects as being potentially of very high value, because they all are.

2. Value is potentially nearly unlimited, budgets are not.

   It might be well and dandy to say that it doesn’t matter if the budget might slip because the value is so high, but budgets are fixed.

   Thousands of software builds are being worked on right now where a 5% budget slip is potentially going to kill the project because there just isn’t more money after the budget is gone, and without being feature-complete the software is going to have near-zero value.

   Yes, incremental development is the way to do things, yes we should be able to stop and ship after each day’s work. However to fulfil the project’s objectives (and maximise its potential returnable value), we must aim to get all the features in to the final build within budget.

3. There are other factors beyond budget, schedule and value to control.

   John Glenn once quipped about being sat on the launchpad, “I felt about as good as anybody would, sitting in a capsule on top of a rocket that were both built by the lowest bidder.

   However, whilst budget was important, he shouldn’t have felt in danger: the political ramifications of a death would have meant that the people who built that rocket and capsule were going to make sure he survived.

   And some software systems are the same. What is the “value” of a nuclear power station control system? Or the control systems in a fly-by-wire aircraft? Do we care more about the financial value or the safety value? I would suggest that controlling the build – engineering the build – in these situations is critical because the safety value is very high.

   In fact, it betrays immediately the fallacy of Tom’s that only the marginal projects need tight controlling – if you’re not controlling the processes around a software build that keeps 400 souls alive at 35,000 feet, you really should be in another profession.

There is a lot of cruft in software engineering. Waterfall methodologies are now considered almost laughable. However, it’s a young discipline and we have barely begun to understand the correct way to engineer software.

Agile philosophies are taking over in development shops because it allows for less process, more communication and ultimately better software being produced at the other end.

Behaviour-Driven and Test-Driven Development methodologies are providing useful benefits to our clients in a way we could barely imagine just a couple of years ago. In certain situations, formal methods and algebraic proofs are confirming that software is “correct” and “complete” in such a way we can literally bet our lives on it.

Tom admits himself he’s not on the front-line of software development these days. Perhaps his metric-centric universe is correctly condemned, but most of us aren’t using those methods anyway. We’re looking at perhaps a few numbers instead: test coverage, remaining days in budget/schedule, remaining features to be implemented. We’ve moved on. Perhaps he just doesn’t get what we’re doing in dev shops these days.

And we’re still learning new techniques, new ways of doing less process, more quality software. Software Engineering is definitely not something that has “come and gone”: it has barely got its foot in the door.

Towards the end though, Tom does make one point I can’t agree with more:

Consistency and predictibility are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

This reminded me of a talk given by Ted Nelson at OpenTech 2005 where at one point he almost screamed:

I didn’t get into computers to automate trivial crap! I got into it to change the World!

Didn’t we all? But I think we can do it without having to just free-ride our way along with our fingers crossed, hoping we don’t run out of money.

In this article, I’ll be exploring some of the issues around creative thinking and software development. This is a theme I expect to be re-visiting a lot over the next few weeks, so these are more high-level riffs. Each of the major points here needs expanding upon, but I’m interested in seeing what people out there lock into as the most interesting aspects of this.

I believe that raw creativity is not normally associated with the development side of the software industry. This is an error that hampers development, innovation and even the core process of programming. Creative thinking should be taught to all programmers and embraced as a way of working.

Typically only the people involved in the user interface, information architecture and marketing stages of the development process get to do the “original vision thing”. Whilst this goes on, programmers are expected to behave like glorified plumbers by connecting interactions together with a logic that meets the spec and passes tests. This skews the development process towards the wrong people.

Project Managers and Information Architects are going to be aghast – Designers will be screaming – but it is the “Developers” who are the best placed to understand what is and isn’t possible. When asked to be involved in the earliest stages, or in stages they’re not normally invited into (yes, even marketing meetings) and encouraged to contribute to the creative process, several benefits are likely to occur:

1. The programmer becomes more involved in the project and is less likely to think about “those idiots” down the hall

2. The project itself will improve – programmers can often point out things that are possible that nobody else had even thought doable. Typically programmers are early-adopters and keen to produce things that impress. Mediocre software is always designed by marketing, great software by people who really understand what software can do when given a chance.

3. The project constraints will be better understood right from the word ‘go’ and proposals made by IAs or Designers who don’t understand the ‘code impact’ can be quickly managed and dealt with rather than getting in front of a client who signs off and expects to see it in the final shipped product.

Some programmers will protest, because they don’t want to go to meetings. This is because most meetings that pretend to be about original thinking aren’t anything of the sort. If you create an environment where genuine thoughtfulness, creativity and interesting ideas are produced and managed, programmers would have a different opinion of those meetings. The sad truth is, most people in the software industry who aren’t programmers are there because they aren’t very interested in this kind of thing. That’s one of the reasons why so many programmers are going it alone right now.

One argument from the other side is that programmers aren’t very good at creative thinking. Those people might be right, but that’s because they think about the process inaccurately.

The truth is, where most programmers learn their trade at first is in building algorithms. There isn’t much scope in the minds of most programmers for creativity at the algorithm level. They have been taught to look for patterns they’ve seen before – the reason why every undergraduate has to undergo at least one course in Data Structures & Algorithms during their college years – and code reuse is preferred over “reinventing the wheel”.

Knowing certain sort algorithms are better in some ways than others is certainly beneficial, as is code reuse. However, the idea that programming is something you can learn once and then just repeat over and over again is absurd.

This has been scaled up from the algorithm level into the project level over the years. It was the “building block” approach to programming: if you look at an algorithm and implement it like so, and it becomes efficient to remove creative thinking here, then it must scale up. The result is that programmers have been trained to think of projects at times in the same way they’ve been taught to think of algorithms: look for patterns, reuse where possible, don’t break the mold.

It is only in recent years that academics have even started considering teaching their students any other method than waterfall for managing a software project, and when a programmer is confronted with their team at the first job working in an agile fashion (if you can find me a team still dealing 100% with waterfall, I’ll show you an expensive team), they are going to find themselves ill-equipped for the average work day. They are going to have to find a way to keep their head above water, and what tends to happen is they hack it. They don’t have the tools – because they’ve been told they’ll never need them – to think creatively.

What’s more this problem is compounded by the fact that it’s no longer enough in an agile World for a programmer to just know how to write algorithms. All of a sudden, the information architect has become redundant as the software engineer takes the role of seeing how the whole system plugs together. There is even an argument that the information architect was always redundant.

Addressing these issues aren’t simple. If creativity is so important, why don’t more programmers engage with it?

Programmers are experts on thinking through complex problems. In theory, all that needs to be done is to teach creative and original thinking as thinking through a complex problem. Even the worst “creativity consultant” can tell you however that the enemy of creativity is habit, so they would argue this is likely to fail. The creative process looks so bizarre to an analytical mind specifically because it can’t be easily explained. This, however, is a misunderstanding.

The real issue with creative thinking is that many people are scared of failure. Programmers, especially so. This is because in real life, as in a software project, failure is expensive. Time lost is time lost – we fear doing something that we can’t take back. We look forward to our next actions in life through the lens of our past, trying to make sure we don’t make the same mistakes again. Programmers thrive on doing this – it’s what makes the analytical mind tick. This leads to unassertive caution and bashful timidity. We are too scared to put our hand up and say “let’s do something reckless” in a project meeting for exactly the same reason we won’t say it to a stranger in a bar we’re attracted to: if it fails, we feel it, and we have to deal with it.

Creative thinking is therefore not thinking about breaking the habit of doing something repetitive, but breaking the habit of doing nothing at all. It’s about doing whatever we can, over and over again, just to get one little gem of an idea out there. In the world of brainstorming, failure is cheap, and risk-taking gets the prize. It is this that should be encouraged within our industry because we are so desperate for it: brazen, reckless, failure that costs nothing, followed by the sparkle of genius that changes the World.

If creativity in programmers might help projects, or help develop programmers themselves, one question needs to be asked: Where exactly does creativity need to be applied in the software industry?

In short: Everywhere.

Software development is a creative industry, as all pinnacles of civilisation ultimately are, from stock markets to museums. The fact that we still describe the disciplines in terms of engineering or science confuses me: it may be that when a deep understanding of electronics was needed before you could begin to write software, this seemed sensible. But in the age of abstract languages like Ruby or Python? Dare I say even Java?

At the core of this is how we go about deciding what software to write in the first place. This is an area, which you may be surprised to hear that there is little creativity.

In fact, nearly all software is a derivative of a previously-available software. Even open-source, where you would expect creativity to thrive is often a collection of re-writes of commercial software which in itself is nearly always a derivative of a new way of making a quick buck. Firefox might be swell, but it is rooted in Netscape which in itself was just a way of making some money out of somebody else’s bright idea. Tim Berners-Lee might have had an original thought once, but don’t think that anything you’ve seen since then around web servers or web browsers has been original.

Then there is the implementation of the software. Programmers like code reuse. They love libraries. They would marry their preferred framework if it were philosophically sensible or legal. However, it is producing a generation of software that is derivative.

Right now, developers are in a “group-think” position that we follow the principles of doing things the way they always have been done. Sometimes this makes sense: windowing systems, cryptographic libraries, etc. However, whilst Rails and Django sure are swell, is it really the case that what is needed in the World is more CRUD-interfaces to relational databases? So often you see projects that could have been implemented more elegantly in less code, but rarely are. “_why the lucky stiff” is a notable exception to this trend: learning from watching him is recommended.

This brings us to the marketing of software. Right now you can sell your time, sell the code, or find a way to incorporate adverts. Sure “monetisation” is important: critical if you want to keep in your XXXL shape. That said, are there really only 3 ways to making money? Of course not, but whenever a new product is thought up, it’s always put into one of a few different silos. Try this: think of all the ways people are able to make money legally in the world from commissions to referrals to services, and then try and find a way to get your ‘great idea’ to fit each of those silos. You might end up inventing something completely original.

Creative thinking isn’t hard – there are thousands of ways of doing it. The question is: why aren’t we programmers doing more of it?