Innovation in Software

In this article, I’ll be exploring some of the issues around creative thinking and software development. This is a theme I expect to be re-visiting a lot over the next few weeks, so these are more high-level riffs. Each of the major points here needs expanding upon, but I’m interested in seeing what people out there lock into as the most interesting aspects of this.

I believe that raw creativity is not normally associated with the development side of the software industry. This is an error that hampers development, innovation and even the core process of programming. Creative thinking should be taught to all programmers and embraced as a way of working.

Typically only the people involved in the user interface, information architecture and marketing stages of the development process get to do the “original vision thing”. Whilst this goes on, programmers are expected to behave like glorified plumbers by connecting interactions together with a logic that meets the spec and passes tests. This skews the development process towards the wrong people.

Project Managers and Information Architects are going to be aghast – Designers will be screaming – but it is the “Developers” who are the best placed to understand what is and isn’t possible. When asked to be involved in the earliest stages, or in stages they’re not normally invited into (yes, even marketing meetings) and encouraged to contribute to the creative process, several benefits are likely to occur:

1. The programmer becomes more involved in the project and is less likely to think about “those idiots” down the hall

2. The project itself will improve – programmers can often point out things that are possible that nobody else had even thought doable. Typically programmers are early-adopters and keen to produce things that impress. Mediocre software is always designed by marketing, great software by people who really understand what software can do when given a chance.

3. The project constraints will be better understood right from the word ‘go’ and proposals made by IAs or Designers who don’t understand the ‘code impact’ can be quickly managed and dealt with rather than getting in front of a client who signs off and expects to see it in the final shipped product.

Some programmers will protest, because they don’t want to go to meetings. This is because most meetings that pretend to be about original thinking aren’t anything of the sort. If you create an environment where genuine thoughtfulness, creativity and interesting ideas are produced and managed, programmers would have a different opinion of those meetings. The sad truth is, most people in the software industry who aren’t programmers are there because they aren’t very interested in this kind of thing. That’s one of the reasons why so many programmers are going it alone right now.

One argument from the other side is that programmers aren’t very good at creative thinking. Those people might be right, but that’s because they think about the process inaccurately.

The truth is, where most programmers learn their trade at first is in building algorithms. There isn’t much scope in the minds of most programmers for creativity at the algorithm level. They have been taught to look for patterns they’ve seen before – the reason why every undergraduate has to undergo at least one course in Data Structures & Algorithms during their college years – and code reuse is preferred over “reinventing the wheel”.

Knowing certain sort algorithms are better in some ways than others is certainly beneficial, as is code reuse. However, the idea that programming is something you can learn once and then just repeat over and over again is absurd.

This has been scaled up from the algorithm level into the project level over the years. It was the “building block” approach to programming: if you look at an algorithm and implement it like so, and it becomes efficient to remove creative thinking here, then it must scale up. The result is that programmers have been trained to think of projects at times in the same way they’ve been taught to think of algorithms: look for patterns, reuse where possible, don’t break the mold.

It is only in recent years that academics have even started considering teaching their students any other method than waterfall for managing a software project, and when a programmer is confronted with their team at the first job working in an agile fashion (if you can find me a team still dealing 100% with waterfall, I’ll show you an expensive team), they are going to find themselves ill-equipped for the average work day. They are going to have to find a way to keep their head above water, and what tends to happen is they hack it. They don’t have the tools – because they’ve been told they’ll never need them – to think creatively.

What’s more this problem is compounded by the fact that it’s no longer enough in an agile World for a programmer to just know how to write algorithms. All of a sudden, the information architect has become redundant as the software engineer takes the role of seeing how the whole system plugs together. There is even an argument that the information architect was always redundant.

Addressing these issues aren’t simple. If creativity is so important, why don’t more programmers engage with it?

Programmers are experts on thinking through complex problems. In theory, all that needs to be done is to teach creative and original thinking as thinking through a complex problem. Even the worst “creativity consultant” can tell you however that the enemy of creativity is habit, so they would argue this is likely to fail. The creative process looks so bizarre to an analytical mind specifically because it can’t be easily explained. This, however, is a misunderstanding.

The real issue with creative thinking is that many people are scared of failure. Programmers, especially so. This is because in real life, as in a software project, failure is expensive. Time lost is time lost – we fear doing something that we can’t take back. We look forward to our next actions in life through the lens of our past, trying to make sure we don’t make the same mistakes again. Programmers thrive on doing this – it’s what makes the analytical mind tick. This leads to unassertive caution and bashful timidity. We are too scared to put our hand up and say “let’s do something reckless” in a project meeting for exactly the same reason we won’t say it to a stranger in a bar we’re attracted to: if it fails, we feel it, and we have to deal with it.

Creative thinking is therefore not thinking about breaking the habit of doing something repetitive, but breaking the habit of doing nothing at all. It’s about doing whatever we can, over and over again, just to get one little gem of an idea out there. In the world of brainstorming, failure is cheap, and risk-taking gets the prize. It is this that should be encouraged within our industry because we are so desperate for it: brazen, reckless, failure that costs nothing, followed by the sparkle of genius that changes the World.

If creativity in programmers might help projects, or help develop programmers themselves, one question needs to be asked: Where exactly does creativity need to be applied in the software industry?

In short: Everywhere.

Software development is a creative industry, as all pinnacles of civilisation ultimately are, from stock markets to museums. The fact that we still describe the disciplines in terms of engineering or science confuses me: it may be that when a deep understanding of electronics was needed before you could begin to write software, this seemed sensible. But in the age of abstract languages like Ruby or Python? Dare I say even Java?

At the core of this is how we go about deciding what software to write in the first place. This is an area, which you may be surprised to hear that there is little creativity.

In fact, nearly all software is a derivative of a previously-available software. Even open-source, where you would expect creativity to thrive is often a collection of re-writes of commercial software which in itself is nearly always a derivative of a new way of making a quick buck. Firefox might be swell, but it is rooted in Netscape which in itself was just a way of making some money out of somebody else’s bright idea. Tim Berners-Lee might have had an original thought once, but don’t think that anything you’ve seen since then around web servers or web browsers has been original.

Then there is the implementation of the software. Programmers like code reuse. They love libraries. They would marry their preferred framework if it were philosophically sensible or legal. However, it is producing a generation of software that is derivative.

Right now, developers are in a “group-think” position that we follow the principles of doing things the way they always have been done. Sometimes this makes sense: windowing systems, cryptographic libraries, etc. However, whilst Rails and Django sure are swell, is it really the case that what is needed in the World is more CRUD-interfaces to relational databases? So often you see projects that could have been implemented more elegantly in less code, but rarely are. “_why the lucky stiff” is a notable exception to this trend: learning from watching him is recommended.

This brings us to the marketing of software. Right now you can sell your time, sell the code, or find a way to incorporate adverts. Sure “monetisation” is important: critical if you want to keep in your XXXL shape. That said, are there really only 3 ways to making money? Of course not, but whenever a new product is thought up, it’s always put into one of a few different silos. Try this: think of all the ways people are able to make money legally in the world from commissions to referrals to services, and then try and find a way to get your ‘great idea’ to fit each of those silos. You might end up inventing something completely original.

Creative thinking isn’t hard – there are thousands of ways of doing it. The question is: why aren’t we programmers doing more of it?