Innovation in Software

Development is a creative pursuit. Whilst many think of it as a purely technical challenge, it requires a level of lateral thinking about the World that is a cross between doing a crossword puzzle, composing a symphony and having an argument with people who don’t exist. It’s not surprising some of us are a little eccentric.

It reminds me of the writing process a lot. You sit down at a blank screen after having conducted your research and you have to just dig in and find some way of making progress. Many a developer struggles with a blank IDE screen much in the same way many a writer struggles to find influence. When I was learning how to write properly, I was told that “a professional writer can not be like the poet who spends a morning taking out a comma, and the afternoon putting it back in”. We need to work hard. Same with code. A block then is a real problem.

Slashdot this weekend asked How to Get Out of Developer’s Block?, or rather a user asked:

I have spent the past six months working on a software project, and while I can come up with ideas, I just can’t seem to sit down in front of the computer to code. I sit there and I just can’t concentrate. I don’t know whether this is akin to writer’s block, but it feels like it. Have any other Slashdotters run into this and if so how did you get out of it? It is bothering me since the project has ground to a halt and I really want to get started again. I am the sole developer on the project, if that makes a difference.

The comments that follow in the thread that range from the sensible to the bizarre. I have a bunch of tricks I use when I’m struggling, so thought I’d put them together

1. Get enough sleep – you have no idea how sleep deprivation can mess you up when you’re trying to concentrate. When I’m working on code, I take a minimum of 10 hours sleep a night. Anything less, and I’m not going to be able to think in purely abstract terms for 8 hours straight during the day.
2. Exercise – and whilst those of you who know me might laugh, it’s important. I actually do get regular exercise when I’m coding full-time. Just a long walk at the start or end of the day can be enough. Something that gets the hear rate up helps though (perhaps explaining why I always code better the day after… errr… private stuff that gets my heart rate up!).
3. Don’t drink alcohol – this was something I got when I was trying to sort out my pilot’s license. When you’re going flying, I don’t drink for 24 hours before getting into the plane. I found my workload was easier, my writing got more fluid and my code went up a gear. On big client projects I don’t drink at all on school nights. If I’m drinking in the evenings whilst on a project, it’s because the project isn’t challenging me and I’m bored.
4. Clear your environment out – I’m currently sat at a desk with perhaps 150 items of paperwork on it. In this environment, I can not focus on code. My mental processes are cluttered because my physical processes are. Tidying up might seem like a stupid way to get out of a block, but I genuinely find that a clear working environment leads to much clearer mental processes. I don’t know how or why, it fascinates me, but just get your physical environment fixed up and suddenly your mental environment starts to fire a little better than before.
5. Write a trivial test – this is the code version of “free-writing” that I sometimes use to unblock on writing an article. Basically write a small test (or spec if you’re BDD) for something almost trivial and then get it to pass. Repeat. Now you’re back in the game.
6. Work on the design – it’s amazing how bad we collectively are at really thinking through a problem. Go and work on some wireframes or develop some sketches of the underlying schemas and try and simplify them. Reduce things down, and suddenly you’ll see areas you can work on right away outside of the problem you’re blocked on. If you’re not able to delve into design or architecture because of the nature of the project, quite frankly you need another bunch of guys to work with.
7. Try and find it done already – I once spent a lot of time trying to work out how to solve a particular problem. The answer was non-trivial to implement in my mind. I kept putting it off. I was scared of how bad I could end up making my solution. In a fit of procrastination I spent an hour digging around the problem area and eventually found an open-source tool that did exactly what I needed, out of the box. Well, that solves that problem…
8. Are you scared of success? It might sound like a stupid question because success is good, right? But when we succeed at something, we conquer some barrier we have worked to overcome for a period, things change. Suddenly people might look at you differently. Perhaps you end up having to work on a less interesting project. You might want your current project to be a success for other reasons. Ask yourself whether you really want this project to succeed. And then realise there’s no getting out of it: failing, or staying where you are is just as bad an outcome and harms you, your self-confidence and your reputation.
9. Find a SCRUM meeting somewhere – one of the very best things about daily stand-up meetings in SCRUM projects is that the meeting only has three topics of conversation: outcomes from stuff you agreed to do in the last meeting; what you plan to do today to further the project, if anything; and obstacles in your way. Not everybody has a team (and sole development is the hardest form there is, trust me), so find a SCRUM somewhere else. Use Twitter, your blog, a group of friends down the pub, anything. Just talk about what’s stopping you and see if anybody can help you in any way, or offer suggestions. Obviously asking a friend about a tricky problem relating to class inheritance isn’t going to yield results if they don’t know what you’re on about, but ask around more liberally than you have done to date.
10. Work on something else – we all have other projects on the go. If the above isn’t working, just go and get on with something else. Your subconscious is dealing with the problem and will come up with a solution. Just make sure you hit your deliverables schedule if you have one!

Now comments are back up, I look forward to hearing of any other tips people might have.

Tonight, the BBC finally got around to airing the RTS Huw Weldon Memorial Lecture recorded in September. The speaker this year was Anthony Lilley talking on issues relating to social media and ironically given the subject of his talk, no online archive of the talk appears to be available. For all the embracing of “The Me in Media” – as the talk was titled – and the power of the network, we on the network are not allowed to see it.

No matter. The point, the thesis, the element, is quite communicable: we the audience are now in charge. What’s more, Lilley makes a compelling argument that this isn’t a sudden and new development, but something that happened the moment the audience started to appear on screen from quiz shows to our scribbles being sent to Tony Hart.

The power of networked media is considerable, and it’s pretty odd to see a bunch of network TV executives try and grasp the content that they, their ideas, their employers and their money don’t matter much any more. Within 10 years, Lilley hinted that networks like UKTV which mostly show repeats will essentially be pointless with the advent of VoD and PVR systems being widespread. Within 20 years the people being asked to pay the license fee will have no real memory of broadcast media. Within 30 years… who knows?

I was minded of a certain cartoon and then thought about how the BBC Innovation Labs is on again this year. I wonder how daring the BBC and others want us to be. Personally, I think they’re just outright scared.

In a recent article on creativity and software, I made an argument that software developers need to become more involved in the creative process. One of the problems I discussed was that most programmers and developers have never been taught to think creatively, and many don’t know how to go about it. Whilst there are issues of where developers get brought into the brain-storming process, I thought it might be useful to look at some linear techniques that programmers might find useful given the way most coders think about problems.

There is a myth that creative sparks are something that “just happen” to the “right people”. Innovation is hard work, and there is nothing celestial or gifted about it. There are many methods of being able to get new ideas. Almost all of these have one common theme:

Disrupt your habitual thinking

We all have different perspectives on the World, but we rarely shift those perspectives internally. If you are a liberal, open-source loving, dope-smoking college student you are going to have a very different perspective to that held by a 60-year old CEO of a multinational. Most creative thinking exercises are about trying to shift you out of your skin and to see the World from somebody else’s perspective.

If you’re a programmer, you might be more comfortable stating this as “introducing pseudo-random thinking”. You need to shake up your habits, your processes, and get out of your own skin.

I’m going to go through some linear techniques here that might show you what I’m talking about. These techniques are great for trying to innovate when dealing with software problems (my core audience here), but can also be applied to writers trying to get rid of writer’s block or entrepreneurs trying to come up with new ideas for business plans.

As I go through these, you will see each one is about shaking up your ideas. Try all of them, try one of them, but just try.

Reverse your assumptions

Write down your challenge or problem, and then list all the assumptions you’ve made – e.g. “I must write this in Rails”, “I have one week to do this”, “I can not get any more resources”.

Challenge each of those assumptions by writing down a reversal, i.e. – “I can write this in any language that does the job well”, “I could delay the launch by a week”, “I could post it on an online contractor site”. Some of those opposite viewpoints will stand out a little. Create a list of them and ask yourself how to accomplish each one of those useful reversals, and decide if more research is needed.

This technique is perhaps the simplest, but in some cases the most effective. The trick is to try and embrace the exact opposite of your World view.

Drill down into your assumption

As above, write down all your assumptions, but this time write down all the attributes of that assumption. It could be you need to write it in Rails because that’s the only framework you know, or because that’s company policy. You might need to launch in a week because the PR department has already announced it.

For each assumption ask yourself “Why?” or “How else could I do this?”. Try and be flexible in your thinking, and for each of the assumption you make here, try and find an opposite as above, thereby cascading out all sorts of new assumptions.

This builds on reversals, and allows for more sophisticated thinking. It can feel simplistic at first, but with some assumptions you just need to keep asking “why?” like that annoying game children sometimes play.

Dinner for Six

You ever play that game where you choose six people dead or living as dinner guests, and compare with friends? Do the same again, but ask that table for ideas around your problem or challenge. They don’t have to be famous people – it could be college professors you admired, your family, anybody. Just so long as you get some contrasting voices in your head, it’ll start producing some chaos and randomness into your thought processes.

Some people like to go and do deep research into their dinner table – it could be the six most successful CEOs in your industry, or people who would be interested in your idea for example.

It’s possible with business people to go and find out how they did the things they did and work out their key motivations. Whenever I go down this route, I always end up with Peter Drucker and Seth Godin “in the room” because I admire their work and know their attitudes to certain parts of business so well from their books. This is actually re-enforcing learning: I might read a book by Godin, but this process causes me to apply it, and until I’ve done that, it’s not really learned behaviour.

Alternatively, when faced with a development problem, you might want to ask yourself “What would the 37signals guys do here?” or “What would happen if this were a problem in the Linux kernel?” and work from there. Ideally, choose projects you’re familiar with but which aren’t linked to the area you work in: you might respect 37signals web work, but what would they do with a device driver problem? Linus might know his kernel issues, but how would he develop a shopping cart?

Tug of War

Take a problem you’re trying to solve. Write down the best-case “win” and the worst-case “lose” for that problem. For example in the situation “produce a new blog”, the best-case might be “Technorati Top 100” and worst case scenario might be “Permanent writer’s block on day one, nobody reading”.

Write down all of the conditions needed to achieve each one, and you’ll notice a tug-of-war happening on the page in front of you. You’ll start to get a feel for what can happen and start seeing opportunities. Try and work out how to get from each worst-case scenario by putting down an opposite into the best-case scenario, and vice versa.

It’s amazing how this develops, simply because it’s so common for people to assume an idea is going to fail they find themselves producing a deluge of ‘worst case’ issues. By forcing a recast into ideas for producing the best scenario, new ideas just seem to pour out.

Binary Chopping

I’ve heard this called all sorts of things, and sometimes it uses more than just two words, but I like to start with two words to describe the problem I’m dealing with. For example: “Syndicated Content”

I then take each word and I split it into two more words. For “Syndicated” I might choose “subscription” and “contract”. For “Content” I might choose “articles” and “comments”. I can then drill each of those. Keep on going until you’ve got a nice big long list: “syndicated”, “subscription, “credits”, “annual”, “contract”, “services”, “fixed-term”, “content”, “articles”, “wiki”, “aggregator”, “comments”, “trackbacks”, “votes”.

Then, it’s time to look at each of those and combine them to see where you can get ideas:

• “fixed-term” + “votes”
• “credits” + “services”
• “wiki” + “comments”

When you have a list of 20+ words, the power of this can be pretty amazing, and ideas spring out. Don’t dismiss them, just work through them and see where it takes you.

Introduce Random Words

This can have variable results, but often produces something interesting.

Write down something your problem, challenge, assumption or something you’re familiar with: e.g. “Produce a successful social network site”.

Now, come up with a random word. If you’re doing it without the help of a dictionary or some other random method you’re going to be acting sub-consciously (almost like the binary chop), and so you might be surprised at how close a connection it could be. Alternatively you could just a random word generator like this one.

Then, try and force connections and see what happens. Evaluate your ideas, and if need be use one of the other techniques here to drill down or evaluate your new ideas.

“Produce a successful social network site” + “housed” could produce an idea for college students who are looking to share a house on or near campus. With “cynical” it could be a site where people gather as common enemies to somebody(!), or with “pope” you could find yourself thinking about a site focused on the Christian community and churches.

Summary

I hope that there is at least one exercise in there that’s new to you and that you can use. I’m interested in hearing about others out there I’ve not heard before.

In particular, what techniques do people use when stuck at different levels of a project, like deep down in algorithm issues? Are we just sheep dealing with patterns, or can these help us work out alternatives when designing classes and implementing methods?

So, I’m evaluating Wiki software, but figure I may as well look at what else is out there. I’m rather shocked to find that there are almost as many superfluous wiki codebases out there as there are CMS systems built in PHP and MySQL.

All that development effort to provide what is essentially the same tool, over and over again.

This is why open source struggles – duplication of effort actually wastes developer time AND user time – it leaves users struggling to decide which application is the right one. Not one of those wiki choices is so significantly different from the others that, on being given the source code, you couldn’t get it to have the features of all the others eventually.

If all we do is reinvent the wheel over and over again, we’re never going to get around to inventing the light-bulb. The whole purpose of a mature civilisation is that we don’t all need to learn how to do everything from the basic building blocks up over and over again. Open source is in need of getting some civilisation in place, I think.

This is one of the motivators behind the new Vagueware site – I want us to talk about what comes next, whereas most sites that help teams collaborate on software deal with what just happened or what is going on right now. I want to think about stuff that might be 2-3 years away, and start working out how to make it happen, what risks and rewards there might be, do we need it, are there ethical or legal considerations, is it even a good idea, etc., etc.

Or am I missing something? Is the abundance of choice and the lack of monoculture within OSS a strength that needs to be embraced and not admonished? Is treading water just the safe thing to do in a shark-filled pool?

According to CNET Eric Schmidt is saying filtering of copyright material on YouTube is about to be turned on.

I say that is an amazingly good thing for the Web in general, and UGC sites in particular.

If you go to YouTube today it is full of clips from mainstream media interspersed with attention-seeking idiots and experiments with Mentos and Diet Coke. Right now, mainstream media have nothing to fear from YouTube because it is merely TV cut up and made more convenient for people with ADD. By demanding that MSM-sourced material is removed, they are relinquishing control and demanding that the users compete with them directly.

If we get rid of the mainstream media from UGC video sites, we’ll actually see a proliferation of UGC. Nature abhors a vacuum, and nothing fills it quicker than YouTube users. I think people will quickly get bored of op-ed pieces to camera, and we’ll see a more creative spirit flourish because suddenly the audience will be looking for it, rather than just finding clips of comedy shows and watching them in lieu of doing something more creative instead.

The week of the 23rd April through the 29th is TV Turnoff Week and my own experience shows a proliferation of creative activity when my time is not being stolen by a piece of furniture. Content filtering on UGC sites will have a similar effect – when you can’t watch clips of Family Guy any more, you might try and fill that time with producing something yourself, and if that happens in just 1% of cases, we’ll see a sharp uptick in quality UGC found on the video sites.

People shouldn’t bemoan copyright material filtering on YouTube – they should applaud it, and encourage it as the dawn of a new era in users taking control of the media they consume. Viacom are willing Google to ensure Viacom can’t control their audience any more: that’s to Viacom’s detriment and to our benefit. Viva la revolution! :-)

Another useful write-up from Creating Passionate Users, this time on the subject of learning from people who hate your products.

For a long, long time now I’ve known this simple truth: I have learnt far more from criticism than I’ve ever learnt from compliments.

This isn’t some masochistic outlook on life: if you want to improve something you need to know what’s wrong with it. Sometimes that something is you, sometimes it’s your team, sometimes it’s your writing, your code, your broken down car, whatever. If something is wrong, living in denial and assuming the fault lies with the World at large means you’ll never be able to fix it.

All too often I’ve sat in on meetings where people have told me what I had built was “fine”, it was “OK”. I could see the product sucked, that it would fall apart when pushed, but it was their sign-off. If they’d been honest with me, I would have had the momentum and back-up for changing it and making it better. With the key people just sat there happy to sign off something “OK”, all I could do was let them be complacent.

When somebody criticises your code/service/product, don’t justify your position and block out what they’re saying. Listen. Take notes. Go away and have a coffee and think about what they said. Then work out what to do about it. It might be too late to do anything, or it might be an oppurtunity to do something great. Nobody will know until you’ve heard about the complaints in the first place.

There is one caveat though: remember being positive is harder than being negative. Negative is easy. When you ask people to be positive they may struggle, but when you ask them to be negative expect a cascade of venting to come your way. Be prepared for it, and don’t be surprised if after the exercise the number of negative comments outweighs the positive comments 10:1 or worse. Just accept it, deal with it, get on with fixing it.

I care a lot about innovation and creative thinking. It’s one of the things that inspired me to learn how to write software when I was just 11 years old: if you can imagine it and it can be turned into a process a computer can run, you can create it. The potential has always been amazing, but now with tools like Ruby on Rails, the idea-to-product lead time is shorter than ever before.

The problem is, most people are awful at creative thinking and forming new ideas. In the UK we seem to be particularly bad at this, maybe because we tend to be more self-concious than many other cultures about looking foolish, or being mocked. However, it’s preventing so many businesses here from flourishing that it drives me crazy. Here’s my take on it, and how I go about coming up with hundreds of new ideas a week (note: coming up with hundreds of ideas is a good thing, acting on them is something completely different).

When traditional business consultants talk about innovation they use phrases like “blue sky thinking” or they may encourage thinking “outside of the box”. It is often thought that if we are constrained by how things are done now, the best way to be creative and innovative is to throw away everything and start from thin air.

It sounds good, and it has a catchy ring to it, but it’s flawed. It’s no surprise that the only original thought these consultants had was to become a business consultant and take your money off you for repeating clichés. They are effectively selling snake-oil, and that’s hardly an original idea.

Innovation does not come from the kind of freedom that means starting with nothing. It comes from changing the boundaries of what you already have, to evolve rather than throw everything away. Throwing everything away might be the idea you come up with at the end of the process, but it’s not how you should start.

A writer’s worst nightmare is often the blank page. A blank page represents the consultant’s “blue sky” – you can put anything on a blank page. This can cause your brain to ‘freeze’ if you don’t already know what is going to happen next. Writer’s block is caused by being able to do anything, not by being unable to do anything.

You need something to work with – you need to give yourself a boundary. One exercise to remove writer’s block is to write one word – any word, just a word – in the middle of the page. Then think about another word that would go with it, contradict it, oppose it, complement it, whatever. Soon, you find your brain giving you ideas, and off you go. This is giving your brain a set of boundaries, something to work with.

Comedy writers know this better than anybody. They have to, because writing good comedy is harder than any other type of writing (trust me, I’ve tried). Take sitcoms: the ‘sit’ is obviously short for ‘situation’ and if you watch a sitcom closely you’ll often find the really funny bits come from a set of situations rather than some cheesy lines. However, there is more to it than that.

Every sitcom starts with a context. In fact, all good comedy is often about the limits of something – a coffee shop, a relationship, a bunch of weird friends, the lead’s self-doubt. Think about all your favourite shows, and you’ll realise that the first thing the writers will have done is define the situation and boundaries the comedy will be written within.

Friends: group of 20-somethings who are close friends dealing with life in modern day New York.

Yes, Minister: the battle between the politician with a desire to do something worthwhile, and the civil service who desire nothing at all to happen if possible.

The Office: the excruciating banality and confinement of working with somebody who doesn’t understand reality.

If I asked you to write a sicom from scratch, you’d find it hard unless you had already defined a set of boundaries like those above. Get the idea? Still don’t believe me? OK, try this:

Write a new joke right now. Come up with something original, a joke the World has never heard before. It can be about anything.

I bet after a few minutes you’re finding it hard. Where do you start?

OK, I’ll help you. The joke has to involve a monk and an elephant. Easier? Probably. If you’re still struggling, I’ll give you a push and suggest you must use a play on the word ‘habit’ as in an item of monk’s clothing and being an established custom. Got a joke yet? Your brain is probably finding it easier now to come up with something funny.

This is what comedy writers do – they find some boundaries, and twist around inside them. It’s not just comedy either.

So how does this relate to technology and software? Well, firstly, it explains why all commercially successful technical advancement comes ultimately from evolution of ideas rather than revolutions.

People often resist revolutions – they need to find a way of relating to something via an evolutionary step. Early-adopters are not always grabbing hold of the next revolution: it’s because they’re always adopting early, that the stuff they do now is an evolution of where they were 6 months ago. We’ll all catch them up in a couple of years when our evolutionary path catches up, but to us it might look like a revolution, but that’s just because we’re being slow.

The evolutionary path we follow in our uptake of ideas is merely us accepting the twisting of the boundaries we’re comfortable with. Some people for example, will have produced a joke about the monk and the elephant that others will have found obscure, obscene or just outright unfunny. But that’s because the way you shape your box and then push the edges is different to how other people shape their box and push the edges. This is the core of all technical advancement, selling new technologies, and ultimately, creative commercially successful innovation.

People talk about the industrial “revolution”, but really it was an evolution of ideas on how to make spinning cotton more productive, mixed with the idea that a steam engine was more efficient than a horse (and didn’t need as much care and handling), mixed again with the idea that selling goods all over the World from a factory in the North of England made more money than selling it all down the local market. Evolution, after evolution, after evolution. It took decades, a lot of pain and failure along the way, and was all driven by people who liked twisting the rules inside a set of boundaries.

It took special circumstances, and a group of people to take a series of steps that others found uncomfortable.

The other reason this relates to technology, is that if we know that innovation is evolution, not revolution, and we know that giving ourselves boundaries to push is the way to be creative, we can use that to produce something valuable. Remember at the start when I said there was no other field more open to new ideas being created than computing? Once you formulate how to be creative, it opens up the playing field completely – you can become a money machine.

Want to come up with a new super-cool Web 2.0 website? Fine, let’s see – it needs to be based on communities, it needs to feel like a desktop app, and ideally it should get better the more people use it. That’s a good start, but I bet you’re still struggling to come up with dozens of ideas. Problem is, they’re all technical boundaries, and what we want is a business boundary – we want to generate new businesses, not new technologies, so we need to focus on the right box.

That said, trying to build a business with no thought to what is possible with the technology is a dumb move, and sometimes taking an old business idea and updating it with the latest tech (cf. Shopify) can produce a great business. However, for this example let’s add another arbitrary business bounday: it needs to be based on an e-Commerce model – people should use the site to buy things.

Right now, your brain might be starting to kick into gear and you’re coming up with ideas. Chances are, if you’re still concentrating on the technical boundaries they’re just re-hashing what has been done before – adding Ajax to OScommerce clones or something – and that’s no good: we want to get a bunch of really original ideas.

First thing we could do is say we’ll do the exact opposite of what everybody else does. So let’s do a web-site that when you visit it, it’s full of people wanting to buy things, and you can sell them what they need – the 180 degree turn-around. That’s an interesting and innovative twist.

We could just as easily brainstorm how to make something bigger, smaller, easier, harder, cooler, less cool, more popular, more niche, more intelligent, less intelligent, and so on. Look at every feature of eBay or Shopify and ask yourself “How would I do that differently?”

If we built a site where instead of a list of products, it was a list of wanted ads, we could then look at it again from fresh and find an evolutionary step from there. It may be that what we actually do is aggregate specially-tagged content from people’s blogs and allow marketers to find what bloggers are after, rather than hosting the ads ourselves. We may decide to produce a version that is highly targets to a very small niche. We might want to make it something well suited to market researchers trying to produce the Big Toy This Christmas.

Then, once we have those new boundaries, we start to twist within them and we may end up turning out a whole bunch of new ideas.

When you see the World through a lens that makes every product feature a boundary to be pushed, twisted and warped, creative thinking becomes infectious. And in the new World of software development, we need all the creative thinking we can get.

These are just some of my thoughts that I’ll be expanding on in the coming weeks – I’d be really keen to hear your ideas in the comments section.