Innovation in Software

Service Level Agreements are a must-have for Enterprise clients and it has surprised me that so few web companies have used them as a route to making money: if you don’t need an SLA, take the app for free. If you do want an SLA (because say your entire email operation is running on our web service, say), then you need to pony up some cash. It’s worked in open source, so I think it’s a no-brainer for an industry that is service-orientated at its core.

Good news then that Amazon S3 has today announced an SLA which means if they drop below 99.9% uptime per month you can have some cash back. You get even more money back if they drop below 99% uptime. They also agree to give you 60 days notice if they want to get rid of you for any reason – but don’t have to give that reason.

It’s a step in the right direction, but they could make even more money by offering even better SLAs if customers are prepared to spend more money to get them. That money would be capital Amazon would be free to invest in infrastructure which not only enhances S3, but Amazon’s core systems and business.

In the current Web-wacky-world, money is a curious talking point. Everybody, at some point, needs to think about ‘monestisation’. It’s not even a real word, but the general plan for most companies seems to go something like:

1. Get funding for an idea
2. Build it
3. ???
4. Profit!

The fact that this is a business plan stolen from Slashdot shows you just how weak it is. Slashdot remains (to its credit?), a bastion of unprofessionalism.

Step 3 – making money – is something I believe should actually be in there at step 1. You should have a plan right before you start. Most developers are so excited by the idea they want to build that they take the easy route out: “we’ll put some ads on there”.

Big mistake.

Firstly, advertising is what I rather emotively call “brain rape”. It has no other intention than to disrupt users so their emotions can be manipulated into working with your advertiser. The idea Google had for Gmail is to my mind insane: I want to read my e-mail when I check my e-mail, not be distracted into doing something else.

Secondly, advertising is about to get a great deal more innovative, and the truth is: advertisers don’t need you any more.

Steve Bowbrick has written a great article this morning inspired by a trip to a bunch of creative types experimenting with online media campaigns.

The point is this: it is no longer enough to stick a 30 second ad spot in between your favourite shows, or to place a banner advert on a site they don’t control. The next generation of advertising is to build online media presence that people hunt down. A series of videos on YouTube, a wiki, a group on Facebook discussing the product.

Advertisers have talked for generations about establishing a relationship with the consumer. This has been to date complete poppy-cock. It is only in recent years with the advance of loyalty cards that they’ve really been able to get under our skins and look at what brand of sanitary towel or condom we prefer (in order to establish a correlation, if any, with our choice of toothpaste and breakfast cereal), but the plans for the future are ones for direct audience participation. It will not be enough for them to tell us what we should think of their product: we will, as a form of passing the time, be helping tell them what their product means to us.

Never before has this been possible, because never before have we had the means to be able to allow media to be consumed, shared and ‘user-generated’ the way we have now. The innovators were the artists, and now it’s getting mainstream it’s time for the ad men to get involved.

I shall leave it for another day to point out just how debased we become as a civilisation when we seek out sales people as entertainment, but for now I just want to point out one thing: unless you’re building something advertisers can use as a platform in itself like video or social media, they’re planning to cut you and your app out of the loop within 3-5 years. They don’t want to give you money unless they have to, and right now they’re starting to think there may be a way out.

Out of all the areas that I research for this blog and the site (at the time of writing, down for a week awaiting a new build of some code), the one that intrigues me most is the economics of software. I have watched – painfully at times – as we have turned from an industry dominated by license fees and support contracts into one trying to find a way of announcing that washing machines are half price this weekend next to our elegant user interfaces.

I for one welcome the anticipated shift in advertising away from banner ads and disruptive behaviour. I think it will completely destroy entertainment, but that’s another problem to be solved when people get there. What intrigues me right now, is how are we going to get software development funded.

• Open source doesn’t have a real business model outside of the FUD enterprises hear, that goes away when you dual-license your code
• Users don’t want to pay subscription fees
• We are no longer in an age where license fees have a viable future, particularly with web software
• Advertisers are getting ready to avoid as many people as possible

All of this points to something that could burst. Not a bubble, not something where there is an inflation that is out of synch with reality. But an adjustment. A shift of priorities. Something which we need to start thinking about now, because within a few years we won’t have any way to make money from this industry with 95% of the apps out there and that means we’re going to need to rethink everything.

In this article, I’ll be exploring some of the issues around creative thinking and software development. This is a theme I expect to be re-visiting a lot over the next few weeks, so these are more high-level riffs. Each of the major points here needs expanding upon, but I’m interested in seeing what people out there lock into as the most interesting aspects of this.

I believe that raw creativity is not normally associated with the development side of the software industry. This is an error that hampers development, innovation and even the core process of programming. Creative thinking should be taught to all programmers and embraced as a way of working.

Typically only the people involved in the user interface, information architecture and marketing stages of the development process get to do the “original vision thing”. Whilst this goes on, programmers are expected to behave like glorified plumbers by connecting interactions together with a logic that meets the spec and passes tests. This skews the development process towards the wrong people.

Project Managers and Information Architects are going to be aghast – Designers will be screaming – but it is the “Developers” who are the best placed to understand what is and isn’t possible. When asked to be involved in the earliest stages, or in stages they’re not normally invited into (yes, even marketing meetings) and encouraged to contribute to the creative process, several benefits are likely to occur:

1. The programmer becomes more involved in the project and is less likely to think about “those idiots” down the hall

2. The project itself will improve – programmers can often point out things that are possible that nobody else had even thought doable. Typically programmers are early-adopters and keen to produce things that impress. Mediocre software is always designed by marketing, great software by people who really understand what software can do when given a chance.

3. The project constraints will be better understood right from the word ‘go’ and proposals made by IAs or Designers who don’t understand the ‘code impact’ can be quickly managed and dealt with rather than getting in front of a client who signs off and expects to see it in the final shipped product.

Some programmers will protest, because they don’t want to go to meetings. This is because most meetings that pretend to be about original thinking aren’t anything of the sort. If you create an environment where genuine thoughtfulness, creativity and interesting ideas are produced and managed, programmers would have a different opinion of those meetings. The sad truth is, most people in the software industry who aren’t programmers are there because they aren’t very interested in this kind of thing. That’s one of the reasons why so many programmers are going it alone right now.

One argument from the other side is that programmers aren’t very good at creative thinking. Those people might be right, but that’s because they think about the process inaccurately.

The truth is, where most programmers learn their trade at first is in building algorithms. There isn’t much scope in the minds of most programmers for creativity at the algorithm level. They have been taught to look for patterns they’ve seen before – the reason why every undergraduate has to undergo at least one course in Data Structures & Algorithms during their college years – and code reuse is preferred over “reinventing the wheel”.

Knowing certain sort algorithms are better in some ways than others is certainly beneficial, as is code reuse. However, the idea that programming is something you can learn once and then just repeat over and over again is absurd.

This has been scaled up from the algorithm level into the project level over the years. It was the “building block” approach to programming: if you look at an algorithm and implement it like so, and it becomes efficient to remove creative thinking here, then it must scale up. The result is that programmers have been trained to think of projects at times in the same way they’ve been taught to think of algorithms: look for patterns, reuse where possible, don’t break the mold.

It is only in recent years that academics have even started considering teaching their students any other method than waterfall for managing a software project, and when a programmer is confronted with their team at the first job working in an agile fashion (if you can find me a team still dealing 100% with waterfall, I’ll show you an expensive team), they are going to find themselves ill-equipped for the average work day. They are going to have to find a way to keep their head above water, and what tends to happen is they hack it. They don’t have the tools – because they’ve been told they’ll never need them – to think creatively.

What’s more this problem is compounded by the fact that it’s no longer enough in an agile World for a programmer to just know how to write algorithms. All of a sudden, the information architect has become redundant as the software engineer takes the role of seeing how the whole system plugs together. There is even an argument that the information architect was always redundant.

Addressing these issues aren’t simple. If creativity is so important, why don’t more programmers engage with it?

Programmers are experts on thinking through complex problems. In theory, all that needs to be done is to teach creative and original thinking as thinking through a complex problem. Even the worst “creativity consultant” can tell you however that the enemy of creativity is habit, so they would argue this is likely to fail. The creative process looks so bizarre to an analytical mind specifically because it can’t be easily explained. This, however, is a misunderstanding.

The real issue with creative thinking is that many people are scared of failure. Programmers, especially so. This is because in real life, as in a software project, failure is expensive. Time lost is time lost – we fear doing something that we can’t take back. We look forward to our next actions in life through the lens of our past, trying to make sure we don’t make the same mistakes again. Programmers thrive on doing this – it’s what makes the analytical mind tick. This leads to unassertive caution and bashful timidity. We are too scared to put our hand up and say “let’s do something reckless” in a project meeting for exactly the same reason we won’t say it to a stranger in a bar we’re attracted to: if it fails, we feel it, and we have to deal with it.

Creative thinking is therefore not thinking about breaking the habit of doing something repetitive, but breaking the habit of doing nothing at all. It’s about doing whatever we can, over and over again, just to get one little gem of an idea out there. In the world of brainstorming, failure is cheap, and risk-taking gets the prize. It is this that should be encouraged within our industry because we are so desperate for it: brazen, reckless, failure that costs nothing, followed by the sparkle of genius that changes the World.

If creativity in programmers might help projects, or help develop programmers themselves, one question needs to be asked: Where exactly does creativity need to be applied in the software industry?

In short: Everywhere.

Software development is a creative industry, as all pinnacles of civilisation ultimately are, from stock markets to museums. The fact that we still describe the disciplines in terms of engineering or science confuses me: it may be that when a deep understanding of electronics was needed before you could begin to write software, this seemed sensible. But in the age of abstract languages like Ruby or Python? Dare I say even Java?

At the core of this is how we go about deciding what software to write in the first place. This is an area, which you may be surprised to hear that there is little creativity.

In fact, nearly all software is a derivative of a previously-available software. Even open-source, where you would expect creativity to thrive is often a collection of re-writes of commercial software which in itself is nearly always a derivative of a new way of making a quick buck. Firefox might be swell, but it is rooted in Netscape which in itself was just a way of making some money out of somebody else’s bright idea. Tim Berners-Lee might have had an original thought once, but don’t think that anything you’ve seen since then around web servers or web browsers has been original.

Then there is the implementation of the software. Programmers like code reuse. They love libraries. They would marry their preferred framework if it were philosophically sensible or legal. However, it is producing a generation of software that is derivative.

Right now, developers are in a “group-think” position that we follow the principles of doing things the way they always have been done. Sometimes this makes sense: windowing systems, cryptographic libraries, etc. However, whilst Rails and Django sure are swell, is it really the case that what is needed in the World is more CRUD-interfaces to relational databases? So often you see projects that could have been implemented more elegantly in less code, but rarely are. “_why the lucky stiff” is a notable exception to this trend: learning from watching him is recommended.

This brings us to the marketing of software. Right now you can sell your time, sell the code, or find a way to incorporate adverts. Sure “monetisation” is important: critical if you want to keep in your XXXL shape. That said, are there really only 3 ways to making money? Of course not, but whenever a new product is thought up, it’s always put into one of a few different silos. Try this: think of all the ways people are able to make money legally in the world from commissions to referrals to services, and then try and find a way to get your ‘great idea’ to fit each of those silos. You might end up inventing something completely original.

Creative thinking isn’t hard – there are thousands of ways of doing it. The question is: why aren’t we programmers doing more of it?

My eye was caught by a post on the excellent indexed blog which if nothing else, inspires a new slant on data visualisation. It also makes me smile.

The cartoon in question is this:

Apologies to anybody using a screen reader, but it comes down to this: certain objectives in life can be classified as being desirable and accessible. The less desirable, and the less accessible, the more likely it is you don’t want it. The less accessible, the more desirable it needs to be to shoot for it.

One of my bugbears with open source right now is when it comes to this simple understanding, the community is mostly out of balance. We’re sat unevenly on lumpy soil. We produce things that are desirable to niche audiences (ourselves, usually) and nobody else. We produce tools that are hard to understand because it’s easy for us to do that, and hard to make it so that people don’t need to think about what is going on.

In the last couple of weeks I’ve been rolling around ideas for Vagueware in my head, and if I were to plot where the code I produce from here on should be on that graph, it would be somewhere between “hard to get” and “daydreams”. I want to make it easy for anybody to drop an idea in, and within a few weeks to see their idea (if it’s a good one) take form. But I’m not interested in producing a clone of existing software, even though I know I’m going to have to at some level.

Going through my box of ideas, I note some of them are evolutionary steps on what already exists, some of them are a little ‘out there’. I need to plan carefully which ones I’m going to put up first: too ‘traditional’ and ‘samey’ and I send out the wrong message and alienate the interesting ideas. Too pie-in-the-sky, I get people fluffing it off as a fantasy World.

Thinking this through, I see that there are several categories of code I’m going to want to produce:

• Core Components – it is likely that in any idea I bring forward, there are going to be certain tools needed over and over again, most of which don’t have any high-quality Rails open source components in the wild: a Drupal-like CMS, social networking tools, e-commerce plugins, etc.
• Evolutionary – Then, it’s time to push things out a little. What happens when I combine an RSS aggregator with a Bayesian classifier so you see posts clustered around topics as opposed by feed? What happens if I take an auction-site model, but update the entire system so different auction models can be tried, and sales can be easily embedded into blogs and other pages? What happens when you take a real-estate platform, but plug it in to a social networking tool so you can see who your new neighbours are going to be? What happens when you take a web-based spreadsheet and make it aware of APIs out there, a la Seth Godin’s “websheets” idea?
• Revolutionary – The ‘out there’ ideas that try to shake up things in truly bizarre ways. A news sites anybody can contribute to (not just post links to blog articles on), experiments in micropayments, etc.

That means there is room for movement in terms of what is accepted – if we say up front “these ideas have been done before, we’re just re-executing them in open source” or “these are mad as hell”, people won’t walk away with the wrong idea. Give things a label, people believe it.

I’ve also had a bit of a re-think about monetisation. I need to clear this with my accountant, but there’s potential here for something a little bit more interesting.

Let’s suppose an idea gets posted, and it’s sane at some level and doable. It may be I’m too busy to do it, but various people like it so much that they’d pay to see it completed sooner rather than later. They can become ‘sponsors’ of the idea, and anybody could bid to take the sponsorship pool in return for providing working code. I might grab it, somebody else might grab it whatever. If two or more parties are involved, they have to agree between them how it’s going to be shared, with the sponsor’s agreement. It could turn into a way of being able to fund open source development of that kind of code where scratching your own itch isn’t enough.

Just some thoughts. For what it’s worth, it’s likely that the new version of the site won’t be ready for a couple of weeks yet. I’m still trying to decide what to do with this blog when it happens, and need to firm up some of the above ideas before real progress can be made. Watch this space, etc.

Oh, and I’m also spending way too much time thinking about changing the logo. Any suggestions appreciated, although it shouldn’t become too much of a bikeshed.