Innovation in Software

About two years ago, job adverts started appearing in the industry that had interesting phrases in them: Ruby on Rails Rockstar; Ajax Master; Software Architect Ninja.

Eyebrows were raised. Was this just another example of the industry’s machismo playing out? Were we witnessing the next generation of kids coming up through the industry and rejecting the corporate ties of traditional job titles? Did HR really expect their latest technology recruit to play to stadiums, or attack enemies silently in the middle of the night?

What these companies were actually trying to say was “we need somebody self-motivated, dedicated to their craft, and who can take a lead in front of our audience (composed of customers, investors and employees)”.

Alas, the whole meme was misunderstood, those recruited into such roles were ridiculed, and adverts stopped trying to feel like they were recruiting heroes from Manga comics or NME.

However, we still need that mindset all over this industry. I’d go as far as to say it needs to become the default. We just need to cut out the hyperness and stop drenching the industry in irrelevant masculinity.

Let me explain what I mean by “that mindset” in more detail. Most of you will be familiar, but I think it’s worth recapping:

• Self-motived

  Anybody can learn how to write code, or fix a server. The difference between the majority of people in the industry today and the ones we want to see more of is how they learn, not what they learn. Some years ago I was working in a public sector gig where I had a couple of different people working under me. The difference between how they thought about their work was considerable. If I asked the first person to do something, the response might be “I’ve never done that before, so I’m going to need a few minutes to work it out, Google around, work out what I’m doing, spot the drawbacks, OK?”. Brilliant. The other guy would simply stop at “I’ve not been trained to do that. You need to send me on a training course”.

  There is no room in this industry for people with the second mindset any more. Rockstars don’t normally go to music school. Yes, training courses can be valuable, but unless you’re the kind of person who teaches themselves constantly, you’re not suited to this industry.

• Dedicated to their craft

  A craft you say? Better believe it. Most of us are taking a blank piece of framework and sculpting it into a finished product. “Rockstars” care about aesthetics, tools, even to some extent materials (or at least in the Ruby world we do). Software people should have ideals of craftsmanship, even when they’re pretending to be engineers.

  However, not everybody is dedicated to their craft. Many will not go to a conference unless somebody else is paying. Most do not re-factor their code to make it better in every way possible. 98% will never attempt to help other developers improve their code without an assumption of compensation. The really great people in this sector have the mannerisms and dedication of the minority, mannerisms we need to see become the norm.

  I’m not suggesting hair-shirts or years of religious solitude here, just an interest in the industry that seeps a little bit from time to time out of normal work hours and uses up some of their spare time and cash because the interest is genuine.

• Taking a lead

  I’ve met and worked with many “back-room mindset” developers. I once worked in a “pit” of developers and engineers at a household-name firm where team members were told to go home, have a bath, and come back tomorrow with clean clothes without any holes in them. To the uninitiated, this sounds disgusting (and to be fair, the pit did have a ‘funkiness’ about it), but in years gone by this was what small development teams were like. Guys who were in the back room, never asked for an opinion, never brought into management meetings, and who liked it that way. An invitation to a meeting was often met with a sneer by many of the people I worked with back in the late 1990s.

  Those attitudes are now (thankfully) disappearing. If you are the kind of developer who never wants to talk to the customer or management, you are useless in this industry. You’re dead wood. Sorry, but you need to realise that 80% of development is about communication, and good developers are the ones who lead that communication. Taking the lead is what we do. We are sculpting the future of organisations and job roles, we don’t get to sit in the cupboard and meekly accept specification documents handed down to us by non-developers any more.

There are more traits out there that we need to collectively foster, each of us at a personal level. This is just a starting point, this is what I thought of when people mentioned “Rockstar” developers. We just need a new name for it, something that doesn’t have all the connotations of “rockstar”, “ninja” or “master”. “Crafter” could work, but is too folksy for most people.

If the industry doesn’t start seeing these traits more in the kids coming up out of college, we’re going to struggle. The massive teams of old are dead. Small, focused development teams filled with people who think about their work the right way are the future. How do we foster these people?

Many years ago I ran FreeBSD on pretty much every system I touched. I loved it. The lack of political polarisation that touches the GNU/Linux community combined with the power of Unix under the hood sucked me in.

I even, almost infamously, had a rant about another operating system I now use daily. I was a zealot. I am ashamed of my attitude in that email – my only excuse is that I was still quite young, a rather naive and opinionated 26 year old.

Today, I use exclusively OS X, and occasionally boot up a virtual Windows instance to test code on MS code. I have a terminal open nearly all the time and often dive down to the Unix goodness, but more often than not the bulk of my work is in the browser, text editor, IDE or Mail app.

Why then, I was asked the other day, do I not just ditch Apple and get a Linux (or even BSD), machine to work with? The bulk of my work (except for iPhone application development) would continue as before.

My answer is very simple: carpenters don’t build lathes.

To explain, let’s first go back to that rant I had about OS X. About a month later, I ordered my first Apple laptop – an iBook G4 – and was very happy with it. What happened in that month?

I realised when shopping around, I needed to check out the chipsets used for WiFi in each laptop because FreeBSD wireless support was pretty hit-and-miss back then. I also needed to understand the graphics chip-set because I’d need to compile X if I wanted to run a graphical desktop environment. I then realised I’d need to assess pretty much every feature in terms of hardware compatibility to make a purchasing decision. And then, on receipt of my new laptop I would have to spend a few hours doing all the compiling and fixing, and I would then need to do this work every few months as part of an upgrade cycle.

Yes, I’d get the power and flexibility of my own tailored operating system environment, but isn’t that a lot of work?

I recalled a few months previously a friend was playing with his new smart phone. It was running Symbian and he seemed to be doing lots of prodding and poking with it. Enquiring what he was doing, he said he was “doing some maintenance” to keep it in working order. Hang on, was he effectively sysadmin’ing his phone? “Yes, I suppose”, came the reply. When you have to do systems administration work on your phone, your phone is no longer a tool to assist with your work, it is an object of work in itself.

It was this insight when thinking about my operating system choices that directed me to OS X: I wanted the power of Unix, sure, but I wanted it to just work. I wanted to be able to get on with my work, the laptop and operating system as tools rather than objects of work themselves.

This small insight has made my life a great deal easier, and still dictates not using any other operating system. Some of my peers see it as weakness (even more point to my original rant above), but I see it as spending my time doing the things I love.

And it’s also an insight I think that as developers we forget: are we developing tools that assist with objectives, or tools that are objects of work themselves. Are we building interfaces and suggesting business logic that means our customers spend time managing the behaviour to fit around them? If so, why?

In the last few weeks I’ve seen some really interesting “reductions” in functionality that aim to make tools more directly useful, rather than requiring some administration.

Take for example, the role of authentication. You need to “contain” all of your customer’s “stuff” in a way that is linked to their “account”. So we have user registration, user profiles, account activations, password resets, etc. Seems like a lot of stuff. Are we sure we need it?

Take for example, Posterous. Their home page explains it all:

All you need to do is email stuff to post@posterous.com and you’re up and running. No signup, no captchas, no password strength indicators unless you want to add them. You’re a person capturing stuff – why do you need to admin yet another web application?

BootStrap UK (which is worthy of an article in its own right as a concept), needs accounts but has a simpler registration system: just follow @bootstrapuk on Twitter and they’ll follow you and direct message you a password. Done.

There are other examples out there as well, sometimes using software to help break out “lathe obsession” elsewhere in society. One iPhone app I’ve seen reviewed gets the banking system to work around the customer, rather than demand the customer head into a branch to suit the banking system. Brilliant.

We often spend too much time trying to work out how a customers will need to behave around our application, rather than how we need to get out of the way and let our customers use what we produce in order to do their thing. In every customer we need to start seeing the carpenter more, and assume they – perhaps unlike many of us developers – don’t like tinkering with lathes for its own sake.

Over 2008 some remarkable technologies emerged quietly that seem to be gaining traction within the industry. Whilst around for years, I am confident their time is about to come proper.

If you were to ask most people on the street who the most innovative technology firm was of 2008, you are likely to see a familiar list: Google, Apple, maybe Microsoft or perhaps some of the smaller outfits that have crossed into the mainstream like Facebook or Twitter.

Few people will mention Amazon.

In fact, if you point out that Amazon is right now perhaps one of the most innovative technology firms on the planet, people will raise an eyebrow and ask “What? The online shop? Where I get my my books and DVDs from?”.

This typical reaction is perhaps caused by only industry participants having seen so far just how Amazon are disrupting the economics of doing business online. Now anybody can have access to infinite storage arrays, huge compute clouds, masses of humans to complete complex tasks and distribute content across the globe as fast as possible, all without a penny of capital expenditure: you pay only for what you use. You can even send your physical products to be stored in Amazon’s warehouse and they will for a fee handle order fulfilment for you, again only paying for what you use.

Capital expenditure is a start-up company’s biggest problem. Remove it and suddenly anything becomes possible in the start-up World. This is big. Very big.

The real beauty is perhaps the fact Amazon made this move originally to use their own infrastructure more efficiently. If Amazon has lots of spare compute capacity ready to serve pages during their busy season (the run up to Christmas), why not lease it out the rest of the year? And yet this strategy has started to offset their own infrastructure costs so much I wouldn’t be surprised if within a few years their operational costs tend towards zero. The most powerful e-commerce platform on the planet, and everybody else is paying for it for them.

This has caused people to sit up and notice. “The Cloud” is now the hottest buzz term in the industry and all players are trying to figure out a strategy to compete.

One of the issues is that Amazon’s infrastructure is not as simple to use as it could be. Plenty of firms see a gap to try and make things simpler: one of the biggest complaints about S3 is that you need to use custom APIs instead of open standards like SFTP or WebDAV; EC2 needs a more complex understanding of systems administration and data storage than traditional models; for many applications it’s overkill or too generic, and so on.

If you break the components needed for a web application into its constituent parts from platform and compute capability through to storage services, you realise that at each level there are numerous companies trying to find a place in this market from Google and Microsoft through to unknown start ups, some of whom are attempting to make access to other cloud services easier to use and therefore are some sort of “meta-cloud” service.

This is thought to be a paradigm shift in how developers think about developing and delivering applications, but what we have seen to date is likely just the tip of the iceberg. For a number of years now traditional engineering firms (notably Rolls Royce in particular), have realised substantial revenue growth comes not from product sales – competitors can easily counter advances in product development – but in services. It seems the computer industry is starting to cotton on, and companies like AMD are thinking about how to ride the Cloud hype into the services sector. Even Microsoft are considering versions of Windows that users pay for by the hour.

It is perhaps because these service revenues lock a user into a provider’s business that some point out the dangers but I believe in time we will allay such fears by changing how we describe, define and use cloud capacity: it’s perfectly possible for us to control our own data and rent storage and compute capability as we need it, perhaps without realising that is what we are doing, without surrendering our rights and privacy. It is not yet a trivial job to do so, but surely over 2009 we are likely to see services emerge that allow consumers to harness cloud concepts and capabilities without needing to understand the detail.

Ultimately though, the real benefit over the years ahead will be the possibilities these services offer the programmers wise enough to harness them: without the CAPEX requirements, the only limitation developers seem to encounter is that their imagination struggles to break free from the bonds that have dominated careers to date. It is hard after decades of worrying about RAM, storage and CPU limitations to have all them removed, or at a minimum re-shaped. This is the beginning – if we can struggle to imagine it – of something huge.

This weekend just gone, I attempted to compete in Railsrumble 2007 with an application I call ‘Kagtum’. The idea of Rails Rumble is to take 48 hours to build a Rails app from scratch in a competitive scenario. At around 7:30pm BST last night I realised I wasn’t going to finish the app and so I asked to be withdrawn, as I explained:

Maybe it was the fact I decided to try and compete on my own, and therefore didn’t have the advantage of a team. Maybe it was the fact it took half the first day just to get a working application stack up on linode. Maybe it was the Saturday afternoon spent in the company of friends rather than coding. Maybe the idea was too ambitious.

Maybe I just wasn’t good enough.

Whatever it was, I didn’t get enough of my idea implemented on the weekend of the 8th and 9th September 2007, that I wanted other people to look at it. I failed. There is no app here.

The ideas I played with in those 48 hours though, intrigue me, and they will be worked on over the coming weeks and months. The end goal is going to either be very interesting, or an exercise in futility. If you want to find out which, you can keep an eye on the blog and I’ll be making announcements there.

I will be judging, and I look forward to seeing other apps, so good luck. Until next year…

The first day did kill me – linode was under heavy load (not surprisingly, with over 100 teams trying to get their application stacks set up) and the guidance we had been given by way of a screencast was inaccurate in places. Top tip: when you’re in a hurry leave the rdoc behind and always pass “-d” to gem install.

Anyway, I still hope to judge – and I’d advise anybody with an interest in innovation to look out for the announcement that you can sign up for judging and take a look at the apps that were finished – but I thought I’d talk about Kagtum a little bit here, because the core is almost done and I’m confident I can get a working app out of the door soon. I’m also tempted to open source it.

It all started about 2 years ago when I was left distinctly underwhelmed by Wikinews. The problems with wikinews are many and pretty obvious to anybody who spends a few minutes digging.

The primary problem to my mind is that they’re using a piece of software designed to build an encyclopedia to build a news website which means all articles are given equal footing. It seems reasonable that they should be given equal footing, until you realise that unlike an encyclopedia, not all news items are equal. A world-famous opera singer dying is not equal to a drunken brawl in my local town centre, and neither are equal to the Iraqi PM losing the confidence of the Iraqi citizenship.

However, the core idea – news written by, and for, everybody is a great idea. I’ve spent the last two years playing with lots of ideas in my head and watched emerging developments in the online news and journalism scene before I came up with an answer: quite simply it comes down to targeting relevance.

If I am in Manchester UK, there are stories that are local to me I’m interested in that somebody in New York doesn’t want to see. Likewise, there are stories happening on the other side of the planet which are important to me because they have an impact on me, or because they are in an area I have an interest in. The “perfect” news website will know this, and present just the articles I need to see. Ideally, I also don’t want to be bogged down with partisan and opinionated pieces – I want impartial, simple, Economist-news-page-style articles that give me the leader and then show me what is out there being written about it.

Thus was born the concept of Kagtum – the phrase “kag tum” means “to bring news” in Sumerian, the script of which is the oldest written language currently known to mankind. Kagtum will be a wiki news site that helps target articles based on relevance to you and your life. Relevance is everything.

The idea is quite simple, but the algorithm needs some polish before I can roll it out: we create a news story perhaps based on a report in MSM, or perhaps as a first-hand eyewitness account, that points to online sources if available. We then attach to that story some “impact profiles” based on location or a tags.

For example, a story happening on my street (say, planning permission for a new development) would have a geocoded location and an impact radius of the local neighbourhood. A story happening in 10 Downing Street would also be tagged with that location but could have an impact on the whole of the UK. Suppose the latter story was a policy announcement on Iraq – we’d add Iraq as a location impact as well.

I then login and give my location as my postcode or street, which is geocoded, and the software knows that the story in my street is relevant to me, so is the story in Downing Street. It knows therefore, what is relevant to each and every user and displays the appropriate stories.

Let’s suppose however I have no interest in Iraq. We can tag stories and users can also add tags to their profile that they’re very interested in or very disinterested in. If I said I wanted all stories marked “Iraq” to be pushed down the queue then its relevance to me would be lowered – it might still appear, just not as prominently.

In theory then, when I log in to kagtum, I would see stories about technology, politics and cricket, particularly with stories about my local neighbourhood (stories about technology in my neighbourhood would be even more prominent), whilst my friend who doesn’t care about anything but beer and football will see something perfectly tailored for his interests.

It may also be the case that there are multiple profiles for each user (home vs. work) and that a user can add multiple locations – where they live, where they work, where they grew up, where their parents live – and sees a mixture of stories about places they care about.

The biggest problem I had this weekend was developing the specifics of knowing which stories to show to each user. The problem isn’t hard algorithmically, but providing a technique that doesn’t harm performance and can scale to more than a few hundred users online at a time is proving a little tricky using standard ActiveRecord associations and using the methods baked into GeoKit by default.

There is also an issue of what we mean by “radius”. Saying “this story is important to everybody within 5 miles” is simple enough, but what if I say “to everybody within Greater Manchester”? I somehow need to know if a given longitude and latitude is within that district or not. The Radii Problem (as I came to call it whilst muttering to myself) is important and it’s difficult. I discovered it as I added a story in Washington that was important for the whole of the US – if I added a simple radius of 3,500 miles (to take in California) it of course also covered a huge chunk of Canada, Middle America, the Caribbean, the whole of the North Atlantic (including Ireland!) and most of the North Pole. For a story about domestic US politics, this is obviously needlessly “grabby”.

I have ideas on how to solve this problem, but they’re going to take a few weeks of playing with datasets from the UN and other agencies to be able to get them working smoothly.

There are other aspects I have planned for the site around developing narratives and helping individuals become kagtum journalists, but I’ll keep discussion of those for after the roll-out of both kagtum and the new vagueware.com.

I’ve turned comments on for this article, so if you have thoughts, ideas or suggestions, please leave them.

Adrian Holovaty announced a while back templatemaker which sounds more complicated than it is.

Take a pile of web pages all built using the same template – say, restaurant listings, or a timetable – and throw it at the code. It produces a copy of the ‘template’ with the ‘data holes’ marked up. You then look at a new page, and it gives you the raw data. No more screen scraping, it just learns what it needs to find the data that changes each time. Nice.

A really useful aspect is that whilst he’s implemented it with Python bindings, the core is written in C and uses the Python C bindings. That means with a bit of hackery, porting it to Ruby should be quite simple. A project for the weekend, perhaps?

I’m shouting this up for two reasons: it’s a piece of code I’m sure all of us have thought about implementing, but never got around to producing because it just ‘felt’ too difficult, so it’s definitely very innovative from that perspective; secondly, it’s using the C bindings in an intelligent way to get performance where it counts – something I’m going to need to do more of over the next few months in other languages.

I suspect the future isn’t going to be about using the right language for a project, but the right language for the right part of the job. You could implement something in Erlang, which can interface with Ruby, which in turn can interface with C. There are points of complexity there that a competent software engineer would shudder over, I’m starting to feel that being competent in one language or framework is definitely not going to be enough for the next generation of software: we need the flexibility and power of scripting languages, and the raw power of compiled C/C++.

Almost twenty years ago a process was started that culminated in the recent announcement that the game of checkers (or ‘draughts’) can now be played flawlessly by computer. The technique is a brute-force attack, and so now there is a computer somewhere storing every single possible combination of pieces and able to work out the optimum next move in every scenario.

It turns out that the ‘best’ result will be a draw. It might not like look tic-tac-toe, but when you have mapped the complexity out enough, it is eventually just as futile. It’s going to take the computation and storage of 10²⁰ more positions than checkers took, but within a few decades we should have the computing ability to map out every possible game of chess. My prediction on that: white will always win if both sides play flawlessly.

Ron Evans predicts that it is:

“… only a matter of time before sufficient computing power allows the machines to contemplate eventualities of which we have not even postulated the existence.

The Singularity IS coming…”

“Only a matter of time” is a lovely prediction to lob into conversation. Sure, it’s only a matter of time. It could be 15,000 years mind, but we’re in no hurry…

I’m sceptical about this happening any time soon, simply because the entropy of the natural World is so vast and immense that it makes a chess game look like child’s play. Sure, us humans might have a problem understanding the size and scope of every conceivable move in chess, but compared to every conceivable possibility within the World around us, it’s nothing.

There are however, interesting applications we can start to conceive of now. Game theory has been abused by war-mongers in the Rand corporation to the point it no longer has any real credibility, but if the flaws are removed, it’s possible for computers to start finding new theories we had not yet considered. The impact within systems that are bound by rules and predictable behaviour (currency trading, for example) could be huge.

What really intrigued me however was something the creators of the ultimate checkers program had to create as a byproduct of their work. The limitations of current hardware meant they had to get innovative about how they actually stored all of those possible positions:

“For example, they stored the outcomes for the 39 trillion possible positions for endgames in a mere 237 gigabytes of computer-storage space, an average of 154 positions per byte. The mathematicians are now applying these techniques to bioinformatics, looking for ways to manage the massive quantity of data generated by the sequencing of genomes.”

We’re delving into work where the storage requirements are becoming immense, and the last decade of having “enough” storage for most work meant we didn’t need to get creative around storage and search algorithms. The future is perhaps not about taking generic algorithms and applying them to our data, but finding new mathematical models of representing what we need to and creating domain-specific algorithms.

The singularity might be a way off, but finding a way of getting there is going to be intellectually stimulating, regardless.

Michael Kavis has posted up a short note on The Lost Art of Software Engineering and asks whether we have we lost the skills that once dominated the industry. Are we just looking after the business side of things, and is that wrong?

As somebody whose degree was Software Engineering – as opposed to ‘Computer Science’ that dominates the UK Universities – I feel that I can give a little insight into this.

First of all, I believe he’s talking about the wrong thing. Software Engineering is literally applying formal methods to the process of development in order to make sure that it always behaves in a predictable manner. The software can fail, but it should fail in a way you expect and have accounted for in terms of providing failover or re-instantiation (i.e. you can reboot without killing anybody). Typically, it is used for designing fly-by-wire systems on aircraft, management of nuclear power stations, systems that have a direct effect on health like life support or MRI scanners, or for embedded systems that might be difficult to maintain such as satellites or marine equipment.

Software engineering is not about transactions per second or response times unless those are business requirements. Usually software engineering is purely about applying formal methods to ensure the most critical business requirement of all is met: nobody’s death is caused by the system.

This leads me to where I think Michael has taken a wrong turn: the quantifiable measures he talks about in his post are not showing up in specification or tender documents because the business requirements don’t require them to. There might be a point that business requirements aren’t discussing system requirements like response time enough, but we are now at a point where we are dealing more with humans and their requirements than we are computers and their requirements.

This isn’t something we should be ashamed of. We now have the memory, CPU and tools to be able to build software that fits around our environment and we should embrace that. It’s actually quite hard to build a web application that doesn’t answer within a second, and if it does become an issue we then use the most useful tool in our box: re-factoring. Never optomise too early, scale when you need to. The business requirements in most cases is to get something shipping, and fast.

Over the last few years there has been an upswing in the interest around Agile and specifically design principles as they apply to software: as programmers we are beginning to understand our work at a philosophical level as much as we are a technical level. This is going to lead to developments that we couldn’t dream of a few years ago, and it’s because we’re freeing our mental capacity from having to worry about malloc() and free() every few minutes.

Where the skills we have as engineers come into play is making sure that the business requirements are understood and expressed in the simplest possible system. Instead of thinking about responses per second, we are free to concentrate on model validation. Instead of mathematically proving an algorithm will always work, we are building automated test frameworks. We actually spend more time engineering and less time hacking than we did just three or four years ago.

Software engineering isn’t a lost art, and neither is systems engineering and design. They might have evolved, but they’re far from lost: they’re just getting started, but now they’re much more hands-on.

In this article, I’ll be exploring some of the issues around creative thinking and software development. This is a theme I expect to be re-visiting a lot over the next few weeks, so these are more high-level riffs. Each of the major points here needs expanding upon, but I’m interested in seeing what people out there lock into as the most interesting aspects of this.

I believe that raw creativity is not normally associated with the development side of the software industry. This is an error that hampers development, innovation and even the core process of programming. Creative thinking should be taught to all programmers and embraced as a way of working.

Typically only the people involved in the user interface, information architecture and marketing stages of the development process get to do the “original vision thing”. Whilst this goes on, programmers are expected to behave like glorified plumbers by connecting interactions together with a logic that meets the spec and passes tests. This skews the development process towards the wrong people.

Project Managers and Information Architects are going to be aghast – Designers will be screaming – but it is the “Developers” who are the best placed to understand what is and isn’t possible. When asked to be involved in the earliest stages, or in stages they’re not normally invited into (yes, even marketing meetings) and encouraged to contribute to the creative process, several benefits are likely to occur:

1. The programmer becomes more involved in the project and is less likely to think about “those idiots” down the hall

2. The project itself will improve – programmers can often point out things that are possible that nobody else had even thought doable. Typically programmers are early-adopters and keen to produce things that impress. Mediocre software is always designed by marketing, great software by people who really understand what software can do when given a chance.

3. The project constraints will be better understood right from the word ‘go’ and proposals made by IAs or Designers who don’t understand the ‘code impact’ can be quickly managed and dealt with rather than getting in front of a client who signs off and expects to see it in the final shipped product.

Some programmers will protest, because they don’t want to go to meetings. This is because most meetings that pretend to be about original thinking aren’t anything of the sort. If you create an environment where genuine thoughtfulness, creativity and interesting ideas are produced and managed, programmers would have a different opinion of those meetings. The sad truth is, most people in the software industry who aren’t programmers are there because they aren’t very interested in this kind of thing. That’s one of the reasons why so many programmers are going it alone right now.

One argument from the other side is that programmers aren’t very good at creative thinking. Those people might be right, but that’s because they think about the process inaccurately.

The truth is, where most programmers learn their trade at first is in building algorithms. There isn’t much scope in the minds of most programmers for creativity at the algorithm level. They have been taught to look for patterns they’ve seen before – the reason why every undergraduate has to undergo at least one course in Data Structures & Algorithms during their college years – and code reuse is preferred over “reinventing the wheel”.

Knowing certain sort algorithms are better in some ways than others is certainly beneficial, as is code reuse. However, the idea that programming is something you can learn once and then just repeat over and over again is absurd.

This has been scaled up from the algorithm level into the project level over the years. It was the “building block” approach to programming: if you look at an algorithm and implement it like so, and it becomes efficient to remove creative thinking here, then it must scale up. The result is that programmers have been trained to think of projects at times in the same way they’ve been taught to think of algorithms: look for patterns, reuse where possible, don’t break the mold.

It is only in recent years that academics have even started considering teaching their students any other method than waterfall for managing a software project, and when a programmer is confronted with their team at the first job working in an agile fashion (if you can find me a team still dealing 100% with waterfall, I’ll show you an expensive team), they are going to find themselves ill-equipped for the average work day. They are going to have to find a way to keep their head above water, and what tends to happen is they hack it. They don’t have the tools – because they’ve been told they’ll never need them – to think creatively.

What’s more this problem is compounded by the fact that it’s no longer enough in an agile World for a programmer to just know how to write algorithms. All of a sudden, the information architect has become redundant as the software engineer takes the role of seeing how the whole system plugs together. There is even an argument that the information architect was always redundant.

Addressing these issues aren’t simple. If creativity is so important, why don’t more programmers engage with it?

Programmers are experts on thinking through complex problems. In theory, all that needs to be done is to teach creative and original thinking as thinking through a complex problem. Even the worst “creativity consultant” can tell you however that the enemy of creativity is habit, so they would argue this is likely to fail. The creative process looks so bizarre to an analytical mind specifically because it can’t be easily explained. This, however, is a misunderstanding.

The real issue with creative thinking is that many people are scared of failure. Programmers, especially so. This is because in real life, as in a software project, failure is expensive. Time lost is time lost – we fear doing something that we can’t take back. We look forward to our next actions in life through the lens of our past, trying to make sure we don’t make the same mistakes again. Programmers thrive on doing this – it’s what makes the analytical mind tick. This leads to unassertive caution and bashful timidity. We are too scared to put our hand up and say “let’s do something reckless” in a project meeting for exactly the same reason we won’t say it to a stranger in a bar we’re attracted to: if it fails, we feel it, and we have to deal with it.

Creative thinking is therefore not thinking about breaking the habit of doing something repetitive, but breaking the habit of doing nothing at all. It’s about doing whatever we can, over and over again, just to get one little gem of an idea out there. In the world of brainstorming, failure is cheap, and risk-taking gets the prize. It is this that should be encouraged within our industry because we are so desperate for it: brazen, reckless, failure that costs nothing, followed by the sparkle of genius that changes the World.

If creativity in programmers might help projects, or help develop programmers themselves, one question needs to be asked: Where exactly does creativity need to be applied in the software industry?

In short: Everywhere.

Software development is a creative industry, as all pinnacles of civilisation ultimately are, from stock markets to museums. The fact that we still describe the disciplines in terms of engineering or science confuses me: it may be that when a deep understanding of electronics was needed before you could begin to write software, this seemed sensible. But in the age of abstract languages like Ruby or Python? Dare I say even Java?

At the core of this is how we go about deciding what software to write in the first place. This is an area, which you may be surprised to hear that there is little creativity.

In fact, nearly all software is a derivative of a previously-available software. Even open-source, where you would expect creativity to thrive is often a collection of re-writes of commercial software which in itself is nearly always a derivative of a new way of making a quick buck. Firefox might be swell, but it is rooted in Netscape which in itself was just a way of making some money out of somebody else’s bright idea. Tim Berners-Lee might have had an original thought once, but don’t think that anything you’ve seen since then around web servers or web browsers has been original.

Then there is the implementation of the software. Programmers like code reuse. They love libraries. They would marry their preferred framework if it were philosophically sensible or legal. However, it is producing a generation of software that is derivative.

Right now, developers are in a “group-think” position that we follow the principles of doing things the way they always have been done. Sometimes this makes sense: windowing systems, cryptographic libraries, etc. However, whilst Rails and Django sure are swell, is it really the case that what is needed in the World is more CRUD-interfaces to relational databases? So often you see projects that could have been implemented more elegantly in less code, but rarely are. “_why the lucky stiff” is a notable exception to this trend: learning from watching him is recommended.

This brings us to the marketing of software. Right now you can sell your time, sell the code, or find a way to incorporate adverts. Sure “monetisation” is important: critical if you want to keep in your XXXL shape. That said, are there really only 3 ways to making money? Of course not, but whenever a new product is thought up, it’s always put into one of a few different silos. Try this: think of all the ways people are able to make money legally in the world from commissions to referrals to services, and then try and find a way to get your ‘great idea’ to fit each of those silos. You might end up inventing something completely original.

Creative thinking isn’t hard – there are thousands of ways of doing it. The question is: why aren’t we programmers doing more of it?

In my last post to the blog, I talked about open sourcing code and building a business off it. I’m now getting more convinced that’s the way to go, and aim to pull down this site and replace it with something more suited to that purpose in the next week. The content here will be retained, but in another form and will not sit on the front page.

For those of you reading via RSS, this is a heads-up that this feed might break in the next week, but I’ll do my best to make sure everything gets directed appropriately and that all existing content on the site gets carried over to the new site – I get too much traffic from old articles to throw it away.

Anyway, onto the details of “the plan”.

One thing to note is that Vaueware’s language of choice will be Ruby, and therefore 100% (or as close to it as possible) of the code produced and put out into the wild will be Ruby. One of the goals is to lower the barrier of participation, and I believe Ruby to be one of the simplest programming languages in the World to understand, whilst also being one of the most powerful. Over the course of this year, I’ll be putting considerable effort into producing Ruby and Rails tutorials to help in getting more people involved in producing actual code: I’m big on participation.

Another thing to note is that I hope to break a paradigm here in one regard: open source is very good at copying what has been done before but very bad at genuine innovation in any one field. There are examples where this is not true, but virtually every successful open source project in existence is a clone of a commercial piece of software in some regard. If not in function, at least in genre and paradigm, we follow where others lead. I really don’t want to see Vagueware go down that route: the World has enough clones of Drupal, and in the Rails world Mephisto does nicely, thanks.

A central part of the idea for the next stage of Vagueware is an area I call “The Labs”. The idea is that anybody can sketch out ideas for new ways of doing things, for new types of application, even if they have no experience of developing software, and then interesting ideas get developed by Vagueware. This isn’t another “let’s have a competition where the best idea gets money” site, this is about encouraging that tiny corner of inspiration in all of us into a collaborative effort to produce better tools. We then build all those ideas, or as many as possible, together. And give it all away for free. Then, build revenue models around services on those tools.

So how will this work? Well, here’s a scenario:

Johnny has an idea about how to use a web application to do something interesting. He realises that he doesn’t have the skill and/or time to develop it himself, and he is more interested in seeing the idea in action than he is in making money off it.

Johnny vists the Vagueware labs site, and submits a short proposal. This gets moderated along the following criteria:

1. Can it be developed as an open source project without landing us in trouble?
2. Can it be developed by the current skill base of Vagueware and the wider community?
3. Is it an interesting and good idea?

The last one is important, but not as important as the first two. However, originality is key as the focus is on open source research and development, as opposed to open source software. I want to try weird and wonderful things nobody else has done before. It’s the only way this will be fun.

Did I mention this had to be fun? If it’s not challenging, there’s no point. Profit follows meaning, so if we create meaning out of fun and interesting projects, somehow I’ll end up being able to pay my rent. Yes, I really do believe that.

So, then Johhny’s idea is accepted, and it gets a whole area of the labs to itself, and anybody visting the labs can see it. A bunch of wiki pages are there for people to bulk the idea out, add business plans and possible uses, catches and possible risks, etc. All of this activity is open to anybody, regardless of who they are, what their background is, etc. Attached will be some basic forums for community discussion (I hate Wikipedia’s discussion pages, the lack of structure makes communication harder rather than easier), and the idea ferments.

So far, this will sound familiar to those who were around for v1.0 of Vagueware back in 2003/04. At this point though, things get a little more focused than they did back then.

At this point I code something basic – the absolute minimum to get the idea to v0.1 and place it in an SVN repository. I invite patches, I develop it further myself. If I can, I’ll pay people to work on it and add to it. As it gets closer to v1.0 hosted/packaged versions are made available for non-techs to be able to use, but anybody can download it and do what they want with it. Vagueware’s revenue will be based on offering support, maintenance and other services around that core concept. Licensing will be MIT/BSD in virtually all cases.

What about Johnny doesn’t he make something? Well, his software is now written, and he can download it and run it for free. He can bask in the glow his idea came to fruitition and he made the World a better place. He can install it on a server and market it himself. It won’t have cost him a penny to ‘develop’ the idea so that represents some decent ROI right there.

I expect in the early stages, there will be a lot of ideas around open sourcing existing platforms like YouTube, MySpace, etc. and providing a platform for people to develop those ‘traditional’ Web 2.0 ideas (if that isn’t a contradiction), but eventually it would be nice to think revenue can come in from the more revolutionary ideas that require real R&D. I’m particularly interested in Vagueware Labs focusing on software as a mode of design as opposed to churning out code.

Will it work? I have absolutely no idea. If it doesn’t, the worst case scenario is I have to go and get a real job. Best case scenario? Everybody gets involved in some interesting ideas. What if nobody submits ideas? I have a stack of about 400 of my own I’ll make public over time. The Web is full of interesting ideas. What if people take the idea and make millions off it and I end up broke? C’est la vie, personally I don’t think it’ll happen though: the key to this is in the execution and strategy, not in the idea. If I’m too lazy to execute delivery properly, I deserve to go broke.

Like I said in my last e-mail, all of this takes a reasonable amount of courage because I don’t know of anybody else doing this, and nobody is telling me this business model will work. I think it’ll pay in the long term though. If it doesn’t, back to the drawing board.

Right now, almost 100% of Vagueware’s revenue is through services. The objective for 2007 was to switch this so that nearly 100% of the revenue would be through products. Offering excellent little applications for SMEs and corporates, who would pay a license fee to oil their squeaking machines of information flow. Oh, the ideas. Many of them unique, some of them just better executions of what has been done before. You know the dream, you’ve had it yourself: build a customer base, sell the company and retire inside five years.

But there is a niggle. I have come to realise that as a one-man company, in need to grow the business I need to do something rather extraordinary. I’ve realised it’s not enough to produce really good products, and to execute the marketing well if I want to develop a solid business: I might score lucky and get a company I can sell pretty quickly that way, but if I want something sustainable and to grow now without taking extra finance on board, I need to do something different.

I want to build something people trust isn’t going to go under tomorrow, that can develop without me being at the helm all day long, that can go off on weird tangents without me spending all my time trying to think up new ideas, and that can get the benefits of a diverse and enthusiastic development team behind it without me spending huge amounts of money on coders.

In other words, I need to open source my code.

By making my code base open source I get around all those issues of “what happens if you get run over by a bus” and I help foster a far larger development community than I could ever hire. By giving the code away for free I help develop competition which – call me a free marketeer if you wish – I believe helps my company as much as it helps those who didn’t come up with the idea in the first place.

The disadvantage of course comes in the monetisation stage: by not locking the code base down, it’s not enough to just offer a product licensing fee. I need to think in terms of services people will pay for. Hosting, appliance boxes, customisation, support and maintenance deals, training, etc. all have to be considered. No longer is it just Software as a Service, but the software is free with an optional service. That requires a big re-think.

It also means a fundamental re-think of how I approach development of new ideas. I’ve realised that it’s not enough for me to just come up with an idea, throw out v0.1 and hope it flies. There’s a lot of scribbling in notepads as to how to deliver something a little “different” in that regard. With regards to those services, the I’ve filled several dozen pages with ideas, and I hope to experiment with all of them at some stage.

However, I have a confession to make: I’m scared.

All of this requires me to be ridiculously brave, and say to hell with the money: the money will take care of itself. That would be easy to do if I were rich and didn’t need to think about rent, bills and food, but I’m not. I’m pretty much broke and if it weren’t for that bespoke development work, I’d be pretty hungry too. But I’ll find a way, somehow. If it means taking on bespoke work through the rest of this year whilst I wait for revenues to pick up on other services, so be it.

This is going to take some courage I think, but I believe it’s the right thing to do in the long run. Along the way, I expect this site (and blog, and therefore feed) to get trashed in the coming weeks whilst I build something more aligned to where this is heading. Those of you who remember a previous incarnation of Vagueware will find some recurring themes in what gets released in a few weeks: that’s a good thing.

It’s going to be an interesting ride. Anybody want a ticket?