This weekend just gone, I attempted to compete in Railsrumble 2007 with an application I call ‘Kagtum’. The idea of Rails Rumble is to take 48 hours to build a Rails app from scratch in a competitive scenario. At around 7:30pm BST last night I realised I wasn’t going to finish the app and so I asked to be withdrawn, as I explained:

Maybe it was the fact I decided to try and compete on my own, and therefore didn’t have the advantage of a team. Maybe it was the fact it took half the first day just to get a working application stack up on linode. Maybe it was the Saturday afternoon spent in the company of friends rather than coding. Maybe the idea was too ambitious.

Maybe I just wasn’t good enough.

Whatever it was, I didn’t get enough of my idea implemented on the weekend of the 8th and 9th September 2007, that I wanted other people to look at it. I failed. There is no app here.

The ideas I played with in those 48 hours though, intrigue me, and they will be worked on over the coming weeks and months. The end goal is going to either be very interesting, or an exercise in futility. If you want to find out which, you can keep an eye on the blog and I’ll be making announcements there.

I will be judging, and I look forward to seeing other apps, so good luck. Until next year…

The first day did kill me - linode was under heavy load (not surprisingly, with over 100 teams trying to get their application stacks set up) and the guidance we had been given by way of a screencast was inaccurate in places. Top tip: when you’re in a hurry leave the rdoc behind and always pass “-d” to gem install.

Anyway, I still hope to judge - and I’d advise anybody with an interest in innovation to look out for the announcement that you can sign up for judging and take a look at the apps that were finished - but I thought I’d talk about Kagtum a little bit here, because the core is almost done and I’m confident I can get a working app out of the door soon. I’m also tempted to open source it.

It all started about 2 years ago when I was left distinctly underwhelmed by Wikinews. The problems with wikinews are many and pretty obvious to anybody who spends a few minutes digging.

The primary problem to my mind is that they’re using a piece of software designed to build an encyclopedia to build a news website which means all articles are given equal footing. It seems reasonable that they should be given equal footing, until you realise that unlike an encyclopedia, not all news items are equal. A world-famous opera singer dying is not equal to a drunken brawl in my local town centre, and neither are equal to the Iraqi PM losing the confidence of the Iraqi citizenship.

However, the core idea - news written by, and for, everybody is a great idea. I’ve spent the last two years playing with lots of ideas in my head and watched emerging developments in the online news and journalism scene before I came up with an answer: quite simply it comes down to targeting relevance.

If I am in Manchester UK, there are stories that are local to me I’m interested in that somebody in New York doesn’t want to see. Likewise, there are stories happening on the other side of the planet which are important to me because they have an impact on me, or because they are in an area I have an interest in. The “perfect” news website will know this, and present just the articles I need to see. Ideally, I also don’t want to be bogged down with partisan and opinionated pieces - I want impartial, simple, Economist-news-page-style articles that give me the leader and then show me what is out there being written about it.

Thus was born the concept of Kagtum - the phrase “kag tum” means “to bring news” in Sumerian, the script of which is the oldest written language currently known to mankind. Kagtum will be a wiki news site that helps target articles based on relevance to you and your life. Relevance is everything.

The idea is quite simple, but the algorithm needs some polish before I can roll it out: we create a news story perhaps based on a report in MSM, or perhaps as a first-hand eyewitness account, that points to online sources if available. We then attach to that story some “impact profiles” based on location or a tags.

For example, a story happening on my street (say, planning permission for a new development) would have a geocoded location and an impact radius of the local neighbourhood. A story happening in 10 Downing Street would also be tagged with that location but could have an impact on the whole of the UK. Suppose the latter story was a policy announcement on Iraq - we’d add Iraq as a location impact as well.

I then login and give my location as my postcode or street, which is geocoded, and the software knows that the story in my street is relevant to me, so is the story in Downing Street. It knows therefore, what is relevant to each and every user and displays the appropriate stories.

Let’s suppose however I have no interest in Iraq. We can tag stories and users can also add tags to their profile that they’re very interested in or very disinterested in. If I said I wanted all stories marked “Iraq” to be pushed down the queue then its relevance to me would be lowered - it might still appear, just not as prominently.

In theory then, when I log in to kagtum, I would see stories about technology, politics and cricket, particularly with stories about my local neighbourhood (stories about technology in my neighbourhood would be even more prominent), whilst my friend who doesn’t care about anything but beer and football will see something perfectly tailored for his interests.

It may also be the case that there are multiple profiles for each user (home vs. work) and that a user can add multiple locations - where they live, where they work, where they grew up, where their parents live - and sees a mixture of stories about places they care about.

The biggest problem I had this weekend was developing the specifics of knowing which stories to show to each user. The problem isn’t hard algorithmically, but providing a technique that doesn’t harm performance and can scale to more than a few hundred users online at a time is proving a little tricky using standard ActiveRecord associations and using the methods baked into GeoKit by default.

There is also an issue of what we mean by “radius”. Saying “this story is important to everybody within 5 miles” is simple enough, but what if I say “to everybody within Greater Manchester”? I somehow need to know if a given longitude and latitude is within that district or not. The Radii Problem (as I came to call it whilst muttering to myself) is important and it’s difficult. I discovered it as I added a story in Washington that was important for the whole of the US - if I added a simple radius of 3,500 miles (to take in California) it of course also covered a huge chunk of Canada, Middle America, the Caribbean, the whole of the North Atlantic (including Ireland!) and most of the North Pole. For a story about domestic US politics, this is obviously needlessly “grabby”.

I have ideas on how to solve this problem, but they’re going to take a few weeks of playing with datasets from the UN and other agencies to be able to get them working smoothly.

There are other aspects I have planned for the site around developing narratives and helping individuals become kagtum journalists, but I’ll keep discussion of those for after the roll-out of both kagtum and the new vagueware.com.

I’ve turned comments on for this article, so if you have thoughts, ideas or suggestions, please leave them.

Adrian Holovaty announced a while back templatemaker which sounds more complicated than it is.

Take a pile of web pages all built using the same template - say, restaurant listings, or a timetable - and throw it at the code. It produces a copy of the ‘template’ with the ‘data holes’ marked up. You then look at a new page, and it gives you the raw data. No more screen scraping, it just learns what it needs to find the data that changes each time. Nice.

A really useful aspect is that whilst he’s implemented it with Python bindings, the core is written in C and uses the Python C bindings. That means with a bit of hackery, porting it to Ruby should be quite simple. A project for the weekend, perhaps?

I’m shouting this up for two reasons: it’s a piece of code I’m sure all of us have thought about implementing, but never got around to producing because it just ‘felt’ too difficult, so it’s definitely very innovative from that perspective; secondly, it’s using the C bindings in an intelligent way to get performance where it counts - something I’m going to need to do more of over the next few months in other languages.

I suspect the future isn’t going to be about using the right language for a project, but the right language for the right part of the job. You could implement something in Erlang, which can interface with Ruby, which in turn can interface with C. There are points of complexity there that a competent software engineer would shudder over, I’m starting to feel that being competent in one language or framework is definitely not going to be enough for the next generation of software: we need the flexibility and power of scripting languages, and the raw power of compiled C/C++.

Almost twenty years ago a process was started that culminated in the recent announcement that the game of checkers (or ‘draughts’) can now be played flawlessly by computer. The technique is a brute-force attack, and so now there is a computer somewhere storing every single possible combination of pieces and able to work out the optimum next move in every scenario.

It turns out that the ‘best’ result will be a draw. It might not like look tic-tac-toe, but when you have mapped the complexity out enough, it is eventually just as futile. It’s going to take the computation and storage of 10²⁰ more positions than checkers took, but within a few decades we should have the computing ability to map out every possible game of chess. My prediction on that: white will always win if both sides play flawlessly.

Ron Evans predicts that it is:

“… only a matter of time before sufficient computing power allows the machines to contemplate eventualities of which we have not even postulated the existence.

The Singularity IS coming…”

“Only a matter of time” is a lovely prediction to lob into conversation. Sure, it’s only a matter of time. It could be 15,000 years mind, but we’re in no hurry…

I’m sceptical about this happening any time soon, simply because the entropy of the natural World is so vast and immense that it makes a chess game look like child’s play. Sure, us humans might have a problem understanding the size and scope of every conceivable move in chess, but compared to every conceivable possibility within the World around us, it’s nothing.

There are however, interesting applications we can start to conceive of now. Game theory has been abused by war-mongers in the Rand corporation to the point it no longer has any real credibility, but if the flaws are removed, it’s possible for computers to start finding new theories we had not yet considered. The impact within systems that are bound by rules and predictable behaviour (currency trading, for example) could be huge.

What really intrigued me however was something the creators of the ultimate checkers program had to create as a byproduct of their work. The limitations of current hardware meant they had to get innovative about how they actually stored all of those possible positions:

“For example, they stored the outcomes for the 39 trillion possible positions for endgames in a mere 237 gigabytes of computer-storage space, an average of 154 positions per byte. The mathematicians are now applying these techniques to bioinformatics, looking for ways to manage the massive quantity of data generated by the sequencing of genomes.”

We’re delving into work where the storage requirements are becoming immense, and the last decade of having “enough” storage for most work meant we didn’t need to get creative around storage and search algorithms. The future is perhaps not about taking generic algorithms and applying them to our data, but finding new mathematical models of representing what we need to and creating domain-specific algorithms.

The singularity might be a way off, but finding a way of getting there is going to be intellectually stimulating, regardless.

Michael Kavis has posted up a short note on The Lost Art of Software Engineering and asks whether we have we lost the skills that once dominated the industry. Are we just looking after the business side of things, and is that wrong?

As somebody whose degree was Software Engineering - as opposed to ‘Computer Science’ that dominates the UK Universities - I feel that I can give a little insight into this.

First of all, I believe he’s talking about the wrong thing. Software Engineering is literally applying formal methods to the process of development in order to make sure that it always behaves in a predictable manner. The software can fail, but it should fail in a way you expect and have accounted for in terms of providing failover or re-instantiation (i.e. you can reboot without killing anybody). Typically, it is used for designing fly-by-wire systems on aircraft, management of nuclear power stations, systems that have a direct effect on health like life support or MRI scanners, or for embedded systems that might be difficult to maintain such as satellites or marine equipment.

Software engineering is not about transactions per second or response times unless those are business requirements. Usually software engineering is purely about applying formal methods to ensure the most critical business requirement of all is met: nobody’s death is caused by the system.

This leads me to where I think Michael has taken a wrong turn: the quantifiable measures he talks about in his post are not showing up in specification or tender documents because the business requirements don’t require them to. There might be a point that business requirements aren’t discussing system requirements like response time enough, but we are now at a point where we are dealing more with humans and their requirements than we are computers and their requirements.

This isn’t something we should be ashamed of. We now have the memory, CPU and tools to be able to build software that fits around our environment and we should embrace that. It’s actually quite hard to build a web application that doesn’t answer within a second, and if it does become an issue we then use the most useful tool in our box: re-factoring. Never optomise too early, scale when you need to. The business requirements in most cases is to get something shipping, and fast.

Over the last few years there has been an upswing in the interest around Agile and specifically design principles as they apply to software: as programmers we are beginning to understand our work at a philosophical level as much as we are a technical level. This is going to lead to developments that we couldn’t dream of a few years ago, and it’s because we’re freeing our mental capacity from having to worry about malloc() and free() every few minutes.

Where the skills we have as engineers come into play is making sure that the business requirements are understood and expressed in the simplest possible system. Instead of thinking about responses per second, we are free to concentrate on model validation. Instead of mathematically proving an algorithm will always work, we are building automated test frameworks. We actually spend more time engineering and less time hacking than we did just three or four years ago.

Software engineering isn’t a lost art, and neither is systems engineering and design. They might have evolved, but they’re far from lost: they’re just getting started, but now they’re much more hands-on.

In this article, I’ll be exploring some of the issues around creative thinking and software development. This is a theme I expect to be re-visiting a lot over the next few weeks, so these are more high-level riffs. Each of the major points here needs expanding upon, but I’m interested in seeing what people out there lock into as the most interesting aspects of this.

I believe that raw creativity is not normally associated with the development side of the software industry. This is an error that hampers development, innovation and even the core process of programming. Creative thinking should be taught to all programmers and embraced as a way of working.

Typically only the people involved in the user interface, information architecture and marketing stages of the development process get to do the “original vision thing”. Whilst this goes on, programmers are expected to behave like glorified plumbers by connecting interactions together with a logic that meets the spec and passes tests. This skews the development process towards the wrong people.

Project Managers and Information Architects are going to be aghast - Designers will be screaming - but it is the “Developers” who are the best placed to understand what is and isn’t possible. When asked to be involved in the earliest stages, or in stages they’re not normally invited into (yes, even marketing meetings) and encouraged to contribute to the creative process, several benefits are likely to occur:

1. The programmer becomes more involved in the project and is less likely to think about “those idiots” down the hall

2. The project itself will improve - programmers can often point out things that are possible that nobody else had even thought doable. Typically programmers are early-adopters and keen to produce things that impress. Mediocre software is always designed by marketing, great software by people who really understand what software can do when given a chance.

3. The project constraints will be better understood right from the word ‘go’ and proposals made by IAs or Designers who don’t understand the ‘code impact’ can be quickly managed and dealt with rather than getting in front of a client who signs off and expects to see it in the final shipped product.

Some programmers will protest, because they don’t want to go to meetings. This is because most meetings that pretend to be about original thinking aren’t anything of the sort. If you create an environment where genuine thoughtfulness, creativity and interesting ideas are produced and managed, programmers would have a different opinion of those meetings. The sad truth is, most people in the software industry who aren’t programmers are there because they aren’t very interested in this kind of thing. That’s one of the reasons why so many programmers are going it alone right now.

One argument from the other side is that programmers aren’t very good at creative thinking. Those people might be right, but that’s because they think about the process inaccurately.

The truth is, where most programmers learn their trade at first is in building algorithms. There isn’t much scope in the minds of most programmers for creativity at the algorithm level. They have been taught to look for patterns they’ve seen before - the reason why every undergraduate has to undergo at least one course in Data Structures & Algorithms during their college years - and code reuse is preferred over “reinventing the wheel”.

Knowing certain sort algorithms are better in some ways than others is certainly beneficial, as is code reuse. However, the idea that programming is something you can learn once and then just repeat over and over again is absurd.

This has been scaled up from the algorithm level into the project level over the years. It was the “building block” approach to programming: if you look at an algorithm and implement it like so, and it becomes efficient to remove creative thinking here, then it must scale up. The result is that programmers have been trained to think of projects at times in the same way they’ve been taught to think of algorithms: look for patterns, reuse where possible, don’t break the mold.

It is only in recent years that academics have even started considering teaching their students any other method than waterfall for managing a software project, and when a programmer is confronted with their team at the first job working in an agile fashion (if you can find me a team still dealing 100% with waterfall, I’ll show you an expensive team), they are going to find themselves ill-equipped for the average work day. They are going to have to find a way to keep their head above water, and what tends to happen is they hack it. They don’t have the tools - because they’ve been told they’ll never need them - to think creatively.

What’s more this problem is compounded by the fact that it’s no longer enough in an agile World for a programmer to just know how to write algorithms. All of a sudden, the information architect has become redundant as the software engineer takes the role of seeing how the whole system plugs together. There is even an argument that the information architect was always redundant.

Addressing these issues aren’t simple. If creativity is so important, why don’t more programmers engage with it?

Programmers are experts on thinking through complex problems. In theory, all that needs to be done is to teach creative and original thinking as thinking through a complex problem. Even the worst “creativity consultant” can tell you however that the enemy of creativity is habit, so they would argue this is likely to fail. The creative process looks so bizarre to an analytical mind specifically because it can’t be easily explained. This, however, is a misunderstanding.

The real issue with creative thinking is that many people are scared of failure. Programmers, especially so. This is because in real life, as in a software project, failure is expensive. Time lost is time lost - we fear doing something that we can’t take back. We look forward to our next actions in life through the lens of our past, trying to make sure we don’t make the same mistakes again. Programmers thrive on doing this - it’s what makes the analytical mind tick. This leads to unassertive caution and bashful timidity. We are too scared to put our hand up and say “let’s do something reckless” in a project meeting for exactly the same reason we won’t say it to a stranger in a bar we’re attracted to: if it fails, we feel it, and we have to deal with it.

Creative thinking is therefore not thinking about breaking the habit of doing something repetitive, but breaking the habit of doing nothing at all. It’s about doing whatever we can, over and over again, just to get one little gem of an idea out there. In the world of brainstorming, failure is cheap, and risk-taking gets the prize. It is this that should be encouraged within our industry because we are so desperate for it: brazen, reckless, failure that costs nothing, followed by the sparkle of genius that changes the World.

If creativity in programmers might help projects, or help develop programmers themselves, one question needs to be asked: Where exactly does creativity need to be applied in the software industry?

In short: Everywhere.

Software development is a creative industry, as all pinnacles of civilisation ultimately are, from stock markets to museums. The fact that we still describe the disciplines in terms of engineering or science confuses me: it may be that when a deep understanding of electronics was needed before you could begin to write software, this seemed sensible. But in the age of abstract languages like Ruby or Python? Dare I say even Java?

At the core of this is how we go about deciding what software to write in the first place. This is an area, which you may be surprised to hear that there is little creativity.

In fact, nearly all software is a derivative of a previously-available software. Even open-source, where you would expect creativity to thrive is often a collection of re-writes of commercial software which in itself is nearly always a derivative of a new way of making a quick buck. Firefox might be swell, but it is rooted in Netscape which in itself was just a way of making some money out of somebody else’s bright idea. Tim Berners-Lee might have had an original thought once, but don’t think that anything you’ve seen since then around web servers or web browsers has been original.

Then there is the implementation of the software. Programmers like code reuse. They love libraries. They would marry their preferred framework if it were philosophically sensible or legal. However, it is producing a generation of software that is derivative.

Right now, developers are in a “group-think” position that we follow the principles of doing things the way they always have been done. Sometimes this makes sense: windowing systems, cryptographic libraries, etc. However, whilst Rails and Django sure are swell, is it really the case that what is needed in the World is more CRUD-interfaces to relational databases? So often you see projects that could have been implemented more elegantly in less code, but rarely are. “_why the lucky stiff” is a notable exception to this trend: learning from watching him is recommended.

This brings us to the marketing of software. Right now you can sell your time, sell the code, or find a way to incorporate adverts. Sure “monetisation” is important: critical if you want to keep in your XXXL shape. That said, are there really only 3 ways to making money? Of course not, but whenever a new product is thought up, it’s always put into one of a few different silos. Try this: think of all the ways people are able to make money legally in the world from commissions to referrals to services, and then try and find a way to get your ‘great idea’ to fit each of those silos. You might end up inventing something completely original.

Creative thinking isn’t hard - there are thousands of ways of doing it. The question is: why aren’t we programmers doing more of it?

In my last post to the blog, I talked about open sourcing code and building a business off it. I’m now getting more convinced that’s the way to go, and aim to pull down this site and replace it with something more suited to that purpose in the next week. The content here will be retained, but in another form and will not sit on the front page.

For those of you reading via RSS, this is a heads-up that this feed might break in the next week, but I’ll do my best to make sure everything gets directed appropriately and that all existing content on the site gets carried over to the new site - I get too much traffic from old articles to throw it away.

Anyway, onto the details of “the plan”.

One thing to note is that Vaueware’s language of choice will be Ruby, and therefore 100% (or as close to it as possible) of the code produced and put out into the wild will be Ruby. One of the goals is to lower the barrier of participation, and I believe Ruby to be one of the simplest programming languages in the World to understand, whilst also being one of the most powerful. Over the course of this year, I’ll be putting considerable effort into producing Ruby and Rails tutorials to help in getting more people involved in producing actual code: I’m big on participation.

Another thing to note is that I hope to break a paradigm here in one regard: open source is very good at copying what has been done before but very bad at genuine innovation in any one field. There are examples where this is not true, but virtually every successful open source project in existence is a clone of a commercial piece of software in some regard. If not in function, at least in genre and paradigm, we follow where others lead. I really don’t want to see Vagueware go down that route: the World has enough clones of Drupal, and in the Rails world Mephisto does nicely, thanks.

A central part of the idea for the next stage of Vagueware is an area I call “The Labs”. The idea is that anybody can sketch out ideas for new ways of doing things, for new types of application, even if they have no experience of developing software, and then interesting ideas get developed by Vagueware. This isn’t another “let’s have a competition where the best idea gets money” site, this is about encouraging that tiny corner of inspiration in all of us into a collaborative effort to produce better tools. We then build all those ideas, or as many as possible, together. And give it all away for free. Then, build revenue models around services on those tools.

So how will this work? Well, here’s a scenario:

Johnny has an idea about how to use a web application to do something interesting. He realises that he doesn’t have the skill and/or time to develop it himself, and he is more interested in seeing the idea in action than he is in making money off it.

Johnny vists the Vagueware labs site, and submits a short proposal. This gets moderated along the following criteria:

1. Can it be developed as an open source project without landing us in trouble?
2. Can it be developed by the current skill base of Vagueware and the wider community?
3. Is it an interesting and good idea?

The last one is important, but not as important as the first two. However, originality is key as the focus is on open source research and development, as opposed to open source software. I want to try weird and wonderful things nobody else has done before. It’s the only way this will be fun.

Did I mention this had to be fun? If it’s not challenging, there’s no point. Profit follows meaning, so if we create meaning out of fun and interesting projects, somehow I’ll end up being able to pay my rent. Yes, I really do believe that.

So, then Johhny’s idea is accepted, and it gets a whole area of the labs to itself, and anybody visting the labs can see it. A bunch of wiki pages are there for people to bulk the idea out, add business plans and possible uses, catches and possible risks, etc. All of this activity is open to anybody, regardless of who they are, what their background is, etc. Attached will be some basic forums for community discussion (I hate Wikipedia’s discussion pages, the lack of structure makes communication harder rather than easier), and the idea ferments.

So far, this will sound familiar to those who were around for v1.0 of Vagueware back in 2003/04. At this point though, things get a little more focused than they did back then.

At this point I code something basic - the absolute minimum to get the idea to v0.1 and place it in an SVN repository. I invite patches, I develop it further myself. If I can, I’ll pay people to work on it and add to it. As it gets closer to v1.0 hosted/packaged versions are made available for non-techs to be able to use, but anybody can download it and do what they want with it. Vagueware’s revenue will be based on offering support, maintenance and other services around that core concept. Licensing will be MIT/BSD in virtually all cases.

What about Johnny doesn’t he make something? Well, his software is now written, and he can download it and run it for free. He can bask in the glow his idea came to fruitition and he made the World a better place. He can install it on a server and market it himself. It won’t have cost him a penny to ‘develop’ the idea so that represents some decent ROI right there.

I expect in the early stages, there will be a lot of ideas around open sourcing existing platforms like YouTube, MySpace, etc. and providing a platform for people to develop those ‘traditional’ Web 2.0 ideas (if that isn’t a contradiction), but eventually it would be nice to think revenue can come in from the more revolutionary ideas that require real R&D. I’m particularly interested in Vagueware Labs focusing on software as a mode of design as opposed to churning out code.

Will it work? I have absolutely no idea. If it doesn’t, the worst case scenario is I have to go and get a real job. Best case scenario? Everybody gets involved in some interesting ideas. What if nobody submits ideas? I have a stack of about 400 of my own I’ll make public over time. The Web is full of interesting ideas. What if people take the idea and make millions off it and I end up broke? C’est la vie, personally I don’t think it’ll happen though: the key to this is in the execution and strategy, not in the idea. If I’m too lazy to execute delivery properly, I deserve to go broke.

Like I said in my last e-mail, all of this takes a reasonable amount of courage because I don’t know of anybody else doing this, and nobody is telling me this business model will work. I think it’ll pay in the long term though. If it doesn’t, back to the drawing board.

Right now, almost 100% of Vagueware’s revenue is through services. The objective for 2007 was to switch this so that nearly 100% of the revenue would be through products. Offering excellent little applications for SMEs and corporates, who would pay a license fee to oil their squeaking machines of information flow. Oh, the ideas. Many of them unique, some of them just better executions of what has been done before. You know the dream, you’ve had it yourself: build a customer base, sell the company and retire inside five years.

But there is a niggle. I have come to realise that as a one-man company, in need to grow the business I need to do something rather extraordinary. I’ve realised it’s not enough to produce really good products, and to execute the marketing well if I want to develop a solid business: I might score lucky and get a company I can sell pretty quickly that way, but if I want something sustainable and to grow now without taking extra finance on board, I need to do something different.

I want to build something people trust isn’t going to go under tomorrow, that can develop without me being at the helm all day long, that can go off on weird tangents without me spending all my time trying to think up new ideas, and that can get the benefits of a diverse and enthusiastic development team behind it without me spending huge amounts of money on coders.

In other words, I need to open source my code.

By making my code base open source I get around all those issues of “what happens if you get run over by a bus” and I help foster a far larger development community than I could ever hire. By giving the code away for free I help develop competition which - call me a free marketeer if you wish - I believe helps my company as much as it helps those who didn’t come up with the idea in the first place.

The disadvantage of course comes in the monetisation stage: by not locking the code base down, it’s not enough to just offer a product licensing fee. I need to think in terms of services people will pay for. Hosting, appliance boxes, customisation, support and maintenance deals, training, etc. all have to be considered. No longer is it just Software as a Service, but the software is free with an optional service. That requires a big re-think.

It also means a fundamental re-think of how I approach development of new ideas. I’ve realised that it’s not enough for me to just come up with an idea, throw out v0.1 and hope it flies. There’s a lot of scribbling in notepads as to how to deliver something a little “different” in that regard. With regards to those services, the I’ve filled several dozen pages with ideas, and I hope to experiment with all of them at some stage.

However, I have a confession to make: I’m scared.

All of this requires me to be ridiculously brave, and say to hell with the money: the money will take care of itself. That would be easy to do if I were rich and didn’t need to think about rent, bills and food, but I’m not. I’m pretty much broke and if it weren’t for that bespoke development work, I’d be pretty hungry too. But I’ll find a way, somehow. If it means taking on bespoke work through the rest of this year whilst I wait for revenues to pick up on other services, so be it.

This is going to take some courage I think, but I believe it’s the right thing to do in the long run. Along the way, I expect this site (and blog, and therefore feed) to get trashed in the coming weeks whilst I build something more aligned to where this is heading. Those of you who remember a previous incarnation of Vagueware will find some recurring themes in what gets released in a few weeks: that’s a good thing.

It’s going to be an interesting ride. Anybody want a ticket?

Last night, a story on MacRumors.com grabbed my interest. It was grabbed for two reasons:

1. I could land a booty of 10 apps from MacHeist for the iBook that would normally cost me $400, but for just $49
2. This, apparently, is highly controversial

Now, most of the apps are OK, but not the type you would run out and buy as if your life depended on it. I already have and use Textmate, so that didn’t add value. I’d been meaning to grab copies of DEVONthink and RapidWeaver for a few months now and $49 for those two alone is a saving. The fact I was able to get Delicious Library, FotoMagico, ShapeShifter, Disco, iClip, a Pangea game (I chose Enigmo 2, which is already sucking me in), and NewsFire as bonuses and for no extra money, just made it a bit more interesting for me.

Now, here’s the thing. The blogosphere is on fire about this, not because of the remarkable value, but because the developers providing the software are apparently being ‘ripped off’ by MacHeist and despite going into this with both eyes wide open, are obviously being conned. Here’s some examples:

for MacHeist to call it “The Week of the Independent Mac Developer” and to practically give away the software… well, that’s just a fucking insult to me and all the other hard working developers out there. – Gus Mueller

My understanding is that the developers taking part in the bundle are getting a flat rate for participating. That means that the more bundles MacHeist sells, the more money MacHeist makes, while the developers will get no additional money. Each new user adds support costs, so the more bundles they sell, the worse off each developer may be. – Paul @ Rogue Amoeba

The argument is not that MacHeist are being underhand. Rather it is that the developers are only making a fixed fee and MH are making a bundle on the back of them.

So are the developers idiots? Have they been conned? If you read Gus Mueller’s fiscal breakdown, you see that MacHeist are canny businessmen if nothing else. Here’s some of the comments from the developers who got involved:

Gus has strong opinions and I love him for that, but none of us who are bundled with MacHeist were forced to do so; we knew ahead of time what the price would be and how much we’d get, and we decided it was worth it for us.

I think events like this get a lot of publicity, so they bring in new customers that I wouldn’t reach on my own. So I’m not really sabotaging my sales; I’m supplementing them. Seriously, if you came to me and said, “I’m going to resell Delicious Library to customers on the moon, who you’ve never met and can’t reach, for $1 a copy,” I’d say, “Go for it!” I don’t care if I only get a penny if it’s a penny more than I would have gotten on my own. – Wil Shipley of Delicious Library

Let’s see, if we sold 2,000 copies of FotoMagico in one day, we would have made $160,000. That would have been really cool. But usually we don’t sell as many. In fact, I think most of the 2,000 people who bought the MacHeist bundle on this first day did not buy it for FotoMagico and would never have bought FotoMagico separately. Many of them probably will never use it. – Oliver Briedenbach of FotoMagic

While MacHeist may be taking a bigger cut of the profit than the ten developers involved, I actually think the guys at Macheist deserve it. It takes a lot of time and effort to set up something as slick as this.

You also have to remember that MacHeist is only going to get a single hit of cash from this project, where as the developers will be reaping the benefits for a long time to come, paid upgrades anyone?

A lot of noise has been made about the cost of supporting all these new users. But, I don’t think that argument holds up very well, especially seeing as we’ve only had two support e-mails from MacHeist customers.

Also, the fact that MacHeist has raised nearly $50,000 for charity and they still have 3 days to go shouldn’t be overlooked. I think this is just incredible! If they continue at this rate they may even hit $100,000 for charity. If I knew they’d raised that much I’d have probably put RapidWeaver in for free! – RealMac blog - developers of RapidWeaver

So what to make of it all? To my mind, the developers have hit a master stroke with this. They realised this was an opportunity to do some marketing and get paid for it, rather than the usual trick of paying to get exposure. Thousands of downloads have happened, and their apps are now on the hard drives of people who had never heard of them before. The legacy of being involved is going to be measured in the tens of thousands of dollars, so what if MacHeist walked away with a bundle of cash up front? When was the last time a PR agency offered to give you $5,000 in exchange for putting your software in front of thousands of users? If somebody offered me, or indeed any developer I know, that deal it would take approximately 0.001 second to bite their hand off.

License keys and software are bits and bytes. The cost of production of 10,000 copies is pretty much the same as for 1,000 copies, so if you’ve only sold 1,000 copies at full price, and 9,000 people are walking around unprepared to pay full price for your software, how much would you pay to get those 9,000 people to pay $1 each? What if it didn’t cost you anything? Would you take the deal?

If you answered ‘no’, you don’t deserve to be in the software business. If you already are, I fully expect you to go bankrupt in the next five years, because you evidently don’t grasp the realities of software distribution in the early 21st-century.

You may be sceptical - will it really work, you may wonder?

Well, I’m a customer. Here’s the apps I bought, what I know of them before downloading them, and what my impressions are now. If a developer of a package reads this, they might get an idea of what the other users are thinking of this great little deal, and how they might build on the legacy 10,000 of us have just paid to give them:

• Delicious Library - When this app came out ages ago, I thought “so what?”. To me, a personal library management system is a little, well, anal. I don’t own an iSight, so the cool barcode scanning stuff is lost on me, too.

  I opened up the app last night though, and started typing in a couple of ISBN numbers of books on my desk. All of a sudden, it made sense - I could catalogue the huge pile of books I had in a corner, and get up-to-date market prices for them from Amazon, export lists for friends if they want to borrow them, you name it. I would never, ever have paid $40 for this, but I’m glad I now have it. If upgrades cost less than $20/year, I may be in.

• DEVONthink - I’ve been planning to buy this for a while. Playing with it last night, I got to build a few interesting setups to help with product development, and I expect I will be pushing it to its limits soon. I realise even now that in a month or so, I may need to upgrade to the Pro or Office edition. They’ve got a customer out of me instead of a procrastinator.

• Disco - Really don’t see the point, but I don’t burn a lot of discs and my graphics card isn’t up to showing the ‘smoke’ effect. I’m sure it’s brilliant, but this is an app I knew I wouldn’t need, don’t need and won’t need. That said, I’m now aware of the developer, and they’re in my RSS feeds, so they now have my eyeballs for the time being, which is a powerful thing to have. I am somebody with a mac and disposable income, so the developers should be glad they got paid to get my attention rather than the other way around.

• FotoMagico - I already have Keynote even though I don’t do many presentations, but I can already see ways this is better. What’s more, where Keynote is built for purely presentations, FotoMagico looks like a better fit for a couple of simple training presentations I have lined up in the New Year. A real gem of a find in the bundle, for me.

• Enigmo 2 - Never heard of it, never seen it, never thought about it. I’m now an addict. They have me as a customer for the long haul, I’ll be in the queue for Enigmo 3 if it ever exists, and I’ll look at all the other titles they release. Great coding, great artwork, great idea.

• iClip - This is one of those apps where you think “I’m not paying $30 for that!” but I’ve used it 7 times just in the production of this blog article. Fantastic little tool, and I can quickly see it being a tool I won’t be able to live without in the very near future.

• NewsFire - I currently use Bloglines as my RSS reader, but have started getting quite hacked off by it recently. For me at least, Bloglines is buggy at times. This looks interesting, and I’m going to give it a run at my huge OPML file in the next few weeks and see where I end up. It may just be they have a convert, but this is the kind of app I need to spend weeks, if not months, with before making a call.

• RapidWeaver - I saw a screenshot of this a while back, and remember wishing “if it was $10 cheaper, I’d probably buy that”. I regularly have to put together small websites that I can’t be bothered with coding up in HTML, I need something better than OSWD.org, and iWeb sucks. As a simple little tool, this should save me time.

  What’s really interesting though, is when I started digging into theme development, I realised that I could use it as a tool for building mock-ups and views for my Rails apps as well, and building themes looks fun. It may very well be that the site re-design for vagueware.com happens in RapidWeaver, because it knows CSS and XHTML way better than I do, and is the first tool I’ve seen to give me the flexibility I need. Throw into the mix I’m going to be buying upgrades and maybe even theme packs in future, this is a good deal for the developer.

• ShapeShifter - I hate UI tweak programmes, because most of the themes produced aren’t actually very easy on the eye for the kind of 12-14 hour days I pull in front of screen. I found a couple of themes for ShapeShifter that are nice on the eye and more comfortable than Aqua, to me at least. I would never have tried this software unless it was in the bundle, but from now on I think they have me as an upgrade customer.

• TextMate - I’ve been using TM for around 9-10 months now. It is not the be-all and end-all everybody seems to make of it, but it’s a daily companion in my work and I’m grateful for its simplicity, power and flexibility. A spare license for the new machine after New Year is a boon.

Whilst I talk about the new hardware, it’s worth pointing out that iClip, DEVONthink, Pangea (Enigmo) and RealMacSoftware (RapidWeaver) are going to be seeing my credit card once more as I buy second licenses for a second machine.

As a marketing exercise, it’s genius to my mind. If somebody offered to give me money for customer exposure, I’d be grateful for it. I think my comments above show that for most of the developers, it has worked in that spirit for me at least very well - I would never have bought their apps before, and now I’m keen on their code. Instead of paying money for clicks, advertising, sponsorship, whatever, they have stumbled on a way of making a few thousand dollars whilst carrying out this marketing exercise and in the long run it is going to make them a lot of money, in my opinion.

Exposure in this case has been far more profitable than trying to make sure every license is paid for at full price, which is of course the underlying argument for giving away books, blog posts, podcasts and software of all flavours: exposure is a better commodity than cashflow when you’re planning on being around for the long haul. In a World where ideas and software are cheap to move around, ideas like this show us the future of marketing and it is a remarkable business model.

As for the cynics, well, they’re either jealous and being sour, or they really are just too stupid to get the fiscal advantage. I hope they change their minds, and I can’t wait for the next MacHeist bundle.

I picked up an article doing the rounds by Joel Spolsky on context switching in Agile development and find myself a bit confused by the way he pitches his conclusion, even though I agree with most of what he says.

Context switching is something programmers first learn about in University purely in terms of what happens when a CPU or operating system has to stop doing what it’s doing, and do something else for a bit. It happens thousands of times a second on a modern computer, and is happening on your machine right now if you have more than one process running. Getting processors to the point where they can do that effortlessly took decades.

The next thing programmers learn about context switching is that they are expected to behave just as well as a modern CPU at context switching when they are expected to juggle dozens of projects at once. Interestingly, the people who expect this, don’t understand what is required to handle a context switch for a programmer and see their new piece of meat as being a bit like a messy version of OS/2 Warp, but perhaps not quite as fast.

In the past I’ve written about what I call alone time - the point where a programmer will try and get into “The Zone” and become as productive as they possibly can. I think those two articles hint at why a context switch is so hard. It disrupts a work-flow that is hard to obtain. You can’t easily drift in and out of a context when you’re being productive.

Joel agrees with all of this, but says “hey, you know, sales are important, sometimes it’s worth the context switch, that’s what Agile development is all about”. The problem is, he’s talking complete rubbish.

Agile is about how to manage the development of a single project: one project on the radar, iterations, feedback, customer contact, the code is the specification. You get the idea.

You don’t use Agile to handle multiple projects. For that you need something completely different. The context switch is something that agile isn’t designed to handle, because it assumes that as a development manager you wouldn’t be so utterly irresponsible, moronic and boneheaded to attempt to try and get a developer handling several projects at once. It’s a limitation of Agile, which is why you need something else in most programming teams to handle it.

Right now I have seven projects on the board, two of them able to wait a while, three of them less than 24 hours from being signed off, and the remaining two due to last through for another week or two - I then intend to take a month off Christmas and sleep. Right now, I need to context switch even though I know it’s not ideal. The way I do that has nothing to do with Agile development, and everything to do with how I plan my time.

In the example Joel points to, the answer is not “this is Agile, deal with it, be Agile”, the answer is to say to the developer “look, do you want some overtime to look at this?” or to find a developer who isn’t busy to look at it instead. It’s about time management and handling more than one project, not about forcing a method designed for one project into something it isn’t.

Juggling multiple projects, handling resource allocation and managing the board in a busy programming team is hard work. It isn’t something Agile can fix, so don’t blame it when it doesn’t, and let programmers stay in context as long as possible if you want them to be productive.

I’m just about coping by handling this my own way, but in the New Year I’m hoping to hire a developer to help me out and let me stay in context for longer.

I have recently decided that to become a better developer, I need to learn how to play chess properly. This might sound eccentric, but let me explain.

I know how to play chess in the same way most of us learnt how to play the guitar as teens - we knew where to put our fingers and strum along, not really attempting to get to grips with the underlying nature of the instrument. In other words I know the names and legal moves of all the pieces, I know the middle of the board if often the most important part to dominate early in the game, and that’s about it.

I am a truly terrible chess player, because I lack the ability to remember techniques and tactics, and I don’t have the patience to understand how to play an over-arching strategy. I just move pieces around the board in a way that seems defensive or attacking depending on where my opponent is in their game.

If I’m honest with myself, quite a large amount of my development time can feel like this as well - I know how to code, I have a general idea of what is sane and what isn’t, but I lack the inner development to be able to see project as a strategic objective with each method and line of code another tactic in the battle.

Battle? Who’s the opponent? Again, here I need to be honest with myself, because whilst customers changing requirements, project managers not knowing what is going on and flaky code on the production servers can be an enemy to productive coding, I have to admit I am my own worst enemy. I know it’s easier not to write the tests, not to refactor my code, not to think if I could replace that twenty lines of code with three, not to go back and make sure that my tests covered every possible fubar the users could throw at it, not to comment code properly, etc.

I hope by encouraging a strategic and tactical perspective in my work, my code will get better. Chess is great at training the brain to think in ways I think will be useful to me in my work. Not convinced? Well after just a half hour of reading up on basic chess strategy and tactics, I’ve come to the following conclusions about how to improve my day-to-day developmet:

• My language and development tools are my ‘material advantage’ in chess terms. Ruby - my language of choice these days - allows me to express ideas more easily than other langagues, and in less code. Rails gives me a lot of stuff for free. Unix does too. These are all advantages my ‘opponent’ does not have - it is often easier to write good code than it is to write bad code.
• Automated testing can be my defence. They give me confidence that code is generally correct and allow me to move forward safe in the knowledge that as I progress I’ve got those pieces at the back there covered.
• Documentation is like a ‘fork’ - I can challenge my understanding now and deal with my lack of understanding 12 months from now by making sure code is commented properly.
• If I encourage myself to think three, four, five moves ahead and think about what my ‘opponent’ might do as counter-attack, that’s going to make me both a better chess player, but a better developer too. Right now I spend way too much time thinking about the problem in front of me and maybe the problem coming after that one.

This might feel like I’m stretching an analogy, but really I’m just trying to stretch my brain to thinking in new ways. If I’m able to develop my skills as a chess player, I can’t see the harm it would do to other areas of my life.

”It is true some rather interesting projects get abandoned by their developers or shall I say their artists. Most of them are sculptures made out of C (or any other language). And this is the biggest handicap of those programs. They aren’t engineered they get crafted.” - Reiner on www.gnomedesktop.org

When I was 11, I wandered into the school library and on the shelves found a book called “Learn to program BASIC on the BBC Micro”. It was about 60 pages long, and most of each page was covered with pictures of robots telling me what ‘PRINT’ did. Coincidentally the time of year was Lent, and it being a Catholic school I had the option of foregoing lunch and instead donating 40 pence to CAFOD to use the computer room for ‘personal projects’. This was how I began my study. It quickly became an obsession.

By the time I got to 14, I had discovered a few things:

• I couldn’t afford a real computer, but it didn’t matter. This was 1992 and yet whilst my richer friends were playing with Amiga 500 and Atart ST machines, I was stuck with an Amstrad CPC 6128 my father had kindly managed to donate. This was where I learnt the true value of engineering within constraints.
• I didn’t want to study Computer Science. This, I still believe was the right decision. However, my failing was thinking that I wanted to study Software Engineering. I didn’t want to be an academic trying to understand the philosophy of bubble sorting - I wanted to change the World and write software that would help run The Future, and I figured Software Engineering would give that to me.
• My school teachers had realised I was the best person to call when a computer broke somewhere in the building, and as a result, I knew I would probably be able to make a living out of my obsession if I needed to. Knowing that you can make a living with the skills you have when you’re 14 is… empowering.

I set my course. I worked just about hard enough (but no harder) to get to UMIST - now merged into Manchester University a campus I still live on the edge of - and studied Software Engineering.

Actually, that’s a lie.

I was on the course, sure, but I chose to get drunk most of the time instead. See, the problem was, fine, I could learn formal specification and learn how to build a compiler, and that would all be swell, but I now realised it wasn’t going to change anything. I was not going to be able to express myself the way I wanted to if I was confined by determining the correctness of my code mathematically. I thought Software Engineering was more concrete than Computer Science, but in actual fact it was just the same abstract notions but sat in front of a screen and expressing them via C instead of in the library expressing them via pseudo-code.

What I then realised is something I should have had the guidance of when I was 14: I didn’t really want to be an engineer. Engineers do what they’re told, follow a plan, make sure the foundations are set correctly and regulations are followed.

I wanted to be an architect, a craftsman, an artisan. The depression of the realisation that I had taken the wrong course actually drove me to the point of barely turning up for class at all. I ended up taking on a variety of full-time jobs whilst attending exams so I at least came out with a piece of paper. I also spent a lot of time getting drunk and playing pool.

This conflict still continues inside me - sometimes I feel I want to be the engineer, building automated tests, writing out my code in formal proofs, knowing that my code is correct. Other times I want to be the scupltor, using the IDE like a blank slab of marble or a lump of clay awaiting moulding. Some of my best programming has come from just writing a line that kind of looks like it might be the core of what I wanted to do and then wrapping it in other code that made it work. That’s hardly the way you would want a fly-by-wire or power-station safety system to be written, but it is immensely satisfying. It’s fun. It’s art.

So now I look at these formal methodologies, whether they be more traditional lifecycle-led with their heavy requirements capture stage at the front, or if they’re agile, and I think “what a bore”. In some ways I’m drawn more to agile methodologies not because they are more productive, but because they are more fun. They are there to provide confidence to the client, sure, but they allow the artisan programmer out, free to roam.

It feels wrong though - when I go out roaming, I feel as though I am not a professional programmer. All of a sudden I feel I am an artist playing amateur programmer, and I live in fear of being discovered and found out. I constrain myself with project management tools and development methodologies, not because I think they do me any good, but because they stop me feeling guilty.

You know how I spend most of my day? I write a lot of Ruby, specifically Ruby on Rails. I reckon I must spend about 20%-25% of my time looking things up in the API docs for either Ruby or Rails. Do I know how to code in them? Sure I do. I just don’t bother memorising the API. I just find the bit I need, the syntax and method parameters that need to be called right there and then. With time some methods become second-nature, but I don’t actively memorise this stuff. I’m too busy coming up with my own ideas to remember other people’s. How un-engineer-like is that? How skittish and flaky is it? I don’t think I even deserve to be called a programmer if I do things like that.

But I bet I have more fun than most engineers. Because as the code comes out, at the end, it feels like a piece of art is being formed. Sure it might be a reporting module that spits out graphs and pie charts, but I know the inner beauty of that little bit of optomisation that made it query the database 12 times rather than 800 times. I know how I flitted from the model, to the view, to the controller, hoisting each little function up like a piece of scaffolding. I know that before I started there was nothing, and now there is code, there is function, there is value. The World is very slightly different for the users.

If I could go back and give advice to my 14-year old self on how to become a professional software engineer, you know what I’d tell him? Go and study Art or Philosophy. It will do more for your ability to create than any Computer Science course ever will - and you can learn all the CS stuff from Knuth anyway, if you need to. I’d tell him that the true beauty of this profession is that it has the advantages of being a writer, a musician or artist except it’s far more profitable. I’d point out that understanding how code works isn’t as important as understanding how you work.

I’d also probably tell him which girls secretly fancied him and to stay off the beer when he got to Uni, but that would spoil the journey of excrutiating self-discovery. :-)

So I’m keen to learn: am I alone in this industry? Should I be stripped of the title ‘Programmer’ or even more laughably ‘Software Engineer’ and just be known as ‘that guy who does that stuff, with you know, the computers and stuff’? Is the real motivation behind the uptake of Agile not the fact it makes people more productive, but it makes people feel more natural, more like artists? What would you tell the younger you about how to harness ‘the art’ or the ‘the science’ of programming?

Just noticed that if you’re an agile developer (like me) you and you’re in the North West of England (like me) you may want to go to the AgileNorth event on the 20th September. It costs £95, and the programme looks pretty reasonable.

Alas, I’ve only just found out and I’m already up for being somewhere else that day, but I’ll hope to see feedback around the blogs from those people going.

I care a lot about innovation and creative thinking. It’s one of the things that inspired me to learn how to write software when I was just 11 years old: if you can imagine it and it can be turned into a process a computer can run, you can create it. The potential has always been amazing, but now with tools like Ruby on Rails, the idea-to-product lead time is shorter than ever before.

The problem is, most people are awful at creative thinking and forming new ideas. In the UK we seem to be particularly bad at this, maybe because we tend to be more self-concious than many other cultures about looking foolish, or being mocked. However, it’s preventing so many businesses here from flourishing that it drives me crazy. Here’s my take on it, and how I go about coming up with hundreds of new ideas a week (note: coming up with hundreds of ideas is a good thing, acting on them is something completely different).

When traditional business consultants talk about innovation they use phrases like “blue sky thinking” or they may encourage thinking “outside of the box”. It is often thought that if we are constrained by how things are done now, the best way to be creative and innovative is to throw away everything and start from thin air.

It sounds good, and it has a catchy ring to it, but it’s flawed. It’s no surprise that the only original thought these consultants had was to become a business consultant and take your money off you for repeating clichés. They are effectively selling snake-oil, and that’s hardly an original idea.

Innovation does not come from the kind of freedom that means starting with nothing. It comes from changing the boundaries of what you already have, to evolve rather than throw everything away. Throwing everything away might be the idea you come up with at the end of the process, but it’s not how you should start.

A writer’s worst nightmare is often the blank page. A blank page represents the consultant’s “blue sky” - you can put anything on a blank page. This can cause your brain to ‘freeze’ if you don’t already know what is going to happen next. Writer’s block is caused by being able to do anything, not by being unable to do anything.

You need something to work with - you need to give yourself a boundary. One exercise to remove writer’s block is to write one word - any word, just a word - in the middle of the page. Then think about another word that would go with it, contradict it, oppose it, complement it, whatever. Soon, you find your brain giving you ideas, and off you go. This is giving your brain a set of boundaries, something to work with.

Comedy writers know this better than anybody. They have to, because writing good comedy is harder than any other type of writing (trust me, I’ve tried). Take sitcoms: the ‘sit’ is obviously short for ‘situation’ and if you watch a sitcom closely you’ll often find the really funny bits come from a set of situations rather than some cheesy lines. However, there is more to it than that.

Every sitcom starts with a context. In fact, all good comedy is often about the limits of something - a coffee shop, a relationship, a bunch of weird friends, the lead’s self-doubt. Think about all your favourite shows, and you’ll realise that the first thing the writers will have done is define the situation and boundaries the comedy will be written within.

Friends: group of 20-somethings who are close friends dealing with life in modern day New York.

Yes, Minister: the battle between the politician with a desire to do something worthwhile, and the civil service who desire nothing at all to happen if possible.

The Office: the excruciating banality and confinement of working with somebody who doesn’t understand reality.

If I asked you to write a sicom from scratch, you’d find it hard unless you had already defined a set of boundaries like those above. Get the idea? Still don’t believe me? OK, try this:

Write a new joke right now. Come up with something original, a joke the World has never heard before. It can be about anything.

I bet after a few minutes you’re finding it hard. Where do you start?

OK, I’ll help you. The joke has to involve a monk and an elephant. Easier? Probably. If you’re still struggling, I’ll give you a push and suggest you must use a play on the word ‘habit’ as in an item of monk’s clothing and being an established custom. Got a joke yet? Your brain is probably finding it easier now to come up with something funny.

This is what comedy writers do - they find some boundaries, and twist around inside them. It’s not just comedy either.

So how does this relate to technology and software? Well, firstly, it explains why all commercially successful technical advancement comes ultimately from evolution of ideas rather than revolutions.

People often resist revolutions - they need to find a way of relating to something via an evolutionary step. Early-adopters are not always grabbing hold of the next revolution: it’s because they’re always adopting early, that the stuff they do now is an evolution of where they were 6 months ago. We’ll all catch them up in a couple of years when our evolutionary path catches up, but to us it might look like a revolution, but that’s just because we’re being slow.

The evolutionary path we follow in our uptake of ideas is merely us accepting the twisting of the boundaries we’re comfortable with. Some people for example, will have produced a joke about the monk and the elephant that others will have found obscure, obscene or just outright unfunny. But that’s because the way you shape your box and then push the edges is different to how other people shape their box and push the edges. This is the core of all technical advancement, selling new technologies, and ultimately, creative commercially successful innovation.

People talk about the industrial “revolution”, but really it was an evolution of ideas on how to make spinning cotton more productive, mixed with the idea that a steam engine was more efficient than a horse (and didn’t need as much care and handling), mixed again with the idea that selling goods all over the World from a factory in the North of England made more money than selling it all down the local market. Evolution, after evolution, after evolution. It took decades, a lot of pain and failure along the way, and was all driven by people who liked twisting the rules inside a set of boundaries.

It took special circumstances, and a group of people to take a series of steps that others found uncomfortable.

The other reason this relates to technology, is that if we know that innovation is evolution, not revolution, and we know that giving ourselves boundaries to push is the way to be creative, we can use that to produce something valuable. Remember at the start when I said there was no other field more open to new ideas being created than computing? Once you formulate how to be creative, it opens up the playing field completely - you can become a money machine.

Want to come up with a new super-cool Web 2.0 website? Fine, let’s see - it needs to be based on communities, it needs to feel like a desktop app, and ideally it should get better the more people use it. That’s a good start, but I bet you’re still struggling to come up with dozens of ideas. Problem is, they’re all technical boundaries, and what we want is a business boundary - we want to generate new businesses, not new technologies, so we need to focus on the right box.

That said, trying to build a business with no thought to what is possible with the technology is a dumb move, and sometimes taking an old business idea and updating it with the latest tech (cf. Shopify) can produce a great business. However, for this example let’s add another arbitrary business bounday: it needs to be based on an e-Commerce model - people should use the site to buy things.

Right now, your brain might be starting to kick into gear and you’re coming up with ideas. Chances are, if you’re still concentrating on the technical boundaries they’re just re-hashing what has been done before - adding Ajax to OScommerce clones or something - and that’s no good: we want to get a bunch of really original ideas.

First thing we could do is say we’ll do the exact opposite of what everybody else does. So let’s do a web-site that when you visit it, it’s full of people wanting to buy things, and you can sell them what they need - the 180 degree turn-around. That’s an interesting and innovative twist.

We could just as easily brainstorm how to make something bigger, smaller, easier, harder, cooler, less cool, more popular, more niche, more intelligent, less intelligent, and so on. Look at every feature of eBay or Shopify and ask yourself “How would I do that differently?”

If we built a site where instead of a list of products, it was a list of wanted ads, we could then look at it again from fresh and find an evolutionary step from there. It may be that what we actually do is aggregate specially-tagged content from people’s blogs and allow marketers to find what bloggers are after, rather than hosting the ads ourselves. We may decide to produce a version that is highly targets to a very small niche. We might want to make it something well suited to market researchers trying to produce the Big Toy This Christmas.

Then, once we have those new boundaries, we start to twist within them and we may end up turning out a whole bunch of new ideas.

When you see the World through a lens that makes every product feature a boundary to be pushed, twisted and warped, creative thinking becomes infectious. And in the new World of software development, we need all the creative thinking we can get.

These are just some of my thoughts that I’ll be expanding on in the coming weeks - I’d be really keen to hear your ideas in the comments section.

One of the worst bits about being a developer is having to come up with documentation. There are great tools to help any coder who comments their code (fewer than you think actually do this) automatically generate programmer’s documentation, but what about user docs? I hate dealing with user docs. As a result, they often suck.

It has been suggested that user manuals need a make-over and whilst I understand the point that is being made, I think we can do something even better. Rather than spend all that money on producing a beautiful looking user manual that marketing got to, how about get rid of the manual completely?

It’s no secret that a lot of the new start-ups have been inspired by the wonderous 37signals philosophy, and in particular the ideas condensed into Getting Real. To my mind, if we follow those philosophies, we should be looking at producing systems that don’t need manuals. If you can’t look at a web application or a camera and work out intuitively how it works, it’s the wrong interface.

When you buy a new digital camera, what’s the first thing you do with it? I take a picture. I don’t care if the exposure is wrong, that it’s writing to the internal memory instead of the memory stick, that the flash isn’t turned on, etc. Once I know how to take a photo with it, I’m ready to start exploring, and most cameras these days have a great interface via their on-screen display. I don’t want to get my new toy and spend a few hours flicking through a manual just to work out how to turn it on.

This philosophy - the simplest interface possible - means we eradicate the problem known as “Stuck in P-mode” that blights so many users. But what if you have to produce an interface that needs to be a bit more ‘condensed’ than a carefully thought-out OSD menu? What if you want to embrace the expert user who wants the short-cuts and so on?

I’d still say the interface is wrong, but if you have to, when the customer buys it, you should be offering that user free training at a session near where they live. If they live in the middle of nowhere, let them have the training online - let them interact with a live session via the web. They shouldn’t have to go out and search for extra training, they should get it from the start from you, the supplier. Once that happens, users will happily start to talk about you and your product, because you are no longer selling them a gadget - you’re selling them a skillset. That’s a powerful marketing device.