Innovation in Software

No matter. Try again. Fail again. Fail better.
– Samuel Beckett

Whenever I am asked by a client why Agile Development is a good idea, why TDD or BDD is the way to build a product, I can sum up the argument pretty succinctly: it reduces the cost of change to as close to zero as possible.

Recently I have been doing some reading around “Lean Development”. I have found Eric Ries’ Lessons Learned blog an invaluable resource for finding jumping off points into resources and ideas that are still very new and fresh to me. This morning whilst leafing through my new copy of The Principles of Product Development Flow, which I bought after seeing it recommended by Eric a few weeks ago, I realised I could succinctly reduce the arguments for lean development down to another mantra:

Lean Development reduces the cost of failure to as close to zero as possible

It doesn’t sound like much of an insight, does it. In fact, everything you read about this area is about testing Potentially Fatal Assumptions (PFAs), and market-validating feature sets: get your product out of the door, don’t invest too much in case it fails, if it succeeds then well done for lucking out. And of course, lean development is not just about failing at things – it’s about reducing the cost and time to delivery whilst also focusing on the only factor that really counts in product development: profitability.

However, when you let the idea about cheap failure seep in, you realise it challenges deep-rooted beliefs held in our society.

From the nursery school chart of “gold stars” all the way through our continuous grading of achievement in our first 20 or so years of life, most of us have learned through trial and error that failure is bad. Failure is (literally), for losers. Failure is not something you want associated with you, your ideas, your relationships or your life. Failure, we believe, is bad.

I think we’ve taken a wrong turn somewhere. Providing that the cost of failure is low – in terms of capital, resource, emotional investment, etc. – failure is useful. Failing quickly at something that was always going to fail, allows us to direct our attention to something new.

Our lives on this planet are short. We can only fit in so many projects, businesses and relationships in our average 70+ years on this planet. Why on earth should we invest a sizeable percentage of that precious fleeting existence on something that is doomed to failure from the outset? Why do we fight on investing more emotional and financial resource into something that will never ultimately work?

So, test for failure early, quickly, and cheaply. Sure, you can try and work out whether you need to riff through something because you’re in The Dip and that might work for you. What I’m saying though is that when you fail at something quickly and cheaply, you have opened up your life to a new possibility, a new project or idea that isn’t flawed and will be successful.

I’m not saying you should aim for failure. I’m saying that if something is fatally flawed, find out quickly, and rejoice. If it’s a success, well, brilliant: that was the point all along.

In project terms, I’ve failed plenty of times. My only regret is that I didn’t fail more cheaply. I invested so much emotionally, physically and financially that my resources were depleted for the next project, which in turn may have caused failure where there wasn’t a fatal flaw. From now on, failing early and failing often will be a sign for me that I’m getting better at understanding fatal assumptions. One day I will run into an idea that thanks to my cumulative experience of flaws will have none that effect its profitability, and it will hopefully succeed in ways I can not yet imagine.

When you realise that something is failing, resist the urge you’ve had all your life to try and “turn it around”. Find the PFA that is no longer just a potential, learn from it, and get on to the next project. Quickly.

Footnote: I remembered whilst writing this post, a friend who got divorced less than a year after getting married. He and his wife found that their attitudes to starting a family had significantly diverged, and so realised (and I quote), “it’s best to just call it a day early, otherwise we could lose years of our lives, wasting them on a relationship that was never going to last long-term” (emphasis mine). That took some guts, however their lives will be the better for admitting the fatal assumption that they both wanted the same things going forward, and they will reap the benefits of not sinking so much of their lives into something fatally flawed. If they can be so mature and pragmatic about something so emotionally raw, I’m sure you can be about your next idea for an iPhone app nobody cares about.

[Up-front Disclaimer: I didn't do Computer Science at University. I did "Software Engineering". It involved formal methods and quite a lot of Z notation. I have some pretty strong views on Software Engineering as a discipline as a result. Most of my CompSci colleagues generally do not share my beliefs.]

Johannes Ernst posted yesterday a pointer to an article from IEEE Software by Tom DeMarco, pronouncing Software Engineering as “an idea whose time has come and gone”. Tom’s argument can basically be summed up as:

[...] you need to distinguish between two drastically different kinds of projects:

• Project A will eventually cost about a million dollars and produce value of around $1.1 million
• Project B will eventually cost about a million dollars and produce value of more than $50 million

What’s immediately apparent is that control is really important for Project A but almost not at all important for Project B. This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.

To my mind, the question that’s much more important than how to control a software project is, why on earth are we doing so many projects that deliver such marginal value?

Can I really be saying that it’s OK to run projects without control or with relatively little control? Almost. I’m suggesting that first we need to select projects where precise control won’t matter so much. Then we need to reduce our expectations for exactly how much we’re going to be able to control them, no matter how assiduously we apply ourselves to control.”

Frankly, I can’t believe such a narrow and misguided argument has come from the pen of Tom DeMarco.

Imagine if you will the next time I go into tender on a project.

Customer: So how much will this project cost to deliver, and how long will it take?
Me: Well, how much value is it going to add to your business?
Customer: Pardon?
Me: Is it going to add a little bit of value to your business, or a lot?
Customer: A lot, I expect.
Me: In that case, I have no idea. Let’s just roll with it and see where it goes, eh?

I’m not convinced that’ll go down well.

There are a few logical mistakes in Tom’s argument (and I can’t believe I’m having to point them out), that basically means he’s talking a great big pile of crap:

1. We can not assess the true value of software until it is shipped.

   We might be able to sense the potential value, but it’s a guess.

   I have one project on my books now with a mid-sized build budget, potential value in the hundreds of millions. Another project for a smaller build budget with a potential value of £200k-£300k. Another project with a tiny build budget that if it goes viral could be worth billions five years from now.

   Or maybe the project with the tiny build budget sinks, the medium project gets on NASDAQ in 5 years and the biggest project eeks out a living but never goes stellar.

   We simply don’t know. We can’t know. We’re guessing. There are too many variables outside of the software engineering process to be able to assess the potential value accurately.

   In essence, we have to treat all projects as being potentially of very high value, because they all are.

2. Value is potentially nearly unlimited, budgets are not.

   It might be well and dandy to say that it doesn’t matter if the budget might slip because the value is so high, but budgets are fixed.

   Thousands of software builds are being worked on right now where a 5% budget slip is potentially going to kill the project because there just isn’t more money after the budget is gone, and without being feature-complete the software is going to have near-zero value.

   Yes, incremental development is the way to do things, yes we should be able to stop and ship after each day’s work. However to fulfil the project’s objectives (and maximise its potential returnable value), we must aim to get all the features in to the final build within budget.

3. There are other factors beyond budget, schedule and value to control.

   John Glenn once quipped about being sat on the launchpad, “I felt about as good as anybody would, sitting in a capsule on top of a rocket that were both built by the lowest bidder.

   However, whilst budget was important, he shouldn’t have felt in danger: the political ramifications of a death would have meant that the people who built that rocket and capsule were going to make sure he survived.

   And some software systems are the same. What is the “value” of a nuclear power station control system? Or the control systems in a fly-by-wire aircraft? Do we care more about the financial value or the safety value? I would suggest that controlling the build – engineering the build – in these situations is critical because the safety value is very high.

   In fact, it betrays immediately the fallacy of Tom’s that only the marginal projects need tight controlling – if you’re not controlling the processes around a software build that keeps 400 souls alive at 35,000 feet, you really should be in another profession.

There is a lot of cruft in software engineering. Waterfall methodologies are now considered almost laughable. However, it’s a young discipline and we have barely begun to understand the correct way to engineer software.

Agile philosophies are taking over in development shops because it allows for less process, more communication and ultimately better software being produced at the other end.

Behaviour-Driven and Test-Driven Development methodologies are providing useful benefits to our clients in a way we could barely imagine just a couple of years ago. In certain situations, formal methods and algebraic proofs are confirming that software is “correct” and “complete” in such a way we can literally bet our lives on it.

Tom admits himself he’s not on the front-line of software development these days. Perhaps his metric-centric universe is correctly condemned, but most of us aren’t using those methods anyway. We’re looking at perhaps a few numbers instead: test coverage, remaining days in budget/schedule, remaining features to be implemented. We’ve moved on. Perhaps he just doesn’t get what we’re doing in dev shops these days.

And we’re still learning new techniques, new ways of doing less process, more quality software. Software Engineering is definitely not something that has “come and gone”: it has barely got its foot in the door.

Towards the end though, Tom does make one point I can’t agree with more:

Consistency and predictibility are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

This reminded me of a talk given by Ted Nelson at OpenTech 2005 where at one point he almost screamed:

I didn’t get into computers to automate trivial crap! I got into it to change the World!

Didn’t we all? But I think we can do it without having to just free-ride our way along with our fingers crossed, hoping we don’t run out of money.

Excuse the self-indulgent nature of this post, but I think I’ve been quite restrained on this for the last 12 months and hey – it’s Christmas.

At this time of year, two things always happen to me:

1. I get a little bit melancholic. My playlist changes to have lots of big sweeping chords (think ‘Atmosphere’ by Joy Division, some Band of Horses, some Vaughan Williams). I watch films about people enjoying a simple life such as Harvey or El Perro last night, and think about de-cluttering my own.

2. I start thinking about why I do this for a living all over again.

2007 has been a pretty good year to me. I’ve written about 10,000 lines of commercially deployed code which is a lot. I’ve written several million words in blog posts, e-mails and essays which for me is average. I have about 40 business cards I didn’t have at the start of the year of people I will undoubtedly work with at some point. I’ve also undoubtedly upset, angered or annoyed people unwittingly (sorry!). I’ve almost certainly got things wrong.

Still, now we’re moving to the point where I ask what it is I do, why I do it, and how. Every 12 months I ask the question about whether I want to hire staff, whether I want to go big or stay small. Whether I need to start phoning some of the VCs in that little stack of business cards. Whether I should just go and get a job and not worry about money again for a while.

Getting rid of all the clutter and thinking about what it is I do, I realised whilst reading this article this morning that there is a vision out that there that is so pure in it’s concentrated common sense that at some point this year I once again stopped thinking about doing good stuff quickly and just started doing stuff that paid the bills.

Make it free. Make it simple. Make it open. That was the plan in 2005, so what happened to it?

I read Getting Real when it was first released. I advocate and often use agile methodologies, but there’s something I’m missing. All my projects seem bigger than they need to be. Sounds like I have an agenda for 2008 forming already.

And this is the point. I can make those changes because I don’t work for somebody else. I can take those risks and make those things happen if that’s what I want to do. I can just get on with it.

In this article, I’ll be exploring some of the issues around creative thinking and software development. This is a theme I expect to be re-visiting a lot over the next few weeks, so these are more high-level riffs. Each of the major points here needs expanding upon, but I’m interested in seeing what people out there lock into as the most interesting aspects of this.

I believe that raw creativity is not normally associated with the development side of the software industry. This is an error that hampers development, innovation and even the core process of programming. Creative thinking should be taught to all programmers and embraced as a way of working.

Typically only the people involved in the user interface, information architecture and marketing stages of the development process get to do the “original vision thing”. Whilst this goes on, programmers are expected to behave like glorified plumbers by connecting interactions together with a logic that meets the spec and passes tests. This skews the development process towards the wrong people.

Project Managers and Information Architects are going to be aghast – Designers will be screaming – but it is the “Developers” who are the best placed to understand what is and isn’t possible. When asked to be involved in the earliest stages, or in stages they’re not normally invited into (yes, even marketing meetings) and encouraged to contribute to the creative process, several benefits are likely to occur:

1. The programmer becomes more involved in the project and is less likely to think about “those idiots” down the hall

2. The project itself will improve – programmers can often point out things that are possible that nobody else had even thought doable. Typically programmers are early-adopters and keen to produce things that impress. Mediocre software is always designed by marketing, great software by people who really understand what software can do when given a chance.

3. The project constraints will be better understood right from the word ‘go’ and proposals made by IAs or Designers who don’t understand the ‘code impact’ can be quickly managed and dealt with rather than getting in front of a client who signs off and expects to see it in the final shipped product.

Some programmers will protest, because they don’t want to go to meetings. This is because most meetings that pretend to be about original thinking aren’t anything of the sort. If you create an environment where genuine thoughtfulness, creativity and interesting ideas are produced and managed, programmers would have a different opinion of those meetings. The sad truth is, most people in the software industry who aren’t programmers are there because they aren’t very interested in this kind of thing. That’s one of the reasons why so many programmers are going it alone right now.

One argument from the other side is that programmers aren’t very good at creative thinking. Those people might be right, but that’s because they think about the process inaccurately.

The truth is, where most programmers learn their trade at first is in building algorithms. There isn’t much scope in the minds of most programmers for creativity at the algorithm level. They have been taught to look for patterns they’ve seen before – the reason why every undergraduate has to undergo at least one course in Data Structures & Algorithms during their college years – and code reuse is preferred over “reinventing the wheel”.

Knowing certain sort algorithms are better in some ways than others is certainly beneficial, as is code reuse. However, the idea that programming is something you can learn once and then just repeat over and over again is absurd.

This has been scaled up from the algorithm level into the project level over the years. It was the “building block” approach to programming: if you look at an algorithm and implement it like so, and it becomes efficient to remove creative thinking here, then it must scale up. The result is that programmers have been trained to think of projects at times in the same way they’ve been taught to think of algorithms: look for patterns, reuse where possible, don’t break the mold.

It is only in recent years that academics have even started considering teaching their students any other method than waterfall for managing a software project, and when a programmer is confronted with their team at the first job working in an agile fashion (if you can find me a team still dealing 100% with waterfall, I’ll show you an expensive team), they are going to find themselves ill-equipped for the average work day. They are going to have to find a way to keep their head above water, and what tends to happen is they hack it. They don’t have the tools – because they’ve been told they’ll never need them – to think creatively.

What’s more this problem is compounded by the fact that it’s no longer enough in an agile World for a programmer to just know how to write algorithms. All of a sudden, the information architect has become redundant as the software engineer takes the role of seeing how the whole system plugs together. There is even an argument that the information architect was always redundant.

Addressing these issues aren’t simple. If creativity is so important, why don’t more programmers engage with it?

Programmers are experts on thinking through complex problems. In theory, all that needs to be done is to teach creative and original thinking as thinking through a complex problem. Even the worst “creativity consultant” can tell you however that the enemy of creativity is habit, so they would argue this is likely to fail. The creative process looks so bizarre to an analytical mind specifically because it can’t be easily explained. This, however, is a misunderstanding.

The real issue with creative thinking is that many people are scared of failure. Programmers, especially so. This is because in real life, as in a software project, failure is expensive. Time lost is time lost – we fear doing something that we can’t take back. We look forward to our next actions in life through the lens of our past, trying to make sure we don’t make the same mistakes again. Programmers thrive on doing this – it’s what makes the analytical mind tick. This leads to unassertive caution and bashful timidity. We are too scared to put our hand up and say “let’s do something reckless” in a project meeting for exactly the same reason we won’t say it to a stranger in a bar we’re attracted to: if it fails, we feel it, and we have to deal with it.

Creative thinking is therefore not thinking about breaking the habit of doing something repetitive, but breaking the habit of doing nothing at all. It’s about doing whatever we can, over and over again, just to get one little gem of an idea out there. In the world of brainstorming, failure is cheap, and risk-taking gets the prize. It is this that should be encouraged within our industry because we are so desperate for it: brazen, reckless, failure that costs nothing, followed by the sparkle of genius that changes the World.

If creativity in programmers might help projects, or help develop programmers themselves, one question needs to be asked: Where exactly does creativity need to be applied in the software industry?

In short: Everywhere.

Software development is a creative industry, as all pinnacles of civilisation ultimately are, from stock markets to museums. The fact that we still describe the disciplines in terms of engineering or science confuses me: it may be that when a deep understanding of electronics was needed before you could begin to write software, this seemed sensible. But in the age of abstract languages like Ruby or Python? Dare I say even Java?

At the core of this is how we go about deciding what software to write in the first place. This is an area, which you may be surprised to hear that there is little creativity.

In fact, nearly all software is a derivative of a previously-available software. Even open-source, where you would expect creativity to thrive is often a collection of re-writes of commercial software which in itself is nearly always a derivative of a new way of making a quick buck. Firefox might be swell, but it is rooted in Netscape which in itself was just a way of making some money out of somebody else’s bright idea. Tim Berners-Lee might have had an original thought once, but don’t think that anything you’ve seen since then around web servers or web browsers has been original.

Then there is the implementation of the software. Programmers like code reuse. They love libraries. They would marry their preferred framework if it were philosophically sensible or legal. However, it is producing a generation of software that is derivative.

Right now, developers are in a “group-think” position that we follow the principles of doing things the way they always have been done. Sometimes this makes sense: windowing systems, cryptographic libraries, etc. However, whilst Rails and Django sure are swell, is it really the case that what is needed in the World is more CRUD-interfaces to relational databases? So often you see projects that could have been implemented more elegantly in less code, but rarely are. “_why the lucky stiff” is a notable exception to this trend: learning from watching him is recommended.

This brings us to the marketing of software. Right now you can sell your time, sell the code, or find a way to incorporate adverts. Sure “monetisation” is important: critical if you want to keep in your XXXL shape. That said, are there really only 3 ways to making money? Of course not, but whenever a new product is thought up, it’s always put into one of a few different silos. Try this: think of all the ways people are able to make money legally in the world from commissions to referrals to services, and then try and find a way to get your ‘great idea’ to fit each of those silos. You might end up inventing something completely original.

Creative thinking isn’t hard – there are thousands of ways of doing it. The question is: why aren’t we programmers doing more of it?

I picked up an article doing the rounds by Joel Spolsky on context switching in Agile development and find myself a bit confused by the way he pitches his conclusion, even though I agree with most of what he says.

Context switching is something programmers first learn about in University purely in terms of what happens when a CPU or operating system has to stop doing what it’s doing, and do something else for a bit. It happens thousands of times a second on a modern computer, and is happening on your machine right now if you have more than one process running. Getting processors to the point where they can do that effortlessly took decades.

The next thing programmers learn about context switching is that they are expected to behave just as well as a modern CPU at context switching when they are expected to juggle dozens of projects at once. Interestingly, the people who expect this, don’t understand what is required to handle a context switch for a programmer and see their new piece of meat as being a bit like a messy version of OS/2 Warp, but perhaps not quite as fast.

In the past I’ve written about what I call alone time – the point where a programmer will try and get into “The Zone” and become as productive as they possibly can. I think those two articles hint at why a context switch is so hard. It disrupts a work-flow that is hard to obtain. You can’t easily drift in and out of a context when you’re being productive.

Joel agrees with all of this, but says “hey, you know, sales are important, sometimes it’s worth the context switch, that’s what Agile development is all about”. The problem is, he’s talking complete rubbish.

Agile is about how to manage the development of a single project: one project on the radar, iterations, feedback, customer contact, the code is the specification. You get the idea.

You don’t use Agile to handle multiple projects. For that you need something completely different. The context switch is something that agile isn’t designed to handle, because it assumes that as a development manager you wouldn’t be so utterly irresponsible, moronic and boneheaded to attempt to try and get a developer handling several projects at once. It’s a limitation of Agile, which is why you need something else in most programming teams to handle it.

Right now I have seven projects on the board, two of them able to wait a while, three of them less than 24 hours from being signed off, and the remaining two due to last through for another week or two – I then intend to take a month off Christmas and sleep. Right now, I need to context switch even though I know it’s not ideal. The way I do that has nothing to do with Agile development, and everything to do with how I plan my time.

In the example Joel points to, the answer is not “this is Agile, deal with it, be Agile”, the answer is to say to the developer “look, do you want some overtime to look at this?” or to find a developer who isn’t busy to look at it instead. It’s about time management and handling more than one project, not about forcing a method designed for one project into something it isn’t.

Juggling multiple projects, handling resource allocation and managing the board in a busy programming team is hard work. It isn’t something Agile can fix, so don’t blame it when it doesn’t, and let programmers stay in context as long as possible if you want them to be productive.

I’m just about coping by handling this my own way, but in the New Year I’m hoping to hire a developer to help me out and let me stay in context for longer.

”It is true some rather interesting projects get abandoned by their developers or shall I say their artists. Most of them are sculptures made out of C (or any other language). And this is the biggest handicap of those programs. They aren’t engineered they get crafted.” – Reiner on www.gnomedesktop.org

When I was 11, I wandered into the school library and on the shelves found a book called “Learn to program BASIC on the BBC Micro”. It was about 60 pages long, and most of each page was covered with pictures of robots telling me what ‘PRINT’ did. Coincidentally the time of year was Lent, and it being a Catholic school I had the option of foregoing lunch and instead donating 40 pence to CAFOD to use the computer room for ‘personal projects’. This was how I began my study. It quickly became an obsession.

By the time I got to 14, I had discovered a few things:

• I couldn’t afford a real computer, but it didn’t matter. This was 1992 and yet whilst my richer friends were playing with Amiga 500 and Atart ST machines, I was stuck with an Amstrad CPC 6128 my father had kindly managed to donate. This was where I learnt the true value of engineering within constraints.
• I didn’t want to study Computer Science. This, I still believe was the right decision. However, my failing was thinking that I wanted to study Software Engineering. I didn’t want to be an academic trying to understand the philosophy of bubble sorting – I wanted to change the World and write software that would help run The Future, and I figured Software Engineering would give that to me.
• My school teachers had realised I was the best person to call when a computer broke somewhere in the building, and as a result, I knew I would probably be able to make a living out of my obsession if I needed to. Knowing that you can make a living with the skills you have when you’re 14 is… empowering.

I set my course. I worked just about hard enough (but no harder) to get to UMIST – now merged into Manchester University a campus I still live on the edge of – and studied Software Engineering.

Actually, that’s a lie.

I was on the course, sure, but I chose to get drunk most of the time instead. See, the problem was, fine, I could learn formal specification and learn how to build a compiler, and that would all be swell, but I now realised it wasn’t going to change anything. I was not going to be able to express myself the way I wanted to if I was confined by determining the correctness of my code mathematically. I thought Software Engineering was more concrete than Computer Science, but in actual fact it was just the same abstract notions but sat in front of a screen and expressing them via C instead of in the library expressing them via pseudo-code.

What I then realised is something I should have had the guidance of when I was 14: I didn’t really want to be an engineer. Engineers do what they’re told, follow a plan, make sure the foundations are set correctly and regulations are followed.

I wanted to be an architect, a craftsman, an artisan. The depression of the realisation that I had taken the wrong course actually drove me to the point of barely turning up for class at all. I ended up taking on a variety of full-time jobs whilst attending exams so I at least came out with a piece of paper. I also spent a lot of time getting drunk and playing pool.

This conflict still continues inside me – sometimes I feel I want to be the engineer, building automated tests, writing out my code in formal proofs, knowing that my code is correct. Other times I want to be the scupltor, using the IDE like a blank slab of marble or a lump of clay awaiting moulding. Some of my best programming has come from just writing a line that kind of looks like it might be the core of what I wanted to do and then wrapping it in other code that made it work. That’s hardly the way you would want a fly-by-wire or power-station safety system to be written, but it is immensely satisfying. It’s fun. It’s art.

So now I look at these formal methodologies, whether they be more traditional lifecycle-led with their heavy requirements capture stage at the front, or if they’re agile, and I think “what a bore”. In some ways I’m drawn more to agile methodologies not because they are more productive, but because they are more fun. They are there to provide confidence to the client, sure, but they allow the artisan programmer out, free to roam.

It feels wrong though – when I go out roaming, I feel as though I am not a professional programmer. All of a sudden I feel I am an artist playing amateur programmer, and I live in fear of being discovered and found out. I constrain myself with project management tools and development methodologies, not because I think they do me any good, but because they stop me feeling guilty.

You know how I spend most of my day? I write a lot of Ruby, specifically Ruby on Rails. I reckon I must spend about 20%-25% of my time looking things up in the API docs for either Ruby or Rails. Do I know how to code in them? Sure I do. I just don’t bother memorising the API. I just find the bit I need, the syntax and method parameters that need to be called right there and then. With time some methods become second-nature, but I don’t actively memorise this stuff. I’m too busy coming up with my own ideas to remember other people’s. How un-engineer-like is that? How skittish and flaky is it? I don’t think I even deserve to be called a programmer if I do things like that.

But I bet I have more fun than most engineers. Because as the code comes out, at the end, it feels like a piece of art is being formed. Sure it might be a reporting module that spits out graphs and pie charts, but I know the inner beauty of that little bit of optomisation that made it query the database 12 times rather than 800 times. I know how I flitted from the model, to the view, to the controller, hoisting each little function up like a piece of scaffolding. I know that before I started there was nothing, and now there is code, there is function, there is value. The World is very slightly different for the users.

If I could go back and give advice to my 14-year old self on how to become a professional software engineer, you know what I’d tell him? Go and study Art or Philosophy. It will do more for your ability to create than any Computer Science course ever will – and you can learn all the CS stuff from Knuth anyway, if you need to. I’d tell him that the true beauty of this profession is that it has the advantages of being a writer, a musician or artist except it’s far more profitable. I’d point out that understanding how code works isn’t as important as understanding how you work.

I’d also probably tell him which girls secretly fancied him and to stay off the beer when he got to Uni, but that would spoil the journey of excrutiating self-discovery. :-)

So I’m keen to learn: am I alone in this industry? Should I be stripped of the title ‘Programmer’ or even more laughably ‘Software Engineer’ and just be known as ‘that guy who does that stuff, with you know, the computers and stuff’? Is the real motivation behind the uptake of Agile not the fact it makes people more productive, but it makes people feel more natural, more like artists? What would you tell the younger you about how to harness ‘the art’ or the ‘the science’ of programming?

Just noticed that if you’re an agile developer (like me) you and you’re in the North West of England (like me) you may want to go to the AgileNorth event on the 20th September. It costs £95, and the programme looks pretty reasonable.

Alas, I’ve only just found out and I’m already up for being somewhere else that day, but I’ll hope to see feedback around the blogs from those people going.

One of the worst bits about being a developer is having to come up with documentation. There are great tools to help any coder who comments their code (fewer than you think actually do this) automatically generate programmer’s documentation, but what about user docs? I hate dealing with user docs. As a result, they often suck.

It has been suggested that user manuals need a make-over and whilst I understand the point that is being made, I think we can do something even better. Rather than spend all that money on producing a beautiful looking user manual that marketing got to, how about get rid of the manual completely?

It’s no secret that a lot of the new start-ups have been inspired by the wonderous 37signals philosophy, and in particular the ideas condensed into Getting Real. To my mind, if we follow those philosophies, we should be looking at producing systems that don’t need manuals. If you can’t look at a web application or a camera and work out intuitively how it works, it’s the wrong interface.

When you buy a new digital camera, what’s the first thing you do with it? I take a picture. I don’t care if the exposure is wrong, that it’s writing to the internal memory instead of the memory stick, that the flash isn’t turned on, etc. Once I know how to take a photo with it, I’m ready to start exploring, and most cameras these days have a great interface via their on-screen display. I don’t want to get my new toy and spend a few hours flicking through a manual just to work out how to turn it on.

This philosophy – the simplest interface possible – means we eradicate the problem known as “Stuck in P-mode” that blights so many users. But what if you have to produce an interface that needs to be a bit more ‘condensed’ than a carefully thought-out OSD menu? What if you want to embrace the expert user who wants the short-cuts and so on?

I’d still say the interface is wrong, but if you have to, when the customer buys it, you should be offering that user free training at a session near where they live. If they live in the middle of nowhere, let them have the training online – let them interact with a live session via the web. They shouldn’t have to go out and search for extra training, they should get it from the start from you, the supplier. Once that happens, users will happily start to talk about you and your product, because you are no longer selling them a gadget – you’re selling them a skillset. That’s a powerful marketing device.