Innovation in Software

[Up-front Disclaimer: I didn't do Computer Science at University. I did "Software Engineering". It involved formal methods and quite a lot of Z notation. I have some pretty strong views on Software Engineering as a discipline as a result. Most of my CompSci colleagues generally do not share my beliefs.]

Johannes Ernst posted yesterday a pointer to an article from IEEE Software by Tom DeMarco, pronouncing Software Engineering as “an idea whose time has come and gone”. Tom’s argument can basically be summed up as:

[...] you need to distinguish between two drastically different kinds of projects:

• Project A will eventually cost about a million dollars and produce value of around $1.1 million
• Project B will eventually cost about a million dollars and produce value of more than $50 million

What’s immediately apparent is that control is really important for Project A but almost not at all important for Project B. This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.

To my mind, the question that’s much more important than how to control a software project is, why on earth are we doing so many projects that deliver such marginal value?

Can I really be saying that it’s OK to run projects without control or with relatively little control? Almost. I’m suggesting that first we need to select projects where precise control won’t matter so much. Then we need to reduce our expectations for exactly how much we’re going to be able to control them, no matter how assiduously we apply ourselves to control.”

Frankly, I can’t believe such a narrow and misguided argument has come from the pen of Tom DeMarco.

Imagine if you will the next time I go into tender on a project.

Customer: So how much will this project cost to deliver, and how long will it take?
Me: Well, how much value is it going to add to your business?
Customer: Pardon?
Me: Is it going to add a little bit of value to your business, or a lot?
Customer: A lot, I expect.
Me: In that case, I have no idea. Let’s just roll with it and see where it goes, eh?

I’m not convinced that’ll go down well.

There are a few logical mistakes in Tom’s argument (and I can’t believe I’m having to point them out), that basically means he’s talking a great big pile of crap:

1. We can not assess the true value of software until it is shipped.

   We might be able to sense the potential value, but it’s a guess.

   I have one project on my books now with a mid-sized build budget, potential value in the hundreds of millions. Another project for a smaller build budget with a potential value of £200k-£300k. Another project with a tiny build budget that if it goes viral could be worth billions five years from now.

   Or maybe the project with the tiny build budget sinks, the medium project gets on NASDAQ in 5 years and the biggest project eeks out a living but never goes stellar.

   We simply don’t know. We can’t know. We’re guessing. There are too many variables outside of the software engineering process to be able to assess the potential value accurately.

   In essence, we have to treat all projects as being potentially of very high value, because they all are.

2. Value is potentially nearly unlimited, budgets are not.

   It might be well and dandy to say that it doesn’t matter if the budget might slip because the value is so high, but budgets are fixed.

   Thousands of software builds are being worked on right now where a 5% budget slip is potentially going to kill the project because there just isn’t more money after the budget is gone, and without being feature-complete the software is going to have near-zero value.

   Yes, incremental development is the way to do things, yes we should be able to stop and ship after each day’s work. However to fulfil the project’s objectives (and maximise its potential returnable value), we must aim to get all the features in to the final build within budget.

3. There are other factors beyond budget, schedule and value to control.

   John Glenn once quipped about being sat on the launchpad, “I felt about as good as anybody would, sitting in a capsule on top of a rocket that were both built by the lowest bidder.

   However, whilst budget was important, he shouldn’t have felt in danger: the political ramifications of a death would have meant that the people who built that rocket and capsule were going to make sure he survived.

   And some software systems are the same. What is the “value” of a nuclear power station control system? Or the control systems in a fly-by-wire aircraft? Do we care more about the financial value or the safety value? I would suggest that controlling the build – engineering the build – in these situations is critical because the safety value is very high.

   In fact, it betrays immediately the fallacy of Tom’s that only the marginal projects need tight controlling – if you’re not controlling the processes around a software build that keeps 400 souls alive at 35,000 feet, you really should be in another profession.

There is a lot of cruft in software engineering. Waterfall methodologies are now considered almost laughable. However, it’s a young discipline and we have barely begun to understand the correct way to engineer software.

Agile philosophies are taking over in development shops because it allows for less process, more communication and ultimately better software being produced at the other end.

Behaviour-Driven and Test-Driven Development methodologies are providing useful benefits to our clients in a way we could barely imagine just a couple of years ago. In certain situations, formal methods and algebraic proofs are confirming that software is “correct” and “complete” in such a way we can literally bet our lives on it.

Tom admits himself he’s not on the front-line of software development these days. Perhaps his metric-centric universe is correctly condemned, but most of us aren’t using those methods anyway. We’re looking at perhaps a few numbers instead: test coverage, remaining days in budget/schedule, remaining features to be implemented. We’ve moved on. Perhaps he just doesn’t get what we’re doing in dev shops these days.

And we’re still learning new techniques, new ways of doing less process, more quality software. Software Engineering is definitely not something that has “come and gone”: it has barely got its foot in the door.

Towards the end though, Tom does make one point I can’t agree with more:

Consistency and predictibility are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

This reminded me of a talk given by Ted Nelson at OpenTech 2005 where at one point he almost screamed:

I didn’t get into computers to automate trivial crap! I got into it to change the World!

Didn’t we all? But I think we can do it without having to just free-ride our way along with our fingers crossed, hoping we don’t run out of money.

Development is a creative pursuit. Whilst many think of it as a purely technical challenge, it requires a level of lateral thinking about the World that is a cross between doing a crossword puzzle, composing a symphony and having an argument with people who don’t exist. It’s not surprising some of us are a little eccentric.

It reminds me of the writing process a lot. You sit down at a blank screen after having conducted your research and you have to just dig in and find some way of making progress. Many a developer struggles with a blank IDE screen much in the same way many a writer struggles to find influence. When I was learning how to write properly, I was told that “a professional writer can not be like the poet who spends a morning taking out a comma, and the afternoon putting it back in”. We need to work hard. Same with code. A block then is a real problem.

Slashdot this weekend asked How to Get Out of Developer’s Block?, or rather a user asked:

I have spent the past six months working on a software project, and while I can come up with ideas, I just can’t seem to sit down in front of the computer to code. I sit there and I just can’t concentrate. I don’t know whether this is akin to writer’s block, but it feels like it. Have any other Slashdotters run into this and if so how did you get out of it? It is bothering me since the project has ground to a halt and I really want to get started again. I am the sole developer on the project, if that makes a difference.

The comments that follow in the thread that range from the sensible to the bizarre. I have a bunch of tricks I use when I’m struggling, so thought I’d put them together

1. Get enough sleep – you have no idea how sleep deprivation can mess you up when you’re trying to concentrate. When I’m working on code, I take a minimum of 10 hours sleep a night. Anything less, and I’m not going to be able to think in purely abstract terms for 8 hours straight during the day.
2. Exercise – and whilst those of you who know me might laugh, it’s important. I actually do get regular exercise when I’m coding full-time. Just a long walk at the start or end of the day can be enough. Something that gets the hear rate up helps though (perhaps explaining why I always code better the day after… errr… private stuff that gets my heart rate up!).
3. Don’t drink alcohol – this was something I got when I was trying to sort out my pilot’s license. When you’re going flying, I don’t drink for 24 hours before getting into the plane. I found my workload was easier, my writing got more fluid and my code went up a gear. On big client projects I don’t drink at all on school nights. If I’m drinking in the evenings whilst on a project, it’s because the project isn’t challenging me and I’m bored.
4. Clear your environment out – I’m currently sat at a desk with perhaps 150 items of paperwork on it. In this environment, I can not focus on code. My mental processes are cluttered because my physical processes are. Tidying up might seem like a stupid way to get out of a block, but I genuinely find that a clear working environment leads to much clearer mental processes. I don’t know how or why, it fascinates me, but just get your physical environment fixed up and suddenly your mental environment starts to fire a little better than before.
5. Write a trivial test – this is the code version of “free-writing” that I sometimes use to unblock on writing an article. Basically write a small test (or spec if you’re BDD) for something almost trivial and then get it to pass. Repeat. Now you’re back in the game.
6. Work on the design – it’s amazing how bad we collectively are at really thinking through a problem. Go and work on some wireframes or develop some sketches of the underlying schemas and try and simplify them. Reduce things down, and suddenly you’ll see areas you can work on right away outside of the problem you’re blocked on. If you’re not able to delve into design or architecture because of the nature of the project, quite frankly you need another bunch of guys to work with.
7. Try and find it done already – I once spent a lot of time trying to work out how to solve a particular problem. The answer was non-trivial to implement in my mind. I kept putting it off. I was scared of how bad I could end up making my solution. In a fit of procrastination I spent an hour digging around the problem area and eventually found an open-source tool that did exactly what I needed, out of the box. Well, that solves that problem…
8. Are you scared of success? It might sound like a stupid question because success is good, right? But when we succeed at something, we conquer some barrier we have worked to overcome for a period, things change. Suddenly people might look at you differently. Perhaps you end up having to work on a less interesting project. You might want your current project to be a success for other reasons. Ask yourself whether you really want this project to succeed. And then realise there’s no getting out of it: failing, or staying where you are is just as bad an outcome and harms you, your self-confidence and your reputation.
9. Find a SCRUM meeting somewhere – one of the very best things about daily stand-up meetings in SCRUM projects is that the meeting only has three topics of conversation: outcomes from stuff you agreed to do in the last meeting; what you plan to do today to further the project, if anything; and obstacles in your way. Not everybody has a team (and sole development is the hardest form there is, trust me), so find a SCRUM somewhere else. Use Twitter, your blog, a group of friends down the pub, anything. Just talk about what’s stopping you and see if anybody can help you in any way, or offer suggestions. Obviously asking a friend about a tricky problem relating to class inheritance isn’t going to yield results if they don’t know what you’re on about, but ask around more liberally than you have done to date.
10. Work on something else – we all have other projects on the go. If the above isn’t working, just go and get on with something else. Your subconscious is dealing with the problem and will come up with a solution. Just make sure you hit your deliverables schedule if you have one!

Now comments are back up, I look forward to hearing of any other tips people might have.

I have been throwing a mantra around at clients in the last year:

Only 20% of software development is about writing code. The other 80% is all about communication

As an industry, we are awful at communication. I don’t mean writing reports, I mean listening. Really listening.

Guess what? The clients have noticed. I’m spending more time picking up pieces after people didn’t listen. Sometimes I look at work I’ve done myself and realise I didn’t do it so well myself. Live and learn…

This last week I was reminded of an old article I wrote here that discussed the rudeness and arrogance of some developers. A client suggest we “resume this phone call when you’re less frustrated with me”, and when I apologised and explained my frustration he simply answered “that’s fine, but I’ve had the same thing for years from my ex-wife”.

Wow.

When I realise I’ve dropped the ball, I’ve upset somebody and I’ve not known how I did it, I step back and re-evaluate myself. It would have been easy to blame the client, it would have been less challenging to just write it off as one of those things. But that doesn’t help him, and therefore I’m failing.

Stories I hear at events tell me I’m one of the few guys who will go through that process – the majority just write off the client as being at fault, and we all end up with a bad reputation.

I’m worried for the next few years this problem is going to get worse across the industry. We have more clued up people in the industry than ever before, and we are building more and more echo chambers where you hear people at events talking about Twitter as if it were as common place in people’s lives as a fridge.

Over the next few years we are going to see an influx of creators, innovators and entrepreneurs with great ideas who don’t know much (yet) about software development. I just sat down with one potential client and explained Behaviour-Driven Development at an abstract level and they were blown away by the sophistication of the philosophy. I then realised that to me this is every-day stuff, to them it’s not a million miles away from arcane magic.

We need to do a better job at explaining development to people who don’t know anything about it. We need to explain why it’s worth spending money on doing less, but that less is of a higher quality. We need to stop and listen to what they need to help build a great business. And that is key: ultimately, we are not providing a service or building a product – we are helping them build a great business.

And the only way for most of us to help them do that, is to produce great software.

And in turn, the only way to do that is to get better at the other 80% of the job we’ve been so used to ignoring until now: communication.

I expect I’m going to be talking about this a lot more over the next few years.

I have had an awful lot on my plate recently. I think most of us in this sector have.

Right now, I have a few choices to make about what Vagueware does and is involved in over the next 9 months, and I feel the time is ripe for some minor re-invention.

Over the last 15 years I have worked on everything from server administration to strategy, development to training, project management to debugging. One of the reasons Vagueware has struggled to grow at the rate it should have is because I’ve not really chosen one small area and really focused on it. I’ve had my fingers in many, many pies.

It is through being over-committed to so many projects and concerns that has led to me being exhausted and the quality of my work being deeply unsatisfying to myself.

That needs to change, and indeed it already is. I am now moving away from client work and towards internal product development (including a joint venture I’m itching to tell you about, but it’s too early yet), but I’m worried I’ve missed a trick and there is something out there that it’s “obvious” I should be working on but I’m not aware of. You have no idea how often people will start a conversation with me with the phrase “I’m amazed you’re not working on…” and on consideration, I’m usually surprised too.

So, this is last call for input. If there is something you think I should be working on, whether it be a product area you’re surprised I haven’t got stuck into, your own project you think I can help on, or just an interesting idea, speak now (via email or twitter please) or forever hold your peace. If you want to hire me for a project you have until Monday to start the discussion, but after that all client work is being put on hold.

It’s about to go dark on the client front and Kagtum development (amongst others) is going to gear up to a whole new level. See you on the other side.

It’s been a while since I blogged, so I thought I’d give a heads-up and explain what I’ve been up to.

I’ve been changing what I do for a living over the last few months. Over the course of that time I have discovered more about myself, this sector and what we are lacking in the region than I ever imagined I would.

When I created Vagueware Ltd I was so tired of meetings after four years in academia I would do anything to avoid them. I basically wanted to sit in a room and write Rails code all day, every day. A year of that made me realise I’d taken a wrong turn. It took me another year to get the guts to change it, and start thinking about what I really care about doing.

Over the last few months I have been asked to sit down with more and more companies and just listen, absorb, and respond. A person has an idea on how to improve services in a sector, they don’t know how to deliver it, they might not know how to finance it, who should they speak to, what skills are they lacking, can I help? I listen, discuss, think, do some research, think some more, discuss some more, write a report and take them forward.

All of a sudden I’m having to connect paid SLAs of hosting providers with finance forecasts. I’m trying to understand the politics of a business and connect it to Agile web development processes. I’m helping people who didn’t know they could be a dot.com six months ago work out how to connect to a billion people as quickly as possible. It’s absorbing, fascinating, tiring, but ultimately what I was hoping it to be: optimistically human.

You might think it’s nice work if you can get it, but it’s absolutely shattering. I’m averaging 80 hours/week now (I’ll be hiring in the next couple of months, so if you like the sound of all this and want to come and play, do get in touch), and I typically don’t feel my head hit the pillow. I am exhausted and need a break, but I have never been so excited about my client portfolio.

In the next few weeks I’m hoping to get back to writing up articles here on innovation in the software space, but for now, I’m too busy doing it to write about it…

In the very near future, things are going to be changing at Vagueware.

Firstly, the site currently at vagueware.com is going down. I’m going to release the code running the idea bank as open source and you’ll be able to also setup a free hosted version of your own on Vagueware’s servers. Think of it as a bit like wordpress.org & .com but for open innovation rather than blogging. This will mean you can create your own IdeaStorm for your company or product.

I think open innovation and getting customers or employees involved in product and service development is going to be big in the next few years, and I want to help people get involved. If you have Ruby on Rails skills, patches to the code base will be appreciated as well – it’s going to be MIT licensed so that it follows the “Rails way”.

That will of course need a new name, and given that it’s all about constantly evolving and changing what you do and how you do it, it’ll be named Fluxish.

There are quite a few major changes needed to get the current build ready for that release, so don’t expect it this week. The ideas on the current site won’t be lost: I’ll be creating a special little hosted fluxish install and moving all the data and users over – I won’t be destroying anything, just giving it a new home.

So what will go in the idea bank’s place at the main site? Well, the new Vagueware site will concentrate on selling my consultancy and development services. There will also be a mini-blog there about the business, freeing this blog up from posts like this where I discuss what is going on inside the business. I’ll be highlighting companies I’ve worked with in the past and occasionally posting a page up as a more detailed article about the process of development.

This blog will become much more focused on innovation and emerging trends within the digital sector. This is an area I’ve drifted away from in the last three months, and I’m keen to get it back on track.

In addition, I’m going to be blogging more elsewhere in partnership with other organisations.

I’ve agreed to start writing more for O’Reilly GMT to try and turn it into a more mature source of information for the technology scene within Europe. I’m still working out and proposing what kind of articles those will be, but obviously they’ll not be about vagueware, not about innovation in software in the sense this blog will be, but aimed at a tech-savvy audience.

Also, I’ve been asked to contribute articles to ‘Manchester is online’, formerly ‘The Mancunian Way’. It’s one of the most read Mancunian blogs, and I’m hoping to bring some insight to a slightly less geeky crowd than the usual readership I get to speak to here. This is more of an experiment right now, but I’m looking forward to seeing how it develops. It’s the first time I’ll be stepping across into blogging for Mainstream Media, and I couldn’t be more pleased that I’m doing it with the team at the Manchester Evening News.

In short then, I’ve got a lot of writing to get on with over the next few months, so please don’t be too upset if this blog gets neglected at times.

In the last few weeks, I’ve been getting some pestering from Manoj over my business and how to develop it. The conversation last week took it a step further: stop trying to do everything within Vagueware and instead concentrate on coming up with ideas and then – critically – finding the team to make the ideas happen. I might be great at ideas, and able to hold my own as a developer and somebody able to build a cashflow model, but I can’t do marketing to save my life. Nor can I handle PR, design, or think of everything else that needs to be thought of.

Of course, this took the usual pattern of my conversations with Manoj: we started with “Manoj, you’re such a doofus”, and ended with me thinking about it and conceding he might have a point.

Right now, I have so many ideas to work on they just sit here. They do nothing whilst on my desk. That was why I built vagueware originally – push the ideas out there, somebody, somewhere is bound to get on with them. I left myself with just a few ideas to work on:

• My own consultancy: helping businesses develop software solutions. Two years ago that took the form of offering services as a Rails developer and has evolved to the point that next year it will take the form of managing a range of developers and being the bridge between the business World and the geek World
• The Idea Bank: allow people to post ideas, and build a business around encouraging innovation
• Kagtum: A new way of thinking about news, relevant content and what people need to know about the World around them
• Fluxish: An idea I’ve not blogged about, but ultimately comes to down to very scalable “it just works” web application hosting that takes the pain away from growing an online business

Guess what? That’s still too many ideas. I figured with 400+ ideas, taking just 1% of them for myself and shoving the other 99% “out there”, the 1% would become manageable. That hasn’t happened and it means whilst the consultancy is doing “OK” it’s not doing “great” and all the other ideas are suffering from neglect.

I’ve made a decision then. I’m now reducing the number of projects I work on: my day job is now the consultancy and the idea bank.

Except I still want to make Kagtum happen, and I still want to make Fluxish happen, and there are four or five ideas outside of those I’d like to see come to life in the next 12 months.

How am I going to do this? Well, I’m going to start putting teams together who want to take equity in an idea and with a mixture of design, developer, PR and other skills, we’re all going to own a share in a business that we get to the point of being of interest to external funding – or even better, making money on its own two feet – with a view to exit.

So, if you think this sounds like something you want to work on (particularly if you’re interested in Kagtum or Fluxish), or you have skills outside of development such as PR and marketing, and you think you could give up 5-10 hours/week for a business you’d have equity in, you might want to get in touch with me. If you’re not convinced, you need some idea as to what is involved and want me to blog some more, leave a message in the comments.

I’m currently reading Wikinomics and finding it incredibly engaging. I’ll write a fuller review when I get to the end of it sometime later this week, but I’m that enthusiastic I had to give people a bit of a heads-up before the weekend. The full review is likely to be long. This post won’t be.

To date, the only truly successful wiki has probably been Wikipedia – it’s probably the only one that the mythical ‘man in the street’ can name. In Wikinomics, Don Tapscott and Anthony D. Williams document an emerging trend and show that it’s not just wiki software that is describing the new spirit of collaborative development, but blogs, UGC sites like YouTube and social networks. It is the interactive element that adds value into the business, not the technical definition of what a wiki actually is.

Where the really interesting things are going to happen though are where collaboration happens between end customer and producer, and the middle men who connect half a dozen businesses to a single customer desire.

Outsourcing has reached a point where an industrial designer and a marketeer can design a product over coffee, firm up designs overnight, have prototype units being developed by a Chinese company within a matter of weeks, and support provided by an Indian company the day the unit goes on sale. The flexibility of this kind of out-sourcing is allowing start-ups to get very big, very quickly.

Some are beginning to realise they can even outsource the product design to the customer. It’s not just small companies either – major companies are seeing the value of a porous membrane between internal R&D and the rest of the World.

Vagueware obviously has a vested interest in this model. I haven’t quite worked out the dynamics, the money side of things and how we go about making developers take notice, but I’m hoping that others who like the idea of open collaborative design in the software industry will help work that out with me. I’m currently toying with ideas on how to reward those outside my business who directly add value to it. If you have ideas on how that can happen, you know what to do

I’m not alone. We’re about to enter an era of real businesses with real products being built this way. The knowledge economy is going to be very flat, with each of us having the ability to act as independent agents working on the ideas that interest us. Economically, this is going to be fascinating.

From what I’ve read so far, Wikinomics is a good introduction to how this new World is starting to unfold, and I think if you’re interested in these new models or if you’re interested in what the next 2-3 years of web application development is going to look like (if you’re a bespoke developer or designer, your future clients are either reading this book already or will do soon), you need to grab yourself a copy.

You can buy it from this link if you’re in the UK or this link if you’re in the US. Enjoy!

This weekend just gone, I attempted to compete in Railsrumble 2007 with an application I call ‘Kagtum’. The idea of Rails Rumble is to take 48 hours to build a Rails app from scratch in a competitive scenario. At around 7:30pm BST last night I realised I wasn’t going to finish the app and so I asked to be withdrawn, as I explained:

Maybe it was the fact I decided to try and compete on my own, and therefore didn’t have the advantage of a team. Maybe it was the fact it took half the first day just to get a working application stack up on linode. Maybe it was the Saturday afternoon spent in the company of friends rather than coding. Maybe the idea was too ambitious.

Maybe I just wasn’t good enough.

Whatever it was, I didn’t get enough of my idea implemented on the weekend of the 8th and 9th September 2007, that I wanted other people to look at it. I failed. There is no app here.

The ideas I played with in those 48 hours though, intrigue me, and they will be worked on over the coming weeks and months. The end goal is going to either be very interesting, or an exercise in futility. If you want to find out which, you can keep an eye on the blog and I’ll be making announcements there.

I will be judging, and I look forward to seeing other apps, so good luck. Until next year…

The first day did kill me – linode was under heavy load (not surprisingly, with over 100 teams trying to get their application stacks set up) and the guidance we had been given by way of a screencast was inaccurate in places. Top tip: when you’re in a hurry leave the rdoc behind and always pass “-d” to gem install.

Anyway, I still hope to judge – and I’d advise anybody with an interest in innovation to look out for the announcement that you can sign up for judging and take a look at the apps that were finished – but I thought I’d talk about Kagtum a little bit here, because the core is almost done and I’m confident I can get a working app out of the door soon. I’m also tempted to open source it.

It all started about 2 years ago when I was left distinctly underwhelmed by Wikinews. The problems with wikinews are many and pretty obvious to anybody who spends a few minutes digging.

The primary problem to my mind is that they’re using a piece of software designed to build an encyclopedia to build a news website which means all articles are given equal footing. It seems reasonable that they should be given equal footing, until you realise that unlike an encyclopedia, not all news items are equal. A world-famous opera singer dying is not equal to a drunken brawl in my local town centre, and neither are equal to the Iraqi PM losing the confidence of the Iraqi citizenship.

However, the core idea – news written by, and for, everybody is a great idea. I’ve spent the last two years playing with lots of ideas in my head and watched emerging developments in the online news and journalism scene before I came up with an answer: quite simply it comes down to targeting relevance.

If I am in Manchester UK, there are stories that are local to me I’m interested in that somebody in New York doesn’t want to see. Likewise, there are stories happening on the other side of the planet which are important to me because they have an impact on me, or because they are in an area I have an interest in. The “perfect” news website will know this, and present just the articles I need to see. Ideally, I also don’t want to be bogged down with partisan and opinionated pieces – I want impartial, simple, Economist-news-page-style articles that give me the leader and then show me what is out there being written about it.

Thus was born the concept of Kagtum – the phrase “kag tum” means “to bring news” in Sumerian, the script of which is the oldest written language currently known to mankind. Kagtum will be a wiki news site that helps target articles based on relevance to you and your life. Relevance is everything.

The idea is quite simple, but the algorithm needs some polish before I can roll it out: we create a news story perhaps based on a report in MSM, or perhaps as a first-hand eyewitness account, that points to online sources if available. We then attach to that story some “impact profiles” based on location or a tags.

For example, a story happening on my street (say, planning permission for a new development) would have a geocoded location and an impact radius of the local neighbourhood. A story happening in 10 Downing Street would also be tagged with that location but could have an impact on the whole of the UK. Suppose the latter story was a policy announcement on Iraq – we’d add Iraq as a location impact as well.

I then login and give my location as my postcode or street, which is geocoded, and the software knows that the story in my street is relevant to me, so is the story in Downing Street. It knows therefore, what is relevant to each and every user and displays the appropriate stories.

Let’s suppose however I have no interest in Iraq. We can tag stories and users can also add tags to their profile that they’re very interested in or very disinterested in. If I said I wanted all stories marked “Iraq” to be pushed down the queue then its relevance to me would be lowered – it might still appear, just not as prominently.

In theory then, when I log in to kagtum, I would see stories about technology, politics and cricket, particularly with stories about my local neighbourhood (stories about technology in my neighbourhood would be even more prominent), whilst my friend who doesn’t care about anything but beer and football will see something perfectly tailored for his interests.

It may also be the case that there are multiple profiles for each user (home vs. work) and that a user can add multiple locations – where they live, where they work, where they grew up, where their parents live – and sees a mixture of stories about places they care about.

The biggest problem I had this weekend was developing the specifics of knowing which stories to show to each user. The problem isn’t hard algorithmically, but providing a technique that doesn’t harm performance and can scale to more than a few hundred users online at a time is proving a little tricky using standard ActiveRecord associations and using the methods baked into GeoKit by default.

There is also an issue of what we mean by “radius”. Saying “this story is important to everybody within 5 miles” is simple enough, but what if I say “to everybody within Greater Manchester”? I somehow need to know if a given longitude and latitude is within that district or not. The Radii Problem (as I came to call it whilst muttering to myself) is important and it’s difficult. I discovered it as I added a story in Washington that was important for the whole of the US – if I added a simple radius of 3,500 miles (to take in California) it of course also covered a huge chunk of Canada, Middle America, the Caribbean, the whole of the North Atlantic (including Ireland!) and most of the North Pole. For a story about domestic US politics, this is obviously needlessly “grabby”.

I have ideas on how to solve this problem, but they’re going to take a few weeks of playing with datasets from the UN and other agencies to be able to get them working smoothly.

There are other aspects I have planned for the site around developing narratives and helping individuals become kagtum journalists, but I’ll keep discussion of those for after the roll-out of both kagtum and the new vagueware.com.

I’ve turned comments on for this article, so if you have thoughts, ideas or suggestions, please leave them.

We should do something to fix this you know…