Innovation in Software

Tim O’Reilly is calling for Facebook to share social graph data so that systems can leverage all the data you’ve shoved into Facebook and use it within their own apps.

There are a couple of issues here.

Firstly, Facebook isn’t actually stopping 3rd-party API developers from knowing who your friends are, and if your friends agree to add an application, the app provider can see their graph too. What isn’t agreed yet is whether this should be made more open, or whether there needs to be a standard way of describing this data. There are all sorts of reasons why I might not want my “social graph” to be made available in an easily-manageable format, not least because it raises privacy concerns.

There is also the fact that Facebook’s business model relies on not making this data available. The “expose your data, and they will come” argument relies on a simple metric of conversion.

Within a company like Amazon, exposing the product catalogue by API is a no-brainer. The more places their stock list is available, the more chances they have of getting somebody into the system, the more likely they are to convert them into a sale. The porous membrane an API gives an app developer in this instance means 3rd-prty developers do the hard work of getting stock shifted in countless innovative ways the original company wouldn’t have thought of.

Facebook however, is different. The ‘conversion’ in their instance is getting somebody to look at pages with adverts on it. What they need is for their users to actively recruit more users – invite them inside the walled garden – and then try and keep them there. They’ve out-sourced the “retaining” part of the equation to developers (playing games, taking quizzes, sharing links, glorified e-mail), but by allowing their most valuable asset to be easily exported they are reducing their customer’s incentive to stay within the walled garden.

As always, it comes down to whether you have a right to that data, and whether you have a right to move it. I’d argue you do, but I’m suggesting it’s going to be hard for Facebook to allow you to take it wherever you want.

[UPDATE]: I realised there is a way to do this without Facebook’s permission. I’ve written it up on the site.

My good friend and ex-colleague from way, way back, Andy Stothard, is currently on holiday in Vancouver. He loves the place and last time he came back he raved about it.

He’s just posted an article that struck me as an interesting take on file sharing. He’s sat down in front of his laptop, found a nearby computer sharing iTunes and decided to have a listen. It may be that “Gareth” didn’t intend for that to happen, or maybe he did.

What fascinates me about this is that there is an added context to the file sharing given by geography. It’s only people on the same immediate network you can see, and if Andy had really wanted to, he could have found Gareth and had a chat about his music collection (and maybe IT security) whilst he was there. What if it was “Gillian” instead of “Gareth” and Andy had really liked her taste in music. And she had found his iTunes collection to be interesting as well. And they’d made an effort to find each other?

File sharing has been criticised because it allows for an anonymous, amorphous mass on opposite sides of the planet to steal copies of music easily. What Andy was engaged in there didn’t take anything away from the publishers (he didn’t have a copy of the music, if he wants a copy he’ll need to buy one himself), added to his sense of the people around him in a foreign city, and potentially could have allowed for interesting conversation to break out between two previously unconnected people.

There is a lot to think about in the future with relation to the economics of creative works and the rise of ubiquitous digital access, but we need to realise that it’s not “File Sharing” that is the problem but “File Copying”.

This weekend just gone, I attempted to compete in Railsrumble 2007 with an application I call ‘Kagtum’. The idea of Rails Rumble is to take 48 hours to build a Rails app from scratch in a competitive scenario. At around 7:30pm BST last night I realised I wasn’t going to finish the app and so I asked to be withdrawn, as I explained:

Maybe it was the fact I decided to try and compete on my own, and therefore didn’t have the advantage of a team. Maybe it was the fact it took half the first day just to get a working application stack up on linode. Maybe it was the Saturday afternoon spent in the company of friends rather than coding. Maybe the idea was too ambitious.

Maybe I just wasn’t good enough.

Whatever it was, I didn’t get enough of my idea implemented on the weekend of the 8th and 9th September 2007, that I wanted other people to look at it. I failed. There is no app here.

The ideas I played with in those 48 hours though, intrigue me, and they will be worked on over the coming weeks and months. The end goal is going to either be very interesting, or an exercise in futility. If you want to find out which, you can keep an eye on the blog and I’ll be making announcements there.

I will be judging, and I look forward to seeing other apps, so good luck. Until next year…

The first day did kill me – linode was under heavy load (not surprisingly, with over 100 teams trying to get their application stacks set up) and the guidance we had been given by way of a screencast was inaccurate in places. Top tip: when you’re in a hurry leave the rdoc behind and always pass “-d” to gem install.

Anyway, I still hope to judge – and I’d advise anybody with an interest in innovation to look out for the announcement that you can sign up for judging and take a look at the apps that were finished – but I thought I’d talk about Kagtum a little bit here, because the core is almost done and I’m confident I can get a working app out of the door soon. I’m also tempted to open source it.

It all started about 2 years ago when I was left distinctly underwhelmed by Wikinews. The problems with wikinews are many and pretty obvious to anybody who spends a few minutes digging.

The primary problem to my mind is that they’re using a piece of software designed to build an encyclopedia to build a news website which means all articles are given equal footing. It seems reasonable that they should be given equal footing, until you realise that unlike an encyclopedia, not all news items are equal. A world-famous opera singer dying is not equal to a drunken brawl in my local town centre, and neither are equal to the Iraqi PM losing the confidence of the Iraqi citizenship.

However, the core idea – news written by, and for, everybody is a great idea. I’ve spent the last two years playing with lots of ideas in my head and watched emerging developments in the online news and journalism scene before I came up with an answer: quite simply it comes down to targeting relevance.

If I am in Manchester UK, there are stories that are local to me I’m interested in that somebody in New York doesn’t want to see. Likewise, there are stories happening on the other side of the planet which are important to me because they have an impact on me, or because they are in an area I have an interest in. The “perfect” news website will know this, and present just the articles I need to see. Ideally, I also don’t want to be bogged down with partisan and opinionated pieces – I want impartial, simple, Economist-news-page-style articles that give me the leader and then show me what is out there being written about it.

Thus was born the concept of Kagtum – the phrase “kag tum” means “to bring news” in Sumerian, the script of which is the oldest written language currently known to mankind. Kagtum will be a wiki news site that helps target articles based on relevance to you and your life. Relevance is everything.

The idea is quite simple, but the algorithm needs some polish before I can roll it out: we create a news story perhaps based on a report in MSM, or perhaps as a first-hand eyewitness account, that points to online sources if available. We then attach to that story some “impact profiles” based on location or a tags.

For example, a story happening on my street (say, planning permission for a new development) would have a geocoded location and an impact radius of the local neighbourhood. A story happening in 10 Downing Street would also be tagged with that location but could have an impact on the whole of the UK. Suppose the latter story was a policy announcement on Iraq – we’d add Iraq as a location impact as well.

I then login and give my location as my postcode or street, which is geocoded, and the software knows that the story in my street is relevant to me, so is the story in Downing Street. It knows therefore, what is relevant to each and every user and displays the appropriate stories.

Let’s suppose however I have no interest in Iraq. We can tag stories and users can also add tags to their profile that they’re very interested in or very disinterested in. If I said I wanted all stories marked “Iraq” to be pushed down the queue then its relevance to me would be lowered – it might still appear, just not as prominently.

In theory then, when I log in to kagtum, I would see stories about technology, politics and cricket, particularly with stories about my local neighbourhood (stories about technology in my neighbourhood would be even more prominent), whilst my friend who doesn’t care about anything but beer and football will see something perfectly tailored for his interests.

It may also be the case that there are multiple profiles for each user (home vs. work) and that a user can add multiple locations – where they live, where they work, where they grew up, where their parents live – and sees a mixture of stories about places they care about.

The biggest problem I had this weekend was developing the specifics of knowing which stories to show to each user. The problem isn’t hard algorithmically, but providing a technique that doesn’t harm performance and can scale to more than a few hundred users online at a time is proving a little tricky using standard ActiveRecord associations and using the methods baked into GeoKit by default.

There is also an issue of what we mean by “radius”. Saying “this story is important to everybody within 5 miles” is simple enough, but what if I say “to everybody within Greater Manchester”? I somehow need to know if a given longitude and latitude is within that district or not. The Radii Problem (as I came to call it whilst muttering to myself) is important and it’s difficult. I discovered it as I added a story in Washington that was important for the whole of the US – if I added a simple radius of 3,500 miles (to take in California) it of course also covered a huge chunk of Canada, Middle America, the Caribbean, the whole of the North Atlantic (including Ireland!) and most of the North Pole. For a story about domestic US politics, this is obviously needlessly “grabby”.

I have ideas on how to solve this problem, but they’re going to take a few weeks of playing with datasets from the UN and other agencies to be able to get them working smoothly.

There are other aspects I have planned for the site around developing narratives and helping individuals become kagtum journalists, but I’ll keep discussion of those for after the roll-out of both kagtum and the new vagueware.com.

I’ve turned comments on for this article, so if you have thoughts, ideas or suggestions, please leave them.

I read some pretty weird stuff out there. You might think it odd given I’m all into innovation that I rarely read the likes of Techmeme or Techcrunch – the truth is, there’s little innovation in either.

I want to look at the guys who spend days lying on their backs staring at clouds and thinking to themselves “that cloud looks just like a better way to moderate pseudonymous online discussions”. I prefer artist coders to mathematician coders: they have more interesting things to say. I’ve said this so many times it’s starting to sound like a cliché: I’ve learned more about software from architecture, art and philosophy than I ever have from my University notes on formal methods.

Occasionally, as I scour the websites of the research labs of the World, I encounter something a little bit “special” in the “let’s look at something unique and apply it to the computer industry” niche. This is no exception.

HP Labs have recently published (only appeared in my RSS today, but it’s dated the 12th July) a paper on the election protocol the city of Venice used to elect its ‘Doge’, or Duke/Chief Magistrate, if you prefer. It appears to have been published in the IEEE proceedings for their 20th Computer Security Foundations Symposium which happens to have taken place in Venice, 6th-8th July this year. It could all just be a bit of an in-joke.

However, there are some really interesting ideas in here.

The election of the Doge was – in modern terms – quite absurd. It consisted of 10 rounds of voting, and at the beginning the entire electorate (the Grand Council of oligarchs who were all men over 30) was eligible. Each round was alternately a random lot draw, and then an election. There is an outline of the process given in the paper, and you can see how long it must have taken to elect anybody.

Where the authors (Miranda Mowbray and Dieter Gollman) take it is to show that there are some qualities it would be sensible to consider when designing computer-based election procedures. Specifically they argue that consensus in asynchronous systems, recovery after network partitioning or indeed any scenario where a ‘lead’ has to be taken by some node might be all processes that benefit from such a system. Whilst absurd to us now, it has the virtues of stability and resilience to ‘gaming’ or corruption.

There are some real gems in here as they draw parallels between modern day computer security and this ancient electoral system. Part of the protocol required for a random ballot to ensure that the drawing was fair, was that the person doing the draw would be selected as “the first boy seen after prayer’s at St. Marks” by a particular member of the Council. Mowbray and Gollman describe releasing a young boy into the square at just the right time as a “protocol attack”. They go onto describe the protocol’s arduous and drawn out procedure as “security theatre” – something that anybody who has looked at recent anti-terror legislation is all too familiar with.

They go onto propose a slightly simpler protocol that could be used by computer systems and I’m left with a mind whirring with possible applications. This is the kind of paper that makes me wish I’d got the main Vagueware application online as I can suddenly find myself thinking of applications in search, auctions, community building and more.

This is what I mean when I talk about innovation in software.

Real innovation in this space is not about taking something you’ve seen on the Web and doing it slightly differently like thousands of others. It’s about taking a piece of politics, art, theatre, humanity, and finding something in it – a tiny piece of inspiration – that gives you an idea of how to make software better.

Here we go again. A round-up of links for a Sunday spent in front of the PC, or if you’re a corporate slave, a way to pass Monday morning bunking off doing real work.

What is the Secret Behind Contagious Behaviour – This video from Stanford’s Always On summit has some fascinating discussion about contagious behaviour, implementing innovation and working out how to give up control of marketing and products to your customers. My favourite phrase from it probably has to be “fragile fires”. Mitchell Baker of Mozilla, Perry Klebhan – inventor of the modern snow shoe, and Gil Penchina of Wikia discuss how to get users doing the work for you. Moderated by Bob Sutton of Stanford and Diego Rodriguez of IDEO.

Collaboration doesn’t Work – if you’re afraid of the Kool Aid, this article from inc.com suggests teamwork and collaboration doesn’t work. I think the conclusions drawn here are all wrong (obviously) but for a simple reason: the author thinks that collaboration can be done by anybody, without training. It’s a skill that needs time to develop, much like the skill of being able to write software, write a PR release or do a cashflow forecast. Asking people to just walk in a room and start brain-storming without any prior training is asking them to behave in a business context using skills that to this point they had only learnt in social contexts, i.e. contexts where being polite is more important than being right.

Wikipedia Forks – It’s quite common within open source projects for groups within the project to reach disagreement and one set walks off with a copy of the project (which is legally OK for them to do), set up camp somewhere else and create a new project with the old code as a base. This is known as ‘forking’ and it happens a lot more than the media would have you believe. Now a group from Wikipedia believe it’s time to create a new project that has the good bits of Wikipedia but with the oversight of experts, and so off they march. Initially we as readers won’t see much difference, but the proof of the pudding will be in the long-term eating. I wish them well.

iPod users prefer CDs to iTunes – and who can blame them? The issue here of course is ‘DRM’ or ‘Digital Rights Management’. If you buy music through iTunes, Apple get to tell you which devices you can play that music on and how. If you buy a CD and rip it into iTunes you get a physical back-up, you can play the music where you want, and the ripped files can be copied to any device including ones not made by Apple. I genuinely think that within the next 3 years there will be a massive consumer revolt at DRM and the only way to deal with it will be to completely restructure the way music labels (and in the future, Hollywood studios) make their money.

8 Steps to better IT meetings – In fact, not just IT meetings, but any meeting. A nice round-up of how to keep meetings on track. Personally, I think meetings should be avoided completely – individual conversations work much better and if kept to the point can be a great deal quicker and more productive. Having everybody in the room really is as productive as having nobody in the room. However, if you insist on meetings, this is a great way to make them less brain-dead.

The 25 Worst web-sites in the World, ever – I remember most of these when they launched. Truly horrendous, and in my opinion the number one spot holder deserves it – utterly awful website (I won’t spoil the surprise for you though). That said, none of these were as bad as Microsoft Bob in the ‘I… don’t… believe… they… did this…’ stakes.

50 favourite design resources – for those of you who, like me, find design something that has to be worked on as a skill rather than something that comes naturally, here’s a crib sheet.

The above cartoon is by Hugh MacLeod of gapingvoid.com – warning, these days he talks about a particular brand of wine a lot, but his cartoons are often cool. I’d quite like this one on my business card. Thanks to his nice licensing terms, I might be able to do that.

When he published this, he pointed to a story by Scoble about the bubble people are now seeing. There are some important things to note about this bubble:

1. It’s only being seen by those of us inside the industry. If I talk to most people outside of the industry about social sites, Web 2.0, Ajax, whatever, they haven’t got a clue what I’m talking about.
2. There is a long, long way to go before this one bursts, but I don’t think the web in 12 months time will look close to what we have right now – it won’t burst, it’ll just look more sensible.

Let me give some background to where I’m thinking here, and start by making a controversial point: Technorati sucks from a social software perspective. So does Google.

I think they suck because to be part of the network, to add to the community, you have to be a publisher. Most people don’t have the time, energy or inclination to become a one-person publishing band, or the money to become a 10-person publishing house. Nor do they want to make videos to put on YouTube or podcast their thoughts. Some people think publishing is something attention-seekers do, and they don’t want to be called attention-seekers. They’re not going to participate.

Of course blogging is attention-seeking, but it’s not a given that this is a bad thing. I want to draw attention to all sorts of things, and I’d rather get your attention by talking and educating you about Ruby on Rails or new technology, or innovation, rather than just put up a big billboard and take out radio ads saying “We Rock the Kazbar – Vagueware Ltd”, because I respect your attention way more than that.

So am I an attention seeker? Sure, of course I am. But I’m seeking attention for something I’m passionate about: my business, technology, innovation. I want to share it. You might not give two hoots about my business, technology or innovation, but you’ll be passionate about something else. People who realise that become bloggers.

However, as a result of a relatively small set of attention-seekers having all the kung fu right now, looking at the A-list of blogs – the ultimate list of passionate attention-seekers – is a pointless exercise. The only thing that is being used as a metric is how many attention-seekers like this group a bit more than a different group by way of the number of linked articles. This is a terrible metric to use.

Ultimately, it could be Google’s downfall as well. PageRank – whilst better than previous systems for ranking content on the Web – uses the worst metric possible: what publishers think is relevant, as opposed to what searchers and readers think is relevant (although they have taken measures to improve this).

What could be more useful is to find out what people of all persuasions – non-publishers included – like across the board. That’s why social bookmarking sites have got traction – it doesn’t take much to digg, reddit or del.icio.us something, you’re not some being publisher, you’re just some dude who clicked on ‘digg this’. That metric – people who were prepared to go ‘Yo! This guy has funk!’ – is a lot more useful. It means that people who are only a bit passionate get to play with less effort.

Except it’s still not great.

For starters, game theory gets all over this model, and sites like Alexa’s Top 500 can so easily be manipulated, all you end up doing is creating a whole new form of spam. As the social bookmark sites get attention from other niches outside tech – particularly porn – they’re going to have to deal with this problem.

The other problem is that it’s still not frictionless. Alexa tracks people who bother to install their toolbar, which most people really can’t be bothered to do. Digg and Reddit are contributed to only by people who can be bothered to click on a little button they remembered to save to their bookmarks bar, write up a few words and some tags and hit submit. It’s not as bad as publishing a blog, but it’s still effort.

All the Web 2.0/community sites are going to face this problem: social communities take work to build and you can’t build them, they have to build themselves. And once people realise they’re there, they are going to start spamming the heck out of them and game theory becomes game-on.

The potential though is huge. Reddit is doing something very interesting with niche sites – joel.reddit.com and so on – which has the potential not to just change the way we think about publishing, but how we think about search.

I’m not interested in what the Internet at large as a group of publishers thinks is the best result for ‘Ruby on Rails’, I’m interested in what a community I respect thinks is the best hit. It’s a better metric for me. If they get the sweetspot just right, the social bookmark sites can take market share away from mainstream search. Right now Joel has his own bookmarking site, but what if every community, every blog, what if they all had social bookmarking communities? Every blog becomes a search engine built by people you share a passion with – a niche community with value.

Even better, because this is so niche, you don’t care about scaling, you don’t care about game theory, you just care about trying to reduce the traction of participation. It needs no money, just passion. Passion is the steam engine of the next industrial revolution.

That’s where the Web 2.0 crowd need to head. This bubble is wrong because we think it’s about Ajax and Rails and reflecing logos. Technology and shiny buttons aren’t what is making the net different in the next 10 years. It’s the people, stupid!