Innovation in Software

Back in February I had a little rant about business models in the news industry. A couple of things have prompted me to come back to this story. I want to be more positive and less ranty this time.

First, let’s get the something out of the way. Before this continues one more stupid, misguided step, I want to address the “linking is theft” attack being supported by some newspapers. Even my brother-in-law is starting to get worried about this, and he rarely shows an interest in the inner politics of online business.

Richard Posner has stipulated that linking to copyright content without compensating the publisher who paid for the content to be created, is the culprit of newspapers failing.

At first read, his argument seems almost sound. Almost.

It roughly breaks down like this:

1. Newspapers have fixed costs (staff salaries) that must be reduced if advertising revenues fall
2. Online news websites (as well as general economic conditions), are taking a slice of advertising revenues, thereby harming newspaper revenues
3. These new-comers are only able to operate because they piggy-back on the newspapers content
4. Ultimately this means newspapers go out of business, and the online operators will rake in the cash as consumers have no other source

The flaws are subtle and hard to spot unless you understand a couple of key points:

1. Online, linking is the only currency that counts
2. Linking is currency. It’s like cash. It’s why we trade it.
3. Linking to somebody is like paying for their advertising
4. Currency, online? It’s called linking

I had never heard of Richard Posner before he posted his article. Many of you won’t have, either. Yet there’s a link to his website up there. If you click on the link – the link I gave him in my blog article, without permission and without charging him a penny for advertising his content – I am sharing my readership (you, my valued friend!), with him. He gains. You may choose to subscribe to his blog. He may offer to sell you a t-shirt with a witty motto on it that you buy. I have given him the first thing you need to succeed in business online: audience.

He may choose to ignore that opportunity, but that’s his right.

Handily, as he has a trackback URL, some of his audience might discover me as well. See how this works for everybody? Incidentally if you’re reading this after clicking a link on Posner’s blog, would you like to buy a t-shirt? No? OK, back to business models of newspapers then…

It’s reckoned that Google through its search and news operations drives between 30%-50% of traffic to online newspapers. Imagine what would happen if it switched that traffic off tomorrow, unable to find a way to pay the fees newspapers demand of it. Imagine if the only way you could enter the news industry online was to pay for advertising. Imagine if not one piece of your archive would be available through traditional search. Imagine if in addition to your staff costs, you now had to pay out a couple of hundred thousand a month in advertising on Google, on billboards, on TV, or wherever you could. How does a sharp increase in costs help a business facing sharp decline in revenues survive?

Well, it doesn’t, does it? Quite.

What is worrying is that according to the Guardian he is well respected on the legal scene, and somebody, somewhere, might think this is a good idea. I have a horrid feeling newspaper owners are amongst the crowd cheering him on. I sincerely hope they fail – not only would it kill off online news, it would kill off news, period. I am constantly left astonished at how bad the thinking behind major institutions is. These guys help form societal opinion on major issues? Time to step aside…

Imagine my pleasure then, when another story came along that suggests, somebody, somewhere has got a clue.

Chris Anderson – editor of Wired magazine – was cited as stating:

With newspapers debating their future, the argument has been pitched as free versus paid-for models, but Anderson argued the real decision was free versus “freemium” – not about whether to charge, but choosing carefully which specialised content people will pay for.

[...]

Instead of working on growing the audience more, he believes publishers will need to grow their offerings. Right now, Wired provides three pricing tiers: free content on the web, about $5 for a magazine and 80 cents for subscribers. In the future, he believes Wired will have many tiers.

Even so though, the linking-is-evil campaign was stalking around at the back of the room:

Guardian editor-in-chief Alan Rusbridger asked about Google’s role in this freemium world; 40% of the traffic to its sites comes from Google.

Anderson said: “I consider that a gift, but papers consider it theft.” Newspapers could exclude Google from indexing their sites or could band together and charge Google to index their content. But it might be a self-defeating move.

“Newspapers need to be part of the conversation” on the net, he said. In the end, Anderson thinks that the democratising effect of the internet is a good thing, which will lead to a richer society, but, he added: “I don’t deny that it will get messy.”

Messy? Sounds good to me. Revolutions that have lasting impact normally involve guillotines and blood. Sorry newspapers, but your middle management strata are starting to look a little like the French royal court sometime around 1789, and we peasant revolutionaries are eyeing up your Bastille.

So, freemium, what does that look like?

Well, to be frank it’s going to be a bit strange for newspapers to get used to. Getting rid of most of their ad sales team is going to hurt, and to those working there I am sorry, but the bubble has burst and it’s time to move on. Advertising might bring in some revenue over time, but it’s likely to be more cost-effective to outsource that sales component to an external agency dedicated to online ad sales.

They then need to start thinking about what content they can get away with charging for. Strong candidates include going deep on content, say something like a standard news story of 400 words is available free, the longer 1,500 word version is available to paying subscribers or on a per-story charge basis. Content people are genuine fans of like crosswords could bring in revenues too.

However, there’s more than just content, there are service limitations to lift too.

For example online software applications normally give a free option away with limitations and then offer to remove those limitations in return for cash. It may be a project management tool where you can – for free – have one project and 2 users, but if you want to move all your projects and team onto it, you stump up the cash.

If the future of online news is to become more interactive and newspaper business models become more like those of applications, the limitations are available to exploit all over the place. For example, plenty of people would pay a small charge just to guarantee they’re getting the latest content: delay it for an hour to free users and see what happens.

Online applications have been playing this game for years. If you’re a newspaper exec, go and take a close look at the adult film industry online: they’ve perfected making money in a world of competitive free content. Follow their lead, and you have a proven model. Yes, it might seem distasteful, but that sector are masters of exploiting the audience into paying money without playing tricks on them or insisting nobody ever links to them.

I’m amazed it’s taken so long for an industry insider to spot it. I thought it was obvious, but didn’t mention it because I assumed everybody else had realised that this was the way to go. Boy, do I feel dumb right now.

Perhaps I’m wrong, I’m biased: it is this freemium model that is the basis of the Kagtum business model

We’ll aggregate, link and provide the ability for you to add/edit stories for free. But there will be areas you want to go into a little deeper or tools we can provide to make your life easier that will be available for a small fee. We think you’ll be compelled to want to hand over some cash for those tools and content because of their quality, and then we become your news source, not our advertisers.

“Ah”, you say, “but Kagtum isn’t live yet, is it? Come on, Paul, when is it launching?”. My lips are sealed. Sooner than you might think, but not as soon as I want. Watch this space.

Over the last couple of weeks I have spent more time than usual on the road. I have met some extraordinary people, heard some superb ideas and eaten some truly lousy food.

My notes run long, and would be hard to write up and do justice. I thought instead I’d offer some quick points and revisit the subject areas in depth at a later stage:

• Amazon consider themselves very much a technology business and their event at the British Museum convinced me that even if their tools are not the future of computer architecture, their thinking is: assume everything fails all the time; build resilience, not just redundancy; scaling is something you need to think about now, not when it’s too late; being able to pay for computing at a transactional level might feel 1960s, but who cares if it reduces capital outlay up-front?
• The Hadoop project along with MapReduce and Hive are likely going to be amongst the most important technologies in the next few years. They will do to the back-end operations of the web what Ajax did to front-end user interfaces. Start getting familiar with how they can be of use to you.
• Instead of carrying a phone, a laptop, a 3G dongle, power adapters, and all my other paraphernalia, I have recently resorted to nothing more than an iPhone and sometimes a notepad with pens. Even with an iPhone charger in the pocket, this is the future. Trust me, netbooks are so dead when people work out the right way to go to a conference even if they go for an Android device instead of an iPhone.
• The security guards at Longsight train depot are awesome. Don’t ask how I found out, but the guys at Alstom Transport get to play with the biggest kit going, and are bloody nice people to boot, especially regarding cough lost property.
• Everybody says SELinux is a pain in the backside to configure. Once you spend some time working it out, it’s actually not that awful – you just need patience and a plan.
• The whole “newspapers are dying thing” is getting old. It’s simple: this is a bubble that is bursting, just like the dot.com bubble did 10 years ago. It might be a bubble that took nearly 200 years to get here, but burst it will. News needs to move to a “post-industrial” model, and from what I can see journalists are amongst the first to get this. Interesting times.
• I’ve found when panicking about issues on deadlines that Oblique Strategies really helps. There is an iPhone app if you want it.
• Freedoma are having their launch party on the 13th (i.e. on Wednesday night) at Dough in the Northern Quarter. Just e-mail party [at] freedoma.com if you want to go along. Sounds like fun, and I’m trying hard to get out of a prior double-booking to make it.
• Strangest tweet of the last fortnight by me: “Drinking G&T from a can. This is what the Orient Express would be like if run by industrial manufacturers of sex toys.”

Oh, and Kagtum is getting a kick in the pants. All I’ll say now is that there are clues as to what I’m doing in the above notes – watch this space.

There is an awful lot of consternation going on amongst the suits behind online news websites.

The problem, it seems, is a colossal error made by most newspapers in offering you all their content without asking for some money from you. Their reasoning is that this forced them into the route of monetising their websites through advertising. This has never been profitable from the day they first did it. Now, they argue, the time has come to change all this and you must pay.

I want to briefly point out how this happened, why it’s the news organisations’ own stupid collective faults, why people will never pay for this content, and how business models online will need to evolve over the next few years. I pick “News” as a sector, simply because that’s where my head is right now but it applies to any other sector as much.

I also want to point out that whilst I contribute to a blog over at The Manchester Evening News this isn’t about the MEN. This is about a global industry that is in trouble. I don’t know if a single word of what is written below applies to the MEN, because I don’t work for them and don’t have enough experience of the inside of the organisation to know what happens there. Maybe some of the below does apply, but I’d be surprised if it all does.

Let us travel back in time to the early 1990s. Back then, 28.8K modems were considered speedy and there were but a handful of ISPs offering dial-up access in the UK – typically for about £10/month – to the Internet. The main applications were mail and Usenet. The web did not exist.

But one bright day, it did, and then there was light, etc. There was a rush to this new territory and a few keen early adopters started to think about what this might all mean to newspapers.

Strangely, the gentlemen of perhaps the UK’s then most-traditional newspaper, The Daily Telegraph, were the first to consider publishing their content online. They decided then that advertising was the way to support this venture and so the first UK newspaper website was launched with all manner of gaudy, flashing images helping to prop up its cause.

Why did they do that?

Well, let’s think about how newspaper management typically think about their business. To you or I a newspaper is a source of news. To a journalist it is a means of employment that allows keen types to investigate the World around them.

To newspaper owners, it’s an advertising hoarding they sell space in, and in order to get people to look at it, they shove some news stories in next to the main content in as inexpensive a manner as possible.

That might sound cynical, but reality often does.

When Rupert Murdoch’s advisers were trying to convince him to buy MySpace the argument that sold him was “you’ll be selling advertising next to content you didn’t have to pay for the production of. What gets better than that?”. That mindset is dominant in online news.

When you buy a newspaper (if you still do), your money is not to pay the wages of the staff who produced the paper – the money is to offset distribution costs mostly. The printing, shipping and selling of newspapers is an inconvenient but necessary cost and in order to keep everybody from the newsagent all the way through to the inky-fingered man who produced the paper happy, you pay your pennies for the finished article.

The main source of income always has been and always will be, the advertising. In case you’re wondering, it’s the same story in the magazine industry.

Now, let’s move things online. The distribution costs of a website are tiny compared to that of a physical newspaper. If you were to choose the comparatively costly hosting services of Amazon’s EC2 cloud resource, and had 10 small server instances running and were fortunate enough to be shifting a few hundred gigabytes of data a day, your total costs would be less than $60 a day. That’s about £40. Compare that to the £650m needed for the printing presses of the modern paper editions.

Therefore if distribution costs are near enough to zero for online editions, why bother asking the customer to pay for them? It’s almost sound logic, except they then made a major, major error.

They gave the advertising away for free.

Phone any regional newspaper title in the country and speak to their ad sales team. The conversation will result in them offering you a rate card for the print edition. Sound sceptical about the costs and benefits. They will offer space online for free. Every time. I know, because I’ve spoken to quite a few ad sales teams in the last year.

In essence to secure the advertising for the print edition, they have in the past completely undermined the business they need to survive in the future. They have told every one of their advertisers that online adverts are not worth paying for.

Let’s move forward and look at the state of business affairs within the news sector today.

All of a sudden, as if by magic, the clouds have moved apart and revealed that their website is a cost-centre. In order to keep up with the Jones’ they have had to build dedicated online news teams, larger server infrastructures, bring programmers onto the staff and throw money at their website in order to remain attractive to shareholders and other investors. Slowly the ad sales team realise that in fact the future of their business is online and they start to unpick their years of collective suicide, slowly trying to build up the revenues they have worked so hard to destroy.

It’s all a bit grim. One newspaper group dominant in a Southern part of the UK almost sounds desperate when you phone them and enquire about online sales. I genuinely feel sorry for them sometimes.

In short, none of the people responsible for picking up revenues really know what they’re doing any more. The World has changed around them whilst they have stood still, and it’s all too much. They are thinking in terms of traditional newspapers and traditional ad sales, and struggle to find the revenues they need to break even. Slowly, they realise they can’t fix this. They need people to blame. The usual people they wag their fingers at are:

• The BBC. They get a license fee, they operate the largest online news websites and hey, it’s just not fair, right? They should stop!
• The advertisers for not spending enough, mainly because for 15 years they’ve been told that online adverts are basically free
• The audience for demanding free online news like some sort of self-interested collective mob (which they are!)
• Anybody and everybody but themselves.

But wait, there’s hope. Amazon’s Kindle is being thought of in some quarters as the “news iPod”. The people behind the curtain at major newspapers groups all over the World are lining up to try and sell you subscriptions to their content via eBook readers like the Kindle (and perhaps the iPhone and other mobile platforms), and then slowly their digital operations will move to subscription-only content. News websites will slowly shut down and become adverts for the print and subscription-only content via the new technologies sure to become dominant in coming years.

Except of course, the audience won’t care. They won’t sign up. Why would they if even a few websites remain free and open for business? Nobody at Wikinews is busy trying to find a business model that works. I’m pretty certain the BBC won’t be allowed to. That’s enough to wreck this plan.

And yet, the news organisations lumber on hopeful something will change soon. No changes in business structure, no clear thinking about how to go forward, no understanding of online business models.

Here’s a good example of bad thinking about what it takes to succeed online. Craigslist has about a dozen employees and clears about $100m a year in revenues, thanks to their UGC structure. That’s some margin! The BBC gets £3 billion a year in license fees, so by the same gearing (ignoring the obvious differences in structure and intent of a non-commercial entity like the BBC) they shouldn’t need more than perhaps a few hundred journalists. Do you know how many journalists the BBC employs? 7,000. Seven. Thousand. And they, for their budget, are considered quite an efficient news outfit.

So, let’s summarise where they’ve gone wrong so far:

• Newspapers didn’t understand the web when they got started
• The ad sales teams on newspapers took too long to understand online business models
• They then undermined their own business model
• Newspapers realised they need to take a new mindset and blame everybody else for where they are right now
• They then decide the option is to try and get customers to pay for content they don’t want to pay for
• Absolutely no consideration is made of what being an online newspaper consists of

Not sounding good is it?

What to do? It’s OK me sat here mouthing off about how stupid they all are, but can I offer some insight? Well, here’s some thoughts to people running newspaper groups right now:

• Online news is not the one-way broadcast medium you’re used to. Use your audience. Wikinews is cheap to produce. Your newspaper could be. Kagtum will be.
• Online ads are not the same as print ads and can’t be sold in the same way. I don’t know of a single news website in the UK where I can buy PPC ads with my credit card right now.
• Even worse, online ads aren’t very profitable anyway. Google ads work because they enhance the product. How do you make money whilst enhancing your own content? Price comparison, auction and digital distribution models have figured this out well.
• Customers do not want to pay for anything. Ever. You have to work hard to convince them it’s worthwhile. You’re doing an awful job of that, so stop trying to do it
• Invest in journalists, programmers, graphic designers, information visualisation specialists and people who understand online content. Fire the majority of your middle management who aren’t directly involved in producing content or ad sales.
• People will always value investigative journalism
• Think laterally about revenue in an online medium

It might all sound a bit extreme, and I’m sure many will criticise me for my naive attitude, but I say these things because I care. We need good journalism. Wikinews might be cheap to produce, but it doesn’t allow for investigative journalism, an aspect of news reporting which I believe is critical to democracy. We need newspapers.

I’m writing this because I want to both signal where I’m thinking in terms of my own projects, and because I’m also fed up sifting through dozens of articles a day by everybody in the industry from the Western US to the East of Asia all blaming the dire position of news today on a lack of subscription models. I think there is a future for online news, but I suspect it will need the current generation of newspapers to go bust for the new models to successfully emerge.

What fascinates me about all this though, is that there are so many people whose future relies on online business models, and yet they don’t understand what that means operationally. Interesting times.

This weekend just gone, I attempted to compete in Railsrumble 2007 with an application I call ‘Kagtum’. The idea of Rails Rumble is to take 48 hours to build a Rails app from scratch in a competitive scenario. At around 7:30pm BST last night I realised I wasn’t going to finish the app and so I asked to be withdrawn, as I explained:

Maybe it was the fact I decided to try and compete on my own, and therefore didn’t have the advantage of a team. Maybe it was the fact it took half the first day just to get a working application stack up on linode. Maybe it was the Saturday afternoon spent in the company of friends rather than coding. Maybe the idea was too ambitious.

Maybe I just wasn’t good enough.

Whatever it was, I didn’t get enough of my idea implemented on the weekend of the 8th and 9th September 2007, that I wanted other people to look at it. I failed. There is no app here.

The ideas I played with in those 48 hours though, intrigue me, and they will be worked on over the coming weeks and months. The end goal is going to either be very interesting, or an exercise in futility. If you want to find out which, you can keep an eye on the blog and I’ll be making announcements there.

I will be judging, and I look forward to seeing other apps, so good luck. Until next year…

The first day did kill me – linode was under heavy load (not surprisingly, with over 100 teams trying to get their application stacks set up) and the guidance we had been given by way of a screencast was inaccurate in places. Top tip: when you’re in a hurry leave the rdoc behind and always pass “-d” to gem install.

Anyway, I still hope to judge – and I’d advise anybody with an interest in innovation to look out for the announcement that you can sign up for judging and take a look at the apps that were finished – but I thought I’d talk about Kagtum a little bit here, because the core is almost done and I’m confident I can get a working app out of the door soon. I’m also tempted to open source it.

It all started about 2 years ago when I was left distinctly underwhelmed by Wikinews. The problems with wikinews are many and pretty obvious to anybody who spends a few minutes digging.

The primary problem to my mind is that they’re using a piece of software designed to build an encyclopedia to build a news website which means all articles are given equal footing. It seems reasonable that they should be given equal footing, until you realise that unlike an encyclopedia, not all news items are equal. A world-famous opera singer dying is not equal to a drunken brawl in my local town centre, and neither are equal to the Iraqi PM losing the confidence of the Iraqi citizenship.

However, the core idea – news written by, and for, everybody is a great idea. I’ve spent the last two years playing with lots of ideas in my head and watched emerging developments in the online news and journalism scene before I came up with an answer: quite simply it comes down to targeting relevance.

If I am in Manchester UK, there are stories that are local to me I’m interested in that somebody in New York doesn’t want to see. Likewise, there are stories happening on the other side of the planet which are important to me because they have an impact on me, or because they are in an area I have an interest in. The “perfect” news website will know this, and present just the articles I need to see. Ideally, I also don’t want to be bogged down with partisan and opinionated pieces – I want impartial, simple, Economist-news-page-style articles that give me the leader and then show me what is out there being written about it.

Thus was born the concept of Kagtum – the phrase “kag tum” means “to bring news” in Sumerian, the script of which is the oldest written language currently known to mankind. Kagtum will be a wiki news site that helps target articles based on relevance to you and your life. Relevance is everything.

The idea is quite simple, but the algorithm needs some polish before I can roll it out: we create a news story perhaps based on a report in MSM, or perhaps as a first-hand eyewitness account, that points to online sources if available. We then attach to that story some “impact profiles” based on location or a tags.

For example, a story happening on my street (say, planning permission for a new development) would have a geocoded location and an impact radius of the local neighbourhood. A story happening in 10 Downing Street would also be tagged with that location but could have an impact on the whole of the UK. Suppose the latter story was a policy announcement on Iraq – we’d add Iraq as a location impact as well.

I then login and give my location as my postcode or street, which is geocoded, and the software knows that the story in my street is relevant to me, so is the story in Downing Street. It knows therefore, what is relevant to each and every user and displays the appropriate stories.

Let’s suppose however I have no interest in Iraq. We can tag stories and users can also add tags to their profile that they’re very interested in or very disinterested in. If I said I wanted all stories marked “Iraq” to be pushed down the queue then its relevance to me would be lowered – it might still appear, just not as prominently.

In theory then, when I log in to kagtum, I would see stories about technology, politics and cricket, particularly with stories about my local neighbourhood (stories about technology in my neighbourhood would be even more prominent), whilst my friend who doesn’t care about anything but beer and football will see something perfectly tailored for his interests.

It may also be the case that there are multiple profiles for each user (home vs. work) and that a user can add multiple locations – where they live, where they work, where they grew up, where their parents live – and sees a mixture of stories about places they care about.

The biggest problem I had this weekend was developing the specifics of knowing which stories to show to each user. The problem isn’t hard algorithmically, but providing a technique that doesn’t harm performance and can scale to more than a few hundred users online at a time is proving a little tricky using standard ActiveRecord associations and using the methods baked into GeoKit by default.

There is also an issue of what we mean by “radius”. Saying “this story is important to everybody within 5 miles” is simple enough, but what if I say “to everybody within Greater Manchester”? I somehow need to know if a given longitude and latitude is within that district or not. The Radii Problem (as I came to call it whilst muttering to myself) is important and it’s difficult. I discovered it as I added a story in Washington that was important for the whole of the US – if I added a simple radius of 3,500 miles (to take in California) it of course also covered a huge chunk of Canada, Middle America, the Caribbean, the whole of the North Atlantic (including Ireland!) and most of the North Pole. For a story about domestic US politics, this is obviously needlessly “grabby”.

I have ideas on how to solve this problem, but they’re going to take a few weeks of playing with datasets from the UN and other agencies to be able to get them working smoothly.

There are other aspects I have planned for the site around developing narratives and helping individuals become kagtum journalists, but I’ll keep discussion of those for after the roll-out of both kagtum and the new vagueware.com.

I’ve turned comments on for this article, so if you have thoughts, ideas or suggestions, please leave them.