Innovation in Software

One of the problems with modern journalism is that some of the barriers taken for granted have broken down, and that can mean new ethical dilemmas being created every day.

Take the story of David Rohde’s kidnapping. Who he? You might very well ask. A reporter for the New York Times, he was kidnapped around seven months ago by the Taliban. You didn’t read about it in the papers? Well, no. Despite it being picked up by an Afghan news agency and being reported in some UGC news websites, the New York Times conducted a cover-up operation over the last seven months.

The only people who didn’t play ball the NYTimes were worried about were a couple (perhaps only one) of the Wikipedia editors who spent a reasonable amount of effort trying to insert one single reference to the kidnapping in Rohde’s Wikipedia article.

The Wikipedia team conspired to remove the edit and temporarily block the page from time to time. The New York Time have decided to point out how tricky dealing with this was by way of a free puff piece for Wikipedia and Jimmy Wales.

It’s an interesting case in how truth takes a back seat for a while, and raises some interesting questions for me about how exactly news organisations are meant to behave in a new era of constant information flow.

First, their reasoning for suppressing this information:

Times executives believed that publicity would raise Mr. Rohde’s value to his captors as a bargaining chip and reduce his chance of survival. Persuading another publication or a broadcaster not to report the kidnapping usually meant just a phone call from one editor to another, said Bill Keller, executive editor of The Times.

Well, that doesn’t seem very sound logic. Yes, if it had been splashed across CNN for a couple of news cycles because there wasn’t much going on that weekend, you’d have a problem. However, that wouldn’t happen. Even the original news stories that were published intimated journalists being kidnapped were not big news, and part of daily life in that part of the World.

In fact, the story not getting sympathetic coverage could well have caused more damage – why feed and keep a man who is worthless to you and his fellow journalists? If no ransom is possible,  wouldn’t it be simpler to just kill him?

Then there is their attempt to change history that irks a little:

Two days after the kidnapping, a Wikipedia user altered the entry on Mr. Rohde to emphasize his work that could be seen as sympathetic to Muslims, like his reporting on Guantánamo, and his coverage of the Srebrenica massacre of Bosnian Muslims. Mr. Rohde won a Pulitzer Prize for his Bosnia coverage in 1996, when he worked for The Christian Science Monitor.The Wikipedia editor in that case was Michael Moss, an investigative reporter at The Times and friend of Mr. Rohde who has written extensively about groups like Al Qaeda and the Taliban. Like many Wikipedia editors, he adopted a user name that hid his true identity.

“I knew from my jihad reporting that the captors would be very quick to get online and assess who he was and what he’d done, what his value to them might be,” he said. “I’d never edited a Wikipedia page before.”

With his editors’ blessing, Mr. Moss had already made similar changes to Mr. Rohde’s “topic page” on The Times’s Web site, and in both cases he omitted the name of Mr. Rohde’s former employer, because it contained the word Christian.

Woah, there! That’s some pretty hefty editing going on there. First, Michael Moss edits the page to make Rohde look more sympathetic to Muslims, under a pseudonym. Then he edits up the NYTimes.com topic pages, all the while trying to get rid of the mention of his previous employer.

I have to ask, why? The most prominent article on Rohde found through Google before the story of his escape and this cover-up broke over the weekend, points out quite clearly who his previous employer was. This was, in essence, a futile exercise that did not take or remove any information away from the Internet that was already out there, and simply made the NY Times look like they were practising their Stalinist air-brushing techniques.

I’m disappointed in all involved. I don’t think Rohde would have been killed if things had been left as they were. I don’t think a short mention on the evening news would even have happened. A couple of small pieces in competing papers pointing out his work in highlighting issues Muslims around the World faced might actually have helped him gain an earlier release. Either way, if he was still alive after seven months, there was little chance he would be killed at any point by his captors.

Wikipedia might be the biggest boy in town when it comes to UGC news content, but it won’t be for long. You won’t be able to suppress stories in future that are based in fact, and the final line of the NYTimes pieces:

…the idea of a pure openness, a pure democracy, is a naïve one.”

Harks of naïveness in itself: journalists are no longer gate-keepers to truth. Yes, lives are involved and everybody is glad that Rohde managed to escape over the last couple of days. However the guys sat in news rooms – foreign correspondents whose lives are on the line, even more so – are going to have to accept this behaviour just isn’t going to be possible a few years from now. The real question then is should it be?

There is an awful lot of consternation going on amongst the suits behind online news websites.

The problem, it seems, is a colossal error made by most newspapers in offering you all their content without asking for some money from you. Their reasoning is that this forced them into the route of monetising their websites through advertising. This has never been profitable from the day they first did it. Now, they argue, the time has come to change all this and you must pay.

I want to briefly point out how this happened, why it’s the news organisations’ own stupid collective faults, why people will never pay for this content, and how business models online will need to evolve over the next few years. I pick “News” as a sector, simply because that’s where my head is right now but it applies to any other sector as much.

I also want to point out that whilst I contribute to a blog over at The Manchester Evening News this isn’t about the MEN. This is about a global industry that is in trouble. I don’t know if a single word of what is written below applies to the MEN, because I don’t work for them and don’t have enough experience of the inside of the organisation to know what happens there. Maybe some of the below does apply, but I’d be surprised if it all does.

Let us travel back in time to the early 1990s. Back then, 28.8K modems were considered speedy and there were but a handful of ISPs offering dial-up access in the UK – typically for about £10/month – to the Internet. The main applications were mail and Usenet. The web did not exist.

But one bright day, it did, and then there was light, etc. There was a rush to this new territory and a few keen early adopters started to think about what this might all mean to newspapers.

Strangely, the gentlemen of perhaps the UK’s then most-traditional newspaper, The Daily Telegraph, were the first to consider publishing their content online. They decided then that advertising was the way to support this venture and so the first UK newspaper website was launched with all manner of gaudy, flashing images helping to prop up its cause.

Why did they do that?

Well, let’s think about how newspaper management typically think about their business. To you or I a newspaper is a source of news. To a journalist it is a means of employment that allows keen types to investigate the World around them.

To newspaper owners, it’s an advertising hoarding they sell space in, and in order to get people to look at it, they shove some news stories in next to the main content in as inexpensive a manner as possible.

That might sound cynical, but reality often does.

When Rupert Murdoch’s advisers were trying to convince him to buy MySpace the argument that sold him was “you’ll be selling advertising next to content you didn’t have to pay for the production of. What gets better than that?”. That mindset is dominant in online news.

When you buy a newspaper (if you still do), your money is not to pay the wages of the staff who produced the paper – the money is to offset distribution costs mostly. The printing, shipping and selling of newspapers is an inconvenient but necessary cost and in order to keep everybody from the newsagent all the way through to the inky-fingered man who produced the paper happy, you pay your pennies for the finished article.

The main source of income always has been and always will be, the advertising. In case you’re wondering, it’s the same story in the magazine industry.

Now, let’s move things online. The distribution costs of a website are tiny compared to that of a physical newspaper. If you were to choose the comparatively costly hosting services of Amazon’s EC2 cloud resource, and had 10 small server instances running and were fortunate enough to be shifting a few hundred gigabytes of data a day, your total costs would be less than $60 a day. That’s about £40. Compare that to the £650m needed for the printing presses of the modern paper editions.

Therefore if distribution costs are near enough to zero for online editions, why bother asking the customer to pay for them? It’s almost sound logic, except they then made a major, major error.

They gave the advertising away for free.

Phone any regional newspaper title in the country and speak to their ad sales team. The conversation will result in them offering you a rate card for the print edition. Sound sceptical about the costs and benefits. They will offer space online for free. Every time. I know, because I’ve spoken to quite a few ad sales teams in the last year.

In essence to secure the advertising for the print edition, they have in the past completely undermined the business they need to survive in the future. They have told every one of their advertisers that online adverts are not worth paying for.

Let’s move forward and look at the state of business affairs within the news sector today.

All of a sudden, as if by magic, the clouds have moved apart and revealed that their website is a cost-centre. In order to keep up with the Jones’ they have had to build dedicated online news teams, larger server infrastructures, bring programmers onto the staff and throw money at their website in order to remain attractive to shareholders and other investors. Slowly the ad sales team realise that in fact the future of their business is online and they start to unpick their years of collective suicide, slowly trying to build up the revenues they have worked so hard to destroy.

It’s all a bit grim. One newspaper group dominant in a Southern part of the UK almost sounds desperate when you phone them and enquire about online sales. I genuinely feel sorry for them sometimes.

In short, none of the people responsible for picking up revenues really know what they’re doing any more. The World has changed around them whilst they have stood still, and it’s all too much. They are thinking in terms of traditional newspapers and traditional ad sales, and struggle to find the revenues they need to break even. Slowly, they realise they can’t fix this. They need people to blame. The usual people they wag their fingers at are:

• The BBC. They get a license fee, they operate the largest online news websites and hey, it’s just not fair, right? They should stop!
• The advertisers for not spending enough, mainly because for 15 years they’ve been told that online adverts are basically free
• The audience for demanding free online news like some sort of self-interested collective mob (which they are!)
• Anybody and everybody but themselves.

But wait, there’s hope. Amazon’s Kindle is being thought of in some quarters as the “news iPod”. The people behind the curtain at major newspapers groups all over the World are lining up to try and sell you subscriptions to their content via eBook readers like the Kindle (and perhaps the iPhone and other mobile platforms), and then slowly their digital operations will move to subscription-only content. News websites will slowly shut down and become adverts for the print and subscription-only content via the new technologies sure to become dominant in coming years.

Except of course, the audience won’t care. They won’t sign up. Why would they if even a few websites remain free and open for business? Nobody at Wikinews is busy trying to find a business model that works. I’m pretty certain the BBC won’t be allowed to. That’s enough to wreck this plan.

And yet, the news organisations lumber on hopeful something will change soon. No changes in business structure, no clear thinking about how to go forward, no understanding of online business models.

Here’s a good example of bad thinking about what it takes to succeed online. Craigslist has about a dozen employees and clears about $100m a year in revenues, thanks to their UGC structure. That’s some margin! The BBC gets £3 billion a year in license fees, so by the same gearing (ignoring the obvious differences in structure and intent of a non-commercial entity like the BBC) they shouldn’t need more than perhaps a few hundred journalists. Do you know how many journalists the BBC employs? 7,000. Seven. Thousand. And they, for their budget, are considered quite an efficient news outfit.

So, let’s summarise where they’ve gone wrong so far:

• Newspapers didn’t understand the web when they got started
• The ad sales teams on newspapers took too long to understand online business models
• They then undermined their own business model
• Newspapers realised they need to take a new mindset and blame everybody else for where they are right now
• They then decide the option is to try and get customers to pay for content they don’t want to pay for
• Absolutely no consideration is made of what being an online newspaper consists of

Not sounding good is it?

What to do? It’s OK me sat here mouthing off about how stupid they all are, but can I offer some insight? Well, here’s some thoughts to people running newspaper groups right now:

• Online news is not the one-way broadcast medium you’re used to. Use your audience. Wikinews is cheap to produce. Your newspaper could be. Kagtum will be.
• Online ads are not the same as print ads and can’t be sold in the same way. I don’t know of a single news website in the UK where I can buy PPC ads with my credit card right now.
• Even worse, online ads aren’t very profitable anyway. Google ads work because they enhance the product. How do you make money whilst enhancing your own content? Price comparison, auction and digital distribution models have figured this out well.
• Customers do not want to pay for anything. Ever. You have to work hard to convince them it’s worthwhile. You’re doing an awful job of that, so stop trying to do it
• Invest in journalists, programmers, graphic designers, information visualisation specialists and people who understand online content. Fire the majority of your middle management who aren’t directly involved in producing content or ad sales.
• People will always value investigative journalism
• Think laterally about revenue in an online medium

It might all sound a bit extreme, and I’m sure many will criticise me for my naive attitude, but I say these things because I care. We need good journalism. Wikinews might be cheap to produce, but it doesn’t allow for investigative journalism, an aspect of news reporting which I believe is critical to democracy. We need newspapers.

I’m writing this because I want to both signal where I’m thinking in terms of my own projects, and because I’m also fed up sifting through dozens of articles a day by everybody in the industry from the Western US to the East of Asia all blaming the dire position of news today on a lack of subscription models. I think there is a future for online news, but I suspect it will need the current generation of newspapers to go bust for the new models to successfully emerge.

What fascinates me about all this though, is that there are so many people whose future relies on online business models, and yet they don’t understand what that means operationally. Interesting times.

Tim O’Reilly posted some extracts today from a New York Times piece on Jon Stewart, possibly one of the most famous faces of American comedy satire – and journalism – in the World.

What does this have to do with software innovation? Stay with me.

Here in the UK, his program is broadcast the day after it’s broadcast in the US on More4. I find myself twitching slightly if I’m not near a Freeview box around 8:30pm as a result. For a long time I’ve been trying to work out what this means to the future of journalism – thanks to my vested interest in such questions – but also how this translates into an era of Web editorial. What I’ve begun to realise is my fascination is with the disruptive innovation the Daily Show represents:

‘When Americans were asked in a 2007 poll by the Pew Research Center for the People and the Press to name the journalist they most admired, Mr. Stewart, the fake news anchor, came in at No. 4, tied with the real news anchors Brian Williams and Tom Brokaw of NBC, Dan Rather of CBS and Anderson Cooper of CNN. And a study this year from the center’s Project for Excellence in Journalism concluded that “ ‘The Daily Show’ is clearly impacting American dialogue” and “getting people to think critically about the public square.”’

A comedian who in his own words is “throwing spitballs” is the 4th most admired journalist in America? Here’s the real meat for me though:

“Most important, at a time when Fox, MSNBC and CNN routinely mix news and entertainment, larding their 24-hour schedules with bloviation fests and marathon coverage of sexual predators and dead celebrities, it’s been “The Daily Show” that has tenaciously tracked big, “super depressing” issues like the cherry-picking of prewar intelligence, the politicization of the Department of Justice and the efforts of the Bush White House to augment its executive power.”

When Outfoxed came out I was angry about how news had become entertainment specifically because journalists were not dealing with the big issues and producers were allowing those with very strong political views help a political administration skip over troubling questions by suggesting it was the questioners who were at fault.

Yet here we are with somebody using news as a platform on which to build comedy, attacking the establishment. Do I like this more because I agree with his politics? Or because it’s funny rather than nasty? No, I like it more than Fox because he uses the technique to point out the lies rather than promote them as truth.

So, back to software innovation. For the last year I’ve had this mantra: it’s not about the idea, it’s about the execution. It’s about making sure you know exactly what your intended outcome is, testing your assumptions every step of the way, and aiming for that goal. If that goal isn’t something tangible (money in the bank, simpler processes, helping people see truth), you’re not going to get anywhere.

And just like Jon Stewart, every programmer and software entrepreneur has the opportunity to disrupt and make a hit potentially by looking at something they hate and despise and using the techniques of their “enemy” to promote values of a bygone era (in Stewart’s case journalistic, in our case scientific).

I’m starting to get a hint from programmers I know that they realise that during the last decade of dot.com boom and bust and “Web 2.0” hype, they’ve lost some rigour in the process. I’m just as guilty as any one. I think we’re going to see a scientific renaissance in the web industry and some smash hits will emerge as a result.

I also think Tim O’Reilly’s point that potentially we’re going to see bloggers adopt the same techniques to try and push the envelope a little further is intriguing. If they do, they’re going to need the tools to help them do that. Perhaps UK bloggers could try and take the lead and reverse the trend of poor success rates around commercial blogging here in the UK.

This weekend just gone, I attempted to compete in Railsrumble 2007 with an application I call ‘Kagtum’. The idea of Rails Rumble is to take 48 hours to build a Rails app from scratch in a competitive scenario. At around 7:30pm BST last night I realised I wasn’t going to finish the app and so I asked to be withdrawn, as I explained:

Maybe it was the fact I decided to try and compete on my own, and therefore didn’t have the advantage of a team. Maybe it was the fact it took half the first day just to get a working application stack up on linode. Maybe it was the Saturday afternoon spent in the company of friends rather than coding. Maybe the idea was too ambitious.

Maybe I just wasn’t good enough.

Whatever it was, I didn’t get enough of my idea implemented on the weekend of the 8th and 9th September 2007, that I wanted other people to look at it. I failed. There is no app here.

The ideas I played with in those 48 hours though, intrigue me, and they will be worked on over the coming weeks and months. The end goal is going to either be very interesting, or an exercise in futility. If you want to find out which, you can keep an eye on the blog and I’ll be making announcements there.

I will be judging, and I look forward to seeing other apps, so good luck. Until next year…

The first day did kill me – linode was under heavy load (not surprisingly, with over 100 teams trying to get their application stacks set up) and the guidance we had been given by way of a screencast was inaccurate in places. Top tip: when you’re in a hurry leave the rdoc behind and always pass “-d” to gem install.

Anyway, I still hope to judge – and I’d advise anybody with an interest in innovation to look out for the announcement that you can sign up for judging and take a look at the apps that were finished – but I thought I’d talk about Kagtum a little bit here, because the core is almost done and I’m confident I can get a working app out of the door soon. I’m also tempted to open source it.

It all started about 2 years ago when I was left distinctly underwhelmed by Wikinews. The problems with wikinews are many and pretty obvious to anybody who spends a few minutes digging.

The primary problem to my mind is that they’re using a piece of software designed to build an encyclopedia to build a news website which means all articles are given equal footing. It seems reasonable that they should be given equal footing, until you realise that unlike an encyclopedia, not all news items are equal. A world-famous opera singer dying is not equal to a drunken brawl in my local town centre, and neither are equal to the Iraqi PM losing the confidence of the Iraqi citizenship.

However, the core idea – news written by, and for, everybody is a great idea. I’ve spent the last two years playing with lots of ideas in my head and watched emerging developments in the online news and journalism scene before I came up with an answer: quite simply it comes down to targeting relevance.

If I am in Manchester UK, there are stories that are local to me I’m interested in that somebody in New York doesn’t want to see. Likewise, there are stories happening on the other side of the planet which are important to me because they have an impact on me, or because they are in an area I have an interest in. The “perfect” news website will know this, and present just the articles I need to see. Ideally, I also don’t want to be bogged down with partisan and opinionated pieces – I want impartial, simple, Economist-news-page-style articles that give me the leader and then show me what is out there being written about it.

Thus was born the concept of Kagtum – the phrase “kag tum” means “to bring news” in Sumerian, the script of which is the oldest written language currently known to mankind. Kagtum will be a wiki news site that helps target articles based on relevance to you and your life. Relevance is everything.

The idea is quite simple, but the algorithm needs some polish before I can roll it out: we create a news story perhaps based on a report in MSM, or perhaps as a first-hand eyewitness account, that points to online sources if available. We then attach to that story some “impact profiles” based on location or a tags.

For example, a story happening on my street (say, planning permission for a new development) would have a geocoded location and an impact radius of the local neighbourhood. A story happening in 10 Downing Street would also be tagged with that location but could have an impact on the whole of the UK. Suppose the latter story was a policy announcement on Iraq – we’d add Iraq as a location impact as well.

I then login and give my location as my postcode or street, which is geocoded, and the software knows that the story in my street is relevant to me, so is the story in Downing Street. It knows therefore, what is relevant to each and every user and displays the appropriate stories.

Let’s suppose however I have no interest in Iraq. We can tag stories and users can also add tags to their profile that they’re very interested in or very disinterested in. If I said I wanted all stories marked “Iraq” to be pushed down the queue then its relevance to me would be lowered – it might still appear, just not as prominently.

In theory then, when I log in to kagtum, I would see stories about technology, politics and cricket, particularly with stories about my local neighbourhood (stories about technology in my neighbourhood would be even more prominent), whilst my friend who doesn’t care about anything but beer and football will see something perfectly tailored for his interests.

It may also be the case that there are multiple profiles for each user (home vs. work) and that a user can add multiple locations – where they live, where they work, where they grew up, where their parents live – and sees a mixture of stories about places they care about.

The biggest problem I had this weekend was developing the specifics of knowing which stories to show to each user. The problem isn’t hard algorithmically, but providing a technique that doesn’t harm performance and can scale to more than a few hundred users online at a time is proving a little tricky using standard ActiveRecord associations and using the methods baked into GeoKit by default.

There is also an issue of what we mean by “radius”. Saying “this story is important to everybody within 5 miles” is simple enough, but what if I say “to everybody within Greater Manchester”? I somehow need to know if a given longitude and latitude is within that district or not. The Radii Problem (as I came to call it whilst muttering to myself) is important and it’s difficult. I discovered it as I added a story in Washington that was important for the whole of the US – if I added a simple radius of 3,500 miles (to take in California) it of course also covered a huge chunk of Canada, Middle America, the Caribbean, the whole of the North Atlantic (including Ireland!) and most of the North Pole. For a story about domestic US politics, this is obviously needlessly “grabby”.

I have ideas on how to solve this problem, but they’re going to take a few weeks of playing with datasets from the UN and other agencies to be able to get them working smoothly.

There are other aspects I have planned for the site around developing narratives and helping individuals become kagtum journalists, but I’ll keep discussion of those for after the roll-out of both kagtum and the new vagueware.com.

I’ve turned comments on for this article, so if you have thoughts, ideas or suggestions, please leave them.