Innovation in Software

Tim O’Reilly posted some extracts today from a New York Times piece on Jon Stewart, possibly one of the most famous faces of American comedy satire – and journalism – in the World.

What does this have to do with software innovation? Stay with me.

Here in the UK, his program is broadcast the day after it’s broadcast in the US on More4. I find myself twitching slightly if I’m not near a Freeview box around 8:30pm as a result. For a long time I’ve been trying to work out what this means to the future of journalism – thanks to my vested interest in such questions – but also how this translates into an era of Web editorial. What I’ve begun to realise is my fascination is with the disruptive innovation the Daily Show represents:

‘When Americans were asked in a 2007 poll by the Pew Research Center for the People and the Press to name the journalist they most admired, Mr. Stewart, the fake news anchor, came in at No. 4, tied with the real news anchors Brian Williams and Tom Brokaw of NBC, Dan Rather of CBS and Anderson Cooper of CNN. And a study this year from the center’s Project for Excellence in Journalism concluded that “ ‘The Daily Show’ is clearly impacting American dialogue” and “getting people to think critically about the public square.”’

A comedian who in his own words is “throwing spitballs” is the 4th most admired journalist in America? Here’s the real meat for me though:

“Most important, at a time when Fox, MSNBC and CNN routinely mix news and entertainment, larding their 24-hour schedules with bloviation fests and marathon coverage of sexual predators and dead celebrities, it’s been “The Daily Show” that has tenaciously tracked big, “super depressing” issues like the cherry-picking of prewar intelligence, the politicization of the Department of Justice and the efforts of the Bush White House to augment its executive power.”

When Outfoxed came out I was angry about how news had become entertainment specifically because journalists were not dealing with the big issues and producers were allowing those with very strong political views help a political administration skip over troubling questions by suggesting it was the questioners who were at fault.

Yet here we are with somebody using news as a platform on which to build comedy, attacking the establishment. Do I like this more because I agree with his politics? Or because it’s funny rather than nasty? No, I like it more than Fox because he uses the technique to point out the lies rather than promote them as truth.

So, back to software innovation. For the last year I’ve had this mantra: it’s not about the idea, it’s about the execution. It’s about making sure you know exactly what your intended outcome is, testing your assumptions every step of the way, and aiming for that goal. If that goal isn’t something tangible (money in the bank, simpler processes, helping people see truth), you’re not going to get anywhere.

And just like Jon Stewart, every programmer and software entrepreneur has the opportunity to disrupt and make a hit potentially by looking at something they hate and despise and using the techniques of their “enemy” to promote values of a bygone era (in Stewart’s case journalistic, in our case scientific).

I’m starting to get a hint from programmers I know that they realise that during the last decade of dot.com boom and bust and “Web 2.0” hype, they’ve lost some rigour in the process. I’m just as guilty as any one. I think we’re going to see a scientific renaissance in the web industry and some smash hits will emerge as a result.

I also think Tim O’Reilly’s point that potentially we’re going to see bloggers adopt the same techniques to try and push the envelope a little further is intriguing. If they do, they’re going to need the tools to help them do that. Perhaps UK bloggers could try and take the lead and reverse the trend of poor success rates around commercial blogging here in the UK.

We can all remember the futurology clips from the 1960s suggesting we’d all have hover-cars and silver jump-suits by now, so it surprises me that newspapers still try and predict the future. I definitely expected more from The Independent

What they fail to appreciate is:

• The forecasts have always been wrong in the past. There is no reason to be confident they will be right in the future. Not just immigration figures, but shifts in employment, prosperity, house prices, interest rates (which are managed by the central banks!), share prices, the weather tomorrow…
• They’re not taking into account movement in the opposite direction. Yes, your parents’ house in the South of France or Spain is considered immigration by the French.
• We actually need to get to 75 million in the UK by 2030 to keep the state pension system working as it does today, but they ignore that because it doesn’t fit the political agenda de jour.

I’m a liberal (in the European sense of the word), and I find the “Immigration is always bad” argument morally repugnant, but that’s not what I want to focus on in this article. Rather, I’m interested in how mainstream media goes about handling forward forecasts. They typically:

• Don’t ask questions about whether forecasted figures are accurate, or how they could be accurate
• Don’t ask questions about whether figures are representative of the complete picture, or even what “complete” means
• Don’t ask questions about what the forecast actually means in real terms beyond the gut reaction

In short, they’re terrible at understanding possible future outcomes and analysing them. If they were working in the technology industry they would instead know:

• Forecasting is inaccurate
• We never have full sets of data
• We can’t really know what it means until we get there

That would however make for lousy newspaper copy. “Immigration may or may not be a bad thing, or a good thing, depending on what actually happens” is not as catchy as “Immigration out of control”.

How do we educate politicians, journalists and the public about how best to deal with these figures though? This industry is about as good as it gets with future forecasting (although we struggle to see paradigm shifts and anything > 3 years out), so is there anything we can share that others might learn from?

[UPDATE]: Comments like the one I’ve approved below have started coming in. Hmph.

I know from past experience that whenever you talk about immigration, a bunch of BNP/nationalist nutjobs (who are very clued up as to how to find and comment on blogs that even lightly touch on the subject) tend to pass by and start making silly statements. I’ve let this one through to demonstrate my point – whilst glancing on racism in a way the author probably hoped I wouldn’t notice, it’s not as bad as others – but let me please, please stress: I’m not having a debate about immigration here, I’m discussing how the media handles forecasting in general. Immigration is the story today, but I’m talking about a trend that applies to everything from viral epidemics to what the weather will look like next week.

Please do not add comments of the “but we’re going to run out of space” or “the schools will collapse” vein here – they simply won’t get approved, as they’re not relevant to what I’m really talking about above. Making vague statements about “ethnicity” is likely to just make me think you’re an idiot.

Chris Morrison writes up some conclusions on a story suggesting that when Yahoo! told Congress last year that they didn’t know Shi Tao was a dissident and not really a spy, they might have been a bit disingenuous if not outright lying.

The truly exceptional part of his post is the knuckle-dragging moronic comments that Chris has been subjected to. Stuff like this:

“You spoiled brats need to understand that China is not USA. They don’t enjoy the freedoms and rights we do. The police/gov can arrest anyone at anytime with little or no evidence. Many are still wrongfully executed today.

So if you were Yahoo and the police and gov. agencies are knocking at your door what can you do? You either comply or you’re in violation of some bogus law.

As Ven stated, the Chinese Gov. is the problem here. You think any other company would have done something different/morally responsible?”

Superb proof that comments are completely futile if you needed it, but this content is just so astonishingly unintelligent I’m almost lost for words.

It seems to suggest that Yahoo! found themselves playing the part of victim here, that they were just ambling through China one day innocently scooping up buckets full of cash minding their own business and then found themselves being subject to an attack by the Government smashing batons over their heads.

We’re facing an uphill struggle against this kind of idiocy. People appear to have problems understanding that technology companies – the most globalised of all companies in today’s World – have moral responsibilities and instead prefer to blame it all on technocrats getting in the way of good ‘ole capitalist fun.

Yes, Yahoo! has to comply with local laws. That’s why they should be careful about which countries they decide to work in. If I throw an application out the door this afternoon, there is nothing stopping somebody in China using it until the Great Firewall team decide it’s “dangerous”. If I actively set up an office in Beijing and do the paperwork so that I have to comply with state police requests I’m agreeing to step over a line that has certain consequences.

My argument is that we should be careful about making that step with some countries.

Imagine if this were the 1980s and Yahoo! had been asked to help the South Africans track members of the ANC. Would we have said they were perfectly OK to comply with local laws, or would we be screaming that they should get the hell out of Dodge and just accept they aren’t going to make some money there whilst the people in charge are incapable of understanding human rights?

In the same way that IBM has been criticised for its role in 1930s Germany (it is alleged they helped the Nazis identify and round up Jews using their patented card index system) Yahoo! is going to find itself on the wrong side of history until they just accept that running a mail service in China is dangerous as hell. Google might censor search, but they don’t store anything of interest to the police because they tread that line carefully: Yahoo! just want market share no matter what.

It’s not just Yahoo! and Google who have to think about this: it’s something all software companies need to consider in the modern era. If we truly are the people responsible for this iteration of civilisation, do we not have a responsibility to have guide our decisions with a moral or political framework that guarantees certain basic rights to our users? And does running a service in China that requires personal information to be collected put that framework in jeopardy?

Now it looks like the morality at Yahoo! could be so warped that it might involve being less than honest with Congress. And don’t think this is all the fault of the management – they hired an ethics officer but at the last shareholder meeting, they were told by shareholders they couldn’t commit to not proactively censor content or create a committee on human rights in relation to their business: the shareholders are starting to look like the real scum bags here. Given that Yahoo! is having serious business problems with this strategy, perhaps it’s time for a change of direction. Interesting times indeed…