Innovation in Software

Disclaimer: it’s no secret I’m not a fan of Microsoft, and I know some of you are. This is just where I am at the moment, it’s not a troll but an observation. I seek constructive feedback only.

John Dvorak is possibly the crankiest man on Earth. Now he’s aiming it all at Microsoft.

Microsoft is a software company. It has been distracted too easily by the success of others in essentially unrelated fields. Here are but a few examples (and there are dozens more):

• Years ago in the pre-Internet era, AOL was the talk of the town, so Microsoft had to copy it with MSN. No money was made; no strategic advantage was gained.
• Netscape was the rage for a while, so Microsoft threw together a browser and got in that business. The browser was given away for free. No money was made; the strategy got the company in trouble with government trustbusters.
• During the early days of the Internet, new online publications appeared. Microsoft decided to become a publisher too, rolling out a slew of online properties including a computer magazine and a women’s magazine. They were all folded.

[snip another half dozen or so examples...]

This is a company that began making development tools for programmers, beginning with a programming language. Does anyone see a pattern here?

[...]

Maybe Microsoft cannot come to grips with the reason for its success. After all, Ballmer is not a computer programmer, and has never been too interested in software or computers and seems to want to run a media company.

Ballmer may get his wish by turning Microsoft into one, but I don’t think he’ll like it.

It’s true that Microsoft was taken a few twists and turns. Developing bad ideas is what Microsoft does, and have done for over 25 years. The only truly successful products they have in their stable – the products that finance the entire empire – are the Windows operating system(s) and Office. Nearly everybody expects both to take a massive hit on market share within a few years.

Hugh Macleod has, perhaps in the hope of getting Scoble’s old gig as Microsoft evangelist, tried to change the culture within from the outside with his blue monster meme. He’s had limited (but sometimes notable) success in the nearly three years since he started it, and I expect that might have been Microsoft’s last great chance: it was an excuse to change the culture into something more dynamic.

Talking to people within Microsoft there are two cultures: the old guard who want to run things as normal, and the newer breed who want to mix things up. The simple truth is senior management have seemingly let both sides down in the last decade (if not longer).

Without a fundamental culture change, and an ability to focus on core skills (rather than dancing everywhere and anywhere as Dvorak points out), means Microsoft are risking everything.

Nobody cares about Windows any more, because the applications of the future are on the web and the OS is becoming nothing more than a local file store. Nobody cares much about search beyond the engines they already use, no matter how much you try and get them to switch. Everybody hates the Zune. The development tools are over-complex, but that’s perhaps because the underlying libraries are over-complex and the bizarre insistence that an application written in Windows 3.1 should run smoothly in Vista makes developer life awkward.

So what are Microsoft’s core skills? Well, despite Visual Studio being a pig, it has a fan base. MSDN is loved by the people who love it, and as Apple realised with their ADC programme in the move to OS X, it’s those guys who are key to the future. Go and ask developers what they need to build the tools of the future and focus on it.

Apple took a gutsy move in clearing the decks with OS X and basically stopping support for System, but in the process they were able to focus the APIs to make programming for their platform much simpler, cleaner, more fun.

They targeted the very best developers on the planet, who in turn produced applications so desirable that “alpha users” wanted to buy Apple kit to run them. Go to a gathering of leading technologists, designers, writers or other alpha users today and the Windows machines will be notable by their absence (or extremely small presence). If the laptop hasn’t got an Apple logo on it, it’s odds on it’ll be running a flavour of GNU/Linux.

Microsoft need to do the same. They need to focus on the aesthetics of software, and take their base of developers and make them champions.

Then they need to think about how to help their customers become the very best customers they can be. When I sit down at a machine to work, to play, whatever, I don’t want to think about using a computer: I want to think about the job I’m doing. I want to think about how to get what I’m doing, done.

In short, Microsoft lost my business because BSD Unix and OS X allowed me to get to the pub sooner.

This needs to be the focus of the Office team: how do we make things so easy, users don’t even need to think about what they need to do for more than a few seconds before we’re helping them do it.

If the culture internally wakes up to the reality off campus that they need to change, and get senior management backing to focus on those changes, they can build a platform for the future that keeps Microsoft in the top flight for another generation. If instead they continue to stick their heads in the sand and think they can be any company they want to be, well…

Trying to shift to a monetisation strategy based on advertising in this economic climate is just pure foolishness, just as building a strategy on your competitor’s leading product is going to make you forget about making your own products the very best they can be. Microsoft’s current strategy is akin to Adobe announcing they’re going to launch a search engine: most of their base are going to ask very loudly “WTF?”

I suspect though that Ballmer will be allowed to continue playing in the sandpit that is Microsoft, the cultures will continue to clash, and nothing useful will be produced as a result. Potentially they’re going to find themselves in the same position as GM within a few years.

Shame. Who will I moan about when Microsoft goes under?

We sometimes take for granted the knowledge we have of how the Internet works. We know that an image in a search result might be linked to a site that has nothing to do with the image. We know that just because a reader of a blog comments on a post and links to a picture, it doesn’t mean the blog owner has endorsed or in any way taken ‘control’ of that picture.

We know this.

Some people though, aren’t quite as smart as us. They think that you have more control over how Google sees you than you do. They think that if you link to a picture you are ‘trying to take it over’. They don’t understand hypertext, they don’t understand indexing algorithms and they certainly don’t understand how this all applies in terms of copyright. Don’t believe me?

TechCrunch is currently dealing with perhaps the most technically inept man on Earth representing a photographer in an argument over online copyright and image distribution.

The problem is that he has a little knowledge – pictures can drive traffic, and that drives revenue – but not enough knowledge to understand what TechCrunch’s role is in this instance.

Even worse, he’s decided to act in a way I would consider unethical by phoning advertisers and threatening to name them in a lawsuit explaining he “just wanted to let [them] know”, in that I’m-doing-you-a-favour-don’t-look-at-me-like-I’m-a-leech kind of way.

This makes me come to the following conclusions:

1. If I ever need to hire a photographer, I’m never going to hire Beth Boldt as she clearly hires idiots to represent her legally (although he doesn’t appear to be a lawyer), and I really don’t want to deal with idiots working on her behalf
2. If you’re ever going to threaten to sue somebody, maybe you shouldn’t threaten Mike Arrington who is, you know, a lawyer, and knows what he’s doing… (top tip Mike learned at law school: use spell check before hitting ‘send’).
3. All of us have a responsibility to make sure the people acting on our behalf – personally, or within our companies – understand the issues as they really are.

If you’re working in a corporate environment in the UK, you should make sure at least some of your directors or somebody over at legal checks out Out-Law.com once in a while, and if you’re freelancing or a SME, its RSS feed should be part of your morning coffee ritual.

A few years from now, people new to the Internet will find it impossible to imagine an era before video online. What use is that content though, if your user is visually impaired? How do we go about indexing and searching it?

A few weeks ago IBM announced an “Accessibility Internet Browser for Multimedia” over at alphaWorks. It addresses the short-comings of JAWS or voice-enabled browsers, and is built on top to of the Eclipse Rich Client Platform as a stand-alone application. It means the play/pause buttons (normally unavailable to visually impaired users) are stripped out and made accessible, and that playback doesn’t ‘clash’ with voice-synthesised browser operation.

Developers can add metadata to reorganise or simplify the content, provide additional information, add special navigation or even provide audio descriptions for movies using XML metadata.

It’s this last point that I think is going to be interesting. If this tool – or one like it – becomes standard, and video producers are encouraged to mark-up their content properly, existing search and relationship algorithms can be applied to video content. Right now searching video content is pretty limited – maybe by making it accessible, we all benefit.

Almost twenty years ago a process was started that culminated in the recent announcement that the game of checkers (or ‘draughts’) can now be played flawlessly by computer. The technique is a brute-force attack, and so now there is a computer somewhere storing every single possible combination of pieces and able to work out the optimum next move in every scenario.

It turns out that the ‘best’ result will be a draw. It might not like look tic-tac-toe, but when you have mapped the complexity out enough, it is eventually just as futile. It’s going to take the computation and storage of 10²⁰ more positions than checkers took, but within a few decades we should have the computing ability to map out every possible game of chess. My prediction on that: white will always win if both sides play flawlessly.

Ron Evans predicts that it is:

“… only a matter of time before sufficient computing power allows the machines to contemplate eventualities of which we have not even postulated the existence.

The Singularity IS coming…”

“Only a matter of time” is a lovely prediction to lob into conversation. Sure, it’s only a matter of time. It could be 15,000 years mind, but we’re in no hurry…

I’m sceptical about this happening any time soon, simply because the entropy of the natural World is so vast and immense that it makes a chess game look like child’s play. Sure, us humans might have a problem understanding the size and scope of every conceivable move in chess, but compared to every conceivable possibility within the World around us, it’s nothing.

There are however, interesting applications we can start to conceive of now. Game theory has been abused by war-mongers in the Rand corporation to the point it no longer has any real credibility, but if the flaws are removed, it’s possible for computers to start finding new theories we had not yet considered. The impact within systems that are bound by rules and predictable behaviour (currency trading, for example) could be huge.

What really intrigued me however was something the creators of the ultimate checkers program had to create as a byproduct of their work. The limitations of current hardware meant they had to get innovative about how they actually stored all of those possible positions:

“For example, they stored the outcomes for the 39 trillion possible positions for endgames in a mere 237 gigabytes of computer-storage space, an average of 154 positions per byte. The mathematicians are now applying these techniques to bioinformatics, looking for ways to manage the massive quantity of data generated by the sequencing of genomes.”

We’re delving into work where the storage requirements are becoming immense, and the last decade of having “enough” storage for most work meant we didn’t need to get creative around storage and search algorithms. The future is perhaps not about taking generic algorithms and applying them to our data, but finding new mathematical models of representing what we need to and creating domain-specific algorithms.

The singularity might be a way off, but finding a way of getting there is going to be intellectually stimulating, regardless.

I read some pretty weird stuff out there. You might think it odd given I’m all into innovation that I rarely read the likes of Techmeme or Techcrunch – the truth is, there’s little innovation in either.

I want to look at the guys who spend days lying on their backs staring at clouds and thinking to themselves “that cloud looks just like a better way to moderate pseudonymous online discussions”. I prefer artist coders to mathematician coders: they have more interesting things to say. I’ve said this so many times it’s starting to sound like a cliché: I’ve learned more about software from architecture, art and philosophy than I ever have from my University notes on formal methods.

Occasionally, as I scour the websites of the research labs of the World, I encounter something a little bit “special” in the “let’s look at something unique and apply it to the computer industry” niche. This is no exception.

HP Labs have recently published (only appeared in my RSS today, but it’s dated the 12th July) a paper on the election protocol the city of Venice used to elect its ‘Doge’, or Duke/Chief Magistrate, if you prefer. It appears to have been published in the IEEE proceedings for their 20th Computer Security Foundations Symposium which happens to have taken place in Venice, 6th-8th July this year. It could all just be a bit of an in-joke.

However, there are some really interesting ideas in here.

The election of the Doge was – in modern terms – quite absurd. It consisted of 10 rounds of voting, and at the beginning the entire electorate (the Grand Council of oligarchs who were all men over 30) was eligible. Each round was alternately a random lot draw, and then an election. There is an outline of the process given in the paper, and you can see how long it must have taken to elect anybody.

Where the authors (Miranda Mowbray and Dieter Gollman) take it is to show that there are some qualities it would be sensible to consider when designing computer-based election procedures. Specifically they argue that consensus in asynchronous systems, recovery after network partitioning or indeed any scenario where a ‘lead’ has to be taken by some node might be all processes that benefit from such a system. Whilst absurd to us now, it has the virtues of stability and resilience to ‘gaming’ or corruption.

There are some real gems in here as they draw parallels between modern day computer security and this ancient electoral system. Part of the protocol required for a random ballot to ensure that the drawing was fair, was that the person doing the draw would be selected as “the first boy seen after prayer’s at St. Marks” by a particular member of the Council. Mowbray and Gollman describe releasing a young boy into the square at just the right time as a “protocol attack”. They go onto describe the protocol’s arduous and drawn out procedure as “security theatre” – something that anybody who has looked at recent anti-terror legislation is all too familiar with.

They go onto propose a slightly simpler protocol that could be used by computer systems and I’m left with a mind whirring with possible applications. This is the kind of paper that makes me wish I’d got the main Vagueware application online as I can suddenly find myself thinking of applications in search, auctions, community building and more.

This is what I mean when I talk about innovation in software.

Real innovation in this space is not about taking something you’ve seen on the Web and doing it slightly differently like thousands of others. It’s about taking a piece of politics, art, theatre, humanity, and finding something in it – a tiny piece of inspiration – that gives you an idea of how to make software better.

One of the most fascinating parts of looking at web server log files, has always been what search terms people type in to discover content on the site. There are few things that get you closer to staring into the eyeball of all humanity than seeing what they ask Google for.

I use the usual tools, but in recent weeks I’ve been trying out 103bees which, as well as providing all sorts of things similar to Google Analytics, throws you an e-mail every week with the search queries people have been using.

It took me a while to remember why it might be relevant to this site for somebody to type in “organised pigeon hunting”. It would appear I am not alone in finding some of my search traffic quite odd.