Innovation in Software

There’s a lot of interesting things happening out there right now. HTML 5 is about to make a whole suite of new applications possible thanks to:

• Much better rendering of graphics on the fly
• Client-side storage of application data
• Drag-and-drop interfaces that make web apps feel more desktop-like

But there has to be something we are missing out on that is niggling us all at the back of our collective group-think mind. Perhaps watching the Google Wave introduction got you psyched about something that suddenly became possible. Perhaps the very way the web inherently works bothers you, and you envisage a new platform.

I’m interested in hearing about it now the comments are getting a little bit of love across articles. Go crazy. Throw them in there…

Over 2008 some remarkable technologies emerged quietly that seem to be gaining traction within the industry. Whilst around for years, I am confident their time is about to come proper.

If you were to ask most people on the street who the most innovative technology firm was of 2008, you are likely to see a familiar list: Google, Apple, maybe Microsoft or perhaps some of the smaller outfits that have crossed into the mainstream like Facebook or Twitter.

Few people will mention Amazon.

In fact, if you point out that Amazon is right now perhaps one of the most innovative technology firms on the planet, people will raise an eyebrow and ask “What? The online shop? Where I get my my books and DVDs from?”.

This typical reaction is perhaps caused by only industry participants having seen so far just how Amazon are disrupting the economics of doing business online. Now anybody can have access to infinite storage arrays, huge compute clouds, masses of humans to complete complex tasks and distribute content across the globe as fast as possible, all without a penny of capital expenditure: you pay only for what you use. You can even send your physical products to be stored in Amazon’s warehouse and they will for a fee handle order fulfilment for you, again only paying for what you use.

Capital expenditure is a start-up company’s biggest problem. Remove it and suddenly anything becomes possible in the start-up World. This is big. Very big.

The real beauty is perhaps the fact Amazon made this move originally to use their own infrastructure more efficiently. If Amazon has lots of spare compute capacity ready to serve pages during their busy season (the run up to Christmas), why not lease it out the rest of the year? And yet this strategy has started to offset their own infrastructure costs so much I wouldn’t be surprised if within a few years their operational costs tend towards zero. The most powerful e-commerce platform on the planet, and everybody else is paying for it for them.

This has caused people to sit up and notice. “The Cloud” is now the hottest buzz term in the industry and all players are trying to figure out a strategy to compete.

One of the issues is that Amazon’s infrastructure is not as simple to use as it could be. Plenty of firms see a gap to try and make things simpler: one of the biggest complaints about S3 is that you need to use custom APIs instead of open standards like SFTP or WebDAV; EC2 needs a more complex understanding of systems administration and data storage than traditional models; for many applications it’s overkill or too generic, and so on.

If you break the components needed for a web application into its constituent parts from platform and compute capability through to storage services, you realise that at each level there are numerous companies trying to find a place in this market from Google and Microsoft through to unknown start ups, some of whom are attempting to make access to other cloud services easier to use and therefore are some sort of “meta-cloud” service.

This is thought to be a paradigm shift in how developers think about developing and delivering applications, but what we have seen to date is likely just the tip of the iceberg. For a number of years now traditional engineering firms (notably Rolls Royce in particular), have realised substantial revenue growth comes not from product sales – competitors can easily counter advances in product development – but in services. It seems the computer industry is starting to cotton on, and companies like AMD are thinking about how to ride the Cloud hype into the services sector. Even Microsoft are considering versions of Windows that users pay for by the hour.

It is perhaps because these service revenues lock a user into a provider’s business that some point out the dangers but I believe in time we will allay such fears by changing how we describe, define and use cloud capacity: it’s perfectly possible for us to control our own data and rent storage and compute capability as we need it, perhaps without realising that is what we are doing, without surrendering our rights and privacy. It is not yet a trivial job to do so, but surely over 2009 we are likely to see services emerge that allow consumers to harness cloud concepts and capabilities without needing to understand the detail.

Ultimately though, the real benefit over the years ahead will be the possibilities these services offer the programmers wise enough to harness them: without the CAPEX requirements, the only limitation developers seem to encounter is that their imagination struggles to break free from the bonds that have dominated careers to date. It is hard after decades of worrying about RAM, storage and CPU limitations to have all them removed, or at a minimum re-shaped. This is the beginning – if we can struggle to imagine it – of something huge.

So whilst I’m thinking opportunity, it seems the incumbents in the web application industry are starting to separate the wheat from the chaff, or at least according to ReadWriteWeb, Google is at any rate.

I’m actually genuinely surprised at the announcement on Google Video simply because it’s a better service, despite being less popular.

If Yahoo! do this, expect it to become a usable service again.

Interesting times…

Whilst I’ve worked in data centres before – and am all too familiar with how hot, noisy, industrial and dangerous they can be – I sometimes forget how the software industry I now work in has an industrial footprint in those rooms. It’s easy to think of my business as being ‘clean’, because the dirt is so well hidden.

Plans for Google’s new data centre in Dalles, as the blueprints published by Harper’s shows, should remind us just how industrial our business really is.

Combined with the annotation by Ginger Strand, we get a picture of how big this data centre is. Three buildings of over 68,000 square foot each and electricity consumption equivalent to that needed to power 82,000 homes, a third of which will be used just to keep the building temperature at a reasonable level.

Thanks to its location much of the energy used every day will be supplied via hydroelectric power, however its very existence has caused other technology firms to up their data centre spending, and it’s unlikely all of that capacity will be run on renewable power. And besides, every watt of clean energy powering a server is a watt not powering a domestic home.

It’s also worth remembering this isn’t “the” Google data centre. It’s “a” Google data centre.

For years now they have been pushing racks into peering sites and DCs around the globe as well as smaller facilities of their own – an estimated million servers are out there running Google sites, and there are more data centres planned by Google and their competitors over the next four years. Already data centres consume more power in the United States than the army of some 100-million-plus American monster-sized televisions. As the magazine itself says, the Web “is no ethereal store of ideas, shimmering over our heads like the aurora borealis. It is a new heavy industry, an energy glutton that is only growing hungrier.”

Better virtualisation of servers is going to help, but there’s a limit to how much you can virtualise. Is the time now right for us to get smarter again about how we use clock cycles? Is the efficiency-first stance of programming we’ve consigned to the era of the 8-bit machine now going to become fashionable again?

Maybe though, we could do a little to educate the public to make use of this vast industry a little more efficiently. Does the quest for the top 100 current hot trends at Google really suggest that we’re using this power wisely?

Via RoughType

It’s taken a while for Google to see sense, but they’ve finally started including the maps their user base have been creating on top of their platform in their own search results.

Collective, localised information gathering is a big step forward, and is what Tim O’Reilly really meant when he first talked about “Web 2.0” all that time ago (OK, for a tired man like me, it seems a long time ago). It’ll take a while to get traction, but once this switch is noticed it will fundamentally change the way we think about map data.

There is however a problem – how do we make a decision of trust on a user-contributed map? Google are marking out community maps in the result so you can make your own mind up, but what would be even better is if once a map is started the community could rate and even – wiki-style – improve a set of map data?

I still think that the OpenStreetMap community has answers to questions Google have yet to ask.

Some time ago I setup an online bookshop with Amazon.co.uk and their aStore tool. It’s not had much love for the last six months, but I’m going to be putting a fair bit more time into it in the next couple of weeks. If you have any suggestions for products or books that should be featured, please let me know in the comments.

The observant will also notice that as of today, at the bottom of each post above the comments is a little ‘tag cloud’ of relevant Amazon products. Amazon’s algorithms are a little whacky at times, so it might take a while for genuinely useful products to show up. I’m tempted to change this into something more traditional, but I want to see how this new kind of advertising might work.

I also intend in the next few weeks to introduce a very small Google AdSense block onto individual story pages placed between the headline and the article body. If you’re reading this site through the feed – and if you can’t or don’t want to use a feed reader, you can now subscribe to feed updates via email using the form in the sidebar – you won’t see any adverts, ever.

These moves are just meant for the 11,000+ people who pass by here every year without ever stopping by again: they obviously didn’t get any value out of what I wrote, so perhaps somebody else can help them.

If people find these moves really intrusive, I’ll obviously reconsider them.

In an article in Harvard Business Review Johnathan Zittrain takes a look at the risks facing the Internet in the immediate and medium term and how best to tackle the issues.

It’s interesting to me that his argument can be boiled down like this:

1. Technologies that can be used unintentionally by others to build useful things (generative technologies) are great.

2. The PC is great because of this. So is the Internet.

3. Because they are so generative people are able to create technologies that are actually harmful.

4. In order to avoid harmful software being run on their PCs connected to the Internet, users may choose to use less generative technologies, and move to appliance-like systems or services (e.g. iPhone, or only ever staying within Google/Yahoo sites)

5. We need to stop them adopting appliance-like systems in order to keep the Internet strong.

He goes about spelling that out in a few thousand words, but his ideas on how to do all this are a little light and tucked up at the end almost like an after-thought. It’s ideas though that I’m most interested in here, so here they are:

• Netizenship

  He suggests that all code should be subject to a Wikipedia-like editorial process where self-chosen experts vouch for whether the code you download and run on your system is harmful or not.

  This sounds novel, but I’m concerned about how such a system could be gamed by nefarious types. It may be that you would need to establish a trust network with certain users, but this would require you to expose information about the software you’re running: giving up your privacy to people who can name you is considered a greater threat by many than giving it up to criminals.

• Virtual Machines

  You download software and you put it in a special “container” on your machine where it can destroy what it wants to: it can’t get out to destroy your precious data. Once it has been running for a while and you trust it more and more, you can move it over into the “special” area where it has more rights.

  One question: why would I ever want to run the software outside of the container? Why not just provide a proper security model for all applications running on the system and take it from there? Unix sorted this one out better than Microsoft ever has in the 1970s, and with jailed/container environments from the BSDs and Solaris likely to hit the desktop Unix market within a couple of years, this might have some traction.

• More help from ISPs

  Zittrain argues here that ISPs turn a blind eye to zombie machines because they don’t want to help their users. This is completely and totally wrong.

  About a year ago I was in the ops center of a major multi-national ISP, sat with their abuse team watching how they worked. I was being considered for a contract to help them automate their processes (a contract cancelled as it was announced the ops team was to be relocated down South shortly afterwards).

  One of the jobs they took on was handling the automatic e-mails sent to them by AOL informing them of spam received by a machine on their network. They would then cross-correlate the IP address to customer records by hand, and give the customer a call and talk them through de-zombie-fying the machine. This *is* industry standard practice. ISPs who didn’t do this would eventually just find themselves being disconnected by their peers as they became more and more of a harbour for zombie machines.

  What would be better is more ways for ISPs to handle this automatically and to establish better trust frameworks. I also think that ISPs could relatively trivially prevent the most common spyware passing through their transparent reverse proxy cache boxes without causing major damage – the problem would be whether such action would be legal.

  I also think one area that ISPs could improve is education: a lot of people are pretty naive when they first get online and it’s only once the real story is told (or experienced first-hand) that users wise up. I think that at current rates enough people will be clued up enough to stop downloading spyware within a decade, but that’s a dangerous prediction, and there’s nothing stopping ISPs getting into the mix.

• Network neutrality for mashups

  This is an idea that sounds simple in practice, but actually defeats the point of generative technologies being adaptable, flexible, unregulated and novel.

  The premise is simply that if you write an application for a an API, your application is locked into that API’s vendor. They could withdraw service completely or just for your API key, or they could decide that they want to change the API at any time thereby breaking your application. In other words, if you invest time in developing to Google’s code base, you are suddenly beholden to Google: you can’t switch to Yahoo! or Microsoft APIs instead. This appliance-like style of development suddenly makes the “generative technology” of the Internet look quite stale compared to say the PC revolution.

  Zittrain argues that certain basic functions should be standardised so that once you write the application, it can work with similar services from other providers. If you write a Google maps mashup, changing it to work with Yahoo! should be as simple as changing a URL and an API key reference.

  It sounds simple, and standardisation is sure to come about over the next five years, but it’s actually a major undertaking. It would require all the major players to agree a baseline set of functionality, and that’s unlikely to happen with a group of commercial players.

  Where standards have arisen on the Internet – the IETF or the W3C – it has been a group of people acting in the spirit of non-commercial interest. Sure, the IETF meetings are dominated by Cisco, Juniper, Nortel and the ilk, but they are adopting a process first initiated by academics and amateurs: they’re second to the party, and are kept in their place by the tradition of open collaboration the IETF encourages.

  To create a standards body for web APIs now would require open source developers to somehow subvert the progress made by the commercial guys, or for the companies to stop behaving the way they have done for the last decade or so. It is an onerous task.

I think the error Zittrain is making here is that he believes these are problems that need to be solved in a way that is 100% efficient, or near 100% efficient. He also seems to be forgetting the human factors.

In terms of protecting users from harmful software, I believe this problem will go away within the next five years. It’s a dangerous prediction, but I believe that once users become more aware of the risks of software they are more likely to question where they source it from. It could be argued that without some kind of trust metric start-ups might find it hard to get traction, but start-ups have always benefited from the attention of early adopters who are clued up enough to know how to assess the risks of software like this.

In other words: if it’s getting good blog press, it’s OK to download. Most companies have woken up to the fact that you can lose every shred of credibility on the Internet within a day or less now, and credible company are not likely to push code that is going to harm your machine. By 2012, companies prepared to compromise their credibility will likely have no future. Zittrain’s assumption is a little like that of Cold War game theory: everybody is out to harm you, and the only right course of action is trust with verification, or outright suspicion.

As for the issue around APIs, again I think Zittrain is missing the human angle here. Yes, Google or Yahoo! could shut down their APIs at any point in time. Google have in fact stopped supporting an API or two in the past, but were careful in how they did so: they just stopped taking on new users for that API. If they were to leave developers high and dry, developers would feel the trust they had in the company had been broken, and they would make the effort to switch. Switching might be a pain, but it’s not impossible: most mashup apps out there right now are almost toy-like in their simplicity and can be recoded to a new API within a few hours. Once that happens, Google may as well drop all their APIs, and they know that.

I think eventually some form of standardisation is going to have to emerge, but all I can say about that with any degree of certainty is that it is unlikely to come from the commerical players.

According to CNET Eric Schmidt is saying filtering of copyright material on YouTube is about to be turned on.

I say that is an amazingly good thing for the Web in general, and UGC sites in particular.

If you go to YouTube today it is full of clips from mainstream media interspersed with attention-seeking idiots and experiments with Mentos and Diet Coke. Right now, mainstream media have nothing to fear from YouTube because it is merely TV cut up and made more convenient for people with ADD. By demanding that MSM-sourced material is removed, they are relinquishing control and demanding that the users compete with them directly.

If we get rid of the mainstream media from UGC video sites, we’ll actually see a proliferation of UGC. Nature abhors a vacuum, and nothing fills it quicker than YouTube users. I think people will quickly get bored of op-ed pieces to camera, and we’ll see a more creative spirit flourish because suddenly the audience will be looking for it, rather than just finding clips of comedy shows and watching them in lieu of doing something more creative instead.

The week of the 23rd April through the 29th is TV Turnoff Week and my own experience shows a proliferation of creative activity when my time is not being stolen by a piece of furniture. Content filtering on UGC sites will have a similar effect – when you can’t watch clips of Family Guy any more, you might try and fill that time with producing something yourself, and if that happens in just 1% of cases, we’ll see a sharp uptick in quality UGC found on the video sites.

People shouldn’t bemoan copyright material filtering on YouTube – they should applaud it, and encourage it as the dawn of a new era in users taking control of the media they consume. Viacom are willing Google to ensure Viacom can’t control their audience any more: that’s to Viacom’s detriment and to our benefit. Viva la revolution! :-)

Do you remember when, as a geek, you thought Eric Schmidt was cool? Maybe it was about the time that he was working at Xerox PARC. Maybe it was when he was working for Bell or Zilog. Maybe it was when he led the development of Java – man, actually, he deserves a slap for that one – or when he moved to Novell. In recent months, his straddling of Google and Apple has been analysed in some depth around the blogosphere.

The problem is, as a businessman, his strategy is all wrong. If you’re a Google shareholder, you should be asking quite serious questions about whether he really understands the economics of software development in a consumer market correctly.

The “road to Damascus moment” for me was when I read that he thinks mobile phones should be paid for by advertisers and that Google will be lining up to deliver the advertising platform needed. This is an idiotic business model. Thinking about it, I now realise that their entire business model is flawed and can be defeated by somebody offering equivalent execution of a product/service but with a low-cost subscription in lieu of advertising. Just because something is free, it doesn’t mean people will want it if it comes with adverts: if you don’t believe me, ask ITV.

The problem with advertising as an economic model behind any business, is that it requires you, the advertiser or publisher, to distract your customer/target/victim. If I build a web-mail system and say “hey, this will let you get your mail done in no time at all!” don’t you think I would be clearly in a position of hypocrisy if my business model required me to distract you constantly with “partner offers” down the side of your inbox?

If I offer you a ‘free’ spreadsheet or word processing tool, don’t you think I might be harming your productivity just a tad by hoping you won’t mind me looking at what you’re working on and making some useful suggestions for websites you should go and visit that want to sell you stuff? You thought clippy the paperclip was annoying? Wait until Google start running ads inside Writely.

The issue for me, is that advertising in the UI is anathema to good software design. We should be aiming to produce high-quality, bug-free software that lets users do the right thing as quickly as possible, and make it difficult for them to do the wrong thing at all. If we accept that the way this development will be paid for is through advertising, we are telling the user “actually, we don’t care about your productivity as much as we should”.

Advertising on mobile phones would be even worse. I look at my mobile phone maybe a dozen times a day. When I use it to browse something on the web, I make sure it’s as quick and painless and ad-free as possible. It is a tool, a device I carry around because I need to. It’s not my best friend that I feel the need to consult whenever I don’t know what to do with my evening.

If people are going to give me a £250 phone for free just so I can watch their ads, they’re going to quickly get burnt and end up having to cut costs and reduce their quality of service to me. They’ll do this as they realise I’m not going to be interrupted by them all day, and their maths of how excited I would be to hear about their special offer collapse under the weight of my considerable apathy.

What I’d prefer is what I have – a contract for 12 months where I give O2 my £25/month, and they give me a free/subsidised phone every year and a few free minutes of calls chucked in as well. I pay this for them to provide me a high quality of service, and to not send me adverts. To be there when I need them to be, and to stay out of my way when I’m, you know, living my life.

I think a lot of customers are going to start buying into this as well – they realise the problems with advertising supported businesses because it hurts their productivity, and when somebody produces a webmail tool as well designed as Google’s, but costing a small subscription year with no ads, Google will see market share slide. People will start to realise that whilst money is expensive and scarce, compared to time it’s cheap and freely available. If you ask somebody “do you want free, but takes longer, or a small amount of money and it’s less time with fewer distractions?” when they want to get something done, they’ll reach for their wallet every time.

The result will be the only people using Google and other ad-supported software being those who can’t pay a subscription, or use the tools so infrequently a subscription is of little value to them: exactly the wrong demographic advertisers want to talk to. As software developers, it is in our economic interest to notice this before Google and others do.

There is another scenario as well. Thinking about this, I came to the conclusion I don’t use Gmail because it is ad supported and I want my mail client to be a productivity tool as much as anything else – I don’t want to be distracted. But I do use Google search. Why don’t I get distracted there? And the answer hit me: my brain filters out the ads. I now just don’t see the links in the blue box at the top, or the boxes down the side of the page: I have adapted to focus on what I need. This is an interesting observation from the perspective of how the human brain adapts, but the scary question for Google is: what does this mean for an advertising-funded business model when everybody adapts?