A friend sent me a link yesterday with the short description of “Intriguing…”

It was a link to Clay Shirky’s article Gin, Television, and Social Surplus and indeed it is an intriguing article where he sets out his initial thesis thus:

I was recently reminded of some reading I did in college, way back in the last century, by a British historian arguing that the critical technology, for the early phase of the industrial revolution, was gin.

The transformation from rural to urban life was so sudden, and so wrenching, that the only thing society could do to manage was to drink itself into a stupor for a generation. The stories from that era are amazing– there were gin pushcarts working their way through the streets of London.

It’s not a new theory, however I’m not entirely sure it’s completely accurate. Urban life is not a new invention: Rome at one point is reckoned to have had 1 million citizens, and Athens had 300,000 citizens before it. Whilst they both had their debauchery, nobody has ever suggested that Rome needed wine and orgies in order to function as a city.

His parallel starts to get more interesting however:

If I had to pick the critical technology for the 20th century, the bit of social lubricant without which the wheels would’ve come off the whole enterprise, I’d say it was the sitcom. Starting with the Second World War a whole series of things happened–rising GDP per capita, rising educational attainment, rising life expectancy and, critically, a rising number of people who were working five-day work weeks. For the first time, society forced onto an enormous number of its citizens the requirement to manage something they had never had to manage before–free time.

And what did we do with that free time? Well, mostly we spent it watching TV.

I’m not sure the “wheels would have come off”, but there is no doubt that even people on very poor incomes have more free time than people of similar economic standing would have had for many millenia - if ever.

He goes on to talk about this surplus of time as something useful, interesting and powerful. His first example however directly contradicts my thoughts around The Vision Thing:

And I’m willing to raise that to a general principle. It’s better to do something than to do nothing. Even lolcats, even cute pictures of kittens made even cuter with the addition of cute captions, hold out an invitation to participation. When you see a lolcat, one of the things it says to the viewer is, “If you have some sans-serif fonts on your computer, you can play this game, too.” And that’s message–I can do that, too–is a big change.

Actually, it’s a change, but it’s not one we should embrace unless we say it’s the thin end of the wedge. That eventually something useful and interesting is going to happen and society starts working on interesting things. Clay goes on to talk about how if even one slither of that time of staring at the flashing box in the corner is used to do something productive, it means something interesting is going to happen.

Let’s say that everything stays 99 percent the same, that people watch 99 percent as much television as they used to, but 1 percent of that is carved out for producing and for sharing. The Internet-connected population watches roughly a trillion hours of TV a year. That’s about five times the size of the annual U.S. consumption. One per cent of that is 10,000 Wikipedia projects per year worth of participation.

I think that’s going to be a big deal. Don’t you?

To an extent I agree. I don’t know what 10,000 Wikipedia projects per year is going to look like, but there is no doubt that something, somewhere is going to happen of interest.

But what are those 10,000 projects? Do we have the creative ability to do 10,000 useful things every year? Do we have the will to do something more interesting than throw sheep at each other or spending our entire time photoshopping memes? Time will tell.

So, with just a couple of days of throwing ideas around, some interesting strands are coming together.

Firstly, those who are developers tend to agree with me: we’re a broken industry in need of a fix. Secondly, those who aren’t developers think I personally am broken and in need of a fix. Some think both are true.

With the private e-mail I was getting and IM chatter, I figured it would be better to produce a sandpit where anybody could get stuck in and post things up, link to articles, etc.

http://visionthing.vagueware.com is the current sandpit. We need to give this a better name and identity, but as you can see there is now a little momentum building.

You can also see that a few of us are playing with the idea of a Manifesto. It’s not enough to say what we’re angry about: we should be talking about what we want this industry to be, what we believe it is capable of, to lay a framework down to make sure we look after ourselves, our users and potentially our investors without breaking a fundamental philosophical barrier.

I think it might be worth just touching on a couple of the responses though, specifically those who suggest the problem is me.

OK, I’ll admit it, I’m tired. I need a break. I might not know what I really want out of life (who the heck does?), however those aren’t the problems we’re talking about.

The real problem is the abundance of froth in this industry right now, with no real substance and meaning to it. I am not condemning the entire industry - I just question the meaning of parts of it, and whether we can’t use our collective skills to do something better. This is ultimately a philosophical and political position to take, and it’s one many seem to share with me. We don’t know the details yet, but we know we want to try and hammer them out and at the end say “this is what we believe in”. I don’t expect everybody to agree with that position: a philosophy that has 100% belief isn’t a philosophy, it’s a law of nature.

We know we’re jaded and tired, but we’re jaded and tired for a reason.

As a group of geeks we hate spin, bullshit, lies, marketing speak and so on. We are an industry moving to a foundation built on those principles rather than the ones we admire: hard problems being solved with skill, and adding value to society. We want to help those with a financial interest satisfy their curiosity, but we want to encourage them to do it with the same sense of purpose that drives us to all-night hacking sessions.

I just had a niggle in my head the other day. Now we have the beginnings of a community prepared to work out how to make this industry better. We’re going to have detractors labeling one or all of as burned out as they look at that angry stare in our eyes when they explain their “social networking for lampshades” idea to us, but I feel all the better for knowing there are people who are thinking about this the way I am, and I hope they feel better too.

I have a problem with “the vision thing” in the industry at the moment. I don’t know where we’re going, or why. The technology - and our insight on how it can be applied - available to us has the ability to change the World, and instead we’re producing pointless crap and obsessing over details of page animations as if they alone will save the World.

If I hear one more wannabe-startup tell me that they plan to change the World and get rich off the back of social networking I will scream. If I see one more aggressive pitch for a site that a teenager could put together in a weekend under the guise of it being “World leading” I will hurt somebody. If I’m asked just one more time to give a quote to develop a site “a bit like eBay but with a social graph” I’m going to quit and go and be a farmer or something.

When I first got into computing, it was because of the potential of what you could do with this technology. When I first got on the Internet in 1996 I genuinely believed we were on the cusp of changing everything. An anarchic communication system where ideas could flow easily, people connect and work together to have a positive impact on society? I’ll take one of those please! Except 12 years later I think we’ve used it for mostly plastic, soulless and philosophically bankrupt ideas because “that’s where the money is”. There are exceptions, but the fact they are by definition exceptional means I feel we’ve taken a wrong turn somewhere.

I know I haven’t made any great personal contribution so am as guilty as anybody, but at least through efforts around BarCamp, Co-working and support of GeekUp I’m trying to get the conversation rolling. Maybe I’m just hacked off we’re moving so slowly. I know we have the collective talent so why are we all - me included - not getting on with it?

Over the next ten years we have the potential to fundamentally change the way the World works. Not just our World, but the Third World too. There are threats, opportunities, risks and rewards - not necessarily all financial - but for me the “vision thing” we should be working towards is about making people’s lives better not by trying to replace TV or other media, not by providing entertainment, but helping enhance people’s relationships - somehow providing meaning. And no, I don’t mean social networking, business networking or anything else where a friendship is defined as clicking on a “Confirm” button. I mean genuine enhancement of whatever it is we’re here to do.

I have spent a lot of time recently thinking about whether I want to stay in this industry. Last week I was ready to serve out the 6-month contract I’m on at the moment and then go and do something else for a while. Over the long weekend something niggled me and I think I know what I’m going to do next. I’ll explain more in coming weeks/months, but right now I want to see if I’m alone in having the niggle.

Comments are still broken here, so maybe I shouldn’t be doing this, but I want to know what other people think of the “Big Picture”. Specifically I’m going to do one of those slightly annoying “tag things” where I point to the people whose opinions I’m genuinely interested in who will hopefully respond to this post with one of their own and then tag another five people and so on so we can try and get this conversation going. Tagging posts with “thevisionthing” with Technorati might help us keep track of where it ends up.

What the hell are we doing in this industry? Why do we spend so much time talking about Ajax and definitions of “Web 2.0” and virtually no time whatsoever trying to work out what people want? Is this just all an aspect of the industry being over-run by complete geeks, or is the industry lacking any sense of philosophy? Are we being over-run by ideas and concepts from the advertising industry and mass media generally, because they’re becoming more dominant in the industry? Should I turn my dev environment off and go and do something less boring instead? I just want to hear what people think.

I tag (in alphabetical order):

• Guy Dickinson
• Matt Lee
• Katie Lips
• Manoj Ranaweera
• Tom Smith

That said, if anybody else wants to respond to this - say, Hugh who sounds about as burned out right now as I feel, Seth Godin who was in the industry way before me, Guy Kawasaki who is simply on the ball constantly, or anybody shoving their feed into NorthPack - I’d love to hear about it. I have a horrible feeling this will fall flat on its face and people will simply suggest I take a holiday (probably a good point), but it’s worth a punt…

Update: we have a few responses in:

• Andy Mitchell makes my point better than I did
• Guy Dickinson broadly thinks I’m wrong
• Tom Smith feels the pain, but thinks I just need a holiday or to do something different

They all make valid points, but there is something here. Via e-mail Andy and Steve Ireland have continued the discussion a little more. I think there is something in trying to advance this a little. Stay tuned.

The way I understand it, there are two groups inside Microsoft right now: I shall refer to them (even if nobody else does) as the Old Guard and the New Guys.

The Old Guard are the guys who built Microsoft in the first place. They’re the ones that we might consider the Evil Empire. They thought about software as a means to make money in itself. IPR and tools like DRM were critical to their thinking about how software should work. They’re the ones the EU don’t like. They’re the ones we’re a bit tired of in the open source community. They want your money.

The New Kids have seen a little more of the World as it really is. They think that software is a tool to sell services, training, knowledge, and that things like IPR and DRM get in the way of incredible creative freedoms. To them being able to mix in with as many people as possible is more important than trying to make sure that Microsoft locks you in: they want to win by producing the best tools possible. They want your heart and soul, feeling your money will follow.

The Old Guard is, naturally, getting older. They’re retiring. The New Kids are getting more important. They’re rising through the ranks. They’re able to make decisions. They’re the future.

We’ve just seen another small move in the battle that the New Kids are winning.

Give them time. They’re getting it.

Whilst I’ve worked in data centres before - and am all too familiar with how hot, noisy, industrial and dangerous they can be - I sometimes forget how the software industry I now work in has an industrial footprint in those rooms. It’s easy to think of my business as being ‘clean’, because the dirt is so well hidden.

Plans for Google’s new data centre in Dalles, as the blueprints published by Harper’s shows, should remind us just how industrial our business really is.

Combined with the annotation by Ginger Strand, we get a picture of how big this data centre is. Three buildings of over 68,000 square foot each and electricity consumption equivalent to that needed to power 82,000 homes, a third of which will be used just to keep the building temperature at a reasonable level.

Thanks to its location much of the energy used every day will be supplied via hydroelectric power, however its very existence has caused other technology firms to up their data centre spending, and it’s unlikely all of that capacity will be run on renewable power. And besides, every watt of clean energy powering a server is a watt not powering a domestic home.

It’s also worth remembering this isn’t “the” Google data centre. It’s “a” Google data centre.

For years now they have been pushing racks into peering sites and DCs around the globe as well as smaller facilities of their own - an estimated million servers are out there running Google sites, and there are more data centres planned by Google and their competitors over the next four years. Already data centres consume more power in the United States than the army of some 100-million-plus American monster-sized televisions. As the magazine itself says, the Web “is no ethereal store of ideas, shimmering over our heads like the aurora borealis. It is a new heavy industry, an energy glutton that is only growing hungrier.”

Better virtualisation of servers is going to help, but there’s a limit to how much you can virtualise. Is the time now right for us to get smarter again about how we use clock cycles? Is the efficiency-first stance of programming we’ve consigned to the era of the 8-bit machine now going to become fashionable again?

Maybe though, we could do a little to educate the public to make use of this vast industry a little more efficiently. Does the quest for the top 100 current hot trends at Google really suggest that we’re using this power wisely?

Via RoughType

Have you ever been shown a ‘database’ by somebody who doesn’t really know what a database is? You know it’s going to go badly when showing it, they double-click on an Excel file and you are confronted with a grid with ‘Name’, ‘Address1’, ‘City’, etc. across the top and a huge number of rows below.

Given that most people use spreadsheets like databases in that way, it was only a matter of time before one of the big online spreadsheet applications promoted it as a feature. And so it came to pass.

Seth thinks this is great but I’m not so sure. In a sense, it allows for gathering of data for analysis at a quantitative level, but at the same time it breaks the line between polling, database work and spreadsheet crunching in a way that might confuse more people than it will help.

Perhaps on the other hand it’ll help stop us thinking about applications and more about data. We are moving towards an era where we care about who and what more than the how and detail of collecting the data.

One of the advantages of being almost 30 is that people less than a decade younger than you tend to think of you as being “wise”. Some of the staff in my local bar will ask me about everything from US politics, the Renaissance, Alan Turing, 1980s TV commercials and arcane facts about the early forms of Parliament. Cultured bunch, the staff in my local.

Last night however, it was my turn to learn. One of them had asked me last week about the phrase “Jumping the Shark” and where it had come from. Last night she told me the very next day after I’d explained it to her, she watched an episode of Scooby Doo (OK, maybe they’re not that cultured) where Scooby jumps a shark and that it had made more sense to her knowing what it was a reference to - it is one of the classic insider jokes within TV comedy. I then had to re-explain it all to the other people assembled. The conversation that followed was… interesting:

It went on in a similar vein for a few more minutes. More examples of the futility of the network, the silliness of the apps. Admittedly, none of them had left Facebook yet, but that might be that it’s rather hard to leave, as GeekUp and Co-working day regular Alan Burlison found out

These are people the social networks need. In their early 20s. University students. Bright, intelligent, aspirational. I have no doubt that within a decade most of them will be in the upper 25% of earners in the UK. Malcolm Gladwell would call them “sneezers” or something - they spread their likes and dislikes around their friends quickly. They set trends.

And in the last couple of months they have come to hate Facebook and MySpace.

Specifically, they hate that these networks have been opened up to people engaging in what is effectively a developed and sophisticated form of spam. They hate that they are being hassled via the social graph into doing “fun” things that are actually about as fun as receiving a hoax virus e-mail. They understand that their time and attention is important and its being wasted by sites that don’t respect that.

I have ideas for applications that will actually add value to the social graph and be of use to people in this group, but by the time I get to roll them out it could be too late - the people that make the platform interesting to me as a recruitment base for customers may have moved onto something else.

Facebook are adding features to improve the user experience as they learn how developers are gaming the system. They might win the battle in time, but ultimately they might have to give more control to users to block invites from apps that are not even remotely in their realm of interest.

This isn’t over yet. 2008 could easily be the year the social networks died.

On the back of my business cards I have 10 quotes which on discovering them the first time, I found to be something that resonated with me, and that I hope might resonate with potential clients, business partners and friends.

The first of those is a famous quote by E.W. Dijkstra that for me sums up the reason I got into the industry in the first place:

“Computer Science is no more about computers than astronomy is about telescopes”

I also recall Ted Nelson’s talk about Transliterature at OpenTech 2005, where he also summed up why computers fascinated me as an 11-year old learning to program the first time:

“I studied Computer Science to help change the World, not to automate trivial crap”

There is something bigger here in our industry we refuse to acknowledge. There is something deeper beneath the surface that all the talk of social networks, long tails and user-generated content doesn’t get anywhere near.

This ember of a notion has been inside me for a while now, and it’s starting to turn into a small fire. I don’t know where it’s going, but what I do know is that I’m now getting more and more passionate for “big picture” stuff. The kind of things that need investment and great people.

I’m rather pleased then, with all this “big picture stuff” going on in my head, that this year’s Turing Lecture is being held again at Manchester University and that it has just been announced as being given by James Martin, producer of the film Target Earth - note, not the 1950s B-Movie, alas! However, it’s big in its approach, and I’m looking forward to watching it just before Dr Martin gives his talk.

I still haven’t decided what 2008 is going to be about for me professionally, but I do know it’s going to be less about me and finding ways to reconnect to that Dijkstra quote in my work. The Turing lecture will be a timely reminder of some of the issues facing us - and maybe sometime this year those of us in Manchester can start thinking about how to work out some of the solutions. Maybe.

I’ve just decided “Maybe” is my new favourite word.

Happy New Year.

This year has seen this blog grow by around 50% in terms of visits, page views and visitors. It’s a topic for another article, but I don’t want to think how much it would have cost me to get the vagueware name in front of the 14,000 unique visitors who have passed through here any other way.

Anyway, one of the most popular articles at the start of 2007 was my predictions for 2007 which was written before I realised the futility of prediction. Anyway, I figured it would only be fair if I now sat down and went through them and worked where I was right, where I was wrong.

Hardware

Apple will make great strides in 2007: Well, their share price more than doubled over 2007 and is now flirting with $200. One colleague in the states who does freelance installation/network admin work tells me orders through him for Apple kit in SMEs now beats PC orders five to one. Yes, five Apple orders for every PC order. Either something weird is happening in Florida/his neighbourhood, or something more interesting is happening.

I said I was expecting an Apple mobile phone: now seems so obvious, but at the start of 2007 there was absolutely no hint of the iPhone. I should have predicted the absurd pricing as well, but hey-ho.

[E]ither a 12” Macbook Pro, or possibly a tablet, perhaps with a smaller screen: No such luck.

8-core Mac Pros: Yay!

iTV, which I expect to emerge with better Video-on-demand services here in the UK: Early days on this one. It’s getting there though.

Video-On-Demand is going to be a big player in 2007 in the UK with BT, Sky, ntl:Telewest and others all edging into the market: I think I get partial credit on this one. I’ve been predicting VoD going big since 1998, but am constantly surprised by the ineptitude of the major networks. Even so, all major players have got into the mix.

By year end, it should be possible to buy for a smallish fee a programme without advertising embedded in it: Hmmmm. Kind of. I expected this to be a lot more solid, but at the moment the providers are still flirting with the tech.

[B]roadcasters and content-owners will be attempting to chase Apple and the record labels down the long tail: this one, I get a win on. BBC, ITV, C4 are all trying to build systems (and starting to co-operate on a system) that allows them to get iTunes out of the mix so that they aren’t held over a barrel as the US networks are.

Score: 5.5 out of 8 predictions

Software

Winner will be Linux: Some will argue over this, but ultimately we’ve seen open source operating systems fly this year. Dell will ship you a Linux box pre-installed, and Asus ship their EEE PC with a build of Linux on-board by default. You might argue it’s not getting dominant yet, but consider it in the context of my other prediction for OS roll-outs this year:

Vista will not fly: Consumers, SMEs, Enterprises alike all claim that Vista is not ready for them. They’re really asking what all that money is for, and I expect a lot of people are just waiting for more fixes before they’ll take a serious look at it. But if XP or Linux does everything you need, why upgrade? The innovation just isn’t coming out of Microsoft right now in the right places.

Ubuntu gain[s] market share: It has, but by nowhere near as much as I expected.

Apple will leverage consumer-friendly hardware, combined with consumer/family-focused enhancements in Leopard: See above. They are, but the late release of Leopard hurt them.

It may be we see a management shake-up at Microsoft: What I meant here was Ballmer needs to go. Ah well…

Firefox will gain market share: Ding! It has from all the figures I’ve seen, but again by nowhere near the amount I expected.

OpenOffice and similar well-developed open-source tools will gain audience: Ditto.

Score: 6 right out of 7 predictions

The Web

A paradigm shift in search: The team I was thinking of when I wrote this are still in closed beta. I’m getting worried. A lose for me here.

Huge improvements in machine translation of foreign language material: Ditto. I saw beta code in November 2006 that was good enough to launch with, and they’ve just gone quiet all year. I’m seeing a trend here with companies with great coders and great ideas going nowhere.

The Web 2.0 fiscal bubble will burst: I think the phrase has burst and people now just accept that social graphs and ‘collective intelligence’ as well as AJAX are just standard business practice. However the financials are actually getting scarier, with the big headline being an effective evaluation of Facebook touching $15 billion.

Adobe’s Apollo will be a much talked-about release: It was, but nowhere enough. Microsoft’s competing tech that I won’t even dignify with a name-check due it’s inate craptard-ness got more PR coverage.

I suggested that Apollo would also contribute to a new era of local application and web application integration: alas, not yet. Then again, like VoD I’ve been predicting this for years and thinking it was just around the corner.

Microformats will take a leap forward: I can’t in all honesty call this a win. They are exactly where they were a year ago - pretty much stale and needing easier adoption.

Popularism and effectiveness will start to replace academic philosophy in terms of determining the right standards for the Web to follow: Another loss. The W3C is still in charge, and still going exactly nowhere in the vain belief their discussions and papers are adequate replacements for working software.

Mobile web access will take off: I’ll take a win on this. 3G dongles, more WiFi everywhere, the iPhone and Nokia N95 providing better web experience on the move means in the UK this is happening. In the developing World, most people’s first use of the net is through a mobile phone, and that trend is only accelerating.

Uptake in Skype and similar free/flat-rate services on mobile devices: Almost. Data tariffs have got a lot more sensible over 2007 with T-Mobile and Vodafone leading the charge, but it’s still not at the point where most people are shoving their calls onto VoIP services.

Advertising to handsets will be resisted by consumers: Nope. Not really been tried in the sense I meant - operators using location data to target advertising - but Google is only just starting to ramp up their efforts.

The click-fraud problem will finally hit home: Alas not. We still think those Google CTRs are real.

More services will charge the consumer directly: I’ve got no data on this, but I think it’s a loser. I’m seeing more ads (or rather my ad blocker is blocking more) and the move seems to be about trying to get users on-board for free.

Score: I’m giving myself 2.5 out of 12 predictions here.

Programming

Agile methods will become the dominant lingua franca of programming: I’m only really looking at the Rails/Web industry here but I think this is a win. By the end of 2007 I was seeing projects being down in Waterfall and almost fell over. I’m not sure if that’s the case outside of my niche though.

Bespoke design and development web-shops will struggle: An unfortunate win for me on this one. In the North of England alone I know of one major web development firm going into receivership, and just before Christmas half a dozen SMEs closed up shop permanently. I think companies with great reputations are getting repeat work, but it’s getting harder to start up and compete against foreign operations charging nearly nothing. Personally, I concluded by the end of the year I want out of this sector: it’s dying, and dying hard.

Instead of going to a ‘web design company’ for a site, by the end of 2007 we’ll see clients starting to go to specialists/developers will need to start to know market sectors just as well: Many will argue this hasn’t happened. I say it has. I’m getting work now because I have experience of two sectors: online video and extranet applications. I would find it hard to get something in say retail right now, customers are looking for portfolios as much as they are skills.

Costs of development will come down: I claimed at the time this was only going to start in 2007 and it would 3-4 years to shake down. A win for me, I think.

Score: 4 out of 4 predictions here. I said at the start of 2007 this was the one I expected to get wrong the most, but it looks like I have a better handle on the trends underneath the industry than I thought.

Society

Now onto my more wishy-washy “armchair economist/politician” predictions.

Gordon Brown will struggle to become the next PM: Wrong.

All parties will try and find a way to re-connect to membership: Another loss, but the funding issues are still there. Personally I’m amazed that cronyism is still alive and well in Whitehall.

The EU Constitution will get a kick in the pants: Win. It’s back on the board, and we’re now arguing whether it’s a constitution or not.

Crash caused by consumer debt: Is a ‘squeeze’ the same as a ‘crash’? I’ll give myself half a win on this.

I went on to say I thought it would effect the house price market only by way of a slow-down: Win!

China will grow it’s GDP by about 10%: We won’t get the full figures until March next year, but looks like a win

I said environmentalism will be front and centre for 2007, with all countries, including the USA, putting it at the top of the agenda: I only got this as a win by a couple of weeks. We’re now on-course to stopping the madness, albeit it too slowly.

CFL lightbulbs will become standard: Win. You might be able to buy standard bulbs still, but do you know anybody who actually still buys them?

At least one new form of renewable energy, distinguishable from wind, solar, etc. get announced in 2007: Nope, a loss. But we’re seeing better/more efficient uses of wind and solar.

Iraq will head into full-blown civil war: I’m surprised by this one. It’s not in full-blown civil war, and where the US are in control the surge is actually having an effect. Where the British just pulled back though is sounding very grim.

Iran will end up extending their borders and taking control of at least part of Iraq: Maybe not officially, but in practice this has happened. They’re running operations all over the place, and I think it’s only a matter of time before they just stick their flag down and say “this is ours” somewhere in the South. Given that’s where the oil is, expect a reaction. Half-a-win here.

Score: 6 out of 11 predictions correct.

I also sneakily put in a prediction for 2008, that being Barack Obama will become the 44th President of the United States: well, technically he won’t get inaugurated until January 2009 but it’s starting to shape up well and his campaign has what politicos in the states call “the big mo” - i.e. unstoppable momentum.

Conclusion

Over the course of 2007 I realised the futility of prediction, and gave a talk at BarCamp Leeds (still to go online) about how rubbish Futurology is. I’ve surprised myself here by being better at some predictions than others. Strangely, the areas where I took less outside influence and opinions - programing and software - I scored better at than those where I had read a lot of ‘professional’ analysis like politics, the web industry and hardware.

Out of 42 predictions I got just 24 right. Statistically, that’s in the noise, but it’s not too shabby. Some of my predictions were long-shots and the gambler in me knows that if I’d managed to get odds on them all, I’d be in profit. Even so, I’m not if I want to try and make predictions for 2008. I’ll sleep on it.

Who would have thought that one of the most innovative players in the hosting and web application industry would be a bookshop?

One of the big problems with Amazon’s web services is that they aren’t that great for permanently hosted web applications. There’s the dynamic IP addressing issue (which weoceo will look after if you have the cash) and the serious problem of how to store your database.

S3 is very nice, but it stores flat data, and certainly not anything as fancy as SQL tables. Until recently there was a hacky way to do it with a special storage engine for MySQL, but just looking at it made me nervous about my data.

Well, Amazon have decided to fix this issue. I received this email from them this morning.

“Dear AWS Developers,

This is a short note to let a subset of our most active developers know about an upcoming limited beta of our newest web service: Amazon SimpleDB, which is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database - real-time lookup and simple querying of structured data - without the operational complexity.

Were excited about this upcoming service and wanted to let you know about it as soon as possible. We anticipate beginning the limited beta in the next few weeks. In the meantime, you can read more about the service, and sign up to be notified when the limited beta program opens and a spot becomes available for you. To do so, simply click the “Sign Up For This Web Service” button on the web site below and we will record your contact information.

Learn more and sign up

Sincerely,

The Amazon Web Services Team”

So, there we have it. No more managing DB clusters. Scalable database tables, which once the beta is over will likely come with an SLA. Assuming that this just sits on top of S3, we might even be able to host our data inside the EU and get ll warm and fuzzy about protecting customer data properly.

I’m not sure this will be based on a standard set of DB libs but I expect we’ll see 1-line hacks to make it work with Rails, PHP and a host of other app frameworks within a few weeks.

I’m in.

This afternoon, I’ve been playing around with Facebook’s ad platform. Partly for Vagueware, partly for other businesses, I’ve been looking at what Facebook says about its user base to advertisers.

The level of targeting is just outright astonishing. It allows for ads not only to be targeted on demographics such as age range and city, but even on interests and relationship status.

For example, I now know there are approximately (all figures given are approximate to the nearest 20 or so), 120 people in Manchester interested in Programming.

Out of the 2,017,440 UK citizens who describe themselves on Facebook as ‘single’, 998,900 are male, 904,960 are female. The numbers don’t add up because some people don’t define a gender which makes the point that if you don’t fill info in, you can’t be targeted via that info.

There are 1,180 females in the UK who declare an interest in ‘Computers’. The figure for males is around 8,540.

580 UK men say they’re really into shoes, with 14,300 British women aspiring to be Imelda Marcos.

There are around 5,680 people working for BT in the UK on Facebook. In the US, there are around 40 people working for O’Reilly Media who mention it in their profile. I could target either with an advert - handy if you have a product or idea you want to pitch.

This is just the tip of the iceberg. Advertisers don’t need to know who your friends are (the social graph), to target you this tightly. If a member of GeekUp wanted to put up a singles ad for all single women between the ages of 24 and 32 who are into computers resident in Manchester (approx. 100 of them), they now theoretically could. Lucky ladies.

The question is, is this really a bad thing? Doesn’t it mean we’re not all going to see advertising that really has no relevance to us? Or does this kind of marketing mean that we are the perfect willing victims for advertisers to go deep into our psyche? I knew this day was coming, but I thought it was still some way off.

Vagueware is not my only gig. My other Directorship is very low-key, doesn’t take much time, and is where I and my business partner experiment with various marketing revenue models. It’s primarily been a learning experience, and the frustration we’ve had over the last two years in getting various complex projects rolled out has meant we’ve been looking at partnering with technology companies and focusing on the marketing and customer communications side.

Our latest venture is quite a departure for us. Excuse me whilst I shill for a couple of paragraphs:

Whilst cash back websites are not brand new, we’re hoping that with a really solid technology platform underneath us, we’re going to be able to do something special in the way of helping people make shopping a little more fun - and save cash too. I do however, need some eyes and ears because I’ve had no control over technology roll-out, so I’m interested to hear of problems people might have.

As you can see, the concept is really simple. You sign up, we give you a fiver. You shop online, we give you a percentage of what you spend. You refer friends, we give them a fiver, and we give you a fiver as well once they qualify for payout. We’ll point you in the direction of freebies that pay you money as well. We’ll be launching a blog to highlight particularly good offers. Occasionally e-mails with super secret codes will land in your inbox and you will consider yourself a wise old bean for signing up with us. It makes things cheaper if you’re doing a lot of Christmas shopping online, although for various reasons we’re late to the party for that one, so our strategy is a little more long-term.

End of shill

I mention it here, because I’m interested in problems an educated audience (that’s you, dear reader), might see. We know for example that the back end systems are rock-solid and everything is nice and secure, but are there ‘quirks’ we’ve yet to spot that only a geek can spot? Maybe you just think the business model is odd, or we haven’t explained it very well. Either way, I wanted people whose opinions I respect to take a look before the big marketing push over the next 12 months, and see where we can make improvements.

Tonight, the BBC finally got around to airing the RTS Huw Weldon Memorial Lecture recorded in September. The speaker this year was Anthony Lilley talking on issues relating to social media and ironically given the subject of his talk, no online archive of the talk appears to be available. For all the embracing of “The Me in Media” - as the talk was titled - and the power of the network, we on the network are not allowed to see it.

No matter. The point, the thesis, the element, is quite communicable: we the audience are now in charge. What’s more, Lilley makes a compelling argument that this isn’t a sudden and new development, but something that happened the moment the audience started to appear on screen from quiz shows to our scribbles being sent to Tony Hart.

The power of networked media is considerable, and it’s pretty odd to see a bunch of network TV executives try and grasp the content that they, their ideas, their employers and their money don’t matter much any more. Within 10 years, Lilley hinted that networks like UKTV which mostly show repeats will essentially be pointless with the advent of VoD and PVR systems being widespread. Within 20 years the people being asked to pay the license fee will have no real memory of broadcast media. Within 30 years… who knows?

I was minded of a certain cartoon and then thought about how the BBC Innovation Labs is on again this year. I wonder how daring the BBC and others want us to be. Personally, I think they’re just outright scared.

A friend just forwarded to me an article called “The Outsourced Brain” over at the New York Times. A sample:

” Until that moment, I had thought that the magic of the information age was that it allowed us to know more, but then I realized the magic of the information age is that it allows us to know less. It provides us with external cognitive servants — silicon memory systems, collaborative online filters, consumer preference algorithms and networked knowledge. We can burden these servants and liberate ourselves.

Musical taste? I have externalized it. Now I just log on to iTunes and it tells me what I like.”

This is going to gradually become a debate over the next few years as we pass more and more of our thinking and life over to algorithms. Stroustrup once said “Software runs civilisation”. I think we’re approaching the point where we can say “Software runs civilians”. There are obviously issues with this that need to be explored.

About a year ago I developed a hypothesis of what humanity would broadly look like 100 years from now. Some friends found my synopsis of this vision a little ridiculous: “You know the borg in Star Trek? That’ll be us”.

What I mean is that we are slowly moving our thinking out into the cloud and acting as one. Individualism is being lost, group-think is being encouraged. If that sounds a little Orwellian, can I just point out that we’re the ones encouraging it on ourselves - from CCTV cameras to collaborative filtering on Amazon - it is not being imposed on us.

The irony is that for all the menace of Borg assimilation and Orwellian dystopia in fiction, we are shaping parts of our society into something that mimics it in the hope it will lead to peace and harmony within society. Maybe it will, I don’t know.

The hope we have is that if we spend less time thinking about what music to buy, which directions to use to get somewhere, and trying to remember things we can get out of Google and Wikipedia anyway, we’ll have time for more important things. The question is what things are we doing with that time? Are we just filling that brain capacity with other trivia we don’t need?

Not for the first time, I feel that those of us styled “Software Engineers” have a responsibility to ask some questions here.

OK, so I’m having some fun with the title. This is Part III (the final part, you’ll be pleased to hear) of my write-up about the Open Schools Alliance even last week. Part I and Part II are worth a look if you just got here.

First up straight after the break was Liberal Democrat MP for Southport, John Pugh. This particular Honourable Member is well-known to those of us on the open source side of the digital divide: he has a habit of asking what must be for the mandarins on the receiving end really annoying questions of the government about their IT procurement policies. He has a particular interest in IT in schools as he himself started out as a teacher, but in recent years has found an ally in Private Eye for his questioning in the House around the tax credits fiasco and other IT blunders.

He made the argument that the government is progressively getting worse at procurement in that it’s not learning from its mistakes. He argued that many within government departments are unaware of what open source is, are unaware of what it can do, or what it can save. The quote for me from this session was “whilst the government have a road building programme, they don’t argue roads must be built so that they may only accomodate Fords”.

It seemed to me though, that his real bugbear was open standards more than open source - it is the fact we’re producing systems that lock us into a vendor for a lifetime that is causing us problems.

We then moved into a panel discussion featuring John, Ian Lynch, Mark Taylor, Mike Partridge and our strawman for the day, Dr. Stephen Lucy of BECTA.

This discussion ultimately came down to panellists and the audience expressing dismay at BECTA’s attitude towards OSS, and how they were allowing for the propping up of what can be described as state aid of Microsoft. I was quite impressed by how Dr Lucy handled the situation, but was informed by another attendee later that this was characteristic of how he worked - he would attend these events and “play a dead bat” to the air of hostility. I can’t blame him, but BECTA are going to have to realise that it’s going to get worse unless they start looking at how to bring open source into the mix.

There are huge issues around OSS and IT procurement in general - probably more urgent in education than anywhere else - and it’ll be interesting to see the direction the Open Schools Alliance.

There needs to be a shift from centralised procurement to bottom-up organisation, but even with centralised projects like CLEO it has been shown OSS can provide amazing value for money. If BECTA were willing to play ball a little, who knows how much more great software we could see in classrooms over the next few years? As it is, it feels to me like a few senior players in BECTA are positioning themselves for consultancy positions in Microsoft and WebCT.

Whatever happens, it’s going to be interesting to watch, and I only hope that eventually parents and teachers see sense, and give the OSA all the support they need.