So, with just a couple of days of throwing ideas around, some interesting strands are coming together.

Firstly, those who are developers tend to agree with me: we’re a broken industry in need of a fix. Secondly, those who aren’t developers think I personally am broken and in need of a fix. Some think both are true.

With the private e-mail I was getting and IM chatter, I figured it would be better to produce a sandpit where anybody could get stuck in and post things up, link to articles, etc.

http://visionthing.vagueware.com is the current sandpit. We need to give this a better name and identity, but as you can see there is now a little momentum building.

You can also see that a few of us are playing with the idea of a Manifesto. It’s not enough to say what we’re angry about: we should be talking about what we want this industry to be, what we believe it is capable of, to lay a framework down to make sure we look after ourselves, our users and potentially our investors without breaking a fundamental philosophical barrier.

I think it might be worth just touching on a couple of the responses though, specifically those who suggest the problem is me.

OK, I’ll admit it, I’m tired. I need a break. I might not know what I really want out of life (who the heck does?), however those aren’t the problems we’re talking about.

The real problem is the abundance of froth in this industry right now, with no real substance and meaning to it. I am not condemning the entire industry - I just question the meaning of parts of it, and whether we can’t use our collective skills to do something better. This is ultimately a philosophical and political position to take, and it’s one many seem to share with me. We don’t know the details yet, but we know we want to try and hammer them out and at the end say “this is what we believe in”. I don’t expect everybody to agree with that position: a philosophy that has 100% belief isn’t a philosophy, it’s a law of nature.

We know we’re jaded and tired, but we’re jaded and tired for a reason.

As a group of geeks we hate spin, bullshit, lies, marketing speak and so on. We are an industry moving to a foundation built on those principles rather than the ones we admire: hard problems being solved with skill, and adding value to society. We want to help those with a financial interest satisfy their curiosity, but we want to encourage them to do it with the same sense of purpose that drives us to all-night hacking sessions.

I just had a niggle in my head the other day. Now we have the beginnings of a community prepared to work out how to make this industry better. We’re going to have detractors labeling one or all of as burned out as they look at that angry stare in our eyes when they explain their “social networking for lampshades” idea to us, but I feel all the better for knowing there are people who are thinking about this the way I am, and I hope they feel better too.

I have a problem with “the vision thing” in the industry at the moment. I don’t know where we’re going, or why. The technology - and our insight on how it can be applied - available to us has the ability to change the World, and instead we’re producing pointless crap and obsessing over details of page animations as if they alone will save the World.

If I hear one more wannabe-startup tell me that they plan to change the World and get rich off the back of social networking I will scream. If I see one more aggressive pitch for a site that a teenager could put together in a weekend under the guise of it being “World leading” I will hurt somebody. If I’m asked just one more time to give a quote to develop a site “a bit like eBay but with a social graph” I’m going to quit and go and be a farmer or something.

When I first got into computing, it was because of the potential of what you could do with this technology. When I first got on the Internet in 1996 I genuinely believed we were on the cusp of changing everything. An anarchic communication system where ideas could flow easily, people connect and work together to have a positive impact on society? I’ll take one of those please! Except 12 years later I think we’ve used it for mostly plastic, soulless and philosophically bankrupt ideas because “that’s where the money is”. There are exceptions, but the fact they are by definition exceptional means I feel we’ve taken a wrong turn somewhere.

I know I haven’t made any great personal contribution so am as guilty as anybody, but at least through efforts around BarCamp, Co-working and support of GeekUp I’m trying to get the conversation rolling. Maybe I’m just hacked off we’re moving so slowly. I know we have the collective talent so why are we all - me included - not getting on with it?

Over the next ten years we have the potential to fundamentally change the way the World works. Not just our World, but the Third World too. There are threats, opportunities, risks and rewards - not necessarily all financial - but for me the “vision thing” we should be working towards is about making people’s lives better not by trying to replace TV or other media, not by providing entertainment, but helping enhance people’s relationships - somehow providing meaning. And no, I don’t mean social networking, business networking or anything else where a friendship is defined as clicking on a “Confirm” button. I mean genuine enhancement of whatever it is we’re here to do.

I have spent a lot of time recently thinking about whether I want to stay in this industry. Last week I was ready to serve out the 6-month contract I’m on at the moment and then go and do something else for a while. Over the long weekend something niggled me and I think I know what I’m going to do next. I’ll explain more in coming weeks/months, but right now I want to see if I’m alone in having the niggle.

Comments are still broken here, so maybe I shouldn’t be doing this, but I want to know what other people think of the “Big Picture”. Specifically I’m going to do one of those slightly annoying “tag things” where I point to the people whose opinions I’m genuinely interested in who will hopefully respond to this post with one of their own and then tag another five people and so on so we can try and get this conversation going. Tagging posts with “thevisionthing” with Technorati might help us keep track of where it ends up.

What the hell are we doing in this industry? Why do we spend so much time talking about Ajax and definitions of “Web 2.0” and virtually no time whatsoever trying to work out what people want? Is this just all an aspect of the industry being over-run by complete geeks, or is the industry lacking any sense of philosophy? Are we being over-run by ideas and concepts from the advertising industry and mass media generally, because they’re becoming more dominant in the industry? Should I turn my dev environment off and go and do something less boring instead? I just want to hear what people think.

I tag (in alphabetical order):

• Guy Dickinson
• Matt Lee
• Katie Lips
• Manoj Ranaweera
• Tom Smith

That said, if anybody else wants to respond to this - say, Hugh who sounds about as burned out right now as I feel, Seth Godin who was in the industry way before me, Guy Kawasaki who is simply on the ball constantly, or anybody shoving their feed into NorthPack - I’d love to hear about it. I have a horrible feeling this will fall flat on its face and people will simply suggest I take a holiday (probably a good point), but it’s worth a punt…

Update: we have a few responses in:

• Andy Mitchell makes my point better than I did
• Guy Dickinson broadly thinks I’m wrong
• Tom Smith feels the pain, but thinks I just need a holiday or to do something different

They all make valid points, but there is something here. Via e-mail Andy and Steve Ireland have continued the discussion a little more. I think there is something in trying to advance this a little. Stay tuned.

The way I understand it, there are two groups inside Microsoft right now: I shall refer to them (even if nobody else does) as the Old Guard and the New Guys.

The Old Guard are the guys who built Microsoft in the first place. They’re the ones that we might consider the Evil Empire. They thought about software as a means to make money in itself. IPR and tools like DRM were critical to their thinking about how software should work. They’re the ones the EU don’t like. They’re the ones we’re a bit tired of in the open source community. They want your money.

The New Kids have seen a little more of the World as it really is. They think that software is a tool to sell services, training, knowledge, and that things like IPR and DRM get in the way of incredible creative freedoms. To them being able to mix in with as many people as possible is more important than trying to make sure that Microsoft locks you in: they want to win by producing the best tools possible. They want your heart and soul, feeling your money will follow.

The Old Guard is, naturally, getting older. They’re retiring. The New Kids are getting more important. They’re rising through the ranks. They’re able to make decisions. They’re the future.

We’ve just seen another small move in the battle that the New Kids are winning.

Give them time. They’re getting it.

No matter where it happens, if somebody out there is coming up with crazy ideas in software, I get to hear about it eventually. It’s just rare I get time to write it up.

This evening I caught on to Breach Candy Group’s idea for what they call AML or ‘Aesthetics Markup Language’ and decided to talk about it pretty much straight away. This article is under the “Philosophy” category for a reason. If you’re not in for some deep thinking, move along.

Their idea is to be able to define the aesthetics of a piece of film or art in a standardised markup language so that systems may be able to perhaps generate new content of a similar aesthetic style. For example, as they say:

“Let’s start with images. We could start off with the following variables:

• movement (speed of movement = speed of change in pixels?) This could be later used to analyze some rhythm of change.

• brightness and contrast (how would this be tracked = the relationship or average of pixels in any given location on the video?)

This could also later be use to analyze things such as harmony of composition, direction of lines in the mise en scene, etc. We would have to come up with a set of principles from art history and composition and see how these could be determined in the screen etc?

• color range (this would probably have to be RGB values in the image itself). This would probably move us into the realm of things such as monochromatic color schemes, bright colors, harmonious colors, contrasting / oppositional color … ie to use some notion of color theory to provide patterns in certain styles of video etc. I’ve studied this in high school so will be fun to revisit some principles of classical painting.

So I suspect what we need to do is set up a very simple experiment / structure in place that can be developed and extended depending on need. In other words, we need to develop … AML (Aesthetics Meta Language) … a basic language structure that would describe what the variables are within any analyzed video. This language, I suspect, could be then developed into the interface between the language of aesthetics and the computer.

Something like this:

//AML: “DEBBIE DOES DALLAS”

134 12 58 22

It’s a nice idea, but they’re missing a trick. The thinking that got them to this point to me seems much more interesting.

They argue that most of the complexity we witness in the World is a repetition of simple things that go on to form complexity. This theory exists behind cellular automata, fractals and more.

Cellular automata are incredibly important in our understanding of how complexity is created out of the simplest building blocks within the Universe. Take for example the image to the left of this text. It might look like something taken from a microscope, but it’s an image created in software using a CA routine that was less than 100 lines of code, simulating “dictyostelium slime mold”. You can find out more about exactly how it was constructed at the site I got the image from.

This idea of complexity emerging from simple rules is particularly important in the field of Artificial Intelligence for reasons described in a philosophically entertaining manner in the AML article:

“Here the key question is that what algorithms could be used to model the way humans think and thus be used to guide machines to perform complex tasks. The philosophical implications of this are even more profound than getting a robot to recognize faces or clean a non-linear toilet bowl. That is, if human intelligence is, in fact, highly programmable, what then defines humans from machines? This goes two ways: machines-as-humans and humans-as-machines. In other words, AI defines rationality a certain way with certain presupposition of what logic, thinking and consciousness are and how they can be pragmatically simulated in computers. But as importantly, if we look at the concept of rationality and how it has been historically constructed, this has always presupposes a certain “image of thought” that has excluded all that would not fit into the sphere of rationality (intuitions, insanity, madness, illogic, spontaneity, absurdity ….). So how would we then understand the blurred boundaries of man and computer (as intelligent forms, which neither technically speaking are) and the human-computer assemblage that is making the old notions of rationality/humanity perhaps increasingly difficult to defend? Humans as (programmed?) repetitions: computers as programmed repetitions: natural intelligence: artificial intelligence: natural stupidity: artificial stupidity …”

Think about that for a moment. The point is not to be “correct” in the sense of making an algorithm “smart”, but make it mimic so it has the ability to be just as insane, dumb and mad as humanity. If you understand the Turing Test correctly, we will have produced an algorithm capable of passing it correctly, when we produce an AI capable of insanity, melancholy and psychopathic behaviours. Asimov’s 3 rules of robotics can not exist in a machine capable of passing a Turing Test, in other words.

This is a good development of an argument on the part of the Breach Candy Group, partly because it’s intuitively correct but also because it follows the science we have to hand quite nicely. It’s a shame then that in itself it probably undermines the need for AML on which they built on these foundations.

AML is about how a computer would process very specific values - contrast ratios, line measurements, etc. - yet they accept themselves that what they need to do is instead mimic the way a human would describe the aesthetic. No film director would talk about precise values of contrast settings or line movement, but instead would find a way to talk about colour, tone, depth, warmth, speed and so on in a much more abstract way.

In other words, to achieve what they’re hoping for they need to develop an algorithm which is able to mimic the human way of parsing film aesthetics, train it by making it “watch” films and then ask it to produce something “like” a subset of them. They’re trying to find a way through by producing a way of retaining knowledge about an aesthetic in a standard form, but as anybody who has read “Zen and the Art of Motorcycle Maintenance” can tell you, there is much more to an aesthetic quality than how you describe it mechanically.

In fact, the point about a distinctive aesthetic quality is that whilst we know it when we see it, we all see something different and would all describe it differently. Let me try and make my point by using an image that has probably had its aesthetic qualities dissected, written about and analysed more than any other in the history of aesthetics.

The Wikipedia article on the painting has quite a detailed summary of some of the aesthetic judgements made. There are two groups these arguments can be placed: those that are algorithmic and those that are subjective.

It might be reasonable to produce in AML those that are algorithmic. They can be measured precisely - golden ratios, pyramid composition. However how exactly do you describe something as subjective as “the composition of the figure evokes an ambiguous effect: we are attracted to this mysterious woman but have to stay at a distance as if she were a divine creature” in a markup language?

I don’t have answers, only questions on this one. Interesting thoughts though. And if they can be resolved, we’d be a major step forward to understanding AI - and ourselves - much better.

On the back of my business cards I have 10 quotes which on discovering them the first time, I found to be something that resonated with me, and that I hope might resonate with potential clients, business partners and friends.

The first of those is a famous quote by E.W. Dijkstra that for me sums up the reason I got into the industry in the first place:

“Computer Science is no more about computers than astronomy is about telescopes”

I also recall Ted Nelson’s talk about Transliterature at OpenTech 2005, where he also summed up why computers fascinated me as an 11-year old learning to program the first time:

“I studied Computer Science to help change the World, not to automate trivial crap”

There is something bigger here in our industry we refuse to acknowledge. There is something deeper beneath the surface that all the talk of social networks, long tails and user-generated content doesn’t get anywhere near.

This ember of a notion has been inside me for a while now, and it’s starting to turn into a small fire. I don’t know where it’s going, but what I do know is that I’m now getting more and more passionate for “big picture” stuff. The kind of things that need investment and great people.

I’m rather pleased then, with all this “big picture stuff” going on in my head, that this year’s Turing Lecture is being held again at Manchester University and that it has just been announced as being given by James Martin, producer of the film Target Earth - note, not the 1950s B-Movie, alas! However, it’s big in its approach, and I’m looking forward to watching it just before Dr Martin gives his talk.

I still haven’t decided what 2008 is going to be about for me professionally, but I do know it’s going to be less about me and finding ways to reconnect to that Dijkstra quote in my work. The Turing lecture will be a timely reminder of some of the issues facing us - and maybe sometime this year those of us in Manchester can start thinking about how to work out some of the solutions. Maybe.

I’ve just decided “Maybe” is my new favourite word.

Happy New Year.

I never “got” code reviews. I never understood how sitting around talking about somebody else’s code would help them or you, beyond being able to weed out the obviously awful coders.

This week I’ve been doing some consultancy, in-sourced to help a firm evaluate what their developers have built for them. As part of that process I’ve had to review documentation, specifications and code. The process has been at times confusing and enlightening.

It’s one thing to be able to say “this is how it should be done” when you’re not a developer, but it’s another thing when you are a developer who hasn’t done some of those things yourself. This process has taught me as much about myself and my own code as it has about these guys’ code.

Interestingly, I now see things from a buyer’s perspective much more clearly, and I can see how frustrating this must be from the other side. I can see how things which we don’t consider important are to a buyer critical. I can see how clever solutions can sometimes be too clever. I can see how something simple you forget to do can make all the difference.

After a brief spurt of articles, I’ve slowed right down posting. It’s not accidental - it’s very considered in fact - and I thought it might be worth sharing a few plans.

I had a queue of about 90+ articles in draft ready to be finished and posted. At a rate of three per day, plus adding at least two more in draft form onto the queue I would have had the momentum to keep me going through to the end of the year.

I stepped back though and thought about why I wanted to publish here, what this blog was for, and whether that was a sensible strategy. I started asking whether I should care if this tool does this or that tool does that.

There are much bigger ideas we’re heading towards that need something more thoughtful than twenty blog articles a week.

I have yet decide my approach, but what readers enjoy matters. I get virtually no feedback about what people like beyond Google Analytics tells me, and it tells me little. I get few people quoting me and linking back to articles. I get few comments. In other words, it’s a little hard to know what is working and what isn’t beyond “being angry about Leopard” gets me traffic.

Questions for you then: what articles do you enjoy or hate? What do you want to see more of or less of?

A friend just forwarded to me an article called “The Outsourced Brain” over at the New York Times. A sample:

” Until that moment, I had thought that the magic of the information age was that it allowed us to know more, but then I realized the magic of the information age is that it allows us to know less. It provides us with external cognitive servants — silicon memory systems, collaborative online filters, consumer preference algorithms and networked knowledge. We can burden these servants and liberate ourselves.

Musical taste? I have externalized it. Now I just log on to iTunes and it tells me what I like.”

This is going to gradually become a debate over the next few years as we pass more and more of our thinking and life over to algorithms. Stroustrup once said “Software runs civilisation”. I think we’re approaching the point where we can say “Software runs civilians”. There are obviously issues with this that need to be explored.

About a year ago I developed a hypothesis of what humanity would broadly look like 100 years from now. Some friends found my synopsis of this vision a little ridiculous: “You know the borg in Star Trek? That’ll be us”.

What I mean is that we are slowly moving our thinking out into the cloud and acting as one. Individualism is being lost, group-think is being encouraged. If that sounds a little Orwellian, can I just point out that we’re the ones encouraging it on ourselves - from CCTV cameras to collaborative filtering on Amazon - it is not being imposed on us.

The irony is that for all the menace of Borg assimilation and Orwellian dystopia in fiction, we are shaping parts of our society into something that mimics it in the hope it will lead to peace and harmony within society. Maybe it will, I don’t know.

The hope we have is that if we spend less time thinking about what music to buy, which directions to use to get somewhere, and trying to remember things we can get out of Google and Wikipedia anyway, we’ll have time for more important things. The question is what things are we doing with that time? Are we just filling that brain capacity with other trivia we don’t need?

Not for the first time, I feel that those of us styled “Software Engineers” have a responsibility to ask some questions here.

We can all remember the futurology clips from the 1960s suggesting we’d all have hover-cars and silver jump-suits by now, so it surprises me that newspapers still try and predict the future. I definitely expected more from The Independent

What they fail to appreciate is:

• The forecasts have always been wrong in the past. There is no reason to be confident they will be right in the future. Not just immigration figures, but shifts in employment, prosperity, house prices, interest rates (which are managed by the central banks!), share prices, the weather tomorrow…
• They’re not taking into account movement in the opposite direction. Yes, your parents’ house in the South of France or Spain is considered immigration by the French.
• We actually need to get to 75 million in the UK by 2030 to keep the state pension system working as it does today, but they ignore that because it doesn’t fit the political agenda de jour.

I’m a liberal (in the European sense of the word), and I find the “Immigration is always bad” argument morally repugnant, but that’s not what I want to focus on in this article. Rather, I’m interested in how mainstream media goes about handling forward forecasts. They typically:

• Don’t ask questions about whether forecasted figures are accurate, or how they could be accurate
• Don’t ask questions about whether figures are representative of the complete picture, or even what “complete” means
• Don’t ask questions about what the forecast actually means in real terms beyond the gut reaction

In short, they’re terrible at understanding possible future outcomes and analysing them. If they were working in the technology industry they would instead know:

• Forecasting is inaccurate
• We never have full sets of data
• We can’t really know what it means until we get there

That would however make for lousy newspaper copy. “Immigration may or may not be a bad thing, or a good thing, depending on what actually happens” is not as catchy as “Immigration out of control”.

How do we educate politicians, journalists and the public about how best to deal with these figures though? This industry is about as good as it gets with future forecasting (although we struggle to see paradigm shifts and anything > 3 years out), so is there anything we can share that others might learn from?

[UPDATE]: Comments like the one I’ve approved below have started coming in. Hmph.

I know from past experience that whenever you talk about immigration, a bunch of BNP/nationalist nutjobs (who are very clued up as to how to find and comment on blogs that even lightly touch on the subject) tend to pass by and start making silly statements. I’ve let this one through to demonstrate my point - whilst glancing on racism in a way the author probably hoped I wouldn’t notice, it’s not as bad as others - but let me please, please stress: I’m not having a debate about immigration here, I’m discussing how the media handles forecasting in general. Immigration is the story today, but I’m talking about a trend that applies to everything from viral epidemics to what the weather will look like next week.

Please do not add comments of the “but we’re going to run out of space” or “the schools will collapse” vein here - they simply won’t get approved, as they’re not relevant to what I’m really talking about above. Making vague statements about “ethnicity” is likely to just make me think you’re an idiot.

Apple Insider is giving us a thorough run-down on the history of the Virtual Desktop in anticipation of Apple’s launch of OS X Leopard. Leopard will ship with ‘Spaces’, Apple’s own take on the Virtual Desktop.

What has surprised me most in recent years is that nobody seems interested in why we continue to try and fit everything into the ‘Desktop’ metaphor at all.

As I understand it, when they were trying to decide an interface at Xerox PARC, they thought about who would use the tools they were developing. It is a sign of the times that they thought the only person likely to be sat in front of a keyboard any time in the future would be a secretary - so the developers went down to the typing pool and looked at how a typist might understand the World of work.

It’s for that reason, and that reason alone, we have the metaphor of a ‘file’ and a ‘folder’ and a ‘desktop’ within modern user interfaces.

There have been experimental interfaces to shift the metaphor, but these experiments have normally failed because the chosen metaphor has been so weak. Surely there has to be a better way to think about the way we work now we have fully embraced the hypertext document, the network as the computer and the social network?

Whilst some may ask what could you do with infinite bandwidth, others are actually trying to get there. Internet2 - a research project that is surprisingly low-profile outside of those directly involved - has recently reached 100Gbps and there are, as ever, plans to go faster.

We’re at an odd period in the history of the Internet when it comes to bandwidth. We’re at speeds fast enough to provision most people’s textual and audio requirements just fine, and a few years away from being able to provide enough space for everybody’s HD video requirements. The question is, what next? What uses can we put higher speeds to? We’re quickly reaching the point where we can send data around between nodes faster than the nodes can do something useful with the data.

Once we’re at the point where data can consistently be transferred quicker than it can be processed - either by a computer or a human - we’re at a new point in the history of the network. Suddenly the big powerful boxes stitched together with string become mere silos for the data. And we, the users, reach a point where there is true saturation. At what point will the capacity for data transfer outreach the collective human capacity for making use of it?