A friend sent me a link yesterday with the short description of “Intriguing…”

It was a link to Clay Shirky’s article Gin, Television, and Social Surplus and indeed it is an intriguing article where he sets out his initial thesis thus:

I was recently reminded of some reading I did in college, way back in the last century, by a British historian arguing that the critical technology, for the early phase of the industrial revolution, was gin.

The transformation from rural to urban life was so sudden, and so wrenching, that the only thing society could do to manage was to drink itself into a stupor for a generation. The stories from that era are amazing– there were gin pushcarts working their way through the streets of London.

It’s not a new theory, however I’m not entirely sure it’s completely accurate. Urban life is not a new invention: Rome at one point is reckoned to have had 1 million citizens, and Athens had 300,000 citizens before it. Whilst they both had their debauchery, nobody has ever suggested that Rome needed wine and orgies in order to function as a city.

His parallel starts to get more interesting however:

If I had to pick the critical technology for the 20th century, the bit of social lubricant without which the wheels would’ve come off the whole enterprise, I’d say it was the sitcom. Starting with the Second World War a whole series of things happened–rising GDP per capita, rising educational attainment, rising life expectancy and, critically, a rising number of people who were working five-day work weeks. For the first time, society forced onto an enormous number of its citizens the requirement to manage something they had never had to manage before–free time.

And what did we do with that free time? Well, mostly we spent it watching TV.

I’m not sure the “wheels would have come off”, but there is no doubt that even people on very poor incomes have more free time than people of similar economic standing would have had for many millenia - if ever.

He goes on to talk about this surplus of time as something useful, interesting and powerful. His first example however directly contradicts my thoughts around The Vision Thing:

And I’m willing to raise that to a general principle. It’s better to do something than to do nothing. Even lolcats, even cute pictures of kittens made even cuter with the addition of cute captions, hold out an invitation to participation. When you see a lolcat, one of the things it says to the viewer is, “If you have some sans-serif fonts on your computer, you can play this game, too.” And that’s message–I can do that, too–is a big change.

Actually, it’s a change, but it’s not one we should embrace unless we say it’s the thin end of the wedge. That eventually something useful and interesting is going to happen and society starts working on interesting things. Clay goes on to talk about how if even one slither of that time of staring at the flashing box in the corner is used to do something productive, it means something interesting is going to happen.

Let’s say that everything stays 99 percent the same, that people watch 99 percent as much television as they used to, but 1 percent of that is carved out for producing and for sharing. The Internet-connected population watches roughly a trillion hours of TV a year. That’s about five times the size of the annual U.S. consumption. One per cent of that is 10,000 Wikipedia projects per year worth of participation.

I think that’s going to be a big deal. Don’t you?

To an extent I agree. I don’t know what 10,000 Wikipedia projects per year is going to look like, but there is no doubt that something, somewhere is going to happen of interest.

But what are those 10,000 projects? Do we have the creative ability to do 10,000 useful things every year? Do we have the will to do something more interesting than throw sheep at each other or spending our entire time photoshopping memes? Time will tell.

Many years ago I did some freelance writing. Some of it was painfully dull (filler articles for free magazines), some of it bizarre and seedy (your suspicions about readers’ letters in porn mags are well-founded: they are sometimes written by paid writers), but the biggest lesson I got from it was that it’s hard to make a decent living with that as your main gig.

When you need to rely on artistic output to pay the rent, it doesn’t take you long to realise that unless you’re going to get picked up by a large publisher or music label, you’re going to need another job.

Recently I’ve been thinking about this problem and the related crisis in the music and film industries in some detail. At its simplest, the problem is this:

People want to consume entertainment, but they do not wish to pay for it.

Artists do not have the right to be paid whatever they feel they are worth, they must compete in a market and persuade people to hand over cash just like any other industry. Punitive measures such as taxing consumer products in order to force payment of artists is in my opinion pure idiocy. We need to think instead about encouraging people to pay for the entertainment they love. I think that requires a few things:

1. Consumers should not have to fork out more money than they feel comfortable spending
2. More of that money needs to land in the artist’s pockets rather than distributor’s, so that artists on the ‘long tail’ can make a living off a smaller fan base
3. Artists need to find new ways to grow and engage with their fan base

Thankfully, the Internet makes all of these much more practicable than ever before.

One solution to the first problem has recently been played out with mixed results by Radiohead, but wouldn’t it be interesting if we had some sort of “tip jar” system in place for all artists? You download something via P2P, like it, and you can make a donation - of whatever size you want - to the creator. Well somebody is working on that but the question is whether it will ever work.

One artist working with a non-digital medium (paintings) has given this a whirl and it seems to be working. Ali Spagnola will - when it’s your turn - paint a picture just for you based on a theme you suggest and then send it you free of charge anywhere in the World. It’s not a con. I know this, because I’m currently staring at this picture painted for me sometime last year. Payment is completely voluntary. I’m ashamed to admit I still haven’t got around to throwing some money into the tip jar, but I’ll rectify that mistake this week. The painting has grown on me. I would miss it if I lost it. Ali deserves to be able to eat for giving it to me.

Does Ali make money? Perhaps. Do Radiohead? Definitely. So, it’s a model with potential.

As for the distribution problem, well I think it’s clear now that the current relationship with artists and the distribution chain is going to die within a matter of years. A band or a writer can now distribute directly via their website, and even authors can publish books cost-effectively without the need to get men in suits and lawyers involved. There is an issue of how to manage all this and as Kevin Kelly discovered when researching this, being your own tour manager, promoter, lawyer and roadie can be a gruelling and unprofitable exercise.

And then we get to audience engagement. The Internet has blown that apart as well - artists can now have a direct conversation with their fan base via blogs, social networking websites and video sites. It doesn’t scale (how do you stay personal with fifty million fans in 150 countries?), but that would be as they say “a nice problem to have”. Most artists don’t know how to do this well - they’re musicians, writers and film directors, not PR specialists - which suggests there will exist a niche industry helping bands do this very cost effectively within a few years. The current promotion and PR industries are not a good fit for where the industry is heading, they need to change.

As for growing your fan base, I agree with Robert Rich’s words in his message to Kevin Kelly:

Companies can use demographic models and track people’s search patterns to pander to their initial tastes and to strengthen those tastes, rather than broaden their horizons. This problem doesn’t lie within the technology of the internet, but within the realities of capitalism and human psychology.

There is a problem here with collaborative filtering - it’s locking us into tastes, not broadening them. However, it can also be the most powerful tool an artist can have working for them.

Four months ago I had never heard of The Courteeners and yet last Saturday was in the crowd at their sell-out gig at Manchester Academy having paid several times face value for the tickets off eBay. That only happened because last.fm algorithmically said “you should listen to these guys, because you like James”. So far The Courteeners and their label, promoters and distributors have directly received at least £30 off me they would never have got without that technology helping them. I expect they will get hundreds off me over the next decade providing they keep doing something I like.

However, I’d like to share that music. I’d like to say to my friends “look, listen to this, you’ll like it” and give them a copy. DRM and the law prevents me. It is working against them, because I know for a fact I could recruit at least another half dozen fans for their next tour and album release. They are working against me by insisting I do not put their album up on a website for anybody to download and listen to. I will happily work as their unpaid promoter and recruit whoever I can into giving them money, but that little circled “C” prevents me. They could have licensed it under a creative commons license, but they chose not to.

This one act alone has probably cost them a couple of thousand pounds in future lost revenue just through me. Scale it up to the 2,000 people who were at that gig the other night, they’re probably losing millions. Not millions in five years when they try and break America: millions of pounds right now, this week.

So, we need to find more new ways to openly and cheaply distribute art and leverage a fan base so as to be able to make a decent living - perhaps even an indecent living - for artists and fans alike. I have more ideas on how to make that happen, but I will share those with you tomorrow.

As most of you now know, I’ve spent a lot of time thinking recently. I’ve even been ranting a fair bit. I sound downbeat when I talk about it (as I did at GeekUp this week), but there are small shards of optimism I can extract from all of the discussion, and it’s those I’m going to focus on.

For several years now, an idea has been bugging me. It addresses hard problems, big social issues I care about, and I believe I can actually do something useful, innovative and entertaining in the space. I have called it Kagtum.

The article I found the most upbeat about my rant has actually make me think it’s worth dealing with this set of problems again from a fresh perspective and push the ideas forward into code.

In the years thinking about these problems I’ve spent a lot of time thinking about how to make it happen. I’d spoken to a lot of people. I’d sniffed around investors. I’d watched blogs by people in the same space and saw where they were heading and thought about where they might trip up. I’ve watched people trip up (90% of them by focusing on the wrong thing), and made mental notes.

Today, something clicked. I realised can actually make the beta happen, in my spare time quicker than I thought and not break a sweat whilst having fun. And, even better, I don’t need to drop any of my current commitments around vagueware, idea banks, other business ideas.

That sounds like a plan.

So, it’s time to kick things off. I’m trying to recruit a user base of people interested in news, current affairs and emerging technology against which to bounce ideas off. I’ve started that process by setting up a page on Facebook and CPC’ing people on FB interested in those topics. If you want to become a fan, please do so and you’ll be behind the wall on the first release.

The Vision Thing will continue. However, now I’m going to try and deal with the issues not by complaining, whining, ranting and criticising, but trying to find a way to be optimistic and beat a path. I will aim to show, not tell. Maybe I’ll mess up and people will laugh, maybe I won’t. Should be fun finding out, either way.

At tonight’s GeekUp (Briton’s Protection - just 50 yards from the usual venue - 6pm), there is another social discussion event. This time, another area I spend a lot of time thinking about is up for consideration. From the announcement:

Discussion Topic: “The Future of the Internet”

• How do you see people using the internet in 5/10 years time?
  • What features do you want to see browsers supporting?
  • Will people still be using browsers? If not, what will they be using?
  • Does anyone actually use 3G video chat? Will VoIP mainstream follow too?
  • Will Google always be the number one search engine?
  • Will Google be even bigger? Perhaps it might run our lives …
  • Will IPv6 actually be adopted by the masses?
  • Anyone up for a 3G wireless dongle biometric implant to hook your memory up to the net?!
  • Semantic Web - is it the future? what does it mean?

How we intend to get through that lot in a couple of hours I have no idea. I expect I will be writing up notes and reporting back tomorrow if people can’t make it, but if you can make it, you should.

For some reason, Mephisto isn’t behaving on my server right now, so comments aren’t working. I plan to move the whole shebang over to Wordpress sometime “soon”. That said, I’d like as a gentle introduction back into me blogging here again about “Innovation in Software” (less ‘what is happening in Manchester’ in future), to talk about comment systems.

This cartoon from the excellent xkcd strip sums up my problem with comments right now:

The problem I have is this: on any popular system where users are allowed to comment, as the number of users able to comment without fear of genuine peer review increases, the signal-to-noise ratio drops exponentially.

In other words, if I and 5 million other people can comment on a YouTube video without any fear of us being reminded of what we said not just in the future but the very next time we meet a friend, we are more likely to be flippant, irrelevant, and “noisy” than if we knew people whose opinion of ourselves mattered to us were going to be reading that comment and evaluating it.

It’s why social network status updates and posted items are relatively sane and measured and why blog posts are more considered. We care about what the readers think, because what they think will have a direct impact on our future relationship with them.

So, whilst thinking about kagtum a lot recently (background if you’re unaware), I’ve been thinking about this problem. How do you allow for user comments without them descending into noise?

The “mission statement” for kagtum in its current form is something along the lines of “delivering relevant news and event information”, where “relevancy” is the secret sauce that gets quite complicated. How do we make sure every comment you see is relevant in order for it to stay within that mission statement?

I have a possible answer, but I need to keep it close to my chest for now. Normally my ideas are thrown out into the wind as being worthless, however my answer has a direct consequence on execution of a business plan. That said, if you come and meet me at an event and ask me, I’ll tell you what it is if I trust you. ;-)

The way I understand it, there are two groups inside Microsoft right now: I shall refer to them (even if nobody else does) as the Old Guard and the New Guys.

The Old Guard are the guys who built Microsoft in the first place. They’re the ones that we might consider the Evil Empire. They thought about software as a means to make money in itself. IPR and tools like DRM were critical to their thinking about how software should work. They’re the ones the EU don’t like. They’re the ones we’re a bit tired of in the open source community. They want your money.

The New Kids have seen a little more of the World as it really is. They think that software is a tool to sell services, training, knowledge, and that things like IPR and DRM get in the way of incredible creative freedoms. To them being able to mix in with as many people as possible is more important than trying to make sure that Microsoft locks you in: they want to win by producing the best tools possible. They want your heart and soul, feeling your money will follow.

The Old Guard is, naturally, getting older. They’re retiring. The New Kids are getting more important. They’re rising through the ranks. They’re able to make decisions. They’re the future.

We’ve just seen another small move in the battle that the New Kids are winning.

Give them time. They’re getting it.

I have a running battle with Yahoo! in terms of their “foreign markets policies”, but not with their core design and tech teams. I think then, the news that comes to me via Information Aesthetics that they’ve shut down their entire design innovation team is utter lunacy.

Yahoo! is struggling to keep up. Innovation and creativity is how you leap-frog and out-Google Google. It would seem they’re no longer that keen on doing that.

As such I have to say if you’re holding onto Yahoo! stock, consider selling: I think this is the first step to them going bust about four or five years from now. They’d best just hope that Microsoft still want them.

No matter where it happens, if somebody out there is coming up with crazy ideas in software, I get to hear about it eventually. It’s just rare I get time to write it up.

This evening I caught on to Breach Candy Group’s idea for what they call AML or ‘Aesthetics Markup Language’ and decided to talk about it pretty much straight away. This article is under the “Philosophy” category for a reason. If you’re not in for some deep thinking, move along.

Their idea is to be able to define the aesthetics of a piece of film or art in a standardised markup language so that systems may be able to perhaps generate new content of a similar aesthetic style. For example, as they say:

“Let’s start with images. We could start off with the following variables:

• movement (speed of movement = speed of change in pixels?) This could be later used to analyze some rhythm of change.

• brightness and contrast (how would this be tracked = the relationship or average of pixels in any given location on the video?)

This could also later be use to analyze things such as harmony of composition, direction of lines in the mise en scene, etc. We would have to come up with a set of principles from art history and composition and see how these could be determined in the screen etc?

• color range (this would probably have to be RGB values in the image itself). This would probably move us into the realm of things such as monochromatic color schemes, bright colors, harmonious colors, contrasting / oppositional color … ie to use some notion of color theory to provide patterns in certain styles of video etc. I’ve studied this in high school so will be fun to revisit some principles of classical painting.

So I suspect what we need to do is set up a very simple experiment / structure in place that can be developed and extended depending on need. In other words, we need to develop … AML (Aesthetics Meta Language) … a basic language structure that would describe what the variables are within any analyzed video. This language, I suspect, could be then developed into the interface between the language of aesthetics and the computer.

Something like this:

//AML: “DEBBIE DOES DALLAS”

134 12 58 22

It’s a nice idea, but they’re missing a trick. The thinking that got them to this point to me seems much more interesting.

They argue that most of the complexity we witness in the World is a repetition of simple things that go on to form complexity. This theory exists behind cellular automata, fractals and more.

Cellular automata are incredibly important in our understanding of how complexity is created out of the simplest building blocks within the Universe. Take for example the image to the left of this text. It might look like something taken from a microscope, but it’s an image created in software using a CA routine that was less than 100 lines of code, simulating “dictyostelium slime mold”. You can find out more about exactly how it was constructed at the site I got the image from.

This idea of complexity emerging from simple rules is particularly important in the field of Artificial Intelligence for reasons described in a philosophically entertaining manner in the AML article:

“Here the key question is that what algorithms could be used to model the way humans think and thus be used to guide machines to perform complex tasks. The philosophical implications of this are even more profound than getting a robot to recognize faces or clean a non-linear toilet bowl. That is, if human intelligence is, in fact, highly programmable, what then defines humans from machines? This goes two ways: machines-as-humans and humans-as-machines. In other words, AI defines rationality a certain way with certain presupposition of what logic, thinking and consciousness are and how they can be pragmatically simulated in computers. But as importantly, if we look at the concept of rationality and how it has been historically constructed, this has always presupposes a certain “image of thought” that has excluded all that would not fit into the sphere of rationality (intuitions, insanity, madness, illogic, spontaneity, absurdity ….). So how would we then understand the blurred boundaries of man and computer (as intelligent forms, which neither technically speaking are) and the human-computer assemblage that is making the old notions of rationality/humanity perhaps increasingly difficult to defend? Humans as (programmed?) repetitions: computers as programmed repetitions: natural intelligence: artificial intelligence: natural stupidity: artificial stupidity …”

Think about that for a moment. The point is not to be “correct” in the sense of making an algorithm “smart”, but make it mimic so it has the ability to be just as insane, dumb and mad as humanity. If you understand the Turing Test correctly, we will have produced an algorithm capable of passing it correctly, when we produce an AI capable of insanity, melancholy and psychopathic behaviours. Asimov’s 3 rules of robotics can not exist in a machine capable of passing a Turing Test, in other words.

This is a good development of an argument on the part of the Breach Candy Group, partly because it’s intuitively correct but also because it follows the science we have to hand quite nicely. It’s a shame then that in itself it probably undermines the need for AML on which they built on these foundations.

AML is about how a computer would process very specific values - contrast ratios, line measurements, etc. - yet they accept themselves that what they need to do is instead mimic the way a human would describe the aesthetic. No film director would talk about precise values of contrast settings or line movement, but instead would find a way to talk about colour, tone, depth, warmth, speed and so on in a much more abstract way.

In other words, to achieve what they’re hoping for they need to develop an algorithm which is able to mimic the human way of parsing film aesthetics, train it by making it “watch” films and then ask it to produce something “like” a subset of them. They’re trying to find a way through by producing a way of retaining knowledge about an aesthetic in a standard form, but as anybody who has read “Zen and the Art of Motorcycle Maintenance” can tell you, there is much more to an aesthetic quality than how you describe it mechanically.

In fact, the point about a distinctive aesthetic quality is that whilst we know it when we see it, we all see something different and would all describe it differently. Let me try and make my point by using an image that has probably had its aesthetic qualities dissected, written about and analysed more than any other in the history of aesthetics.

The Wikipedia article on the painting has quite a detailed summary of some of the aesthetic judgements made. There are two groups these arguments can be placed: those that are algorithmic and those that are subjective.

It might be reasonable to produce in AML those that are algorithmic. They can be measured precisely - golden ratios, pyramid composition. However how exactly do you describe something as subjective as “the composition of the figure evokes an ambiguous effect: we are attracted to this mysterious woman but have to stay at a distance as if she were a divine creature” in a markup language?

I don’t have answers, only questions on this one. Interesting thoughts though. And if they can be resolved, we’d be a major step forward to understanding AI - and ourselves - much better.

Last week I was asked to comment for Crain’s article this morning on Yuuguu. I had to offer up a disclaimer, as I do now, that I have done a little bit of work for Yuuguu and I’m under NDA on what I know about the specifics of the internals of their technology.

Typically when asked to quote I give the journalist way more than they need in the knowledge they’ll pick out the one sentence that fits the story they want to tell. On this occasion what I said in full was:

“Yuuguu is interesting because they’ve executed a plan quite wisely. Rather than get overly clever about technology as many start-ups in the web sector do, they’ve used a suite of established technologies, understood user expectations and then combined them expertly. You don’t know how hard it is to do that right until you try.

They’re also very different to the other IM services out there - they’ve skirted around the problems people have with VoIP in a way that gives them a solid, proven business model.

They’ve taken on multiple markets at once in a way established players in those sectors are going to have a problem responding to quickly.

Even better, they haven’t spent years trying to come up with proprietary protocols and re-inventing the wheel, but instead cleverly blended together the best of what works and extended it to produce something greater than the sum of its parts.

They’re in a tough area and they’re competing on multiple fronts, but I think they’re in a strong position. The IM sector is not engaging with the audience Yuuguu is and uses technology that would scare most IT admins away from deploying it anyway, the web conferencing sector still don’t “get” the modern Web in my opinion, and the companies selling shared desktop solutions have just had Yuuguu chop their business model out from under them - but many have yet to realise it yet, so aren’t responding.

The only real threat might come from better SIP services threatening their revenue model and customers communicating on voice outside of the Yuuguu system. Having spoken to the guys at Yuuguu though, I wouldn’t be surprised if they don’t already have an answer to that.”

I think Yuuguu are a clever outfit that are doing something quite unique. They aren’t innovating in the madcap “let’s reinvent the wheel way”, nor are they jumping on a bandwagon and trying to use the words “social networking” in their business plan. They’ve looked at what does and doesn’t work, found a way to make something that works better and then established a set of technologies based on best industry practice to make those ideas happen. And all the while, the business model is sat right at the core of what they’re doing.

I hope Yuuguu does take off, and does make considerable profits in the long-term. It would be great to see a local tech start-up fly.

Have you ever been shown a ‘database’ by somebody who doesn’t really know what a database is? You know it’s going to go badly when showing it, they double-click on an Excel file and you are confronted with a grid with ‘Name’, ‘Address1’, ‘City’, etc. across the top and a huge number of rows below.

Given that most people use spreadsheets like databases in that way, it was only a matter of time before one of the big online spreadsheet applications promoted it as a feature. And so it came to pass.

Seth thinks this is great but I’m not so sure. In a sense, it allows for gathering of data for analysis at a quantitative level, but at the same time it breaks the line between polling, database work and spreadsheet crunching in a way that might confuse more people than it will help.

Perhaps on the other hand it’ll help stop us thinking about applications and more about data. We are moving towards an era where we care about who and what more than the how and detail of collecting the data.

One of the advantages of being almost 30 is that people less than a decade younger than you tend to think of you as being “wise”. Some of the staff in my local bar will ask me about everything from US politics, the Renaissance, Alan Turing, 1980s TV commercials and arcane facts about the early forms of Parliament. Cultured bunch, the staff in my local.

Last night however, it was my turn to learn. One of them had asked me last week about the phrase “Jumping the Shark” and where it had come from. Last night she told me the very next day after I’d explained it to her, she watched an episode of Scooby Doo (OK, maybe they’re not that cultured) where Scooby jumps a shark and that it had made more sense to her knowing what it was a reference to - it is one of the classic insider jokes within TV comedy. I then had to re-explain it all to the other people assembled. The conversation that followed was… interesting:

It went on in a similar vein for a few more minutes. More examples of the futility of the network, the silliness of the apps. Admittedly, none of them had left Facebook yet, but that might be that it’s rather hard to leave, as GeekUp and Co-working day regular Alan Burlison found out

These are people the social networks need. In their early 20s. University students. Bright, intelligent, aspirational. I have no doubt that within a decade most of them will be in the upper 25% of earners in the UK. Malcolm Gladwell would call them “sneezers” or something - they spread their likes and dislikes around their friends quickly. They set trends.

And in the last couple of months they have come to hate Facebook and MySpace.

Specifically, they hate that these networks have been opened up to people engaging in what is effectively a developed and sophisticated form of spam. They hate that they are being hassled via the social graph into doing “fun” things that are actually about as fun as receiving a hoax virus e-mail. They understand that their time and attention is important and its being wasted by sites that don’t respect that.

I have ideas for applications that will actually add value to the social graph and be of use to people in this group, but by the time I get to roll them out it could be too late - the people that make the platform interesting to me as a recruitment base for customers may have moved onto something else.

Facebook are adding features to improve the user experience as they learn how developers are gaming the system. They might win the battle in time, but ultimately they might have to give more control to users to block invites from apps that are not even remotely in their realm of interest.

This isn’t over yet. 2008 could easily be the year the social networks died.

Let’s imagine you are a toymaker. No, not some carpenter in a little workshop deep in Old Europe - a multinational that commissions studies on “pester-power” and are only slightly embarrassed by the fact some of your toys contain lead paint.

You have a trademark over a game, that quite frankly went out of fashion in the 1980s. Nobody wants to play it any more because it’s seen as dull, boring and just a little bit “fuddy”.

Then, one day, you notice sales are starting to rise. People are buying the game again. You can’t understand why, so you commission another report (hey, that’s your job) to find out where this new interest is coming from. A few months later, you have an answer - somebody has created an electronic copy of your game and made it available as an application in a social networking site. People are so crazy for it as a result, your brand is now gaining value and you’re going to have to think about how to cater for this new generation of players.

What do you do?

Well, if you’re Mattel or Hasbro and your games is Scrabble and the online app is Scrabulous on Facebook, you naturally send out cease & desist letters and hack off your new fan base.

The idiocy of this decision is monumental. Yes, you need to protect your trademark. Yes, you need to show that you’ve acted to protect it otherwise you can end up losing it anyway. Do they really think this is the way forward though?

I have to admit I’m a tad biased here. Here’s my Scrabulous stats screen:

As you can see, I’m one of those people who plays daily, and plays a lot. I’d hate to see it go. But that’s not why I’m writing about it here.

There is something new about the economy that is spreading around us. In the past ideas, trademarks, patents all were treated as if they had some inherent power that should not be discussed. People say they won’t discuss things because they need to be secret, that they fear the legal consequences. People don’t give up ideas until they’re “protected”. People guard words they invented as if they alone are the secret sauce to great riches.

Here’s the thing: that’s all bullshit now.

You want people to talk about your product, your ideas. You want them to talk, talk, talk, talk all day long. You want people to stand up and shout from the rooftops about your products, your patents, your trademarks. You want them to share their ideas of how your products could be made better. When they start doing that, especially when other people are providing them the tools to do it, you should think very carefully about whether you want to tell them to shut up.

Excuse the self-indulgent nature of this post, but I think I’ve been quite restrained on this for the last 12 months and hey - it’s Christmas.

At this time of year, two things always happen to me:

1. I get a little bit melancholic. My playlist changes to have lots of big sweeping chords (think ‘Atmosphere’ by Joy Division, some Band of Horses, some Vaughan Williams). I watch films about people enjoying a simple life such as Harvey or El Perro last night, and think about de-cluttering my own.

2. I start thinking about why I do this for a living all over again.

2007 has been a pretty good year to me. I’ve written about 10,000 lines of commercially deployed code which is a lot. I’ve written several million words in blog posts, e-mails and essays which for me is average. I have about 40 business cards I didn’t have at the start of the year of people I will undoubtedly work with at some point. I’ve also undoubtedly upset, angered or annoyed people unwittingly (sorry!). I’ve almost certainly got things wrong.

Still, now we’re moving to the point where I ask what it is I do, why I do it, and how. Every 12 months I ask the question about whether I want to hire staff, whether I want to go big or stay small. Whether I need to start phoning some of the VCs in that little stack of business cards. Whether I should just go and get a job and not worry about money again for a while.

Getting rid of all the clutter and thinking about what it is I do, I realised whilst reading this article this morning that there is a vision out that there that is so pure in it’s concentrated common sense that at some point this year I once again stopped thinking about doing good stuff quickly and just started doing stuff that paid the bills.

Make it free. Make it simple. Make it open. That was the plan in 2005, so what happened to it?

I read Getting Real when it was first released. I advocate and often use agile methodologies, but there’s something I’m missing. All my projects seem bigger than they need to be. Sounds like I have an agenda for 2008 forming already.

And this is the point. I can make those changes because I don’t work for somebody else. I can take those risks and make those things happen if that’s what I want to do. I can just get on with it.

I’ve been keeping a distant eye on Evan Williams for a while all stalker-like because he’s just so damned inventive. He created Blogger, Twitter and I’m sure there’s more in the pipeline. It’s interesting to see a great article about him at The Economist

Accidental innovation is fascinating at all sorts of levels to me. History is filled with stories of people stumbling into an idea, or taking things away and using radical constraints to improve creative thinking.

I built vagueware entirely by accident. I was filling shoe boxes with index cards of ideas for the things I really wanted to build and realised I should start putting them out there on the web, and I needed a tool to do it. Oddly, I’ve still not got around to putting the ideas online - they’re still on index cards waiting to be released.

Who would have thought that one of the most innovative players in the hosting and web application industry would be a bookshop?

One of the big problems with Amazon’s web services is that they aren’t that great for permanently hosted web applications. There’s the dynamic IP addressing issue (which weoceo will look after if you have the cash) and the serious problem of how to store your database.

S3 is very nice, but it stores flat data, and certainly not anything as fancy as SQL tables. Until recently there was a hacky way to do it with a special storage engine for MySQL, but just looking at it made me nervous about my data.

Well, Amazon have decided to fix this issue. I received this email from them this morning.

“Dear AWS Developers,

This is a short note to let a subset of our most active developers know about an upcoming limited beta of our newest web service: Amazon SimpleDB, which is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database - real-time lookup and simple querying of structured data - without the operational complexity.

Were excited about this upcoming service and wanted to let you know about it as soon as possible. We anticipate beginning the limited beta in the next few weeks. In the meantime, you can read more about the service, and sign up to be notified when the limited beta program opens and a spot becomes available for you. To do so, simply click the “Sign Up For This Web Service” button on the web site below and we will record your contact information.

Learn more and sign up

Sincerely,

The Amazon Web Services Team”

So, there we have it. No more managing DB clusters. Scalable database tables, which once the beta is over will likely come with an SLA. Assuming that this just sits on top of S3, we might even be able to host our data inside the EU and get ll warm and fuzzy about protecting customer data properly.

I’m not sure this will be based on a standard set of DB libs but I expect we’ll see 1-line hacks to make it work with Rails, PHP and a host of other app frameworks within a few weeks.

I’m in.