Innovation in Software

Who would have thought that one of the most innovative players in the hosting and web application industry would be a bookshop?

One of the big problems with Amazon’s web services is that they aren’t that great for permanently hosted web applications. There’s the dynamic IP addressing issue (which weoceo will look after if you have the cash) and the serious problem of how to store your database.

S3 is very nice, but it stores flat data, and certainly not anything as fancy as SQL tables. Until recently there was a hacky way to do it with a special storage engine for MySQL, but just looking at it made me nervous about my data.

Well, Amazon have decided to fix this issue. I received this email from them this morning.

“Dear AWS Developers,

This is a short note to let a subset of our most active developers know about an upcoming limited beta of our newest web service: Amazon SimpleDB, which is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database – real-time lookup and simple querying of structured data – without the operational complexity.

Were excited about this upcoming service and wanted to let you know about it as soon as possible. We anticipate beginning the limited beta in the next few weeks. In the meantime, you can read more about the service, and sign up to be notified when the limited beta program opens and a spot becomes available for you. To do so, simply click the “Sign Up For This Web Service” button on the web site below and we will record your contact information.

Learn more and sign up

Sincerely,

The Amazon Web Services Team”

So, there we have it. No more managing DB clusters. Scalable database tables, which once the beta is over will likely come with an SLA. Assuming that this just sits on top of S3, we might even be able to host our data inside the EU and get ll warm and fuzzy about protecting customer data properly.

I’m not sure this will be based on a standard set of DB libs but I expect we’ll see 1-line hacks to make it work with Rails, PHP and a host of other app frameworks within a few weeks.

I’m in.

Service Level Agreements are a must-have for Enterprise clients and it has surprised me that so few web companies have used them as a route to making money: if you don’t need an SLA, take the app for free. If you do want an SLA (because say your entire email operation is running on our web service, say), then you need to pony up some cash. It’s worked in open source, so I think it’s a no-brainer for an industry that is service-orientated at its core.

Good news then that Amazon S3 has today announced an SLA which means if they drop below 99.9% uptime per month you can have some cash back. You get even more money back if they drop below 99% uptime. They also agree to give you 60 days notice if they want to get rid of you for any reason – but don’t have to give that reason.

It’s a step in the right direction, but they could make even more money by offering even better SLAs if customers are prepared to spend more money to get them. That money would be capital Amazon would be free to invest in infrastructure which not only enhances S3, but Amazon’s core systems and business.

Almost twenty years ago a process was started that culminated in the recent announcement that the game of checkers (or ‘draughts’) can now be played flawlessly by computer. The technique is a brute-force attack, and so now there is a computer somewhere storing every single possible combination of pieces and able to work out the optimum next move in every scenario.

It turns out that the ‘best’ result will be a draw. It might not like look tic-tac-toe, but when you have mapped the complexity out enough, it is eventually just as futile. It’s going to take the computation and storage of 10²⁰ more positions than checkers took, but within a few decades we should have the computing ability to map out every possible game of chess. My prediction on that: white will always win if both sides play flawlessly.

Ron Evans predicts that it is:

“… only a matter of time before sufficient computing power allows the machines to contemplate eventualities of which we have not even postulated the existence.

The Singularity IS coming…”

“Only a matter of time” is a lovely prediction to lob into conversation. Sure, it’s only a matter of time. It could be 15,000 years mind, but we’re in no hurry…

I’m sceptical about this happening any time soon, simply because the entropy of the natural World is so vast and immense that it makes a chess game look like child’s play. Sure, us humans might have a problem understanding the size and scope of every conceivable move in chess, but compared to every conceivable possibility within the World around us, it’s nothing.

There are however, interesting applications we can start to conceive of now. Game theory has been abused by war-mongers in the Rand corporation to the point it no longer has any real credibility, but if the flaws are removed, it’s possible for computers to start finding new theories we had not yet considered. The impact within systems that are bound by rules and predictable behaviour (currency trading, for example) could be huge.

What really intrigued me however was something the creators of the ultimate checkers program had to create as a byproduct of their work. The limitations of current hardware meant they had to get innovative about how they actually stored all of those possible positions:

“For example, they stored the outcomes for the 39 trillion possible positions for endgames in a mere 237 gigabytes of computer-storage space, an average of 154 positions per byte. The mathematicians are now applying these techniques to bioinformatics, looking for ways to manage the massive quantity of data generated by the sequencing of genomes.”

We’re delving into work where the storage requirements are becoming immense, and the last decade of having “enough” storage for most work meant we didn’t need to get creative around storage and search algorithms. The future is perhaps not about taking generic algorithms and applying them to our data, but finding new mathematical models of representing what we need to and creating domain-specific algorithms.

The singularity might be a way off, but finding a way of getting there is going to be intellectually stimulating, regardless.