Innovation in Software

Last weekend Rails Rumble took place, which is where teams of developers attempt to write an entire web application in a weekend. It is a testament to the power of Ruby on Rails that professional quality web applications can be built from scratch in just a couple of days, and more development teams should attempt to work this way.

Next week I’ll review each of the applications in more detail to look at how the lean development process worked and didn’t work in each case. For now however, consider going to vote. And I’d ask you to consider voting high for the Bartender application – it’s a Mancunian team, and the perfect accompaniment to a bank holiday weekend.

In the last few weeks, I’ve been evaluating source code I’ve written over the last year under closed source commercial agreements. I did this partly to try and look at what I did wrong and learn from those mistakes at a business level and technical level, but also to find out if there are common themes to anything I was working on that I could extract to make my life easier in future.

What I found shouldn’t surprise most people. Everything I did was about building forms, allowing data from those forms to be collected, and then looked at in some way. Every project looked like this:

• Define a form
• Build form
• Users input data into form
• Save data
• Look at data

Sometimes I would define/build the form and deliver the app, but at least two apps included the ability for users to build forms them (basically, survey software).

In fact, all web sites with UGC follow this model at some level – a blog engine is a form for a post, followed by the display of entered data in a particular way or set of ways. YouTube is a set of forms to handle file uploads followed by some data processing, and then displaying that content.

That’s quite interesting, because we have so far thought about web applications being custom little pieces of code in almost every case. Rails and similar frameworks enforce this model however, and make it easier by doing a lot of the heavy lifting involved in describing those processes. However, ultimately all we’re doing is putting lipstick onto relational databases. If that’s all we’re doing, surely there is a quicker way of doing it?

I was thinking that each of those stages in each application is already defined somehow as a real world business process we want to map into software. Why can’t we find a way of being able to quickly define a business process in code, and then just generate an application based on that definition?

It seems I’m a few years behind in my thinking. Business Process Management (BPM) is a mature area of development with a gazillion standards out there and it would seem XPDL is the XML-love of choice for this crowd. Digging further, I found something even cooler for the Rubyists amongst you: OpenWFEru nicknamed ‘Rufus’ amongst developers seems to have most of the heavy lifting done for you. The example looks promising.

This is interesting for me, because I have to build a couple of pieces of software soon which really are understanding workflows within teams, and I’d rather spend more time reading 19th-century books on political philosophy (hey, everybody’s got to have a hobby).

As a result I’m trying to work out how to hammer this into those projects. If I can find a way of being able to take the “I’ll look after the forms for you” aspect of Rails with the ease of defining workflows found in OpenWFEru and coupling them could you define a bunch of schemas and a workflow and then generate a better scaffolded Rails application? Would the controllers be able to mature over CRUD and do something more useful? Or could you build a CMS in Rails that uses user-generated workflows to allow for quickly hacked-together custom applications? It’s an intriguing bunch of thoughts…

On the 1st March (next Thursday, as I write this), I’m planning to do a mini re-launch of Vagueware.com and with a new website comes a new business model.

I will be phasing things in over the next 6-8 weeks. First, the “open innovation” phase, where ideas are thrown around, submitted and developed by anybody with an interest. Then the time to turn some of those ideas into real open source code, so anybody and everybody can develop the ideas further. And of course, run useful software. Finally a services model to make all your Rails deployment nightmares go away, and to bring some cashflow around those open source tools – support, maintenance, bespoke customisation will feature as well as hosted SaaS solutions.

However, I have a conundrum.

I’m busy finishing up work elsewhere right now. I have some todo items on older projects to close, and I am also involved in a second company completely unrelated to Vagueware that desperately needs some of my coding love right now. Time between now and next Thursday is tight. And the code base for the next version of Vagueware.com? Well, to be honest, barely started.

Right now, I need the splash page, and the ability for users to create accounts, edit wiki pages, for me to be able to structure some CMS around it all, add some forums, etc. It’s not a huge job, but to code it up from scratch even with the existing plugins out there might take longer than a week given other pulls on my time. It’s obvious that given the focus on it being a Ruby/Rails led set of projects, the time should be taken to develop the site itself in Rails.

There is an alternative though. Use something else other than Rails for now. Drupal does nearly everything I need and I could configure it all up this weekend ready to go. But it’s PHP. And that feels like a lie to me, and a lie to the other people I want to involve in this.

I could go meta and suggest that the first project is a tool to replace the site in Rails that would then allow Vagueware to become self-hosting. At least then the ideas would get started, and providing I was able to export everything from the inter-rim site when the job was done, nothing would be lost. It would allow those people who want to show up on day one to be involved in developing a community tool useful to them, and even have an input into how projects should be handled in the future. It could be an advantage.

It still feels like a major hack though. Anybody have any other ideas? I could delay the launch by another couple of weeks, but I’m keen to get started ASAP.

All over the Ruby on Rails blogs right now, people are getting excited about the upcoming release of 1.2 and all that it brings.

However, one of the comments on the announcement post above, strikes a chord with me:

jesus on 23 Nov 08:42: Release early, release often. not exactly right? Why do we had to wait 8 month were a lot of features could have been released as say rails 1.5? Watching rails_edge an break some stuff with an update is not for everybody. I don’t want to figure out stuff with no documentation for why it breaks my app now. So please do more interim releases.

(sic, throughout)

One of the biggest problems I have with myself is not releasing early and often. I spend way, way too much time trying to get things perfect, rather than just get something up no matter what state it’s in. I try to change my behaviour, but it’s not clicked yet that users just want regular, small, incremental updates.

It’s nice to know I’m in good company, but the way 1.2 has been handled is a little annoying. I’m glad it’s here but I would have much preferred a regular schedule of small releases every 8 weeks and deal with 4 small incremental updates rather than face one great big update. I have no idea how much work this update is going to create for me right now, but I’m thinking that there are some dozen or so applications I have belonging to clients that will need checking carefully.

Even so, I’m grateful for all the hard work done, and will hold myself back from biting the hand that feeds me too hard. It must be a bit of a slog as a coder if all you face is criticism and no praise whatsoever. So, if DHH or any of the Rails core are reading this: thanks for all the work.

Sun is going to be giving full support to Ruby, and will be shipping a Ruby parser built on top of the Java Virtual Machine apparently with Java and Ruby both being fully supported by the VM.

I still am not sure if this isn’t some weirdo-World practical joke. I expect to wake tomorrow morning to find this is just another version of the Parrot Joke (which eventually became a real project).

If true, it means Ruby is about to get incredibly mainstream, and I’m going to spend the next five years having some real fun. It also means Joel is going to look even more stupid with regards to stating that Ruby is not ‘enterprise-ready’.

One of the biggest problems with Ruby right now is the current parser is slow and we’re having to jump through hoops just to get performance up. With web applications it’s not a major problem until you start to scale, and then most of your performance problems can be ironed out with some cacheing and database schema optomisation. Through more clock cycles at the database server, and most web apps get quicker – Ruby isn’t the slow part in that equation. However, this changes the game – _if_ the parser Sun ships is nice and fast.

The mailing lists however are strangely quiet. It may be that too many people want to adore Matz for longer, or that the performance will suck against the new official parser in development, or that nobody believes it. Or it really doesn’t matter.

But I say that when a language gets mainstream commercial support from one of the largest tech companies on the planet, that language has arrived and we should be grateful for it. I only hope the Java kids don’t steal our lunch money when we turn up and we all play nicely together.

The above link is to a thread that points eventually here and here that both help convince me this isn’t some warped joke.

I always thought there was something interesting about Sun. I think they might be about to become my favourite company again. I might even take another look at Solaris in coming weeks to see if they’ve made it suck less than the last time I played about five years ago. See – this is smart thinking on their part. They’ve got me interested in what they’re doing again. Tres cool, non?

The above cartoon is by Hugh MacLeod of gapingvoid.com – warning, these days he talks about a particular brand of wine a lot, but his cartoons are often cool. I’d quite like this one on my business card. Thanks to his nice licensing terms, I might be able to do that.

When he published this, he pointed to a story by Scoble about the bubble people are now seeing. There are some important things to note about this bubble:

1. It’s only being seen by those of us inside the industry. If I talk to most people outside of the industry about social sites, Web 2.0, Ajax, whatever, they haven’t got a clue what I’m talking about.
2. There is a long, long way to go before this one bursts, but I don’t think the web in 12 months time will look close to what we have right now – it won’t burst, it’ll just look more sensible.

Let me give some background to where I’m thinking here, and start by making a controversial point: Technorati sucks from a social software perspective. So does Google.

I think they suck because to be part of the network, to add to the community, you have to be a publisher. Most people don’t have the time, energy or inclination to become a one-person publishing band, or the money to become a 10-person publishing house. Nor do they want to make videos to put on YouTube or podcast their thoughts. Some people think publishing is something attention-seekers do, and they don’t want to be called attention-seekers. They’re not going to participate.

Of course blogging is attention-seeking, but it’s not a given that this is a bad thing. I want to draw attention to all sorts of things, and I’d rather get your attention by talking and educating you about Ruby on Rails or new technology, or innovation, rather than just put up a big billboard and take out radio ads saying “We Rock the Kazbar – Vagueware Ltd”, because I respect your attention way more than that.

So am I an attention seeker? Sure, of course I am. But I’m seeking attention for something I’m passionate about: my business, technology, innovation. I want to share it. You might not give two hoots about my business, technology or innovation, but you’ll be passionate about something else. People who realise that become bloggers.

However, as a result of a relatively small set of attention-seekers having all the kung fu right now, looking at the A-list of blogs – the ultimate list of passionate attention-seekers – is a pointless exercise. The only thing that is being used as a metric is how many attention-seekers like this group a bit more than a different group by way of the number of linked articles. This is a terrible metric to use.

Ultimately, it could be Google’s downfall as well. PageRank – whilst better than previous systems for ranking content on the Web – uses the worst metric possible: what publishers think is relevant, as opposed to what searchers and readers think is relevant (although they have taken measures to improve this).

What could be more useful is to find out what people of all persuasions – non-publishers included – like across the board. That’s why social bookmarking sites have got traction – it doesn’t take much to digg, reddit or del.icio.us something, you’re not some being publisher, you’re just some dude who clicked on ‘digg this’. That metric – people who were prepared to go ‘Yo! This guy has funk!’ – is a lot more useful. It means that people who are only a bit passionate get to play with less effort.

Except it’s still not great.

For starters, game theory gets all over this model, and sites like Alexa’s Top 500 can so easily be manipulated, all you end up doing is creating a whole new form of spam. As the social bookmark sites get attention from other niches outside tech – particularly porn – they’re going to have to deal with this problem.

The other problem is that it’s still not frictionless. Alexa tracks people who bother to install their toolbar, which most people really can’t be bothered to do. Digg and Reddit are contributed to only by people who can be bothered to click on a little button they remembered to save to their bookmarks bar, write up a few words and some tags and hit submit. It’s not as bad as publishing a blog, but it’s still effort.

All the Web 2.0/community sites are going to face this problem: social communities take work to build and you can’t build them, they have to build themselves. And once people realise they’re there, they are going to start spamming the heck out of them and game theory becomes game-on.

The potential though is huge. Reddit is doing something very interesting with niche sites – joel.reddit.com and so on – which has the potential not to just change the way we think about publishing, but how we think about search.

I’m not interested in what the Internet at large as a group of publishers thinks is the best result for ‘Ruby on Rails’, I’m interested in what a community I respect thinks is the best hit. It’s a better metric for me. If they get the sweetspot just right, the social bookmark sites can take market share away from mainstream search. Right now Joel has his own bookmarking site, but what if every community, every blog, what if they all had social bookmarking communities? Every blog becomes a search engine built by people you share a passion with – a niche community with value.

Even better, because this is so niche, you don’t care about scaling, you don’t care about game theory, you just care about trying to reduce the traction of participation. It needs no money, just passion. Passion is the steam engine of the next industrial revolution.

That’s where the Web 2.0 crowd need to head. This bubble is wrong because we think it’s about Ajax and Rails and reflecing logos. Technology and shiny buttons aren’t what is making the net different in the next 10 years. It’s the people, stupid!