Innovation in Software

Last weekend Rails Rumble took place, which is where teams of developers attempt to write an entire web application in a weekend. It is a testament to the power of Ruby on Rails that professional quality web applications can be built from scratch in just a couple of days, and more development teams should attempt to work this way.

Next week I’ll review each of the applications in more detail to look at how the lean development process worked and didn’t work in each case. For now however, consider going to vote. And I’d ask you to consider voting high for the Bartender application – it’s a Mancunian team, and the perfect accompaniment to a bank holiday weekend.

If you ask the clichéd man in the street to describe his clichéd idea of Facebook user, he would remove his cliché kebab from his mouth, put down his cliché can of Carling Black Label, and through the use of awful diction and terrible grammar developed through his awfully clichéd state education he would paint a picture for you:

“Teenager, American or European, kind of person who likes to text a lot, probably quite immature, large group of friends, not much into going out getting sloshed, like wot I did when I woz a kid, innit?”

Well, remove yourself from this yob, and look at the actual figures.

North America and Europe contribute 65.8% of all Facebook users, and only around 10% of their users are teenagers. That means less than 7% of the actual Facebook user base is the cliché most of us – especially those of us in the industry – think of as their user base.

The majority are quite different. And I think once you understand the actual group using Facebook, you start to understand its popularity in some groups (and the derision it receives in others).

Obviously (given its history), people above graduation age are the norm. Most of us (myself included), use it as a platform to keep distant touch with people we ordinarily would not speak to year-to-year, never mind day-to-day or week-to-week. We are a generation of nomads, spreading out across the globe after school and University, keeping in touch with people via the light touch of the status update. Twitter is for people who are perhaps a little more frantic (and for me at least, it’s a great way to keep in touch with my peer group, people who I might not have met, but whose work I’m interested in).

It’s not, however, a tool for teenagers. Not really. They have a social group they see almost every day – they don’t want or need Facebook. Likewise, the people who don’t like Facebook are typically people who see their social groups regularly.

For those of us who are geeks, well our groups tend to be a little more dispersed. I can name friends and family in a dozen cities around the World I wouldn’t keep up with as much if FB wasn’t there. Those people who don’t have such far-flung friends, teenagers included, simply have no use for it: they can, you know, actually talk to them.

For some reason, until seeing that graph, that thought had never occurred to me.

I’m changing what appears over at the main website over the course of this week. I’ve been playing with some ideas, and right now I’m focusing on the structure rather than the aesthetic. The most common feedback I’ve had over recent weeks and months is that it’s not entirely clear what Vagueware does, how it does it, or for whom. That is a pretty major failure in marketing communication.

So, ignoring the specifics of the aesthetics, here is a glimpse of the structure of the new page. It will hopefully look more polished than this, but you can see where it’s going in terms of message. Let me know what you think in the comments.

Click for full-size image

I have my “Chairman* of Fly The Coop” hat on here.

Fly The Coop, as some of you will be aware, is an Independent & Provident Society (read: non-profit co-operative), here to help all you freelancers and SMEs out there collaborate and co-work on a regular basis.

We have an opportunity to get involved in the new Hackspace in the Northern Quarter and take over the first floor to call our very own. However, we need to do a feasibility study and work out whether this is even viable before we commit ourselves to taking the space.

So, if you have 5 minutes please do go and fill in the survey.

We’re also interested in finding companies who want to “sponsor” the space, without perhaps making use of it in the same way. Perhaps you’re a service provider who doesn’t need a desk, or you only need one desk but want to provide more support than one desk gives us, whatever. This sponsorship would help us offset move-in and first year costs whilst we get settled in. If you’re interested in sponsorship, please do get in touch with either myself or email info@flythecoop.co.uk

* Until the next AGM, anyway

Many years ago I ran FreeBSD on pretty much every system I touched. I loved it. The lack of political polarisation that touches the GNU/Linux community combined with the power of Unix under the hood sucked me in.

I even, almost infamously, had a rant about another operating system I now use daily. I was a zealot. I am ashamed of my attitude in that email – my only excuse is that I was still quite young, a rather naive and opinionated 26 year old.

Today, I use exclusively OS X, and occasionally boot up a virtual Windows instance to test code on MS code. I have a terminal open nearly all the time and often dive down to the Unix goodness, but more often than not the bulk of my work is in the browser, text editor, IDE or Mail app.

Why then, I was asked the other day, do I not just ditch Apple and get a Linux (or even BSD), machine to work with? The bulk of my work (except for iPhone application development) would continue as before.

My answer is very simple: carpenters don’t build lathes.

To explain, let’s first go back to that rant I had about OS X. About a month later, I ordered my first Apple laptop – an iBook G4 – and was very happy with it. What happened in that month?

I realised when shopping around, I needed to check out the chipsets used for WiFi in each laptop because FreeBSD wireless support was pretty hit-and-miss back then. I also needed to understand the graphics chip-set because I’d need to compile X if I wanted to run a graphical desktop environment. I then realised I’d need to assess pretty much every feature in terms of hardware compatibility to make a purchasing decision. And then, on receipt of my new laptop I would have to spend a few hours doing all the compiling and fixing, and I would then need to do this work every few months as part of an upgrade cycle.

Yes, I’d get the power and flexibility of my own tailored operating system environment, but isn’t that a lot of work?

I recalled a few months previously a friend was playing with his new smart phone. It was running Symbian and he seemed to be doing lots of prodding and poking with it. Enquiring what he was doing, he said he was “doing some maintenance” to keep it in working order. Hang on, was he effectively sysadmin’ing his phone? “Yes, I suppose”, came the reply. When you have to do systems administration work on your phone, your phone is no longer a tool to assist with your work, it is an object of work in itself.

It was this insight when thinking about my operating system choices that directed me to OS X: I wanted the power of Unix, sure, but I wanted it to just work. I wanted to be able to get on with my work, the laptop and operating system as tools rather than objects of work themselves.

This small insight has made my life a great deal easier, and still dictates not using any other operating system. Some of my peers see it as weakness (even more point to my original rant above), but I see it as spending my time doing the things I love.

And it’s also an insight I think that as developers we forget: are we developing tools that assist with objectives, or tools that are objects of work themselves. Are we building interfaces and suggesting business logic that means our customers spend time managing the behaviour to fit around them? If so, why?

In the last few weeks I’ve seen some really interesting “reductions” in functionality that aim to make tools more directly useful, rather than requiring some administration.

Take for example, the role of authentication. You need to “contain” all of your customer’s “stuff” in a way that is linked to their “account”. So we have user registration, user profiles, account activations, password resets, etc. Seems like a lot of stuff. Are we sure we need it?

Take for example, Posterous. Their home page explains it all:

All you need to do is email stuff to post@posterous.com and you’re up and running. No signup, no captchas, no password strength indicators unless you want to add them. You’re a person capturing stuff – why do you need to admin yet another web application?

BootStrap UK (which is worthy of an article in its own right as a concept), needs accounts but has a simpler registration system: just follow @bootstrapuk on Twitter and they’ll follow you and direct message you a password. Done.

There are other examples out there as well, sometimes using software to help break out “lathe obsession” elsewhere in society. One iPhone app I’ve seen reviewed gets the banking system to work around the customer, rather than demand the customer head into a branch to suit the banking system. Brilliant.

We often spend too much time trying to work out how a customers will need to behave around our application, rather than how we need to get out of the way and let our customers use what we produce in order to do their thing. In every customer we need to start seeing the carpenter more, and assume they – perhaps unlike many of us developers – don’t like tinkering with lathes for its own sake.

No matter. Try again. Fail again. Fail better.
– Samuel Beckett

Whenever I am asked by a client why Agile Development is a good idea, why TDD or BDD is the way to build a product, I can sum up the argument pretty succinctly: it reduces the cost of change to as close to zero as possible.

Recently I have been doing some reading around “Lean Development”. I have found Eric Ries’ Lessons Learned blog an invaluable resource for finding jumping off points into resources and ideas that are still very new and fresh to me. This morning whilst leafing through my new copy of The Principles of Product Development Flow, which I bought after seeing it recommended by Eric a few weeks ago, I realised I could succinctly reduce the arguments for lean development down to another mantra:

Lean Development reduces the cost of failure to as close to zero as possible

It doesn’t sound like much of an insight, does it. In fact, everything you read about this area is about testing Potentially Fatal Assumptions (PFAs), and market-validating feature sets: get your product out of the door, don’t invest too much in case it fails, if it succeeds then well done for lucking out. And of course, lean development is not just about failing at things – it’s about reducing the cost and time to delivery whilst also focusing on the only factor that really counts in product development: profitability.

However, when you let the idea about cheap failure seep in, you realise it challenges deep-rooted beliefs held in our society.

From the nursery school chart of “gold stars” all the way through our continuous grading of achievement in our first 20 or so years of life, most of us have learned through trial and error that failure is bad. Failure is (literally), for losers. Failure is not something you want associated with you, your ideas, your relationships or your life. Failure, we believe, is bad.

I think we’ve taken a wrong turn somewhere. Providing that the cost of failure is low – in terms of capital, resource, emotional investment, etc. – failure is useful. Failing quickly at something that was always going to fail, allows us to direct our attention to something new.

Our lives on this planet are short. We can only fit in so many projects, businesses and relationships in our average 70+ years on this planet. Why on earth should we invest a sizeable percentage of that precious fleeting existence on something that is doomed to failure from the outset? Why do we fight on investing more emotional and financial resource into something that will never ultimately work?

So, test for failure early, quickly, and cheaply. Sure, you can try and work out whether you need to riff through something because you’re in The Dip and that might work for you. What I’m saying though is that when you fail at something quickly and cheaply, you have opened up your life to a new possibility, a new project or idea that isn’t flawed and will be successful.

I’m not saying you should aim for failure. I’m saying that if something is fatally flawed, find out quickly, and rejoice. If it’s a success, well, brilliant: that was the point all along.

In project terms, I’ve failed plenty of times. My only regret is that I didn’t fail more cheaply. I invested so much emotionally, physically and financially that my resources were depleted for the next project, which in turn may have caused failure where there wasn’t a fatal flaw. From now on, failing early and failing often will be a sign for me that I’m getting better at understanding fatal assumptions. One day I will run into an idea that thanks to my cumulative experience of flaws will have none that effect its profitability, and it will hopefully succeed in ways I can not yet imagine.

When you realise that something is failing, resist the urge you’ve had all your life to try and “turn it around”. Find the PFA that is no longer just a potential, learn from it, and get on to the next project. Quickly.

Footnote: I remembered whilst writing this post, a friend who got divorced less than a year after getting married. He and his wife found that their attitudes to starting a family had significantly diverged, and so realised (and I quote), “it’s best to just call it a day early, otherwise we could lose years of our lives, wasting them on a relationship that was never going to last long-term” (emphasis mine). That took some guts, however their lives will be the better for admitting the fatal assumption that they both wanted the same things going forward, and they will reap the benefits of not sinking so much of their lives into something fatally flawed. If they can be so mature and pragmatic about something so emotionally raw, I’m sure you can be about your next idea for an iPhone app nobody cares about.

Whilst it was announced by the conference itself a week or two back and it appeared via @vagueware, I realised today I’d not mentioned it here: I’m speaking at ThinkVisibility on the 12th September.

It’s the first scheduled talk I’ve done in a long while – perhaps the first I’ve done since I started attending/organising BarCamp conferences – so it’s a bit weird actually planning it all this far in advance. I am however looking forward to it, as my talk for online marketers is focused on a subject area I’ve not been able to discuss publicly for the several years I’ve been learning it.

Whilst I’ve sworn to the organisers that I won’t discuss Kagtum per se, a lot of my talk will focus on techniques and technologies that underpin it. My abstract reads like this:

Why do marketers exert more effort trying to convince people who have never heard of their companies – whether it be through SEO, CPC campaigns and building social media audiences – to spend money, than they spend using technology to understand people they already have a relationship with?

Do you know how many of your current customers or site visitors are pre-disposed to buying a particular type of product online? Do you have statistics on cross-selling opportunities at checkout? Do you even know where to find that data? Or how to use technology to make sure your e-commerce platform does?

Applied software engineering, advanced data sets and a little bit of lateral thinking means your website can do your customer’s searching for them, before they know they need to search for something themselves. All you need is a little bit of knowledge, a friendly developer or geek and a leap of imagination.

In a talk tailored for people who are more curious about what’s possible rather than studying the status quo, we’ll briefly cover how centuries old mathematics can help your online presence as much as it has helped your inbox stay junk-free, touch on collaborative filtering and the possibilities unlocked in your software through just one or two key pieces of information about a customer. Finally we’ll touch on how software can constantly learn about your customers and automatically work to increase conversions.

An abstract – and hopefully intellectually stimulating – talk, you might leave wanting to learn some maths and computer science to understand how your website can serve your customers better, even if the thought of such a prospect appals you right now.

I’ll be tweaking it a little bit as I always do when developing a talk properly, but that’s the main gist of it.

A short/cut-down version will probably get done as a screen-cast for some time in later September or early Ocotber, but if you want to hear the full thing or discuss what I talk about in detail in person, I’d suggest you go and register now. The focus is around online marketing and SEO, and my talk will of course reflect the audience who will be there, so if that’s your bag there should be plenty of other talks to grab your attention and make the fee worthwhile. I hope to see you there.

People have been asking about Kagtum for some time now. About 3 years, actually. When will it launch? When can we see it? Much excitement abounds, and I’ve wanted to provide firm answers from the start, but for various reasons – mainly caused by a lack of resource to throw at the development which has meant it’s been a “spare time” project – it has been constantly delayed.

As it stands right now, Kagtum could go to private beta in about a fortnight. It would be rough around the edges, half the features I want it to have would be turned off, but it would offer the ability to read news stories and have the system work out (based on where, who and what you are), what stories should appear on your front page each day. It would offer an idea of what the technology is capable of, and I believe garner interest and audience from day one.

However, a clanger has emerged. It’s something I’ve been dismissive of, simply because I didn’t believe anybody would be so stupid to go through with it as a plan. I now think this idiocy is going to be copied by the rest of the industry, and that causes some real problems.

Yesterday, Rupert Murdoch concluded that his review into pay-wall publishing was over and all News International/News Corporation titles will go behind a pay-wall. By the end of the year, if you want to read a story from The Time, Sun or News of the World, or indeed any of the hundreds of other newspapers owned by the empire, you’ll have to pay. I’ve already explained the stupidity in another article, so let me just point you to the bit from the Guardian article on this that I knew might be coming, but hoped would be delayed:

He accepted that there could be a need for furious litigation to prevent stories and photographs being copied elsewhere: “We’ll be asserting our copyright at every point.”

That kills the current Kagtum model, dead.

Source: Reuters

I have no idea how litigious this could get, but at the very minimum Kagtum needs to read in a story, look at every word and word group (it recognises “swine flu pandemic” as one tag common to many stories without prompting, even when it appears in a paragraph), store it, and at the very minimum provides a link to the original article with the original headline.

Until it’s clear whether that’s acceptable by News Corporation and family, I can’t process any of their newspaper articles in a publicly available system: I simply don’t have the cash to go to court over this.

So, all News Corporation media (including websites affiliated with magazines, newspapers, TV channels and other media they own) is now to be excluded from Kagtum until the legal position is clarified. Anybody else who puts up a paywall will also get excluded.

That has a couple of knock-on effects:

1. It makes maintenance of the platform much more difficult as suddenly we’re dealing with websites that explicitly do not want to be linked to – something so alien to the spirit of the web, I’m still confused as to how anybody ever thought it was a good idea
2. The quality of the product for the user is decreased and therefore the take-up rate of Kagtum will suffer
3. People who pay for access to one newspaper will end up in the echo chamber Kagtum is specifically designed to avoid

So, what to do? Throw everything away and regret not proving the model and the technology? Hope NC change their minds, or watch them go bust? Hope people won’t mind lack of access to NC content? Or get the technology launched now and plead ignorance in court when I get sued to high-heaven?

None of the above are viable in the long-term. Being sued isn’t even viable in the short-term.

I’ve had time to think about this for a while, and stand by my arguments. I still genuinely think that pay-walls are bad for newspapers, bad for customers, and ultimately bad for democracy. I also think this is the nail in the coffin for the old media empires – News Corporation have just provided a way for the rest of the industry to kill them off permanently. However, market realities do not often take into account philosophy essays, and the situation is as it is.

If News Corporation are open to systems like Kagtum and competitors scraping data and linking to them, we can re-consider. It’s pretty easy for the site to ask you which titles you have a paid subscription for, so you don’t get links to content you can’t read and you do get links for content you like enough to hand over cash for. However without the ability to read the data in the first place, it can’t be grouped or targeted. Grouping and targeting content is all Kagtum really does, and unless it can do that freely and easily it can’t even reference that content.

There is the ability to do something else with the technology, news related but perhaps softer in its definition of news. There are toolkits kicking around on my hard drive for analysing e-mail, RSS and other sources of information in order to help prioritise content, and that might be one direction to go. Building an application for your phone or desktop that does what Kagtum did, but on your machine with your subscription parameters might even work. I’m also intrigued about the prospect of a different kind of UGC-based news portal that doesn’t have the “all stories are equal” issues of Wikinews or the strong bias of Indymedia.

In essence, I’m having a re-think. I genuinely thought NC would abandon their plans by now. As recently as last week I had a drink with one journalist friend and suggested Kagtum would launch in August, and I believed it – I just did not accept this plan was going to happen. My mistake. Sorry, Sarah, I called it wrong (again).

I expect my next announcement will be firmer, but right now I need to brainstorm my options and let them brew a while.

Kagtum is dead, long live Kagtum!

I wasn’t really expecting many of you to take me up on my Geek Social Responsibility thing. It was a shot in the dark, and I suppose what I was trying to answer was:

1. Can we exert more resource than we do already without hurting ourselves financially?
2. Can we make better use of that resource, so it has a greater impact?

The response was quite interesting, and several ideas proposed are now being planned, investigated and discussed. Something is happening. One point that emerged from the discussion on the GeekUp mailing list was that all of this could be taken advantage of. Steve Richardson said:

I am less inclined to devote substantial time and energy to helping charities develop web sites etc… as there is a disproportionate level of, hmmm I want to use the word abuse but fear it may be a bit strong, from certain charities who expect something for nothing and have very little appreciation of the complexity of the work involved.

Let me put it this way, do you think the directors (and other employees) of charities are working for free? I know they don’t, they are salaried, often pretty well, and I fail to see the difference between charity directors and us techies (we are all real life human beings trying to get by). There are far too many inequalities in remuneration for different jobs as it is – I recently heard that managers of Lidl earn £40K a year; without wanting to belittle Lidl managers, I am certain that the required skills for managing a supermarket are at least comparable with the skills required to develop web sites well and am absolutely certain that many people undertaking this role are paid a lot less.

Perhaps if the charity directors salary was split between the director and the ‘volunteering’ techie I may be more inclined to contribute… until then I think I need to concentrate on keeping my own head above water.

This is a common complaint: I am trying to make a living, these people are expecting the moon on a stick, and I’m falling short for obvious reasons.

Part of the issue is that Steve – like the majority of developers, designers and others in our sector – is a freelancer. The life of a freelancer is defined by the constant chasing of invoices, and trying to make sure as many hours of the day as possible are billable. Asking them to engage in “Socially Responsible Activities” is a bit like asking them “Do you want to risk not making the rent next month?”

Most people engaged in CSR activities are doing so as part of a marketing effort, a strand to their corporate behemoth “image” that makes people feel warm and fuzzy inside when they see the logo. There is an implicit silent agreement between such efforts and the charities who benefit: the company doesn’t really mean it, but needs to show they’re doing something. The charity is grateful, but know they have to fight to wrench what they can out of the company before they change their minds.

But in the geek community, things are different. When we do things, we mean it. We really, really care about doing “the right thing” and doing it well. So when the charity starts complaining that they want more and more doing, it causes resentment – this isn’t a corporate behemoth pouring some marketing budget into a “feel good” brand. It’s an individual putting heart and soul into something. The charity needs to understand that they need to tread gently and co-operate in a different way.

So, time-limited sessions like Speak to a Geek are great because the boundaries are implicit. Providing consortia to help work as a team means if somebody needs to dive out and do something else for a few days, the charity isn’t left hanging. Boundaries and back-ups are the way forward.

I’m meeting with people over the coming weeks to see how we can progress everything on the forum that seems to have grabbed people’s attention (register and vote if you haven’t already – adding ideas is even better). I will report back when I can.