Innovation in Software

The more observant amongst you will have noticed a couple of things about this blog recently:

1. There have been very few updates in recent months
2. When I have updated it, I haven’t been talking much about innovation in software, which given the title is what you’d expect

Most of the cool little things I see out there I am now throwing out via Twitter, and as time is – now more than ever – a premium commodity for all of us, I don’t think it’s fair to keep on dumping my brain into the ether.

For 2010, Vagueware will get a new website (sneak peek the beta if you want) and a new, more “corporate” feel blog. The RSS feed you might be reading this through will move across to that new blog. It might be a little drier, but it will focus on what we’re actually up to over at Vagueware HQ a little more closely.

In addition, in the first quarter of 2010, I’ll be starting up several brand new blogs each with a specific focus. I have a wide range of interests when it comes to software, and I think each area deserves attention all by itself. It helps you as a reader to segment them more, and it helps me stay focused when I write – something this blog has suffered from.

Specifically, you can look out for blogs focusing on digital entrepreneurship, software development process and best practice, and a couple of other ideas I’ve been mulling over for a while. Each of those will be announced at the new blog, along with the product announcements (no, really!), planned for early 2010.

The experiments conducted here will continue in other channels – such as getting the community to pay for half my time for the first part of 2010 – and I can’t see me giving up blogging completely at all any time soon.

However, in essence, this is the last post for Innovation in Software for now. A message will appear on all pages stating this site is now here purely as an archive, and all comments will be turned off in one week. No need to unsubscribe from RSS – you will be moved magically over to the new blog feed soon. Thank you for your continued readership, and I hope that at some point since this blog started on September 1st 2006 , I’ve given you something to think about and/or enjoy.

I have my “Chairman* of Fly The Coop” hat on here.

Fly The Coop, as some of you will be aware, is an Independent & Provident Society (read: non-profit co-operative), here to help all you freelancers and SMEs out there collaborate and co-work on a regular basis.

We have an opportunity to get involved in the new Hackspace in the Northern Quarter and take over the first floor to call our very own. However, we need to do a feasibility study and work out whether this is even viable before we commit ourselves to taking the space.

So, if you have 5 minutes please do go and fill in the survey.

We’re also interested in finding companies who want to “sponsor” the space, without perhaps making use of it in the same way. Perhaps you’re a service provider who doesn’t need a desk, or you only need one desk but want to provide more support than one desk gives us, whatever. This sponsorship would help us offset move-in and first year costs whilst we get settled in. If you’re interested in sponsorship, please do get in touch with either myself or email info@flythecoop.co.uk

* Until the next AGM, anyway

Whilst it was announced by the conference itself a week or two back and it appeared via @vagueware, I realised today I’d not mentioned it here: I’m speaking at ThinkVisibility on the 12th September.

It’s the first scheduled talk I’ve done in a long while – perhaps the first I’ve done since I started attending/organising BarCamp conferences – so it’s a bit weird actually planning it all this far in advance. I am however looking forward to it, as my talk for online marketers is focused on a subject area I’ve not been able to discuss publicly for the several years I’ve been learning it.

Whilst I’ve sworn to the organisers that I won’t discuss Kagtum per se, a lot of my talk will focus on techniques and technologies that underpin it. My abstract reads like this:

Why do marketers exert more effort trying to convince people who have never heard of their companies – whether it be through SEO, CPC campaigns and building social media audiences – to spend money, than they spend using technology to understand people they already have a relationship with?

Do you know how many of your current customers or site visitors are pre-disposed to buying a particular type of product online? Do you have statistics on cross-selling opportunities at checkout? Do you even know where to find that data? Or how to use technology to make sure your e-commerce platform does?

Applied software engineering, advanced data sets and a little bit of lateral thinking means your website can do your customer’s searching for them, before they know they need to search for something themselves. All you need is a little bit of knowledge, a friendly developer or geek and a leap of imagination.

In a talk tailored for people who are more curious about what’s possible rather than studying the status quo, we’ll briefly cover how centuries old mathematics can help your online presence as much as it has helped your inbox stay junk-free, touch on collaborative filtering and the possibilities unlocked in your software through just one or two key pieces of information about a customer. Finally we’ll touch on how software can constantly learn about your customers and automatically work to increase conversions.

An abstract – and hopefully intellectually stimulating – talk, you might leave wanting to learn some maths and computer science to understand how your website can serve your customers better, even if the thought of such a prospect appals you right now.

I’ll be tweaking it a little bit as I always do when developing a talk properly, but that’s the main gist of it.

A short/cut-down version will probably get done as a screen-cast for some time in later September or early Ocotber, but if you want to hear the full thing or discuss what I talk about in detail in person, I’d suggest you go and register now. The focus is around online marketing and SEO, and my talk will of course reflect the audience who will be there, so if that’s your bag there should be plenty of other talks to grab your attention and make the fee worthwhile. I hope to see you there.

People have been asking about Kagtum for some time now. About 3 years, actually. When will it launch? When can we see it? Much excitement abounds, and I’ve wanted to provide firm answers from the start, but for various reasons – mainly caused by a lack of resource to throw at the development which has meant it’s been a “spare time” project – it has been constantly delayed.

As it stands right now, Kagtum could go to private beta in about a fortnight. It would be rough around the edges, half the features I want it to have would be turned off, but it would offer the ability to read news stories and have the system work out (based on where, who and what you are), what stories should appear on your front page each day. It would offer an idea of what the technology is capable of, and I believe garner interest and audience from day one.

However, a clanger has emerged. It’s something I’ve been dismissive of, simply because I didn’t believe anybody would be so stupid to go through with it as a plan. I now think this idiocy is going to be copied by the rest of the industry, and that causes some real problems.

Yesterday, Rupert Murdoch concluded that his review into pay-wall publishing was over and all News International/News Corporation titles will go behind a pay-wall. By the end of the year, if you want to read a story from The Time, Sun or News of the World, or indeed any of the hundreds of other newspapers owned by the empire, you’ll have to pay. I’ve already explained the stupidity in another article, so let me just point you to the bit from the Guardian article on this that I knew might be coming, but hoped would be delayed:

He accepted that there could be a need for furious litigation to prevent stories and photographs being copied elsewhere: “We’ll be asserting our copyright at every point.”

That kills the current Kagtum model, dead.

Source: Reuters

I have no idea how litigious this could get, but at the very minimum Kagtum needs to read in a story, look at every word and word group (it recognises “swine flu pandemic” as one tag common to many stories without prompting, even when it appears in a paragraph), store it, and at the very minimum provides a link to the original article with the original headline.

Until it’s clear whether that’s acceptable by News Corporation and family, I can’t process any of their newspaper articles in a publicly available system: I simply don’t have the cash to go to court over this.

So, all News Corporation media (including websites affiliated with magazines, newspapers, TV channels and other media they own) is now to be excluded from Kagtum until the legal position is clarified. Anybody else who puts up a paywall will also get excluded.

That has a couple of knock-on effects:

1. It makes maintenance of the platform much more difficult as suddenly we’re dealing with websites that explicitly do not want to be linked to – something so alien to the spirit of the web, I’m still confused as to how anybody ever thought it was a good idea
2. The quality of the product for the user is decreased and therefore the take-up rate of Kagtum will suffer
3. People who pay for access to one newspaper will end up in the echo chamber Kagtum is specifically designed to avoid

So, what to do? Throw everything away and regret not proving the model and the technology? Hope NC change their minds, or watch them go bust? Hope people won’t mind lack of access to NC content? Or get the technology launched now and plead ignorance in court when I get sued to high-heaven?

None of the above are viable in the long-term. Being sued isn’t even viable in the short-term.

I’ve had time to think about this for a while, and stand by my arguments. I still genuinely think that pay-walls are bad for newspapers, bad for customers, and ultimately bad for democracy. I also think this is the nail in the coffin for the old media empires – News Corporation have just provided a way for the rest of the industry to kill them off permanently. However, market realities do not often take into account philosophy essays, and the situation is as it is.

If News Corporation are open to systems like Kagtum and competitors scraping data and linking to them, we can re-consider. It’s pretty easy for the site to ask you which titles you have a paid subscription for, so you don’t get links to content you can’t read and you do get links for content you like enough to hand over cash for. However without the ability to read the data in the first place, it can’t be grouped or targeted. Grouping and targeting content is all Kagtum really does, and unless it can do that freely and easily it can’t even reference that content.

There is the ability to do something else with the technology, news related but perhaps softer in its definition of news. There are toolkits kicking around on my hard drive for analysing e-mail, RSS and other sources of information in order to help prioritise content, and that might be one direction to go. Building an application for your phone or desktop that does what Kagtum did, but on your machine with your subscription parameters might even work. I’m also intrigued about the prospect of a different kind of UGC-based news portal that doesn’t have the “all stories are equal” issues of Wikinews or the strong bias of Indymedia.

In essence, I’m having a re-think. I genuinely thought NC would abandon their plans by now. As recently as last week I had a drink with one journalist friend and suggested Kagtum would launch in August, and I believed it – I just did not accept this plan was going to happen. My mistake. Sorry, Sarah, I called it wrong (again).

I expect my next announcement will be firmer, but right now I need to brainstorm my options and let them brew a while.

Kagtum is dead, long live Kagtum!

I wasn’t really expecting many of you to take me up on my Geek Social Responsibility thing. It was a shot in the dark, and I suppose what I was trying to answer was:

1. Can we exert more resource than we do already without hurting ourselves financially?
2. Can we make better use of that resource, so it has a greater impact?

The response was quite interesting, and several ideas proposed are now being planned, investigated and discussed. Something is happening. One point that emerged from the discussion on the GeekUp mailing list was that all of this could be taken advantage of. Steve Richardson said:

I am less inclined to devote substantial time and energy to helping charities develop web sites etc… as there is a disproportionate level of, hmmm I want to use the word abuse but fear it may be a bit strong, from certain charities who expect something for nothing and have very little appreciation of the complexity of the work involved.

Let me put it this way, do you think the directors (and other employees) of charities are working for free? I know they don’t, they are salaried, often pretty well, and I fail to see the difference between charity directors and us techies (we are all real life human beings trying to get by). There are far too many inequalities in remuneration for different jobs as it is – I recently heard that managers of Lidl earn £40K a year; without wanting to belittle Lidl managers, I am certain that the required skills for managing a supermarket are at least comparable with the skills required to develop web sites well and am absolutely certain that many people undertaking this role are paid a lot less.

Perhaps if the charity directors salary was split between the director and the ‘volunteering’ techie I may be more inclined to contribute… until then I think I need to concentrate on keeping my own head above water.

This is a common complaint: I am trying to make a living, these people are expecting the moon on a stick, and I’m falling short for obvious reasons.

Part of the issue is that Steve – like the majority of developers, designers and others in our sector – is a freelancer. The life of a freelancer is defined by the constant chasing of invoices, and trying to make sure as many hours of the day as possible are billable. Asking them to engage in “Socially Responsible Activities” is a bit like asking them “Do you want to risk not making the rent next month?”

Most people engaged in CSR activities are doing so as part of a marketing effort, a strand to their corporate behemoth “image” that makes people feel warm and fuzzy inside when they see the logo. There is an implicit silent agreement between such efforts and the charities who benefit: the company doesn’t really mean it, but needs to show they’re doing something. The charity is grateful, but know they have to fight to wrench what they can out of the company before they change their minds.

But in the geek community, things are different. When we do things, we mean it. We really, really care about doing “the right thing” and doing it well. So when the charity starts complaining that they want more and more doing, it causes resentment – this isn’t a corporate behemoth pouring some marketing budget into a “feel good” brand. It’s an individual putting heart and soul into something. The charity needs to understand that they need to tread gently and co-operate in a different way.

So, time-limited sessions like Speak to a Geek are great because the boundaries are implicit. Providing consortia to help work as a team means if somebody needs to dive out and do something else for a few days, the charity isn’t left hanging. Boundaries and back-ups are the way forward.

I’m meeting with people over the coming weeks to see how we can progress everything on the forum that seems to have grabbed people’s attention (register and vote if you haven’t already – adding ideas is even better). I will report back when I can.

For the last three years, this blog was using the Mephisto blogging platform – a Ruby on Rails application that I intended to expand myself in all sorts of ways.

To be frank, the code base for Mephisto suffers a little from bit-rot, and it seems to me that the original authors have almost given up on supporting the system. Hey, it’s still not made a 1.0 release, so I’m sure it’s just a lack of time, we’re all busy, etc. but I need something with a more active developer community (and I know I’m part of the problem by not diving in myself).

Handily, in the time that has elapsed, Wordpress has gone from being a rather annoying and slightly odd PHP blogging platform into a really slick CMS with tons of widgets, themes and plugins. So I’ve switched.

And at the same time, I’ve also decided that comments need to be opened up on articles. Right now, all articles published from 1st June 2009 have comments open so if you saw something you liked (or hated), feel free to make a comment now.

P.S. thanks to Jason Morris for the script modification that helped me move 3 years of blog posts in about 15 minutes, itself based on a script by David Murphy

Vagueware sometimes does all the delivery of a project when doing service provision, sometimes it partners up, and sometimes we get asked just to keep an eye on things from afar.

More recently however we’ve been asked several times to sit down, talk through an idea, codify it into a loose specification and manage putting it out to tender for any firms interested in working on the project. This allows us to do what we do well, and others to get access to interesting and profitable projects.

We have recently been engaged by a client to produce a video portal site, the main selling point of which is the business execution behind it. We can’t go into explicit details on that front, but if you’re interested in looking at a sub-£10,000 build for a relatively simple video-to-mobile solution, we’d ask you to consider taking a look at the invitation to tender below.

Click here for invitation to tender

Please send all proposals back to me at paul AT vagueware DOT com by Tuesday the 30th June. Any further queries or questions should be directed via that address too.

Last night Sarah Hartley announced the closure of the Mancunian Way blog that I’ve been contributing to for a couple of years now.

Sarah explains on her own blog that she is on to new adventures.

I have slightly mixed emotions about this myself. I won’t miss the mild panic when Sarah’s email goes around asking for a blog of the week and I have to think smartly as I realise I haven’t contributed in a while, but that’s about the only thing I’ll miss.

The highlight for me will almost certainly be the the write-up I did for the Tony Wilson Experience last Summer, which the council liked so much they shoved it in their own press section and asked for permission afterwards. Always nice to have your writing enjoyed by an audience, and the feedback I got from the MEN blogs in particular was fantastic.

I also managed to cover b.tween, the anniversary of the Baby, several Northern Startup events, and managed to riff about things I thought of as interesting and useful.

So, I will miss it, quite a bit. A significant part of my online identity was “blogs for the MEN”, however it gives a little more space and time for other things in my writing life, and I’m off to do some new writing adventures of my own – all will be announced soon.

I will also watch with interest what the MEN decide to do in future post-Sarah. I don’t know of another regional newspaper in the UK that has as well developed a blog section (and for that Sarah should definitely be commended), so it would be a huge shame for it to lose momentum now. I have no idea what comes next over there, my only contact at MEN was Sarah.

Good luck to Sarah, those of you finding me via that blog please stay tuned for my next projects, and let me just say: The blog is dead. Long live the blog.

Many moons ago I worked in the world of ISP systems administration. That broadly meant I had to be intimately familiar with the inner workings of mail, DNS and Radius servers (the latter still gives me cold shivers when I think back), as well as connectivity issues. Back then, broadband but was a distant dream but thanks to my knowledge of AAA protocols – Authentication, Authorisation & Accounting – I gradually had to become familiar with modem racks and being able to diagnose a problem purely by listening to the screeching noise of a 56K modem negotiating with its telephone-connected partner.

Since then, I have gradually moved away from all that jazz and more into development. Occasionally I pop my head back into that market and see where things are, and am constantly surprised how stagnant the market is. It’s still at a place where it expects people to know much more than they probably want to, just to get started.

And I still remain surprised at how bad support offerings are around connectivity, especially for SoHo customers. Business users pay a premium for their broadband, but still have to spend either a lot of time learning how to configure routers, or pay over-inflated support contracts for others to do it for them.

Add into the mix the growing area of cloud services like EC2 and its competitors and things get more complex. Setting up and managing cloud services so that they automatically scale and adapt to your application’s needs (burstable needs, even), is complex and expensive. There is the inkling of an idea within me to build toolkits that would dramatically reduce the pain of moving to cloud services and managing/supporting burst-tolerant applications.

The issue for me is, do I work on any of these ideas? Some of them are labour intensive, but clearly ideas I can easily recruit people into once a critical mass of customers are on board. There are also other ideas kicking around the Vagueware workshop and I’m sure many of you who know me can think of areas I should be working in that I’m not.

So, I’ve decided to kick the tires of UserVoice, a competitor (considerably more polished) than the toolkit I had here at vagueware.com for a number of years.

In the Services & Products Forum I’ve seeded half a dozen things I could be working on right now, but for various reasons are mostly on hold. I want you, dear reader, to vote on the ones you like. Even better, I want you to suggest ideas that if popular will benefit from dedicated resource – time, cash and sleepless nights – to bring them to fruition. It’s a little like how I started all those years ago, but far more organised.

Once some of the products and services I’m working on right now get launched, they will benefit from their own forums as well. Watch this space, etc.

I have had an awful lot on my plate recently. I think most of us in this sector have.

Right now, I have a few choices to make about what Vagueware does and is involved in over the next 9 months, and I feel the time is ripe for some minor re-invention.

Over the last 15 years I have worked on everything from server administration to strategy, development to training, project management to debugging. One of the reasons Vagueware has struggled to grow at the rate it should have is because I’ve not really chosen one small area and really focused on it. I’ve had my fingers in many, many pies.

It is through being over-committed to so many projects and concerns that has led to me being exhausted and the quality of my work being deeply unsatisfying to myself.

That needs to change, and indeed it already is. I am now moving away from client work and towards internal product development (including a joint venture I’m itching to tell you about, but it’s too early yet), but I’m worried I’ve missed a trick and there is something out there that it’s “obvious” I should be working on but I’m not aware of. You have no idea how often people will start a conversation with me with the phrase “I’m amazed you’re not working on…” and on consideration, I’m usually surprised too.

So, this is last call for input. If there is something you think I should be working on, whether it be a product area you’re surprised I haven’t got stuck into, your own project you think I can help on, or just an interesting idea, speak now (via email or twitter please) or forever hold your peace. If you want to hire me for a project you have until Monday to start the discussion, but after that all client work is being put on hold.

It’s about to go dark on the client front and Kagtum development (amongst others) is going to gear up to a whole new level. See you on the other side.