Innovation in Software

Adrian Holovaty announced a while back templatemaker which sounds more complicated than it is.

Take a pile of web pages all built using the same template – say, restaurant listings, or a timetable – and throw it at the code. It produces a copy of the ‘template’ with the ‘data holes’ marked up. You then look at a new page, and it gives you the raw data. No more screen scraping, it just learns what it needs to find the data that changes each time. Nice.

A really useful aspect is that whilst he’s implemented it with Python bindings, the core is written in C and uses the Python C bindings. That means with a bit of hackery, porting it to Ruby should be quite simple. A project for the weekend, perhaps?

I’m shouting this up for two reasons: it’s a piece of code I’m sure all of us have thought about implementing, but never got around to producing because it just ‘felt’ too difficult, so it’s definitely very innovative from that perspective; secondly, it’s using the C bindings in an intelligent way to get performance where it counts – something I’m going to need to do more of over the next few months in other languages.

I suspect the future isn’t going to be about using the right language for a project, but the right language for the right part of the job. You could implement something in Erlang, which can interface with Ruby, which in turn can interface with C. There are points of complexity there that a competent software engineer would shudder over, I’m starting to feel that being competent in one language or framework is definitely not going to be enough for the next generation of software: we need the flexibility and power of scripting languages, and the raw power of compiled C/C++.

The people who have to listen to my diatribes about what is wrong with programming in person know there is one thing that bothers me more than anything: we’re not getting kids interested in programming, and we’re going to pay for it in the future.

When I think back to how I got into writing software, I realise I just don’t see it the way teenagers do. They see it as a career choice and will actively avoid writing a single line of software until they get to University. For me, code was something I delved into from the age of 11 and never really returned. I was writing C when I was 14, and I think it actually changed the way I thought about software permanently – it meant that something changed inside my brain as it was still growing. I don’t think that’s a bad thing.

That said, I understand the barrier of entry is now way too high. Most kids do not want to get into an IDE. When they think “it might be nice to learn to program” and go to the bookstore, they are confronted with a wall of very expensive, very dull looking books. The first book I ever read on programming was on BBC Basic and was illustrated with pictures of robots in factories pretending to be FOR loops. That isn’t really comparable to what kids have access to today.

It’s wonderful then that _why has put some effort into fixing this little conundrum and released the first build of HacketyHack which calls itself “the Coder’s Starter Kit”. He’s only released for Windows so far, so I’ve not been able to try it out (still less get involved in helping him close tickets – I’d throw time at this without thinking about it for a second), but his Manifesto is something I wish I had written myself. He’s on the same page I’ve been whining on about for the last year.

I hope it the best of success, and the moment _why releases it for other platforms that I actually have access to, I’ll make sure I get stuck in on helping submit patches and spreading the word further.

Right now, almost 100% of Vagueware’s revenue is through services. The objective for 2007 was to switch this so that nearly 100% of the revenue would be through products. Offering excellent little applications for SMEs and corporates, who would pay a license fee to oil their squeaking machines of information flow. Oh, the ideas. Many of them unique, some of them just better executions of what has been done before. You know the dream, you’ve had it yourself: build a customer base, sell the company and retire inside five years.

But there is a niggle. I have come to realise that as a one-man company, in need to grow the business I need to do something rather extraordinary. I’ve realised it’s not enough to produce really good products, and to execute the marketing well if I want to develop a solid business: I might score lucky and get a company I can sell pretty quickly that way, but if I want something sustainable and to grow now without taking extra finance on board, I need to do something different.

I want to build something people trust isn’t going to go under tomorrow, that can develop without me being at the helm all day long, that can go off on weird tangents without me spending all my time trying to think up new ideas, and that can get the benefits of a diverse and enthusiastic development team behind it without me spending huge amounts of money on coders.

In other words, I need to open source my code.

By making my code base open source I get around all those issues of “what happens if you get run over by a bus” and I help foster a far larger development community than I could ever hire. By giving the code away for free I help develop competition which – call me a free marketeer if you wish – I believe helps my company as much as it helps those who didn’t come up with the idea in the first place.

The disadvantage of course comes in the monetisation stage: by not locking the code base down, it’s not enough to just offer a product licensing fee. I need to think in terms of services people will pay for. Hosting, appliance boxes, customisation, support and maintenance deals, training, etc. all have to be considered. No longer is it just Software as a Service, but the software is free with an optional service. That requires a big re-think.

It also means a fundamental re-think of how I approach development of new ideas. I’ve realised that it’s not enough for me to just come up with an idea, throw out v0.1 and hope it flies. There’s a lot of scribbling in notepads as to how to deliver something a little “different” in that regard. With regards to those services, the I’ve filled several dozen pages with ideas, and I hope to experiment with all of them at some stage.

However, I have a confession to make: I’m scared.

All of this requires me to be ridiculously brave, and say to hell with the money: the money will take care of itself. That would be easy to do if I were rich and didn’t need to think about rent, bills and food, but I’m not. I’m pretty much broke and if it weren’t for that bespoke development work, I’d be pretty hungry too. But I’ll find a way, somehow. If it means taking on bespoke work through the rest of this year whilst I wait for revenues to pick up on other services, so be it.

This is going to take some courage I think, but I believe it’s the right thing to do in the long run. Along the way, I expect this site (and blog, and therefore feed) to get trashed in the coming weeks whilst I build something more aligned to where this is heading. Those of you who remember a previous incarnation of Vagueware will find some recurring themes in what gets released in a few weeks: that’s a good thing.

It’s going to be an interesting ride. Anybody want a ticket?

For a long time, I’ve had a problem explaining exactly why software engineering doesn’t work like other kinds of engineering. One of the issues I’ve had is trying to find the wording needed to explain that programmers don’t really build anything, and that to manage building software like building a bridge makes no sense. I couldn’t quite put my finger on the right words though.

Until now.

Jack W. Reeve’s essay on ‘What is Software Design?’ is the best work I’ve seen to date on the subject. He argues – convincingly – that programmers are designers, architects, draftsmen at a drawing board. We don’t build anything: compilers, linkers, interpreters do that for us. He points out how flawed an idea it is to produce a design document that can never truly represent the details of how the software will work, because it is an abstraction of an abstraction. The only accurate design is the software source code itself.

What’s more, it makes sense that in the same way that an architect can build a model, which can be turned into a few more detailed drawings, which can then be turned into engineering drawings, it makes sense to start all software development with a set of mock-ups which are then turned into a UI (say, static HTML) which then become plugged together with code. It’s like building a skyscraper by defining what the bits ‘users’ will see will look like, then working out how the nuts and bolts will hold it together: exactly what an architect does.

His essay dates from a time when the best language available to programmers to express their design was C++, and he argues that C++ helps his model of ‘design as code’ along nicely. It was the best language for the job because as an object-orientated language, it allowed an expression that most closely resembled how we abstractly thought about objects and data in the real World. Since then, of course, things have moved along a little bit.

I would argue today that Ruby is the best design language we have available to us. I claim this, because I actually find it easier and quicker to express an abstract design – which in turn can then execute – through Ruby than I do through a diagram or notation or pseudo-code.

At this point you might be thinking “yeah, right, whatever!”, but this isn’t some idle boast, I eat my own dog food.

The other day I needed to clarify a design point with another programmer via e-mail. I needed to express a HABTM relationship being used as a way of listing ‘exceptions’ to a ‘everybody/nobody’ flag stored in a model. The details aren’t important, but I considered for a moment the very best way of communicating this. I considered drawing a UML diagram. I thought about a quick sketch with some boxes, I considered a little pseudo-code. I then remembered that the programmer I was discussing this with could code Ruby, and maybe if I just developed a few stubs with some comments he’d get the idea.

Five minutes later, I had not just produced stub code. I had some migrations that would build the tables needed in Rails, and I had a few methods I was proposing adding to a model. It was untested, and some DateTime handling code would need a little fleshing out, but I’d discovered it was just quicker to write the code than it would be to write the comments explaining what the code would do. I actually ended up writing a software design in my mail client, that I could copy and paste over into a code editor and with a little bit of expansion and testing, get running.

In other words, the perfect design notation seems to have been discovered – it’s not UML that generates stub C++ in a CASE tool, or some formal design notation. It’s Ruby that just, well, runs. It is perfect because it is the perfect balance of programming language and design notation.

So not only was Reeves right in saying that source code is the best detailed design possible, that we are designers at heart, that programming languages that work at an abstract level are a step in the right direction, that formal design tools are useless and pointless, but he was even right that eventually we’d produce a programming language that was just a compilable design notation.

We now have that tool and its name is Ruby.

UPDATE: Two separate comments – each without seeing the other, as I didn’t approve them until this morning – suggest that LISP is a better design tool than Ruby.

It is worth pointing out that Ruby is kind of a grandchild of Lisp because it inherits so much from Scheme, itself a dialect of Lisp.

So is Ruby just another Lisp dialect? No. Whilst borrowing aspects of Lisp, it is more closely related to Smalltalk in terms of its object-orientation. It also borrows from Python, Dylan and CLU. It uses the best of all of them – Lisp included – and makes something better.

Also, there is another important part of Ruby that Lisp (and Smalltalk to be fair) do not yet have: an easy way to rapidly produce production-level, deployable, useful software. I can build a web application in Ruby – thanks to Rails, or Camping – in relatively little time compared to 10 years ago in PHP or Perl. I suspect, although haven’t researched it fully, that to do the same in Lisp might take me a little longer.

Ruby has many of the great features of Lisp baked in, but many of its own bolted in to boot. What’s more, the design ethos of Ruby is its strongest point: Ruby is designed to make programming pleasurable. When you enjoy writing code, you write more of it in less time, and with fewer bugs. Lisp might be fun for those who enjoy it, but I agree to some extent with Larry Wall on this one: it “has the visual appeal of oatmeal with fingernail clippings mixed in”.

If you like Lisp, I suspect you will love Ruby, and you’ll find yourself more productive in it. Either way, I’ll concede that for some people in love with it, Lisp is as useful to them as a design language as Ruby is to me. We’re both lucky, we both know we are, and we’re both a step ahead of the C# and Java guys.

I still say Ruby rocks more though. :-P

UPDATE 2: Changed link to the PDF to point to one held at the official home and authorised version of these PDFs. Thanks to Daniel Read for pointing out my error in directing you all to a pirate copy.

I’ve just spent an hour re-factoring some of my code. I also put some comments in, because there weren’t any when I started. I also wrote some unit tests I should have done before writing the code, and actually ended up fixing a bug that would happen in rare cases. I also made it smaller, faster and less of a CPU hog. I increased the readability and fixed a couple of indenting issues.

What surprises me is that I didn’t do any of this when I wrote the code in the first place.

I think part of the problem is that when you have a deadline, you just care about getting a solution together that works, and you want it ASAP. You don’t care about elegance when the bosses are crying for a customer sign-off at the end of the week. As a result, you take shortcuts, jump around and deploy code that “will kinda work” and just hope it never fails. Or at least, hope it never fails whilst you still work there.

Strangely, some of the best code I’ve ever seen out there is open source, which feels counter-intuitive – it’s a bunch of people doing hobby code, not something they are being paid to produce. I think it’s because there is no deadline with open source that makes it better. It has been said that programmers work on open source projects in the evening because it’s relaxing. Without the deadlines, pressures and insistence things are done a certain way, developers can revel in the intellectual nature of development. As a result, they take the time to unit test, to comment, to document to re-factor properly.

One of the interesting conversations developers could have with customers and their management is to ask “Which is more important? Code that kind of works right now, or code that is built to last?” and I think the conclusions would shock us. Most customers don’t care that you got a big block of code reduced down to a really nice recursive loop. Most bosses aren’t bothered about comments in code, because they don’t know the real price of not having them.

This exercise tonight has convinced me I need to change my working practices, and I need to stop trying to undercut on prices – I need to price as a job, raise the time estimates to remove the deadlines and start promising to produce code that is built to last. What I’m worried about is that this isn’t a very good sales strapline: we’re more expensive and take longer because we do stuff you don’t care about.

Even so, I think it’s worth doing. Quality code makes me happier than quickly developed code.

Many years ago, I didn’t program just for money. I actually did it because I loved the problem solving aspect. I kind of miss that – I spend way too much time formatting reports and generating graphs these days, not enough time just messing around for the hell of it.

The other day, I discovered CodeGolf.com as a new hobby/competitive sport. The idea is simplicity itself: take one of the challenges, produce working code in your interepreted language of choice (Perl, Python, PHP or my favourte, Ruby) that pumps out the right output and then get it working in the smallest number of bytes possible. It’s an exercise in obfuscation and confusion, sure, but it’s great fun for somebody like me.

The challenge that drew me in was Oblongular Number Spirals which looks deceptively simple. Let’s just say that after about an hour I realised it wasn’t – it’s actually very, very hard. I ended up with a solution that uses a really bad edge detection algorithm as I walk across a virtual grid (actually a multi-dimensional array) making sure I don’t bump into other rows and the edges.

A working solution must have taken me a total of six hours coding time – which shows how out of practice I am with this pure algorithm work. Reducing it down to the smallest number of bytes was the easiest bit, but falling asleep last night I realised that there were ways I could easily reduce it further from my poor show of 426 bytes. Anyway, it was great fun, and for the masochists amongst you, here is my slightly longer (line breaks are in this one) script:

s,t=gets.split.map!{|s|s.to_i};b=[];
k,d,x,y,e,f,g,h=s*t,0,0,0,-1,-1,s,t;o=k.to_s;
(1..k).each{|i|b[y]=[]if b[y].nil?;b[y][x]=i;
case d;
when 0;if x+1==g;d=1;f=y;y+=1;else;x+=1;end;
when 1;if y+1==h;d=2;g=x;x-=1;else;y+=1;end;
when 2;if x-1==e;d=3;h=y;y-=1;else;x-=1;end;
when 3;if y-1==f;d=0;e=x;x+=1;else;
y-=1;end;end};
(0...t).each{|r|m="";
(0...s).each{|q|z=b[r][q].to_s;m<<" "*(o.length-z.length);
m<<" "unless q==0;
m<< z;};puts m;}

Isn’t that awful? Taking a language as beautiful as Ruby and making it look like that! I should be ashamed of myself. For some reason, in some browsers, the last line of the above block gets chopped but should read ‘m << z;};puts m;}'

Here's a slightly less painful version:

s, t = gets.split.map!{ |s| s.to_i }
b=[]
k, d, x, y, e, f, g, h = s*t, 0, 0, 0, -1, -1, s, t
o = k.to_s

(1..k).each{ |i|
  b[y] = []
  if b[y].nil?
    b[y][x] = i
    case d
    when 0
      if x + 1 == g
        d = 1; f = y; y += 1
      else
        x += 1
      end
    when 1
      if y + 1 == h
        d = 2; g = x; x -= 1
      else
        y += 1
      end
    when 2
      if x - 1 == e
        d = 3; h = y; y -= 1
      else
        x -= 1
      end
    when 3
      if y - 1 == f
        d = 0; e = x; x += 1
      else
        y -= 1
      end
    end
}

(0...t).each{ |r|
  m = ""
  (0...s).each{ |q|
    z = b[r][q].to_s
    m << " " * (o.length - z.length)
    m << " " unless q == 0
    m << z
  }
  puts m
}

That produces output that passes the codegolf tests, so if somebody wants to take it and make the obvious space-saving optimisations (yes, I spelt that word incorrectly the last time I used it), that will get it down to less than 200 bytes, feel free.

And to any potential clients reading this right now, let me promise you, hand on heart, I would never dream of putting code that awful into your production environments. The code you buy would be elegant, commented, and use meaningful variable names. Honest. ;-)