Who would have thought that one of the most innovative players in the hosting and web application industry would be a bookshop?

One of the big problems with Amazon’s web services is that they aren’t that great for permanently hosted web applications. There’s the dynamic IP addressing issue (which weoceo will look after if you have the cash) and the serious problem of how to store your database.

S3 is very nice, but it stores flat data, and certainly not anything as fancy as SQL tables. Until recently there was a hacky way to do it with a special storage engine for MySQL, but just looking at it made me nervous about my data.

Well, Amazon have decided to fix this issue. I received this email from them this morning.

“Dear AWS Developers,

This is a short note to let a subset of our most active developers know about an upcoming limited beta of our newest web service: Amazon SimpleDB, which is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database - real-time lookup and simple querying of structured data - without the operational complexity.

Were excited about this upcoming service and wanted to let you know about it as soon as possible. We anticipate beginning the limited beta in the next few weeks. In the meantime, you can read more about the service, and sign up to be notified when the limited beta program opens and a spot becomes available for you. To do so, simply click the “Sign Up For This Web Service” button on the web site below and we will record your contact information.

Learn more and sign up

Sincerely,

The Amazon Web Services Team”

So, there we have it. No more managing DB clusters. Scalable database tables, which once the beta is over will likely come with an SLA. Assuming that this just sits on top of S3, we might even be able to host our data inside the EU and get ll warm and fuzzy about protecting customer data properly.

I’m not sure this will be based on a standard set of DB libs but I expect we’ll see 1-line hacks to make it work with Rails, PHP and a host of other app frameworks within a few weeks.

I’m in.

I never “got” code reviews. I never understood how sitting around talking about somebody else’s code would help them or you, beyond being able to weed out the obviously awful coders.

This week I’ve been doing some consultancy, in-sourced to help a firm evaluate what their developers have built for them. As part of that process I’ve had to review documentation, specifications and code. The process has been at times confusing and enlightening.

It’s one thing to be able to say “this is how it should be done” when you’re not a developer, but it’s another thing when you are a developer who hasn’t done some of those things yourself. This process has taught me as much about myself and my own code as it has about these guys’ code.

Interestingly, I now see things from a buyer’s perspective much more clearly, and I can see how frustrating this must be from the other side. I can see how things which we don’t consider important are to a buyer critical. I can see how clever solutions can sometimes be too clever. I can see how something simple you forget to do can make all the difference.

Adrian Holovaty announced a while back templatemaker which sounds more complicated than it is.

Take a pile of web pages all built using the same template - say, restaurant listings, or a timetable - and throw it at the code. It produces a copy of the ‘template’ with the ‘data holes’ marked up. You then look at a new page, and it gives you the raw data. No more screen scraping, it just learns what it needs to find the data that changes each time. Nice.

A really useful aspect is that whilst he’s implemented it with Python bindings, the core is written in C and uses the Python C bindings. That means with a bit of hackery, porting it to Ruby should be quite simple. A project for the weekend, perhaps?

I’m shouting this up for two reasons: it’s a piece of code I’m sure all of us have thought about implementing, but never got around to producing because it just ‘felt’ too difficult, so it’s definitely very innovative from that perspective; secondly, it’s using the C bindings in an intelligent way to get performance where it counts - something I’m going to need to do more of over the next few months in other languages.

I suspect the future isn’t going to be about using the right language for a project, but the right language for the right part of the job. You could implement something in Erlang, which can interface with Ruby, which in turn can interface with C. There are points of complexity there that a competent software engineer would shudder over, I’m starting to feel that being competent in one language or framework is definitely not going to be enough for the next generation of software: we need the flexibility and power of scripting languages, and the raw power of compiled C/C++.

It’s only Tuesday afternoon, and already this week I’ve learned a great deal about producing better software, albeit inadvertently - nothing teaches you more about code than having to fix stupid mistake. Each week I always learn some new insight but this week it’s been like a waterfall of learning the hard way. I thought I’d moan at you lot, because it’s more fun than whining about it all inside my own head. :-)

• Users are lazier than you think - to me, it’s pretty obvious that if I’ve already entered data into a system, I’m not going to enter it all over again, but this time full of typos and changing reference numbers around. I’m just going to enter the data in correctly first time, and if I have a problem, I’ll go back and edit it. Not so users of one system I’ve been working on this week. Cue an afternoon of going through data and checking it for consistency errors
• Users don’t read the screen - we noticed a load of weird consistencies elsewhere in the same system. It turns out that I’d upgraded ferret on the system, and then the ferret index on the app was obviously foo-barred (note: when doing a gem update on ferret you MUST either clear out the index dir in your Rails app or you must script/console and Model.rebuildindex on each Model you’ve got actsas_ferret rolling with - you don’t have a choice in the matter). Strangely, in the period the app was doing this, people must have been putting data in and getting the “Application error” screen. Did they stop? Not likely - they just carried on, hoping it wouldn’t matter. Cue another half an afternoon of checking the database for errors.
• Logs are more useful than you think - Rails logs all parameters it gets, even in production mode. Some people think this is a security risk, and for password forms you should definitely turn it off (look up filterparameterlogging to see how). However, when you have a load of weird consistency errors in your DB, my god do those logs help reconstruct what was going on and help you fix what happened
• script/console doesn’t do what it says it just did - try this: take a model with a datetime column. In script/console, try and assign a new date to a record and save it. Note how that didn’t work. I’m not quite at the bottom of why, but it’s curious that it doesn’t “just work” - it may be a config issue, but I can’t see what right now.
• Nobody cares about my problems, they’ve got their own - I don’t even know I bother bitching about this stuff: I just find myself shouting into a big empty echo chamber that is the Web. :-)

Anyway, I’m at a careers fair tomorrow helping out MDDA up in Cheetham Hill. I hope to return with all my limbs intact.

After that, I’ve got tickets for all five days of the cricket at Old Trafford, so I’ll be pretty much in holiday mode after tomorrow.

The people who have to listen to my diatribes about what is wrong with programming in person know there is one thing that bothers me more than anything: we’re not getting kids interested in programming, and we’re going to pay for it in the future.

When I think back to how I got into writing software, I realise I just don’t see it the way teenagers do. They see it as a career choice and will actively avoid writing a single line of software until they get to University. For me, code was something I delved into from the age of 11 and never really returned. I was writing C when I was 14, and I think it actually changed the way I thought about software permanently - it meant that something changed inside my brain as it was still growing. I don’t think that’s a bad thing.

That said, I understand the barrier of entry is now way too high. Most kids do not want to get into an IDE. When they think “it might be nice to learn to program” and go to the bookstore, they are confronted with a wall of very expensive, very dull looking books. The first book I ever read on programming was on BBC Basic and was illustrated with pictures of robots in factories pretending to be FOR loops. That isn’t really comparable to what kids have access to today.

It’s wonderful then that _why has put some effort into fixing this little conundrum and released the first build of HacketyHack which calls itself “the Coder’s Starter Kit”. He’s only released for Windows so far, so I’ve not been able to try it out (still less get involved in helping him close tickets - I’d throw time at this without thinking about it for a second), but his Manifesto is something I wish I had written myself. He’s on the same page I’ve been whining on about for the last year.

I hope it the best of success, and the moment _why releases it for other platforms that I actually have access to, I’ll make sure I get stuck in on helping submit patches and spreading the word further.

In the last few weeks I’ve been seeing more and more chatter about Erlang, a language from Ericsson that is primarily used in telecommunications applications with soft real-time, distributed, concurrent, fault-tolerant properties. It would seem that the section of the blogosphere I’m following (mostly trend-setter developers) are taking an interest in how useful it may be outside of applications like running telco switch equipment.

I first stumbled across Erlang almost a decade ago when it was first released and was considering a roll-out of Eddie, which at the time was the only software solution to load-balancing that looked like it might actually work. That project got canned, but I’ve always had this niggle that I should go and play with Erlang some more and that there had to be ways of being able to use the unique qualities of Erlang to do interesting work in terms of Internet - and specifically Web - applications.

David Pollak’s article on why implementing Twitter in Erlang might be a good idea confirmed in me the idea that at least Erlang’s fans think its time might have come for concurrent web applications and got me thinking. What would I do with this highly scalable, distributed and concurrent system in terms of a web application that would scare me if I were to attempt doing them in Rails/Ruby?

So, I rolled this around in my head a little bit, and am thinking that in a month or two’s time it might be nice to try and build some interesting applications in Erlang - nothing ground-breaking, just experiments in scalability, maybe a Twitter-clone to see whether David’s hunch is right. In the process, pick up a new vocabulary to think about software in terms of how Erlang sees the World. I’ve already noticed from the FAQ that it has some quirks (a couple of basic string manipulation functions in the httpd_util library for ‘historical reasons’ is making me think it needs some love as a language), but I’m intrigued enough to want to learn more.

Whilst the free resources on the official site look comprehensive, I was mildly relieved from my experiences with their Rails/Ruby titles to notice that the Pragmatic Programmer’s have released a beta of an Erlang title that at first glance seems to be pretty comprehensive already, and of a similar style to the Ruby/Rails titles from that stable.

I always get a little nervy when making friends with a new language, especially one that describes the World in a way that’s new to me. Over the next month or two I’m going to be going through the resources and building a few toy applications to help me on my way. I’ve got a lot of work to get done in Ruby and Rails though, so I hope I don’t get too deep, too quick, or the landlord will be calling again…

In the last few weeks, I’ve been evaluating source code I’ve written over the last year under closed source commercial agreements. I did this partly to try and look at what I did wrong and learn from those mistakes at a business level and technical level, but also to find out if there are common themes to anything I was working on that I could extract to make my life easier in future.

What I found shouldn’t surprise most people. Everything I did was about building forms, allowing data from those forms to be collected, and then looked at in some way. Every project looked like this:

• Define a form
• Build form
• Users input data into form
• Save data
• Look at data

Sometimes I would define/build the form and deliver the app, but at least two apps included the ability for users to build forms them (basically, survey software).

In fact, all web sites with UGC follow this model at some level - a blog engine is a form for a post, followed by the display of entered data in a particular way or set of ways. YouTube is a set of forms to handle file uploads followed by some data processing, and then displaying that content.

That’s quite interesting, because we have so far thought about web applications being custom little pieces of code in almost every case. Rails and similar frameworks enforce this model however, and make it easier by doing a lot of the heavy lifting involved in describing those processes. However, ultimately all we’re doing is putting lipstick onto relational databases. If that’s all we’re doing, surely there is a quicker way of doing it?

I was thinking that each of those stages in each application is already defined somehow as a real world business process we want to map into software. Why can’t we find a way of being able to quickly define a business process in code, and then just generate an application based on that definition?

It seems I’m a few years behind in my thinking. Business Process Management (BPM) is a mature area of development with a gazillion standards out there and it would seem XPDL is the XML-love of choice for this crowd. Digging further, I found something even cooler for the Rubyists amongst you: OpenWFEru nicknamed ‘Rufus’ amongst developers seems to have most of the heavy lifting done for you. The example looks promising.

This is interesting for me, because I have to build a couple of pieces of software soon which really are understanding workflows within teams, and I’d rather spend more time reading 19th-century books on political philosophy (hey, everybody’s got to have a hobby).

As a result I’m trying to work out how to hammer this into those projects. If I can find a way of being able to take the “I’ll look after the forms for you” aspect of Rails with the ease of defining workflows found in OpenWFEru and coupling them could you define a bunch of schemas and a workflow and then generate a better scaffolded Rails application? Would the controllers be able to mature over CRUD and do something more useful? Or could you build a CMS in Rails that uses user-generated workflows to allow for quickly hacked-together custom applications? It’s an intriguing bunch of thoughts…

On the 1st March (next Thursday, as I write this), I’m planning to do a mini re-launch of Vagueware.com and with a new website comes a new business model.

I will be phasing things in over the next 6-8 weeks. First, the “open innovation” phase, where ideas are thrown around, submitted and developed by anybody with an interest. Then the time to turn some of those ideas into real open source code, so anybody and everybody can develop the ideas further. And of course, run useful software. Finally a services model to make all your Rails deployment nightmares go away, and to bring some cashflow around those open source tools - support, maintenance, bespoke customisation will feature as well as hosted SaaS solutions.

However, I have a conundrum.

I’m busy finishing up work elsewhere right now. I have some todo items on older projects to close, and I am also involved in a second company completely unrelated to Vagueware that desperately needs some of my coding love right now. Time between now and next Thursday is tight. And the code base for the next version of Vagueware.com? Well, to be honest, barely started.

Right now, I need the splash page, and the ability for users to create accounts, edit wiki pages, for me to be able to structure some CMS around it all, add some forums, etc. It’s not a huge job, but to code it up from scratch even with the existing plugins out there might take longer than a week given other pulls on my time. It’s obvious that given the focus on it being a Ruby/Rails led set of projects, the time should be taken to develop the site itself in Rails.

There is an alternative though. Use something else other than Rails for now. Drupal does nearly everything I need and I could configure it all up this weekend ready to go. But it’s PHP. And that feels like a lie to me, and a lie to the other people I want to involve in this.

I could go meta and suggest that the first project is a tool to replace the site in Rails that would then allow Vagueware to become self-hosting. At least then the ideas would get started, and providing I was able to export everything from the inter-rim site when the job was done, nothing would be lost. It would allow those people who want to show up on day one to be involved in developing a community tool useful to them, and even have an input into how projects should be handled in the future. It could be an advantage.

It still feels like a major hack though. Anybody have any other ideas? I could delay the launch by another couple of weeks, but I’m keen to get started ASAP.

Reading an article on Ajaxian.com just now, I was struck by this screenshot:

Note, in the bottom left-hand corner, the keyboard shortcuts. This, to my mind is an example when keyboard shortcuts are just going to leave the user confused.

Don’t get me wrong - I love shortcuts. I rarely move my hands off they keyboard unless I have to, and one of the problems I had when I first moved to a Mac was learning and configuring things the way I like them. I don’t like using a mouse to repeat an action I think I should be able to do with a keyboard alone.

However, whilst it makes sense for my OS to have them, my browser (which I spend four or five hours a day staring at) to have them, and for my text editor (another three or four hours a day), a trivial app that I might visit once in a while just doesn’t need them. If a web application is something I’m going to spend hours per day or week looking at, such as Gmail or a calendar or an RSS reader, it might make sense. But for a survey tool site? I think they got confused somewhere along the line.

A developer I’m working with points me to an announcement of a serious DoS vulnerability in Ruby’s CGI library that would allow Rails apps to get locked up with a trivial attack if they process MIME parts at all.

Crikey.

So I follow the paper-trail to the original source and get here: Mongrel Temporary Fix For cgi.rb 99% CPU DoS Attack

Wow. That looks serious. It can kill the server! 99% lock-up! Urgent patch!

I don’t dispute the seriousness of the effect of the vulnerability, but having looked at the patch file, I’m confused this wasn’t picked up in testing, and it took me a whole five minutes to stop laughing after thinking about people running around complaining that their Windows gem couldn’t be updated:

--- /opt/local/lib/ruby/1.8/cgi.rb      2005-10-06 19:01:22.000000000 -0600
+++ cgi.rb      2006-09-22 16:38:08.000000000 -0600
@@ -1017,7 +1017,7 @@
               else
                 stdinput.read(content_length)
               end
-          if c.nil?
+          if c.nil? || c.empty?
             raise EOFError, "bad content body"
           end
           buf.concat(c)

No, I’m not making that up. It really is just a one line change from ‘if c.nil?’ to ‘if c.nil? || c.empty?’. Reminds of something I read earlier about the Rails method ‘blank?’ which basically tests if an object is nil or empty, and nothing else. Looks like it should be in the base of Ruby itself. :-)

And before you test whether I’m patched - yes, I am.

For a long time, I’ve had a problem explaining exactly why software engineering doesn’t work like other kinds of engineering. One of the issues I’ve had is trying to find the wording needed to explain that programmers don’t really build anything, and that to manage building software like building a bridge makes no sense. I couldn’t quite put my finger on the right words though.

Until now.

Jack W. Reeve’s essay on ‘What is Software Design?’ is the best work I’ve seen to date on the subject. He argues - convincingly - that programmers are designers, architects, draftsmen at a drawing board. We don’t build anything: compilers, linkers, interpreters do that for us. He points out how flawed an idea it is to produce a design document that can never truly represent the details of how the software will work, because it is an abstraction of an abstraction. The only accurate design is the software source code itself.

What’s more, it makes sense that in the same way that an architect can build a model, which can be turned into a few more detailed drawings, which can then be turned into engineering drawings, it makes sense to start all software development with a set of mock-ups which are then turned into a UI (say, static HTML) which then become plugged together with code. It’s like building a skyscraper by defining what the bits ‘users’ will see will look like, then working out how the nuts and bolts will hold it together: exactly what an architect does.

His essay dates from a time when the best language available to programmers to express their design was C++, and he argues that C++ helps his model of ‘design as code’ along nicely. It was the best language for the job because as an object-orientated language, it allowed an expression that most closely resembled how we abstractly thought about objects and data in the real World. Since then, of course, things have moved along a little bit.

I would argue today that Ruby is the best design language we have available to us. I claim this, because I actually find it easier and quicker to express an abstract design - which in turn can then execute - through Ruby than I do through a diagram or notation or pseudo-code.

At this point you might be thinking “yeah, right, whatever!”, but this isn’t some idle boast, I eat my own dog food.

The other day I needed to clarify a design point with another programmer via e-mail. I needed to express a HABTM relationship being used as a way of listing ‘exceptions’ to a ‘everybody/nobody’ flag stored in a model. The details aren’t important, but I considered for a moment the very best way of communicating this. I considered drawing a UML diagram. I thought about a quick sketch with some boxes, I considered a little pseudo-code. I then remembered that the programmer I was discussing this with could code Ruby, and maybe if I just developed a few stubs with some comments he’d get the idea.

Five minutes later, I had not just produced stub code. I had some migrations that would build the tables needed in Rails, and I had a few methods I was proposing adding to a model. It was untested, and some DateTime handling code would need a little fleshing out, but I’d discovered it was just quicker to write the code than it would be to write the comments explaining what the code would do. I actually ended up writing a software design in my mail client, that I could copy and paste over into a code editor and with a little bit of expansion and testing, get running.

In other words, the perfect design notation seems to have been discovered - it’s not UML that generates stub C++ in a CASE tool, or some formal design notation. It’s Ruby that just, well, runs. It is perfect because it is the perfect balance of programming language and design notation.

So not only was Reeves right in saying that source code is the best detailed design possible, that we are designers at heart, that programming languages that work at an abstract level are a step in the right direction, that formal design tools are useless and pointless, but he was even right that eventually we’d produce a programming language that was just a compilable design notation.

We now have that tool and its name is Ruby.

UPDATE: Two separate comments - each without seeing the other, as I didn’t approve them until this morning - suggest that LISP is a better design tool than Ruby.

It is worth pointing out that Ruby is kind of a grandchild of Lisp because it inherits so much from Scheme, itself a dialect of Lisp.

So is Ruby just another Lisp dialect? No. Whilst borrowing aspects of Lisp, it is more closely related to Smalltalk in terms of its object-orientation. It also borrows from Python, Dylan and CLU. It uses the best of all of them - Lisp included - and makes something better.

Also, there is another important part of Ruby that Lisp (and Smalltalk to be fair) do not yet have: an easy way to rapidly produce production-level, deployable, useful software. I can build a web application in Ruby - thanks to Rails, or Camping - in relatively little time compared to 10 years ago in PHP or Perl. I suspect, although haven’t researched it fully, that to do the same in Lisp might take me a little longer.

Ruby has many of the great features of Lisp baked in, but many of its own bolted in to boot. What’s more, the design ethos of Ruby is its strongest point: Ruby is designed to make programming pleasurable. When you enjoy writing code, you write more of it in less time, and with fewer bugs. Lisp might be fun for those who enjoy it, but I agree to some extent with Larry Wall on this one: it “has the visual appeal of oatmeal with fingernail clippings mixed in”.

If you like Lisp, I suspect you will love Ruby, and you’ll find yourself more productive in it. Either way, I’ll concede that for some people in love with it, Lisp is as useful to them as a design language as Ruby is to me. We’re both lucky, we both know we are, and we’re both a step ahead of the C# and Java guys.

I still say Ruby rocks more though. :-P

UPDATE 2: Changed link to the PDF to point to one held at the official home and authorised version of these PDFs. Thanks to Daniel Read for pointing out my error in directing you all to a pirate copy.

One of the most pleasurable places for a developer to be in is “the zone”. It’s where everything else - the local environment around the desk, the street outside, the family at home, the bills, the workload, even sometimes hunger and thirst, are all completely forgotten. The only thing that matters at that moment is the problem in front of you. For some developers - particularly those who are Unix systems administrators at heart, like me - it can go on for days at a time. I sometimes call it “my coding head” - I hunger for the problem, and nothing else can get in the way.

Getting into the zone is not always easy. Different people have different techniques, nearly all of which involve a bit of peace and quiet and no interruptions. You need to start somehow and somewhere, and I’ve heard that people sometimes procrastinate fearing that they won’t get in the zone - they’re worried about phone calls to make, etc. so they’ll do those first. I find saying “I’ll just do half an hour” to myself, can sometimes result in a ten hour coding marathon.

Once you’re there, the worst thing that can happen is to be interrupted - I frequently shut down mail and IM clients when I want to make progress. When I used to work in shared offices, I would find a quiet corner. I shouted at people who interrupted me with trivial problems and put signs on my desk saying “Please don’t disturb unless building is on fire”. My bosses hated me because of it. Even now, I sometimes set alarms on my phone to remind me to eat - it has been known for me to forget, and suddenly think “hmmm, think I’ll have a bit of tea”, only to discover it’s 4am.

Today, the zone is just jumping and down all over my head and the feeling is wonderful - I can slip in and out of it even with light interruption. My technique involves a decent high-carb meal, some coffee, headphones and iTunes. Musically, almost anything that isn’t too heavy is fine, and my playlist gets tailored based on the kind of code I’m dealing with. I tend to do weird abstract things better with pop/rock (although strangely, always with female vocals), and simpler but more ‘big wedge of code that needs sorting’ projects get shifted easiest with some Mozart, Saint-Saens, or Delibes. I think that’s because drudgery is easier to cope with when I have some complicated music patterns going on in the background, and abstract/complicated work needs nothing more complex than a simple and melodic tune playing.

Another reason that The Zone is so useful, is that it actually relaxes me. My workload right now is very high - almost stressful - but I’m getting through it. I’ve been beset by problems with my development environment this week, clients’ work is backing up in the queue and I’m trying to get to grips with two new projects whilst simultaneously putting two old projects to bed for good. Last night I had a pretty restless night - this morning I overslept - to be greeted on the phone by a client who themselves was so worried about a deliverable that she hadn’t been sleeping properly. For a few moments this morning, I was relating to Atlas - the World upon my shoulders, etc.

This afternoon though, it was smiles all the way. I became detached from it all and was able to just make progress, gliding forwards, pieces of code jigsaws suddenly came slotting down into place, and the rest of the World could go to hell. It was relaxing, fun, enjoyable. Me, code, my mp3 collection, and a handy kettle.

Then, something odd happened that triggered an old memory. ‘Sunshine on a Rainy Day’ by Zoe came onto the play-list. A dodgy piece of cheese you might think, but it’s in my collection for one reason and one reason only: it was once used on a Commodore Amiga advert. The kid in the advert was playing Lemmings, looked out onto a rain-swept street, and just didn’t care: he was in the zone, happy with the amazing tool he had in front of him. Geeky, but happy. I didn’t know any hardcore geeks when I was learning to code, and that was the first time I saw somebody else enjoying the zone.

I looked up, and outside saw on my balcony that it was raining, just like in that old advert. Peering out into the gloom I could see Manchester was wet, rainy, cold. But it didn’t matter, I was warm, dry, my face lit by the glow of the screen, just like the advert. I was in the zone. Removed from the reality of the World, I was playing my own version of Lemmings - coding and strategy games are remarkably similar from a problem-solving perspective - and I remembered what it was that got me into this gig in the first place all those years ago when the Amiga was the ultimate computer, but all I could afford was a beaten-up second-hand Amstrad CPC6128: The Zone.

It’s not the money. It’s not the freedom of my own business. It’s not even the end product. It’s the process, the thinking, the working stuff out. It doesn’t matter how stressed I feel about the code that needs to be done, I can escape it all by doing the code itself. How many people can honestly say they have a legal occupation that gives them a buzz and helps melt away stress without the need for chemicals?

Nat Torkington on the O’Reilly blog points out some interesting figures on COBOL published this week.

There’s something not right about those figures. That depth of market penetration just doesn’t stack up. At the time of writing there have been in the last 7 days a total of 154 jobs with the word ‘COBOL’ in it posted on JobServe. For comparison, there were 4,771 with ‘Java’ in there, 3,016 with ‘C#’ and 3,033 with C++ in the description or title. To suggest COBOL is a major force in IT these days just doesn’t make sense.

It’s not as if there is much money in it right now either - £35k/year salaried maximum, or £313/day contract tops. Compare that to a minimum average salary of £37k (with maximum touching almost £47k) or average day rate of £486 for Java.

Ruby’s figures are all over the place right now - 45% lower on average compared to this time last year - because there are so many kids who learnt Rails 15 minutes ago asking for PHP rates, but I don’t think they actually know what they’re doing. I certainly don’t think people paying that rate are going to be impressed with the people walking in the door, and may be turned off Rails/Ruby completely.

I don’t know anybody doing serious Ruby development work who actually knows and has experience of the language on less than £275/day - and for quality coders, that figure is going to rise soon, I figure. My ‘sticker price’ has been around £300/day and has held up pretty well over the last year, but I’m starting to give discounts to some projects because I actually like them and the people involved in them, and I want to be competitive to get those jobs.

That said, there is something interesting about COBOL and the way we see languages like it too. Some people are out on the front edge finding the best way to do things - cheaper, faster, better - by exploring whole new language paradigms, methodologies and tools. I consider myself to be one of that crowd: I’m already looking at what might be next after Ruby hits critical mass. Consider me an ‘early adopter coder’ interested in doing things better.

Meanwhile, there are plenty of people out there who just want things to keep on working as they have done for the last 30 years and they don’t care it could be done better. They’re just glad it’s being done and want programmers to make sure it stays that way. I think they’re sensible in one regard, but I wonder how much money they’re losing in service and maintenance charges?

I have been offered COBOL work in the last few months, and if I hadn’t been so busy I might have taken it on. I suspect there might be a big market in helping all those old tech houses upgrade without having to go and get ripped off by one of the more expensive consultancies - it might be that as we shift business paradigms from call centers and shops with staff over to buying online, this is the perfect time for those businesses to move over, upgrade, and put 30 years of hurt behind them.

Either way, COBOL is dying regardless of what the surveys say, and it’s dying fast. Java is stable, but nobody loves it. Meanwhile, the Ruby figures scare the hell out of me - I know there is scarcity in this market, so why the low prices being put out by IT recruitment agencies? What other weird languages are there out there with weird market prices?

I’ve just spent an hour re-factoring some of my code. I also put some comments in, because there weren’t any when I started. I also wrote some unit tests I should have done before writing the code, and actually ended up fixing a bug that would happen in rare cases. I also made it smaller, faster and less of a CPU hog. I increased the readability and fixed a couple of indenting issues.

What surprises me is that I didn’t do any of this when I wrote the code in the first place.

I think part of the problem is that when you have a deadline, you just care about getting a solution together that works, and you want it ASAP. You don’t care about elegance when the bosses are crying for a customer sign-off at the end of the week. As a result, you take shortcuts, jump around and deploy code that “will kinda work” and just hope it never fails. Or at least, hope it never fails whilst you still work there.

Strangely, some of the best code I’ve ever seen out there is open source, which feels counter-intuitive - it’s a bunch of people doing hobby code, not something they are being paid to produce. I think it’s because there is no deadline with open source that makes it better. It has been said that programmers work on open source projects in the evening because it’s relaxing. Without the deadlines, pressures and insistence things are done a certain way, developers can revel in the intellectual nature of development. As a result, they take the time to unit test, to comment, to document to re-factor properly.

One of the interesting conversations developers could have with customers and their management is to ask “Which is more important? Code that kind of works right now, or code that is built to last?” and I think the conclusions would shock us. Most customers don’t care that you got a big block of code reduced down to a really nice recursive loop. Most bosses aren’t bothered about comments in code, because they don’t know the real price of not having them.

This exercise tonight has convinced me I need to change my working practices, and I need to stop trying to undercut on prices - I need to price as a job, raise the time estimates to remove the deadlines and start promising to produce code that is built to last. What I’m worried about is that this isn’t a very good sales strapline: we’re more expensive and take longer because we do stuff you don’t care about.

Even so, I think it’s worth doing. Quality code makes me happier than quickly developed code.

Many years ago, I didn’t program just for money. I actually did it because I loved the problem solving aspect. I kind of miss that - I spend way too much time formatting reports and generating graphs these days, not enough time just messing around for the hell of it.

The other day, I discovered CodeGolf.com as a new hobby/competitive sport. The idea is simplicity itself: take one of the challenges, produce working code in your interepreted language of choice (Perl, Python, PHP or my favourte, Ruby) that pumps out the right output and then get it working in the smallest number of bytes possible. It’s an exercise in obfuscation and confusion, sure, but it’s great fun for somebody like me.

The challenge that drew me in was Oblongular Number Spirals which looks deceptively simple. Let’s just say that after about an hour I realised it wasn’t - it’s actually very, very hard. I ended up with a solution that uses a really bad edge detection algorithm as I walk across a virtual grid (actually a multi-dimensional array) making sure I don’t bump into other rows and the edges.

A working solution must have taken me a total of six hours coding time - which shows how out of practice I am with this pure algorithm work. Reducing it down to the smallest number of bytes was the easiest bit, but falling asleep last night I realised that there were ways I could easily reduce it further from my poor show of 426 bytes. Anyway, it was great fun, and for the masochists amongst you, here is my slightly longer (line breaks are in this one) script:

s,t=gets.split.map!{|s|s.to_i};b=[];
k,d,x,y,e,f,g,h=s*t,0,0,0,-1,-1,s,t;o=k.to_s;
(1..k).each{|i|b[y]=[]if b[y].nil?;b[y][x]=i;
case d;
when 0;if x+1==g;d=1;f=y;y+=1;else;x+=1;end;
when 1;if y+1==h;d=2;g=x;x-=1;else;y+=1;end;
when 2;if x-1==e;d=3;h=y;y-=1;else;x-=1;end;
when 3;if y-1==f;d=0;e=x;x+=1;else;
y-=1;end;end};
(0...t).each{|r|m="";
(0...s).each{|q|z=b[r][q].to_s;m<<" "*(o.length-z.length);
m<<" "unless q==0;
m<< z;};puts m;}

Isn’t that awful? Taking a language as beautiful as Ruby and making it look like that! I should be ashamed of myself. For some reason, in some browsers, the last line of the above block gets chopped but should read ‘m << z;};puts m;}'

Here's a slightly less painful version:

s, t = gets.split.map!{ |s| s.to_i }
b=[]
k, d, x, y, e, f, g, h = s*t, 0, 0, 0, -1, -1, s, t
o = k.to_s

(1..k).each{ |i|
  b[y] = []
  if b[y].nil?
    b[y][x] = i   
    case d
    when 0
      if x + 1 == g
        d = 1; f = y; y += 1
      else
        x += 1
      end
    when 1
      if y + 1 == h
        d = 2; g = x; x -= 1
      else
        y += 1
      end
    when 2
      if x - 1 == e
        d = 3; h = y; y -= 1
      else
        x -= 1
      end
    when 3
      if y - 1 == f
        d = 0; e = x; x += 1
      else
        y -= 1
      end
    end
}

(0...t).each{ |r|
  m = ""
  (0...s).each{ |q|
    z = b[r][q].to_s
    m << " " * (o.length - z.length)
    m << " " unless q == 0
    m << z
  }
  puts m
}

That produces output that passes the codegolf tests, so if somebody wants to take it and make the obvious space-saving optimisations (yes, I spelt that word incorrectly the last time I used it), that will get it down to less than 200 bytes, feel free.

And to any potential clients reading this right now, let me promise you, hand on heart, I would never dream of putting code that awful into your production environments. The code you buy would be elegant, commented, and use meaningful variable names. Honest. ;-)