Innovation in Software

Development is a creative pursuit. Whilst many think of it as a purely technical challenge, it requires a level of lateral thinking about the World that is a cross between doing a crossword puzzle, composing a symphony and having an argument with people who don’t exist. It’s not surprising some of us are a little eccentric.

It reminds me of the writing process a lot. You sit down at a blank screen after having conducted your research and you have to just dig in and find some way of making progress. Many a developer struggles with a blank IDE screen much in the same way many a writer struggles to find influence. When I was learning how to write properly, I was told that “a professional writer can not be like the poet who spends a morning taking out a comma, and the afternoon putting it back in”. We need to work hard. Same with code. A block then is a real problem.

Slashdot this weekend asked How to Get Out of Developer’s Block?, or rather a user asked:

I have spent the past six months working on a software project, and while I can come up with ideas, I just can’t seem to sit down in front of the computer to code. I sit there and I just can’t concentrate. I don’t know whether this is akin to writer’s block, but it feels like it. Have any other Slashdotters run into this and if so how did you get out of it? It is bothering me since the project has ground to a halt and I really want to get started again. I am the sole developer on the project, if that makes a difference.

The comments that follow in the thread that range from the sensible to the bizarre. I have a bunch of tricks I use when I’m struggling, so thought I’d put them together

1. Get enough sleep – you have no idea how sleep deprivation can mess you up when you’re trying to concentrate. When I’m working on code, I take a minimum of 10 hours sleep a night. Anything less, and I’m not going to be able to think in purely abstract terms for 8 hours straight during the day.
2. Exercise – and whilst those of you who know me might laugh, it’s important. I actually do get regular exercise when I’m coding full-time. Just a long walk at the start or end of the day can be enough. Something that gets the hear rate up helps though (perhaps explaining why I always code better the day after… errr… private stuff that gets my heart rate up!).
3. Don’t drink alcohol – this was something I got when I was trying to sort out my pilot’s license. When you’re going flying, I don’t drink for 24 hours before getting into the plane. I found my workload was easier, my writing got more fluid and my code went up a gear. On big client projects I don’t drink at all on school nights. If I’m drinking in the evenings whilst on a project, it’s because the project isn’t challenging me and I’m bored.
4. Clear your environment out – I’m currently sat at a desk with perhaps 150 items of paperwork on it. In this environment, I can not focus on code. My mental processes are cluttered because my physical processes are. Tidying up might seem like a stupid way to get out of a block, but I genuinely find that a clear working environment leads to much clearer mental processes. I don’t know how or why, it fascinates me, but just get your physical environment fixed up and suddenly your mental environment starts to fire a little better than before.
5. Write a trivial test – this is the code version of “free-writing” that I sometimes use to unblock on writing an article. Basically write a small test (or spec if you’re BDD) for something almost trivial and then get it to pass. Repeat. Now you’re back in the game.
6. Work on the design – it’s amazing how bad we collectively are at really thinking through a problem. Go and work on some wireframes or develop some sketches of the underlying schemas and try and simplify them. Reduce things down, and suddenly you’ll see areas you can work on right away outside of the problem you’re blocked on. If you’re not able to delve into design or architecture because of the nature of the project, quite frankly you need another bunch of guys to work with.
7. Try and find it done already – I once spent a lot of time trying to work out how to solve a particular problem. The answer was non-trivial to implement in my mind. I kept putting it off. I was scared of how bad I could end up making my solution. In a fit of procrastination I spent an hour digging around the problem area and eventually found an open-source tool that did exactly what I needed, out of the box. Well, that solves that problem…
8. Are you scared of success? It might sound like a stupid question because success is good, right? But when we succeed at something, we conquer some barrier we have worked to overcome for a period, things change. Suddenly people might look at you differently. Perhaps you end up having to work on a less interesting project. You might want your current project to be a success for other reasons. Ask yourself whether you really want this project to succeed. And then realise there’s no getting out of it: failing, or staying where you are is just as bad an outcome and harms you, your self-confidence and your reputation.
9. Find a SCRUM meeting somewhere – one of the very best things about daily stand-up meetings in SCRUM projects is that the meeting only has three topics of conversation: outcomes from stuff you agreed to do in the last meeting; what you plan to do today to further the project, if anything; and obstacles in your way. Not everybody has a team (and sole development is the hardest form there is, trust me), so find a SCRUM somewhere else. Use Twitter, your blog, a group of friends down the pub, anything. Just talk about what’s stopping you and see if anybody can help you in any way, or offer suggestions. Obviously asking a friend about a tricky problem relating to class inheritance isn’t going to yield results if they don’t know what you’re on about, but ask around more liberally than you have done to date.
10. Work on something else – we all have other projects on the go. If the above isn’t working, just go and get on with something else. Your subconscious is dealing with the problem and will come up with a solution. Just make sure you hit your deliverables schedule if you have one!

Now comments are back up, I look forward to hearing of any other tips people might have.

Who would have thought that one of the most innovative players in the hosting and web application industry would be a bookshop?

One of the big problems with Amazon’s web services is that they aren’t that great for permanently hosted web applications. There’s the dynamic IP addressing issue (which weoceo will look after if you have the cash) and the serious problem of how to store your database.

S3 is very nice, but it stores flat data, and certainly not anything as fancy as SQL tables. Until recently there was a hacky way to do it with a special storage engine for MySQL, but just looking at it made me nervous about my data.

Well, Amazon have decided to fix this issue. I received this email from them this morning.

“Dear AWS Developers,

This is a short note to let a subset of our most active developers know about an upcoming limited beta of our newest web service: Amazon SimpleDB, which is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database – real-time lookup and simple querying of structured data – without the operational complexity.

Were excited about this upcoming service and wanted to let you know about it as soon as possible. We anticipate beginning the limited beta in the next few weeks. In the meantime, you can read more about the service, and sign up to be notified when the limited beta program opens and a spot becomes available for you. To do so, simply click the “Sign Up For This Web Service” button on the web site below and we will record your contact information.

Learn more and sign up

Sincerely,

The Amazon Web Services Team”

So, there we have it. No more managing DB clusters. Scalable database tables, which once the beta is over will likely come with an SLA. Assuming that this just sits on top of S3, we might even be able to host our data inside the EU and get ll warm and fuzzy about protecting customer data properly.

I’m not sure this will be based on a standard set of DB libs but I expect we’ll see 1-line hacks to make it work with Rails, PHP and a host of other app frameworks within a few weeks.

I’m in.

I never “got” code reviews. I never understood how sitting around talking about somebody else’s code would help them or you, beyond being able to weed out the obviously awful coders.

This week I’ve been doing some consultancy, in-sourced to help a firm evaluate what their developers have built for them. As part of that process I’ve had to review documentation, specifications and code. The process has been at times confusing and enlightening.

It’s one thing to be able to say “this is how it should be done” when you’re not a developer, but it’s another thing when you are a developer who hasn’t done some of those things yourself. This process has taught me as much about myself and my own code as it has about these guys’ code.

Interestingly, I now see things from a buyer’s perspective much more clearly, and I can see how frustrating this must be from the other side. I can see how things which we don’t consider important are to a buyer critical. I can see how clever solutions can sometimes be too clever. I can see how something simple you forget to do can make all the difference.

Adrian Holovaty announced a while back templatemaker which sounds more complicated than it is.

Take a pile of web pages all built using the same template – say, restaurant listings, or a timetable – and throw it at the code. It produces a copy of the ‘template’ with the ‘data holes’ marked up. You then look at a new page, and it gives you the raw data. No more screen scraping, it just learns what it needs to find the data that changes each time. Nice.

A really useful aspect is that whilst he’s implemented it with Python bindings, the core is written in C and uses the Python C bindings. That means with a bit of hackery, porting it to Ruby should be quite simple. A project for the weekend, perhaps?

I’m shouting this up for two reasons: it’s a piece of code I’m sure all of us have thought about implementing, but never got around to producing because it just ‘felt’ too difficult, so it’s definitely very innovative from that perspective; secondly, it’s using the C bindings in an intelligent way to get performance where it counts – something I’m going to need to do more of over the next few months in other languages.

I suspect the future isn’t going to be about using the right language for a project, but the right language for the right part of the job. You could implement something in Erlang, which can interface with Ruby, which in turn can interface with C. There are points of complexity there that a competent software engineer would shudder over, I’m starting to feel that being competent in one language or framework is definitely not going to be enough for the next generation of software: we need the flexibility and power of scripting languages, and the raw power of compiled C/C++.

It’s only Tuesday afternoon, and already this week I’ve learned a great deal about producing better software, albeit inadvertently – nothing teaches you more about code than having to fix stupid mistake. Each week I always learn some new insight but this week it’s been like a waterfall of learning the hard way. I thought I’d moan at you lot, because it’s more fun than whining about it all inside my own head. :-)

• Users are lazier than you think – to me, it’s pretty obvious that if I’ve already entered data into a system, I’m not going to enter it all over again, but this time full of typos and changing reference numbers around. I’m just going to enter the data in correctly first time, and if I have a problem, I’ll go back and edit it. Not so users of one system I’ve been working on this week. Cue an afternoon of going through data and checking it for consistency errors
• Users don’t read the screen – we noticed a load of weird consistencies elsewhere in the same system. It turns out that I’d upgraded ferret on the system, and then the ferret index on the app was obviously foo-barred (note: when doing a gem update on ferret you MUST either clear out the index dir in your Rails app or you must script/console and Model.rebuildindex on each Model you’ve got actsas_ferret rolling with – you don’t have a choice in the matter). Strangely, in the period the app was doing this, people must have been putting data in and getting the “Application error” screen. Did they stop? Not likely – they just carried on, hoping it wouldn’t matter. Cue another half an afternoon of checking the database for errors.
• Logs are more useful than you think – Rails logs all parameters it gets, even in production mode. Some people think this is a security risk, and for password forms you should definitely turn it off (look up filterparameterlogging to see how). However, when you have a load of weird consistency errors in your DB, my god do those logs help reconstruct what was going on and help you fix what happened
• script/console doesn’t do what it says it just did – try this: take a model with a datetime column. In script/console, try and assign a new date to a record and save it. Note how that didn’t work. I’m not quite at the bottom of why, but it’s curious that it doesn’t “just work” – it may be a config issue, but I can’t see what right now.
• Nobody cares about my problems, they’ve got their own – I don’t even know I bother bitching about this stuff: I just find myself shouting into a big empty echo chamber that is the Web. :-)

Anyway, I’m at a careers fair tomorrow helping out MDDA up in Cheetham Hill. I hope to return with all my limbs intact.

After that, I’ve got tickets for all five days of the cricket at Old Trafford, so I’ll be pretty much in holiday mode after tomorrow.

The people who have to listen to my diatribes about what is wrong with programming in person know there is one thing that bothers me more than anything: we’re not getting kids interested in programming, and we’re going to pay for it in the future.

When I think back to how I got into writing software, I realise I just don’t see it the way teenagers do. They see it as a career choice and will actively avoid writing a single line of software until they get to University. For me, code was something I delved into from the age of 11 and never really returned. I was writing C when I was 14, and I think it actually changed the way I thought about software permanently – it meant that something changed inside my brain as it was still growing. I don’t think that’s a bad thing.

That said, I understand the barrier of entry is now way too high. Most kids do not want to get into an IDE. When they think “it might be nice to learn to program” and go to the bookstore, they are confronted with a wall of very expensive, very dull looking books. The first book I ever read on programming was on BBC Basic and was illustrated with pictures of robots in factories pretending to be FOR loops. That isn’t really comparable to what kids have access to today.

It’s wonderful then that _why has put some effort into fixing this little conundrum and released the first build of HacketyHack which calls itself “the Coder’s Starter Kit”. He’s only released for Windows so far, so I’ve not been able to try it out (still less get involved in helping him close tickets – I’d throw time at this without thinking about it for a second), but his Manifesto is something I wish I had written myself. He’s on the same page I’ve been whining on about for the last year.

I hope it the best of success, and the moment _why releases it for other platforms that I actually have access to, I’ll make sure I get stuck in on helping submit patches and spreading the word further.

In the last few weeks I’ve been seeing more and more chatter about Erlang, a language from Ericsson that is primarily used in telecommunications applications with soft real-time, distributed, concurrent, fault-tolerant properties. It would seem that the section of the blogosphere I’m following (mostly trend-setter developers) are taking an interest in how useful it may be outside of applications like running telco switch equipment.

I first stumbled across Erlang almost a decade ago when it was first released and was considering a roll-out of Eddie, which at the time was the only software solution to load-balancing that looked like it might actually work. That project got canned, but I’ve always had this niggle that I should go and play with Erlang some more and that there had to be ways of being able to use the unique qualities of Erlang to do interesting work in terms of Internet – and specifically Web – applications.

David Pollak’s article on why implementing Twitter in Erlang might be a good idea confirmed in me the idea that at least Erlang’s fans think its time might have come for concurrent web applications and got me thinking. What would I do with this highly scalable, distributed and concurrent system in terms of a web application that would scare me if I were to attempt doing them in Rails/Ruby?

So, I rolled this around in my head a little bit, and am thinking that in a month or two’s time it might be nice to try and build some interesting applications in Erlang – nothing ground-breaking, just experiments in scalability, maybe a Twitter-clone to see whether David’s hunch is right. In the process, pick up a new vocabulary to think about software in terms of how Erlang sees the World. I’ve already noticed from the FAQ that it has some quirks (a couple of basic string manipulation functions in the httpd_util library for ‘historical reasons’ is making me think it needs some love as a language), but I’m intrigued enough to want to learn more.

Whilst the free resources on the official site look comprehensive, I was mildly relieved from my experiences with their Rails/Ruby titles to notice that the Pragmatic Programmer’s have released a beta of an Erlang title that at first glance seems to be pretty comprehensive already, and of a similar style to the Ruby/Rails titles from that stable.

I always get a little nervy when making friends with a new language, especially one that describes the World in a way that’s new to me. Over the next month or two I’m going to be going through the resources and building a few toy applications to help me on my way. I’ve got a lot of work to get done in Ruby and Rails though, so I hope I don’t get too deep, too quick, or the landlord will be calling again…

In the last few weeks, I’ve been evaluating source code I’ve written over the last year under closed source commercial agreements. I did this partly to try and look at what I did wrong and learn from those mistakes at a business level and technical level, but also to find out if there are common themes to anything I was working on that I could extract to make my life easier in future.

What I found shouldn’t surprise most people. Everything I did was about building forms, allowing data from those forms to be collected, and then looked at in some way. Every project looked like this:

• Define a form
• Build form
• Users input data into form
• Save data
• Look at data

Sometimes I would define/build the form and deliver the app, but at least two apps included the ability for users to build forms them (basically, survey software).

In fact, all web sites with UGC follow this model at some level – a blog engine is a form for a post, followed by the display of entered data in a particular way or set of ways. YouTube is a set of forms to handle file uploads followed by some data processing, and then displaying that content.

That’s quite interesting, because we have so far thought about web applications being custom little pieces of code in almost every case. Rails and similar frameworks enforce this model however, and make it easier by doing a lot of the heavy lifting involved in describing those processes. However, ultimately all we’re doing is putting lipstick onto relational databases. If that’s all we’re doing, surely there is a quicker way of doing it?

I was thinking that each of those stages in each application is already defined somehow as a real world business process we want to map into software. Why can’t we find a way of being able to quickly define a business process in code, and then just generate an application based on that definition?

It seems I’m a few years behind in my thinking. Business Process Management (BPM) is a mature area of development with a gazillion standards out there and it would seem XPDL is the XML-love of choice for this crowd. Digging further, I found something even cooler for the Rubyists amongst you: OpenWFEru nicknamed ‘Rufus’ amongst developers seems to have most of the heavy lifting done for you. The example looks promising.

This is interesting for me, because I have to build a couple of pieces of software soon which really are understanding workflows within teams, and I’d rather spend more time reading 19th-century books on political philosophy (hey, everybody’s got to have a hobby).

As a result I’m trying to work out how to hammer this into those projects. If I can find a way of being able to take the “I’ll look after the forms for you” aspect of Rails with the ease of defining workflows found in OpenWFEru and coupling them could you define a bunch of schemas and a workflow and then generate a better scaffolded Rails application? Would the controllers be able to mature over CRUD and do something more useful? Or could you build a CMS in Rails that uses user-generated workflows to allow for quickly hacked-together custom applications? It’s an intriguing bunch of thoughts…

On the 1st March (next Thursday, as I write this), I’m planning to do a mini re-launch of Vagueware.com and with a new website comes a new business model.

I will be phasing things in over the next 6-8 weeks. First, the “open innovation” phase, where ideas are thrown around, submitted and developed by anybody with an interest. Then the time to turn some of those ideas into real open source code, so anybody and everybody can develop the ideas further. And of course, run useful software. Finally a services model to make all your Rails deployment nightmares go away, and to bring some cashflow around those open source tools – support, maintenance, bespoke customisation will feature as well as hosted SaaS solutions.

However, I have a conundrum.

I’m busy finishing up work elsewhere right now. I have some todo items on older projects to close, and I am also involved in a second company completely unrelated to Vagueware that desperately needs some of my coding love right now. Time between now and next Thursday is tight. And the code base for the next version of Vagueware.com? Well, to be honest, barely started.

Right now, I need the splash page, and the ability for users to create accounts, edit wiki pages, for me to be able to structure some CMS around it all, add some forums, etc. It’s not a huge job, but to code it up from scratch even with the existing plugins out there might take longer than a week given other pulls on my time. It’s obvious that given the focus on it being a Ruby/Rails led set of projects, the time should be taken to develop the site itself in Rails.

There is an alternative though. Use something else other than Rails for now. Drupal does nearly everything I need and I could configure it all up this weekend ready to go. But it’s PHP. And that feels like a lie to me, and a lie to the other people I want to involve in this.

I could go meta and suggest that the first project is a tool to replace the site in Rails that would then allow Vagueware to become self-hosting. At least then the ideas would get started, and providing I was able to export everything from the inter-rim site when the job was done, nothing would be lost. It would allow those people who want to show up on day one to be involved in developing a community tool useful to them, and even have an input into how projects should be handled in the future. It could be an advantage.

It still feels like a major hack though. Anybody have any other ideas? I could delay the launch by another couple of weeks, but I’m keen to get started ASAP.

Reading an article on Ajaxian.com just now, I was struck by this screenshot:

Note, in the bottom left-hand corner, the keyboard shortcuts. This, to my mind is an example when keyboard shortcuts are just going to leave the user confused.

Don’t get me wrong – I love shortcuts. I rarely move my hands off they keyboard unless I have to, and one of the problems I had when I first moved to a Mac was learning and configuring things the way I like them. I don’t like using a mouse to repeat an action I think I should be able to do with a keyboard alone.

However, whilst it makes sense for my OS to have them, my browser (which I spend four or five hours a day staring at) to have them, and for my text editor (another three or four hours a day), a trivial app that I might visit once in a while just doesn’t need them. If a web application is something I’m going to spend hours per day or week looking at, such as Gmail or a calendar or an RSS reader, it might make sense. But for a survey tool site? I think they got confused somewhere along the line.