Innovation in Software

I’ve been thinking more about building windmills recently.

I know quite a few musicians. Some of them are mildly successful getting regular royalty cheques and making a living. Some run nights at clubs or bars that supplements their main income, steadily building their profile. Others are failing at making money and struggling to work out what to do to fix it. The difference between them is what they hear when somebody talks about “the music business”.

The successful ones hear the word “business”, the struggling ones the word “music”.

It’s the same in the software business. We grow up as developers, loving the experience of writing code and learning how to do incredible things with nothing more than our minds and a piece of commodity hardware with a free language compiler or interpreter installed. It’s liberating, it’s creative, it’s what we decide we want to do with our lives.

Except the software business is a business. It’s there to serve client needs, not the needs of software or artistic expression. Not enough people seem to grasp that – they seem stuck in the mindset that by producing code, they’re running a software business. It’s taken me several years of running my own company to finally see this for the lie that it is.

Imagine an architect firm where all the firm produced was building designs that nobody wanted or needed. Imagine a firm that only worked on concepts that nobody had asked for, or on receiving a client brief they threw it in the bin and just did what they felt was “better”. Right now, that’s what a lot of software firms are looking like: and like that fictional architect firm, they’re going to struggle to pay the bills.

Client need, user expectation, it’s all this industry is about. It’s the main thing we should think about. We’re not in a university lab playing with data structures and algorithms any more. In the same way architecture is about understanding a brief and imagining solutions rather than drawing, so software development is about listening and understanding more than it is about writing code. Sure drawing is important to an architect, and writing code is important to a developer, but they are an output of the real job not the job itself.

I’m not arguing there isn’t a place for understanding better code, or producing beautiful projects, merely that such activities either need to take place outside of an industrial setting or should at least not conflict with the business needs. It’s OK for an architect to produce beautiful buildings that are also functional, so it’s OK for a programmer to produce beautiful code that also meets client needs: it’s just that client needs should come first on both occasions.

And if you like the idea of development as akin to architecture, or you are struggling to see the parallels, you may want to read this article written nearly two years ago. We should look to the architecture industry way more than we do right now, and we should all use Ruby way more than we do.

I have a running battle with Yahoo! in terms of their “foreign markets policies”, but not with their core design and tech teams. I think then, the news that comes to me via Information Aesthetics that they’ve shut down their entire design innovation team is utter lunacy.

Yahoo! is struggling to keep up. Innovation and creativity is how you leap-frog and out-Google Google. It would seem they’re no longer that keen on doing that.

As such I have to say if you’re holding onto Yahoo! stock, consider selling: I think this is the first step to them going bust about four or five years from now. They’d best just hope that Microsoft still want them.

The Design Council is a body associated with ‘traditional’ design and innovation, but often some of their ideas and programmes are worth looking at here.

Designing Demand, for example, is a free workshop businesses can apply for. A team of independent experts in branding, product development, and design management is led by a Design Council associate. They are dispatched to a company to identify where design can be used to develop new products, services or branding. The experts spend time mentoring management to understand possibilities, and if the company wants to go about implementing the recommendations, the Design Council will point them in the direction of local qualified designers.

I’m wondering: what would a software design workshop look like? Would it be possible to sit down with a company and point out how their use of systems is inefficient or out-of-date? If they were developing bespoke software, is there value in talking about design first? Are we still so abstract from the real World that traditional design practices don’t offer any insight or innovation in this space, or can we take lessons from the design community? Is it possible to apply design principles in software to identify opportunities for innovation?

My gut instinct is there’s something here, but I’m not quite sure what.

Brian MCallister argues quite convincingly that “Popularity, in technology, is shit. Seriously.”

He has a point. When we aim to make something popular we are doing so for reasons of ego, and therefore attempt to compromise what it is we’re trying to achieve. We can’t do complex and useful to niché audience if we’re worried about being popular.

One version of this internal corruption of objectives is sometimes known as the “What would your mother think?” test in development. Would your (presumably technically illiterate, possibly senile) mother make of the gizmo you’ve just made? If the answer is “she wouldn’t understand it” then the trend is to simplify and to make things better.

But your Mum probably doesn’t care about your widget. What’s more useful is whether the people who are going to use it can. And that’s why, so the argument goes, that commons-based peer collaboration might be a better design practice than what we currently do.

It’s also why I think the future of innovation in software is going to be governed by companies making money whilst putting the source code out into the open. They quietly execute, iterating out improvements, making things better with each step, and then eventually the larger market catches on. The market catches on quicker if the source code is out there, resulting in better revenue streams.

Alas, we’re still in an age where the “intellectual property” myth still permeates our society, and trying to produce popular software seems more important than producing useful software. Sometimes it’s like the last decade was a dream…

Michael Kavis has posted up a short note on The Lost Art of Software Engineering and asks whether we have we lost the skills that once dominated the industry. Are we just looking after the business side of things, and is that wrong?

As somebody whose degree was Software Engineering – as opposed to ‘Computer Science’ that dominates the UK Universities – I feel that I can give a little insight into this.

First of all, I believe he’s talking about the wrong thing. Software Engineering is literally applying formal methods to the process of development in order to make sure that it always behaves in a predictable manner. The software can fail, but it should fail in a way you expect and have accounted for in terms of providing failover or re-instantiation (i.e. you can reboot without killing anybody). Typically, it is used for designing fly-by-wire systems on aircraft, management of nuclear power stations, systems that have a direct effect on health like life support or MRI scanners, or for embedded systems that might be difficult to maintain such as satellites or marine equipment.

Software engineering is not about transactions per second or response times unless those are business requirements. Usually software engineering is purely about applying formal methods to ensure the most critical business requirement of all is met: nobody’s death is caused by the system.

This leads me to where I think Michael has taken a wrong turn: the quantifiable measures he talks about in his post are not showing up in specification or tender documents because the business requirements don’t require them to. There might be a point that business requirements aren’t discussing system requirements like response time enough, but we are now at a point where we are dealing more with humans and their requirements than we are computers and their requirements.

This isn’t something we should be ashamed of. We now have the memory, CPU and tools to be able to build software that fits around our environment and we should embrace that. It’s actually quite hard to build a web application that doesn’t answer within a second, and if it does become an issue we then use the most useful tool in our box: re-factoring. Never optomise too early, scale when you need to. The business requirements in most cases is to get something shipping, and fast.

Over the last few years there has been an upswing in the interest around Agile and specifically design principles as they apply to software: as programmers we are beginning to understand our work at a philosophical level as much as we are a technical level. This is going to lead to developments that we couldn’t dream of a few years ago, and it’s because we’re freeing our mental capacity from having to worry about malloc() and free() every few minutes.

Where the skills we have as engineers come into play is making sure that the business requirements are understood and expressed in the simplest possible system. Instead of thinking about responses per second, we are free to concentrate on model validation. Instead of mathematically proving an algorithm will always work, we are building automated test frameworks. We actually spend more time engineering and less time hacking than we did just three or four years ago.

Software engineering isn’t a lost art, and neither is systems engineering and design. They might have evolved, but they’re far from lost: they’re just getting started, but now they’re much more hands-on.

Reading an article on Ajaxian.com just now, I was struck by this screenshot:

Note, in the bottom left-hand corner, the keyboard shortcuts. This, to my mind is an example when keyboard shortcuts are just going to leave the user confused.

Don’t get me wrong – I love shortcuts. I rarely move my hands off they keyboard unless I have to, and one of the problems I had when I first moved to a Mac was learning and configuring things the way I like them. I don’t like using a mouse to repeat an action I think I should be able to do with a keyboard alone.

However, whilst it makes sense for my OS to have them, my browser (which I spend four or five hours a day staring at) to have them, and for my text editor (another three or four hours a day), a trivial app that I might visit once in a while just doesn’t need them. If a web application is something I’m going to spend hours per day or week looking at, such as Gmail or a calendar or an RSS reader, it might make sense. But for a survey tool site? I think they got confused somewhere along the line.

For a long time, I’ve had a problem explaining exactly why software engineering doesn’t work like other kinds of engineering. One of the issues I’ve had is trying to find the wording needed to explain that programmers don’t really build anything, and that to manage building software like building a bridge makes no sense. I couldn’t quite put my finger on the right words though.

Until now.

Jack W. Reeve’s essay on ‘What is Software Design?’ is the best work I’ve seen to date on the subject. He argues – convincingly – that programmers are designers, architects, draftsmen at a drawing board. We don’t build anything: compilers, linkers, interpreters do that for us. He points out how flawed an idea it is to produce a design document that can never truly represent the details of how the software will work, because it is an abstraction of an abstraction. The only accurate design is the software source code itself.

What’s more, it makes sense that in the same way that an architect can build a model, which can be turned into a few more detailed drawings, which can then be turned into engineering drawings, it makes sense to start all software development with a set of mock-ups which are then turned into a UI (say, static HTML) which then become plugged together with code. It’s like building a skyscraper by defining what the bits ‘users’ will see will look like, then working out how the nuts and bolts will hold it together: exactly what an architect does.

His essay dates from a time when the best language available to programmers to express their design was C++, and he argues that C++ helps his model of ‘design as code’ along nicely. It was the best language for the job because as an object-orientated language, it allowed an expression that most closely resembled how we abstractly thought about objects and data in the real World. Since then, of course, things have moved along a little bit.

I would argue today that Ruby is the best design language we have available to us. I claim this, because I actually find it easier and quicker to express an abstract design – which in turn can then execute – through Ruby than I do through a diagram or notation or pseudo-code.

At this point you might be thinking “yeah, right, whatever!”, but this isn’t some idle boast, I eat my own dog food.

The other day I needed to clarify a design point with another programmer via e-mail. I needed to express a HABTM relationship being used as a way of listing ‘exceptions’ to a ‘everybody/nobody’ flag stored in a model. The details aren’t important, but I considered for a moment the very best way of communicating this. I considered drawing a UML diagram. I thought about a quick sketch with some boxes, I considered a little pseudo-code. I then remembered that the programmer I was discussing this with could code Ruby, and maybe if I just developed a few stubs with some comments he’d get the idea.

Five minutes later, I had not just produced stub code. I had some migrations that would build the tables needed in Rails, and I had a few methods I was proposing adding to a model. It was untested, and some DateTime handling code would need a little fleshing out, but I’d discovered it was just quicker to write the code than it would be to write the comments explaining what the code would do. I actually ended up writing a software design in my mail client, that I could copy and paste over into a code editor and with a little bit of expansion and testing, get running.

In other words, the perfect design notation seems to have been discovered – it’s not UML that generates stub C++ in a CASE tool, or some formal design notation. It’s Ruby that just, well, runs. It is perfect because it is the perfect balance of programming language and design notation.

So not only was Reeves right in saying that source code is the best detailed design possible, that we are designers at heart, that programming languages that work at an abstract level are a step in the right direction, that formal design tools are useless and pointless, but he was even right that eventually we’d produce a programming language that was just a compilable design notation.

We now have that tool and its name is Ruby.

UPDATE: Two separate comments – each without seeing the other, as I didn’t approve them until this morning – suggest that LISP is a better design tool than Ruby.

It is worth pointing out that Ruby is kind of a grandchild of Lisp because it inherits so much from Scheme, itself a dialect of Lisp.

So is Ruby just another Lisp dialect? No. Whilst borrowing aspects of Lisp, it is more closely related to Smalltalk in terms of its object-orientation. It also borrows from Python, Dylan and CLU. It uses the best of all of them – Lisp included – and makes something better.

Also, there is another important part of Ruby that Lisp (and Smalltalk to be fair) do not yet have: an easy way to rapidly produce production-level, deployable, useful software. I can build a web application in Ruby – thanks to Rails, or Camping – in relatively little time compared to 10 years ago in PHP or Perl. I suspect, although haven’t researched it fully, that to do the same in Lisp might take me a little longer.

Ruby has many of the great features of Lisp baked in, but many of its own bolted in to boot. What’s more, the design ethos of Ruby is its strongest point: Ruby is designed to make programming pleasurable. When you enjoy writing code, you write more of it in less time, and with fewer bugs. Lisp might be fun for those who enjoy it, but I agree to some extent with Larry Wall on this one: it “has the visual appeal of oatmeal with fingernail clippings mixed in”.

If you like Lisp, I suspect you will love Ruby, and you’ll find yourself more productive in it. Either way, I’ll concede that for some people in love with it, Lisp is as useful to them as a design language as Ruby is to me. We’re both lucky, we both know we are, and we’re both a step ahead of the C# and Java guys.

I still say Ruby rocks more though. :-P

UPDATE 2: Changed link to the PDF to point to one held at the official home and authorised version of these PDFs. Thanks to Daniel Read for pointing out my error in directing you all to a pirate copy.

Here we go again. A round-up of links for a Sunday spent in front of the PC, or if you’re a corporate slave, a way to pass Monday morning bunking off doing real work.

What is the Secret Behind Contagious Behaviour – This video from Stanford’s Always On summit has some fascinating discussion about contagious behaviour, implementing innovation and working out how to give up control of marketing and products to your customers. My favourite phrase from it probably has to be “fragile fires”. Mitchell Baker of Mozilla, Perry Klebhan – inventor of the modern snow shoe, and Gil Penchina of Wikia discuss how to get users doing the work for you. Moderated by Bob Sutton of Stanford and Diego Rodriguez of IDEO.

Collaboration doesn’t Work – if you’re afraid of the Kool Aid, this article from inc.com suggests teamwork and collaboration doesn’t work. I think the conclusions drawn here are all wrong (obviously) but for a simple reason: the author thinks that collaboration can be done by anybody, without training. It’s a skill that needs time to develop, much like the skill of being able to write software, write a PR release or do a cashflow forecast. Asking people to just walk in a room and start brain-storming without any prior training is asking them to behave in a business context using skills that to this point they had only learnt in social contexts, i.e. contexts where being polite is more important than being right.

Wikipedia Forks – It’s quite common within open source projects for groups within the project to reach disagreement and one set walks off with a copy of the project (which is legally OK for them to do), set up camp somewhere else and create a new project with the old code as a base. This is known as ‘forking’ and it happens a lot more than the media would have you believe. Now a group from Wikipedia believe it’s time to create a new project that has the good bits of Wikipedia but with the oversight of experts, and so off they march. Initially we as readers won’t see much difference, but the proof of the pudding will be in the long-term eating. I wish them well.

iPod users prefer CDs to iTunes – and who can blame them? The issue here of course is ‘DRM’ or ‘Digital Rights Management’. If you buy music through iTunes, Apple get to tell you which devices you can play that music on and how. If you buy a CD and rip it into iTunes you get a physical back-up, you can play the music where you want, and the ripped files can be copied to any device including ones not made by Apple. I genuinely think that within the next 3 years there will be a massive consumer revolt at DRM and the only way to deal with it will be to completely restructure the way music labels (and in the future, Hollywood studios) make their money.

8 Steps to better IT meetings – In fact, not just IT meetings, but any meeting. A nice round-up of how to keep meetings on track. Personally, I think meetings should be avoided completely – individual conversations work much better and if kept to the point can be a great deal quicker and more productive. Having everybody in the room really is as productive as having nobody in the room. However, if you insist on meetings, this is a great way to make them less brain-dead.

The 25 Worst web-sites in the World, ever – I remember most of these when they launched. Truly horrendous, and in my opinion the number one spot holder deserves it – utterly awful website (I won’t spoil the surprise for you though). That said, none of these were as bad as Microsoft Bob in the ‘I… don’t… believe… they… did this…’ stakes.

50 favourite design resources – for those of you who, like me, find design something that has to be worked on as a skill rather than something that comes naturally, here’s a crib sheet.