Who would have thought that one of the most innovative players in the hosting and web application industry would be a bookshop?

One of the big problems with Amazon’s web services is that they aren’t that great for permanently hosted web applications. There’s the dynamic IP addressing issue (which weoceo will look after if you have the cash) and the serious problem of how to store your database.

S3 is very nice, but it stores flat data, and certainly not anything as fancy as SQL tables. Until recently there was a hacky way to do it with a special storage engine for MySQL, but just looking at it made me nervous about my data.

Well, Amazon have decided to fix this issue. I received this email from them this morning.

“Dear AWS Developers,

This is a short note to let a subset of our most active developers know about an upcoming limited beta of our newest web service: Amazon SimpleDB, which is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database - real-time lookup and simple querying of structured data - without the operational complexity.

Were excited about this upcoming service and wanted to let you know about it as soon as possible. We anticipate beginning the limited beta in the next few weeks. In the meantime, you can read more about the service, and sign up to be notified when the limited beta program opens and a spot becomes available for you. To do so, simply click the “Sign Up For This Web Service” button on the web site below and we will record your contact information.

Learn more and sign up

Sincerely,

The Amazon Web Services Team”

So, there we have it. No more managing DB clusters. Scalable database tables, which once the beta is over will likely come with an SLA. Assuming that this just sits on top of S3, we might even be able to host our data inside the EU and get ll warm and fuzzy about protecting customer data properly.

I’m not sure this will be based on a standard set of DB libs but I expect we’ll see 1-line hacks to make it work with Rails, PHP and a host of other app frameworks within a few weeks.

I’m in.

Vagueware is not my only gig. My other Directorship is very low-key, doesn’t take much time, and is where I and my business partner experiment with various marketing revenue models. It’s primarily been a learning experience, and the frustration we’ve had over the last two years in getting various complex projects rolled out has meant we’ve been looking at partnering with technology companies and focusing on the marketing and customer communications side.

Our latest venture is quite a departure for us. Excuse me whilst I shill for a couple of paragraphs:

Whilst cash back websites are not brand new, we’re hoping that with a really solid technology platform underneath us, we’re going to be able to do something special in the way of helping people make shopping a little more fun - and save cash too. I do however, need some eyes and ears because I’ve had no control over technology roll-out, so I’m interested to hear of problems people might have.

As you can see, the concept is really simple. You sign up, we give you a fiver. You shop online, we give you a percentage of what you spend. You refer friends, we give them a fiver, and we give you a fiver as well once they qualify for payout. We’ll point you in the direction of freebies that pay you money as well. We’ll be launching a blog to highlight particularly good offers. Occasionally e-mails with super secret codes will land in your inbox and you will consider yourself a wise old bean for signing up with us. It makes things cheaper if you’re doing a lot of Christmas shopping online, although for various reasons we’re late to the party for that one, so our strategy is a little more long-term.

End of shill

I mention it here, because I’m interested in problems an educated audience (that’s you, dear reader), might see. We know for example that the back end systems are rock-solid and everything is nice and secure, but are there ‘quirks’ we’ve yet to spot that only a geek can spot? Maybe you just think the business model is odd, or we haven’t explained it very well. Either way, I wanted people whose opinions I respect to take a look before the big marketing push over the next 12 months, and see where we can make improvements.

When I started my business, I didn’t do things the way you’re “meant” to. I didn’t go out and get a nice office, or spend money on a brochure. To this day I don’t own a printer, and I send all my invoices electronically. For the first 18 months I worked on creaking hardware. My website was nothing more than a blog and I concentrated on just trying to get customers and pushing code out of the door - the shift into working for myself was big enough that it kept me busy without worrying about letterheads.

And at meetings, people politely laugh when I make a joke about “being too Web 2.0 and signed up to the digerati to bother with business cards”. Except I need business cards for all sorts of reasons these days. A year ago, I didn’t. Today, I do.

Last week a long-standing friend (and occasional colleague) launched Doddle, a printing service aimed at designers needing plain, simple, easy printing at low prices. It’s not true that I modeled for his logo - “Mr Doddle” - however I concede the resemblance is uncanny.

With a bit of prepped artwork - I needed help getting it into CMYK, because I don’t ‘do’ design packages - I went along, uploaded a zip file with the front and back graphics in there in TIFF, and filled in my billing and delivery details. It took about 5 minutes. That was Tuesday, and 10 minutes ago my new cards arrived.

I have to say I’m really pleased with the result. My cards are a little ‘unique’ in that they have a large block of text [1] on the back in quite small print, so I was worried if that would become a splodgy mess, but thanks to them being litho-printed, I’m pleased that it’s perfectly legible.

If anybody is looking for business cards, I’d challenge you to find a comparable quality at the price (especially for double-sided). I’m pleased therefore that recommendation of my friend’s business isn’t down to a form of nepotism, but because the product is actually worth the money.

[1] If you want to find out what exactly that block of text is, you’ll have to ask for a card. :-)

There are two kinds of innovation I want to talk about on vagueware:

• Ideas for whole new products and services that can be delivered with software
• Incremental changes to existing software products and services

I spend most of my time thinking about new products and services, the kind of thing that you can start a business off. Those ideas are generally jealously guarded by the people who think they thought of them first, but the simple truth is they have little value without execution: vagueware.com is about trying to get people executing on those ideas.

I have hundreds of ideas on my desk, on my wall, in my head, on my laptop, in notebooks, everywhere. They’re not going anywhere where they are. I do not have the time or the capital to make every single one of them happen. By placing them in the public domain over the coming months, I hope to do a couple of things:

• Somebody, somewhere will do something with them
• I will get the satisfaction that whilst not benefiting monetarily, I helped an entrepreneur and his customers

I hope that if you have an idea that you realise you’re never going to make happen, you’re going to have the courage to place it in the public domain and allow open source developers, start-ups and hobbyists in need of a way to spend their evening get started with it. They might even give you money, you never know.

Then there’s the second kind of idea - the incremental idea. The idea where you see a product or service out on the web or on your machine and you think “that’s great, but if it did…”

I’ve started cataloguing ideas I’ve had for vagueware and tagging them ‘vagueware’ - you can add ideas for vagueware too, and tag them so I see them - and as votes move up and down I’ll see what’s popular and what isn’t. I’ll use the tool itself to decide what to work on next within the tool. Yay for recursion!

I hope other developers use the site to do the same. By putting up ideas on the site for your own product and asking customers to go along and vote, you can get an assessment of what is going to fly and what isn’t.

Every page is editable, wiki-style, so your customers can improve your idea. Every idea has comments so you can have a little conversation around an individual idea.

By looking at a list of ideas in vote order, you can decide what is going to make customers happiest. By putting it on a 3rd-party site like vagueware, you get exposure to a whole bunch of people interested in people like you - innovative developers - who might not have heard of you anywhere else.

Or maybe you can just put an idea up and tag it with a publisher or product name in the hope that somebody at HQ will see it one day and act on it.

We sometimes take for granted the knowledge we have of how the Internet works. We know that an image in a search result might be linked to a site that has nothing to do with the image. We know that just because a reader of a blog comments on a post and links to a picture, it doesn’t mean the blog owner has endorsed or in any way taken ‘control’ of that picture.

We know this.

Some people though, aren’t quite as smart as us. They think that you have more control over how Google sees you than you do. They think that if you link to a picture you are ‘trying to take it over’. They don’t understand hypertext, they don’t understand indexing algorithms and they certainly don’t understand how this all applies in terms of copyright. Don’t believe me?

TechCrunch is currently dealing with perhaps the most technically inept man on Earth representing a photographer in an argument over online copyright and image distribution.

The problem is that he has a little knowledge - pictures can drive traffic, and that drives revenue - but not enough knowledge to understand what TechCrunch’s role is in this instance.

Even worse, he’s decided to act in a way I would consider unethical by phoning advertisers and threatening to name them in a lawsuit explaining he “just wanted to let [them] know”, in that I’m-doing-you-a-favour-don’t-look-at-me-like-I’m-a-leech kind of way.

This makes me come to the following conclusions:

1. If I ever need to hire a photographer, I’m never going to hire Beth Boldt as she clearly hires idiots to represent her legally (although he doesn’t appear to be a lawyer), and I really don’t want to deal with idiots working on her behalf
2. If you’re ever going to threaten to sue somebody, maybe you shouldn’t threaten Mike Arrington who is, you know, a lawyer, and knows what he’s doing… (top tip Mike learned at law school: use spell check before hitting ‘send’).
3. All of us have a responsibility to make sure the people acting on our behalf - personally, or within our companies - understand the issues as they really are.

If you’re working in a corporate environment in the UK, you should make sure at least some of your directors or somebody over at legal checks out Out-Law.com once in a while, and if you’re freelancing or a SME, its RSS feed should be part of your morning coffee ritual.

Nat Torkington writes about the effects data flows are having on him. I also find that once every 2-3 months I suddenly notice there are many sources of information I’m not really reading any more, and so I have a purge.

He makes a point about Facebook which I find interesting. With some customisation, it is possible to tailor what you see and what you don’t (yes, you can turn off the stuff you’re not interested in). As such, we’re able to tailor how we spend our time on Facebook to make it useful for us. That doesn’t mean we spend any more time on Facebook, it just makes that time more productive.

What if all applications were able to work like that? What if we took the “News Feed” concept of Facebook and dropped it into the front page of all our applications and then allowed the user to specify what they wanted to see more or less of, and allow them to tailor the productivity?

In the last 24 hours, I have probably written more code than I have in the previous week. It’s tested, solid, and ready to roll out. I now know how to get into this zone more easily in future. Even with Fresher’s Flu having an effect on my performance this last week, I’m surprised by my findings.

I’m typing this sat in a coffee shop overlooking Albert Square in Manchester. The sun is just about to dip below the buildings to the West, and the cobbles have that weird liquid gold colour they get in late Autumn evening light. It’s irresistible to sit here and stare and watch Mancunians go about their business. People all around me are talking, laughing, joking, arguing. A couple across the road just married at the Town Hall are having their photos taken, and the crowd is making a little noise.

Meanwhile I’m writing code (and this article), and it’s just struck me that right now, right here is when I am at my most productive.

I always thought my method was the exact opposite of this. Around a year ago, I wrote an article on “The Zone” and at the time observed:

‘Getting into the zone is not always easy. Different people have different techniques, nearly all of which involve a bit of peace and quiet and no interruptions.

…

Once you’re there, the worst thing that can happen is to be interrupted - I frequently shut down mail and IM clients when I want to make progress. When I used to work in shared offices, I would find a quiet corner. I shouted at people who interrupted me with trivial problems and put signs on my desk saying “Please don’t disturb unless building is on fire”.’

The key to my mind has always been to avoid interruptions. Peace. Quiet. That’s all you need, right? Well, not quite.

You see, I am not in a peaceful situation right now. It certainly isn’t very quiet. It’s actually quite a bit of a racket going on around me - coffee machines, frappes being mixed with ice, a dozen conversations, now I notice a kid is being a bit brattish in the corner. Yet here I am, working in perfect flow.

In the last year and a half I’ve mostly been working from home, and I’ve found it harder to get into the flow, the zone has been alluding me. My usual practice is to make sure absolutely nothing can disturb me and then try and work in complete silence. Except that’s really, really hard.

I think the important thing is, I need trivial noise in the background as that forces me to focus and try and block things out. Complete silence leaves a big empty noise in my head that needs to be filled with thought and it’s too easy to lose focus. A mild noise I need to work very slightly at blocking out forces me to really lock into the task in front of me. Hours could pass, and I wouldn’t notice, but for the cobbles dimming outside the window.

I still can’t get into zone thought if even one single phone call or IM message calls me away in the middle of something. Having a direct interruption is as dangerous as ever. So I’ve revised my thinking: “trivial distraction = good, direct interruption = bad”. I am starting to get into the habit of setting my status to “Away” when I’m working so I only get distracted by urgent/important issues.

It really has caught me by surprise though, just how solid the work I produce is when I’m sat in a coffee shop full of students. A year ago when I wrote the first article I would have laughed at anybody suggest anything like it. Now I know better, and will seek out coffee shops and places of mild, indirect distraction as often as possible.

Adrian Holovaty announced a while back templatemaker which sounds more complicated than it is.

Take a pile of web pages all built using the same template - say, restaurant listings, or a timetable - and throw it at the code. It produces a copy of the ‘template’ with the ‘data holes’ marked up. You then look at a new page, and it gives you the raw data. No more screen scraping, it just learns what it needs to find the data that changes each time. Nice.

A really useful aspect is that whilst he’s implemented it with Python bindings, the core is written in C and uses the Python C bindings. That means with a bit of hackery, porting it to Ruby should be quite simple. A project for the weekend, perhaps?

I’m shouting this up for two reasons: it’s a piece of code I’m sure all of us have thought about implementing, but never got around to producing because it just ‘felt’ too difficult, so it’s definitely very innovative from that perspective; secondly, it’s using the C bindings in an intelligent way to get performance where it counts - something I’m going to need to do more of over the next few months in other languages.

I suspect the future isn’t going to be about using the right language for a project, but the right language for the right part of the job. You could implement something in Erlang, which can interface with Ruby, which in turn can interface with C. There are points of complexity there that a competent software engineer would shudder over, I’m starting to feel that being competent in one language or framework is definitely not going to be enough for the next generation of software: we need the flexibility and power of scripting languages, and the raw power of compiled C/C++.

As a rule, I hate blog posts about blogging. I also particularly hate blogs that cite posts that in themselves are citations on other blog posts. It is then, with regretful hypocrisy that I find myself citing Joel Spolsky whose post in turn cites Dave Winer. Even worse, Joel is citing a blog post made nearly 8 months ago. That said, I think the point being made is of interest and might change the way we think about user-generated content over the next 12 months.

Joel’s main point is that comments are next to pointless. He doesn’t allow comments on his blog, neither does Dave Winer. Their argument is that as a means to self-expression, the author has a right not to be bombarded with people shouting them down and they shouldn’t have to see “their space” polluted with garbage.

The flaw with this idea is the idea that blogs are a means of self-expression. They are not. They are a way to find like-minded people and to provide a low barrier of entry to getting involved in w global discussion about the things you care about. They are not billboards, they are not magazines, they are not leader columns in newspapers. Nor are they journals or fascinating insights into your unique and tortured soul (you delicate little snowflake, you).

They are about a two-way dialogue, a means to advance ideas and to further understanding in the most cost-effective and uncensored way possible. I could produce a magazine about innovation in software but that would bankrupt me - instead, when I get time, I put articles up here as and when. I might not get tens of thousands of readers, but I certainly get hundreds (in fact, 2,167 unique visitors in the last 30 days have visited blog.vagueware.com which is not too shabby). And some of them - very occasionally - leave a comment here.

The purpose of all this is to encourage a conversation. Comments and trackbacks are typically an important part of inviting people and saying “Please! Come on in and tell me what you think!” and help move things along.

The real problem is that really interesting ideas are never in the comments. I don’t get the turf wars that spark off on other blogs because every comment is moderated - I don’t allow through spam, abuse, or one-liners and so the comments are worth reading normally, but normally nobody ever reads them. The comments are typically idle thoughts, occasional insights, sometimes intelligent but often just a throwaway quip. They intrigue me, but the comments are not what excites me about the conversation around here - it’s the people who link here.

When somebody sits down and decides to push the conversation further by spending 30 minutes writing a piece about something I’ve written, edited it, made a decision that they’d be happy for their name to be next to those words for all eternity (the Web never forgets), and then publishes it with a link back to here, the quality is generally higher than somebody who quickly fills in a little box at the bottom of the page.

I haven’t had many inbound links, but the ones I’ve had have always made me think more about what I do and why I do it than any comment ever has. There used to be a load of links when this blog lived at a different URL, fewer now, but in time I expect that’s where the interesting ideas will come from.

This line of thinking seems to be an emerging trend out there.

Recently, there has been a lot of love for Tumblr, a blogging platform without comment facilities. I think it’s a daring tactic, and one that will improve the quality of the content they carry. It’s also wise engineering with a direct impact on profitability. All of a sudden they don’t need to worry (quite as much) about spam. They don’t need to worry (quite as much) about authentication. They need less storage on their servers. The code is simpler. It even helps their authors. If I see something on a tumblr blog I have a choice:

• Do nothing
• Link to it in an article here that expands my own thoughts on the subject

Either way, the noise level within the conversation is minimal and providing I’m not an illiterate moron, the signal goes up. What’s more, the author of the tumblr blog doesn’t just get his own audience thinking about what he wrote, but perhaps some of mine as well. A few people might link to my piece so I get their audience, and so does the tumblr blog. This is how it’s meant to happen.

The author has no worries about spam, or about people cluttering up their space with thoughts that just don’t “get it”. They just get to go on linking to people and watching out for who links to them by way of Technorati or similar.

In short, comments aren’t a bad idea because you have a right to not be shouted down, they’re a bad idea because they slow the conversation down. They raise noise, deplete signal and keep the A-list static.

As such, I’ve decided that as of today comments are closed here. It makes my life easier, and hopefully yours too. If you’re a friend of mine on Facebook, you can make comments on the notes that get imported from here automatically if you want, but if you really have something to say about anything you read here, I’d ask you to post a link on your own blog. I’ll see it (I track every inbound link) and if I think it’s relevant I’ll update the article you point to. I may even write a new article pointing back at you. We both benefit.

If you don’t have a blog, I have to ask: what on earth are you waiting for? Tumblr awaits…

In a recent article on creativity and software, I made an argument that software developers need to become more involved in the creative process. One of the problems I discussed was that most programmers and developers have never been taught to think creatively, and many don’t know how to go about it. Whilst there are issues of where developers get brought into the brain-storming process, I thought it might be useful to look at some linear techniques that programmers might find useful given the way most coders think about problems.

There is a myth that creative sparks are something that “just happen” to the “right people”. Innovation is hard work, and there is nothing celestial or gifted about it. There are many methods of being able to get new ideas. Almost all of these have one common theme:

Disrupt your habitual thinking

We all have different perspectives on the World, but we rarely shift those perspectives internally. If you are a liberal, open-source loving, dope-smoking college student you are going to have a very different perspective to that held by a 60-year old CEO of a multinational. Most creative thinking exercises are about trying to shift you out of your skin and to see the World from somebody else’s perspective.

If you’re a programmer, you might be more comfortable stating this as “introducing pseudo-random thinking”. You need to shake up your habits, your processes, and get out of your own skin.

I’m going to go through some linear techniques here that might show you what I’m talking about. These techniques are great for trying to innovate when dealing with software problems (my core audience here), but can also be applied to writers trying to get rid of writer’s block or entrepreneurs trying to come up with new ideas for business plans.

As I go through these, you will see each one is about shaking up your ideas. Try all of them, try one of them, but just try.

Reverse your assumptions

Write down your challenge or problem, and then list all the assumptions you’ve made - e.g. “I must write this in Rails”, “I have one week to do this”, “I can not get any more resources”.

Challenge each of those assumptions by writing down a reversal, i.e. - “I can write this in any language that does the job well”, “I could delay the launch by a week”, “I could post it on an online contractor site”. Some of those opposite viewpoints will stand out a little. Create a list of them and ask yourself how to accomplish each one of those useful reversals, and decide if more research is needed.

This technique is perhaps the simplest, but in some cases the most effective. The trick is to try and embrace the exact opposite of your World view.

Drill down into your assumption

As above, write down all your assumptions, but this time write down all the attributes of that assumption. It could be you need to write it in Rails because that’s the only framework you know, or because that’s company policy. You might need to launch in a week because the PR department has already announced it.

For each assumption ask yourself “Why?” or “How else could I do this?”. Try and be flexible in your thinking, and for each of the assumption you make here, try and find an opposite as above, thereby cascading out all sorts of new assumptions.

This builds on reversals, and allows for more sophisticated thinking. It can feel simplistic at first, but with some assumptions you just need to keep asking “why?” like that annoying game children sometimes play.

Dinner for Six

You ever play that game where you choose six people dead or living as dinner guests, and compare with friends? Do the same again, but ask that table for ideas around your problem or challenge. They don’t have to be famous people - it could be college professors you admired, your family, anybody. Just so long as you get some contrasting voices in your head, it’ll start producing some chaos and randomness into your thought processes.

Some people like to go and do deep research into their dinner table - it could be the six most successful CEOs in your industry, or people who would be interested in your idea for example.

It’s possible with business people to go and find out how they did the things they did and work out their key motivations. Whenever I go down this route, I always end up with Peter Drucker and Seth Godin “in the room” because I admire their work and know their attitudes to certain parts of business so well from their books. This is actually re-enforcing learning: I might read a book by Godin, but this process causes me to apply it, and until I’ve done that, it’s not really learned behaviour.

Alternatively, when faced with a development problem, you might want to ask yourself “What would the 37signals guys do here?” or “What would happen if this were a problem in the Linux kernel?” and work from there. Ideally, choose projects you’re familiar with but which aren’t linked to the area you work in: you might respect 37signals web work, but what would they do with a device driver problem? Linus might know his kernel issues, but how would he develop a shopping cart?

Tug of War

Take a problem you’re trying to solve. Write down the best-case “win” and the worst-case “lose” for that problem. For example in the situation “produce a new blog”, the best-case might be “Technorati Top 100” and worst case scenario might be “Permanent writer’s block on day one, nobody reading”.

Write down all of the conditions needed to achieve each one, and you’ll notice a tug-of-war happening on the page in front of you. You’ll start to get a feel for what can happen and start seeing opportunities. Try and work out how to get from each worst-case scenario by putting down an opposite into the best-case scenario, and vice versa.

It’s amazing how this develops, simply because it’s so common for people to assume an idea is going to fail they find themselves producing a deluge of ‘worst case’ issues. By forcing a recast into ideas for producing the best scenario, new ideas just seem to pour out.

Binary Chopping

I’ve heard this called all sorts of things, and sometimes it uses more than just two words, but I like to start with two words to describe the problem I’m dealing with. For example: “Syndicated Content”

I then take each word and I split it into two more words. For “Syndicated” I might choose “subscription” and “contract”. For “Content” I might choose “articles” and “comments”. I can then drill each of those. Keep on going until you’ve got a nice big long list: “syndicated”, “subscription, “credits”, “annual”, “contract”, “services”, “fixed-term”, “content”, “articles”, “wiki”, “aggregator”, “comments”, “trackbacks”, “votes”.

Then, it’s time to look at each of those and combine them to see where you can get ideas:

• “fixed-term” + “votes”
• “credits” + “services”
• “wiki” + “comments”

When you have a list of 20+ words, the power of this can be pretty amazing, and ideas spring out. Don’t dismiss them, just work through them and see where it takes you.

Introduce Random Words

This can have variable results, but often produces something interesting.

Write down something your problem, challenge, assumption or something you’re familiar with: e.g. “Produce a successful social network site”.

Now, come up with a random word. If you’re doing it without the help of a dictionary or some other random method you’re going to be acting sub-consciously (almost like the binary chop), and so you might be surprised at how close a connection it could be. Alternatively you could just a random word generator like this one.

Then, try and force connections and see what happens. Evaluate your ideas, and if need be use one of the other techniques here to drill down or evaluate your new ideas.

“Produce a successful social network site” + “housed” could produce an idea for college students who are looking to share a house on or near campus. With “cynical” it could be a site where people gather as common enemies to somebody(!), or with “pope” you could find yourself thinking about a site focused on the Christian community and churches.

Summary

I hope that there is at least one exercise in there that’s new to you and that you can use. I’m interested in hearing about others out there I’ve not heard before.

In particular, what techniques do people use when stuck at different levels of a project, like deep down in algorithm issues? Are we just sheep dealing with patterns, or can these help us work out alternatives when designing classes and implementing methods?

It’s only Tuesday afternoon, and already this week I’ve learned a great deal about producing better software, albeit inadvertently - nothing teaches you more about code than having to fix stupid mistake. Each week I always learn some new insight but this week it’s been like a waterfall of learning the hard way. I thought I’d moan at you lot, because it’s more fun than whining about it all inside my own head. :-)

• Users are lazier than you think - to me, it’s pretty obvious that if I’ve already entered data into a system, I’m not going to enter it all over again, but this time full of typos and changing reference numbers around. I’m just going to enter the data in correctly first time, and if I have a problem, I’ll go back and edit it. Not so users of one system I’ve been working on this week. Cue an afternoon of going through data and checking it for consistency errors
• Users don’t read the screen - we noticed a load of weird consistencies elsewhere in the same system. It turns out that I’d upgraded ferret on the system, and then the ferret index on the app was obviously foo-barred (note: when doing a gem update on ferret you MUST either clear out the index dir in your Rails app or you must script/console and Model.rebuildindex on each Model you’ve got actsas_ferret rolling with - you don’t have a choice in the matter). Strangely, in the period the app was doing this, people must have been putting data in and getting the “Application error” screen. Did they stop? Not likely - they just carried on, hoping it wouldn’t matter. Cue another half an afternoon of checking the database for errors.
• Logs are more useful than you think - Rails logs all parameters it gets, even in production mode. Some people think this is a security risk, and for password forms you should definitely turn it off (look up filterparameterlogging to see how). However, when you have a load of weird consistency errors in your DB, my god do those logs help reconstruct what was going on and help you fix what happened
• script/console doesn’t do what it says it just did - try this: take a model with a datetime column. In script/console, try and assign a new date to a record and save it. Note how that didn’t work. I’m not quite at the bottom of why, but it’s curious that it doesn’t “just work” - it may be a config issue, but I can’t see what right now.
• Nobody cares about my problems, they’ve got their own - I don’t even know I bother bitching about this stuff: I just find myself shouting into a big empty echo chamber that is the Web. :-)

Anyway, I’m at a careers fair tomorrow helping out MDDA up in Cheetham Hill. I hope to return with all my limbs intact.

After that, I’ve got tickets for all five days of the cricket at Old Trafford, so I’ll be pretty much in holiday mode after tomorrow.

One of the most fascinating parts of looking at web server log files, has always been what search terms people type in to discover content on the site. There are few things that get you closer to staring into the eyeball of all humanity than seeing what they ask Google for.

I use the usual tools, but in recent weeks I’ve been trying out 103bees which, as well as providing all sorts of things similar to Google Analytics, throws you an e-mail every week with the search queries people have been using.

It took me a while to remember why it might be relevant to this site for somebody to type in “organised pigeon hunting”. It would appear I am not alone in finding some of my search traffic quite odd.

I remember once having a boss that had what seemed an endless supply of energy. He was at his desk from 8am and normally didn’t stop until 2am the next morning. His energy was seemingly infinite and it was rare to see him flag. At the time I thought that was amazing. It wasn’t until after I’d left the company I learned he was a regular user of cocaine. Ah. Yes. Well…

As it is, in recent days and weeks I have discovered my own way of being able to get an endless stream of energy, all without illegal narcotics. I have been working 9am until 3am - occasionally 10am until 4am - for the last couple of weeks, and feel fine. I’m not tired, I’m not bored, I’m getting real stuff done, and I figured I’d share how I got here for those who need a hand getting focused for a year-end coding rush:

• Variety is the spice of life. Right now I have 5 open projects which means when I start getting tired of one, I can move to another. It’s all Ruby, true, but they’re different problems needing different approaches. This helps break the day up.
• 50 minutes work, 10 minutes break. Every hour. Without fail. Get up, walk around, have a drink, listen to music, stretch, whatever. Don’t sit there all day.
• Don’t read RSS, Mail, Usenet, IRC, whatever, unless you really have to. You have no idea how easy it is to get distracted.
• Time slowly becomes irrelevant, and that has interesting effects. I now look at a clock reading ‘00:12’ and instead of thinking “gosh, it’s late!” I think “Classic FM get less adverts now, I’ll put that on for the next few hours”. I can’t quite remember how many days ago it was that I listened to the opening session of the Ashes, nor am I quite sure what day it is today without looking at the top right-hand corner of my screen, and I don’t remember the last weekend I had. This is an acquired taste, and would scare most people.
• Coffee is my new friend. This is not an excuse for drinking it constantly though, I find if I drink more than four or five cups in a 24-hour period, I can’t concentrate.
• Taking a break for an hour is not cheating. This morning I went for a walk for an hour. I enjoyed looking up and seeing sky and clouds, which is odd but necessary.
• No alcohol. I used to drink quite heavily, but about a month ago I read a book on alcohol which left me not wanting to touch it ever again. The last month has been the most productive of my life, which I could have predicted: I knew that if I had even one pint, it would change my ability to concentrate with the level of intensity that is optimal for me for perhaps 24-36 hours afterwards.
• Lists. There is more power in ticking off an item on a list than you can possibly imagine. Looking back at a list of things that have been crossed out makes you feel as though it was worth it.
• Let your body do what it wants. The other night I went to get ‘a quick hour’ of rest around 8pm with the intention of working through until about 5-6am that night. I woke up the next day at 9am. Clearly my body needed the sleep. It would be unwise in the extreme to ignore it.
• Also it is important to realise this is only a temporary measure. If I tried doing this for a year, it would kill me. I’m doing it for a month to get a load of work out of the way, and then from the 15th December to the 15th January, you will do well to find me near a line of code or a phone

Momentum is definitely the best productivity tool I know of. Years ago I read a book on procrastination that suggested you promise yourself to “just do half an hour”, and then you would just find yourself carrying on - that is basically what I did, except I did it weeks ago and haven’t let the momentum slide yet. When you see pieces dropping into place, and you ignore the boundaries of time it is easy to say to yourself “oh, I can do just one more bit here and it’ll be loads better”.

I can’t stress how important it is that this is something I can do for a month, and then need to take a month off. If people start thinking “that’s insane”, they are right, and I wouldn’t advocate it for somebody who doesn’t know what they’re getting into. Nor would I suggest it to anybody who isn’t single. It’s also a false economy in terms of the long game because whilst I might be able to get over 100 hours a week in right now, I won’t get 5 hours a week done for the next six weeks. But that’s not the game I’m playing this month.

You might wonder what game it is playing, and why I’m prepared to put myself through this. Well, my company has its first year end at the end of December, and I want to put everything outstanding to bed by the end of November giving me a month to tie up my accounts and outstanding invoices. Two projects have ‘hard’ deadlines that can’t slip. One is for a friend who wants a site for her business. The other two are important for other reasons.

I would never do this if it wasn’t my own company, and I wouldn’t expect anybody else to do it either even for their own company - I accept the madness this entails. Hard work is always part of running your own business, but like I say, it’s a temporary measure. If your mind is moving towards New Year resolutions this early and you’re thinking about setting up your own business, ask yourself if this kind of work blitz is something you feel you could do once in a while whilst still enjoying it and if the answer is not an unequivocal “yes”, perhaps you should just stay on your comfy salary and stick to 9-5.

Reading an article on Ajaxian.com just now, I was struck by this screenshot:

Note, in the bottom left-hand corner, the keyboard shortcuts. This, to my mind is an example when keyboard shortcuts are just going to leave the user confused.

Don’t get me wrong - I love shortcuts. I rarely move my hands off they keyboard unless I have to, and one of the problems I had when I first moved to a Mac was learning and configuring things the way I like them. I don’t like using a mouse to repeat an action I think I should be able to do with a keyboard alone.

However, whilst it makes sense for my OS to have them, my browser (which I spend four or five hours a day staring at) to have them, and for my text editor (another three or four hours a day), a trivial app that I might visit once in a while just doesn’t need them. If a web application is something I’m going to spend hours per day or week looking at, such as Gmail or a calendar or an RSS reader, it might make sense. But for a survey tool site? I think they got confused somewhere along the line.

I picked up an article doing the rounds by Joel Spolsky on context switching in Agile development and find myself a bit confused by the way he pitches his conclusion, even though I agree with most of what he says.

Context switching is something programmers first learn about in University purely in terms of what happens when a CPU or operating system has to stop doing what it’s doing, and do something else for a bit. It happens thousands of times a second on a modern computer, and is happening on your machine right now if you have more than one process running. Getting processors to the point where they can do that effortlessly took decades.

The next thing programmers learn about context switching is that they are expected to behave just as well as a modern CPU at context switching when they are expected to juggle dozens of projects at once. Interestingly, the people who expect this, don’t understand what is required to handle a context switch for a programmer and see their new piece of meat as being a bit like a messy version of OS/2 Warp, but perhaps not quite as fast.

In the past I’ve written about what I call alone time - the point where a programmer will try and get into “The Zone” and become as productive as they possibly can. I think those two articles hint at why a context switch is so hard. It disrupts a work-flow that is hard to obtain. You can’t easily drift in and out of a context when you’re being productive.

Joel agrees with all of this, but says “hey, you know, sales are important, sometimes it’s worth the context switch, that’s what Agile development is all about”. The problem is, he’s talking complete rubbish.

Agile is about how to manage the development of a single project: one project on the radar, iterations, feedback, customer contact, the code is the specification. You get the idea.

You don’t use Agile to handle multiple projects. For that you need something completely different. The context switch is something that agile isn’t designed to handle, because it assumes that as a development manager you wouldn’t be so utterly irresponsible, moronic and boneheaded to attempt to try and get a developer handling several projects at once. It’s a limitation of Agile, which is why you need something else in most programming teams to handle it.

Right now I have seven projects on the board, two of them able to wait a while, three of them less than 24 hours from being signed off, and the remaining two due to last through for another week or two - I then intend to take a month off Christmas and sleep. Right now, I need to context switch even though I know it’s not ideal. The way I do that has nothing to do with Agile development, and everything to do with how I plan my time.

In the example Joel points to, the answer is not “this is Agile, deal with it, be Agile”, the answer is to say to the developer “look, do you want some overtime to look at this?” or to find a developer who isn’t busy to look at it instead. It’s about time management and handling more than one project, not about forcing a method designed for one project into something it isn’t.

Juggling multiple projects, handling resource allocation and managing the board in a busy programming team is hard work. It isn’t something Agile can fix, so don’t blame it when it doesn’t, and let programmers stay in context as long as possible if you want them to be productive.

I’m just about coping by handling this my own way, but in the New Year I’m hoping to hire a developer to help me out and let me stay in context for longer.