Innovation in Software

This afternoon, I’ve been playing around with Facebook’s ad platform. Partly for Vagueware, partly for other businesses, I’ve been looking at what Facebook says about its user base to advertisers.

The level of targeting is just outright astonishing. It allows for ads not only to be targeted on demographics such as age range and city, but even on interests and relationship status.

For example, I now know there are approximately (all figures given are approximate to the nearest 20 or so), 120 people in Manchester interested in Programming.

Out of the 2,017,440 UK citizens who describe themselves on Facebook as ‘single’, 998,900 are male, 904,960 are female. The numbers don’t add up because some people don’t define a gender which makes the point that if you don’t fill info in, you can’t be targeted via that info.

There are 1,180 females in the UK who declare an interest in ‘Computers’. The figure for males is around 8,540.

580 UK men say they’re really into shoes, with 14,300 British women aspiring to be Imelda Marcos.

There are around 5,680 people working for BT in the UK on Facebook. In the US, there are around 40 people working for O’Reilly Media who mention it in their profile. I could target either with an advert – handy if you have a product or idea you want to pitch.

This is just the tip of the iceberg. Advertisers don’t need to know who your friends are (the social graph), to target you this tightly. If a member of GeekUp wanted to put up a singles ad for all single women between the ages of 24 and 32 who are into computers resident in Manchester (approx. 100 of them), they now theoretically could. Lucky ladies.

The question is, is this really a bad thing? Doesn’t it mean we’re not all going to see advertising that really has no relevance to us? Or does this kind of marketing mean that we are the perfect willing victims for advertisers to go deep into our psyche? I knew this day was coming, but I thought it was still some way off.

A friend just forwarded to me an article called “The Outsourced Brain” over at the New York Times. A sample:

” Until that moment, I had thought that the magic of the information age was that it allowed us to know more, but then I realized the magic of the information age is that it allows us to know less. It provides us with external cognitive servants — silicon memory systems, collaborative online filters, consumer preference algorithms and networked knowledge. We can burden these servants and liberate ourselves.

Musical taste? I have externalized it. Now I just log on to iTunes and it tells me what I like.”

This is going to gradually become a debate over the next few years as we pass more and more of our thinking and life over to algorithms. Stroustrup once said “Software runs civilisation”. I think we’re approaching the point where we can say “Software runs civilians”. There are obviously issues with this that need to be explored.

About a year ago I developed a hypothesis of what humanity would broadly look like 100 years from now. Some friends found my synopsis of this vision a little ridiculous: “You know the borg in Star Trek? That’ll be us”.

What I mean is that we are slowly moving our thinking out into the cloud and acting as one. Individualism is being lost, group-think is being encouraged. If that sounds a little Orwellian, can I just point out that we’re the ones encouraging it on ourselves – from CCTV cameras to collaborative filtering on Amazon – it is not being imposed on us.

The irony is that for all the menace of Borg assimilation and Orwellian dystopia in fiction, we are shaping parts of our society into something that mimics it in the hope it will lead to peace and harmony within society. Maybe it will, I don’t know.

The hope we have is that if we spend less time thinking about what music to buy, which directions to use to get somewhere, and trying to remember things we can get out of Google and Wikipedia anyway, we’ll have time for more important things. The question is what things are we doing with that time? Are we just filling that brain capacity with other trivia we don’t need?

Not for the first time, I feel that those of us styled “Software Engineers” have a responsibility to ask some questions here.

Tim O’Reilly is calling for Facebook to share social graph data so that systems can leverage all the data you’ve shoved into Facebook and use it within their own apps.

There are a couple of issues here.

Firstly, Facebook isn’t actually stopping 3rd-party API developers from knowing who your friends are, and if your friends agree to add an application, the app provider can see their graph too. What isn’t agreed yet is whether this should be made more open, or whether there needs to be a standard way of describing this data. There are all sorts of reasons why I might not want my “social graph” to be made available in an easily-manageable format, not least because it raises privacy concerns.

There is also the fact that Facebook’s business model relies on not making this data available. The “expose your data, and they will come” argument relies on a simple metric of conversion.

Within a company like Amazon, exposing the product catalogue by API is a no-brainer. The more places their stock list is available, the more chances they have of getting somebody into the system, the more likely they are to convert them into a sale. The porous membrane an API gives an app developer in this instance means 3rd-prty developers do the hard work of getting stock shifted in countless innovative ways the original company wouldn’t have thought of.

Facebook however, is different. The ‘conversion’ in their instance is getting somebody to look at pages with adverts on it. What they need is for their users to actively recruit more users – invite them inside the walled garden – and then try and keep them there. They’ve out-sourced the “retaining” part of the equation to developers (playing games, taking quizzes, sharing links, glorified e-mail), but by allowing their most valuable asset to be easily exported they are reducing their customer’s incentive to stay within the walled garden.

As always, it comes down to whether you have a right to that data, and whether you have a right to move it. I’d argue you do, but I’m suggesting it’s going to be hard for Facebook to allow you to take it wherever you want.

[UPDATE]: I realised there is a way to do this without Facebook’s permission. I’ve written it up on the site.

The most interesting observation from the last few months of Facebook usage for me, is how people manage their online identities: they don’t.

You can have multiple profiles and link them together in Facebook if you want. To my knowledge, nobody I know does this – everything they do just piles into one profile. In fact, I’m not even sure people know you can do this. Being completely open with one profile would traditionally be considered a high-risk strategy when mixing business contacts and personal contacts.

We’ve all heard stories about the graduate applying for a job at a top firm and being the perfect picture of potential leadership until somebody finds their MySpace page, complete with pictures of drug use, snippets from their starring role in a ‘Girls Gone Wild’ video and profanity-filled exchanges with friends. We are told that “They” don’t like this – “we” should be wary.

I think paranoia is more to blame for these warnings than reality.

We are no longer living in a parody of the 1950s. I don’t think that society was ever really as rigid as the films and documentaries make it look – where everybody calls each other by their surname and a tipple too many after dinner left you a social outcast, but even if it was real we’re leaving that mode of thinking about each other and quickly moving into an era where authenticity matters.

As a potential employer, I would rather know a new manager has an “interesting past”. It would help me understand his/her character more than pretending they were conceived a perfect model of professionalism.

I don’t know if the rest of the World is going to see things the way I do, but I know that Facebook and social network apps like it are making more people face up to the reality of dealing with people as they are, not how we’d like them to be.

Right now, I’m quite an angry man. I’ve just read The GIFT of Giving at Yodel Anecdotal, where Michael Samway gives us an account of his trip to the State Department, to talk about Yahoo! is really ‘doing the right thing’. The only comment that he makes about the huge problem human rights protestors have with Yahoo! is this:

The tense moment on the first panel arrived when an Amnesty International representative opened his remarks by directly accusing Yahoo! and the other companies of cooperating with repressive regimes, including handing over information on political dissidents and limiting the free flow of information.

That’s undercutting it a bit. For those of you unaware of the story, you may want to catch up on some background reading

I’m a supporter of Amnesty International. I believe we need to be responsible with the data that we hold as companies working across international borders. Michael goes on to defend Yahoo! by stating that in the second panel:

… we each raised some of the vexing questions we all wrestle with in the field of business and human rights. Partly in response to comments from the first panel, I explained that we condemn the punishment of any activity internationally recognized as free expression and that the relationship between law enforcement entities and technology companies around the world is more complex than commonly understood. Rarely, if ever, will a company know the name, identity, or occupation of an individual connected to a user ID demanded by a law enforcement agency, whether in Munich, Mexico City, or Mumbai. What we do know is we protect user privacy through rigorous compliance practices and careful adherence to law governing government demands for user information.

Vexing questions, eh? The only ‘vex’ is “Do we sacrifice human rights and work with repressive regiemes in order to gain market share?” to which Yahoo!, Microsoft and Google all answer a resounding “YES PLEASE!”. As for not knowing all the details, Yahoo! know as well as I do that you don’t need all the details. An IP address is often more than enough to track down a user if you need to – I know this, because when I worked for a major multinational ISP, it was one of my jobs to work with the Police in order to catch paedophiles, and all I needed to help them get their man was an IP address and a date/time.

Yahoo’s atttitude is not atypical of the industry wanting to break into China right now, but it is typical of the flagrant disregard that they have for human beings, and certain values we consider in a civilised society to be universally true. They claim that they take a stand, but the moment that stand looks like China will lock them out of the market, they buckle. Their reasoning?

…the presence of companies like Yahoo! in markets abroad can have a transformative effect on peoples’ lives and on local and national economies. Information is power. Access to information, especially through the Internet, has changed what people know about the world around them and about events, people, and issues that directly impact their lives day-to-day. People know more about local public health issues, environmental causes, politics, consumer choices, and job opportunities. They communicate and interact like never before with family, friends, neighbors, and people locally, regionally, and even globally with similar interests. And the Internet drives innovation across sectors, including in science, medicine, business, and journalism to name a few.

That would be true, but for one simple fact. The information Yahoo shows to their Chinese audience is regulated in its entirety by the Chinese government, and the moment there is something truly powerful on the net – say, a website suggesting democracy would be a better way to run the World’s most populous nation – Yahoo! calmly assist the Chinese government in making sure nobody sees it, and if the poster of the content was using a Yahoo! mail account, well here, have the IP address they last used when logging in to check their mail, our pleasure.

What really sickens me about this, is Yahoo! just refuses to accept they’re doing anything morally objectionable. They just sit around humming and hawwing and making noises of “difficult questions… it’s a tough one to call… lot of factors to consider…” without once stopping, and thinking to themselves “We’re responsible for people being persecuted, jailed and possibly tortured”, and doing the one thing that any civilised human would do: get the hell out of Dodge.

I doubt my comment on that post making a direct attack will ever get approved, so I’ll just post it here:

I’ll bet a large wedge of cash that this never gets approved, but if somebody at Yahoo! reads this and tries to change internal policy, that would be “nice”

I’m glad you had a nice day out, and that you think Yahoo! is doing something important in helping people change their view of the World.

It doesn’t, however, change the fact that Yahoo! are responsible for handing over information on several dozen democratic reformers in China, who are now rotting to death in jail.

The line “we were just complying with a legitimate governmental request” doesn’t cut it – you guys know you were in the wrong to do it, but you don’t care about doing wrong as long as you are able to keep, and grow, market share.

Yahoo! in China is no different to IBM in Nazi Germany – “we’re not involved, we’re just doing business, our shareholders expect it of us” – but history will judge that Yahoo! were involved in a disgusting chapter of Chinese history and didn’t do a thing to get in the way if it meant it would hurt the bottom line.

I hope you enjoy more cups of coffee with important people, but if you want to make a difference, you have to club together with the rest of civilised society and make a stand that you’re not going to hand over data on people who just want to be able to vote – and take the consequences of loosing market share, or being thrown out of the market. Your shareholders will have more belief in you for doing the right thing, then they will for you doing anything to make a buck.

Until then, I, and many others like me, refuse to use Yahoo! services of any form unless I have no choice. As people hear what it is you guys have done in the past, our numbers will grow. I only hope one day you will realise that helping the Chinese government find and torture democratic ‘dissidents’ hurts your share price more than not being in China.

As ever, opinion and thoughts invited in the comments.