I’ve been working with Leila Johnston on a new thing. It’s a fortnightly podcast called Shift Run Stop and as she explains it’s “an ambient soundscape sort of production, an undulation of chatter and noise, ideas, games and food”. Editing it is a lot of fun, as are the weekly recording sessions.

It lives at shiftrunstop.co.uk and in iTunes for your subscribing pleasure. Hope you enjoy it as much as we do.

Duncan Campbell’s piece in ‘Comment is free’ points out the very patchy reporting that the death of Ian Tomlinson has had thus far in much of Britain’s press. I think that’s about to change.

Please take a couple of minutes to watch this video and read this report. The video shows what seems to be Ian Tomlinson, the man who died during the G20 protests on April 1st, being pushed from behind by a police officer dressed in riot gear, including a balaclava. Unlike the others in the group, the officer who pushed Mr Tomlinson to the floor doesn’t seem to be wearing his identification epaulets, or ‘collar numbers‘, on his shoulders.

I’d already left the area by this point, but were you there? Did you see what happened? The Independent Police Complaints Commission is still gathering evidence in this case.

The investigation is continuing to look through CCTV footage to see whether the incident inside Royal Exchange Passage has been captured and we already have a number of witness accounts from the area. However, I would ask anyone else who saw Mr Tomlinson at about 7.20 p.m. or who may have taken a photo of him around that time to contact us so that we can build up a full picture of what happened.

Anybody who saw Mr Tomlinson in Royal Exchange Square is asked to contact the IPCC on 0800-096 9071 or email Tomlinson@ipcc.gov.uk

And who took this important video? Not the police (though they were certainly filming the protesters). Not a protestor. Not even a legal observer (though I saw at least one of those in the crowd). No, the person who filmed this was an independent bystander. He was apparently a fund manager from New York, working in London.

We all need to be observers. It seems to me that we need more people taking photos of the police during events like this, not to target them with yet more anti-terror legislation such as the Counter Terrorism Act 2008. (See also this article outlining the new legislation too. Especially the comments).

We need to be able to hold to police accountable, especially in tense and difficult situations like a protest. Imagine how much more difficult the IPCC’s job would be now if it were not for all those cameras in the hands of the public. If the police start using section 76 to prevent people from filming them in public, that’s a protest I’ll be attending as participant, not just an observer.

With that in mind, I installed MeeTimer over the weekend. It’s a Firefox plugin which…

records where you spend your time online. It does it in a rather useful way, by allowing you to group websites into activities … so you can make sense of where your time is going. Finally, it accumulates time spent on a site over the course of a day…

I’ve been using it for 3 days and it’s giving some interesting food for thought.

You can even optionally set up ‘tab warnings’ on specific groups (sites you’ve labeled ‘Procrastination’, say) which will pop up with a nice overlay telling you exactly how much time you’ve wasted in this site, and others in the same category (though allows you to click through and ignore the warning just this once or for the current browsing session if you still want to). I’m already finding this feature useful on the handful sites whose feed I’m subscribed to but for some reason still find myself visiting out of habit. (For me, it’s Waxy links and Boing Boing. I love them, but I’d rather be reminded to enjoy them as part of my feed reading routine rather than browsing out of habit. I bet you have your own which make you ask is this really what you want to be doing right now?). A little reminder is really useful for habit-breaking here.

Mostly MeeTimer is just quietly keeping track of a bunch of per-site accumulators, cleverly based on whether Firefox has focus and which is the currently active tab. The results are already interesting. I realised that I was spending a bit less time on Twitter and Flickr, and a bit more time on work webmail, than I thought.

This is all very well, but I want more. Specifically, I wanted to get at the data. Not just the accumulated weekly/daily/monthly (etc) totals and averages, but the number of visits to each site per day. The raw visits. In as much detail as possible. I want CSV exports, or an API, or something. If I’m spending a daily average of 21 minutes on Twitter, how many visits comprise that time? MeeTimer simply doesn’t tell me.

Or does it?

Digging around my Firefox profile, I find a very interesting file at /Library/Application Support/Firefox/Profiles/{profile-id}/meetimer.sqlite. Ooh, I bet I know what that is. So I open up SQLite and start poking.

Sorry. It’s about to get a bit dull from here on in. Unless you get excited about the idea of being able to manipulate this data you’ll probably want to scroll down to the end. Honestly, I won’t mind.

They’ve gone? Right. Let’s get hacking.

$ sqlite
SQLite version 3.6.12
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .restore meetimer.sqlite
sqlite> .tables
deterrent_stats  groups           log
deterrentlinks   groups_urls      url_maps
deterrents       ignored_urls     urls

Excellent. We’ve got tables with sensible names and everything. Let’s see what log looks like.

sqlite> .headers on
sqlite> select * from log limit 3;
url_id|startdate|duration|day|week
4|1238324612508|3|200987|200913
5|1238324617244|44|200987|200913
6|1238324647668|17|200987|200913
sqlite> select * from urls limit 3;
id|url
1|mail.google.com
2|www.google.com
3|www.google.co.uk

Lovely. Easy enough then. The groups and groups_urls tables do what you’d expect too. For now, let’s make url_id more meaningful by doing a join with the url table.

sqlite> select
  url_id, duration, day, week, url
from log
left join urls on log.url_id=urls.id
limit 5;
url_id|duration|day|week|url
4|3|200987|200913|google.co.uk
8|40|200987|200913|meetimer.productivefirefox.com
4|16|200987|200913|google.co.uk
11|10|200987|200913|technorati.com
12|14|200987|200913|google.com/reader/

What if we wanted to show the number of visits, the total duration, and the maximum length of duration for visits to Twitter…

sqlite> select
  count(url_id), sum(duration), max(duration), url
from log
left join urls on log.url_id=urls.id
where url = 'twitter.com';
count(url_id)|sum(duration)|max(duration)|url
34|2712|455|twitter.com

Excellent. I wonder what the top seven URLs when ordered by the number of visits?

sqlite> select
  url_id, count(url_id), sum(duration), max(duration), day, week, url
from log
left join urls on log.url_id=urls.id
group by url
order by count(url_id) desc
limit 7;
url_id|count(url_id)|sum(duration)|max(duration)|day|week|url
9|34|2712|455|200989|200914|twitter.com
10|30|1075|249|200989|200914|search.twitter.com
1|22|2505|928|200989|200914|mail.google.com
4|20|206|57|200989|200914|google.co.uk
17|18|476|114|200989|200914|flickr.com
21|10|2480|2125|200989|200914|bbc.co.uk
39|8|13152|10212|200989|200914|webmail.bbc.co.uk

Twitter, with 34 visits. Sheesh. And for comparison, the top 7 sites by total duration of visit?

sqlite> select
  url_id, count(url_id), sum(duration), max(duration), day, week, url
from log
left join urls on log.url_id=urls.id
group by url
order by sum(duration) desc
limit 5;
url_id|count(url_id)|sum(duration)|max(duration)|day|week|url
39|8|13152|10212|200989|200914|webmail.bbc.co.uk
9|34|2712|455|200989|200914|twitter.com
1|22|2505|928|200989|200914|mail.google.com
21|10|2480|2125|200989|200914|bbc.co.uk
12|6|1355|633|200989|200914|google.com/reader/

13152 seconds (3.6 hours) on my work webmail between Sunday morning and Wednesday aftenoon. And all done in 8 visits. Yuck.

Ok. Let’s start thinking about daily summaries. Grouping by day, and then by URL (since I’m not very good at SQL, and don’t know how to limit it to 5 per day, I’ll just manually snip out all but the top 5 for each day for now)…

sqlite> select
  url_id, count(url_id), sum(duration), max(duration), day, url from log
left join urls on log.url_id=urls.id
group by day, url
order by day, sum(duration) desc;
url_id|count(url_id)|sum(duration)|max(duration)|day|url
1|2|306|228|200987|mail.google.com
9|6|296|217|200987|twitter.com
12|2|225|211|200987|google.com/reader/
28|1|128|128|200987|hunch.com
21|1|66|66|200987|bbc.co.uk
[...]
39|3|10222|10212|200988|webmail.bbc.co.uk
21|3|2155|2125|200988|bbc.co.uk
9|18|1494|235|200988|twitter.com
1|12|1003|185|200988|mail.google.com
10|14|777|249|200988|search.twitter.com
[...]
39|5|2930|2667|200989|webmail.bbc.co.uk
1|8|1196|928|200989|mail.google.com
9|10|922|455|200989|twitter.com
12|1|394|394|200989|google.com/reader/
21|6|259|151|200989|bbc.co.uk
[...]

And returning to the original question of just how many visits do I make to Twitter

sqlite> select
  count(url_id) as visits,
  round(sum(duration) / 60.0, 2) as total,
  round(max(duration) / 60.0, 2) as longest
from log
left join urls on log.url_id=urls.id
where url = 'twitter.com'
group by day
order by day;
visits|total|longest
6|4.93|3.62
18|24.9|3.92
10|15.37|7.58

So it seems that on Sunday I made 6 visits for a total of about 5 minutes and a single longest session of 3 and a half minutes. On Monday it was 18 visits for a total of 25 minutes including one session of nearly 4 minutes, while today, 10 visits so far (including one of over 7 minutes) have already added up to over 15 minutes.

.mode csv

in SQLite is handy too, because it changes that list format to look like

visits,total,longest
6,4.93,3.62
18,24.9,3.92
10,15.37,7.58

so it’s trivial to open it in a spreadsheet.

Even better will be something cunning and programmatic. Maybe in PHP or Ruby or something. Even this exploratory manual approach is fun though. It will obviously be better once I’ve built up a bit more history but now I know that MeeTimer is storing my data in a way that I can access it, I’m even more excited about it. Thanks, MeeTimer. You rock.

I was invited along to the announcement, and took some hurried notes during the introductory talks. They’re by no means comprehensive but I did manage to capture a few quotes from some of the speakers and most of the Q&A session too. Here are the highlights…

Tim Brooks, Managing Director

Before the web, we reached around 6 million people per week. We now reach 33 million in a good week.

Emily Bell, Director of Digital Content

We take risks. … Handing this over to you guys is a risk. But one that I’m sure will pay back in multiple dividends in terms of the creativity it unlocks. It’s a significant step towards the idea of ‘Guardian Everywhere’.

Mike Bracken – Director of Technology Development

showing Chalkboards…

We can’t do everything ourselves … We want you to show us how to improve

Stephen Dunn, Head of technology strategy

shared a timeline of some of the Guardian’s online activity, pointing out that today’s announcement is a step in a journey.

• 1995 – Guardian first on the web
• 1996 – Guardian New Media Lab
• 1998 – Unmoderated Talkboards launched
• 1999 – Guardian Unlimited launched with registration sustem, but was removed in the same year.
• 1999 – RSS feeds and headlines distribution service
• 2001 – first Guardian blog
• 2006 – Free Our Data
• 2006 – Comment Is Free
• 2007 – RSS Everywhere
• 2008 – Full feeds (with ads)
• 2008 – First guardian hack day

and their web principles:

1. Permanent – (a cool URI is one that does not change)
2. Addressable – (resources are about something, ready fort the social web. We live in ‘the age of Point-at-Things‘)
3. Discoverable – (multiple routes to content. Tagging drives discovery)
4. Open – (hackable URLs, using and contributing to open source tech)

Matt McAlister, Heard of Guardian Developer Network

Today the Guardian announces the open platform. It’s the suite of services allowing partners to build services with the Guardian

Some more detail from Matt on the terms of use. You get rich, tagged article content. Full content, not just the headline and abstract. And you can publish it in your apps

We decided that the best price point for fueling growth is free. We want you to be able to make money on your site too.

Looking at the site, there are Commercial partner programs to enable commercial use. The FAQ says that “The Content API is free to use. There are some terms that you must adhere to for the free access level. For example, our default limit on queries per day is 5,000 calls, and we will in the future ask partners to display advertising from our ad network on pages with our full content.”

Explaining that it’s a beta trial, and they’ll be approving API keys on a limited basis (which has pissed off Dave Winer), Matt said

We’ll be doing that slowly so that we can understand what the dynamics are, and what you want to build … But we do plan to open more widely soon.

Simon Willison

Simon demonstrated the API explorer (the first app written using the API) and how each set of results also specifies a set of usable filters which are available in source, and shown in explorer.

New concepts can be prototyped in less time than the meeting you’d have needed to discuss it in the first place

and demonstrated this point using several demos including contenttagger.org (from Chris Thorpe) and APIMaps from Stamen Design. Pleasingly, AMEE are another launch partner for the Data Store too. Exciting stuff.

Q&A session

Q – (Jon Slattery from Press Gazette) – Are you happy for news organisations to use your material?
A – (Emily Bell) – They do already! Frequently. The free model is built with developer community in mind. If other news orgs want to use the data because it’s better then theirs, that’s fine within the terms.

Q – (James Cronin) – Can we store data in your data store?
A – (Matt McAlister) – we’re publishing data into the store at the moment, but tell us what you’ve got, we’d love to look after it.

Q – (Stefan Magdalinski from Moo) [re 5000 hit daily API limit]
A (Simon Willison) – 5k hits is what you can ask from us in 24 hours. You’re allowed to cache that data though, so could serve as many pages as you like. We’re also serving smart caching headers with 50 min expiry.

Q – Are you exposing photos & media?
A – (Matt McAlister) – Not yet. But we return headline and description and URL for our photo pages.

Q – (Andy Pipes) – Any plans to share personal data, like the New York Times?
A – (Matt McAlister) – No. None at all.

[photo credit: Steve Bowbrick]

Here’s some advice for anyone at the BBC wanting to get involved, which includes some things to consider if you’re not already familiar with contributing to Wikipedia. Feel free to ignore it if you don’t work for the Beeb, but perhaps it will be interesting and useful to other people too and of course I’m keen to hear what (presumably many) important things I’ve missed.

First of all, it’s worth knowing that the BBC has editorial guidelines about using open access online encyclopedias.

“…When correcting errors about the BBC, we should be transparent about who we are. We should never remove criticism of the BBC. Instead, we should respond to legitimate criticism. We should not remove derogatory or offensive comments but must report them to the relevant administrators for them to take action.

Before editing an online encyclopedia entry about the BBC, or any entry which might be deemed a conflict of interest, BBC staff should consult the house rules of the site concerned and, if necessary, ask permission from the relevant wikieditor. They may also need to seek advice from their line manager.”

Once you’re comfortable with all of that, the next place to look is Wikipedia’s own documentation.

A good places to being in the guide on contributing to Wikipedia, which says that although you do not have to create an account to edit articles on Wikipedia, there are many good reasons for you to do so. See especially the advice on why create an account. BBC employees should be open and transparent about their BBC status (which will be obvious from their IP addresses anyway, like this well publicised example) and the best way of doing this is by creating and using a user account.

More good places to get started include the Five Pillars, avoiding common mistakes and the perfect article (although it’s worth remembering that perfection is not required).

The policies and guidelines are important. Anyone considering editing Wikipedia you take their time in absorbing and understanding all the policies and guidelines. Here are some highlights. What follows it not a complete list, just a taster to get you started.

Policies

Neutral point of view

“All Wikipedia articles and other encyclopedic content must be written from a neutral point of view, representing fairly, and as far as possible without bias, all significant views that have been published by reliable sources.”…

Verifiability

“The threshold for inclusion in Wikipedia is verifiability, not truth—that is, whether readers are able to check that material added to Wikipedia has already been published by a reliable source, not whether we think it is true.”…

No original research

“Wikipedia does not publish original research or original thought. This includes unpublished facts, arguments, speculation, and ideas; and any unpublished analysis or synthesis of published material that serves to advance a position. This means that Wikipedia is not the place to publish your own opinions, experiences, or arguments.”…

What Wikipedia is not

“Wikipedia is not an indiscriminate collection of information; merely being true or useful does not automatically make something suitable for inclusion in an encyclopedia” …

see particularly the policy on news reports

“Wikipedia considers the historical notability of persons and events. News coverage can be useful source material for encyclopedic topics, but not all events warrant an encyclopedia article of their own. Routine news coverage of such things as announcements, sports, and tabloid journalism are not sufficient basis for an article.”…

Guidelines

Conflicts of interest

“Activities regarded by insiders as simply “getting the word out” may appear promotional or propagandistic to the outside world. If you edit articles while involved with organizations that engage in advocacy in that area, you may have a conflict of interest.”…

see particularly How to avoid COI edits and How to handle conflicts of interest

External Links

“Wikipedia’s purpose is not to include a comprehensive list of external links related to each topic. No page should be linked from a Wikipedia article unless its inclusion is justifiable”…

plus What to link, including What should be linked, Links to be considered and Links normally to be avoided

Reliable sources

“Keep in mind that if the information is worth reporting, an independent source is likely to have done so.”…

Notability

“Within Wikipedia, notability is an inclusion criterion based on encyclopedic suitability of a topic for a Wikipedia article. The topic of an article should be notable, or “worthy of notice.” Notability is distinct from “fame,” “importance,” or “popularity,” although these may positively correlate with it.”…

see particularly General notability guideline and Notability of article content

“Keep in mind that an encyclopedia article is a summary of accepted knowledge regarding its subject, not a complete exposition of all possible details”…

You’ll want to be careful to follow Wikipedia’s policies and guidelines to ensure that any proposed edits, new pages or external links are worthy of inclusion, and always be open to correction from Wikipedia’s users and editors.

In 2050, the permanent and nomadic residents of Lyddle End use the community Fabricator to build whatever they need. They do their paperwork and receive legal advice in the Tuck Shop, the owner of which can deliver a very personal service. (people still like to go to the shops.)

Nanotech/ Biotech agreements between Asia and Europe mean that creation at the atomic level is now possible. This photo shows the local butcher building a pig. (Those that eat meat still like to have it cut from something previously sentient, rather than the petri-pork that has replaced tofu.)

This is an early model fabricator. It needs to assemble from other matter. Absorbtion rods at the back suck C02 from the air (cleaning the planet) for raw material and re-sculpt this matter on the fab-deck. Other ‘fabs’ resculpt landfill or rock – but recycling the Earth’s mess is obviously the priority. There are no Home fabricators (legally) – the act of creation cannot yet happen in a small or enclosed space.

Each community fabricator is named after the first thing made. This one is called Dodo.

You can learn more about, and how to get involved in, Russell’s delightful Lyddle End 2050 project as well as see the results as they accumulate.

(Update: it’s now on Russell’s blog. Hurrah.)

During the recent US election, I was being rather traditional, tucked up in bed listening to Radio 4 (quite different from my approach in the 2005 UK general election, when Nick and I were even live-blogging the action). While I was being sleepy and passive this year, my friend Jo was being social online. Here’s what her screen looked like, complete with live-streaming BBC News, IM chat and Twitter.

I’ve been building this list for ages, but it’s finally time for a roundup of social viewing tools. Here are some examples of how the web is being used to make different sorts of conversations possible around television:

Curation and communities

• There are a few blogs about television. Watchification is “selecting the really good stuff from the BBC iPlayer…” and other sources. (Disclaimer: I’m the tech geek behind the curtains at Watchification). Curation is interesting. By highlighting Twitter, Delicious and Flickr content, the tag pages are getting (IMHO) more useful too.

• Smashing Telly is “a hand edited collection of the best free, instantly available TV on the web”. Like Watchification, it’s an example of comments around curated programmes rather than live chat.

Social recommendations

• I keep hearing people asking ‘what’s the last.fm of television?’ Dan recently sent me an invite to Boxee, which apparently
  

  “gives you a true entertainment experience to enjoy your movies, TV shows, music and photos, as well as streaming content from websites like Hulu, CBS, Comedy Central, Last.fm, and flickr.”

  I’ve only just started using it, and although it seems far from perfect it is only an alpha at this stage. The integration with other platforms, the desktop app and the last.fm-like scrobbling looks interesting.

• TIOTI has been around for a bit longer than Boxee. It invites you to:
  

  Find your favorite TV shows and brand new ones you’ll love, Share shows you like with your friends and see what they are watching, Download or stream TV shows from dozens of places online, Get involved and post your thoughts, improve our guide or add pics and vids.

Annotations

• YouTube started offering video annotations after Google acquired Omnisio but only (so far) gives the video uploader a way to add annotations to the video, so it’s not (yet?) a social annotation tool.
• Viddler, on the other hand, offer time-stamped comments and tagging, which are displayed along the video timeline and (by default) pop up at the appropriate time.

Playing the backchannel

• CurrentTV recently partnered with Twitter to display relevant Twitter updates live on-screen. Discuss the presidential debates while watching it (using Twitter tags) and have your comment displayed on TV.

• MTV’s Backchannel takes a different approach to annotating episodes of The Hills, turning the process of ‘tagging’ and ‘clicking’, to endorse a tag, into a game. Playing Backchannel won’t (as far as I can tell) stream the show to you, you just play in the browser while you’re watching the show at the same time.

Live chat

• When I think of live chat around TV, I think of Joost. Joost’s ‘channel chat’ has been overhauled a couple of times since the early days (I seem to remember it being initially based on IRC, then in 2007 they announced a partnership with Meebo) and more recently it seems to have gone away completely since they moved to Flash (or am I missing it?).

• BanterTV combines iPlayer simulcast embeds with real-time chat.

• The Electric Sheep Company’s WebFlock provides features for social viewing including
  

  a visually immersive environment for social interaction, media consumption and game play

• Lycos Cinema is “is Synchronized Movie Watching” (complete with a frankly horrible introductory video) and invites you to
  

  Watch thousands of movies and tv shows with your friends and chat live while you watch.

• What about the BBC? Anthony Rose recently announced some prototype work with Microsoft, in which iPlayer was hooked up to Windows Live Mesh. This was announced recently at Microsoft’s Professional Developer’s Conference and blogged in more detail by Microsoft’s Marc Holmes. Strategic, or experiment? You decide.

Of all of them, I find the asychronous chat using comments in the timeline on Viddler, and the game-playing elements of MTV’s Backchannel to be the most interesting. There must be lots of examples I’ve missed, but it’s an area I’ll continue to watch with interest.

Part of the enjoyment of both sites is the idea that we’re not just curating our favourite stuff, we’re weaving links between it, with those links becoming increasingly fun to explore. Tonight’s hack was a small improvement in order to pull in a few things that the web knows about every tag, but also every presenter, director, editor, and in fact any search term. Here’s what happens when you search Speechification for shows in which Stephen Fry is the presenter.

It works for normal tags too, and they’re handled in the same way. Ever wondered what Speechification has on Malcolm X? or Chris Watson?

I took exactly the same approach at Watchification, though had a bit more room to play with in the layout.

Other examples: ‘politics’, ‘music’, and shows in which Simon Amstell is the presenter.

As well as the obvious embed of (Creative Commons licensed) photos from Flickr, I used Monitter for the realtime Twitter monitoring, and a delicious.com feed of relevant bookmarks.

I have lots more ideas for other things to usefully enhance the page too. Watch this spaces. Um. These spaces.

Update: at the suggestion of Dan Hil, I’ve moved the widgety stuff to the bottom of the page.

After watching Chris Delay from UK games company Introversion Software presenting at Playful last week, I went digging around their back catalog. I’m excited about their upcoming game, ‘Subversion‘, but I was pleased to be able to download demos of Uplink, DEFCON and Darwinia.

DEFCON is already a personal favourite. I was glad to see that, as with their other games, as well as a Windows version it’s also been ported to both Mac and Linux (the Mac version is being distributed by Ambrosia GTames).

It’s strikingly beautiful. Based on a gorgeous world map (with nostalgic Cold War era national borders) the careful placement of silos, airfields, radars and naval units, the escalation of hostilities and eventual all-out nuclear war. Timing is everything. It’s actually pretty tense.

The gameplay is simple, with defined countdowns from DEFCON 5 all the way to 1, at which point (as we all know from watching War Games) the use of nuclear force is authorised. You can accelerate time at will, unless you’re playing the real-time ‘office mode’ game mode.

Unlike most war games, DEFCON doesn’t glorify it. The ambient background music interlaced with atmospheric sounds include a very disconcerting coughing noise and the messages which pop up when a city is hit, telling you how many million people just died, are shockingly stark.

After playing 3 games, I’m now reliably beating the computer. It must be time to start playing online against human opponents; multiplayer games with strategic alliances look like fun.

More detailed reviews (going back a couple of years now) from Gamespot. IGN and many more.

Roo Reynolds plays “Hotel California” from Tom Armitage on Vimeo.

Thanks to Tom for capturing this video. That was fun.