Category Archives: Data

Validating host names – top level domains (TLDs)

For a recent project I needed to update a hard-coded list of TLDs. Keeping it fresh seemed like a good idea, so I took IANA’s list of TLDs (IANA TLDs also available as text here) and made a little API service to provide the data in different formats:

ICANN provides the TLDs in upper case, but the Firtl API lower-cases them for convenience.

Generating coloured-in election map images with SVG

The Election Boundary Map generator in Firtl’s sandpit will attempt to match a text data file with an SVG image of the UK (actually only Great Britain since Northern Ireland is not on the Ordnance Survey map) showing political constituencies. Simply upload a text file with two fields on each line. The first field should be the name of the constituency, the second a numeric value such as a percentage. Separate the two fields with a space.

A coloured map will be generated to reflect the value in each constituency. The colours are drawn from a red-green-blue spectrum from 0 to 100%, or 0 to the maximum value found in the file if the “Stretch values” checkbox is set. Constituencies for which no matching name is found in the uploaded file will be left uncoloured.

ElectionMapSpectrum

The map images are available in PNG, JPEG and SVG. Here are PNG examples of the percentages of votes cast for some of the better-known options at the 2010 General Election, in alphabetical order:

election-boundary-map.bnp

BNP

election-boundary-map.conservative

Conservative

election-boundary-map.green

Green

election-boundary-map.independent

Independent

election-boundary-map.labour

Labour

election-boundary-map.libdem

Liberal Democrat

election-boundary-map.plaidcymru

Plaid Cymru

election-boundary-map.snp

SNP

election-boundary-map.ukip

UKIP

The SVG maps were generated from data from the Ordnance Survey’s OpenData Boundary-Line product (sorry, can’t give a direct in-page link as OS’s HTML is broken). I used the free QGIS mapping application to export SVG from the OS data, but had to edit QGIS’ SimpleSvg plugin to embed constituency names from the ‘Attribute Table’ to allow more straightforward colouring of the map.

The data I used to colour the maps I found by visiting http://en.wikipedia.org/wiki/Results_of_the_2010_United_Kingdom_general_election_by_parliamentary_constituency and following the link to the Electoral Commission’s General Election 2010 Results Data (sorry, no direct in-page link as the Electoral Commission’s HTML is insufficiently marked-up).

The electoral data is in one great Excel file (why not an open format?), which I transformed into individual files for upload to my map generator. Here are the files:

election2010bnp election2010conservative election2010green election2010independent election2010labour election2010libdem election2010plaidcymru election2010snp election2010ukip

Firtl ready to log – I think

There’s obviously a lot of work still to do to get a desktop/smartphone version of the logger ready, but the ‘minimal/mobile’ site seems to work just fine with the two elderly mobile phones (a Nokia 6500 Classic and a Samsung Throwing Phone) I’ve tried it on. At the moment there’s only a single subject on which to record log entries. The MorrisonsTavistockCheckoutTime subject was inspired by a thread on a local forum about queueing times at a local supermarket. It’s not currently possible to add new subjects, but will be any day now.

So how does one log an entry for time spent at the checkout? If you’re using an older mobile phone which doesn’t make it easy for you to browse your email, you may want to register for an account at firtl.com from your desktop at home first. The log application won’t accept anonymous contributions as a matter of policy. Like a lot of other sites, firtl.com will send you a registration email with a verification link in it which you must click before you can log in.

Find your subject


The minimal/mobile method assumes no clever support on the phone, so it involves several steps, exchanging data with the firtl.com server at each stage. You can start by browsing firtl.com and following links to the log from there. If you already know on what subject you want to log, you can follow a direct link such as this one: MorrisonsTavistockCheckoutTime. You should see the subject page, showing some explanatory text, recent entries and a chart of the log entries.

 

Select a logger

The long-term idea is to provide different ways of adding data for different subjects, so each subject will list known appropriate logging methods. There’s only one working at the moment – the MinimalStopwatch. That’s a multi-stage stopwatch for basic mobile phones. It won’t look very nice on more powerful devices, but it will work. Follow the link to MinimalStopwatch. To log an entry you must be logged in, so you’ll be prompted to login now if you were not already. You can register from your mobile phone, but you’ll need to be able to read the email firtl.com sends you on your mobile phone.

 

 The MinimalStopwatch

The stopwatch shows the subject you’ve chosen and a couple of buttons. You can edit the subject name here, although it’s probably easiest to follow a link from somewhere else (an article about the subject, for example). Pressing the “Start” button will start the stopwatch running, reset will cancel it. A suggested policy for recording time logs with the MinimalStopwatch is to get to this point (ready to press ‘Start’) before you commence your activity-to-be-measured. That way you can press ‘Start’ as soon as the activity begins.

 

 

Stopwatch running

What happens when you press start is that your phone will send a request to firtl.com to temporarily record a ‘start time’ for the stopwatch. Your phone will show you a link to the selected subject, confirmation of the time the stopwatch started and several buttons. The ‘…’ button allows you to see how much time has elapsed so far without stopping the stopwatch. The ‘Stop’ button will permanently stop the time ready for logging, and ‘Reset’ will cancel the stopwatch.

 

 

Time elapsed so far

Pressing the ‘…’ button while the stopwatch is running will request a time-so-far update from firtl.com. Bear in mind that this update is obtained by mobile networking, may take some time to complete and by the time it arrives at your phone may be several seconds ‘short’. If mobile networking is working well for you, the running time you see should be within a few seconds of what a proper stopwatch would show.

 

 

Stopping the watch

Pressing the ‘Stop’ button will give you a final duration for the stopwatch. The duration is the difference between the time at which firtl.com received the ‘Start’ request and the time at which it receives the ‘Stop’ request. Those timings are not precisely when you press the buttons on your phone, but may be delayed by several seconds due to mobile networking. The Stopped page allows you to add a text note to your log entry before finally logging it. Note that stopping the watch is not enough to record an entry – you must press the ‘Log’ button to add your measurement to the subject log.

 

Log entry text

You can add a small amount of text to a log entry. It’s not necessary to do so, but may be useful later when you compare or analyse measurements. Remember to press ‘Log’ – and wait for the page to update – to commit your entry to the subject log.

 

 

 

 

Log entry complete

Pressing the ‘Log’ button will commit your log entry to the subject log. After this, other users of the log will be able to see your entry. If you follow the link to the subject, you’ll see your entry in the list and in the chart plotted for the subject.

 

 

 

 

That’s it!

That’s all there is to it. On the subject page you’ll be able to see your log entry and those of anybody else contributing data. There’s a list showing the last few log entries and an all-time chart plotting the time entries against the time when they were recorded. There will be a few more charts available soon, showing average times over time (so you can see whether queueing times are increasing or decreasing), average times over day-of-week and average times over time-of-day (so you can see when are the best / worst times to go through the checkout!).

Of course none of these graphs will show anything useful at all unless there are data to chart! Please, if you’re interested in what can be learned about a subject by pooling social measurements, register for an account and log some entries. Don’t worry about doing it wrong – users can easily permanently delete their own log entries.

Morrisons

I’m not affiliated with Morrisons in any way, nor am I waging any kind of train-spotting war on their checkouts. I quite like shopping at Morrisons; the deli more than makes up for the sometimes empty shelves elsewhere =p They just happened to be the subject of the article at TavistockPeople which gave me the idea for this project.