Category Archives: Uncategorized

Reading a smart meter with a webcam

Yeah I know – the title, right? “What the faq is he doing with a webcam if he has a smart meter?” A contemporary UK “smart meter” is very smart: it holds your data hostage. What I had installed recently is an electronic meter with communications adapter and a little head unit to show me near-instantaneous consumption data. As soon as the meter was fitted, I went looking for the actual data that must exist at least temporarily to allow the head unit to work. The executive summary: the data is not accessible to the homeowner.

Buy a Software Defined Radio for fun, but not this

I didn’t go looking for the data straight away, I had a quick look at the head unit and the meter to see if there was anyway of connecting to them or exporting the data. There are none. “The meter must be communicating with the head unit somehow, maybe I can sniff / connect the same way?” I thought, and read several articles suggesting ‘Software Defined Radio’ would give me access to not only my meter’s data, but those of similar meters in my neighbourhood. This is either not a thing in the UK or was only briefly a thing with early models of even-less-smart meter, broadcasting your data in an easily accessible way being incompatible with privacy.

The smart meter

The standards to which UK smart meters must adhere pretty much preclude any easy tapping or sniffing of data. In fact, it appears the only way to gain access to your own smart meter data is to arrange with your supplier to send the data to them or to a 3rd party selected by your supplier, so they may offer you your own data ‘as a service’. This is a miserable shortcoming of the smart meter standard, which should be rectified as soon as possible. It must be possible for a reasonably competent homeowner to gain direct and private access to their own consumption data.

That little head unit is okay for instantaneous power and a running total, but when it gets to £20, where did the money go? There are no clues. This falls far short of any definition of ‘smart’. I believe it should be illegal to call a device ‘smart’ unless its smartness is self-contained. If you have a device that can deliver its “smart” behaviour while disconnected from the Internet, that’s a smart device. If it needs to communicate with a service hosted elsewhere, the device itself shouldn’t be marketed as anything other than a proxy, agent or facade.

The government’s publications for smart meters seem at first glance to have the intent of giving the homeowner control over their data. Where they fall short is in describing only how the homeowner grants access to their data to suppliers and third parties: there’s nothing to describe making the data accessible to the homeowner. In the government’s standards for smart meters, the best you’ll find is a circular buffer “capable of storing four hours of UTC date and time stamped six minute Consumption data”. When you can see your head unit updating the ‘live’ consumption every few seconds and billing periods are rarely less than 1 month, 40 records is effin’ risible! Thanks gov!

Where the webcam comes in

Light on the meter with impulses per kWh legend

Previously, I’d read a meter with a webcam by looking at an old-school meter’s analogue dials and ‘recognising’ the digits. This time there was something a bit more ‘digital’ accessible to the webcam: a blinking red light. Next to the light is a legend that says “1,000 impulses per kWh”. To my pedantic mind, that looks like the light flashes every time one Wh (Watt-hour) is consumed, or 3.6kWs (kilowatt-seconds).

The average power consumption over the interval from one flash to the next is 3.6kWs divided by the duration of the interval. For example, if the light flashes once per second, the average power consumption is 3.6kW (3.6kWs / 1s). Another example – if the light flashes 1,000 times over an hour (the legend’s “1kWh” over an hour) the interval is 3.6s and the average power is 1kW. If you fire up an 11kW ‘power shower’ you’d expect to see about 3 blinks per second, and with everything off at night time, you’d expect some quite long delays between blinks.

The scheme

The basic scheme is to take a picture of the meter with the webcam, crop out everything but the square holding the light, get an average colour for that square and (waves hands) detect when the light switches from off to on. The interval between off-on transitions can then be used to compute average power over the interval. The equipment is same as last time: my venerable Dell Mini 9 running Ubuntu 18.04 and a cheap-and-cheerful Logitech C250 webcam.

For a first shot I used uvccapture and ImageMagick from a shell script to generate a stream of RGB values for the average pixel color, then wrote a little utility that converted these to a datetime and average power pair. This worked, but not well: the capture rate was 3-5 frames per second. Even this was useful though – it was obvious from the first few hours’ data that “Child number one” was taking a half hour power shower compared to just a few minutes for everyone else. I’ve lost that data, so there’s no real “steaming gun” in the data that follows.

Here’s a reasonably representative 24 hours collected with a little Java program (attached below) driving the webcam at 50 frames per second and logging timestamps and power:

You can see from most of the night that there’s a baseline around 50W – I suspect that’s mostly low-power devices we don’t turn off overnight – our Internet router and other devices with electronic displays. The nearly-1kW ‘pulses’ between midnight and 7am are from an electric underfloor heating circuit. It’s interesting to see it spend about twice as much time off as on, so its average power is somewhere around 300W. The UFH is turned off in the morning. Some spikes visible through the day are kitchen devices being used at the same time. The kettle is around 2kW, there’s a 950W microwave, a toaster around 1kW – none of which run for very long.

Parent number one was off-shift that day, so the slightly increased baseline through the afternoon could have involved Netflix. Children numbers one and two arrive home from school either side of sunset in the late afternoon, so there’s a bit more power being consumed by lights, anime and Minecraft. Cooking / tidying up around dinner results in some 2.5kW pulses from an ‘instant’ hot water tap on the kitchen sink which has a small tank of its own. The big spikes late in the evening are from a ‘power shower’ which seems to be about 5kW. They’re mixed in with some added electric heating in the bathroom.

There’s also a regular ‘pock’ of about 300-400W which becomes more obvious around the evening ablutions (and on damp days!). I believe this is the dehumidifier (with humidistat) in our bathroom which is doing a pretty good job (there’s no convenient place to put an extractor and the bathroom has multiple damp-inducing construction issues which we plan to fix … one day) but perhaps costing more to run than we expected.

This quality of data tells me almost exactly what’s responsible for my power bill. It’s not perfect – I suspect the webcam misses the occasional blink from the meter and there may also be spurious false-positive readings recorded which manifest themselves as horrifying 20-30kW spikes! Effectively polling the meter blink also quantises the data in a way that produces equally-ish spaced horizontal stripes of points around a reading. I’ve got a tiny microcontroller and photocell on order which I’m hoping will allow me to stick (temporarily) something on the face of the meter which will much more precisely capture (by interrupt rather than poll) the timing of the blink’s off-to-on transition.

What I’ve collected with the webcam is around about the quality and quantity of data I expected from my smart meter, so I’m bitterly disappointed by what was installed. I have to say that what’s installed is exactly what the standard says should be installed, so my disappointment is reserved wholly for the government. We shouldn’t have to try this hard to get data about us from devices which are recording us in our homes.

Simple web form generator to PDF

Recently I was asked how to complete a Refugee Action Microsoft Word application form by someone who had no Microsoft software. Nothing I tried (third party applications, web-based document editors) was able to even lay out the form correctly, let alone edit the fields without causing damage to the document structure. Here is a PDF version of Refugee Action’s Word document generated by LibreOffice. Some of the formatting / form control issues present in the original document when opened by non-Word applications are present in the PDF.

Organisations with a web presence should be collecting form data with HTML to accommodate the widest possible audience of WWW users and styling it for physical presentation in a portable format (such as PDF) as a separate activity. Separating the data from the form presentation also offers the opportunity of directly importing form data into office databases.

I’ve thrown together a little demo in Firtl’s sandpit to demonstrate a form-filler service.  I chose a text-based Document/Section/Field scheme and created a text file to describe the fillable form. I use a zip file for export / import, but it would be straightforward to provide import/export in other (CSV, XML, JSON etc) formats.

The URL of a form definition is passed to the form-filler service as the ‘def’ query parameter. See how the Refugee Action application looks as a web-based form here. The form submission controls (at the bottom of the web page) allow you to ‘Update’ the page, ‘Download PDF’ and ‘Download Zip’. ‘Update’ does mostly nothing except in the case of image upload: it shows a thumbnail of the uploaded image. It could easily provide some generic validation such as ensuring mandatory fields are filled and checking data types (date / number etc).

The ‘Download PDF’ button uses the form data, the document definition file and some server-side LaTeX to produce a static PDF document containing the form data. The ‘Download Zip’ button allows you to download just the form data on its own. It allows users to save the form content and upload it later (see bottom of every form-filler page for upload form) to continue editing. I tested the form-filler on an Android smartphone by creating the Zipped form content on my desktop PC and then sending it to the Android device via email.

The web forms are plain HTML so should work perfectly with any recent* browser. The Refugee Action demo has their logo applied with a simple CSS stylesheet specified in the form definition file. If no stylesheet is specified, some default styles are included in the page source. Note that the styles apply only to the web form and not to the generated PDF.

Firtl’s form-filler service uses HTTP POST to send form data to the web service, so it’s not possible to provide a clickable link to a convincing example of the PDF-generation in action. Instead I’ve uploaded my test form data zip file to this blog. Download the zip file, navigate to the foot of the demonstration form, select the downloaded zip file and click ‘Upload’. You should see nearly every form field filled with test data. Clicking on the ‘Download PDF’ button should give you the completed document.

The Refugee Action example demonstrates almost all the features built into this basic form generation demo. I’ve included a couple more examples below.

Note that there’s currently nothing stopping you hosting your own form definition file and using the form-filler service to generate web forms, zipped form content and PDF documents. I’d appreciate a mention if you do!

Hello, World (form definition file is here)

Maths Test (form definition file is here)

*’recent browser’ – I tested this on a 2007 Nokia 6500 Classic’s S40 browser. Editing the document, creating the zip and downloading the PDF worked perfectly. The Nokia has no software for viewing zips or PDFs, but can still send them as MMS or email attachments. Try editing a Word document on a phone like that!

Measuring tube question generator

Here’s a little demo of on-the-fly image creation based on a Key Stage 1 homework my daughter brought home. Every time you visit this page you should see a different question. A new question is asked after each time you press the ‘Am I right?’ button.

This page requests a ‘measuring tube’ JSON object from the Firtl server. Server-side code selects a measuring tube template from a set of different capacities and major/minor measurement ‘ticks’, calculates a random fill that lines up nicely with one of the ticks and returns a JSON object containing the fill level and an URL for the image. The image itself is not drawn by the server until it receives the request caused by setting the img’s src attribute.

Screenshot (PNG) of large web page from Firefox

Was having little luck with printing-to-PDF and addons until I saw a comment to the effect that Firefox has a command line interface since many version ago. Just press shift-F2 and a prompt appears at the foot of the Firefox window. Entering ‘help’ lists the available commands, one of which is ‘screenshot’.

You can get help for screenshot with ‘help screenshot’ but the option format seems to be out of date (in version 22.0). The way to enter the command line swithces is to precede them with ‘–‘. As soon as you type the double-dash you should be prompted with the available arguments. So for example to copy the entire current webpage to your clipboard, you’d type shift-F2 (the CLI doesn’t seem to notice if you change tabs) and enter:

screenshot --clipboard --fullpage

To save the whole current webpage press shift-F2 and enter:

screenshot screen.png --fullpage

Hope this helps!

LEGO NXT robot head with webcam and JavaCV

While I was finding out how to use a webcam for motion detection with Java via JavaCV, I built this little robot head from LEGO Mindstorms with LeJOS on the NXT. Having Java on both sides of the USB connection makes communicating the direction ‘to look at’ simple and reliable.

I use JavaCV to find the largest difference (what moved the most) between successive images from the webcam and send its position in the webcam’s view to the NXT via the USB cable. The NXT reads the position and controls yaw-pitch on the two ‘neck’ motors to match.

Emily made the robot’s head from a small, plastic soft fruit punnet (the holes in the bottom fit just right onto the ends of LEGO struts) covered with kitchen tissue strips fixed with glue stick. Make-up is by washable marker pen.