Making sense of our ‘big data’

By Robbie MacIntosh, Director of Technical Operations Centre

Last month, we officially opened our new office, Brabazon House, in Manchester. It not only holds our labs with over 2,400 smart meter testing spaces, it is where we are working to understand our network’s data in new ways. The size and scope of our network is constantly growing and we’re using state-of-the-art methods to visualise and understand the big data produced by the smart meter rollout.

Live data at the DCC control centre showing progress on second-generation smart meter installs

Understanding behaviours

With 2.5million meters, each regularly generating multiple messages, this produces millions of messages an hour. A day’s worth of data would take a single person a lifetime to analyse, that’s why we turn to machine learning to sift through this data. We run complex algorithms against that data and within minutes we can see what’s happening. We monitor every request and we can use that information to see if it’s expected or abnormal behaviour and whether we need to step in and help.

At the frequency of the data we have access to, we have been able to monitor via our operations centres the near real time customer experience for the installation of smart metering across the country. Using our analytical tools and insights, we are able to see the expected and predicted behaviours of customers and assist them when their behaviour is not matching what we usually see. For example, our algorithms can find if a single installer is missing a step in their meter install, and tell the energy supplier the issue before it becomes a big problem.


We also use this data to support customers with their end-to-end processes. We can help them to understand how their installations can be more efficient by sequencing differently or by introducing pauses in the process to allow messages to process completely before trying the next in the queue.

We are able to advise customers where their process is working and where they can improve upon their success rates. We have also carried out similar analysis on prepayment customers and used industry wide data to advise customers on how successful the overall process is.

Our main advantage is that we have industry wide data and can overlay it with other information sources to give higher degrees of predictability and reasons for deviation from the norm, whereas a customer only has their lens to the world. By adding our lens, we can help them to determine whether the issue is in their domain, a specific device combination, a regional variation or a general issue affecting all customers.

This data helps us make Britain more connected, so we can all lead smarter, greener lives.