Curated data and the business of insights

“Not everything that can be counted counts, and not everything that counts can be counted.” - Albert Einstein

There’s a well-quoted statistic that says that over the last two years alone ninety percent of the data in the world was generated. This is truly staggering, and it also means that businesses have new and powerful data sources at their disposal.

But we have to be careful of the data that we use, the right data is essential. Einstein was onto something – just because we can count something doesn’t mean we should. And the reverse is true too that there is incredible untapped value hiding in more qualitative data that we cannot ‘count’ in traditional ways.

Numbers on their own can be difficult to action. While data relies on logic and reasoning, decisions are often made based on emotion. Being able to merge logic and emotion through curated data sources is essential to actionable insights.

What the curation of multiple data sources provides is a more complete understanding of the world. A model is just a simplified representation of the real world. So it’s only useful if that representation holds up in the real world. When you’ve only got one data source your view is always going to be more limited and your representation of reality can only be so good. By curating multiple data sets we get closer to how a human brain processes information, using various stimuli and pieces of information to form a picture of the world, which provides much more richness than drawing conclusions from a single input.

A lot of the value in our work lies in the types of data we now have at our fingertips, unlocking another level of human analysis beyond an organisation’s internal data. In particular, the ability to combine qualitative data with quantitative data enables us to provide a deeper level of understanding into motivations and behaviour, making the numbers more usable than they’ve ever been before.

We’ve been doing a lot of work recently gathering, curating and mining third-party data sources for valuable insight, particularly around things like reviews, social media and news. The value we add is in sourcing third party data that complements our client’s data, adding valuable insight that would otherwise not be achieved if we looked at the client’s data in isolation.

Data can be loosely grouped into three useful areas

Proprietary

The data that an organisation owns (such as sales and revenue information, customer data and website analytics) that can be tracked over time. Many businesses are looking to commercialise their data, for example Paymark’s transactional data from over 140,000 EFTPOS terminals around the country, and Tesco famously does this in the UK through their agency dunnhumby.

Government

Data that is collected by the government and made publicly available, such as Census, Statistics NZ, Inland Revenue, geographic and infrastructure data. This rich information can tell us things like which geographic areas of the country have particular health outcomes and levels of student loan debt by demographic group.

Digital

The biggest and fastest growing data type covering all publicly available information on the internet. This predominantly comes from digital products like Facebook, Google and Wikipedia that build in a way to extract data from the source via an API (application programming interface). Alternatively, digital data is sourced through scraping of public information on the internet such as reviews or news stories.

Measuring the value of social media

We can see the benefits of this approach playing out with Zavy, the social media analytics platform we launched a couple of years ago. We’ve been working recently on measuring the ROI of a brand’s social media activity – that is, tying social media marketing to the bottom line using social media data and transactional data across thousands of different businesses.

To do this we’ve assessed a group of businesses in the Paymark data based on factors such as revenue growth, return business and basket size. This data has then been matched to the business’ social media data to assess the impact of posting activity on revenue. The results so far have been enlightening – what we’ve found is that certain types of social media activity translate into revenue value, to the point that we can say that if a business gets one hundred shares or posts daily rather than weekly they will see a corresponding increase in revenue.

The next step will be to drill down to assess the revenue impact of particular words used in social media posts. We can begin looking at the impact that language like ‘free’, ‘promo’ and ‘giveaway’ has versus language around brand and customer experience, for example. We’ll quickly be able to see the sales impact that promotional posts provide versus long-term branding plays and the longitudinal effects of these activities. Zavy can then begin making recommendations to users based on maximising the long-term brand building value of their activity.

This is a rich territory for exploration and shows just how powerful curated data can be. Enmeshing logic and emotion through combined data sources enables us to humanise the numbers and count that which previously could not be counted.

Putting data to the test

We’ve been working with a large retailer to understand their customer behaviour. We matched the retailer’s own customer data with Paymark transaction data and other third party information such as Google reviews to understand not only how customers are behaving but why.

These third-party data sources also allow us to look at how the retailer is performing in terms of customer behaviour against its various competitors. From the retailer’s data we can get a sense of what category a customer is shopping in and therefore the competitive set. Paymark data enables us to link back to a range of transactional outcomes for those competitors so we can assess that other retailers in our competitive set are, for example, seeing higher loyalty rates than ours. We can then use more qualitative information sources like Google reviews to provide context around that, for example uncovering if there is a pattern in review comments about the friendliness of our staff or the length of the line to check out.