When sentiment is not enough

In recent posts, we have explored the value of social media as a source of insight but we wanted to go further and illustrate why insight and customer experience professionals in particular are paying more attention to social media as a source of insight.

Our aim was to demonstrate how social media can be used to truly understand what customers feel, think and do.

We wanted to add weight to the long standing argument that to understand customers and provide actionable insight we need to go beyond sentiment and look at the language customers use at different stages of their journey.

We conducted a piece of research focusing on customer comments around the main providers of Pay TV in the UK.

We sought to answer two questions:

Does the level of customer satisfaction, as played out by customer comments in social media, reflect other sources of research, namely Ofcom complaints per 1,000 and Which? customer satisfaction scores?

What are the key drivers of advocacy and detraction for the Pay TV providers?

As with any form of research, the closer social media researchers are to business needs, the more actionable the research and measurement become. So we put ourselves in the shoes of marketers at one of the Pay TV providers and assumed they were working towards the following business goals;

  • Improve customer experience
  • Improve client own survey and Which? customer satisfaction scores
  • Reduce rate of Ofcom complaints
  • Acquire and retain customers

These objectives helped inform our research framework, what to look out for and how to structure our approach to unstructured, unprompted data.

We focused our analysis on customer comments in social media for each Pay TV provider during the same time frame as the Ofcom and Which? research (October-December 2013).

  • First, we gathered all the content in connection with the Pay TV providers across Twitter (Oct-Dec 2013).
  • Second, for Twitter we coded a sample of customer comments for feeling (very strong – moderate), emotions, sentiment, detraction, advocacy and topic.
  • Third, we conducted a depth analysis on a sample of Twitter conversations between customers and the Pay TV providers. We also coded for the triggers of customer recommendations /complaints /issues.

We also conducted a specific analysis of forum conversations, which required a more nuanced approach:

  • First, we identified several forum threads where each of the Pay TV providers was the main focus of the thread.
  • Second, we coded the initiator and contributors’ comments for their feeling, favored brand and reason for recommending an alternative to each provider if at all.

Does the level of customer satisfaction as played out by customer comments in social media agree with other research, namely Ofcom complaints per 1,000 and the Which? customer satisfaction score?

To help answer this question we created a Social Net Promoter score (advocacy minus detraction), mirroring the well-known customer satisfaction measure normally based on a recommendation question. We also calculated a customer sentiment score and then compared these two metrics to see which was most closely aligned to the results of the Which? survey and level of customer complaints as reported by Ofcom.

Our analysis showed that the Precise Social Media Net Promoter score correlated closely with the Which? survey rankings. Sky TV was the outright winner in the sector across all metrics with BT and TalkTalk ranking 3rd / 4th for Ofcom complaints, Which? and the Precise Social Media Net Promoter score. Virgin Media came second on all the metrics with the exception of sentiment.



This can be explained by the fact that between October and December 2013 Virgin Media saw a large number of negative mentions related to the ‘Anywhere’ app not working as expected. Interestingly though, although the app drove negative mentions, and therefore affected the overall sentiment towards the brand, it did not lead to more (Ofcom) complaints or drive detraction. This partially illustrates the need to go beyond sentiment to truly understand customer satisfaction. One-off events can impact sentiment but not necessarily drive outright detraction or indeed recommendation. In order to truly understand satisfaction, we therefore need to look at what is driving advocacy and detraction amongst customers as well as understand their feelings, context, tone and the voice of the customer.

Before we advocate Social NPS as the KPI to measure it is important to note that social media, in particular Twitter, is used by customers to get a quicker response to a complaint or problem. Customers often go to Twitter because they may have failed to resolve the problem through other channels, for example via call centers or email. So as could be expected, we find higher levels of complaints and dissatisfaction in our data. Customers also report positive experiences more readily in social media, sharing those with their community. This results in lower NPS scores than those you would find from a customer survey.

Social NPS cannot be automated though as it requires humans to understand what is being said by customers and in what context. If the social Net Promoter Score appears to be a good barometer of customer satisfaction, it is more actionable still when integrated with other scores such as Which? and trended over time.

What were the key drivers of advocacy and detraction?

A key benefit of using social media as a source of insight is our ability to explain the why, more regularly than with traditional research. We have talked at length before about how this adds value to existing brand or customer satisfaction trackers. 

Our analysis found that the key drivers of advocacy and detraction varied according to each provider. For some providers the inability to resolve repeated problems, poor reception and engineers not showing up for appointments drove detraction. Key drivers of advocacy included receiving excellent customer service, problems being resolved quickly and receiving a value for money service or receiving the service that had been sold to them.

Going beyond sentiment brings additional value for insight and customer experience professionals through a deeper understanding of feelings and language at different stages of the customer journey. It can also help inform what messages might resonate with customers and tactics to acquire and retain them.

An example of this could be;

  • Acquisition: Engineers not showing up is a key driver of detraction for competitors but a key driver of advocacy for your brand. This could be highlighted as a prominent USP to potential customers through key touch points such as direct mail and owned media.
  • Acquisition and Retention: If more of your customers are sharing instances of excellent customer experience remind all customers, through testimonials, how much the brand’s focus on customer service matters. Target your competitors’ customers with evidence to show you resolve problems quickly.

Social media insights are actionable when highlighted in the right context and through the right metrics and can help provide further understanding to existing research. As with any other forms of research, it is a question of being clear as to what your research objectives are.



Search article