This blog post will briefly cover some of the reasons why natural language processing (NLP) sentiment analysis is such a challenging prospect, even though it can be extremely valuable to sales and marketing. Specifically, it will focus on the difficulty of zeroing in on a type of opinion which will be insightful and computationally accessible to business analysis. The follow-up to this post focuses specifically on how this is used to reduce customer churn for organisations.
Despite your rage, your emotions are still quantifiable
The Space of NLP Sentiment Analysis
With the popularisation of online reviews and widespread sharing of opinion via social media, there is unprecedented amounts of material accessible to companies in order to gauge the reactions that people are having to their products and services. Additionally, speech-to-text systems are enabling companies to quickly and easily analyse audio transcriptions.
Manually sifting through all of the available data, of which the vast majority is irregular and unstructured, and gleaning anything usable out of it is a daunting and tedious task. One part of the field of sentiment analysis has been concerned with automating this process and putting it in the hands of everyday marketing professionals.
The Fundamental Problem of Opinion Mining
Sentiment can basically be described as an individual’s positive or negative attitude either towards entities generally, or certain aspects of them such as their price, usefulness etc. As described in A Practical Guide to Sentiment Analysis the reasons behind these attitudes can be broken down into two categories.
The first are emotionally-driven native preferences, which stem from motivation deeply rooted in an individual’s psyche. These are generally intangible and cannot be meaningfully scrutinised.
The second, however, relate specifically to goal-fulfilment, in which people act to meet short-term or long-term goals and have a range of distinct emotional reactions in response to achieving or not achieving them. In this case, there is a distinct traceable line of causality between an event and a sentiment.
Because of this, these goal-oriented sentiments are quantifiable and therefore of interest to marketers.
Analysing The Hierarchy of Needs
This variety of human goals and the importance that people place on meeting them is commonly placed on a scale of priority called Maslow’s Hierarchy of Needs. Maslow’s hierarchy separates deficit needs from growth needs, further refining the category of sentiments around goal fulfilment into those which are easily quantifiable and those which are not.
Growth needs refer to those aspects of human fulfilment that are more subjective, higher-level and obscure, typically related to self-actualisation. Deficit needs refer to more concrete concepts such as food, shelter, health and intimacy. These, if not possessed by an individual, will have relatively well defined and commonly understood pathways towards fulfilment.
In its current form, sentiment analysis is focused on this territory of affect which is both goal-oriented and growth-need focused. Zeroing in on this category of opinion offers the best possible terrain for algorithmic customer feedback processing.
Regular vs Comparative Opinion
Another important distinction that needs to be made when opinion mining is the difference between a regular opinion and a comparative opinion. Even if a text has been broken down into entities and aspects, as will be explained below, the results that are given can be misleading if they don’t factor in the fact that some declarations about a product may be standalone and general, while others may be relative.
Within the space of comparative opinions, just as within the space of sentiment analysis in general, there are statements that can be processed algorithmically to yield relevant and useful data, and ones that can’t.
The comparisons that fall within the former, useful category are called gradable comparisons and assume some kind of discernible hierarchy between the elements being compared. An example of this would be saying that something is better, worse or the same as another thing. If these two objects were broken down into, a value can be attributed to the relationship between them.
Comparisons which are not useful on a computational level are called non-gradable comparisons. These refer to a distinction made between two elements that do not imply a hierarchy or a grading of any kind. It instead represents a more abstract relation which cannot be as easily quantified, such as an oblique reference of a vague point of distinction.
Having covered the space within semantics and language that algorithmic opinion mining is concerned with, this post will now go over some of the ways in which this process actually tackles the task of processing opinion text.
How ARE your customers feeling?
Document-Level NLP Sentiment Analysis
Earliest forms of opinion mining were focused on providing document-level sentiment analysis. At this level, the focus was on providing insight for an entire opinion text taken as a whole. According to computer science professor Bin Liu, it is a relatively unsophisticated method which is ultimately a form of document classification. This kind of document classification can be done through supervised machine learning. However, similar results are achieved through a much more simple scoring system in which the text is compared to word-sets associated with positive and negative reviews and weighted accordingly.
The insight that this method can provide necessarily follows the assumption that the overall text expresses an opinion on a single tangible element. The limitations of this approach are obvious, since such an overall judgement is more useful to consumers rather than businesses, who are more interested in a more rigorous breakdown of which features are succeeding, which aren’t succeeding and why.
Sentence-Level NLP Sentiment Analysis
A more granular version of this is sentence-level sentiment analysis, in which a text is parsed sentence by sentence with the aim of automatically painting a more comprehensive view of the opinions being presented. Another way to describe this method is “subjectivity classification”, which firstly distinguishes between sentences that present factual information and sentences that present subjective opinion. These subjective sentences are then weighted based on whether they offer positive, negative or neutral sentiment.
Sentence-level opinion mining, however, is limited in the fact that it fails to deliver any meaningful information once the sentence starts to have any complexity to it. For example, if there is more than one sentiment expressed or if the sentence is comparative (x is better than y), the ability to use each sentence as an indicator of specific, useful information on sentiment breaks down. Although it offers more granularity than the document-level analysis, it can offer only vague and relatively simple insights.
Despite appearances, it could be a treasure trove of actionable sentiment data.
Aspect-Level NLP Sentiment Analysis
The most fine-grain, useful and challenging form of opinion mining is at the aspect or feature level. What this means is the text is probed to extract a number of very specific entities and the relationship between them: the opinion target, a particular aspect of that target, the sentiment associated with the target and the holder of the opinion. The valuable specificity of this method is the fact that it zeroes in on the entity/aspect relationship, providing sentiment information not only about a brand or a product, but also what part or feature is being singled out.
Different Ways of Extracting at an Aspect Level
There are a number of different approaches that are possible when extracting these entities or aspects via algorithm. The first is frequency based, which is based on the idea that the most relevant words and phrases are also usually the most commonly and consistently used within a large available dataset. Nouns and noun phrases are tagged and a data mining algorithm creates a list of candidates using an association mining rule.
Another method of extraction is to use a syntactic relations detecting rule. Because the relationship between an entity and aspect have a finite range of expression within the English language, aiming to discern these blocks of conjoining text will help an algorithm identify and extract the required elements.
It is also possible to extract aspects using a supervised learning algorithm. This requires a preliminary dataset that has been manually tagged by a user in advance to use as reference.
Finally, there is the topic model method which is an unsupervised specific algorithmic process which generates word clusters, referred to as “topics”. Each individual word may belong to more than one of these topics and further refinement and analysis of the data creates a sort of map which provides an insightful overview.
From a marketing perspective, if this grain of information is successfully extracted from a dataset, it provides very direct and useful information that is highly actionable.
This is above and beyond the vague sentiment scores and broader-scope insights that document-level and sentence-level processing provides. Knowing specifically about what features of a product, service or business the customer has issue with is a targeted and valuable information point.
Our follow-up post goes into the specifics of how sentiment analysis is deployed within the field of marketing to detect clients that are at risk of dropping, reducing customer churn.
Additionally, if you’d like to know more about how sentiment analysis is applied to real world solutions, watch free webinar on the topic: