The task is to classify the sentiment of potentially long texts for several aspects. The key idea is to build a modern NLP package which supports explanations of model predictions. The approximated decision explanations help you to infer how reliable predictions are. The package is standalone, scalable, and can be freely extended to your needs. We sum up thoughts in the article:

Do You Trust in Aspect-Based Sentiment Analysis? Testing and Explaining Model Behaviors

There are over 100 repositories on GitHub around sentiment analysis ^{1 2 3 4 5 6 7 8 9} . All of them are hard to commercialize and reuse open-source research projects. We clean up this excellent research. Please give a star if you like the project. This is important to keep this project alive.

The aim is to classify the sentiments of a text concerning given aspects. We have made several assumptions to make the service more helpful. Namely, the text being processed might be a full-length document, the aspects could contain several words (so may be defined more precisely), and most importantly, the service should provide an approximate explanation of any decision made, therefore, a user will be able to immediately infer the reliability of a prediction.

import aspect_based_sentiment_analysis as absa

nlp = absa.load()
text = ("We are great fans of Slack, but we wish the subscriptions "
        "were more accessible to small startups.")

slack, price = nlp(text, aspects=['slack', 'price'])
assert price.sentiment == absa.Sentiment.negative
assert slack.sentiment == absa.Sentiment.positive

Above is an example of how quickly you can start to benefit from our open-source package. All you need to do is to call the load function which sets up the ready-to-use pipeline nlp. You can explicitly pass the model name you wish to use (a list of available models is below), or a path to your model. In spite of the simplicity of using fine-tune models, we encourage you to build a custom model which reflects your data. The predictions will be more accurate and stable.

The pipeline provides an easy-to-use interface for making predictions. Even a highly accurate model will be useless if it is unclear how to correctly prepare the inputs and how to interpret the outputs. To make things clear, we have introduced a pipeline that is closely linked to a model. It is worth to know how to deal with the whole process, especially if you plan to build a custom model.

The diagram above illustrates an overview of the pipeline stages. As usual, at the very beginning, we pre-process the inputs. We convert the text and the aspects into a task which keeps examples (pairs of a text and an aspect) that we can then further tokenize, encode and pass to the model. The model makes a prediction, and here is a change. Instead of directly post-processing the model outputs, we have added a review process wherein the independent component called the professor supervises and explains a model prediction. The professor might dismiss a model prediction if the model internal states or outputs seem suspicious. In the article [here], we discuss in detail how the model and the professor work.

import aspect_based_sentiment_analysis as absa

name = 'absa/classifier-rest-0.2'
model = absa.BertABSClassifier.from_pretrained(name)
tokenizer = absa.BertTokenizer.from_pretrained(name)
professor = absa.Professor(...)     # Explained in detail later on.
text_splitter = absa.sentencizer()  # The English CNN model from SpaCy.
nlp = absa.Pipeline(model, tokenizer, professor, text_splitter)

# Break down the pipeline `call` method.
task = nlp.preprocess(text=..., aspects=...)
tokenized_examples = nlp.tokenize(task.examples)
input_batch = nlp.encode(tokenized_examples)
output_batch = nlp.predict(input_batch)
predictions = nlp.review(tokenized_examples, output_batch)
completed_task = nlp.postprocess(task, predictions)

Above is an example how to initialize the pipeline directly, and we revise in code the process being discussed by exposing what calling the pipeline does under the hood. We have omitted a lot of insignificant details but there's one thing we would like to highlight. The sentiment of long texts tends to be fuzzy and neutral. Therefore, you might want to split a text into smaller independent chunks, sometimes called spans. These could include just a single sentence or several sentences. It depends on how the text_splitter works. In this case, we are using the SpaCy CNN model, which splits a document into single sentences, and, as a result each sentence can then be processed independently. Note that longer spans have richer context information, so a model will have more information to consider. Please take a look at the pipeline details here.

It's time to explain model reasoning, something which is extremely hard. The key concept is to frame the problem of explaining a model decision as an independent task wherein an aux. model, the pattern recognizer, predicts patterns (weighted compositions of tokens, presented below) given model inputs, outputs, and internal states. Due to time constraints, at first we did not want to research and build a trainable pattern recognizer. Instead, we decided to start with a pattern recognizer that originates from our observations, prior knowledge. The model, the aspect-based sentiment classifier, is based on the transformer architecture wherein self-attention layers hold the most parameters. Therefore, one might conclude that understanding self-attention layers is a good proxy to understanding a model as a whole. Accordingly, there are many articles that show how to explain a model decision in simple terms, using attention values (internal states of self-attention layers) straightforwardly. Inspired by these articles, we have also analyzed attention values (processing training examples) to search for any meaningful insights. This exploratory study has led us to create the BasicPatternRecognizer (details are here).

import aspect_based_sentiment_analysis as absa

recognizer = absa.aux_models.BasicPatternRecognizer()
nlp = absa.load(pattern_recognizer=recognizer)
completed_task = nlp(text=..., aspects=['slack', 'price'])
slack, price = completed_task.examples

absa.summary(slack)
absa.display(slack.review)

absa.summary(price)
absa.display(price.review)

The explanations are only useful if they are correct. To form the basic pattern recognizer, we have made several assumptions (prior beliefs), therefore we should be careful about interpreting the explanations too literally. Even if the attention values have thought-provoking properties, for example, they encode rich linguistic relationships, there is no proven chain of causation. There are a lot of articles that illustrate various concerns why drawing conclusions about model reasoning directly from attentions might be misleading. In the article here, we validate and analyse explanations in detail.

In the table below, we present the State of the Art results on the SemEval 2014 evaluation dataset (dataset details are here). There are two available models for the restaurant and the laptop domains. The model implementation details here. The hyper-parameters optimization (with the explanation how to train a model) is here. You can easily reproduce our evaluations, look at the performance tests here.

You can use the pip:

pip install aspect-based-sentiment-analysis

Otherwise, clone the code and create the new environment via conda:

git clone [email protected]:ScalaConsultants/Aspect-Based-Sentiment-Analysis.git
conda env create -f=environment.yml
conda activate Aspect-Based-Sentiment-Analysis

The package works with the Python in the version 3.7 (the same as in Colab 2021).

How to use language models in the Aspect-Based Sentiment Analysis:

• Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence (NAACL 2019) [code][paper]
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (NAACL 2019) [code][paper]
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis [code][paper]

Introduction to the BERT interpretability:

• Are Sixteen Heads Really Better than One? [code][paper]
• A Primer in BERTology: What we know about how BERT works [paper]
• What Does BERT Look At? An Analysis of BERT's Attention [code][paper]
• Visualizing and Measuring the Geometry of BERT [code][paper]
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment [paper]
• Adversarial Training for Aspect-Based Sentiment Analysis with BERT [paper]
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT [paper]
• exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models [code][paper]
• Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings [code][paper]
• Attention is not Explanation [code][paper]
• Attention is not not Explanation [code][paper][blog post]
• Hierarchical interpretations for neural network predictions [code][paper]
• Analysis Methods in Neural NLP [code][paper]
• Visualization for Sequential Neural Networks with Attention [code]
• NeuroX: Toolkit for finding and analyzing important neurons in neural networks [code][paper]

The State of the Art results:

• A Multi-task Learning Model for Chinese-oriented Aspect Polarity Classification and Aspect Term Extraction [code][paper]
• Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification [code][paper]
• Adversarial Training for Aspect-Based Sentiment Analysis with BERT [code][paper]

Other interesting:

• Multi-Dimensional Explanation of Ratings from Reviews [paper]
• Extracting Syntactic Trees from Transformer Encoder Self-Attentions [paper]
• Master Thesis: Transfer and Multitask Learning for Aspect-Based Sentiment Analysis Using the Google Transformer Architecture [code]
• Create interactive textual heat maps for Jupiter notebooks [code]
• A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc [code]
• More you can find here.

Developed by Scalac