Customer feedback analysis: Frequently asked questions and answers_

Published on 25|4|2022 by Lucie Kolářová & Peter Kesch

Customer feedback analysis is an extensive topic. We've compiled the most common questions from people interested in feedback analysis. Naturally, they are related to open text analytics, which SentiSquare analyzes using AI. Some of the questions are very tricky. They may be able to answer your questions as well. Let's take a look at them:

Question: When feedback is rated by 2 different people - an optimist and a pessimist, they have different ratings. How does the machine rate the texts?

Answear: The machine needs input and annotations. So if it gets annotated data from the optimist, it will rate according to it. In these contradictory cases, it is ideal to choose a so-called “super annotator”, a neutral human who will annotate the feedback in a neutral way. According to this neutral annotation, the machine will learn the right way to classify the feedback.

Comment: Text classification is in the most cases very subjective. It is up to the customer to set up the rules on how text has to be evaluated. Some customers prefer a more pessimistic approach, others a more optimistic one. This also heavily depends on the type of text and the use-case. There is no general right way to do it.

Question: Do you evaluate texts in the context of a number that the customer gives to the company (e.g. NPS)?

Answear: At SentiSquare, we work independently of the numerical rating from the customer so that the machine is not influenced by the number. It may happen that the customer gives a 9 but criticizes. Or he or she may make a mistake and give a 1, meaning grading like in the school.

Comment: Our artificial intelligence is helping companies when they need to analyze open questions such as: "What should we do to get the score 10 from you next time?" Text analytics is needed to analyze such open-ended questions, as it is virtually impossible to analyze these answers manually. The more customers you have, the more urgent this gets. Feedback Analysis is popular because it provides clear results. As a result you will know what your customers want, improve the customer experience, design better products and increase sales. SentiSquare artificial intelligence also evaluates the so-called sentiment. The Sentiment defines whether the feedback from your customer is written in a negative, neutral, positive, or mixed way.

Question: You analyze emails, what if there are multiple topics in an email? How does the AI determine the topic? Will it understand that the customer is talking about 3 topics, will it show 3 categories? Can your AI have a conversation about 3 topics with customers?

Answear: Our AI can't yet do communication with customers about 3 topics at once. However, it can recognize that there are multiple topics in an email and tag those correctly.

Comment:Our AI does not do communication with customers. We analyse communication or any type of text. However, the outcome of such analysis can be used to automate customer communication. In case there are multiple topics mentioned in an email, our AI will identify them and rate the email accordingly. As a result, the email will be rated with multiple topics and the information to which probability each of the topics is mentioned. For a classification model to be successful, we need to know what it should look for. Otherwise, it will dump a lot of incoming communication into an “unknown Other” category. With the help of our clustering functionality we can detect most common used topics based on historical data. Based on those topics we can then analyse any future communication and sort them into those categories. Of course, the AI model can be adjusted over time and can learn new topics as they appear by time.

Question: Do you analyse the tone of voice? Can you tell from the voice that the customer is angry and therefore it would be appropriate not to offer the product?

Answear: No, we really only focus on text. You can get the most information from text analysis anyway. Tone of voice has only little impact and it's also a difficult task, for example someone has a loud, aggressive voice like shouting, yet they talk like that all the time.

Comment: We trust the experience of our voice-to-text partners. It has been shown that only 10% of voice tone has an impact on the correct determination of the sentiment. The rest can also be misleading, for example a screaming customer on a tram who is just trying to overcome the surrounding noise. Or just take a lady with a soft voice who is really really angry. You'll find out the most from the content of the text anyway. Besides, whether someone is shouting in a conversation is not that important information for you. What matters is whether the customer is happy at the end of the call. And we can tell that from the text.

Now we will look at questions related to an interesting project Processing customer feedback for Rohlík.cz and Kifli.hu:

SentiSquare analyzes tens of thousands of pieces of open text feedback per month for Rohlik. SentiSquare AI automatically recognizes the topics customers are talking about in feedback. SentiSquare AI allows you to look at customer feedback in virtually real time, either in Tableau or directly in SentiSquare Analytics. The classification works for two languages - Czech and Hungarian, with more languages to come. Martin Boček, Group Head of Market Insights, Customer Experience from Rohlík group says: "SentiSquare has been very useful for us. We get feedback from the vast majority of customers. They send us an enormous amount of feedbacks, after the purchase, also via call center, email and chat communication. We are interested in all the channels through which the customer collects the experience with us."

Question: Are you collecting feedback from other channels then emails? Are there differences in those channels - are the outputs different or similar?

Answear: After a purchase, the feedbacks are both, positive and negative. The phone channel is used by the customer when they want to resolve something. Positive feedback is predominant.

Comment: SentiSquare AI can deal with any text in any language because it learns patterns directly from data. Not all NLP models can deal with difficult, messy text data – Call transcripts, for instance, are tricky as they contain a lot of errors. Emails are messy since they contain fluff such as footers etc. To create real business value, NLP engines need to be fine-tuned to work with the specific types of data they process. NLP scientists at SentiSquare have mastered the fine art of adapting NLP algorithms and preprocessing text data. Messy and difficult data gets ready for machine learning – no matter the language or channel.

Question: I see you have the keyword “moldy” in 2 categories, how is that possible?

Answear: Yes, keywords can be repeated in multiple categories if the machine determines it that way and finds them in multiple categories.

Comment: A common obstacle to building an effective classification system is not quite knowing what the content of the data is about. Sometimes there are hidden high-value patterns or unexpected trends in the data. That is why, every time we get new data, we put our AI on Discovery Mode – we use unsupervised learning to generate clusters of text pieces with similar meanings, and identify words that carry the most meaning in the dataset. This way, we quickly uncover the most important themes and patterns within the dataset, and flag possible false assumptions about the data. We use the resulting knowledge to build classification systems that reflect what customers are saying. Clustering not only provides insight to our clients to support CX improvement; it also provides a basis for building models for use cases from feedback categorisation to routing automation to churn prediction. In sum, our machine finds the best way to process data and offers solutions by itself.

Question: Customer complains about a courier who delivered the goods late, but the root cause may be an unmanaged stock situation.

Answear: The fact is that a machine doesn't know more than a human/customer writes in the text. We won't know the root cause from feedback because the client doesn't know why the courier was late.

Comment: We can, for example, compare a category from June 2020 with June 2021 and find out customer satisfaction related to that category, which will give us an answer to a question to find out satisfaction related to a change of supplier for that category.

Question: You have a link with feedbacks to the CRM to a specific person. Do you work with that at the data level?

Answear: Not now, we are not connected to a specific customer. Although that would not be a problem.

Comment: Certainly linking to a specific customer in the CRM would open up additional opportunities to use text analytics of customer feedback. Any enrichment of text data with metadata (like gender, place of residence,...) brings interesting possibilities and extensions of activities and we are happy to recommend it to our clients.

Question: And multilingual markets? Are you pairing across English?

Answear: It's searchable in English, but only for search.

Comment: To deliver superior precision and deeper understanding, we use the most advanced NLP methods. We simulate a human ability to read and understand text. Our recipe for superior NLP products is the use of distributional semantics and semi-supervised learning. This makes our AI precise and language independent.

Question: You talked about an 80% success rate. What does that mean exactly?

Answear: 20 % machine unassigned versus human, 1 out of 5 category assignments is wrong.

Comment: Comparing AI with human accuracy is tricky. Should our goal be a 100 % success rate? Imagine the case of classifying emails in a contact centre. If multiple agents carry out the same task, their agreement rate is never 100 %. Usually, there are differences in more than 15 % of cases. The 100% is therefore practically unattainable. Moreover, it is human to make mistakes - even the best flesh-and-blood operator achieves a success rate of no more than 95%. And to find a person like this for a task is divine. While accuracy remains a key metric, principal reasons for adopting AI lie elsewhere, in the disciplines where humans fail: speed, cost-efficiency, reliability and flexibility.