Coherence Score of a Topic Model

The coherence score is a measure used to evaluate the quality of topics generated by a topic model. It quantifies the degree of semantic similarity between the words that define the topic. A topic model that generates topics with a high coherence score suggests that the topic is semantically consistent and therefore more interpretable. Here is an example.

Let us say you are building a topic model on a collection of news articles, and one of the topics generated has the following top words: Apple, Watermelon, Banana, Grape. Now, consider another topic with these top words: Apple, Computer, Watermelon, Mouse. The first topic clearly revolves around fruits. Words in this topic are closely related to each other in terms of meaning, so will get a high coherence score. The second topic seems mixed. While “Apple” and “Watermelon” are fruits, “Computer” and “Software” are related to technology. Since these words are less semantically consistent as a group, this topic will receive a lower coherence score.

Coherence score is calculated by measuring pairwise word similarity scores for the top words in a topic, and then averaging them. The similarity scores are often based on word co-occurrence statistics in the text corpus.