Creating reliable topic models with Custom BERTopic
Topic models can do a great job summarising large amounts of text. But, it’s difficult to guage how good the topic models are. In this post, I will discuss different metrics I have explored for evaluating topic models and how they can help make BERTopic a more robust approach.
What is topic model?
First, it’s useful to know how a model like BERTopic works. It’s important all users understand BERTopic is a stochastic model, which clusters sentences into a number of topics set by the user. The advantage of this model is how customisable it is. The problem is that it is stochastic, as any clustering approach is liable to be. This means that the topics output may not be stable, so every time the topic model is run different answers may be spit out.
It may be helpful to view the model as equivalent to a collection of somewhat sophisticated monkeys with typewriters to recreate your Shakespearean documents. Now, this image undersells sophisticated workflow of the model, but does convey to any user that each model is not always Shakespeare. It is doing the best job it can to cluster diverse sentences into meaningful topics. So it is important to understand the workflow for topic modelling should be more trial and error than running the model and receiving the perfect representation of the text.
Practically, BERTopic has the following steps:
- Split corpus into sentences
- Pass sentences through embedding model
- Dimensionality reduction with UMAP (fine tune : n_neighbors)
- Cluster with HDBscan (fine tune: min_cluster_size)
- Tokeniser
- Count Tokens
I have taken this from the docs, but included the first step which they do not. From my research, I found this to be a challenge and create a custom workflow for sentence splitting and normalisation which is another topic.
From this pipeline, the key arguments to change are n_neighbors which sets the number of nearest neighbours in UMAP. I have suggested the list, 5, 10, 15. BERTopic sets advises 15 as default.
min_cluster_size can go from 10 to 50, in my code with an interval of 5. This forces the size of the clusters. So you’d expect 50 to produce less topics, and 10 to produce more topics. However, this will interact with n_neighbors and min_topic size
min_topic_size is set in the BERTopic arguments. This has a similar effect to min_cluster_size and interacts with it. Thus, changing this argument and using a similar scale is useful. min_topic_size adjusts the topics after HDBscan, which is where it differs.
Depending on the amount of sentences, models with high min_cluster_size and min_topic_size can fail to produce 20 topics. As a rule of thumb, I consider any model which cannot produce 20 topics as ‘junk’. This is somewhat arbitrary, and part of a selection of criteria I have developed for evaluating topic models
🔍 What Do We Want from a Topic Model?
This is an often overlooked step in topic modelling projects. However, it is crucial to first establish what expectations we have of a “good” topic model. Without clear goals, evaluating model performance becomes subjective and inconsistent.
A high-quality topic model should exhibit the following features:
Relatively even topic counts
Distinct topics
Stability across multiple runs
High coverage of the dataset
Human interpretability
These criteria are intuitive, yet essential. Unlike numerical data, text is non-normalised and sparse, which means that traditional statistical tests (e.g. p-values) cannot be easily applied to validate model performance. Instead, we must rely on these proxy metrics to compare models across multiple runs and assess how well they meet our defined criteria.
1. Topic Count Distribution
We analyse how evenly sentences are distributed across topics. A model in which one or two topics dominate (e.g., the first topic capturing a large majority of sentences) often indicates poor topic separation and overly broad clusters.
To quantify this, we focus on:
The 20th-largest topic size, as a proxy for lower-bound topic quality.
The Gini coefficient over the top 20 topics, to assess distributional imbalance.
Using the top 20 topics is an arbitrary threshold, but provides a consistent basis for evaluation. Users may use fewer than 20 topics, but strong performance in the top 20 ensures the model’s breadth.
2. Topic Quality and Stability
Get Thomas Compton’s stories in your inbox
Join Medium for free to get updates from this writer.
Topic quality refers to how semantically distinct and repeatable the topics are.
We evaluate this in two ways:
Stability: Comparing the top 20 topic names to a reference table (e.g., from past runs or curated labels), using fuzzy matching to assess how many familiar topics reappear.
Distinctiveness: Optionally, we may compute a PUV (pairwise uniqueness value) score to estimate topic separation.
These metrics help indicate whether the model is capturing real semantic structure or simply artefacts of stochastic processes. While not definitive on their own, comparing across multiple runs enables relative performance benchmarking.
3. Corpus Coverage
We want the model to explain as much of the corpus as possible.
BERTopic’s -1 topic reflects the number of unclustered sentences — a proxy for coverage.
Keyword coverage is also assessed by comparing the top topic keywords to a pre-built term frequency dictionary. If many high-frequency n-grams appear in the top keywords, it suggests that topics reflect central corpus content.
4. Human Interpretability
Finally, and most importantly, a topic model must be useful to humans. Regardless of how well a model performs on quantitative metrics, if the output is not interpretable or meaningful to users, then it has failed.
Interpretability is inherently subjective, and cannot be fully captured by automated metrics. However, the metrics described above provide guidance and confidence that the model is coherent and robust.
| Corpus | Gini Score | Appearance Percentage (%) | Topic 20 Size | PUV | Ngram Value |
| — — — — | — — — — — — | — — — — — — — — — — — — — — | — — — — — — — — | — — — | — — — — — — — |
| B&S | 0.38 | 54.73 | 984 | 0.97 | 0.19 |
| UC | 0.28 | 58.33 | 261 | 0.95 | 0.27 |
Trade-offs and Recommendations
There is a known trade-off in BERTopic between coverage (minimising the -1 topic) and even topic distribution. Models that achieve better coverage often exhibit more skewed topic distributions, and vice versa.
We advise users not to prioritise any single metric in isolation. Instead:
Use the metrics to establish that the model performs better than average across 30 runs.
Then, apply qualitative judgement to select the most appropriate model for the task.
Running Topic Model
In the code provided, you can see the ability to run 30 models in a loop first. This creates a baseline for evaluating which model is best. I will explore the criteria in the next section. So, for this loop you won’t see the outputs.
Topic model outputs include the list of topics, their count in sentences and example sentences. So they need to be labeled through custom_topic_labels. These need to be generated by the user. The best way to do this is looking at the topic 5 words from each topic and having the label reflect that. Do refrain from creating labels based on speculation and try to be as close to these terms as possible. This step is crucial in making sure your model is credible to all audiences as the labelling stage can be controversial
You can represent the topic models visually with barcharts and umap projections found in the BERTopic docs. These are strongly advised because this visualisation is one of the best advantages of topic models and lends credibility to your models when the audience can see them. For the average reader, they will respond better to that than the topic info table put out by the model which is a little too complicated for them
Exploring the trade offs
From running the models multiple times it became clear that there is a strong relationship between the error size and all the metrics for topic model evaluation. With Gini Score here, it is clear that the more sentences are covered by the model, the worse the gini coefficient is. Simplistically, by forcing the model to create larger clusters, the clusters are becoming more skewed towards the top 10. Equally, the model is accepting more lower frequency key terms. This does suggest that artificially inflating the coverage would cause a worse model for the sake of more coverage.
At the same time, this error metric is not particularly meaningful. This is the official metric from the model, but if we are planning on only using the top 20, we should be adding the rest of the topics to that count. Essentially, by focusing on the -1 topic, we risk making worse topics in the top 20 to make better topics beyond this count that we may not use.
On the other hand, the question should be more broad. It is whether the goal of the topic model is to produce 20 sound topics or to represent all of the topics within a database. As the evidence suggests, it is impossible to do both. Most importantly, it is a question of whether we want to include low value sentences in out topic clusters, potentially diluting their meaning.
Conclusion
BERTopic is a great topic model in how customisable it is. With the rise in interest in AI, it is only fair for people to more interested in topic models. Yet, there is always a danger of misunderstanding the purpose of the topic models and the complexity of their usage can be deterring. So, the key point I wish to make is that the best model is the one which fits the quantitative criteria, falls within the top 5, but also visually works best.
It also helps to know the dataset you’re using. While topic models are great, there are still reasons to be sceptical of what they output. It should be clear that the best BERTopic model is still a guess at the topics. It does not accurately summarise everything in the text. It does, however, give some very useful insights into large quantities of text and can be used alongside other methods to produce excellent analysis
