You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: search.json
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@
18
18
"href": "tutorial.html",
19
19
"title": "BERTopic Tutorial",
20
20
"section": "",
21
-
"text": "Warning\n\n\n\nNe pas se focaliser que sur l’exploratoire\nNeed to rerun it all and double check comments"
21
+
"text": "What are the main issues adressed in a set of documents ? Are those documents similar or discussing different issues ? What is the most important topic ? Delineating the main topics in a set of documents is a common task in social sciences; especially when working with a new corpus. Of course, identifying the topics can be done by hand. But when dealing with large corpora, this task is almost impossible; hence, for quite some time now, social scientists have used topic modelling techniques to quickly extract the main themes in their corpus (Asmussen & Møller, 2019), either to explore them or to attribute a label to each document to conduct further analysis.\n\nTopic modelling is a natural language processing task that extracts latent topics structuring a corpora. For instance, Jockers & Mimno (2013) extracted broad themes in the English literature from the 19th century.\nNatural language processing (NLP) is a subfield of computer science that analyses textual data. Main NLP tasks are text generation (like chatGPT), text classification, or topic modelling.\n\n\n\n\n\n\n\nFigure 1: Topic Modelling in-out sketch\n\n\n\nUntil recently, topic modelling techniques — and natural language processing (NLP) in general — heavily relied on word counts, especially bag-of-words approaches. Amongst other limits, those approaches do not take into account the context in which a word is used — the order of the words in a sentence does not matter to those models, thus failing to grasp the complexity of the human language.\nTo tackle this limit, researchers developed transformations that generate richer and denser representations based on deep learning models. Those models take the text as an entry and generate a vector representation called embeddings. BERTopic is a topic model tool designed for Python that leverages transformer models1 to generate coherent topics based on the semantic similarity of texts. Since its inception in 2022, BERTopic has already proven to be useful in various research across social sciences. For instance, Bizel-Bizellot et al. (2024) used it to identify the main circumstances of infection with COVID-19 based on survey-free-text answers. In the context of climate misinformation, Törnberg & Törnberg (2025) analysed images and texts to highlight trends.\nBERTopic is a brilliant tool that unlocks diverse paths but requires time to master. In this tutorial, we focus on its general philosophy and how to use it in a social science project. We will demonstrate how to start with a text corpus and create a topic model that makes sense. As a use case, we will use the database of all the theses defended between 2010 and 2022 in France and create a topic model to describe what keeps French PhD students busy!\nBy the end of this tutorial, you should be able to:\n\nGet an idea of what you can do with topic modelling in social science\nSet up a topic model on your data and make sense of the results\nUnderstand each step of the BERTopic pipeline and customise it\n\nWe conclude this tutorial with a discussion about the evaluation of topic modelling and good practices for reproducibility.\nPython, Machine Learning and NLP prerequisites\nWe don’t assume that you have any knowledge about NLP and try our best to explain every step in an agnostic manner. We also provide numerous references if you want to dig deeper.\nNevertheless, you need to have some notions of Python. If you need to refresh your Python skills, you can use Lino Galina’s courses. We assume that:\n\nyou have a working environment and can install packages\nyou know the basic syntax of Python (functions, variables, if-statements, for loops) and you’re comfortable enough with Pandas to load your documents and proceed to simple manipulations such as creating, dropping and renaming columns and rows.\n\nIn this tutorial, we use python 3.12, and you can install the packages we will need with :\npip install -U bertopic pandas scikit-learn datasets plotly kaleido stopwordsiso nbformat ipykernel\n\n\n\n\n\n\nNote\n\n\n\nWe provide a detailed requirement file that should work for Linux and MacOS.\n\n\nMaterial\nThe tutorial comes with some material uploaded on Zenodo :\n\na notebook with all the code\nThe original dataset (which can be downloaded here).\nA clean dataset with the cleaning code."
<p>Need to rerun it all and double check comments</p>
254
-
</div>
255
-
</div>
256
242
<sectionid="introduction" class="level1">
257
243
<h1>Introduction</h1>
258
244
<p>What are the main issues adressed in a set of documents ? Are those documents similar or discussing different issues ? What is the most important topic ? Delineating the main topics in a set of documents is a common task in social sciences; especially when working with a new corpus. Of course, identifying the topics can be done by hand. But when dealing with large corpora, this task is almost impossible; hence, for quite some time now, social scientists have used <strong>topic modelling</strong> techniques to quickly extract the main themes in their corpus (Asmussen & Møller, 2019), either to explore them or to attribute a label to each document to conduct further analysis.</p>
@@ -466,7 +452,7 @@ <h2 class="anchored" data-anchor-id="breaking-down-the-process">Breaking down th
466
452
<h3class="anchored" data-anchor-id="how-to-generate-the-embeddings">How to generate the embeddings?</h3>
467
453
<p>To generate the embeddings we use encoder models. Encoder models are a type of pre-trained transformer models<ahref="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a> whose job is to encapsulate the semantic of textual data. A good example of encoder is the BERT model and all it’s successors like RoBERTa, DeBERTa … For generating embeddings, BERTopic uses <strong>SBERT</strong>. Encoder models can take in a limitted number of tokens (parts of words), this is called the <strong>context window size</strong>. For smaller models, the context window size is of about 500 tokens (200 words on average) like for BERT and larger models like ModernBERT have a context window size of 8,000 tokens.</p>
468
454
<divclass="callout callout-style-default callout-tip callout-titled" title="Learn more about embedding techniques">
<li><ahref="https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html">BERTopic documentation on embeddings</a></li>
@@ -487,7 +473,7 @@ <h3 class="anchored" data-anchor-id="how-to-generate-the-embeddings">How to gene
487
473
</div>
488
474
<p>The embeddings — ie the generated vectors, contain hundreds of dimensions (for instance, the dimension of BERT’s embeddings is 512). Clustering algorithms work poorly with this many dimensions so we need to reduce the dimensionality of the embedding space (typically between 2 and 10). To reduce the dimensionality, the BERTopic pipeline uses the <strong>UMAP</strong> algorithm for it’s ability to grasp local and global structures (McInnes et al., 2018)<ahref="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>. This mean that, despite moving from several hundreds of dimensions to only a couple, documents that are close together will stay close and distant ones will stay further apart. This is a critical step as we are heavily changing the structure of the data.</p>
489
475
<divclass="callout callout-style-default callout-tip callout-titled" title="Learn more about dimensionality reduction and UMAP">
<li><ahref="https://www.ibm.com/think/topics/dimensionality-reduction">“What is dimensionality reduction?” by Eda Kavlakoglu (IBM)</a></li>
@@ -513,7 +499,7 @@ <h3 class="anchored" data-anchor-id="how-to-generate-clusters">How to generate c
513
499
<p>The goal for the clustering algorithm is to create groups of documents that are semantically close. We are not certain that the output clusters will be “real” topics, but we tune the BERTopic pipeline in order for the clusters to be representatives of topics that are latent in our corpus.</p>
514
500
<p>Different algorithms can be used for clustering. <strong>HDBSCAN</strong> was chosen for it’s ability to detect clusters based on their density, hence it is able to detect clusters of various shapes and density. HDBSCAN also allows for documents to be labeled as noise to primarily focus on dense and coherent groups.</p>
515
501
<divclass="callout callout-style-default callout-tip callout-titled" title="Learn more about clustering techniques and HDBSCAN">
<p>To achieve that, it needs to come back to the text document and discompose it at the word level<ahref="#fn4" class="footnote-ref" id="fnref4" role="doc-noteref"><sup>4</sup></a>. It uses word-count-based techniques that will count the number of occurrences of each word<ahref="#fn5" class="footnote-ref" id="fnref5" role="doc-noteref"><sup>5</sup></a> in each document. However, there are plenty of words we don’t want to count because they do not carry much semantic information (ex: “the”, “I”, “is”, “but”, … ). These words are called stop words and are skipped.</p>
541
527
<p>To do so, we use the <code>CountVectorizer</code> object that will create a word x document matrix.</p>
<p>The word x document matrix is never used as is and needs transformation to outline relevant words. The usual transformation is called TF-IDF. This transformation raises the score of words that appear often in a document and decreases the score of words that appear in many documents. In BERTopic, they use an alternative called <strong>c-TF-IDF</strong>. This transformation raises the score of words that appear often in documents of the same group and decreases the score of words appearing in other groups. With this transformation, we retrieve words that make a group unique!</p>
600
586
<divclass="callout callout-style-default callout-tip callout-titled" title="Learn more about bag-of-words and TF-IDF">
<li><ahref="https://web.stanford.edu/~jurafsky/slp3/3.pdf">“N-gram Language Models” Chapter 3 of Speech and Language Processing by D.Jurafsky and J.H.Martin</a></li>
0 commit comments