diff --git a/README.md b/README.md index e2a96b68..6866dd4d 100644 --- a/README.md +++ b/README.md @@ -9,12 +9,12 @@ # BERTopic - + BERTopic is a topic modeling technique that leverages 🤗 transformers and c-TF-IDF to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions. -BERTopic supports all kinds of topic modeling techniques: +BERTopic supports all kinds of topic modeling techniques: @@ -46,7 +46,7 @@ BERTopic supports all kinds of topic modeling techniques: Corresponding medium posts can be found [here](https://medium.com/data-science/topic-modeling-with-bert-779f7db187e6?sk=0b5a470c006d1842ad4c8a3057063a99 ), [here](https://medium.com/data-science/using-whisper-and-bertopic-to-model-kurzgesagts-videos-7d8a63139bdf?sk=b1e0fd46f70cb15e8422b4794a81161d ) and [here](https://medium.com/data-science/interactive-topic-modeling-with-bertopic-1ea55e7d73d8?sk=03c2168e9e74b6bda2a1f3ed953427e4 -). For a more detailed overview, you can read the [paper](https://arxiv.org/abs/2203.05794) or see a [brief overview](https://maartengr.github.io/BERTopic/algorithm/algorithm.html). +). For a more detailed overview, you can read the [paper](https://arxiv.org/abs/2203.05794) or see a [brief overview](https://maartengr.github.io/BERTopic/algorithm/algorithm.html). ## Installation @@ -75,8 +75,8 @@ pip install bertopic[vision] For a *light-weight installation* without transformers, UMAP and/or HDBSCAN (for training with Model2Vec or inference), see [this tutorial](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html#lightweight-installation). ## Getting Started -For an in-depth overview of the features of BERTopic -you can check the [**full documentation**](https://maartengr.github.io/BERTopic/) or you can follow along +For an in-depth overview of the features of BERTopic +you can check the [**full documentation**](https://maartengr.github.io/BERTopic/) or you can follow along with one of the examples below: | Name | Link | @@ -99,7 +99,7 @@ We start by extracting topics from the well-known 20 newsgroups dataset containi ```python from bertopic import BERTopic from sklearn.datasets import fetch_20newsgroups - + docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data'] topic_model = BERTopic() @@ -120,7 +120,7 @@ Topic Count Name ... ``` -The `-1` topic refers to all outlier documents and are typically ignored. Each word in a topic describes the underlying theme of that topic and can be used +The `-1` topic refers to all outlier documents and are typically ignored. Each word in a topic describes the underlying theme of that topic and can be used for interpreting that topic. Next, let's take a look at the most frequent topic that was generated: ```python @@ -136,7 +136,7 @@ for interpreting that topic. Next, let's take a look at the most frequent topic ('software', 0.0034415334250699077), ('email', 0.0034239554442333257), ('pc', 0.003047105930670237)] -``` +``` Using `.get_document_info`, we can also extract information on a document level, such as their corresponding topics, probabilities, whether they are representative documents for a topic, etc.: @@ -151,7 +151,7 @@ Think! It's the SCSI card doing... 49 49_windows_drive_dos_file windows - dr 1) I have an old Jasmine drive... 49 49_windows_drive_dos_file windows - drive - docs... 0.038983 ... ``` -**`🔥 Tip`**: Use `BERTopic(language="multilingual")` to select a model that supports 50+ languages. +**`🔥 Tip`**: Use `BERTopic(language="multilingual")` to select a model that supports 50+ languages. ## Fine-tune Topic Representations @@ -177,18 +177,18 @@ representation_model = OpenAI(client, model="gpt-4o-mini", chat=True) topic_model = BERTopic(representation_model=representation_model) ``` -**`🔥 Tip`**: Instead of iterating over all of these different topic representations, you can model them simultaneously with [multi-aspect topic representations](https://maartengr.github.io/BERTopic/getting_started/multiaspect/multiaspect.html) in BERTopic. +**`🔥 Tip`**: Instead of iterating over all of these different topic representations, you can model them simultaneously with [multi-aspect topic representations](https://maartengr.github.io/BERTopic/getting_started/multiaspect/multiaspect.html) in BERTopic. ## Visualizations -After having trained our BERTopic model, we can iteratively go through hundreds of topics to get a good -understanding of the topics that were extracted. However, that takes quite some time and lacks a global representation. Instead, we can use one of the [many visualization options](https://maartengr.github.io/BERTopic/getting_started/visualization/visualization.html) in BERTopic. -For example, we can visualize the topics that were generated in a way very similar to +After having trained our BERTopic model, we can iteratively go through hundreds of topics to get a good +understanding of the topics that were extracted. However, that takes quite some time and lacks a global representation. Instead, we can use one of the [many visualization options](https://maartengr.github.io/BERTopic/getting_started/visualization/visualization.html) in BERTopic. +For example, we can visualize the topics that were generated in a way very similar to [LDAvis](https://github.com/cpsievert/LDAvis): ```python topic_model.visualize_topics() -``` +``` @@ -208,18 +208,18 @@ You can swap out any of these models or even remove them entirely. The following ## Functionality -BERTopic has many functions that quickly can become overwhelming. To alleviate this issue, you will find an overview -of all methods and a short description of its purpose. +BERTopic has many functions that quickly can become overwhelming. To alleviate this issue, you will find an overview +of all methods and a short description of its purpose. ### Common -Below, you will find an overview of common functions in BERTopic. +Below, you will find an overview of common functions in BERTopic. -| Method | Code | +| Method | Code | |-----------------------|---| | Fit the model | `.fit(docs)` | | Fit the model and predict documents | `.fit_transform(docs)` | | Predict new documents | `.transform([new_doc])` | -| Access single topic | `.get_topic(topic=12)` | +| Access single topic | `.get_topic(topic=12)` | | Access all topics | `.get_topics()` | | Get topic freq | `.get_topic_freq()` | | Get all topic information| `.get_topic_info()` | @@ -238,9 +238,9 @@ Below, you will find an overview of common functions in BERTopic. ### Attributes -After having trained your BERTopic model, several attributes are saved within your model. These attributes, in part, -refer to how model information is stored on an estimator during fitting. The attributes that you see below all end in `_` and are -public attributes that can be used to access model information. +After having trained your BERTopic model, several attributes are saved within your model. These attributes, in part, +refer to how model information is stored on an estimator during fitting. The attributes that you see below all end in `_` and are +public attributes that can be used to access model information. | Attribute | Description | |------------------------|---------------------------------------------------------------------------------------------| @@ -260,7 +260,7 @@ public attributes that can be used to access model information. ### Variations There are many different use cases in which topic modeling can be used. As such, several variations of BERTopic have been developed such that one package can be used across many use cases. -| Method | Code | +| Method | Code | |-----------------------|---| | [Topic Distribution Approximation](https://maartengr.github.io/BERTopic/getting_started/distribution/distribution.html) | `.approximate_distribution(docs)` | | [Online Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/online/online.html) | `.partial_fit(doc)` | @@ -277,11 +277,11 @@ There are many different use cases in which topic modeling can be used. As such, ### Visualizations -Evaluating topic models can be rather difficult due to the somewhat subjective nature of evaluation. -Visualizing different aspects of the topic model helps in understanding the model and makes it easier -to tweak the model to your liking. +Evaluating topic models can be rather difficult due to the somewhat subjective nature of evaluation. +Visualizing different aspects of the topic model helps in understanding the model and makes it easier +to tweak the model to your liking. -| Method | Code | +| Method | Code | |-----------------------|---| | Visualize Topics | `.visualize_topics()` | | Visualize Documents | `.visualize_documents()` | @@ -293,7 +293,7 @@ to tweak the model to your liking. | Visualize Term Score Decline | `.visualize_term_rank()` | | Visualize Topic Probability Distribution | `.visualize_distribution(probs[0])` | | Visualize Topics over Time | `.visualize_topics_over_time(topics_over_time)` | -| Visualize Topics per Class | `.visualize_topics_per_class(topics_per_class)` | +| Visualize Topics per Class | `.visualize_topics_per_class(topics_per_class)` | ## Citation
Guided