Text Preprocessing in NLP: Bag of Words (BoW) and TF-IDF

V V Srimannarayana Reddy Tetali

18 min read

·

Oct 5, 2026

--

2

Listen

Share

Introduction

Consider the sentence:

“I love machine learning.”

A human can recognize the words, understand their relationships, and interpret the meaning of the sentence.

A traditional machine learning model cannot simply receive this sentence as input and understand it in the same way. Most traditional machine learning algorithms operate on numerical features such as:

[0.2, 1.0, 0.0, 0.7]

So before a model can use text, we need a way to convert the text into numbers.

This process is called text representation or feature extraction.

There are several ways to represent text. Two classical approaches are:

• Bag of Words (BoW) — represents text primarily using word occurrence or word counts.
• TF-IDF — assigns weights to words based on how frequently they occur in a document and how common they are across the entire collection of documents.

For example, suppose we have:

Document 1: I love machine learning
Document 2: I love Python

We can build a vocabulary containing the words appearing in these documents and then represent each document as a numerical vector.

That simple idea is the foundation of Bag of Words.

TF-IDF takes the idea further by asking not only “How often does this word appear?”, but also “How common is this word across all documents?”

Understanding these techniques gives us a useful foundation for understanding more advanced NLP methods later.

Where BoW and TF-IDF Fit in the NLP Pipeline

Before looking at Bag of Words and TF-IDF, it helps to understand where they fit into a typical NLP workflow.

Press enter or click to view image in full size

A simplified NLP pipeline can look like this:

Raw Text
   ↓
Text Cleaning
   ↓
Tokenization
   ↓
Normalization
   ↓
Feature Extraction
   ↓
Machine Learning Model

Each stage has a different purpose.

1. Raw Text

This is the original text collected from a source.

For example:

"I really enjoyed this movie!"

It may contain punctuation, uppercase letters, numbers, emojis, special characters, or other elements depending on the source.

2. Text Cleaning

Cleaning removes or modifies unwanted parts of the text according to the requirements of the task.

For example:

"I REALLY enjoyed this movie!!!"

might be converted to:

"really enjoyed this movie"

The exact cleaning operations depend on the problem. Not every NLP task requires aggressive cleaning.

3. Tokenization

Tokenization means breaking text into smaller units called tokens.

For example:

"I love NLP"

can become:

["I", "love", "NLP"]

Tokens are often words, but they can also be subwords, characters, or other units depending on the NLP system.

4. Normalization

Normalization attempts to make different forms of text more consistent.

Depending on the application, this may include converting text to lowercase, handling different word forms, or other transformations.

5. Feature Extraction

This is where BoW and TF-IDF become important.

The processed text is converted into numerical features that a machine learning algorithm can use.

For example:

"I love NLP"

might eventually become something like:

[1, 1, 1, 0]

The numbers themselves do not carry meaning like a human-readable sentence. They represent measurable features derived from the text.

6. Machine Learning Model

Once the text has been converted into numerical features, those features can be given to traditional machine learning algorithms for tasks such as classification, clustering, or retrieval.

One important distinction is worth remembering:

BoW and TF-IDF are not text-cleaning techniques. They are text representation / feature extraction techniques.

What is Text Representation?

The fundamental problem is straightforward:

Humans work naturally with language, while traditional machine learning algorithms work with numerical data.

Text representation is the process of converting text into a numerical form.

Consider:

"I love NLP"

We could identify three words:

I
love
NLP

Suppose another document contains:

"I love Python"

Now we have several unique words.

These unique words can form our vocabulary.

1. Vocabulary

A vocabulary is the collection of unique terms being used as features.

For example:

[I, love, NLP, Python]

2. Document

A document is an individual piece of text.

It could be a sentence, paragraph, email, review, article, or an entire book, depending on the application.

For example:

"I love NLP"

can be considered one document.

3. Corpus

A corpus is a collection of documents.

For example:

Document 1: I love NLP
Document 2: I love Python

Together, these documents form a corpus.

4. Features

In a text representation, vocabulary terms can become features.

If our vocabulary is:

[I, love, NLP, Python]

then each of these four words can become a feature.

5. Vector

A document can then be represented using a numerical vector.

For example:

[1, 1, 1, 0]

The first value might indicate that I occurs, the second that love occurs, the third that NLP occurs, and the fourth that Python does not occur.

This is the basic idea behind Bag of Words.

6. Document-Term Matrix

When we represent multiple documents using the same vocabulary, we can arrange their vectors into a table called a document-term matrix.

Each row represents a document.

Each column represents a vocabulary term.

The values represent some measurement of that term in the document, such as its count or TF-IDF weight.

Press enter or click to view image in full size

Part 1 — Bag of Words (BOW)

What is Bag of Words?

Bag of Words (BoW) is a text representation technique that represents a document using the words it contains and their occurrence or frequency.

The key idea is simple:

Build a vocabulary from the documents, then represent each document using numerical values that indicate how its vocabulary words occur.

The name “Bag of Words” comes from the fact that the method treats the text approximately like a bag containing words.

The method does not care about the original sequence of the words.

For example:

"The cat chased the dog"

and

"The dog chased the cat"

contain the same words with the same counts.

A basic Bag of Words representation can therefore contain similar numerical information for both sentences even though their meanings are different.

This is one of the most important limitations of BoW.

How Bag of Words Works

Consider two documents:

Document 1:
"I love NLP"

Document 2:
"I love Python"

Step 1: Collect the documents

We start with our collection of documents:

D1 = "I love NLP"
D2 = "I love Python"

Step 2: Tokenize the documents

We split the documents into words:

D1 → [I, love, NLP]
D2 → [I, love, Python]

Step 3: Create the vocabulary

We collect the unique words:

[I, love, NLP, Python]

This vocabulary determines the columns of our numerical representation.

Step 4: Count the occurrence of each vocabulary word

For Document 1:

I       → 1
love    → 1
NLP     → 1
Python  → 0

So its vector becomes:

[1, 1, 1, 0]

For Document 2:

I       → 1
love    → 1
NLP     → 0
Python  → 1

Its vector becomes:

[1, 1, 0, 1]

Step 5: Create the document-term matrix

We can put these vectors into a table:

| Document | I | love | NLP | Python |
| -------- |---| ---- | --- | ------ |
| Doc 1    | 1 | 1    | 1   | 0      |
| Doc 2    | 1 | 1    | 0   | 1      |

Now the text has been converted into numerical data.

Each row represents a document, while each column represents one vocabulary term.

Binary Occurrence vs Word Frequency

There are different ways to represent word occurrence.

1. Binary occurrence

Instead of storing the number of times a word appears, we can simply record whether it appears.

For example:

1 → word is present
0 → word is absent

If:

Document = "Python Python is useful"

then the binary representation for Python would still be:

1

because the word exists in the document.

2. Word frequency/count

We can instead count how many times the word appears.

For:

"Python Python is useful"

the count for Python would be:

2

This count-based representation is what CountVectorizer commonly provides.

The choice between binary occurrence and frequency depends on the task.

Example of BoW

Consider two documents:

Document 1:
"The cat sits on the mat"

Document 2:
"The dog sits on the mat"

First, we identify the unique words.

The vocabulary could be:

[the, cat, sits, on, mat, dog]

Now count the words in each document.

| Document | the | cat | sits | on | mat | dog |
| -------- | --- | --- | ---- | -- | --- | --- |
| Doc 1    | 2   | 1   | 1    | 1  | 1   | 0   |
| Doc 2    | 2   | 0   | 1    | 1  | 1   | 1   |

The first document becomes:

[2, 1, 1, 1, 1, 0]

The second becomes:

[2, 0, 1, 1, 1, 1]

The important point is that the computer is no longer receiving sentences. It is receiving numerical vectors.

For example, the model can see that both documents contain the, sits, on, and mat, while one contains cat and the other contains dog.

However, it does not understand that cats and dogs are animals or that changing the position of words can change the meaning of a sentence.

Implementing BoW Using Python

Scikit-learn provides a convenient class called CountVectorizer for creating Bag of Words representations.

from sklearn.feature_extraction.text import CountVectorizer

documents = [
    "I love NLP",
    "I love Python"
]

vectorizer = CountVectorizer()

X = vectorizer.fit_transform(documents)

print(vectorizer.get_feature_names_out())
print(X.toarray())

Let’s understand what each part does.

1. Importing CountVectorizer

from sklearn.feature_extraction.text import CountVectorizer

CountVectorizer is a scikit-learn class that converts a collection of text documents into a matrix of token counts.

2. Creating the documents

documents = [
    "I love NLP",
    "I love Python"
]

This is our corpus containing two documents.

3. Creating the vectorizer

vectorizer = CountVectorizer()

The vectorizer will learn the vocabulary from the documents and count the occurrences of those terms.

4. Calling fit_transform()

X = vectorizer.fit_transform(documents)

This performs two related operations.

fit means the vectorizer learns the vocabulary from the documents.

For example, it identifies terms such as:

love
nlp
python

transform means the documents are converted into numerical vectors using that vocabulary.

fit_transform() performs both operations together.

5. Getting the feature names

vectorizer.get_feature_names_out()

This returns the vocabulary terms used as features.

The output will be similar to:

['love' 'nlp' 'python']

Notice that I is not included by default. Scikit-learn's default tokenization settings ignore tokens that contain fewer than two characters, so the single-character word I is removed.

This is an important practical detail: the exact vocabulary depends not only on the input text but also on the vectorizer’s configuration.

6. Converting the matrix to an ordinary array

X.toarray()

The vectorizer normally returns a sparse matrix.

Calling toarray() converts it into a dense NumPy array so that we can easily see the values.

For these documents, the output is conceptually:

[[1 1 0]
 [1 0 1]]

The columns correspond to:

['love', 'nlp', 'python']

Therefore:

"I love NLP" → [1, 1, 0]
"I love Python" → [1, 0, 1]

The model can now work with these numerical features.

Limitations of Bag of Words

Bag of Words is simple and useful, but it has several limitations.

1. Word order is ignored

Consider:

"The dog chased the cat."

and:

"The cat chased the dog."

Although these sentences have different meanings, BoW produces similar word-count information because it does not consider the order of words.

2. Context is not captured

Consider the word:

bank

It can refer to a financial institution:

"I deposited money in the bank."

or the side of a river:

"We sat on the river bank."

BoW treats bank as the same feature in both cases and does not understand the surrounding context.

3. Semantic meaning is not understood

Words such as:

car
automobile

are treated as separate features. BoW does not automatically recognize that they can have similar meanings.

4. Large vocabulary

A large collection of documents can contain thousands or even millions of unique words. This can result in a very large number of features and a high-dimensional representation.

5. Sparse matrix

Most documents contain only a small portion of the complete vocabulary.

For example, if the vocabulary contains 50,000 words but a document contains only 100 of them, most feature values will be zero. This creates a sparse matrix.

6. Frequent words can dominate

Words that appear frequently can receive high counts, even though they may not be useful for distinguishing documents.

For example, common words such as the may occur frequently but provide little information about a document.

This creates the need for a method that gives less importance to common words and more importance to distinctive ones. TF-IDF addresses this idea.

7. Synonyms are treated as different words

BoW treats:

car
automobile

as separate features. It does not automatically recognize that they can represent similar concepts.

Therefore, although BoW converts text into numerical features, it does not truly understand context or semantic meaning.

Part 2 — TF-IDF

TF-IDF (Term Frequency–Inverse Document Frequency) is a text representation technique used to assign importance to words in a document.

Unlike simple word counting in BoW, TF-IDF gives higher importance to words that are frequent in a particular document but uncommon across the collection of documents.

Why TF-IDF?

BoW mainly counts how many times each word appears. However, some words occur frequently in almost every document and may not be useful for distinguishing between documents.

For example:

"The movie was good."
"The movie was interesting."
"The movie was exciting."

Words such as the and movie appear repeatedly, but they may not help identify what makes each document different.

TF-IDF reduces the importance of common words and gives more importance to words that are relatively unique or informative.

What is TF-IDF?

TF-IDF combines two measures:

• TF (Term Frequency): How often a word appears in a document.
• IDF (Inverse Document Frequency): How uncommon the word is across all documents.

The basic idea is:

High TF + High IDF = Important word

A word that appears frequently in one document but rarely in other documents receives a higher TF-IDF value.

1. Term Frequency (TF)

Term Frequency (TF) measures how often a particular term occurs within a document.

The basic formula is:

Press enter or click to view image in full size

Where:

• t = term
• d = document

Example : Suppose a document contains 10 words, and python appears 2 times.

Press enter or click to view image in full size

So, the term frequency of python is 0.2.

The more frequently a term occurs within a document, the higher its TF value.

However, TF alone has a limitation: a word may occur frequently in many documents. High frequency does not necessarily mean that the word is distinctive. IDF helps solve this problem.

2. Inverse Document Frequency (IDF)

Inverse Document Frequency (IDF) measures how common or rare a term is across the entire collection of documents.

First, we consider document frequency (df), which is the number of documents containing a particular term.

For example, in a corpus of 100 documents:

df(the)        = 95
df(python)     = 10
df(tensorflow) = 2

The basic IDF formula is:

Press enter or click to view image in full size

Example 1: Word appears in every document

Press enter or click to view image in full size

The word receives a very low IDF because it does not help distinguish between documents.

Example 2: Word appears in fewer documents

Press enter or click to view image in full size

This is larger because the word is less common across the corpus.

Key idea

• Common word → lower IDF
• Rare/distinctive word → higher IDF

The exact IDF values used by libraries can differ because implementations may use smoothing or different mathematical conventions.

TF-IDF Formula

TF and IDF are combined to calculate the final TF-IDF value:

Press enter or click to view image in full size

Therefore, TF-IDF depends on:

• how frequently the term occurs within the document
• how rare the term is across the document collection

A high TF-IDF value generally indicates that a word is important or distinctive for a particular document.

A low TF-IDF value generally occurs when the word is common across many documents, occurs less frequently in the document, or both.

Main idea

Frequent in this document
          +
Uncommon across documents
          ↓
     Higher TF-IDF

Whereas:

Common across many documents
          ↓
       Lower IDF
          ↓
     Lower TF-IDF

This is the main difference between TF-IDF and simple word-count representations.

How TF-IDF Works Step by Step

Consider the corpus:

Document 1: "machine learning is useful"
Document 2: "machine learning is powerful"
Document 3: "deep learning is useful"

The vocabulary can be:

[machine, learning, is, useful, powerful, deep]

Step 1: Identify the terms

Create a vocabulary from all the documents.

Step 2: Calculate TF

For Document 1:

"machine learning is useful"

There are 4 terms, and each appears once:

Press enter or click to view image in full size

The same process is applied to the other documents.

Step 3: Calculate Document Frequency

Count how many documents contain each term:

machine  → 2 documents
learning → 3 documents
is       → 3 documents
useful   → 2 documents
powerful → 1 document
deep     → 1 document

Step 4: Calculate IDF

There are: N = 3

For learning: df = 3 and IDF(learning)=log⁡(3/3)=0

For powerful: df = 1 and IDF(powerful) = log⁡(3/1) = log⁡(3)

Thus, powerful gets a larger IDF because it appears in only one document.

Step 5: Calculate TF-IDF

For any term:

Press enter or click to view image in full size

Since powerful occurs only in Document 2, it can become a distinctive feature for that document.

Remember:

Frequent in one document
          +
Rare across the corpus
          ↓
     Higher TF-IDF

Example of TF-IDF

Consider three documents:

Document 1: "Python is popular for data science"
Document 2: "Python is popular for machine learning"
Document 3: "Python is popular for web development"

The words:

Python
is
popular
for

appear in every document, so they provide relatively little information for distinguishing the documents.

Now consider:

data
machine
web

Each is associated with a particular document in this example, so TF-IDF can give these terms greater importance.

Press enter or click to view image in full size

The exact numerical values depend on the TF-IDF calculation and implementation.

TF-IDF does not simply measure how frequently a word occurs. It measures its frequency in relation to the entire document collection.

Implementing TF-IDF Using TfidfVectorizer

Scikit-learn provides TfidfVectorizer for creating TF-IDF representations.

from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    "machine learning is useful",
    "machine learning is powerful",
    "deep learning is useful"
]

vectorizer = TfidfVectorizer()

X = vectorizer.fit_transform(documents)

print(vectorizer.get_feature_names_out())
print(X.toarray())

1. Importing TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

TfidfVectorizer converts a collection of text documents into a matrix of TF-IDF features.

2. Creating the documents

documents = [
    "machine learning is useful",
    "machine learning is powerful",
    "deep learning is useful"
]

These three strings form our corpus.

3. Creating the vectorizer

vectorizer = TfidfVectorizer()

This creates the TF-IDF vectorizer, which learns the vocabulary and calculates TF-IDF weights.

4. Using fit_transform()

X = vectorizer.fit_transform(documents)

fit_transform() performs two operations:

1. Fit → learns the vocabulary from the documents.
2. Transform → converts the documents into numerical TF-IDF vectors.

The result is stored in X.

5. Getting Feature Names

vectorizer.get_feature_names_out()

This returns the vocabulary terms corresponding to the columns of the matrix.

Example output:

['deep' 'is' 'learning' 'machine' 'powerful' 'useful']

The exact ordering follows the vectorizer’s feature ordering.

6. Viewing the TF-IDF Matrix

X.toarray()

This converts the sparse matrix into a regular NumPy array so the TF-IDF values can be viewed.

The resulting matrix contains TF-IDF weights rather than simple word counts.

Scikit-learn performs the TF-IDF calculation internally, so we generally do not need to manually calculate every TF and IDF value when using TfidfVectorizer.

Limitations of TF-IDF

TF-IDF improves on simple word counting, but it still has several limitations.

1. Word order is ignored

TF-IDF represents documents using individual words and does not consider their order.

For example:

"The dog chased the cat."
"The cat chased the dog."

Both sentences contain the same words, so TF-IDF cannot fully capture the difference in meaning caused by word order.

2. Context is not captured

TF-IDF treats a word as the same feature wherever it appears.

For example, the word:

bank

can refer to a financial institution or the side of a river. TF-IDF does not understand this contextual difference.

3. Semantic meaning is not understood

TF-IDF treats related words as separate features.

For example:

car
automobile

are represented as different words even though they can have similar meanings.

4. High-dimensional representation

A large collection of documents can contain thousands or millions of unique words. This can create a large number of features and increase the dimensionality of the data.

5. Sparse representation

Most documents contain only a small part of the complete vocabulary. As a result, many TF-IDF values are zero, producing a sparse matrix.

6. No understanding of relationships between words

TF-IDF assigns weights to individual words but does not learn relationships between words or concepts. It cannot understand deeper linguistic relationships such as synonyms, antonyms, or sentence meaning.

7. Vocabulary dependency

TF-IDF depends on the vocabulary found in the training documents. Words that are not present in the learned vocabulary cannot be represented by the resulting feature matrix.

Therefore, TF-IDF is useful for creating weighted numerical features, but it does not capture word order, context, or deeper semantic relationships.

BOW vs TF-IDF

Both Bag of Words (BoW) and TF-IDF convert text into numerical feature representations, but they assign values differently.

Press enter or click to view image in full size

When Should You Use BoW vs TF-IDF?

Both BoW and TF-IDF convert text into numerical features, but they are useful in different situations.

Use BoW when:

• You want a simple and easy-to-understand text representation.
• The frequency of words is important for your task.
• You are working with a relatively small vocabulary or dataset.
• You need a quick baseline model for a text classification problem.
• You want to know how many times words occur in each document.

For example, BoW can be useful for simple text classification, document analysis, and frequency-based tasks.

Use TF-IDF when:

• Common words should have less importance.
• You want to emphasize distinctive or informative words.
• You are working with a collection of documents where some words appear frequently across many documents.
• You want a representation that considers both word frequency within a document and word rarity across documents.
• You are building traditional NLP or machine learning models such as text classification, document similarity, or information retrieval.

The choice depends on the task. BoW is simpler and directly represents word counts, while TF-IDF provides weighted values that reduce the importance of words that are common across documents.

Modern Alternatives

BoW and TF-IDF are useful traditional methods for text representation, but modern NLP systems use techniques that can capture more information about context and meaning.

1. Word Embeddings

Word embeddings represent words as dense numerical vectors. Words with similar meanings tend to have similar vector representations.

Examples include: Word2Vec, GloVe, FastText

Unlike BoW and TF-IDF, word embeddings can capture relationships between words based on their usage in text.

2. Contextual Embeddings

Contextual embeddings generate word representations based on the surrounding words.

For example, the representation of bank can change depending on whether the sentence discusses money or a river.

Models such as BERT use this approach to capture contextual information.

3. Transformer-Based Models

Transformers are a modern architecture widely used in NLP.

Examples include: BERT, GPT, T5

These models can process context and relationships between words more effectively than traditional methods such as BoW and TF-IDF.

4. Sentence and Document Embeddings

Instead of representing individual words, modern methods can represent an entire sentence or document as a dense vector.

These representations are useful for tasks such as:

• Semantic similarity
• Document search
• Clustering
• Question answering

Traditional vs Modern Representation

Press enter or click to view image in full size

Modern methods generally provide richer representations than BoW and TF-IDF, especially when understanding context and semantic relationships is important. However, BoW and TF-IDF remain useful because they are simple, interpretable, and effective for many traditional NLP and machine learning tasks.

Conclusion

Text needs to be converted into numerical form before it can be processed by most machine learning algorithms. Bag of Words (BoW) and TF-IDF are two important traditional techniques used for this purpose.

BoW represents text using word counts, making it simple and easy to understand. However, it does not consider word order, context, or semantic relationships. TF-IDF improves on basic word counting by giving lower importance to common words and higher importance to words that are more distinctive within a collection of documents.

Both methods are useful for traditional NLP tasks such as text classification, document analysis, similarity, and information retrieval. However, they still have limitations because they do not fully capture context and deeper meaning.

Modern techniques such as word embeddings, contextual embeddings, and Transformer-based models provide richer representations by considering relationships between words and their surrounding context.

Understanding BoW and TF-IDF is therefore important because they provide a strong foundation for learning how text is converted into numerical features and how modern NLP representations have evolved beyond simple word-based methods.