When Poisson Isn’t Enough
A simple guide to zero-inflated Poisson (ZIP), from a classroom example to real model comparison
You’ve got count data. You fit a Poisson model. The residuals are terrible. You look at the histogram, and most of it is zeros.
Most tutorials say “use Poisson” and stop there.
But what if the zeros aren’t all the same? What if some zeros mean “this could never happen” and others mean “this could have happened, but didn’t”?
That’s where Zero-Inflated Poisson (ZIP) comes in.
Back to the Classroom
Imagine I ask 100 students, "How many times were you late to class this month?”
Suppose I get this:
Note: This is a hypothetical dataset created for teaching purposes, chosen because the numbers are clean and easy to follow.
The first thing I notice is obvious. There are a lot of zeros. 72 out of 100 students reported no late arrivals.
But that leads to a more interesting question: Are all those zeros coming from the same process?
At first, that might sound like a strange question. Zero is zero, right? Statistically, maybe not.
First, Meet Poisson
I am dealing with count data: 0, 1, 2, 3, …
A natural starting point is the Poisson distribution. Poisson answers a simple question: How many times does an event occur during a specified period or exposure?
For example:
- calls received per hour
- accidents per month
- customer complaints per day
- insurance claims per year
- purchases per hour
The Poisson model has one parameter, λ, which represents the average event rate.
For the classroom data above, the average number of late arrivals is:
λhat = 0(72)+1(12)+2(7)+3(4)+4(3)+5(2)/100 = 60/100 = 0.60
So if I fit an ordinary Poisson model, I am effectively saying, These students’ late-arrival counts can be described by one underlying count-generating process.
That assumption is about to become important.
The Problem With the Zeros
Under a Poisson model with λ = 0.60,
P(X=0)=e−0.60≈0.549
So Poisson expects about 100×0.549 ≈ 55100×0.549 ≈ 55 students to have zero late arrivals.
But I observed 72. That’s a difference of 17 students—about 17 percentage points.
Now, a word of caution: A 17-point gap on a sample of 100 students could theoretically happen by random chance with an ordinary Poisson process. I’m not claiming this gap alone is definitive proof of a problem—it's just a strong visual signal. In practice, a data scientist would follow this up with a formal test (like a score test for zero-inflation or a chi-square goodness-of-fit test) to confirm whether the gap is statistically ‘real.’ Think of this visual as the appetizer; the formal test comes later in the modeling process.
At this point, I could simply say, "Poisson doesn’t fit very well. Let’s use zero-inflated Poisson.”
But that skips the most interesting question. Instead, I want to ask: Why are there so many zeros?
Maybe the Poisson model is simply inadequate. Maybe there are other factors affecting the counts. Or maybe something more fundamental is happening: Perhaps not all zeros are being generated by the same mechanism. And that is where zero-inflated Poisson begins.
Are All Zeros Equal?
Imagine two students who both have a recorded count of zero. Student A may have had plenty of opportunities to be late but happened not to be late. Student B may belong to a situation where being late was effectively impossible under the process I am studying—for example, a student who lives on campus and has no commute.
Both observations are X=0
But the stories behind those zeros are different. I can think of them as two types.
Zero #1 — Structural Zero
A structural zero comes from the zero-producing part of the model.
The idea is that some observations belong to a group for which a positive count is structurally impossible—or outside the process being modeled.
Examples might include:
- a customer who was never exposed to the offer
- a patient who was not at risk for the event
- a business that does not receive the type of complaint being counted
- a student who officially withdrew from the course mid-semester but is still included in the final class roster—for them, ‘late arrivals’ isn’t a low number; it’s ‘not applicable’ anymore.
For these observations, zero is not simply an unusually lucky outcome from a Poisson process. It comes from the structural-zero mechanism.
Zero #2 — Poisson-Generated Zero (Chance Zero)
Now consider another observation. A positive count was possible, but the event simply did not occur during the observation period. That zero comes from the ordinary Poisson process itself.
So I now have two different routes to the same observed value: 0
One route is structural. The other is generated by the ordinary count process. That is the central idea behind zero-inflated Poisson.
The ZIP Idea
Before I give you the equation, here’s the idea in its simplest form:
I want to hold onto this picture for the rest of the article.
In an ordinary Poisson model, zero and positive counts come from the same count-generating process. In a ZIP model, zero can come from either the structural-zero branch or the Poisson branch. Positive counts only ever come from one place: the Poisson branch.
That’s easier to hold onto before I show you the equations.
The Mathematics Behind ZIP
Let:
- π = probability of belonging to the structural-zero group
- 1 − π = probability of entering the Poisson component
- λ = the Poisson mean
For zero:
P(X = 0) = π + (1 − π) · e^(−λ)
For a positive count:
P(X = k) = (1 − π) · [e^(−λ) · λ^k / k!], for k > 0
In words: the probability of zero has two pieces—one from the structural-zero group and one from the Poisson process producing a zero on its own. Positive counts have exactly one source: the Poisson component.
That’s the whole idea. I’m not going to spend a page deriving the PMF—the mathematics is here to support the story, not to become the story.
Seeing ZIP in Action: A Simulation
Here’s why this matters in practice. I’ll generate data where I know the true mechanism, because I built it myself.
import numpy as np
from scipy.stats import poisson
import matplotlib.pyplot as plt
SEED = 42
rng = np.random.default_rng(SEED)
# True parameters
n = 1000
pi_true = 0.60 # 60% are structural zeros
lambda_true = 1.5 # Poisson rate for the exposed group
# Generate ZIP data
is_structural_zero = rng.random(n) < pi_true
poisson_draws = rng.poisson(lam=lambda_true, size=n)
y = np.where(is_structural_zero, 0, poisson_draws)
# Observed zeros
observed_zero_pct = (y == 0).mean() * 100
# What would a naive Poisson fit expect?
lambda_hat = y.mean()
poisson_expected_zero_pct = poisson.pmf(0, mu=lambda_hat) * 100
print(f"Observed zero percentage: {observed_zero_pct:.1f}%")
print(f"Poisson expected zero percentage: {poisson_expected_zero_pct:.1f}%")
print(f"Gap: {observed_zero_pct - poisson_expected_zero_pct:.1f} pts")Output:
Observed zero percentage: 70.2%
Poisson expected zero percentage: 55.7%
Gap: 14.5 ptsHere’s what that looks like visually:
max_k = int(y.max())
k_values = np.arange(0, max_k + 1)
poisson_pmf_values = poisson.pmf(k_values, mu=lambda_hat) * n
fig, ax = plt.subplots(figsize=(8, 5))
ax.hist(y, bins=np.arange(-0.5, max_k + 1.5, 1),
color="#4C72B0", alpha=0.75, edgecolor="white",
label="Observed (simulated ZIP data)")
ax.plot(k_values, poisson_pmf_values,
"o--", color="#C44E52",
label=f"Naive Poisson fit (λ̂={lambda_hat:.2f})")
ax.set_xlabel("Count")
ax.set_ylabel("Frequency")
ax.set_title("Observed counts vs. naive Poisson fit\n(the 'mountain of zeros')")
ax.legend()
plt.show()
What This Simulation Does and Doesn’t Prove
I want to be explicit about the scope of this simulation.
Because I generated the data myself, I know the true mechanism:
- 60% of observations are structural zeros
- The remaining 40% follow a Poisson process with λ = 1.5
That lets me honestly say, "Poisson missed the mark because of structural zeros—I built it that way.”
Get Sawantpramod’s stories in your inbox
Join Medium for free to get updates from this writer.
But on real data, the same statistical pattern—more zeros than a Poisson fit expects—is only ever evidence consistent with a structural-zero story. It isn’t proof of one.
I can’t look at a real dataset, count the zeros, and declare that some percentage of them are structural. That argument has to come from domain knowledge—a real reason some units literally can’t produce a positive count—not from the numbers alone. The simulation shows the mechanism works and that the method can detect it when it’s really there. It doesn’t tell me a given real-world dataset has that mechanism. That’s a separate case I have to make on its own terms, later in this article.
A Real-World Example: Customer Complaints
Now let me ground this in a business problem. Imagine I’m a data scientist at a company tracking: Number of complaints received from each customer during a year.
I might observe:
- 80% → 0 complaints
- 10% → 1
- 5% → 2
- 3% → 3
- 2% → 4+
Why are there so many zeros?
Possible structural mechanism: Some customers rarely interact with the service and aren’t exposed to the complaint-generating process. Other customers are exposed and may generate 0, 1, 2, 3… complaints.
The business question: Are these customers truly satisfied, or just never exposed to the complaint process?
ZIP separates the “never complainers” from the “could complain but didn’t.” This changes how I allocate customer service resources and interpret customer satisfaction metrics.
But notice what I’m not saying: I’m not claiming that 80% of customers are structural zeros. The 80% observed is a mixture of structural zeros and Poisson-generated zeros. ZIP helps me estimate the underlying components statistically, but the interpretation—that some customers are never exposed to the complaint process—has to come from business knowledge.
Fitting Real Models in Python
Let me fit three models to the same simulated data and compare them.
Important: I’m comparing statistical support, not proving truth.
"""
Zero-Inflated Poisson (ZIP) — Simulation + Model Comparison
------------------------------------------------------------
Generates a ZIP-distributed sample with known true parameters, fits
Poisson, Zero-Inflated Poisson (ZIP), and Negative Binomial (NB) to it,
and compares them on log-likelihood, AIC, and BIC.
"""
import os
import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.discrete.count_model import ZeroInflatedPoisson
from statsmodels.discrete.discrete_model import Poisson, NegativeBinomial
# ----------------------------------------------------------------------
# Output location (shared with Part 2 so both scripts agree on paths)
# ----------------------------------------------------------------------
OUT_DIR = r"C:/Users/admin/Desktop/Medium.Com/M1/Zero-Inflated Poisson Distribution (ZIP)/PNG"
CSV_PATH = os.path.join(OUT_DIR, "zip_simulated_y.csv")
os.makedirs(OUT_DIR, exist_ok=True)
# ----------------------------------------------------------------------
# Reproducibility + true (known) data-generating parameters
# ----------------------------------------------------------------------
SEED = 42
rng = np.random.default_rng(SEED)
n = 1_000 # number of observations
pi_true = 0.60 # true probability of being a STRUCTURAL zero
lambda_true = 1.5 # true Poisson rate for the "exposed" group
# ----------------------------------------------------------------------
# Generate ZIP data
# Step A: decide who is a structural zero (Bernoulli(pi_true))
# Step B: everyone else draws from Poisson(lambda_true)
# ----------------------------------------------------------------------
is_structural_zero = rng.random(n) < pi_true
poisson_draws = rng.poisson(lam=lambda_true, size=n)
y = np.where(is_structural_zero, 0, poisson_draws)
print(f"Generated ZIP data with n={n}, pi_true={pi_true}, lambda_true={lambda_true}")
print(f"Observed zero count: {np.sum(y == 0)} out of {n} "
f"({100 * np.sum(y == 0) / n:.1f}%)")
print(f"Observed mean: {np.mean(y):.3f}")
print("-" * 70)
# ----------------------------------------------------------------------
# Persist the sample so Part 2 can reuse it without regenerating
# ----------------------------------------------------------------------
np.savetxt(CSV_PATH, y, fmt="%d", delimiter=",", header="y", comments="")
print(f"Saved simulated data to {CSV_PATH}")
# ----------------------------------------------------------------------
# Fit the three models — intercept-only for a fair comparison
# ----------------------------------------------------------------------
exog = np.ones((n, 1)) # intercept-only for all three models
poisson_model = Poisson(y, exog).fit(disp=0)
# exog_infl = predictors for the "structural zero" logit component
# (intercept-only here, matching the simple fixed-pi/lambda case
# discussed in the article)
zip_model = ZeroInflatedPoisson(y, exog, exog_infl=exog,
inflation="logit").fit(disp=0)
# Negative Binomial handles overdispersion but has ONE process — no
# separate structural-zero mechanism.
nb_model = NegativeBinomial(y, exog).fit(disp=0)
# ----------------------------------------------------------------------
# Compare on multiple criteria — never rank by a single number alone
# ----------------------------------------------------------------------
results = pd.DataFrame(
{
"Model": ["Poisson", "ZIP", "Negative Binomial"],
"Log-Likelihood": [poisson_model.llf, zip_model.llf, nb_model.llf],
"AIC": [poisson_model.aic, zip_model.aic, nb_model.aic],
"BIC": [poisson_model.bic, zip_model.bic, nb_model.bic],
"Params": [poisson_model.params.size, zip_model.params.size,
nb_model.params.size],
}
).round(2)
print("=" * 70)
print("MODEL COMPARISON — Poisson vs. ZIP vs. Negative Binomial")
print("=" * 70)
print(results.to_string(index=False))
print("-" * 70)
best_aic = results.loc[results["AIC"].idxmin(), "Model"]
best_bic = results.loc[results["BIC"].idxmin(), "Model"]
print(f"Lowest AIC: {best_aic} | Lowest BIC: {best_bic}")
print(
"\nHow to read this (deliberately cautious phrasing):\n"
f"'{best_aic} is better supported by AIC on this sample' — not\n"
"'proven correct.' AIC/BIC compare relative fit under stated\n"
"assumptions; they don't verify that a structural-zero mechanism\n"
"actually exists. That case has to be made on domain grounds\n"
"(is there a real reason some units could never generate a\n"
"positive count?), with the model comparison as supporting,\n"
"not sole, evidence."
)
Here’s what I get with the simulated data (remember, I know ZIP is the true mechanism):
How to read this: ZIP is better supported by AIC and BIC on this sample. That’s not “ZIP is proven correct"—it"'s “ZIP provides better statistical support under the criteria considered.”
And because I simulated the data, I can check the recovered parameters:
# statsmodels orders ZIP params as [inflation params..., count params...]
# with intercept-only exog, that's [inflate_const, count_const]
inflate_const, count_const = zip_model.params
pi_hat = 1 / (1 + np.exp(-inflate_const)) # inverse logit
lambda_hat_zip = np.exp(count_const) # inverse log link
print("\n" + "=" * 70)
print("ZIP recovered parameters vs. true simulation values")
print("=" * 70)
print(f"True pi = {pi_true:.2f} | ZIP-estimated pi ≈ {pi_hat:.3f}")
print(f"True lambda = {lambda_true:.2f} | ZIP-estimated lambda ≈ {lambda_hat_zip:.3f}")
print(
"\nThis recovery check is a sanity check on the METHOD (does ZIP\n"
"estimation get back the parameters we put in?), not evidence that\n"
"any given real-world dataset has a structural-zero process - that\n"
"still has to be argued from domain knowledge."
)
But—and this is crucial—this recovery check is a sanity check on the method (does ZIP estimation get back the parameters I put in?), not evidence that any given real-world dataset has a structural-zero process. That still has to be argued from domain knowledge.
The ZIP Trap
Seeing many zeros does not prove zero inflation. I cannot look at a dataset, count the zeros, and declare that some percentage of them are structural zeros. Excess zeros can have several explanations: heterogeneity, omitted variables, different exposure levels, overdispersion, dependence, measurement issues, or a genuinely separate zero-generating mechanism.
ZIP is useful when the data-generating story supports that two-component interpretation.
But wait—is it really zero inflation?
Excess zeros can arise from several things:
- Heterogeneity between observations
- Omitted variables
- Different exposure levels
- Overdispersion
- Dependence or clustering
- Measurement/data-generation issues
- A genuinely separate zero-generating mechanism
Important: Zero inflation and overdispersion are not the same thing.
A negative binomial model can handle overdispersion—it allows the variance to exceed the mean. But it still assumes all zeros come from one count process. ZIP, on the other hand, assumes zeros can come from two different processes.
The choice between these models should reflect what I know about the data-generating process, not just which model has a lower AIC.
When ZIP Is NOT the Answer
I don’t use ZIP merely because:
- My histogram has many zeros
- My data is right-skewed
- Poisson doesn’t fit perfectly
- AIC happens to be slightly lower
- I want a more sophisticated model
Instead:
The statistical model should reflect a plausible data-generating mechanism.
Before using ZIP, I ask:
- Is there a real reason some units could never generate a positive count?
- Does the improvement in fit come from a defensible structural story?
- Would Negative Binomial explain the data sufficiently?
- Do I have domain knowledge that supports a two-component interpretation?
A Note on Related Models
I’ve focused on ZIP because it’s the most direct answer to the question, "Are all zeros coming from the same process?”
But there are related models worth knowing about:
- Negative Binomial: Handles overdispersion but assumes one count process
- Zero-Inflated Negative Binomial (ZINB): Combines zero inflation with overdispersion
- Hurdle models: Treat zeros and positives as completely separate processes
These are natural extensions, and I may explore them in a follow-up piece. For now, the key insight—that zeros can have different origins—is what matters most.
The Final Lesson
The interesting thing about zero is not that it’s small. It’s that it can mean two completely different things.
A zero can mean, "The event could never happen" or “The event could have happened, but it didn't," and therefore, counting zeros is easy. Understanding where those zeros came from is the real statistical problem.
Reproducibility
Every number in this article can be reproduced exactly with the code provided. I used a fixed random seed (42) throughout to ensure consistent results. The full simulation and model-fitting code is included above.
Key Takeaways
- Poisson assumes one count process—all zeros come from the same mechanism.
- ZIP separates two types of zeros—structural zeros (can’t happen) and Poisson-generated zeros (could happen, didn’t).
- Statistical support ≠ proof—lower AIC/BIC means “better supported,” not “correct.”
- Domain knowledge matters—a plausible structural-zero story must come from the problem context, not the numbers alone.
- Excess zeros can arise from many sources—I don’t default to ZIP without checking alternatives and assumptions.
- The simulation is a demonstration, not a proof—I know the truth in the simulation because I built it. On real data, I’m only ever estimating.
What I’m Not Claiming
Let me be explicit about the boundaries of this article:
- I’m not claiming ZIP is always the right choice.
- I’m not claiming lower AIC proves ZIP is correct.
- I’m not claiming excess zeros automatically mean structural zeros exist.
- I’m not claiming my analysis proves anything about any real dataset.
The simulation is a demonstration of the method on data where I know the truth. On real data, the same workflow tells me which model is better supported, not which model is true.
That distinction—statistical support versus proof—is the whole point.
Connect With Me
If you found this analysis valuable:
- Clap 👏 for this article
- Share your thoughts in the comments
- Connect with me on Medium.com
- Follow for more such content: https://github.com/sawantpramod/Zero-Inflated-Poisson-a-reproducible-walkthrough
