“Which airport is closest to this wildfire?”

A language model can explain what an airport is, describe wildfire risk, and produce a convincing answer. But finding the closest airport requires something more specific: comparing geometries under a defined distance rule.

Closest to the fire’s centroid, its boundary, or any point in its footprint? Which airports are candidates? Does “closest” mean straight-line distance or travel time?

I started with a narrower problem: build an answer key that another person could run and inspect before asking any models to answer the questions.

The result is a public, reproducible spatial benchmark: 240 questions, executable PostGIS queries, a raw reference-results CSV, an executed analysis notebook, a model-evaluation script, and two figures generated from actual database outputs.

What is complete: the benchmark, the PostGIS answer key, and the figures. What is pending: LLM evaluation. No models have been scored, and this article makes no empirical claim that a particular model fails at geography.

An answer key needs a contract

PostGIS returns an answer for particular inputs, predicates, units, and parameters. Those choices are part of the experiment.

For distance questions, the benchmark uses longitude/latitude airport coordinates and the geography overload of PostGIS distance functions. For topology questions, it uses exact synthetic fixtures with declared projected coordinates. “Inside” means strict point containment; boundary contact does not count. “Touches” includes corner-only contact.

Each question stores its natural-language prompt, candidate geometry context, answer type, scene identifier, and SQL query. The model runner sends the question and geometry context, while excluding SQL and expected answers.

The real-world input is a frozen, 40-airport subset of OurAirports’ public-domain data. The selection is explicit: large airports with scheduled service in a contiguous-US bounding box, sorted by airport identifier, taking the first 40. These are the eligible candidates, not a claim to represent every US airport.

The repository contains the small subset and a manifest with the upstream URL, retrieval timestamp, selection rule, and checksums. It links the full upstream dataset instead of uploading the large source file.

The synthetic inputs are clearly labeled: original rectangles, points, lines, and polygon pairs. They are not actual county boundaries, roads, or wildfire perimeters. This first release does not include wildfire data.

Eight tasks, 30 questions each

Nearest candidate: Return the one eligible airport with the smallest spheroidal distance to the reference point.

Radius: Return every eligible airport at most the specified number of meters away, including an empty array when none qualify.

Ranking: Return the nearest three airport IDs in order, resolving equal distances by ID.

Direction: Return airports with strictly greater latitude and strictly smaller longitude than the reference. This defines a northwest coordinate quadrant, not a compass-bearing sector.

Containment: Determine whether a point is strictly inside a synthetic polygon.

Intersection: Determine whether a line and polygon share any point, including boundary contact.

Adjacency: Determine whether polygon boundaries meet while their interiors remain disjoint.

Multi-step selection: Filter points by strict containment and a distance threshold, then rank the qualifying IDs.

Those 240 questions come from 60 base scenes with related tasks. The synthetic scenes translate a small number of deliberately simple patterns. That makes the tests inspectable, but it does not make them 240 independent geographies or a comprehensive assessment of spatial reasoning.

One nearest airport is not a radius result

For an arbitrary reference point near Los Angeles, at longitude −118.3 and latitude 34.1, PostGIS returned KBUR as the nearest eligible candidate, about 12.6 km away. KLAX was about 20.1 km away.

A nearest-one query returns KBUR. A 100 km radius query returns both KBUR and KLAX within this frozen candidate set. The next candidate shown, KFAT, is about 323.6 km away.

The distinction is easy to explain in English. The benchmark asks whether an answer preserves that distinction when it must enumerate the correct IDs.

A representative radius query is:

SELECT ident
FROM airports
WHERE ST_DWithin(
  geom::geography,
  ST_SetSRID(ST_MakePoint(-118.3, 34.1), 4326)::geography,
  100000
)
ORDER BY ident;

The geography threshold is in meters. The geometry overload uses the units of its reference system. Treating unprojected longitude/latitude geometry as meters changes the question. The PostGIS documentation makes that distinction explicit.

A boundary point changes the answer

The second figure uses a simple square and four exact points. These are synthetic geometries evaluated by PostGIS, not illustrative model scores.

For the interior point, all four predicates are true. For the boundary point, Contains is false while Covers and Intersects are true. A point 0.5 projected meters outside the polygon is within 1 m, but it does not intersect the polygon.

“Near,” “inside,” and “intersects” therefore cannot be treated as interchangeable descriptions. The same pair of objects can satisfy a distance threshold without intersecting.

For point-in-polygon tests, ST_Contains excludes a point lying exactly on the boundary; ST_Covers provides the boundary-inclusive alternative. ST_Intersects tests for a shared point. A question about a road crossing through an interior needs a separately declared interpretation.

What actually ran

I generated the 240 questions, executed every reference query, and saved every answer in ground_truth_results.csv. The synthetic questions also passed 120 analytic fixture assertions.

The database ran locally using PGlite’s real PostGIS WebAssembly extension: PostgreSQL 18.6 and PostGIS 3.6.2. Exact package versions are locked in the repository, and the run manifest records engine versions and input/output hashes. No Docker account or external database service is required.

Four scoring tests passed as well: ID sets ignore ordering but reject duplicates; ranked lists preserve ordering; boolean answers reject strings and integers; and model prompts exclude answer-key fields. The model API integration remains untested because no model calls were made.

These checks establish a reproducible reference computation under the stated rules. They do not establish that every airport coordinate perfectly matches reality, or that a language model has any particular accuracy.

The model comparison is the next experiment

The repository includes an API runner and strict scorer. The runner supplies the same declared input context, saves complete prompts and raw responses, records timestamps and latency, and reads credentials only from an environment variable.

A full run requires 240 unique attempted questions. Errors must remain visible in the denominator. An incomplete run should report its coverage instead of being compared to a full run. A header-only model_results.csv and evaluation_status.json explicitly mark this phase as pending.

For a future comparison, I would report correct answers and denominators by task, exact model identifiers and settings, malformed responses, abstentions, and repeated-run variability. Because related questions share scenes, uncertainty estimates should resample scenes rather than treat every question as independent.

The useful error categories are concrete: wrong predicate, wrong units, swapped coordinate order, missing candidate, incorrect ranking, or boundary-policy error. At this stage they are categories to investigate, not observed LLM findings.

The architecture worth comparing

A tool-assisted condition would follow this sequence:

Natural-language question
          ↓
LLM extracts intent and parameters
          ↓
Validated query specification
          ↓
PostGIS executes a parameterized template
          ↓
Structured IDs, distances, and units
          ↓
LLM explains the returned result

For a radius query, the model would identify the reference and radius. Application code would validate those parameters and choose a predefined query template. The database would calculate the relationship.

This separates intent-extraction mistakes from computation mistakes and explanation mistakes. The explanation still needs to be checked against the returned IDs and distances. A correct database result can be misreported in fluent prose.

Whether that division of labor improves model accuracy, and at what cost and latency, is something the next experiment should measure.

Reproduce the benchmark and figures

Everything behind the completed results is in the public GitHub repository:

• README and execution instructions, plus source provenance and licenses.
• Question generator and the frozen public-data subset.
• All PostGIS queries and the executable ground-truth runner.
• Raw PostGIS results CSV and JSONL answer key.
• Executed analysis notebook, plotting code, figure inputs, and both PNG figures.
• Model runner and scorer, with model results explicitly pending.

With Node 24 and Python 3.11, the command-line rebuild is:

npm ci
python -m pip install -r requirements.txt
python generate.py
node ground_truth.mjs
python -m unittest test_evaluation.py
python analyze.py

The default rebuild uses the committed snapshot. Refreshing the upstream dataset is a deliberate operation that creates a new input version and may change results.

First make the claim checkable

A benchmark is useful when someone else can inspect its assumptions, rerun its queries, and challenge its conclusions.

This release establishes the spatial questions and their computational answers. It shows, with executed examples, why nearest selection, radius membership, containment, and intersection are different operations.

It does not yet tell us which language model handles those distinctions best. That headline has to wait for the model runs.

I Built a Spatial Reasoning Benchmark With a PostGIS Answer Key was originally published in Data And Beyond on Medium, where people are continuing the conversation by highlighting and responding to this story.

How a “simple” assumption that is almost always wrong still builds a useful classifier, and where it quietly lets you down

Every evening at six, Mr. Rao sits at the gate of a housing society and decides who gets let in.

He has no computer. He has never heard of machine learning. But he is very good at his job, and if you watch him closely, he follows a system.

A man arrives with a cardboard box. Probably a delivery. He’s wearing a uniform with a logo. Even more likely a delivery. It’s around eleven in the morning, when most deliveries happen. Almost certainly a delivery. He waves the man through and says “Which flat?” without looking up.

Notice what Mr. Rao did. He never asked, “How often does a man with a box, in a uniform, at eleven in the morning, turn out to be a delivery?” Nobody has seen enough of those exact combinations to know. Instead, he took each clue on its own, asked how typical it was for deliveries, and stacked his hunches.

That habit, taking each clue separately and combining the evidence, is the entire idea behind an algorithm called Naive Bayes. And as you’ll see, the way Mr. Rao slightly fools himself is exactly where the algorithm’s weakness comes from too.

Mr. Rao is an invented example I made up to explain the idea. The code and numbers later in this article are real.

I’m learning data science one step at a time, and Naive Bayes was one of those topics where the maths looked scary but the idea turned out to be friendly. So for this article I ran the experiments myself, built a tiny spam filter by hand to check it against the library, and tested the algorithm on real datasets to see where it shines and where it struggles.

What this article covers

1. Bayes’ theorem, using a test result that fools most people.
2. What the “naive” part really means.
3. A spam filter worked out by hand, with code to check it.
4. The zero-probability problem and Laplace smoothing.
5. The three flavours of Naive Bayes.
6. How it compares with logistic regression and KNN on real data.
7. Why it is often too confident, and what to do about it.
8. When to use it, and when to leave it.

Start with a test that fools almost everyone

Before the algorithm, we need the theorem behind it. Here’s a puzzle.

A disease affects 1 in 100 people. A test for it catches 90% of people who have it. But it also wrongly flags 9% of healthy people as positive. You take the test and it comes back positive. What is the chance that you actually have the disease?

Most people say something like 90%. The real answer is about 9%.

The easiest way to see why is to imagine 1,000 people:

• 10 of them are sick, and the test catches 9 of those.
• 990 are healthy, and the test wrongly flags about 89 of them.

That’s 98 positive tests in total, and only 9 of them are real. So 9 out of 98, or roughly 9%. The test result matters, but so does how rare the disease was to begin with. The huge healthy group produces a lot of false positives, and they swamp the few true ones.

That is Bayes’ theorem in action. It tells you how to update your belief once you’ve seen some evidence, and it never forgets to start from how common something was in the first place.

Bayes’ theorem in four pieces

Written as a formula, it looks like this:

P(class | words) = P(class) x P(words | class) / P(words)

Each piece has a plain meaning:

• Prior, P(class). How common is this class before we look at anything? In the puzzle, 1% for the disease. For email, maybe 30% for spam.
• Likelihood, P(words | class). If a message really were spam, how likely is it to contain these words?
• Evidence, P(words). How common are these words overall? This is the same number for every class, so when we only want to compare classes, we can ignore it.
• Posterior, P(class | words). What we actually want: given these words, how likely is each class?

A classifier calculates the posterior for every class and picks the biggest one.

So where does “naive” come from?

Here’s the catch. To use the likelihood, we need P(words | class) for the whole message at once. For a message with three words, that means knowing how often this exact combination of three words shows up in spam. With a vocabulary of thousands of words, almost no combination has ever been seen before. We would need far more data than exists.

Naive Bayes takes a shortcut. It assumes each word is independent of the others, once we know the class. Then the hard joint probability breaks into easy single-word probabilities that you multiply:

P(w1, w2, w3 | class)  ~  P(w1 | class) x P(w2 | class) x P(w3 | class)

This is exactly what Mr. Rao did. He treated the box, the uniform and the time of day as separate clues and combined them.

It’s called naive because that assumption is almost never true. Words in a real message are linked: “free” and “prize” often appear together. A uniform and a box are linked too, since delivery people usually wear one and carry the other. Counting both as separate evidence counts the same fact twice. Hold onto that thought, because it explains the weakness we’ll see later.

What you gain is huge. For a vocabulary of V words, you only need to estimate about V numbers per class, not an astronomical number of word combinations. That’s why Naive Bayes trains fast and works with small amounts of data.

A spam filter, worked out by hand

Let’s build a tiny one. I made up eight short messages. The first four are spam and the last four are legitimate (“ham”):

The new message to classify is: “free cash today”.

Step 1: the priors. Four of the eight messages are spam, so P(spam) = 0.5 and P(ham) = 0.5.

Step 2: the word probabilities. Count how often each word appears in each class. The spam messages contain 15 words in total, and the ham messages contain 14. The whole vocabulary has 16 different words.

For now, I’ll use Laplace smoothing (adding 1 to every count), and I’ll explain why in a moment. The probability of a word given a class is:

P(word | class) = (times the word appears in the class + 1) / (total words in the class + 16)

For the three words in our message:

Step 3: multiply and compare. Start at the prior (50% spam), then update with each word. Here is the result as each word arrives:

And here it is as an animation:

After the whole message, the model says there’s a 78.4% chance it’s spam. Two spammy words and one ham-ish word land on “probably spam, but not certain”. That feels sensible.

Checking my hand calculation against scikit-learn

I don’t fully trust a calculation until a library agrees with it:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

spam = ["win free prize now", "free cash win",
        "claim your free prize", "urgent claim cash now"]
ham = ["meeting at noon", "lunch at noon today",
       "project meeting today", "see you at lunch"]
docs = spam + ham
labels = np.array([1] * 4 + [0] * 4)        # 1 = spam, 0 = ham
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(docs)
model = MultinomialNB(alpha=1.0)            # alpha=1 is Laplace smoothing
model.fit(X, labels)
new = vectorizer.transform(["free cash today"])
print(model.predict_proba(new))             # [[P(ham), P(spam)]]

The output shows a spam probability of 0.784, matching my hand-worked 78.4%.

The zero-probability problem

Why did I add 1 to every count? Try it without.

In my tiny corpus, the word “free” never appears in a ham message, so P(free | ham) would be exactly 0. And “today” never appears in a spam message, so P(today | spam) would be 0 too.

Because Naive Bayes multiplies, a single zero wipes out everything else. For the message “free cash today”, the ham score would be zero because of “free”, and the spam score would be zero because of “today”. Both classes collapse to zero, and the model can’t decide anything. One unseen word can veto an otherwise obvious verdict.

Laplace smoothing fixes this by pretending you’ve seen every word at least once in every class. That’s what alpha=1.0 does. Smaller values like 0.1 trust the data more. Larger ones flatten the probabilities. It's a hyperparameter you can tune with cross-validation.

One more practical detail. Multiplying many small probabilities makes numbers so tiny that a computer rounds them to zero. Real implementations avoid this by adding logarithms of the probabilities instead. You don’t need to do anything, since scikit-learn already does, but it’s why you’ll see “log probabilities” in the documentation.

The three flavours

The same idea comes in versions designed for different kinds of features:

• Gaussian Naive Bayes is for continuous numbers like measurements. It assumes each feature follows a bell curve inside each class and estimates the mean and spread for each.
• Multinomial Naive Bayes is for counts, like word counts or TF-IDF scores. It’s the standard choice for text.
• Bernoulli Naive Bayes is for yes/no features: was this word present or not? It also counts the absence of a word as evidence, which Multinomial doesn’t.

If you’ve read about Bag of Words and TF-IDF, those are exactly the kinds of features Multinomial Naive Bayes likes.

How good is it on real data?

A tiny spam example proves the idea. It doesn’t tell you how the algorithm does on real data, so I compared Gaussian Naive Bayes with logistic regression and KNN on three datasets bundled with scikit-learn, using 5-fold cross-validation:

from sklearn.datasets import load_wine, load_breast_cancer, load_digits
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(5, shuffle=True, random_state=42)
models = {
    "Naive Bayes": GaussianNB(),
    "Logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
    "KNN": make_pipeline(StandardScaler(), KNeighborsClassifier(5)),
}
for name, loader in [("Wine", load_wine), ("Breast cancer", load_breast_cancer), ("Digits", load_digits)]:
    X, y = loader(return_X_y=True)
    for model_name, model in models.items():
        print(name, model_name, cross_val_score(model, X, y, cv=cv).mean())

On the wine dataset, Naive Bayes scored 97.2%, matching KNN (97.2%) and landing about one point behind logistic regression (98.3%). On breast cancer it scored 93.9%, behind logistic regression (97.4%) and KNN (96.3%). On the digits dataset the gap was large: 84.5%, against 97.1% for logistic regression and 97.9% for KNN.

So Naive Bayes isn’t always a bad choice, but the “naive” shortcut clearly costs more on some data than others. The digits images are the extreme case. Neighbouring pixels are strongly linked to each other, and the model treats them as independent. I also only tried Gaussian Naive Bayes there. A different flavour might do better, and I didn’t test one.

Where it did win was speed. On the digits training split, fitting Naive Bayes took about 0.003 seconds, and logistic regression took about 0.026 seconds on my machine. That’s roughly ten times faster, though the exact times will differ on yours. Naive Bayes only has to count things, while logistic regression has to optimise.

Why the independence assumption hurts

Why did the breast cancer result drop? Look at the features. The dataset has 30 measurements of cell nuclei, and many describe the same thing in different ways. “Mean radius” and “mean perimeter” have a correlation of 0.998, which means they’re nearly the same information. Of the 435 pairs of features, about 16% have a correlation above 0.7.

This is Mr. Rao’s problem again. If the radius says “malignant” and the perimeter says “malignant”, Naive Bayes treats these as two separate pieces of evidence and multiplies them. In reality it’s one piece of evidence counted twice.

The overconfidence problem

Here’s the most interesting thing I found, and it isn’t about accuracy. It’s about confidence.

I trained Naive Bayes and logistic regression on the breast cancer data and kept 143 patients aside for testing. Then I looked at the probabilities each model reported:

from sklearn.metrics import log_loss

nb_probs = GaussianNB().fit(X_train, y_train).predict_proba(X_test)
lr = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)).fit(X_train, y_train)
lr_probs = lr.predict_proba(X_test)
print(log_loss(y_test, nb_probs), log_loss(y_test, lr_probs))
print((nb_probs.max(axis=1) > 0.99).mean(), (lr_probs.max(axis=1) > 0.99).mean())

Naive Bayes had an accuracy of 93.7% against 98.6% for logistic regression. But look at the confidence. Naive Bayes was more than 99% sure on 94.4% of the test patients, compared with 69.2% for logistic regression. Its log loss was 0.37, against 0.07 for logistic regression.

And here’s the part that stayed with me: five of the nine mistakes Naive Bayes made were made with over 98% confidence.

This follows directly from double counting. When correlated features all point the same way, the model multiplies them together and the probability shoots towards 0 or 1, even though the real evidence is weaker than that.

What does this mean in practice?

• Trust Naive Bayes’s ranking (which class scores higher) more than its percentages.
• If you need trustworthy probabilities, for example to set a threshold, you can recalibrate them. scikit-learn has CalibratedClassifierCV for this. I didn't test it here, so treat that as a pointer for you to try.
• Don’t read “99.9% spam” as a literal promise.

Why does it work at all?

If the assumption is almost always wrong, why do people still use Naive Bayes? I think the answer is in the difference between two jobs:

1. Estimating the probability correctly.
2. Choosing the right class.

The overconfidence results show that Naive Bayes is mediocre at the first job. But for the second, it only needs the correct class to come out on top. One common explanation is that the errors from double counting often push the scores of both classes in the same direction, so the winner stays the same. In text especially, where there are thousands of features and each one carries a little evidence, that is often good enough. Researchers have studied this question formally, and the Zhang paper in the Resources section is a good place to start if you want the theory.

When to use it, and when to leave it

Reach for Naive Bayes when:

• You’re working with text, such as spam, topics or sentiment, and want a quick baseline.
• You have little training data.
• You need something very fast to train and to predict.
• You want a simple, explainable model: you can literally list the words that pushed the score.

Be careful when:

• Your features are strongly correlated, like pixels or repeated measurements.
• You need well-calibrated probabilities.
• You’ve got plenty of data and accuracy matters more than simplicity. Logistic regression or tree-based models often win.

And one habit from my earlier articles applies here too: compare it against a baseline and look at more than one metric. If you want a refresher on why accuracy alone can mislead, the same lesson about precision and recall applies to a spam filter.

Back to Mr. Rao

One evening, a man arrived at the gate with a cardboard box, wearing a uniform with a logo, at around eleven in the morning. Mr. Rao waved him through.

He was a very polite man, and he was not a delivery person. He had borrowed the uniform and the box and was visiting a friend.

That’s Naive Bayes in a nutshell. It combines many small clues quickly, and most of the time it’s right. But when the clues are secretly connected, it counts the same fact twice and gets a little too sure of itself. Knowing that doesn’t make the algorithm useless. It tells you when to trust the verdict, and when to ask one more question.

Thanks to my trainer, my mentor and Innomatics for the push to keep learning and exploring. If you’re also learning data science in public, follow along for more beginner-friendly breakdowns.

Resources

• scikit-learn documentation, Naive Bayes: https://scikit-learn.org/stable/modules/naive_bayes.html
• scikit-learn documentation, probability calibration: https://scikit-learn.org/stable/modules/calibration.html
• scikit-learn documentation, bundled datasets: https://scikit-learn.org/stable/datasets/toy_dataset.html
• H. Zhang, “The optimality of naive Bayes”, FLAIRS conference, 2004
• A. McCallum and K. Nigam, “A comparison of event models for naive Bayes text classification”, AAAI-98 workshop on learning for text categorization

Naive, but not foolish: the Naive Bayes algorithm, explained with a security guard and a spam… was originally published in Data And Beyond on Medium, where people are continuing the conversation by highlighting and responding to this story.