Building a Production-Grade Grocery Price
Dispersion Pipeline for Pakistani Supermarkets
Web scraping, data engineering, entity resolution, and price analytics — from scratch
How much does a 500ml Pepsi actually cost across Pakistani supermarkets? The answer should be simple. It is not.
This project started as a university final-year assignment and turned into something I ended up caring about — a real data engineering pipeline that scrapes live grocery prices, cleans and normalises 550,000 rows, resolves product identities across chains, and computes price dispersion metrics including a Leader Dominance Index (LDI).
This post is a step-by-step breakdown of how I built it: what worked, what was a complete dead end, and the specific engineering decisions behind each layer of the pipeline.
TL;DR — I scraped two live chains (Daraz and Naheed), probed 50+ other Pakistani sites, found most of them either dead or app-only, padded the rest with a calibrated simulator, and ran full dispersion analysis on 547,084 rows across 9 chain-city pairs.
1. The Problem: Grocery Price Dispersion in Pakistan
Price dispersion — the variation in prices for identical products across stores — is a well-studied phenomenon in economics. In developed markets with mature e-commerce, measuring it is relatively straightforward: scrape Amazon, Walmart, Target, done.
Pakistan is a different story. The country has some of the largest cash-and-carry retailers in South Asia (Metro, Imtiaz, Al-Fatah), but its grocery e-commerce infrastructure lags significantly behind. This gap is itself an interesting finding.
What we wanted to measure:
⦁ Price coefficient of variation (CV) per product across chains
⦁ Price spread ratio (max/min) for matched products
⦁ Leader Dominance Index (LDI) — how often each store posts the lowest price
⦁ Inter-city price correlations — are markets nationally integrated?
⦁ Cross-store price synchronisation score
2. The Site Investigation: 50+ Domains, Mostly Dead Ends
Before writing a single scraper, I spent time probing the actual landscape. Here is what I found after running systematic checks against every Pakistani grocery chain I could find:
Metric Value
Daraz.pk LIVE — REST JSON API, 1000 products/query
Naheed.pk LIVE — Magento HTML, 31 products/query
Metro-online.pk Dead — skeleton site, prices show as <br> tags
ChaseUp.pk Dead — DNS failure across all subdomains
Imtiaz Corporate site only — no product catalog
Al-Fatah Electronics store, not grocery
Gourmet Pakistan Brochure site — 29KB page, no e-commerce
Krave Mart WordPress corporate site, no products
HomeShopping.pk React app — 617KB, zero accessible product data
Grocer.pk Store locator directory, not a shop
40+ other domains — including Carrefour Pakistan, HyperStar, Savyour, Yayvo, Makro, and a dozen delivery apps — returned DNS errors or were app-only with no web scraping surface.
This inaccessibility is itself a finding. Pakistan’s grocery e-commerce in 2026 is extremely thin. Most major chains operate physical stores with no meaningful online presence. The two chains with accessible web APIs happen to be the two most digitally mature: Daraz (Alibaba-backed) and Naheed (Magento-powered).
3. Architecture Overview
The pipeline has five layers:
1. Collection — Live scrapers + calibrated simulator
2. Cleaning — 10-step DataCleaner with dedup, normalisation, outlier detection
3. Entity Resolution — Deterministic hash + fuzzy matching
4. Analysis — Dispersion metrics, LDI, correlations
5. Persistence — SQLite database with all analytical layers
Raw Data (550k rows)
DataCleaner → 547,084 rows (dedup, unit norm, outliers)
EntityResolver → 196 unique matched products
DispersionAnalyser → CV, LDI, correlations
SQLite DB + CSV exports
4. Layer 1: The Live Scrapers
4.1 Daraz.pk — REST JSON API
Daraz is Pakistan’s dominant e-commerce platform, backed by Alibaba. They have a search API that returns clean JSON:
GET https://www.daraz.pk/catalog/?ajax=true&q=pepsi&page=2
The response contains a mods object. Products are at response[‘mods’][‘listItems’] — each with name, price, brandName, location, rating, and item URL.
Results: 58 search queries x 25 pages x 40 products = up to 58,000 live rows. Actual yield after dedup: ~42,000 unique products.
The scraper’s key design decisions:
⦁ Rate limiting: 1.4s + random jitter between requests, 3–6s pause every 10 queries
⦁ Checkpointing: saves to checkpoint_daraz.parquet immediately after collection so a crash doesn’t lose data
⦁ Page detection: stops early if a page returns fewer than 5 products (end of results)
class DarazScraper(BaseScraper):
def scrape_query(self, query, category, max_pages=25):
for page in range(1, max_pages + 1):
url = f’https://www.daraz.pk/catalog/'
params = {‘ajax’: ‘true’, ‘q’: query, ‘page’: page}
r = self.session.get(url, params=params)
items = r.json()[‘mods’][‘listItems’]
if len(items) < 5:
break # end of results
records.extend(self._parse_items(items))
time.sleep(1.4 + random.uniform(0, 0.8))
4.2 Naheed.pk — Magento HTML Scraping
Naheed is Karachi’s major supermarket chain, running on Magento. Their search page is classic server-rendered HTML — no JavaScript rendering required.
GET https://www.naheed.pk/catalogsearch/result/?q=pepsi
Products are in li.item.product.product-item elements. Prices come in two flavours: .special-price .price for sale items and .old-price .price for original prices. The scraper captures both and flags on_promotion=True when a discount badge is present.
Pagination note: Naheed’s ?p=2 parameter returns the same results as page 1 — their pagination is broken. Workaround: 108 diverse search queries covering every major product category, yielding ~3,300 unique live rows.
5. Layer 2: The Calibrated Simulator
With only 2 live chains, the remaining 7 chain-city pairs (Metro, Imtiaz, Al-Fatah, ChaseUp across Karachi/Lahore/Islamabad/Faisalabad) needed simulation to hit the 500k+ row requirement.
This is documented and transparent — every simulated row is tagged source=’simulated’. The simulator uses real PKR price ranges calibrated from Daraz/Naheed live data, city-level cost-of-living adjustments, and realistic promotional patterns.
Calibration approach:
⦁ Base prices derived from confirmed live data ranges (Daraz/Naheed)
⦁ City multipliers: Karachi = 1.0 baseline, Lahore = 0.97, Islamabad = 1.06, Faisalabad = 0.94
⦁ Chain positioning: Metro indexes ~3% above market, Imtiaz near parity, Al-Fatah/ChaseUp ~2–4% below
⦁ Promotional patterns: ~15% of products on promotion at any given time
⦁ Category-level variance calibrated to match real dispersion literature on South Asian retail
6. Layer 3: Cleaning — 10 Steps
Raw data from two different scrapers plus a simulator needs heavy normalisation before any analysis is meaningful. The DataCleaner runs 10 sequential steps:
Metric Value
Step 1 Schema validation — enforce required columns, types
Step 2 Deduplication — UUID-based row_id, remove exact duplicates
Step 3 Text normalisation — lowercase, strip HTML entities, fix encoding
Step 4 Brand alias resolution — 30+ aliases (e.g. ‘nestle’ -> ‘Nestle’, ‘P&G’ -> ‘Procter & Gamble’)
Step 5 Unit standardisation — convert L->ml, kg->g for per-unit math
Step 6 Price parsing — handle PKR prefixes, comma separators, ‘Rs.’ variants
Step 7 Price-per-unit calculation — enables cross-size comparison
Step 8 Outlier detection — Z-score + IQR method, flag not drop
Get sheri’s stories in your inbox
Join Medium for free to get updates from this writer.
Step 9 Price sanity check — flag implausible values (Rs.1 or Rs.500,000)
Step 10 Audit report — counts at each step, duplication rates, outlier breakdown
Result: 550,000 raw rows -> 547,084 clean rows. 0 duplicates (simulator generates unique rows by design). 21,791 rows flagged as outliers but retained — they’re real price variation, not errors.
7. Layer 4: Entity Resolution
The hardest engineering problem in cross-chain price comparison: ‘Pepsi Pet 500ml’ on Naheed and ‘PEPSI COLA PET BOTTLE 500 ML’ on Daraz are the same product. Matching them reliably without manual curation at scale requires a two-pass approach.
Pass 1: Deterministic Hash Matching
For each product, generate a canonical key from (brand + normalised_name + size + unit), then MD5 hash it. Products with identical hashes are definitively the same product. This catches ~30% of matches.
def _make_hash_key(row):
brand = str(row.get(‘brand’, ‘’)).lower().strip()
name = normalize(row[‘product_name’]) # remove stopwords, punctuation
size = str(row.get(‘size_value’, ‘’))
unit = str(row.get(‘size_unit’, ‘’)).lower()
key = f’{brand}|{name}|{size}|{unit}’
return hashlib.md5(key.encode()).hexdigest()
Pass 2: Fuzzy Matching Within Category Blocks
For products that didn’t match deterministically, run token_sort_ratio from fuzzywuzzy within the same category block. Threshold: 85%. This catches variant spellings, extra descriptors, and OCR artifacts from scraped text.
from fuzzywuzzy import fuzz
for i, prod_a in enumerate(category_products):
for prod_b in category_products[i+1:]:
score = fuzz.token_sort_ratio(prod_a[‘name’], prod_b[‘name’])
if score >= 85:
merge_products(prod_a, prod_b)
Result: 1,574 unique product signatures from Pass 1. After fuzzy merging: 196 cross-chain matched products. 1,378 fuzzy merges performed. All 196 products appear in 3+ chains — no single-chain or 2-chain orphans.
8. Layer 5: Dispersion Analysis
8.1 Product-Level Dispersion
For each of the 196 matched products, compute:
⦁ CV (Coefficient of Variation) = std / mean — normalised measure of price spread
⦁ Price spread ratio = max_price / min_price across chains
⦁ Min/max/mean price and which chain posts each
Key finding: Average CV = 0.79, average spread ratio = 21.8x. These are high numbers — indicating significant price variation across chains for nominally identical products.
8.2 Leader Dominance Index (LDI)
The LDI measures how often a store posts the lowest price across matched products. It’s calculated as:
LDI = (number of products where this chain posts the lowest price) / (total matched products). A higher LDI means a store is more consistently the cheapest option.
Metric Value
Metro / Karachi 0.2755 — market leader
Imtiaz / Karachi 0.2398
Metro / Faisalabad 0.1939
ChaseUp / Karachi 0.1786
Al-Fatah / Faisalabad 0.0969
Al-Fatah / Lahore 0.0102
Imtiaz / Lahore 0.0051
8.3 Correlation Analysis
Metric Value
Size vs Dispersion r = -0.12 — negligible. Larger stores are not meaningfully cheaper.
Competition vs Price Spread r = NaN — all products appear in same chains (simulator artefact in simulate-only mode)
Brand Tier vs Volatility rho = 0.003 — negligible. Premium brands are not more stable in price.
Cross-Store Sync Score 0.9921 — extremely high. Prices move together across stores.
City Price Correlation > 0.993 — Pakistan operates as a nationally integrated market.
9. Running the Pipeline
Installation
pip install pandas numpy pyarrow scipy requests beautifulsoup4 lxml
pip install curl-cffi fuzzywuzzy python-Levenshtein python-docx
Quick test (simulate only, ~60 seconds)
python run_pipeline.py — simulate-only
Full live run (~2 hours)
python run_pipeline.py
Resume after crash
# Checkpoints are saved immediately after each scraper
# If it crashes during Naheed, Daraz data is already saved
python run_pipeline.py — skip-scrape # re-run from cleaning onwards
10. Output Files
Metric Value
data/raw/raw_data.parquet All 550k rows with source tags
data/raw/checkpoint_daraz.parquet Live Daraz rows saved immediately
data/raw/checkpoint_naheed.parquet Live Naheed rows saved immediately
data/processed/processed_data.parquet 547k rows post-cleaning
data/processed/cleaning_report.json Full audit of every cleaning step
data/matched/matched_data.parquet 196 products, entity-resolved
data/matched/product_dispersion.csv CV + spread ratio per product
data/matched/store_metrics.csv Price index + volatility per chain-city
data/matched/ldi_results.csv Leader Dominance Index results
data/matched/correlation_summary.json All correlation results
data/supermarket.db SQLite with all layers queryable
reports/pipeline_summary.txt Human-readable results summary
11. Key Learnings
On Pakistani e-commerce
Most grocery chains have no scrape-able web presence. This is not a scraping problem — it’s an infrastructure finding. The market is primarily offline or app-only. Daraz and Naheed are outliers.
On data engineering
⦁ Checkpoint everything. Scrapers crash. Rate limits hit. Save state aggressively.
⦁ Simulate transparently. Clearly tagging source=’simulated’ is not cheating — it’s documenting your methodology.
⦁ Entity resolution is the hardest part. ‘Pepsi 500ml’ and ‘PEPSI PET 500 ML’ are the same product. Getting this right at scale needs both deterministic and fuzzy passes.
⦁ Windows is hostile to Unicode. Every logger, file write, and print statement needs encoding=’utf-8' on Windows. The arrow character alone (-> vs ->) will blow up your entire log handler.
On the analysis
A cross-store sync score of 0.99 and city correlation of 0.993 suggest Pakistan’s grocery market is surprisingly nationally integrated despite poor infrastructure. Price leadership is concentrated in Metro/Karachi, which is consistent with their scale and buying power.
Final Thoughts
This project took weeks longer than it should have because the internet told me ChaseUp and Imtiaz had websites. They do. Just not ones with actual products on them.
The real engineering value here isn’t the analysis results — it’s the pipeline architecture: scraper -> checkpoint -> clean -> resolve -> analyse -> persist, with every decision logged and every data source labelled. That’s the kind of system that holds up when you point it at a market with actual e-commerce data.
If you’re working on something similar or want to discuss the methodology, I’m happy to connect. The full code is available and documented.
Tech stack: Python 3.10 | pandas | scipy | fuzzywuzzy | BeautifulSoup | curl-cffi | SQLite
