Chandra Prakash·3d agoWhy Tuning maxPartitionBytes to Fix Spark’s Small Files Problem Can Silently Backfire
InDataWiseCodebyDipak Shaw·Oct 27Why Your Spark Window Functions Are SlowInside WindowExec, ROWS vs. RANGE, memory spills, and the Top-K optimization that can reduce unnecessary work.
InT3CHbySriw World of Coding·Jul 16Pyspark Coding Interview Question : Here's What Senior Data Engineers Actually DoYou have a static Python dictionary with 195 country-to-continent mappings. You need to enrich a transactional dataset by mapping…
InT3CHbySriw World of Coding·Jun 30Spark’s Catalyst Optimizer vs Tungsten Execution Engine: The Brain and the Brawn Behind Apache…If you’ve ever wondered why Apache Spark is so blazing fast — and where exactly that magic happens — this article is for you.A response icon1A response icon1
Sumit Mittal·Jun 30From 45 Minutes to 21 Minutes: How Adaptive Query Execution (AQE) Saves Your Spark Jobs4 billion records with 10% skewed data and 3 cores sitting idle with 24 GB spill. Sort Merge Join shuffled 57.5 GB data. 2000 partitions…A response icon2A response icon2
InTowards Data EngineeringbyThink Data·Jun 24Adaptive Query Execution (AQE): How Spark Re-Optimizes Queries at RuntimeTwo Spark queries can return exactly the same result.A response icon1A response icon1
InTowards Data EngineeringbyThink Data·Jun 18Cost-Based Optimizer (CBO) in Spark: Statistics, Cardinality Estimation, and Join SelectionTwo Spark queries can return exactly the same result.A response icon1A response icon1
InTowards Data EngineeringbyThink Data·Jun 18Catalyst Optimizer in Apache Spark: The Compiler Behind Every DataFrame QueryMost Spark developers know Catalyst as:
InData And BeyondbySriw World of Coding·May 28Pyspark Interview Question : You Told Spark to Broadcast a 50 MB Table — And It Crashed the Driver.A deep dive into one of the most misunderstood errors in Apache Spark — and how to never fall for it again.
Abderrahmane Oubagha·May 24When Metadata Becomes the Bottleneck : Optimizing Spark Streaming & Delta LakeA deep-dive into checkpoint compaction and Delta Lake transaction log growth in high-frequency file streaming pipelines — and how to fix…A response icon1A response icon1
