InTowards AIbychakshu_salgotra·Oct 21Part XVI — Whole-Stage Codegen: The Tungsten Trick Behind 2x SpeedupsYour Spark job is not slow because of your data. It is slow because Spark is interpreting your query one virtual function call at a time —…
InTowards AIbychakshu_salgotra·Oct 20Part XV — Adaptive Query Execution: What AQE Actually Rewrites at RuntimeSpark plans your query before it has seen your data. AQE is the engine’s admission that the planner was guessing — and the machinery that…A response icon1A response icon1
InTowards AIbychakshu_salgotra·Oct 6Part XI — RDD vs DataFrame vs Dataset: Why Your “Type-Safe Refactor” Made Spark 3x SlowerThree APIs, two execution worlds — how Catalyst and Tungsten treat structured plans, why typed lambdas are black boxes, and when dropping…A response icon1A response icon1
InTowards AIbychakshu_salgotra·Aug 22Part X — Z-Ordering and Data Clustering Explained: Why Your Partitioned Table Still Scans…Data skipping, min/max statistics, space-filling curves, and liquid clustering — the file-layout mechanics that decide whether your query…
InTowards AIbychakshu_salgotra·Jul 30Part VI - The Spark Join Strategies — Broadcast vs.Why the same join ran in 90 seconds one day and shuffled 3.8TB the next — and how to make Spark pick the right physical operator every time
InTowards AIbychakshu_salgotra·Jul 17Part IV - Jobs, Stages & Tasks — How Spark Actually Decomposes Your QueryWhy one line of DataFrame code spawned 14 jobs, and how to read the execution graph before it reads you
Prashant Kumar·Mar 21Spark Internals: Understanding Tungsten (Part 1)Note: This article is part of a 2-part series, In this part, we focus on the why and how behind Tungsten. In Part 2, we’ll cover its…
InData Engineer ThingsbySiddhant·Feb 26Apache Spark 201— Spark Optimisation TechniquesYour Spark job works — but why is it slow, expensive, or unpredictable? If Spark 101 taught you how Spark works, Spark 201 is about making…
InData Engineer ThingsbySiddhant·Jan 30Apache Spark 101 — What is Spark?Hadoop made big data possible. Spark made it fast. Let’s understand what Apache Spark really is.
InTowards DevbyAkhil Varthya·Jul 20, 2025Spark Internal MechanismApache Spark has become the go-to engine for large-scale data processing — but what actually happens after you write your Spark code? In…A response icon1A response icon1
