InTowards AIbychakshu_salgotra·Oct 21Part XVI — Whole-Stage Codegen: The Tungsten Trick Behind 2x SpeedupsYour Spark job is not slow because of your data. It is slow because Spark is interpreting your query one virtual function call at a time —…
InTowards AIbychakshu_salgotra·Oct 20Part XV — Adaptive Query Execution: What AQE Actually Rewrites at RuntimeSpark plans your query before it has seen your data. AQE is the engine’s admission that the planner was guessing — and the machinery that…A response icon1
InTowards AIbychakshu_salgotra·Oct 6Part XI — RDD vs DataFrame vs Dataset: Why Your “Type-Safe Refactor” Made Spark 3x SlowerThree APIs, two execution worlds — how Catalyst and Tungsten treat structured plans, why typed lambdas are black boxes, and when dropping…A response icon1
InTowards AIbychakshu_salgotra·Aug 22Part X — Z-Ordering and Data Clustering Explained: Why Your Partitioned Table Still Scans…Data skipping, min/max statistics, space-filling curves, and liquid clustering — the file-layout mechanics that decide whether your query…
InTowards AIbychakshu_salgotra·Jul 30Part VI - The Spark Join Strategies — Broadcast vs.Why the same join ran in 90 seconds one day and shuffled 3.8TB the next — and how to make Spark pick the right physical operator every time
