InTowards AIbychakshu_salgotra·Oct 20Part XV — Adaptive Query Execution: What AQE Actually Rewrites at RuntimeSpark plans your query before it has seen your data. AQE is the engine’s admission that the planner was guessing — and the machinery that…A response icon1A response icon1
DataPassion·Sep 9Apache Spark Interview Quickie #1 — Adaptive Query Execution (AQE)Crack FAANG-style questions!!A response icon1A response icon1
The Data Forge·Jul 29Spark Tuning: Partitions, Memory, and Why Liquid Clustering Changes the Optimization GameDataForge Series: Databricks Data Engineer Associate - Part 8 of 16
Sumit Mittal·Jun 30From 45 Minutes to 21 Minutes: How Adaptive Query Execution (AQE) Saves Your Spark Jobs4 billion records with 10% skewed data and 3 cores sitting idle with 24 GB spill. Sort Merge Join shuffled 57.5 GB data. 2000 partitions…A response icon2A response icon2
InTowards Data EngineeringbySriw World of Coding·Apr 9Data Engineer Interview Scenario Based Question : How to Optimize SQL Queries in DatabricksYou’ve written a “perfect” SQL query. You run it in Databricks… and it takes 3 minutes instead of 3 seconds.
InData And BeyondbySriw World of Coding·Apr 8How to Handle Small Files in Databricks: Fix the Performance Killer Before It Ruins Your PipelineImagine this: Your Databricks cluster is grinding to a halt on a simple query. Bills skyrocket. Deadlines slip. All because of tiny files…
Muthubabu Sethuraman·Mar 3Adaptive Query Execution AQE in Databricks SparkListen, we’ve all been there. It’s 2 AM, you’re staring at a Spark job that’s been stuck at 99% for forty minutes, and you’re wondering why…
InTowards DevbyAkhil Varthya·Jul 20, 2025Spark Internal MechanismApache Spark has become the go-to engine for large-scale data processing — but what actually happens after you write your Spark code? In…A response icon1A response icon1
Akhil Varthya·Jul 20, 2025Apache Spark Performance: A Guide to Smarter OptimizationApache Spark is a powerhouse for big data processing, but raw speed alone doesn’t guarantee efficiency. Without thoughtful optimization…
InData Engineer ThingsbySantosh Joshi·Feb 24, 2025A Quick Guide to Spark and Databricks Optimization EnginesExploring Catalyst, AQE, Tungsten, Photon, and Enzyme for Enhanced Performance in Spark and Databricks
