Aliakbar Hosseinzadeh·4h agoPySpark Finally Made Sense When I Stopped Thinking Like a Python DeveloperOne practical story to understand Apache Spark, remember its architecture, and prepare for data engineering interviews.
Zeynepbkn·5h agoApache Spark: Temel Kavramlardan Data Lakehouse Mimarisine Kapsamlı Bir RehberApache Spark, büyük verileri dağıtık biçimde işlemek için geliştirilmiş açık kaynaklı, birleşik bir veri işleme motorudur. Veri…
InLevel Up Codingbydk.websolutions·8h agoHow to Keep Apache Iceberg Tables Fast: Compaction, Snapshot Expiry and Orphan File CleanupA Practical Spark SQL Playbook for Small Files, Snapshot Bloat and Storage Leaks in Your Lakehouse
Pandraju Isaac·19h agoSubqueries Without SQL Strings: scalar(), exists() and lateralJoin() in PySpark 4Spark 4.0 added DataFrame methods for scalar, EXISTS and lateral subqueries, and 4.1 let isin() take a DataFrame. I rebuilt the usual SQL…
The Speedcraft Lab·21h agoNetflix Runs 30,000 Flink Jobs, and Open-Source Autoscaling Saved One of Its Teams $1.1M a YearThat team cut Flink compute 58%, and Netflix runs the scaler at 0.45 utilisation, not 0.7
InDatabricks Community ArticlesbyLikith Raj Gollu·1d agoHow Spark Processes Datasets Larger Than Cluster Memory: The Kitchen ArchitectureWhen I first started out, I always wondered: how exactly does Apache Spark process terabytes of data on very modest hardware? I had a tough…
Leandro Silveira·1d agoAWS EVENT-DRIVEN ARCHITECTURE: UNDERSTANDING EVENTBRIDGE, KINESIS, SQS, SNS, AND MANAGED KAFKAIntroduction
Kulveer Singh·1d agoDelta Lake Deep Dive: ACID on Object StorageThe transaction log is the whole trick: how Delta turns dumb files on S3 into real tables with time travel — and how to recover one.
Venugopal Dabbara·1d agoBeyond Micro-Batching: Exploring Real-Time Mode in DatabricksFrom Kafka transactions to operational decisions: a hands-on experiment using Databricks serverless Spark Declarative Pipelines, and…
InIntellytics SolutionsbyBrijesh Dave·1d agoSpark vs. DuckDB vs. DataFusion: A 100GB Transformation BenchmarkThe 100GB Challenge: Why Many Pipelines Are Over-ProvisionedA response icon1A response icon1
