Francisco Morillo·Jul 14Streaming Data Lake: Spark vs Flink into Apache IcebergA reproducible, open-source benchmark of Apache Spark 4.1 and Apache Flink 2.2 streaming a 100,000-records/second stream into Apache…
Random Droid·Jul 7From Kafka to Spark: Building Real-Time Pipelines with Structured StreamingA standalone piece from the OLAP series — if you’re here from the OLAP Architecture series, this is the practical streaming counterpart to…
Angel Conde·Jun 23Spark RTM mode — Low latency streaming pipelinesFor its entire life, Spark Structured Streaming has been a micro-batch engine. It feels continuous, but under the hood it slices your…
Abderrahmane Oubagha·May 24When Metadata Becomes the Bottleneck : Optimizing Spark Streaming & Delta LakeA deep-dive into checkpoint compaction and Delta Lake transaction log growth in high-frequency file streaming pipelines — and how to fix…A response icon1A response icon1
Ketan Kumbhar·Dec 25, 2025Building a Custom Apache Spark Connector for Google Cloud Pub/SubAn open-source journey into Spark Structured Streaming
Dharmik Soni·Oct 28, 2025Taming the Beast: Understanding and Preventing AttributeReference Memory Leaks in Apache SparkIntroduction
Ajr Jain·Aug 15, 2025Spark Evolution: Diving into v2.4.8, v3.5.0, and v4.0Apache Spark has revolutionized big data processing. We’ll focus on architectural shifts, performance leaps, and feature enhancements.
Emre BÜYÜKBAYRAKTAR·Jun 10, 2025Streaming Datas with Dynamic Queries Transferring and Filtering them Using Kafka…In today’s fast-paced digital era, data is no longer confined to static tables from the past. Especially in fields like manufacturing…
Gamal Elkoumy·Apr 16, 2025From Data Chaos to Real-Time Insights: Building a Data Lakehouse with Spark, Kafka, and IcebergBuild a near-real-time data lakehouse with Spark, Kafka, and Iceberg. Merge JSON data, use time-travel queries, and speed insights!A response icon1A response icon1
