Francisco Morillo·Jul 14Streaming Data Lake: Spark vs Flink into Apache IcebergA reproducible, open-source benchmark of Apache Spark 4.1 and Apache Flink 2.2 streaming a 100,000-records/second stream into Apache…
Random Droid·Jul 7From Kafka to Spark: Building Real-Time Pipelines with Structured StreamingA standalone piece from the OLAP series — if you’re here from the OLAP Architecture series, this is the practical streaming counterpart to…
Angel Conde·Jun 23Spark RTM mode — Low latency streaming pipelinesFor its entire life, Spark Structured Streaming has been a micro-batch engine. It feels continuous, but under the hood it slices your…
Abderrahmane Oubagha·May 24When Metadata Becomes the Bottleneck : Optimizing Spark Streaming & Delta LakeA deep-dive into checkpoint compaction and Delta Lake transaction log growth in high-frequency file streaming pipelines — and how to fix…A response icon1
Ketan Kumbhar·Dec 25, 2025Building a Custom Apache Spark Connector for Google Cloud Pub/SubAn open-source journey into Spark Structured Streaming
