Pandraju Isaac·5d agoArrow UDFs in PySpark 4.1: Python UDFs That Skip pandas EntirelySpark 4.1 added @arrow_udf, which hands your function pyarrow arrays instead of pandas Series. I put it next to pandas_udf and the useArrow…A response icon1A response icon1
Antoine Prouvost·6d agoWhen Parquet levels are bitmapsThe next release of Apache Arrow C++ and PyArrow (26.0.0 being the latest to date) will see up to 20% speedup in reading whole Parquet…
InPython in Plain EnglishbyDouble Espresso Data·Oct 25Day 2 — Arrow Columnar Memory: Why Polars Is Fast100 Days of Polars · Phase 1: FoundationsA response icon1A response icon1
Shahzeb Ali·Oct 23The Worst Bugs Do Not Crash. They Quietly Give You the Wrong Answer.A day spent hunting silent bugs in DataFusion, PyIceberg and Arrow
Alex Merced·Oct 23Apache Data Lakehouse Weekly: October 14–21, 2026Many contributors were still flying home from Community Over Code in Glasgow, and the lists picked up where the hallway talks ended…
Alex Merced·Oct 16Apache Data Lakehouse Weekly: October 6–14, 2026Half the lakehouse community spent this week in Glasgow at Community Over Code, and the mailing lists show it. Syncs got cancelled, board…
Alex Merced·Oct 7Apache Data Lakehouse Weekly: September 29 to October 7, 2026Apache Iceberg 1.12.0 shipped this week, and the release set off a chain of follow-up work across the open lakehouse stack. Polaris started…
Alex Merced·Oct 2Apache Data Lakehouse Weekly: September 23 to 30, 2026Release week. Apache Iceberg 1.12.0 passed its vote after a license audit sank the first candidate, Apache Polaris shipped 1.8.0 with a…
InData ArkaivebyAIExplore369·Sep 25Your Apple Silicon GPU Loses to One CPU Core Until a Million Rows.Every GPU data library benchmarks itself at 50 million rows. Your dataframe has 80,000. I know, because I build one of these libraries, and…
