Scientific File Formats

Thanos Vassilakis

5 min read

·

5 days ago

--

Listen

Share

There are hundreds of niche scientific extensions. What’s interesting is how cutting-edge work has consolidated around a smaller set of next-generation, high-performance, multi-dimensional open formats: cloud-optimized chunks, columnar analytics, and parallel I/O that survive petabyte pipelines.

This is a map of those formats by domain, then a comparative look at what you can actually call from Java, C#, C, Zig, Rust, Python, Elixir, Fortran, and Ruby. The library picture changes faster than the formats; treat the tables as a field guide, not a warranty.

Astronomy and space sciences

Modern telescopes — JWST, Roman, and the ground-based survey machines behind them — produce multi-band datasets that need structured metadata as much as raw pixels.

• ASDF (Advanced Scientific Data Format) — modern successor to FITS for many new missions. YAML/JSON-schema metadata plus efficient binary array blocks; designed for validation and extension tags rather than 80-byte card images.
• FITS 4 — still the interchange lingua franca. Version 4 adds better checksums, compression for large images, and tighter world-coordinate / time handling under the IAU standard.
• HDF5 (.h5 / .hdf5) - filesystem-in-a-file for simulations and instruments that outgrew flat FITS tables. Self-describing groups, datasets, attributes, chunking, and filters.

Genomics, bioinformatics, and health

Next-generation sequencing and single-cell multi-omics need streaming reads and brutal compression over millions of base pairs.

• CRAM — reference-based relative encoding; stores differences against a known genome instead of full alignments. Usually much smaller than BAM for the same content.
• BAM — binary Sequence Alignment/Map; the workhorse machine-readable form of SAM.
• VCF / BCF — variants: SNPs, indels, and structural calls with a schema that tools actually agree on.
• h5ad (AnnData) — de facto standard for single-cell transcriptomics in the Python/scanpy world: sparse expression matrix plus observation/variable annotations in one HDF5-shaped container.
• Loom — HDF5 layout for high-dimensional single-cell matrices with row/column attributes.

Geospatial, climate, and environment

Earth observation and climate models are four-dimensional (lat, lon, height/depth, time) and often live on object storage rather than a single POSIX file.

• Zarr — chunked, compressed N-dimensional arrays as many objects (local dir, S3, GCS). Parallel web I/O without opening one giant file handle.
• Cloud Optimized GeoTIFF (COG) — tiled TIFF with overviews so HTTP range requests fetch only the pixels you need.
• NetCDF-4 — climate/ocean/weather currency; NetCDF-4 rides on HDF5 under the hood while keeping the classic scientific data model (dimensions, variables, attributes).
• GeoPackage — SQLite container for vectors, tiles, and rasters; open and non-proprietary.
• LAZ — compressed LiDAR point clouds; the dense 3D counterpart to LAS.

Big data, physics simulations, and HPC

When pipelines hit petabyte scale, science borrows storage ideas from analytics engines.

• Apache Parquet — columnar on disk; predicate pushdown and compression that make analytical scans cheap.
• Apache Arrow / Feather — columnar in memory (and IPC on disk); zero-copy sharing across languages when everyone speaks Arrow.
• Avro — row-oriented binary with an embedded JSON schema; common in streaming pipelines where schemas evolve.
• F3 (Future-proof File Format) — CMU Database Group experiment: columnar layout that can embed WebAssembly decoders so the file carries its own reader. Early, but points at where “format plus codec” is heading.

Format comparison

Comparative library review

Formats only matter if your language can open them. Below: primary libraries for the formats that show up most often in scientific stacks. Blank or “thin” cells mean no serious first-class library I would ship on — use FFI to C, or convert upstream.

Maturity key: reference = de facto / official; strong = production-grade community; usable = works with caveats; thin = early or niche; gap = use another language or bind C.

Coverage matrix (high level)

Python - still the easiest way

If a format matters, Python usually has the nicest API and the most examples.

Verdict: richest ecosystem. Production science pipelines still default here for interactive work and for defining interchange.

C - where the durable cores live

Verdict: if you need longevity, MPI, or something every other language wraps, start in C (or accept that you will).

Fortran - still first-class for arrays on disk

Verdict: excellent for HDF5/NetCDF/FITS in HPC models. Columnar cloud formats are not where Fortran spends its energy.

Java - enterprise science and genomics tooling

Verdict: strongest non-Python home for genomics (htsjdk) and a serious NetCDF/HDF5/Zarr story for Earth science JVMs.

C# / .NET managed readers catching up

Verdict: Parquet and HDF5 are genuinely good on modern .NET. Domain formats (Zarr, CRAM, ASDF) lag.

Rust - safe reimplementations and good FFI

Verdict: Arrow/Parquet/Zarr/noodles are the places Rust already feels production-native. Hierarchical science formats are improving but still dual-track (FFI vs pure).

Elixir - BEAM-native where it counts

Verdict: for cloud-chunked arrays and columnar interchange on the BEAM, the story is real. For FITS/HDF5/CRAM, plan a boundary service or NIF.

Ruby - bindings and BioRuby, not format leadership

Verdict: fine as a consumer via Arrow or bio-samtools. Not where new scientific format work happens.

Zig - early but pointed at columnar I/O

Verdict: Zig’s interesting scientific niche today is columnar (Parquet/Arrow), not the classic HPC hierarchy formats. Expect rapid change.

How to choose

1. Interchange with astronomers — FITS everywhere; ASDF when the mission already standardized on it (Python/C first).
2. Climate / Earth arrays on object stores — Zarr (+ xarray); NetCDF-4 when the model community demands it.
3. Genomics alignments and variants — HTSlib / pysam / htsjdk / noodles; CRAM for cold storage.
4. Analytics over tables — Parquet on disk, Arrow in process; language choice matters less once Arrow is the glue.
5. BEAM services — ExZarr and ex_arrow for Zarr/Arrow/Parquet; do not pretend HDF5 or CRAM are first-class on Elixir yet.
6. Systems languages without a science runtime — C cores, or Rust when you want memory safety and are willing to live with crate churn.

Closing

Scientific formats stopped being “whatever the instrument vendor shipped” and became a small set of architectural bets: hierarchical files (HDF5/NetCDF), cloud chunks (Zarr/COG), reference compression (CRAM), and columnar analytics (Parquet/Arrow). Language support follows those bets unevenly. Python and C still define the center; Java owns much of genomics tooling; Rust and Elixir are credible where formats are specs rather than 30-year C APIs; Zig is entering through Parquet/Arrow; Fortran remains indispensable for the models that never left NetCDF/HDF5; Ruby and C# are selective rather than universal.

If you are picking a stack for a new pipeline, pick the format for the access pattern first, then the language that already has a boring library for it. Inventing a fifteenth niche extension is almost never the hard part anymore — reading it in ten years, from three languages, on object storage, is.

Originally published at https://thanos.github.io on October 31, 2026.