The Superfile is the fundamental on-disk storage unit of Infino. It is designed as a "super-powered" Parquet file that remains 100% spec-compliant while embedding high-performance search indices for Full-Text Search (FTS) and Vector retrieval docs/architecture/superfile.md3-8
By co-locating columnar data with specialized index structures, Infino achieves "object-storage-native" performance, allowing complex hybrid queries to execute directly against files stored on S3, Azure Blob, or local disk without requiring a separate index database docs/architecture/superfile.md14-25
A Superfile consists of three primary logical regions packed into a single binary artifact:
_id). This section is written by a standard Parquet writer and is readable by any standard Parquet tool like DuckDB or Apache Arrow docs/architecture/superfile.md33-37 src/superfile/builder.rs135-138inf.* key-value pairs. These keys act as pointers, storing the byte offsets and lengths of the embedded index blobs, allowing the SuperfileReader to locate them within the byte stream src/superfile/builder.rs17-20 docs/architecture/superfile.md39-43The following diagram illustrates how the SuperfileBuilder assembles disparate data streams into the unified Superfile format.
Superfile Assembly Pipeline
Sources: src/superfile/builder.rs6-21 src/superfile/builder.rs92-105 src/superfile/stats.rs14-20
The Superfile format is a container for three distinct data modalities. While the columnar body is handled by the standard Parquet writer, the search indices are managed by specialized builders and readers.
The SuperfileBuilder is a single-shot factory that consumes Arrow batches and produces an immutable byte buffer src/superfile/builder.rs6-12
SuperfileStats including n_docs and primary key ranges (id_min, id_max) src/superfile/stats.rs14-20SuperfileReader can verify these on open to detect corruption src/superfile/reader.rs67-80 src/superfile/error.rs114-118For details, see Superfile Builder & Format Layout.
The FTS subsystem provides BM25-ranked keyword search. It is integrated into the Superfile by routing specific LargeUtf8 columns into an FtsBuilder src/superfile/builder.rs107-112
For details, see Full-Text Search (FTS) Index.
The Vector subsystem handles high-dimensional similarity search. Unlike FTS columns, vector data is typically "stripped" from the Parquet body and stored exclusively in the embedded vector blob to save space src/superfile/builder.rs149-153
For details, see Vector Index.
The relationship between the high-level SuperfileReader and the underlying format components is shown below.
Superfile Reader Component Mapping
Sources: src/superfile/reader.rs117-122 src/superfile/mod.rs31-33 docs/architecture/superfile.md156-165
A core design principle of the Superfile is no lock-in. Because the file follows the Parquet specification, third-party tools can access the data without the Infino library docs/architecture/superfile.md45-56
| Feature | Infino Reader | Standard Parquet Reader (DataFusion/DuckDB) |
|---|---|---|
| Scalar Columns | Supported src/superfile/reader.rs25-29 | Supported |
Primary Key (_id) | Supported src/superfile/builder.rs135-138 | Supported |
| BM25 Search | Supported src/superfile/fts/reader.rs4-7 | Not Supported (Ignored) |
| Vector Search | Supported src/superfile/vector/reader.rs4-9 | Not Supported (Ignored) |
| Metadata | inf.* keys parsed src/superfile/reader.rs6-9 | inf.* keys ignored |
Sources: docs/architecture/superfile.md58-65 src/superfile/builder.rs17-20