Infino Overview

Relevant source files

• .devin/wiki.json
• .github/workflows/api-docs.yml
• AGENTS.md
• Cargo.lock
• Cargo.toml
• README.md
• crate-docs.md
• docs/architecture/overview.md
• docs/architecture/supertable.md
• infino-node/Cargo.lock
• infino-python/Cargo.lock
• infino-python/pyproject.toml
• llms.txt
• src/lib.rs
• src/supertable/gc/mod.rs
• tests/license_audit.rs

Infino is a high-performance, embedded retrieval engine designed to run SQL, full-text search (BM25), and vector search over a single copy of data stored on object storage README.md9-10 It is built as a Rust library with first-party bindings for Python and Node.js README.md30-51

Infino follows an object-storage-native philosophy: data and indexes reside in standard Parquet-compatible files on S3, Azure Blob, GCS, or local disk, enabling snapshot-isolated reads and atomic commits without requiring a dedicated server or daemon README.md15

Design Philosophy

The engine is built on four core pillars:

1. Speed-per-dollar: Optimized for object-storage economics while maintaining search-engine speeds (microsecond-range BM25 queries on warm data) README.md13
2. Multi-modal Queries: Keyword, vector, and SQL filters can be combined in a single query over the same rows README.md14
3. Embedded & Stateless: Runs in-process as a library. Compute is stateless; state is managed via a manifest pointer in object storage AGENTS.md7-9
4. Open Format: Uses the superfile format—a valid Apache Parquet file with search indexes embedded alongside columnar data—preventing vendor lock-in README.md16 llms.txt63-64

For a detailed breakdown of the fundamental concepts and performance trade-offs, see Core Concepts & Terminology.

System Architecture

Infino's architecture is divided into two primary layers: the Superfile (the on-disk format) and the Supertable (the table management and query layer).

Natural Language to Code Entity Mapping

The following diagram maps high-level system concepts to their specific implementations in the codebase.

Concept to Code Entity Mapping

Sources: src/lib.rs131-152 Cargo.toml154-166 infino-python/pyproject.toml1-3 llms.txt68-71

Subsystem Relationships

The Supertable coordinates multiple superfile instances. It uses a Manifest to track which files belong to a specific snapshot and manages mutations through append, update, and delete operations.

Subsystem Interaction Diagram

Sources: src/lib.rs102-120 src/supertable/gc/mod.rs24-35 docs/architecture/supertable.md3-25 AGENTS.md7-9

Key Concepts

Superfile

The fundamental unit of storage. It is a valid Parquet file where FTS and Vector index regions are spliced into the layout. Standard Parquet readers (like DuckDB or DataFusion) can read the columns directly while skipping the index regions llms.txt63-64 AGENTS.md7-9

• FTS Index: Provides keyword search using BM25 relevance scoring README.md104 docs/architecture/supertable.md77-78
• Vector Index: Supports nearest-neighbor search (cosine, L2, dot) over high-dimensional embeddings README.md105 docs/architecture/supertable.md69-71

For details, see Superfile Format.

Supertable

A multi-file abstraction that provides a consistent view of a table.

• Snapshot Isolation: Readers see a consistent state via a Manifest even while writes are in progress. Writes are append-only and become visible only upon an atomic commit docs/architecture/supertable.md16-21
• SQL Integration: Reached through query_sql. Search functions are exposed as SQL table-valued functions, allowing search to be joined and filtered via SQL docs/architecture/supertable.md94-100

For details, see Supertable Layer.

Storage & Connectivity

Infino uses a URI-based connection string to determine the storage backend crate-docs.md109-113

• memory://: In-process ephemeral storage.
• s3://: Amazon S3.
• az://: Azure Blob Storage.
• gs://: Google Cloud Storage.
• file:// or path: Local filesystem.

For details, see Catalog & Connection API.

Child Pages

• Getting Started & Developer Setup — Environment setup, Rust toolchain requirements, and running tests/benches.
• Core Concepts & Terminology — Deep dive into the mental model, snapshot isolation, and performance tradeoffs.

Sources: README.md9-17 Cargo.toml93-166 src/lib.rs1-152 docs/architecture/supertable.md1-140 AGENTS.md1-58