Error Handling

Relevant source files

• public-api.txt
• src/config/mod.rs
• src/error.rs
• src/memory/datafusion_pool.rs
• src/memory/mod.rs
• src/superfile/builder.rs
• src/superfile/error.rs
• src/superfile/mod.rs
• src/superfile/stats.rs
• src/supertable/error.rs
• src/supertable/optimize/mod.rs
• src/supertable/query/sql.rs
• tests/superfile/pipeline.rs

Infino employs a tiered error taxonomy designed to provide precise failure information within internal subsystems while exposing a stable, coarse-grained API to language bindings. The system distinguishes between validation errors (schema/configuration), transient operational failures (storage retries/OCC contention), memory budget exhaustion, and permanent data corruption or I/O errors.

Error Taxonomy

The error hierarchy is built using the thiserror crate, allowing for structured source chaining and helpful error messages.

InfinoError (The Public Surface)

InfinoError is the top-level enum returned by all public methods in the curated API src/error.rs36-74 It collapses internal complexity into a stable set of variants that language bindings (Python/Node.js) can easily map to native exceptions.

Variant	Source Subsystems	Purpose
NotFound	Storage, Catalog	Requested table, object, or column does not exist src/error.rs37-39
AlreadyExists	Storage, Catalog	Name conflict or OCC precondition failure (e.g., table already exists) src/error.rs41-43
Schema	Superfile/Supertable Builder	Validation failures: missing ID column, wrong types, reserved names src/error.rs45-47
Cardinality	Mutations	Row count mismatches or mutation cap exceeded src/error.rs49-52
Io	Storage, LocalFS	Network failures, permission issues, or exhausted retries src/error.rs54-56
Query	SQL/FTS/Vector Engine	Planning failures or malformed query syntax src/error.rs58-60
OverBudget	Memory Budget	Query exceeded the connection's heap limit src/error.rs62-68
Backend	Manifest, Commits	Internal failures, encoding errors, or manifest corruption src/error.rs70-73

Sources: src/error.rs36-74 src/error.rs76-146

Internal Subsystem Errors

Detailed error enums exist for specific layers to handle internal logic and recovery:

• BuildError: Raised during the construction of a Superfile or SupertableWriter. It covers schema constraints such as ensuring the id_column is Decimal128(38, 0) src/supertable/error.rs36-37 and validating that vector columns are FixedSizeList<Float32> src/supertable/error.rs54-59
• CommitError: Specific to the Supertable manifest publication path. It includes WriteContentionExhausted for OCC failures src/supertable/error.rs172-177 and ManifestError for sharding boundary violations src/supertable/error.rs181-193
• OptimizeError: Covers compaction and GC failures, including SidecarConflict when a mutation hits a file being compacted public-api.txt104-116
• QueryError: DataFusion-specific errors, mapping ResourcesExhausted to OverBudget src/supertable/query/sql.rs73-78
• StorageError: Encapsulates failures from the StorageProvider. It distinguishes between TransientExhausted (retryable) and Permanent (fatal) failures src/error.rs76-87

Sources: src/supertable/error.rs29-138 src/supertable/error.rs147-178 src/supertable/query/sql.rs73-78 src/error.rs76-87 public-api.txt104-116

Error Propagation and Mapping

The following diagram illustrates how low-level subsystem errors are transformed into the public InfinoError.

Mapping Internal Errors to Public API

Sources: src/error.rs89-146

Memory Budgeting and OverBudget Errors

Infino implements a per-connection memory budget to prevent heap exhaustion. The budget operates in two modes: measured (count only) and bounded (refuse allocations) src/memory/mod.rs41-49

• 90% Gate: To account for untracked small allocations, the budget enforces a ceiling at 90% of the configured value src/memory/mod.rs98-102
• SQL Spilling: When DataFusion's MemoryPool (implemented via ConnectionBudgetPool) hits the limit, it attempts to spill to disk (e.g., sort buffers) src/memory/datafusion_pool.rs14-19
• Terminal Refusal: If an operator cannot spill (e.g., hash-join build side) or the minimum reservation fails, it raises QueryError::OverBudget, which maps to InfinoError::OverBudget src/memory/datafusion_pool.rs20-27 src/supertable/query/sql.rs75

Sources: src/memory/mod.rs14-49 src/memory/datafusion_pool.rs4-27 src/memory/datafusion_pool.rs61-67 src/supertable/query/sql.rs71-78

Schema Validation Constraints

The SupertableWriter and SuperfileBuilder enforce strict schema invariants during the ingestion phase. Violations result in a BuildError.

Constraint	Error Variant	Implementation Site
ID column missing	MissingIdColumn	SupertableWriter src/supertable/error.rs33-34
ID column type mismatch	IdColumnWrongType	SupertableWriter src/supertable/error.rs36-37
Vector dimension mismatch	VectorColumnDimMismatch	SupertableWriter src/supertable/error.rs65-69
Nulls in vector columns	VectorColumnHasNulls	SupertableWriter src/supertable/error.rs75-78
Reserved column prefix	ReservedPrefixInColumnName	SupertableWriter src/supertable/error.rs89-90

Sources: src/supertable/error.rs29-138 src/superfile/error.rs14-96

Optimistic Concurrency Control (OCC) Failure Modes

Infino uses OCC for manifest updates. When two writers attempt to commit simultaneously, the StorageProvider primitive put_if_match (or put_atomic) is used to ensure only one succeeds.

OCC Flow and Error Handling

Sources: src/error.rs76-87 src/supertable/error.rs172-177

Compaction and GC Failures

Compaction and Garbage Collection (GC) are triggered via the optimize method src/supertable/optimize/mod.rs15-23

• Sidecar Conflicts: If a mutation (update/delete) lands during the merge window for a sealed sidecar, the operation fails with OptimizeError::SidecarConflict public-api.txt113-115
• GC Failures: The gc process can fail with GcError::Storage if the backend is unreachable, which is then wrapped into an OptimizeError::Gc src/supertable/optimize/mod.rs17-20
• Empty Merges: If compaction results in no remaining documents (e.g., all rows were deleted), it returns OptimizeError::EmptyMergedSuperfile public-api.txt108

Sources: src/supertable/optimize/mod.rs10-23 public-api.txt45-62 public-api.txt104-131