amoxtli

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 28, 2026 License: MIT Imports: 27 Imported by: 0

README

Amoxtli

Amoxtli — « livre, codex » en nahuatl.

Bibliothèque Go pour ingérer des fichiers et y faire de la recherche documentaire, avec plusieurs backends d'index au choix.

  • Recherche: plein-texte (bleve), vectorielle (sqlite-vec) ou hybride PostgreSQL (pgvector + FTS natif). Les résultats de plusieurs index sont fusionnés par Reciprocal Rank Fusion pondérée.
  • Ingestion: conversion de fichiers (pandoc, LibreOffice, OCR/LLM), découpage des documents markdown en sections.
  • Code source: indexation déclaration par déclaration via tree-sitter en pur Go (Go, JS/TS, Python, PHP…).
  • Images: description par LLM vision, fichiers autonomes comme images embarquées, stockage adressé par contenu et réaffichage via MCP.
  • Qualité et exploitation: grounding (récupération vérifiée), sauvegarde et restauration des index.

Installation

go get github.com/bornholm/amoxtli

Attention

Le backend index/sqlitevec impose deux versions précises (ncruces/go-sqlite3 v0.23.0 et wazero ≥ v1.9.0) à préserver côté consommateur : voir Backend SQLite.

Démarrage rapide

Le magasin de documents (WithStore) et les indexeurs (WithIndexers) sont fournis explicitement, chacun construit par son propre constructeur. L'appelant possède les ressources qu'il crée et doit les fermer ; codex.Close() n'arrête que le runner de tâches.

// Magasin de documents (SQLite local, ou gorm.NewPostgresStore).
store, err := gorm.NewSQLiteStore("/data/kb/data.sqlite") // ingest/gorm
if err != nil { /* ... */ }
defer store.Close()

// Index plein-texte (bleve).
bleveIdx, err := bleve.OpenOrCreate(ctx, "/data/kb/index.bleve") // index/bleve
if err != nil { /* ... */ }
defer bleveIdx.Close()

codex, err := amoxtli.New(ctx,
    amoxtli.WithStore(store),
    amoxtli.WithIndexers(amoxtli.Indexer{ID: "bleve", Index: bleveIdx, Weight: 1.0}),
    amoxtli.WithDisableHyDE(), amoxtli.WithDisableJudge(), // pas de client LLM
)
if err != nil { /* ... */ }
defer codex.Close()

collID, _ := codex.CreateCollection(ctx, "docs")
taskID, _ := codex.IndexFile(ctx, collID, "guide.md", file)
results, _ := codex.Search(ctx, "comment faire…", amoxtli.WithSearchMaxResults(5))

Exemples complets et exécutables :

Ligne de commande

Le binaire cmd/amoxtli expose la bibliothèque sous forme d'outil : il indexe des fichiers locaux dans un espace de travail par projet (.amoxtli/), effectue des recherches (dont une recherche itérative --deep pilotée par LLM) et sert un serveur MCP (stdio ou HTTP) pour les agents.

go install github.com/bornholm/amoxtli/cmd/amoxtli@latest   # ou : make build
amoxtli init
amoxtli add ./docs/*.md                              # documentation
amoxtli add $(git ls-files '*.go')                   # code source (type=code, language=go)
amoxtli sync --base-dir . ./docs                     # arborescence, sources relatives (pas de chemin absolu indexé)
amoxtli search "modèle de concurrence"               # doc ET code
amoxtli search "modèle de concurrence" --filter '!type'        # documentation seule
amoxtli search "modèle de concurrence" --filter dirname=/docs  # métadonnées de fichier (filename, extension, size, mtime, dirname, indexed_at)
amoxtli add ./captures/*.png                         # images décrites par LLM vision (type=image)
amoxtli search "tableau de bord des ventes" --filter type=image
amoxtli image list                                   # images stockées
amoxtli image get <hash> -o schema.png               # en extraire une
amoxtli mcp stdio             # serveur MCP sur stdio (un processus par client)
amoxtli mcp http --addr :8080 # serveur MCP HTTP (processus partagé, multi-sessions)

Voir docs/cli.md pour la configuration (config.yaml, interpolation des secrets), les commandes CRUD et l'intégration MCP.

Documentation

  • Ligne de commande — CLI amoxtli : espace de travail, configuration, commandes CRUD, serveur MCP
  • Architecture — packages, indexeurs personnalisés et suite de conformité
  • Grounding (récupération vérifiée) — CheckGrounding, SearchIterative, décomposition, re-retrieval itératif et modes d'application (demote par défaut / filter)
  • Backend SQLite — déploiement local par fichiers (sqlite-vec, build WASM embarqué, versions épinglées)
  • Backend PostgreSQL — déploiement entièrement PostgreSQL (FTS + pgvector, fusion RRF)
  • Convertisseurs de fichiers — pandoc, LibreOffice, OCR/LLM
  • Indexation des images — description par LLM vision, images embarquées, coûts et cache, choix du modèle, garde-fous, réaffichage par un agent (blob store + MCP fetch_image)
  • Indexation de code source — tree-sitter pur Go, WithSourceCode, recherche croisée doc ↔ code, build tags
  • Tests — tests unitaires et d'intégration (Docker, Ollama, PostgreSQL)
  • Évaluation de la pertinence — Recall@k / MRR / nDCG@k, benchmarks SQuAD/BEIR, profils de récupération et résultats de référence
  • Stabilité de l'API — politique de compatibilité (série 0.x) et surface publique couverte
  • CHANGELOG — historique des versions

L'évaluation de la pertinence (Recall@k, MRR, nDCG — avec un benchmark multilingue sur jeux QA Hugging Face) est fournie par le package eval (voir docs/evaluation.md), et l'observabilité (OpenTelemetry) par le package telemetry (activée via amoxtli.WithObservability()).

Licence

MIT

Documentation

Overview

Package amoxtli provides a multi-backend document indexing and file-ingestion library: full-text search (bleve), vector search (sqlite-vec), weighted result merging, markdown chunking, file conversion and snapshot/restore of indexes.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound = ingest.ErrNotFound
	ErrCanceled = task.ErrCanceled
	// ErrCursorFilterMismatch signals a cursor replayed with a different search
	// filter than the one it was issued for: restart from the first page.
	ErrCursorFilterMismatch = ingest.ErrCursorFilterMismatch
)
View Source 
var Stages = []Stage{StageHyDE, StageJudge, StageGrounding, StageRerank, StageDecompose, StageReformulate}

Stages lists every stage accepted by WithStageLLMClient.

Functions

This section is empty.

Types

type Codex 

type Codex struct {
	// contains filtered or unexported fields
}

Codex is the main embedded instance: a store, index pipeline and task runner behind a single API.

func New 

func New(ctx context.Context, funcs ...Option) (*Codex, error)

New creates a new embedded Codex instance.

A store (WithStore) and an index (WithIndex or WithIndexers) are required; build them with the dedicated constructors, e.g. gorm.NewSQLiteStore / gorm.NewPostgresStore for the store and bleve.OpenOrCreate / sqlitevec.NewIndex / postgres.NewIndex for the indexers. The caller owns the resources it constructs and must Close them; Codex.Close only stops the task runner.

func (*Codex) Backup 

func (c *Codex) Backup(ctx context.Context) (io.ReadCloser, error)

Backup streams a snapshot of the index, the document store and — when one is configured — the blob store, as a multipart archive.

func (*Codex) Blobs added in v0.9.0 

func (c *Codex) Blobs() blob.Store

Blobs returns the configured blob store (WithBlobStore), or nil. Consumers use it to serve back the images referenced by `amoxtli://images/<hash>` in the content of the sections they retrieve.

func (*Codex) CancelTask added in v0.0.3 

func (c *Codex) CancelTask(ctx context.Context, id task.ID) error

CancelTask cancels a scheduled or running task.

func (*Codex) CheckGrounding added in v0.0.2 

func (c *Codex) CheckGrounding(ctx context.Context, query string, results []*index.SearchResult) (*retrieval.GroundingResult, error)

CheckGrounding evaluates whether the given search results support a reliable, grounded answer to the query, returning a verdict (status + score + explanation). It is a standalone step, fully decoupled from Search: run Search first, then pass its results here to decide whether to trust them or abstain. Requires WithGroundingCheck and an LLM client; otherwise it returns an error.

func (*Codex) CleanupIndex 

func (c *Codex) CleanupIndex(ctx context.Context, collections ...model.CollectionID) (task.ID, error)

CleanupIndex schedules a cleanup of orphaned documents and obsolete index entries.

func (*Codex) Close 

func (c *Codex) Close() error

Close stops the task runner, waiting up to the configured close timeout (WithCloseTimeout, default 30s) for in-flight indexing tasks to drain, then removes the ingestion staging directory. Resources passed in through WithStore, WithIndex/WithIndexers and WithTaskRunner are owned by the caller and must be closed by them.

func (*Codex) CreateCollection 

func (c *Codex) CreateCollection(ctx context.Context, label string) (model.CollectionID, error)

CreateCollection creates a new collection and returns its ID.

func (*Codex) DeleteBySource 

func (c *Codex) DeleteBySource(ctx context.Context, source *url.URL) error

DeleteBySource removes all documents and index entries for the given source URL.

func (*Codex) GetSectionsByIDs 

func (c *Codex) GetSectionsByIDs(ctx context.Context, ids []model.SectionID) (map[model.SectionID]model.Section, error)

GetSectionsByIDs returns the sections matching the given IDs, typically used to retrieve the content behind search results.

func (*Codex) Index 

func (c *Codex) Index() index.Index

Index returns the underlying index.Index for advanced usage.

func (*Codex) IndexFile 

func (c *Codex) IndexFile(ctx context.Context, collectionID model.CollectionID, filename string, r io.Reader, funcs ...IndexFileOption) (task.ID, error)

IndexFile indexes a file into the given collection. Returns a task.ID that can be used to track progress via TaskState.

func (*Codex) ListTasks added in v0.0.3 

func (c *Codex) ListTasks(ctx context.Context) ([]task.StateHeader, error)

ListTasks returns the headers of every task known to the runner.

func (*Codex) Manager 

func (c *Codex) Manager() *ingest.Manager

Manager returns the underlying ingest.Manager for advanced usage.

func (*Codex) Reindex 

func (c *Codex) Reindex(ctx context.Context) (task.ID, error)

Reindex schedules a rebuild of the whole index.

func (*Codex) ReindexCollection 

func (c *Codex) ReindexCollection(ctx context.Context, collectionID model.CollectionID) (task.ID, error)

ReindexCollection schedules a rebuild of the index for a single collection.

func (*Codex) Restore 

func (c *Codex) Restore(ctx context.Context, r io.Reader) error

Restore synchronously restores a snapshot previously produced by Backup.

func (*Codex) Search 

func (c *Codex) Search(ctx context.Context, query string, funcs ...SearchOption) ([]*index.SearchResult, error)

Search performs a semantic search across the indexed documents. It returns a single page of results (metadata filtering via WithSearchFilter and reranking via WithReranking still apply); use SearchPage when the pagination cursor is needed.

func (*Codex) SearchIterative added in v0.0.2 

func (c *Codex) SearchIterative(ctx context.Context, query string, funcs ...SearchOption) (*retrieval.Result, error)

SearchIterative runs the explicit re-retrieval orchestration on top of Search: optional query decomposition and, when a grounding checker and reformulator are configured, grounding-gated iterative re-retrieval. It returns the fused evidence together with the final grounding verdict (retrieval.Result). It is a separate entry point from Search (which stays a plain, single-shot retrieval) and from CheckGrounding; with none of the orchestration options enabled it is equivalent to Search.

func (*Codex) SearchPage added in v0.0.3 

func (c *Codex) SearchPage(ctx context.Context, query string, funcs ...SearchOption) (*SearchPage, error)

SearchPage performs a search and returns a page of results plus the cursor to resume from (WithSearchCursor). It supports metadata filtering (WithSearchFilter), reranking (WithReranking) and cursor pagination. Unlike Search it always over-fetches a bounded candidate window so that a next cursor can be produced from the first page.

func (*Codex) TaskState 

func (c *Codex) TaskState(ctx context.Context, id task.ID) (*task.State, error)

TaskState returns the current state of an indexing task.

type IndexFileOption 

type IndexFileOption func(*IndexFileOptions)

IndexFileOption configures an IndexFile call.

func WithIndexFileCollections 

func WithIndexFileCollections(ids ...model.CollectionID) IndexFileOption

WithIndexFileCollections associates the indexed file with the given collection IDs.

func WithIndexFileETag 

func WithIndexFileETag(etag string) IndexFileOption

WithIndexFileETag sets the ETag for the indexed file (used for deduplication).

func WithIndexFileMetadata added in v0.0.3 

func WithIndexFileMetadata(metadata map[string]any) IndexFileOption

WithIndexFileMetadata attaches arbitrary metadata to the indexed document, used for metadata filtering at search time (see WithSearchFilter).

func WithIndexFileSource 

func WithIndexFileSource(source *url.URL) IndexFileOption

WithIndexFileSource sets the source URL for the indexed file.

type IndexFileOptions 

type IndexFileOptions struct {
	Source      *url.URL
	ETag        string
	Collections []model.CollectionID
	// Metadata is arbitrary document metadata used for filtering at search time.
	Metadata map[string]any
}

IndexFileOptions holds options for IndexFile calls.

type Indexer 

type Indexer struct {
	// ID identifies the indexer in the pipeline (e.g. "bleve", "postgres").
	ID string
	// Index is any implementation of the index.Index contract.
	Index index.Index
	// Weight is the relative weight of this indexer when merging scores.
	Weight float64
}

Indexer identifies a weighted index.Index inside the search pipeline.

type Option 

type Option func(*options)

Option is a function that configures a Codex instance.

func WithBlobStore added in v0.9.0 

func WithBlobStore(store blob.Store) Option

WithBlobStore makes indexed images displayable again, not merely searchable: each described image is stored, content-addressed, and referenced in the document by the internal URI `amoxtli://images/<hash>`. Unlike a data URI that destination survives the rendering of a chunk, so a consumer (typically an MCP agent) sees the reference next to the description and can fetch the bytes back.

It also feeds the image enrichment (WithImageEnrichment) unless that one was given a store of its own, and lets the cleanup task collect the blobs no document references any more.

func WithCloseTimeout added in v0.0.3 

func WithCloseTimeout(d time.Duration) Option

WithCloseTimeout bounds how long Close waits for in-flight indexing tasks to drain before giving up (default 30s). A non-positive duration keeps the default.

func WithDisableHyDE 

func WithDisableHyDE() Option

WithDisableHyDE disables the HyDE query transformer.

func WithDisableJudge 

func WithDisableJudge() Option

WithDisableJudge disables the Judge results transformer.

func WithFileConverter 

func WithFileConverter(fc convert.Converter) Option

WithFileConverter sets a file converter for converting files before indexing.

func WithGroundingCheck added in v0.0.2 

func WithGroundingCheck() Option

WithGroundingCheck enables the fused evidence evaluator: a single LLM call that both relevance-filters the retrieved evidence and judges whether it supports a reliable answer (the grounding γ verdict). It makes CheckGrounding available as a standalone step and gates the re-retrieval loop of SearchIterative. When enabled it replaces the Judge results transformer in the pipeline (Search then relies on the evaluator for relevance filtering), avoiding a redundant LLM pass over the same evidence. Requires an LLM client (WithLLMClient). Disabled by default.

func WithGroundingFailOpen added in v0.0.3 

func WithGroundingFailOpen() Option

WithGroundingFailOpen makes Search degrade gracefully when the grounding evidence evaluator (an LLM call) fails: instead of returning the error, Search logs a warning and returns the retrieved results unfiltered. Without it, a transient LLM failure in the evaluator fails the whole Search. Only meaningful together with WithGroundingCheck. Disabled by default (fail-closed).

func WithGroundingMinScore added in v0.0.2 

func WithGroundingMinScore(minScore float64) Option

WithGroundingMinScore sets the grounding score threshold below which the verdict is considered not confident (default 0.4). This gates iterative re-retrieval only — whether SearchIterative reformulates and searches again — and does NOT control the relevance filtering/demotion of the evidence itself (that is WithGroundingMode). Only meaningful together with WithGroundingCheck.

func WithGroundingMode added in v0.0.3 

func WithGroundingMode(mode retrieval.GroundingMode) Option

WithGroundingMode selects what the evaluator's relevance signal does to the evidence: retrieval.GroundingDemote (the default) keeps every retrieved document but ranks the irrelevant ones last — preserving recall@k while surfacing the grounded evidence first; retrieval.GroundingFilter drops the sections judged irrelevant — maximising list precision but truncating recall, which suits short-list RAG. Only meaningful together with WithGroundingCheck.

func WithImageEnrichment added in v0.9.0 

func WithImageEnrichment(funcs ...imagetext.OptionFunc) Option

WithImageEnrichment describes the images embedded in indexed documents with a vision LLM and inserts the descriptions, as text, next to each image — making them findable through the same full-text and semantic search as the rest of the document. It applies to native markdown files as well as to the output of the file converters (pandoc, LibreOffice, GenAI OCR); source files are left alone.

Without an explicit describer, one is built from the LLM client set by WithLLMClient, which must then support image attachments. Pass a dedicated vision model — and the description cache — with imagetext.WithDescriber: 

amoxtli.WithImageEnrichment(
    imagetext.WithDescriber(vision.NewLLMDescriber(visionClient)),
    imagetext.WithMaxImagesPerDocument(16),
)

Relative image paths are resolved against the directory of the indexed file (file: sources only) and strictly confined to it; remote images are never fetched unless imagetext.WithHTTPFetcher is given.

func WithIndex 

func WithIndex(idx index.Index) Option

WithIndex provides a ready-made index.Index, bypassing pipeline composition entirely (including the HyDE/Judge/dedup transformers). Mutually exclusive with WithIndexers. The caller owns and must close the index.

func WithIndexers 

func WithIndexers(indexers ...Indexer) Option

WithIndexers declares the set of indexers composing the search pipeline, each with its relative weight. It can be called several times; indexers accumulate.

Any implementation of index.Index can be plugged in; conformance can be verified with the index/testsuite package. Build the backends with their own constructors, e.g. bleve.OpenOrCreate(...), sqlitevec.NewIndex(...) or postgres.NewIndex(...), and wrap each in an Indexer.

func WithIterativeRetrieval added in v0.0.2 

func WithIterativeRetrieval(rounds int) Option

WithIterativeRetrieval enables grounding-driven re-retrieval in SearchIterative: when the evidence is not confidently grounded the query is reformulated and searched again, up to rounds times (rounds <= 0 means 1). Requires WithGroundingCheck and an LLM client.

func WithLLMClient 

func WithLLMClient(client llm.Client) Option

WithLLMClient sets the default LLM client used by every LLM retrieval stage (HyDE, Judge, grounding, reranker, decomposition, reformulation) unless a stage has its own client (WithStageLLMClient).

func WithMaxTotalWords 

func WithMaxTotalWords(n int) Option

WithMaxTotalWords bounds the prompt size (in words) shared by the LLM retrieval stages: the Judge transformer, the LLM reranker and the grounding evidence evaluator. Words are only a coarse proxy for tokens, so keep it low enough that the resulting prompt fits the chat endpoint's context window.

func WithMaxWordsPerSection 

func WithMaxWordsPerSection(n int) Option

WithMaxWordsPerSection sets the maximum number of words per document section.

func WithMaxWordsPerSectionInPrompt added in v0.2.0 

func WithMaxWordsPerSectionInPrompt(n int) Option

WithMaxWordsPerSectionInPrompt bounds how many words of each retrieved section are included in the prompts of the LLM retrieval stages (Judge, reranker, evidence evaluator), on top of the total WithMaxTotalWords budget. Relevance can almost always be judged from the beginning of a section, so a low cap (default 200) cuts the per-search prompt cost with little quality impact. <= 0 keeps the default.

func WithObservability added in v0.0.3 

func WithObservability() Option

WithObservability wraps the configured LLM client (WithLLMClient) with the OpenTelemetry decorator so the HyDE, Judge, grounding and reranker LLM calls emit spans and token/latency metrics under the amoxtli instrumentation scope. Search latency is instrumented unconditionally. It has no effect unless an LLM client is set. Embeddings issued by an index built by the caller (sqlitevec/postgres) are not covered here — wrap that client yourself with llmx.NewObservableClient if you want their metrics too.

func WithPersistentTasks added in v0.0.3 

func WithPersistentTasks(stagingDir string) Option

WithPersistentTasks enables a persistent, gorm-backed task runner that shares the document store's database. Pending indexing tasks — and tasks left running by a previous, interrupted process — are resumed on startup. It requires a gorm-backed store (gorm.NewSQLiteStore / gorm.NewPostgresStore) and a stable stagingDir where files awaiting indexing are kept across restarts (it must be a durable path, not a temporary directory). It is ignored when a runner is supplied explicitly with WithTaskRunner.

func WithQueryDecomposition added in v0.0.2 

func WithQueryDecomposition(maxSubQueries int) Option

WithQueryDecomposition enables splitting a complex question into at most maxSubQueries sub-questions in SearchIterative, searching each and fusing their evidence. Requires an LLM client. maxSubQueries <= 0 keeps the default (3).

func WithReranking added in v0.0.3 

func WithReranking() Option

WithReranking enables LLM-based reranking of the fused search results before pagination: the retrieved candidates are reordered by relevance to the query, reusing the WithMaxTotalWords budget to bound the prompt size. Requires an LLM client (WithLLMClient). Disabled by default.

func WithSnapshotBoundary 

func WithSnapshotBoundary(boundary string) Option

WithSnapshotBoundary overrides the multipart boundary used by Backup/Restore.

func WithSourceCode added in v0.0.4 

func WithSourceCode(registry *sourcecode.Registry) Option

WithSourceCode enables source-code indexing for the file extensions registered in the registry (see sourcecode.DefaultRegistry). Source files are parsed with tree-sitter into declaration-level sections and tagged with `type=code` and `language=<name>` metadata, filterable at search time with WithSearchFilter.

func WithStageLLMClient added in v0.2.0 

func WithStageLLMClient(stage Stage, client llm.Client) Option

WithStageLLMClient assigns a dedicated LLM client to a single retrieval stage, overriding WithLLMClient for that stage. The main lever on the LLM cost of a search: point the cheap, high-volume stages (HyDE, Judge) at a small fast model while keeping a stronger model for the rest. It can be called once per stage; a nil client removes the override.

func WithStore 

func WithStore(store ingest.Store) Option

WithStore sets the document store. It is required. Build it with gorm.NewSQLiteStore or gorm.NewPostgresStore (or any ingest.Store). The caller owns and must close the store.

func WithTaskParallelism 

func WithTaskParallelism(n int) Option

WithTaskParallelism sets the number of concurrent tasks allowed.

func WithTaskRunner 

func WithTaskRunner(runner task.Runner) Option

WithTaskRunner provides a custom task.Runner implementation.

type SearchOption 

type SearchOption func(*SearchOptions)

SearchOption configures a Search call.

func WithSearchCollections 

func WithSearchCollections(ids ...model.CollectionID) SearchOption

WithSearchCollections restricts the search to the given collection IDs.

func WithSearchCursor added in v0.0.3 

func WithSearchCursor(cursor string) SearchOption

WithSearchCursor resumes pagination after the given opaque cursor (the NextCursor returned by a previous SearchPage).

func WithSearchFilter added in v0.0.3 

func WithSearchFilter(conditions ...index.Condition) SearchOption

WithSearchFilter restricts results to documents whose metadata satisfies the given filter (see index.Eq/Ne/Gt/Gte/Lt/Lte/In). Requires a store implementing ingest.MetadataProvider.

func WithSearchMaxResults 

func WithSearchMaxResults(n int) SearchOption

WithSearchMaxResults sets the maximum number of search results (page size).

type SearchOptions 

type SearchOptions struct {
	// MaxResults is the page size (number of results per page).
	MaxResults  int
	Collections []model.CollectionID
	// Filter restricts results to documents whose metadata matches every
	// condition. Requires a store implementing ingest.MetadataProvider (the
	// gorm stores do).
	Filter index.Filter
	// Cursor resumes pagination after a previous SearchPage (empty = first page).
	Cursor string
}

SearchOptions holds options for Search calls.

type SearchPage added in v0.0.3 

type SearchPage struct {
	Results    []*index.SearchResult
	NextCursor string
	// Grounding is the sufficiency verdict computed by the evidence evaluator
	// during this search (nil when grounding is not configured or the evaluator
	// failed under fail-open). It is the same verdict CheckGrounding would
	// return, exposed here so callers need not re-run the evaluation.
	Grounding *retrieval.GroundingResult
}

SearchPage is a page of search results together with the opaque cursor used to fetch the next page (empty when the last page has been reached).

type Stage added in v0.2.0 

type Stage string

Stage identifies an LLM-driven retrieval stage that can be given a dedicated client with WithStageLLMClient. 

const (
	// StageHyDE is the hypothetical-answer query expansion (semantic indexes).
	StageHyDE Stage = "hyde"
	// StageJudge is the relevance judge filtering retrieved sections.
	StageJudge Stage = "judge"
	// StageGrounding is the fused evidence evaluator (relevance + verdict).
	StageGrounding Stage = "grounding"
	// StageRerank is the LLM reranker reordering fused results.
	StageRerank Stage = "rerank"
	// StageDecompose splits a complex question into sub-questions.
	StageDecompose Stage = "decompose"
	// StageReformulate rewrites the query for iterative re-retrieval.
	StageReformulate Stage = "reformulate"
)

Source Files

View all Source files

Directories

Path Synopsis
Package blob stores the binary content amoxtli needs to hand back verbatim — today the images it describes.
 Package blob stores the binary content amoxtli needs to hand back verbatim — today the images it describes.
fs
Package fs implements a filesystem-backed blob store, for the local (SQLite) workspaces: content under <dir>/<2 hex>/<hash>, its media type in a <hash>.json sidecar.
 Package fs implements a filesystem-backed blob store, for the local (SQLite) workspaces: content under <dir>/<2 hex>/<hash>, its media type in a <hash>.json sidecar.
gorm
Package gorm implements a database-backed blob store covering SQLite and PostgreSQL with the same code, on the model of ingest/gorm and task/gorm.
 Package gorm implements a database-backed blob store covering SQLite and PostgreSQL with the same code, on the model of ingest/gorm and task/gorm.
testsuite
Package testsuite is the conformance suite every blob.Store implementation must pass.
 Package testsuite is the conformance suite every blob.Store implementation must pass.
cmd
amoxtli command
Command amoxtli is the command line interface to the amoxtli library: per-project document indexing, hybrid search and an MCP server for agents.
 Command amoxtli is the command line interface to the amoxtli library: per-project document indexing, hybrid search and an MCP server for agents.
genai
libreoffice
pandoc
vision
Package vision converts standalone image files (.png, .jpg, ...) to markdown by describing them with a vision LLM.
 Package vision converts standalone image files (.png, .jpg, ...) to markdown by describing them with a vision LLM.
Package eval provides an offline retrieval-quality harness: it runs a golden dataset (query → expected relevant sources) through a Retriever and reports the standard ranking metrics — Recall@k, Mean Reciprocal Rank (MRR) and nDCG@k — so that changes to the retrieval stack (fusion, reranking, grounding) can be validated objectively rather than by feel.
 Package eval provides an offline retrieval-quality harness: it runs a golden dataset (query → expected relevant sources) through a Retriever and reports the standard ranking metrics — Recall@k, Mean Reciprocal Rank (MRR) and nDCG@k — so that changes to the retrieval stack (fusion, reranking, grounding) can be validated objectively rather than by feel.
beir
Package beir loads datasets in the BEIR format (the de-facto information retrieval benchmark) into an amoxtli evaluation corpus + query set.
 Package beir loads datasets in the BEIR format (the de-facto information retrieval benchmark) into an amoxtli evaluation corpus + query set.
hfqa
Package hfqa loads Hugging Face extractive-QA datasets in the SQuAD JSON format into an amoxtli evaluation corpus + query set.
 Package hfqa loads Hugging Face extractive-QA datasets in the SQuAD JSON format into an amoxtli evaluation corpus + query set.
example
convert command
Command convert demonstrates two things at once:
 Command convert demonstrates two things at once:
postgres command
Command postgres demonstrates amoxtli backed entirely by PostgreSQL: the document store and the hybrid index (index/postgres) share a single database.
 Command postgres demonstrates amoxtli backed entirely by PostgreSQL: the document store and the hybrid index (index/postgres) share a single database.
sourcecode command
Command sourcecode demonstrates indexing source code alongside documentation and searching across both.
 Command sourcecode demonstrates indexing source code alongside documentation and searching across both.
sqlite command
Command sqlite demonstrates amoxtli backed entirely by local files: a SQLite document store and a Bleve full-text index.
 Command sqlite demonstrates amoxtli backed entirely by local files: a SQLite document store and a Bleve full-text index.
vision command
Command vision demonstrates indexing image files: each image is described by a vision LLM (title, detailed description, transcription of the visible text), and that description is indexed as ordinary markdown — so images become findable through the same full-text and semantic search as every other document, and are tagged with type=image for metadata filtering.
 Command vision demonstrates indexing image files: each image is described by a vision LLM (title, detailed description, transcription of the visible text), and that description is indexed as ordinary markdown — so images become findable through the same full-text and semantic search as every other document, and are tagged with type=image for metadata filtering.
bleve
filtertest
Package filtertest is the shared conformance suite for index.Filter semantics.
 Package filtertest is the shared conformance suite for index.Filter semantics.
pipeline
postgres
Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance).
 Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance).
sqlitevec
sqlitevec/internal/vec
testsuite
gorm
internal
build
Package build carries version information injected at build time via -ldflags "-X github.com/bornholm/amoxtli/internal/build.Version=...".
 Package build carries version information injected at build time via -ldflags "-X github.com/bornholm/amoxtli/internal/build.Version=...".
cli
Package cli implements the amoxtli command line interface: a thin layer over the library that manages a per-project workspace (.amoxtli directory) holding the configuration and the indexed data.
 Package cli implements the amoxtli command line interface: a thin layer over the library that manages a per-project workspace (.amoxtli directory) holding the configuration and the indexed data.
cli/config
Package config defines the amoxtli workspace configuration file (.amoxtli/config.yaml): parsing, environment variable interpolation and validation.
 Package config defines the amoxtli workspace configuration file (.amoxtli/config.yaml): parsing, environment variable interpolation and validation.
cli/mcpserver
Package mcpserver exposes an amoxtli workspace over the Model Context Protocol (stdio), so an LLM agent can search the local corpus.
 Package mcpserver exposes an amoxtli workspace over the Model Context Protocol (stdio), so an LLM agent can search the local corpus.
cli/runtime
Package runtime turns a workspace configuration into a live amoxtli.Codex, owning (and closing) the resources the library constructors require: the document store, the index backends and the LLM client.
 Package runtime turns a workspace configuration into a live amoxtli.Codex, owning (and closing) the resources the library constructors require: the document store, the index backends and the LLM client.
cli/workspace
Package workspace locates and describes an amoxtli workspace: a project directory holding a .amoxtli/ configuration directory, discovered by walking up the filesystem from a starting point (like git does).
 Package workspace locates and describes an amoxtli workspace: a project directory holding a .amoxtli/ configuration directory, discovered by walking up the filesystem from a starting point (like git does).
filterexpr
filternorm
Package filternorm holds the value normalization and comparison rules that define the semantics of index.Filter.
 Package filternorm holds the value normalization and comparison rules that define the semantics of index.Filter.
ignore
Package ignore implements .amoxtlignore support: gitignore-style exclusion rules that decide whether a file should be skipped by "amoxtli add".
 Package ignore implements .amoxtlignore support: gitignore-style exclusion rules that decide whether a file should be skipped by "amoxtli add".
ollamatest
Package ollamatest provides shared helpers for the integration tests that run against an Ollama testcontainer.
 Package ollamatest provides shared helpers for the integration tests that run against an Ollama testcontainer.
syncx
text
workflow
Package llmx provides reusable decorators around github.com/bornholm/genai llm.Client.
 Package llmx provides reusable decorators around github.com/bornholm/genai llm.Client.
imagetext
Package imagetext makes the images embedded in a markdown document searchable: each resolvable image is described by a vision LLM and the description is inserted, as plain text, right after the block carrying the image.
 Package imagetext makes the images embedded in a markdown document searchable: each resolvable image is described by a vision LLM and the description is inserted, as plain text, right after the block carrying the image.
Package retrieval implements grounding-driven retrieval orchestration ported from Corpus' MothRAG-derived mechanisms: an evidence evaluator (relevance filtering fused with a grounding (γ) verdict), query reformulation for iterative re-retrieval, and query decomposition.
 Package retrieval implements grounding-driven retrieval orchestration ported from Corpus' MothRAG-derived mechanisms: an evidence evaluator (relevance filtering fused with a grounding (γ) verdict), query reformulation for iterative re-retrieval, and query decomposition.
gorm
memory
Package telemetry wires amoxtli to OpenTelemetry.
 Package telemetry wires amoxtli to OpenTelemetry.
Package vision turns images into searchable text: a vision LLM describes an image, and the resulting title/description/transcription is indexed as ordinary markdown by every backend (full-text, vector, metadata filtering).
 Package vision turns images into searchable text: a vision LLM describes an image, and the resulting title/description/transcription is indexed as ordinary markdown by every backend (full-text, vector, metadata filtering).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.