storage

package
v0.0.0-...-5224f1c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package storage provides interfaces and implementations for persisting agentops session data, indexes, and provenance records.

Index

Constants

View Source 
const (
	// DefaultBaseDir is the default storage directory.
	DefaultBaseDir = ".agents/ao"

	// SessionsDir holds session markdown/jsonl files.
	SessionsDir = "sessions"

	// IndexDir holds the session index.
	IndexDir = "index"

	// ProvenanceDir holds provenance records.
	ProvenanceDir = "provenance"

	// IndexFile is the name of the main index file.
	IndexFile = "sessions.jsonl"

	// ProvenanceFile is the name of the provenance graph.
	ProvenanceFile = "graph.jsonl"

	// SlugMaxLength is the maximum length for URL-safe slugs.
	SlugMaxLength = 50

	// SlugMinWordBoundary is the minimum length before trimming at word boundary.
	SlugMinWordBoundary = 30
)
View Source 
const (
	// SearchIndexFileName is the file name of the search index.
	SearchIndexFileName = "search-index.jsonl"
	// SearchIndexDir is the directory for index files relative to the workspace.
	SearchIndexDir = ".agents/ao/index"
)

Variables

View Source 
var (
	// ErrSessionIDRequired is returned when a session write is attempted without an ID.
	ErrSessionIDRequired = errors.New("session ID is required")

	// ErrEmptySessionFile is returned when a session file has no content.
	ErrEmptySessionFile = errors.New("empty session file")

	// ErrLineTooLong is returned by the JSONL scan helpers (ScanJSONL,
	// ScanJSONLFile) when a single line exceeds the package buffer cap
	// (scanJSONLMaxLine, 8MB). It signals a hard failure — never a silent
	// truncation — and is wrapped with the offending 1-based line number so the
	// caller can locate the bad record. Match it with errors.Is.
	ErrLineTooLong = errors.New("jsonl line exceeds maximum length")
)

Sentinel errors for the storage package. Using sentinels instead of ad-hoc fmt.Errorf allows callers to match with errors.Is for reliable error handling.

View Source 
var ArtifactSubdirs = []string{"learnings", "patterns", "research", "retros", "candidates"}

ArtifactSubdirs lists the subdirectories under .agents/ that contain indexable artifacts.

Functions

func AccumulateEntryStats 

func AccumulateEntryStats(stats *SearchIndexStats, entry SearchIndexEntry, totalUtility *float64, utilityCount *int)

AccumulateEntryStats updates running stats counters from a single index entry.

func AppendCategoryKeywords 

func AppendCategoryKeywords(keywords []string, category string, tags []string) []string

AppendCategoryKeywords appends category and tag values as lowercase keywords.

func AppendJSONL 

func AppendJSONL(path string, v any) error

AppendJSONL appends v, marshaled to a single JSON line, to the file at path. The write is durable and serialized: the file is created if missing, an advisory lock is taken so concurrent appenders do not interleave, the marshaled line plus a trailing newline is written at end-of-file, and the file is fsynced before returning. The parent directory is created if missing. This is the canonical writer for the package's JSONL logs (session index, provenance graph) and the drop-in for a hand-rolled open+seek-end+write+sync append.

func AppendToSearchIndex 

func AppendToSearchIndex(baseDir string, entry *SearchIndexEntry) error

AppendToSearchIndex adds an entry to the index file.

func ArtifactTypeFromPath 

func ArtifactTypeFromPath(path string) string

ArtifactTypeFromPath determines the artifact type based on the file path.

func AtomicWriteFile 

func AtomicWriteFile(path string, data []byte, perm os.FileMode) error

AtomicWriteFile is the canonical atomic file writer for the repo. It writes data to path durably and atomically: a same-directory temp file is written, fsynced, chmod'd to perm, closed, and renamed over the destination, so a concurrent reader sees either the previous content or the complete new content — never a partial write — and the bytes are on disk before the path becomes observable. The parent directory is created with 0755 if missing.

The intermediate temp file is created via os.CreateTemp (mode 0o600 on Unix), so a wider requested perm is never observable on the filesystem until the rename completes. The temp file is removed on every error path.

This is the single implementation behind the package-level atomic-write helpers elsewhere in the CLI (internal/types/quest, internal/llmwiki); new call sites should prefer this directly. It is the proven shape the fleet-lease fsync contract depends on (see internal/types/quest fault tests).

func ComputeSearchScore 

func ComputeSearchScore(entry SearchIndexEntry, queryTerms []string) float64

ComputeSearchScore calculates relevance score for a query.

func CreateSearchSnippet 

func CreateSearchSnippet(content, query string, maxLen int) string

CreateSearchSnippet creates a context snippet around query matches.

func ExtractCategoryAndTags 

func ExtractCategoryAndTags(content string) (category string, tags []string)

ExtractCategoryAndTags derives category/tags from frontmatter or markdown metadata.

func ExtractFrontmatterMeta 

func ExtractFrontmatterMeta(lines []string) (category string, tags []string)

ExtractFrontmatterMeta parses category/tags from YAML frontmatter lines.

func ExtractKeywords 

func ExtractKeywords(content string) []string

ExtractKeywords extracts keywords from content.

func ExtractMarkdownMeta 

func ExtractMarkdownMeta(lines []string) (category string, tags []string)

ExtractMarkdownMeta parses category/tags from bold-key markdown lines.

func ExtractTitle 

func ExtractTitle(content string) string

ExtractTitle gets the title from markdown content.

func FsyncDir 

func FsyncDir(dir string) error

FsyncDir flushes a directory's metadata to disk by opening it and calling fsync, making a prior rename into that directory durable across power loss. It is the single dir-fsync implementation for the CLI; the atomic writers that do their own rename (storage.FileStorage, internal/pool, internal/llmwiki, internal/evalsubstrate) call this after the rename completes.

A directory fsync that reports the operation as unsupported is treated as a no-op: some filesystems (notably macOS APFS) reject fsync on a directory handle with EINVAL/ENOTSUP, and the rename is already durable there, so that error is not one the caller can act on. A genuine I/O error (e.g. EIO) is returned rather than swallowed, so a real durability failure is not hidden. An open failure (e.g. the directory does not exist) is also returned so callers surface a genuinely missing path.

func ParseBracketedList 

func ParseBracketedList(s string) []string

ParseBracketedList parses "[a, b, c]" into a trimmed string slice.

func ParseMemRLMetadata 

func ParseMemRLMetadata(content string) (utility float64, maturity string)

ParseMemRLMetadata extracts utility and maturity from content.

func ScanJSONL 

func ScanJSONL(r io.Reader, fn func(line []byte)) error

ScanJSONL reads r line by line and calls fn for each line's bytes, applying the package JSONL buffer policy: lines up to scanJSONLMaxLine (8MB) are read; a longer line stops iteration and returns an error wrapping ErrLineTooLong that names the 1-based line number. The line slice passed to fn is only valid for the duration of the call — copy it to retain it. A read error from r is returned as-is. This never silently truncates a line.

func ScanJSONLFile 

func ScanJSONLFile(path string, fn func(line []byte)) (err error)

ScanJSONLFile opens the JSONL file at path and calls fn for each line, applying the package JSONL buffer policy (see ScanJSONL). A missing file is not an error (fn is simply never called); any other open error, read error, or an oversized line (ErrLineTooLong, naming the line number) is returned. The file is always closed; a close error is surfaced only if the scan itself succeeded.

func SplitCSV 

func SplitCSV(s string) []string

SplitCSV splits a comma-separated string and returns non-empty trimmed values.

func WalkIndexableFiles 

func WalkIndexableFiles(dir string) ([]string, error)

WalkIndexableFiles returns all .md and .jsonl files under dir. A walk error (for example, a missing root or an unreadable subdirectory) is surfaced to the caller rather than swallowed, so an empty result unambiguously means "no indexable files" instead of masking a failed traversal.

Types

type FileStorage 

type FileStorage struct {
	// BaseDir is the root directory (e.g., .agents/ao).
	BaseDir string

	// Formatters are the output formats to use.
	Formatters []Formatter
	// contains filtered or unexported fields
}

FileStorage implements Storage using the local filesystem.

func NewFileStorage 

func NewFileStorage(opts ...FileStorageOption) *FileStorage

NewFileStorage creates a new file-based storage.

func (*FileStorage) Close 

func (fs *FileStorage) Close() error

Close releases any resources.

func (*FileStorage) GetBaseDir 

func (fs *FileStorage) GetBaseDir() string

GetBaseDir returns the configured base directory.

func (*FileStorage) GetIndexPath 

func (fs *FileStorage) GetIndexPath() string

GetIndexPath returns the full path to the index file.

func (*FileStorage) GetProvenancePath 

func (fs *FileStorage) GetProvenancePath() string

GetProvenancePath returns the full path to the provenance file.

func (*FileStorage) GetSessionsDir 

func (fs *FileStorage) GetSessionsDir() string

GetSessionsDir returns the full path to the sessions directory.

func (*FileStorage) Init 

func (fs *FileStorage) Init() error

Init creates the required directory structure.

func (*FileStorage) ListSessions 

func (fs *FileStorage) ListSessions() ([]IndexEntry, error)

ListSessions returns all session index entries.

func (*FileStorage) QueryProvenance 

func (fs *FileStorage) QueryProvenance(artifactPath string) ([]ProvenanceRecord, error)

QueryProvenance finds provenance records for an artifact.

func (*FileStorage) ReadSession 

func (fs *FileStorage) ReadSession(sessionID string) (*Session, error)

ReadSession retrieves a session by ID.

func (*FileStorage) WriteIndex 

func (fs *FileStorage) WriteIndex(entry *IndexEntry) error

WriteIndex appends an entry to the session index.

func (*FileStorage) WriteProvenance 

func (fs *FileStorage) WriteProvenance(record *ProvenanceRecord) error

WriteProvenance records provenance information.

func (*FileStorage) WriteSession 

func (fs *FileStorage) WriteSession(session *Session) (string, error)

WriteSession writes a session to storage using all configured formatters. Returns the path to the primary session file.

type FileStorageOption 

type FileStorageOption func(*FileStorage)

FileStorageOption configures a FileStorage instance.

func WithBaseDir 

func WithBaseDir(dir string) FileStorageOption

WithBaseDir sets the base directory.

func WithFormatters 

func WithFormatters(formatters ...Formatter) FileStorageOption

WithFormatters sets the output formatters.

type Formatter 

type Formatter interface {
	// Format writes the session to the given writer.
	Format(w io.Writer, session *Session) error

	// Extension returns the file extension for this format.
	Extension() string
}

Formatter transforms sessions into specific output formats.

type IndexEntry 

type IndexEntry struct {
	// SessionID links to the full session.
	SessionID string `json:"session_id"`

	// Date for quick filtering.
	Date time.Time `json:"date"`

	// SessionPath is the path to the session file.
	SessionPath string `json:"session_path"`

	// Summary for search/preview.
	Summary string `json:"summary,omitempty"`

	// Tags for categorization.
	Tags []string `json:"tags,omitempty"`
}

IndexEntry represents a single entry in the session index.

type ProvenanceRecord 

type ProvenanceRecord struct {
	// ID is the unique record identifier.
	ID string `json:"id"`

	// ArtifactPath is the file that was produced.
	ArtifactPath string `json:"artifact_path"`

	// WorkspacePath is the workspace root that owns the artifact.
	WorkspacePath string `json:"workspace_path,omitempty"`

	// ArtifactType classifies the output (session, index, etc).
	ArtifactType string `json:"artifact_type"`

	// SourcePath is the input file.
	SourcePath string `json:"source_path"`

	// SourceType classifies the input (transcript).
	SourceType string `json:"source_type"`

	// SessionID links to the conversation.
	SessionID string `json:"session_id,omitempty"`

	// CreatedAt is when the record was created.
	CreatedAt time.Time `json:"created_at"`

	// Metadata holds additional provenance data.
	Metadata map[string]any `json:"metadata,omitempty"`
}

ProvenanceRecord tracks the origin of an artifact.

type SearchIndexEntry 

type SearchIndexEntry struct {
	Path       string    `json:"path"`
	ID         string    `json:"id"`
	Type       string    `json:"type"`
	Title      string    `json:"title"`
	Content    string    `json:"content"`
	Keywords   []string  `json:"keywords,omitempty"`
	Category   string    `json:"category,omitempty"`
	Tags       []string  `json:"tags,omitempty"`
	Utility    float64   `json:"utility,omitempty"`
	Maturity   string    `json:"maturity,omitempty"`
	IndexedAt  time.Time `json:"indexed_at"`
	ModifiedAt time.Time `json:"modified_at"`
}

SearchIndexEntry represents a single entry in the artifact search index.

func CreateSearchIndexEntry 

func CreateSearchIndexEntry(path string, categorize bool) (*SearchIndexEntry, error)

CreateSearchIndexEntry creates an index entry from a file.

type SearchIndexStats 

type SearchIndexStats struct {
	TotalEntries int            `json:"total_entries"`
	ByType       map[string]int `json:"by_type"`
	MeanUtility  float64        `json:"mean_utility"`
	OldestEntry  time.Time      `json:"oldest_entry"`
	NewestEntry  time.Time      `json:"newest_entry"`
	IndexPath    string         `json:"index_path"`
}

SearchIndexStats holds aggregate stats about the index.

func ComputeSearchIndexStats 

func ComputeSearchIndexStats(baseDir string) (*SearchIndexStats, error)

ComputeSearchIndexStats calculates index statistics.

type SearchResult 

type SearchResult struct {
	Entry   SearchIndexEntry `json:"entry"`
	Score   float64          `json:"score"`
	Snippet string           `json:"snippet,omitempty"`
}

SearchResult represents a single match in the search index.

func SearchIndex 

func SearchIndex(baseDir, query string, limit int) ([]SearchResult, error)

SearchIndex searches the index for matching entries.

type Session 

type Session struct {
	// ID is the unique session identifier (from transcript).
	ID string `json:"session_id"`

	// Date is when the session occurred.
	Date time.Time `json:"date"`

	// Summary is a brief description of the session.
	Summary string `json:"summary,omitempty"`

	// Decisions lists architectural choices made.
	Decisions []string `json:"decisions,omitempty"`

	// Knowledge contains insights and learnings.
	Knowledge []string `json:"knowledge,omitempty"`

	// FilesChanged lists files modified in the session.
	FilesChanged []string `json:"files_changed,omitempty"`

	// Issues lists issue IDs referenced or created.
	Issues []string `json:"issues,omitempty"`

	// ToolCalls counts tool invocations by type.
	ToolCalls map[string]int `json:"tool_calls,omitempty"`

	// Tokens tracks token usage.
	Tokens TokenUsage `json:"tokens,omitempty"`

	// TranscriptPath is the source transcript file.
	TranscriptPath string `json:"transcript_path,omitempty"`

	// SessionType is the detected session type (career, research, debug, implement, brainstorm, general).
	SessionType string `json:"session_type,omitempty"`
}

Session represents extracted knowledge from a transcript session.

type Storage 

type Storage interface {
	// WriteSession writes a session to storage.
	// Returns the path where the session was written.
	WriteSession(session *Session) (string, error)

	// WriteIndex appends an entry to the session index.
	WriteIndex(entry *IndexEntry) error

	// WriteProvenance records provenance information.
	WriteProvenance(record *ProvenanceRecord) error

	// ReadSession retrieves a session by ID.
	ReadSession(sessionID string) (*Session, error)

	// ListSessions returns all session index entries.
	ListSessions() ([]IndexEntry, error)

	// QueryProvenance finds provenance records for an artifact.
	QueryProvenance(artifactPath string) ([]ProvenanceRecord, error)

	// Init creates the required directory structure.
	Init() error

	// Close releases any resources.
	Close() error
}

Storage is the interface for persisting agentops data.

type TokenUsage 

type TokenUsage struct {
	Input     int  `json:"input"`
	Output    int  `json:"output"`
	Total     int  `json:"total"`
	Estimated bool `json:"estimated,omitempty"`
}

TokenUsage tracks token consumption.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.