Breaking Changes

This is a major architectural shift that eliminates the external plugged_in_v3_server (FastAPI + Milvus) dependency for RAG. Document processing and vector search now run in-process using the embedded zvec vector engine backed by RocksDB with HNSW indexes.

Migration Notes

• Docker image changed: docker-compose.yml now uses pgvector/pgvector:pg18 instead of stock postgres:18-alpine. Back up your database before upgrading.
• New env vars: ZVEC_DATA_PATH, EMBEDDING_MODEL, RAG_SEARCH_TOP_K, RAG_CACHE_TTL_MS, ZVEC_ALLOW_EXTERNAL_PATH (see .env.example)
• Removed env vars: RAG_API_URL is no longer used
• Database migration: 3 new migrations (0091–0093) add the document_chunks table. Run pnpm db:migrate after upgrading.
• New dependency: @zvec/zvec requires native build tools (cmake, g++) — included in the updated Dockerfile.

What's New

Embedded RAG Vector Engine (PR #134)

• In-process vector search via zvec (RocksDB + HNSW) — eliminates external HTTP dependency and SSRF attack surface
• Shared vector infrastructure (lib/vectors/) with domain-based collections for RAG, Memory, and future CBP systems
• Server-side PDF text extraction using unpdf — no external service needed
• Smart text chunking with configurable overlap and separators (lib/rag/chunking.ts)
• Shared embedding service using @langchain/openai with model-to-dimension validation
• Re-index menu option in the library UI for recovering from vector corruption
• Relevance-ordered search results with query term highlighting in AI search answers

Security & Robustness

• Filter injection prevention: Field-name allowlist + per-type value validation (UUID regex for UUID fields, alphanumeric regex for string fields) in buildFilter()
• Path traversal protection: ZVEC_DATA_PATH outside app root now blocked unless ZVEC_ALLOW_EXTERNAL_PATH=true
• Corruption recovery: Unhealthy indexes detected via completeness threshold; collections backed up to .bak before recreation with full stats logging
• Idempotent document processing: processDocument cleans existing chunks before re-inserting, preventing orphaned rows and unique constraint violations
• Embedding dimension validation: Model-to-dimension map throws on misconfigured EMBEDDING_MODEL
• HMR safety: zvec collection handles persisted on globalThis to survive Next.js hot module reloads without RocksDB lock conflicts

Removed (Dead Code Cleanup)

• UploadProgressContext, UploadProgressToast, UploadProgress component (~750 lines) — upload polling infrastructure no longer needed with synchronous processing
• /api/upload-status/[uploadId] API route
• getUploadStatus server action stub
• UploadProgress and UploadProgressState types from types/library.ts

Test Coverage

• 28 RAG tests: chunking (5), service (8), schema (2), buildFilter injection prevention (13)
• Vector service, embedding service, and types tests

Files Changed

New files: lib/vectors/ (5 files), lib/rag/ (4 files), tests/rag/ (4 files), tests/vectors/ (3 files), 3 DB migrations
Deleted files: UploadProgress.tsx, upload-progress-toast.tsx, UploadProgressContext.tsx, upload-status API route, types/library.ts (upload types)
Modified: lib/rag-service.ts (full rewrite), app/actions/library.ts, docker-compose.yml, Dockerfile, next.config.ts, 6 locale files

Full Changelog

v2.21.0...v3.0.0