D4S.Indexer.Domain 1.0.21

D4S.Indexer

Document indexing library for Azure AI Search: extracts text, generates vector embeddings, and uploads searchable chunks.

Quick start

var indexer = IndexerBuilder.Create("my-index")
    .WithAzureSearch(searchEndpoint, searchKey)
    .WithAzureOpenAI(aoaiEndpoint, aoaiKey, embeddingDeployment, embeddingDimensions)
    .WithLocalFiles("./documents")
    .WithFileMetadataFields()
    .Build();

var result = await indexer.IndexAsync();

See src/Rag/samples/ for working examples (local files, SharePoint, OCR, agentic retrieval).

Architecture

D4S.Indexer.Domain          Entities, abstractions (interfaces)
D4S.Indexer.Application     Orchestration (DocumentIndexerService, DocumentExtractor)
D4S.Indexer.Infrastructure  Azure implementations, builder, processors, sources

Built-in sources: local filesystem, multi-site SharePoint (PnP Core). Built-in processors: PDF, DOCX, XLSX, PPTX, TXT/Markdown.

Indexing modes

• Full (default): all documents fetched from every source; documents missing from the source list are deleted from the index.
• Delta (.WithDeltaMode()): only changed/new/deleted documents are provided; deletion is driven by DocumentMetadata.DeletedDate (set it and pass null for GetContentAsync). No implicit cleanup.

Both modes compare LastModifiedDate against the index to skip unchanged documents.

Builder options

IndexerBuilder.Create("index-name")
    // Required
    .WithAzureSearch(endpoint, apiKey)
    .WithAzureOpenAI(endpoint, apiKey, deployment, dimensions)
    // Sources (at least one)
    .WithLocalFiles("./docs")                                 // or: opts => { opts.Path = …; opts.FileExtensions = […]; }
    .WithSharePointMultiSite(spOptions, contextFactory)
    .WithCustomDocumentSource<T>(serviceProvider, serviceKey)
    // Optional
    .WithDeltaMode()
    .WithFileMetadataFields()
    .WithChunkSize(maxSize: 1000, overlap: 200)
    .WithBatchSize(50)
    .WithKeywordExtraction(gptDeployment, maxKeywords: 10)
    .WithAzureDocumentIntelligence(endpoint, apiKey)          // OCR
    .WithCustomDocumentProcessor<T>(serviceProvider, serviceKey)
    .ContinueOnError(true)
    .Filter(meta => meta.Extension == ".pdf")
    .ConfigureMetadata(meta => meta with { CustomFields = … })
    .AddCustomField("Status", CustomFieldType.String, filterable: true)
    .AddIndexFieldsFromAttributes<MyModel>()
    .OnProgress(p => Console.WriteLine(p.Phase))
    .WithLogging()
    .Build();