Documentation
¶
Index ¶
- Constants
- func CmpDocumentLengthEntryAndDocumentIndex(e storage.DocumentLengthEntry, t uint32) int
- func CmpTokenFrequencyEntryAndDocumentIndex(e storage.TokenFrequencyEntry, t uint32) int
- func InverseDocumentFrequency(docCount, tokenDocFreq uint64) float32
- func NormalizedTF(tokenFreq, documentLength uint64, ...) (normTf float32)
- func ScoreTermBM25(docCount, tokenDocFreq, tokenFreq, documentLength uint64, ...) (score float32)
- type Clause
- type ClauseEntry
- type ClauseState
- type HandleClauseCondFunc
- type HandleClauseFunc
- type Keyword
- type QueryContext
- type Range
- type RangeCaptureMode
- type Score
- type Scoring
- type Searcher
- func (s *Searcher) BM25Score(ctx *QueryContext, q *SimpleQuery)
- func (s *Searcher) FieldScore(ctx *QueryContext, fieldHash uint64)
- func (s *Searcher) FilterDocuments(ctx *QueryContext, q *SimpleQuery)
- func (s *Searcher) Iter(c *Clause, handle HandleClauseFunc)
- func (s *Searcher) IterCond(c *Clause, handle HandleClauseCondFunc)
- func (s *Searcher) ResolveScores(ctx *QueryContext, doCopy bool) (scores []Score)
- type SimpleQuery
Constants ¶
const ( DefaultSaturation = 1.2 DefaultLengthPenalty = 0.75 )
const ManyIteratorBatchSize = 8
const MinimumBM25Score = 0
Variables ¶
This section is empty.
Functions ¶
func CmpDocumentLengthEntryAndDocumentIndex ¶ added in v1.7.7
func CmpDocumentLengthEntryAndDocumentIndex(e storage.DocumentLengthEntry, t uint32) int
func CmpTokenFrequencyEntryAndDocumentIndex ¶ added in v1.7.7
func CmpTokenFrequencyEntryAndDocumentIndex(e storage.TokenFrequencyEntry, t uint32) int
func InverseDocumentFrequency ¶
IDF returns the Inverse Document Frequency for a single term. It answers: "how surprising is it to see this term in a document?" Document Count is the total number of documents indexed Token Document Frequency is how many documents contains the supplied token at least once
func NormalizedTF ¶
func NormalizedTF(tokenFreq, documentLength uint64, avgDocLength, saturation, lengthPenalty float32) (normTf float32)
NormalizedTF returns the saturated, length-normalized term frequency for one term in one document's field. tokenFreq - raw count: how many times the term appears in this doc's field documentLength - document length: number of tokens in this doc's field avgDocLength - average document length across all docs for this field saturation - saturation: how fast extra occurrences stop mattering (typically 1.2) lengthPenalty - length penalty: how hard to punish long documents (typically 0.75)
func ScoreTermBM25 ¶
Types ¶
type Clause ¶
type Clause struct {
Keywords []*Keyword
FieldKeywords []*ClauseEntry[Keyword]
FieldRanges []*ClauseEntry[Range]
}
func (*Clause) FieldKeyword ¶
func (*Clause) FieldRange ¶
func (c *Clause) FieldRange(field uint64, lo, hi []byte, mode RangeCaptureMode, boost float32)
type ClauseEntry ¶
type ClauseState ¶
type HandleClauseCondFunc ¶ added in v1.7.5
type HandleClauseCondFunc func(state *ClauseState) (next bool)
type HandleClauseFunc ¶
type HandleClauseFunc func(state *ClauseState)
type QueryContext ¶
Query context intended to be cached and reused by caller on each search
type Range ¶
type Range struct {
CaptureMode RangeCaptureMode
Boost float32
Low, High []byte
}
type RangeCaptureMode ¶
type RangeCaptureMode int
const ( RangeCaptureModeNone RangeCaptureMode = iota RangeCaptureModeLeft RangeCaptureModeRight RangeCaptureModeBoth )
type Scoring ¶ added in v1.7.7
type Scoring struct {
Scores []Score
}
type Searcher ¶
type Searcher struct {
Storage *storage.Storage
BM25Saturation float32
BM25LengthPenalty float32
// Maximum amount of entries challenged against levenshtein fuzz algorithm
LevenshteinM int
LevenshteinMaxK int
}
func (*Searcher) BM25Score ¶
func (s *Searcher) BM25Score(ctx *QueryContext, q *SimpleQuery)
BM25Score fills ctx.Scores with the BM25 score of every document in ctx.Bitmap. Musts are scored first, then Shoulds, matching the original summation order so the floating point result is bit-for-bit identical.
The hot path materializes the bitmap once into an ascending candidates slice and accumulates into a dense []float64 aligned to it, rather than writing the score map on every term-document pair. The bitmap is immutable during scoring, so position i in candidates is the same document across every term walk. That single ToArray call replaces one iterator allocation and one heap batch buffer per term walk, turns every walk into a plain slice range, lets every per-term write land on a slice instead of a hashed map, and collapses the map writes into one final pass.
func (*Searcher) FieldScore ¶
func (s *Searcher) FieldScore(ctx *QueryContext, fieldHash uint64)
func (*Searcher) FilterDocuments ¶
func (s *Searcher) FilterDocuments(ctx *QueryContext, q *SimpleQuery)
Filter the documents id index into the destination bitmap the idea is to filter first the score results based on conditions is caller's responsability to clear dst bitmap
func (*Searcher) Iter ¶
func (s *Searcher) Iter(c *Clause, handle HandleClauseFunc)
Iterates about the matching fields + Tokens ignoring those entries that doesn't contain any token
func (*Searcher) IterCond ¶ added in v1.7.5
func (s *Searcher) IterCond(c *Clause, handle HandleClauseCondFunc)
Same as Iter but also call handle when there is no tokens available for the clause. The return value of handle is used to early stop the query
func (*Searcher) ResolveScores ¶
func (s *Searcher) ResolveScores(ctx *QueryContext, doCopy bool) (scores []Score)
Once a filtering and scoring are done, next step of a searching algorithm Resolves the ctx to an actual idx slice
Source Files