Documentation
¶
Index ¶
- Variables
- func Cache(model jpf.Model, cache jpf.ModelResponseCache, opts ...CachedModelOpt) jpf.Model
- func CountUsage(model jpf.Model, counter *UsageCounter) jpf.Model
- func LimitConcurrency(model jpf.Model, limiter *semaphore.Weighted) jpf.Model
- func Log(model jpf.Model, logger jpf.ModelLogger) jpf.Model
- func Map(model jpf.Model, f func(jpf.Message) jpf.Message) jpf.Model
- func MultiDispatch(models []jpf.Model) jpf.Model
- func NewRemote(format APIFormat, name string, key string, opts ...APIModelOpt) jpf.Model
- func RateLimit(model jpf.Model, limiter *rate.Limiter) jpf.Model
- func Retry(model jpf.Model, maxRetries int, opts ...RetryOpt) jpf.Model
- func RetryChain(models []jpf.Model) jpf.Model
- func Timeout(model jpf.Model, timeout time.Duration) jpf.Model
- func TransformByPrefix(prefix string) func(string) string
- func WithDelay(delay time.Duration) func(*retryModel)
- func WithSalt(salt string) func(m *cachedModel)
- type APIFormat
- type APIModelOpt
- func WithHeader(key, value string) APIModelOpt
- func WithHeaders(headers map[string]string) APIModelOpt
- func WithMaxOutput(n int) APIModelOpt
- func WithPrediction(pred string) APIModelOpt
- func WithPresencePenalty(p float64) APIModelOpt
- func WithReasoningEffort(re ReasoningEffort) APIModelOpt
- func WithTemperature(temp float64) APIModelOpt
- func WithTopP(tp int) APIModelOpt
- func WithURL(u string) APIModelOpt
- func WithVerbosity(vb Verbosity) APIModelOpt
- type CachedModelOpt
- type ReasoningEffort
- type RetryOpt
- type UsageCounter
- type Verbosity
Constants ¶
This section is empty.
Variables ¶
var ErrNoMultiDispatchModels = errors.New("no models provided to multi dispatch model, it needs at least 1")
Functions ¶
func Cache ¶
func Cache(model jpf.Model, cache jpf.ModelResponseCache, opts ...CachedModelOpt) jpf.Model
Cache wraps a Model with response caching functionality. It stores responses in the provided ModelResponseCache implementation, returning cached results for identical input messages and salts to avoid redundant model calls.
func CountUsage ¶
func CountUsage(model jpf.Model, counter *UsageCounter) jpf.Model
CountUsage wraps a Model with token usage tracking functionality. It aggregates token usage statistics in the provided UsageCounter, which allows monitoring total token consumption across multiple model calls.
func LimitConcurrency ¶
LimitConcurrency wraps a Model with concurrency control. It ensures that only a limited number of concurrent calls can be made to the underlying model, using the provided semaphore to manage access (this can be shared between different model instances to ensure control at any level).
func Log ¶
Log wraps a Model with logging functionality. It logs all interactions with the model using the provided ModelLogger. Each model call is logged with input messages, output messages, usage statistics, and timing information.
func MultiDispatch ¶
Wrap a number of underlying models. Each model will have the same request sent to it at the same time. The fastest model to respond without error will provide the final response. If all models respond with error, the fastest erroring model will be returned. This is useful when model response times are very variable, and you don't mind paying more to increase the chance of a fast response. WARNING: It is impossible to stream through this, because we do not know which response will be returned. Streaming does not work with this model in the chain. WARNING: Becasue we send off multiple requests but only wait for one to come back, the token usage tracking is inaccurate above this model in the chain. The tokens returned by this model are the tokens of the first response. For accurate token tracking, attach the token tracker below this model in the tree.
func Retry ¶
Retry wraps a Model with retry functionality. If the underlying model returns an error, this wrapper will retry the operation up to a configurable number of times with an optional delay between retries.
func RetryChain ¶
RetryChain creates a Model that tries a list of models in order, returning the result from the first one that doesn't fail. If all models fail, it returns a joined error containing all the errors.
func Timeout ¶
Timeout creates a new model that will cause the context of the child to timeout, a specified duration after Respond is called. Caution: It only tells the context to timeout - it will not forecfully stop the child model if it does not respect the context.
func TransformByPrefix ¶
Types ¶
type APIModelOpt ¶
type APIModelOpt func(*apiModelSettings)
func WithHeader ¶
func WithHeader(key, value string) APIModelOpt
func WithHeaders ¶
func WithHeaders(headers map[string]string) APIModelOpt
func WithMaxOutput ¶
func WithMaxOutput(n int) APIModelOpt
func WithPrediction ¶
func WithPrediction(pred string) APIModelOpt
func WithPresencePenalty ¶
func WithPresencePenalty(p float64) APIModelOpt
func WithReasoningEffort ¶
func WithReasoningEffort(re ReasoningEffort) APIModelOpt
func WithTemperature ¶
func WithTemperature(temp float64) APIModelOpt
func WithTopP ¶
func WithTopP(tp int) APIModelOpt
func WithURL ¶
func WithURL(u string) APIModelOpt
func WithVerbosity ¶
func WithVerbosity(vb Verbosity) APIModelOpt
type CachedModelOpt ¶
type CachedModelOpt func(*cachedModel)
type ReasoningEffort ¶
type ReasoningEffort uint8
const ( LowReasoning ReasoningEffort = iota MediumReasoning HighReasoning XHighReasoning )
type UsageCounter ¶
type UsageCounter struct {
// contains filtered or unexported fields
}
Counts up the sum usage. Is completely concurrent-safe.
func NewUsageCounter ¶
func NewUsageCounter() *UsageCounter
NewUsageCounter creates a new UsageCounter with zero initial usage. The counter is safe for concurrent use across multiple goroutines.
func (*UsageCounter) Add ¶
func (u *UsageCounter) Add(usage jpf.Usage)
Add the given usage to the counter.
func (*UsageCounter) Get ¶
func (u *UsageCounter) Get() jpf.Usage
Get the current usage in the counter.
Source Files