udf

package
v0.0.0-...-18cb0a0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package udf provides the metadata model for Iceberg SQL UDFs as specified by the Iceberg UDF spec. https://iceberg.apache.org/udf-spec/

Index

Constants

View Source 
const (
	SupportedUDFFormatVersion = 1
)

Variables

View Source 
var (
	ErrInvalidUDFMetadata              = errors.New("invalid UDF metadata")
	ErrInvalidUDFMetadataFormatVersion = errors.New("invalid or missing format-version in UDF metadata")
)
View Source 
var ErrInvalidUDFType = errors.New("invalid UDF type")

ErrInvalidUDFType is returned when a UDF parameter or return type does not conform to the UDF spec's Types section.

Functions

func CanonicalDefinitionID 

func CanonicalDefinitionID(params []Parameter) (string, error)

CanonicalDefinitionID derives the definition-id for a parameter list: the canonical string forms of the parameter types joined by commas with no spaces, e.g. "int,list<int>,struct<id:int,name:string>". A definition with no parameters has the empty string as its definition-id.

Types

type Definition 

type Definition struct {
	// DefinitionID is the canonical string derived from the parameter
	// types (see CanonicalDefinitionID).
	DefinitionID string `json:"definition-id"`
	// SpecificName is an optional user-assignable name for this
	// definition, unique among all definitions of the function.
	SpecificName string `json:"specific-name,omitempty"`
	// Parameters is the ordered list of function parameters. Invocation
	// order must match this list.
	Parameters []Parameter `json:"parameters"`
	// ReturnType is the declared return type. All versions must produce
	// values of this type.
	ReturnType Type `json:"return-type"`
	// ReturnNullable hints whether the return value is nullable.
	// Defaults to true when absent.
	ReturnNullable *bool `json:"return-nullable,omitempty"`
	// Versions lists the versioned implementations of this definition.
	Versions []*DefinitionVersion `json:"versions"`
	// CurrentVersionID identifies the current version of this definition.
	CurrentVersionID int `json:"current-version-id"`
	// FunctionType is "udf" for scalar functions or "udtf" for table
	// functions. For "udtf", ReturnType must be a struct describing the
	// output schema.
	FunctionType FunctionType `json:"function-type"`
	// Doc is an optional documentation string.
	Doc string `json:"doc,omitempty"`
}

Definition represents one function signature (e.g. add_one(int) vs add_one(float)). A definition is uniquely identified by its signature: the ordered list of parameter types, canonicalized as its DefinitionID.

func NewDefinition 

func NewDefinition(functionType FunctionType, params []Parameter, returnType Type, opts ...DefinitionOpt) (*Definition, error)

NewDefinition creates a definition for the given signature, deriving its definition-id from the parameter types. The definition starts with no versions; add them with MetadataBuilder.AddDefinitionVersion.

func (*Definition) Clone 

func (d *Definition) Clone() *Definition

func (*Definition) CurrentVersion 

func (d *Definition) CurrentVersion() *DefinitionVersion

CurrentVersion returns the definition version identified by CurrentVersionID, or nil if no such version exists.

func (*Definition) Equals 

func (d *Definition) Equals(other *Definition) bool

Equals compares two definitions for semantic equality: absent optional fields compare equal to their spec defaults.

func (*Definition) ReturnsNullable 

func (d *Definition) ReturnsNullable() bool

ReturnsNullable returns the effective return-nullable hint, applying the spec default (true) when the field is absent.

func (*Definition) UnmarshalJSON 

func (d *Definition) UnmarshalJSON(b []byte) error

func (*Definition) Version 

func (d *Definition) Version(versionID int) *DefinitionVersion

Version returns the definition version with the given id, or nil if no such version exists.

type DefinitionLogEntry 

type DefinitionLogEntry struct {
	// TimestampMS is when the function was updated to use these
	// definition versions (ms from epoch).
	TimestampMS int64 `json:"timestamp-ms"`
	// DefinitionVersions maps each definition to its selected version at
	// this time.
	DefinitionVersions []DefinitionVersionRef `json:"definition-versions"`
}

DefinitionLogEntry records the selected version of every definition at the time of a change, enabling rollback without external state.

func (*DefinitionLogEntry) UnmarshalJSON 

func (e *DefinitionLogEntry) UnmarshalJSON(b []byte) error

type DefinitionOpt 

type DefinitionOpt func(*Definition)

DefinitionOpt configures optional fields of a new Definition.

func WithDefinitionDoc 

func WithDefinitionDoc(doc string) DefinitionOpt

WithDefinitionDoc sets the definition's documentation string.

func WithReturnNullable 

func WithReturnNullable(nullable bool) DefinitionOpt

WithReturnNullable sets the definition's return-nullable hint.

func WithSpecificName 

func WithSpecificName(name string) DefinitionOpt

WithSpecificName sets the definition's optional specific name, a stable user-assignable handle that must be unique among all definitions.

type DefinitionVersion 

type DefinitionVersion struct {
	// VersionID is a monotonically increasing identifier of the
	// definition version.
	VersionID int `json:"version-id"`
	// Representations lists the UDF implementations of this version.
	Representations []Representation `json:"representations"`
	// Deterministic indicates whether the function is deterministic.
	// Defaults to false.
	Deterministic bool `json:"deterministic,omitempty"`
	// OnNullInput defines how the UDF behaves when any input parameter
	// is NULL. Defaults to OnNullInputCall when empty.
	OnNullInput OnNullInput `json:"on-null-input,omitempty"`
	// TimestampMS is the creation timestamp of this version (ms from epoch).
	TimestampMS int64 `json:"timestamp-ms"`
}

DefinitionVersion is a specific implementation of a definition at a given point in time. Versions are immutable: changes to a definition introduce a new version.

func NewDefinitionVersion 

func NewDefinitionVersion(representations []Representation, opts ...DefinitionVersionOpt) (*DefinitionVersion, error)

NewDefinitionVersion creates a definition version from the given representations. The version id is assigned when the version is added to a definition via MetadataBuilder.AddDefinitionVersion.

NewDefinitionVersion seeds TimestampMS to time.Now().UnixMilli(); use WithTimestampMS to override.

func (*DefinitionVersion) Clone 

func (v *DefinitionVersion) Clone() *DefinitionVersion

func (*DefinitionVersion) Equals 

func (v *DefinitionVersion) Equals(other *DefinitionVersion) bool

Equals compares two versions for semantic equality: the default on-null-input behavior compares equal to an explicit "call".

func (*DefinitionVersion) NullInputBehavior 

func (v *DefinitionVersion) NullInputBehavior() OnNullInput

NullInputBehavior returns the effective on-null-input behavior, applying the spec default (OnNullInputCall) when the field is absent.

func (*DefinitionVersion) UnmarshalJSON 

func (v *DefinitionVersion) UnmarshalJSON(b []byte) error

type DefinitionVersionOpt 

type DefinitionVersionOpt func(*DefinitionVersion)

DefinitionVersionOpt configures optional fields of a new DefinitionVersion.

func WithDeterministic 

func WithDeterministic(deterministic bool) DefinitionVersionOpt

WithDeterministic marks whether the function version is deterministic.

func WithOnNullInput 

func WithOnNullInput(behavior OnNullInput) DefinitionVersionOpt

WithOnNullInput sets the version's behavior for NULL input parameters.

func WithTimestampMS 

func WithTimestampMS(timestampMS int64) DefinitionVersionOpt

WithTimestampMS overrides the version's creation timestamp.

type DefinitionVersionRef 

type DefinitionVersionRef struct {
	// DefinitionID may be empty: a zero-parameter definition has the
	// empty string as its definition-id.
	DefinitionID string `json:"definition-id"`
	VersionID    int    `json:"version-id"`
}

DefinitionVersionRef maps a definition to a selected version.

func (*DefinitionVersionRef) UnmarshalJSON 

func (r *DefinitionVersionRef) UnmarshalJSON(b []byte) error

type FunctionType 

type FunctionType string

FunctionType indicates whether a definition is a scalar function (udf) or a table function (udtf). 

const (
	FunctionTypeUDF  FunctionType = "udf"
	FunctionTypeUDTF FunctionType = "udtf"
)

type ListType 

type ListType struct {
	Element Type
}

ListType is a UDF list type carrying only its element type.

func (ListType) Equals 

func (l ListType) Equals(other Type) bool

func (ListType) MarshalJSON 

func (l ListType) MarshalJSON() ([]byte, error)

func (ListType) String 

func (l ListType) String() string

type MapType 

type MapType struct {
	Key   Type
	Value Type
}

MapType is a UDF map type carrying only its key and value types.

func (MapType) Equals 

func (m MapType) Equals(other Type) bool

func (MapType) MarshalJSON 

func (m MapType) MarshalJSON() ([]byte, error)

func (MapType) String 

func (m MapType) String() string

type Metadata 

type Metadata interface {
	// FormatVersion returns the UDF spec format version (1).
	FormatVersion() int
	// FunctionUUID returns the UUID identifying this UDF, generated once
	// at creation.
	FunctionUUID() uuid.UUID
	// Location returns the function's base location, used to create
	// metadata file locations. It may be empty.
	Location() string
	// Definitions returns the function's definitions, one per signature.
	Definitions() []*Definition
	// DefinitionByID returns the definition with the given definition-id.
	DefinitionByID(definitionID string) (*Definition, bool)
	// DefinitionLog returns the history of definition version selections.
	DefinitionLog() []DefinitionLogEntry
	// Properties returns the function's properties. Entries are treated
	// as hints, not strict rules.
	Properties() iceberg.Properties
	// Secure reports whether this is a secure function whose sensitive
	// information engines should not leak to end users.
	Secure() bool
	// Doc returns the function's documentation string.
	Doc() string

	Equals(Metadata) bool
}

Metadata is the self-contained metadata of an Iceberg SQL UDF as specified by the UDF spec. Metadata files are immutable: any change creates a new file, and catalogs atomically swap the file linked to a catalog identifier. UDF names are not part of the metadata; mapping names to metadata locations is the catalog's responsibility.

func ParseMetadata 

func ParseMetadata(r io.Reader) (Metadata, error)

ParseMetadata parses UDF metadata JSON provided by the passed in reader, returning an error if one is encountered.

func ParseMetadataBytes 

func ParseMetadataBytes(b []byte) (Metadata, error)

ParseMetadataBytes is like ParseMetadataString but for a byte slice.

func ParseMetadataString 

func ParseMetadataString(s string) (Metadata, error)

ParseMetadataString is like ParseMetadata, but for a string rather than an io.Reader.

type MetadataBuilder 

type MetadataBuilder struct {
	// contains filtered or unexported fields
}

MetadataBuilder builds UDF metadata, either from scratch or on top of an existing metadata instance. Because metadata files are immutable, every change produces a complete new metadata object; when any definition's selected version changes, Build appends a definition-log entry capturing the selected version of every definition.

func MetadataBuilderFromBase 

func MetadataBuilderFromBase(base Metadata) (*MetadataBuilder, error)

MetadataBuilderFromBase creates a builder seeded with the state of an existing metadata instance.

func NewMetadataBuilder 

func NewMetadataBuilder() (*MetadataBuilder, error)

NewMetadataBuilder creates a builder for a new UDF's metadata.

func (*MetadataBuilder) AddDefinition 

func (b *MetadataBuilder) AddDefinition(def *Definition) *MetadataBuilder

AddDefinition adds a definition for a new signature. The definition's definition-id is derived from its parameters; if the definition already carries an id it must match the canonical form. Adding a definition whose signature or specific name already exists fails: there can be only one definition for a given signature.

func (*MetadataBuilder) AddDefinitionVersion 

func (b *MetadataBuilder) AddDefinitionVersion(definitionID string, version *DefinitionVersion) *MetadataBuilder

AddDefinitionVersion adds a version to the definition with the given definition-id and selects it as the definition's current version. A version id of 0 is assigned the next id for the definition; an explicit version id must not already exist. Use SetCurrentVersion to roll back to an earlier version.

func (*MetadataBuilder) Build 

func (b *MetadataBuilder) Build() (Metadata, error)

Build validates and returns the metadata. When any definition's selected version changed, a definition-log entry capturing the current selection of every definition is appended, with entries ordered by definition-id.

func (*MetadataBuilder) Err 

func (b *MetadataBuilder) Err() error

Err returns the first error encountered by builder operations, if any.

func (*MetadataBuilder) HasChanges 

func (b *MetadataBuilder) HasChanges() bool

HasChanges reports whether any builder operation changed the state relative to the base metadata.

func (*MetadataBuilder) RemoveDefinition 

func (b *MetadataBuilder) RemoveDefinition(definitionID string) *MetadataBuilder

RemoveDefinition removes the definition with the given definition-id.

func (*MetadataBuilder) RemoveProperties 

func (b *MetadataBuilder) RemoveProperties(keys []string) *MetadataBuilder

RemoveProperties removes the given property keys.

func (*MetadataBuilder) SetCurrentVersion 

func (b *MetadataBuilder) SetCurrentVersion(definitionID string, versionID int) *MetadataBuilder

SetCurrentVersion sets the definition's current version to an existing version id, e.g. to roll back to an earlier implementation.

func (*MetadataBuilder) SetDoc 

func (b *MetadataBuilder) SetDoc(doc string) *MetadataBuilder

SetDoc sets the function's documentation string.

func (*MetadataBuilder) SetLoc 

func (b *MetadataBuilder) SetLoc(loc string) *MetadataBuilder

SetLoc sets the function's base location, used to create metadata file locations.

func (*MetadataBuilder) SetLogTimestampMS 

func (b *MetadataBuilder) SetLogTimestampMS(timestampMS int64) *MetadataBuilder

SetLogTimestampMS overrides the timestamp used for the definition-log entry appended by Build. Writers that need deterministic metadata (or a timestamp consistent with an external commit) should set this; when unset it defaults to time.Now().UnixMilli().

func (*MetadataBuilder) SetProperties 

func (b *MetadataBuilder) SetProperties(props iceberg.Properties) *MetadataBuilder

SetProperties adds or updates the given properties.

func (*MetadataBuilder) SetSecure 

func (b *MetadataBuilder) SetSecure(secure bool) *MetadataBuilder

SetSecure marks whether this is a secure function.

func (*MetadataBuilder) SetUUID 

func (b *MetadataBuilder) SetUUID(newUUID uuid.UUID) *MetadataBuilder

SetUUID assigns the function's UUID. The UUID is generated once at creation and cannot be reassigned.

type OnNullInput 

type OnNullInput string

OnNullInput defines how a UDF behaves when any input parameter is NULL. 

const (
	// OnNullInputCall means the function may handle NULLs internally, so
	// engines must execute it even if some inputs are NULL. This is the
	// default behavior when on-null-input is absent.
	OnNullInputCall OnNullInput = "call"
	// OnNullInputReturnNull means the function always returns NULL if any
	// input argument is NULL, allowing engines to skip evaluation.
	OnNullInputReturnNull OnNullInput = "return-null"
)

type Parameter 

type Parameter struct {
	Name string `json:"name"`
	Type Type   `json:"type"`
	Doc  string `json:"doc,omitempty"`
}

Parameter is a single function parameter.

func (*Parameter) UnmarshalJSON 

func (p *Parameter) UnmarshalJSON(b []byte) error

type PrimitiveType 

type PrimitiveType struct {
	// contains filtered or unexported fields
}

PrimitiveType is a primitive or semi-structured UDF type, stored as the verbatim type string (e.g. "int", "decimal(9,2)", "variant"). Values are always validated: construct them with PrimitiveTypeOf.

func PrimitiveTypeOf 

func PrimitiveTypeOf(s string) (PrimitiveType, error)

PrimitiveTypeOf validates s against the Iceberg type system and the UDF spec's rules for type strings (no spaces or quote characters) and returns the corresponding PrimitiveType.

func (PrimitiveType) Equals 

func (p PrimitiveType) Equals(other Type) bool

func (PrimitiveType) MarshalJSON 

func (p PrimitiveType) MarshalJSON() ([]byte, error)

func (PrimitiveType) String 

func (p PrimitiveType) String() string

type Representation 

type Representation interface {
	// RepresentationType returns the representation's type discriminator,
	// e.g. "sql".
	RepresentationType() string
	// contains filtered or unexported methods
}

Representation is a single implementation of a definition version. The only representation type defined by the UDF spec is SQL; unrecognized types round-trip through UnknownRepresentation.

type SQLRepresentation 

type SQLRepresentation struct {
	Dialect string `json:"dialect"`
	SQL     string `json:"sql"`
}

SQLRepresentation stores the function body as a SQL expression in a specific dialect. The SQL must reference parameters using the names declared in the definition's parameters.

func NewSQLRepresentation 

func NewSQLRepresentation(dialect, sql string) (SQLRepresentation, error)

NewSQLRepresentation creates a SQL representation from a dialect identifier (e.g. "spark", "trino") and a SQL expression.

func (SQLRepresentation) MarshalJSON 

func (r SQLRepresentation) MarshalJSON() ([]byte, error)

func (SQLRepresentation) RepresentationType 

func (SQLRepresentation) RepresentationType() string

type StructField 

type StructField struct {
	Name string
	Type Type
}

StructField is a single field of a UDF struct type, carrying only a name and a type.

type StructType 

type StructType struct {
	Fields []StructField
}

StructType is a UDF struct type carrying only its fields.

func (StructType) Equals 

func (s StructType) Equals(other Type) bool

func (StructType) MarshalJSON 

func (s StructType) MarshalJSON() ([]byte, error)

func (StructType) String 

func (s StructType) String() string

type Type 

type Type interface {
	fmt.Stringer

	Equals(Type) bool
	// contains filtered or unexported methods
}

Type represents a UDF parameter or return type as defined by the UDF spec. UDF types are Iceberg types without field IDs: primitives are encoded as plain strings (e.g. "int", "decimal(9,2)") and nested types (list, map, struct) as JSON objects carrying only the fields the UDF spec requires.

String returns the canonical string representation used to build definition IDs (see CanonicalDefinitionID).

type UDF 

type UDF struct {
	// contains filtered or unexported fields
}

UDF is a function (SQL UDF) loaded from a catalog: its catalog identifier, its metadata, and the location the metadata was loaded from.

func New 

func New(ident table.Identifier, meta Metadata, metadataLocation string) *UDF

New creates a UDF from its catalog identifier, metadata and metadata location.

func (UDF) Definitions 

func (u UDF) Definitions() []*Definition

func (UDF) Equals 

func (u UDF) Equals(other UDF) bool

Equals reports whether two UDFs have the same identifier, metadata location and metadata. The value receiver and argument are intentional, matching view.View: a UDF is a cheap value of references, so callers holding the *UDF from New compare with fn.Equals(*other).

func (UDF) Identifier 

func (u UDF) Identifier() table.Identifier

func (UDF) Location 

func (u UDF) Location() string

func (UDF) Metadata 

func (u UDF) Metadata() Metadata

func (UDF) MetadataLocation 

func (u UDF) MetadataLocation() string

func (UDF) Properties 

func (u UDF) Properties() iceberg.Properties

type UnknownRepresentation 

type UnknownRepresentation struct {
	TypeName string
	// contains filtered or unexported fields
}

UnknownRepresentation preserves a representation of an unrecognized type so that metadata written by newer or extended writers round-trips intact.

func (UnknownRepresentation) MarshalJSON 

func (r UnknownRepresentation) MarshalJSON() ([]byte, error)

func (UnknownRepresentation) Raw 

func (r UnknownRepresentation) Raw() json.RawMessage

Raw returns the representation's original JSON.

func (UnknownRepresentation) RepresentationType 

func (r UnknownRepresentation) RepresentationType() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.