lys

package module
v0.3.42 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 15, 2026 License: MIT Imports: 34 Imported by: 2

README

lys - LoveYourStack

Packages for rapid development of REST APIs handling database CRUD actions.

Only available for PostgreSQL. Most suitable for "database-first" Go developers.

Example usage

Define store package (wiki)

A store package contains database access functions for a specific table or view, in this case the "category" table in the "core" schema.

Boilerplate is minimized through the optional use of generic database CRUD functions.

package corecategory

// define constants for this database table, which get used in generic database functions below
const (
	schemaName     string = "core"
	tableName      string = "category"
	viewName       string = "category"
	pkColName      string = "id"
	defaultOrderBy string = "name"
)

// columns required when creating or updating a record
type Input struct {
	Name string `db:"name" json:"name,omitempty" validate:"required"`
}

// columns outputted when selecting a record. Note that Input is embedded
type Model struct {
	Id    int64 `db:"id" json:"id"`
	Input
}

type Store struct {
	Db *pgxpool.Pool
}

// define functions for this table as methods of the Store struct
// use lyspg generic functions if possible, but can also write custom implementations

func (s Store) Delete(ctx context.Context, id int64) error {
	return lyspg.DeleteUnique(ctx, s.Db, schemaName, tableName, pkColName, id)
}

func (s Store) Insert(ctx context.Context, input Input) (newId int64, err error) {
	return lyspg.Insert[Input, int64](ctx, s.Db, schemaName, tableName, pkColName, input)
}

func (s Store) Select(ctx context.Context, params lyspg.SelectParams) (items []Model, unpagedCount lyspg.TotalCount, err error) {
	return lyspg.Select[Model](ctx, s.Db, schemaName, tableName, viewName, defaultOrderBy, plan.DbNames(), params)
}

// etc

Create routes (wiki)

Pass the store package to generic GET, POST, etc handlers to get full REST API CRUD functionality for this table with minimal boilerplate.

package main

func (srvApp *httpServerApplication) getRoutes(apiEnv lys.Env) http.Handler {

	endpoint := "/core-categories"

	// get full CRUD functionality using lys generic handlers, passing the store defined above
	// no framework: free to write custom handlers when needed

	categoryStore := corecategory.Store{Db: srvApp.Db}
	r.HandleFunc(endpoint, lys.Get(apiEnv, categoryStore)).Methods("GET")
	r.HandleFunc(endpoint+"/{id}", lys.GetById(apiEnv, categoryStore)).Methods("GET")
	r.HandleFunc(endpoint, lys.Post(apiEnv, categoryStore)).Methods("POST")
	r.HandleFunc(endpoint+"/{id}", lys.Put(apiEnv, categoryStore)).Methods("PUT")
	r.HandleFunc(endpoint+"/{id}", lys.Patch(apiEnv, categoryStore)).Methods("PATCH")
	r.HandleFunc(endpoint+"/{id}", lys.Delete(apiEnv, categoryStore)).Methods("DELETE")
}

Use routes

We can now start the HTTP server app and use the routes above.

curl localhost:8010/core-categories?name=Seafood
curl localhost:8010/core-categories/1
curl --header "Content-Type: application/json" --request POST --data '{"name":"Fruit"}' localhost:8010/core-categories
# etc

Features

  • Library only: is not a framework, and does not use code generation, so can be overriden at every step to deal with exceptional cases
  • Support for GET many, GET single, POST, PUT, PATCH and DELETE
  • Support for sorting, paging and filtering GET results via customizable URL params
  • Uses pgx for database access and only uses parameterized SQL queries
  • Support for Excel and CSV output
  • Uses generics and reflection to minimize boilerplate
  • Custom date/time types with zero default values and sensible JSON formats
  • Fast rowcount function, including estimated count for large tables with query conditions
  • Struct validation using validator
  • Distinction between user errors (unlogged, reported to user) and application errors (logged, hidden from user)
  • Provides useful bulk insert (COPY) wrapper, and bulk update/delete (batch) wrappers
  • Support for getting and filtering enum values
  • Support for selection from database set-returning functions
  • Supports API obfuscation of database columns via differing JSON tags
  • Database creation function from embedded SQL files
  • Archive (soft delete) + restore functions
  • and more. See the wiki

Current limitations

  • Only supports PostgreSQL

Testing

See CONTRIBUTING.md for setup instructions.

Supported Go and PostgreSQL Versions

Preliminary values:

Go 1.16+ (due to embed.FS)

PostgreSQL 13+ (due to gen_random_uuid)

Documentation

Overview

Package lys is used for rapid development of REST APIs handling database CRUD actions.

Please see the README and wiki for an overview.

Index

Constants

View Source
const (
	FormatCsv   string = "csv"
	FormatExcel string = "excel"
	FormatJson  string = "json"
)

output format consts

View Source
const (
	// status
	ReqSucceeded string = "succeeded"
	ReqFailed    string = "failed"

	// data
	DataArchived string = "archived"
	DataDeleted  string = "deleted"
	DataRestored string = "restored"
	DataUpdated  string = "updated"
)

response constants

View Source
const UserInfoCtxKey lyspgdb.ContextKey = "UserInfoKey"

UserInfoCtxKey is the key that should be used when binding a user info struct to a request via context if you use this key and add a GetUserName() string method to the struct, the username will be included in error logs when using the error handlers in error_handlers.go

Variables

View Source
var (

	// bad requests (default status)
	ErrBodyMissing        = lyserr.User{Message: "request body missing"}
	ErrIdMissing          = lyserr.User{Message: "id missing"}
	ErrIdNotAUuid         = lyserr.User{Message: "id not a uuid"}
	ErrIdNotAnInteger     = lyserr.User{Message: "id not an integer"}
	ErrIdNotUnique        = lyserr.User{Message: "id not unique"} // the handling func was expecting id to be unique, but it is not
	ErrIdParseError       = lyserr.User{Message: "id could not be parsed"}
	ErrInvalidContentType = lyserr.User{Message: "content type must be application/json"}
	ErrInvalidId          = lyserr.User{Message: "invalid id"} // the id sent is not present in the relevant table
	ErrInvalidJson        = lyserr.User{Message: "invalid json"}
	ErrNoAssignments      = lyserr.User{Message: "no assignments found"} // for patch reqs where assignmentMap is expected
	ErrNotParseableToMap  = lyserr.User{Message: "json body could not be parsed into a map of field names to values"}
	ErrRouteNotFound      = lyserr.User{Message: "route not found"}

	// forbidden
	ErrPermissionDenied = lyserr.User{Message: "permission denied", StatusCode: http.StatusForbidden} // authorization failed
	ErrUserInfoMissing  = lyserr.User{Message: "userInfo missing", StatusCode: http.StatusForbidden}  // failed to get ReqUserInfo from context
)

expected user errors during validation

Functions

func Archive added in v0.1.16

func Archive[idT lyspg.PrimaryKeyType](env Env, db *pgxpool.Pool, store iArchiveable[idT]) http.HandlerFunc

Archive handles moving a record from the supplied store into its archived table

func DecodeJsonBody

func DecodeJsonBody[T any](body []byte) (dest T, err error)

DecodeJsonBody decodes the supplied json body into dest and checks for a variety of error conditions. Caller should check that body is valid JSON and should enforce a maximum body size (usually done in ExtractJsonBody). Adapted from https://www.alexedwards.net/blog/how-to-properly-parse-a-json-request-body.

func Delete

func Delete[idT lyspg.PrimaryKeyType](env Env, store iDeletable[idT]) http.HandlerFunc

Delete handles deletion of a single item using the supplied store.

func ExtractFields

func ExtractFields(fieldsParamName, fieldsVal string, dbNames lysset.Set[string], jsonKeyDbNameMap map[string]string) (fields []string, err error)

ExtractFields returns a slice of strings parsed from the request's fields param

func ExtractFilters

func ExtractFilters(urlValues url.Values, jsonKeyDbNameMap map[string]string, additionalFilterParamNames lysset.Set[string], setFuncUrlParamNames []string, getOptions GetOptions) (conds []lyspg.Condition, err error)

ExtractFilters returns a slice of conditions parsed from the request's params to get urlValues from a request: r.Url.Query()

func ExtractFormat added in v0.1.5

func ExtractFormat(formatParamName, formatVal string) (format string, err error)

ExtractFormat returns the output format for the GET req

func ExtractJsonBody

func ExtractJsonBody(r *http.Request, maxBodySize int64) (body []byte, err error)

ExtractJsonBody reads and validates the body of the supplied request.

func ExtractPaging

func ExtractPaging(pageParamName, pageVal, perPageParamName, perPageVal string, defaultPerPage, maxPerPage int) (page int, perPage int, err error)

ExtractPaging returns paging variables parsed from a request's paging params page defaults to 1, perPage defaults to defaultPerPage

func ExtractSetFuncParamValues added in v0.1.39

func ExtractSetFuncParamValues(r *http.Request, setFuncUrlParamNames []string) (setFuncUrlParamValues []any, err error)

ExtractSetFuncParamValues returns the values to be passed to the SQL setFunc each param is currently treated as mandatory

func ExtractSorts

func ExtractSorts(sortParamName, sortVal string, jsonKeyDbNameMap map[string]string) (sortCols []string, err error)

ExtractSorts returns an array of SQL sorting statements parsed from the request's sort param

func FileResponse added in v0.1.5

func FileResponse(filePath, outputFileName string, remove bool, w http.ResponseWriter)

FileResponse opens the supplied file and streams it to w as a file

func Get

func Get[T any](env Env, store iGetable[T], opts *GetOpts[T]) http.HandlerFunc

Get handles retrieval of multiple items from the supplied store

func GetById

func GetById[idT lyspg.PrimaryKeyType, outT any](env Env, store iGetableById[idT, outT]) http.HandlerFunc

GetById handles retrieval of a single item from the supplied store.

func GetEnumValues

func GetEnumValues(env Env, db *pgxpool.Pool, schema, enum string) http.HandlerFunc

GetEnumValues returns enum values from the supplied schema and enum type name

func GetFunc added in v0.1.38

func GetFunc[T any](env Env, store iGetable[T], selectFunc func(ctx context.Context, params lyspg.SelectParams) (items []T, unpagedCount lyspg.TotalCount, err error)) http.HandlerFunc

GetFunc is a wrapper for Get which allows passing an alternative Select func with the same signature

func GetSimple

func GetSimple[T any](env Env, selectFunc func(ctx context.Context) (items []T, err error)) http.HandlerFunc

GetSimple handles retrieval of all items returned by selectFunc, which may only take ctx as param

func GetUserNameFromCtx

func GetUserNameFromCtx(ctx context.Context) string

GetUserNameFromCtx returns the username if it can be obtained from ctx using the UserInfoCtxKey struct if it has a GetUserName() method. Otherwise it returns "Unknown"

func GetValue added in v0.3.22

func GetValue[T any](env Env, db *pgxpool.Pool, stmt string) http.HandlerFunc

GetValue handles retrieval of a single value returned by stmt

func GetWithLastSync added in v0.1.9

func GetWithLastSync[T any](env Env, store iGetableWithLastSync[T]) http.HandlerFunc

GetWithLastSync is a wrapper for Get which adds the lastSyncAt timestamp from the supplied func to the JSON response

func HandleDbError

func HandleDbError(ctx context.Context, line int, stmt string, err error, errorLog *slog.Logger, w http.ResponseWriter)

HandleDbError returns a specific error message to the API user if the error is caused by a bad input, e.g. a check or uniqueness violation. Otherwise it returns a generic error message to the API user and logs the specific error

func HandleError added in v0.1.25

func HandleError(ctx context.Context, err error, errorLog *slog.Logger, w http.ResponseWriter)

HandleError is the general method for handling API errors where err could contain wrapped errors of other types

func HandleExtError added in v0.1.25

func HandleExtError(ctx context.Context, extMessage string, err error, errorLog *slog.Logger, w http.ResponseWriter)

HandleExtError returns the external message to the API user and logs the error

func HandleInternalError

func HandleInternalError(ctx context.Context, err error, errorLog *slog.Logger, w http.ResponseWriter)

HandleInternalError returns a generic error message to the API user and logs the error

func HandleUserError

func HandleUserError(err lyserr.User, w http.ResponseWriter)

HandleUserError returns a helpful message to the API user, but does not log the error. If HTTP status is not provided, BadRequest is assumed.

func Import added in v0.3.8

func Import[inputT any](env Env, db *pgxpool.Pool, store iImportable[inputT], valRepls ...ImportValueRepl) http.HandlerFunc

Import handles creating multiple new items using the supplied store and returning the number of rows inserted the supplied db is used for the MapFunc in valRepls

func JsonResponse

func JsonResponse(resp StdResponse, httpStatus int, w http.ResponseWriter)

JsonResponse marshals the supplied StdResponse to json and writes it to w

func Message

func Message(msg string) http.HandlerFunc

Message returns the supplied msg in the Data field

func NotFound

func NotFound() http.HandlerFunc

NotFound provides a response informing the user that the requested route was not found

func Patch

func Patch[idT lyspg.PrimaryKeyType](env Env, store iPatchable[idT]) http.HandlerFunc

Patch handles changing some of an item's fields using the supplied store.

func PgSleep

func PgSleep(db lyspg.PoolOrTx, errorLog *slog.Logger, sleepSecs, cancelAfterSecs int) http.HandlerFunc

PgSleep creates an artifical longrunning query in the db which can be viewed using pg_stat_activity. Pass cancelAfterSecs as 0 to not cancel the request. Used for testing context cancelation

func Post

func Post[inputT any, outputT any](env Env, store iPostable[inputT, outputT]) http.HandlerFunc

Post handles creating a new item using the supplied store and returning an output (the new item or its ID) in the response

func ProcessSlice added in v0.1.27

func ProcessSlice[T any](env Env, processFunc func(context.Context, []T) (int64, error)) http.HandlerFunc

ProcessSlice extracts a slice from the req body and passes it into the supplied processFunc

func Put

func Put[idT lyspg.PrimaryKeyType, inputT any](env Env, store iPutable[idT, inputT]) http.HandlerFunc

Put handles changing an item using the supplied store.

func Restore

func Restore[idT lyspg.PrimaryKeyType](env Env, db *pgxpool.Pool, store iArchiveable[idT]) http.HandlerFunc

Restore handles moving a record from the store's archived table back to the main table

func RunSimple added in v0.1.48

func RunSimple(env Env, runFunc func(context.Context) error) http.HandlerFunc

RunSimple runs the supplied func

Types

type Env

type Env struct {
	ErrorLog    *slog.Logger
	Validate    *validator.Validate
	GetOptions  GetOptions
	PostOptions PostOptions
}

Env (environment) contains objects and options needed by API calls

type ExtractGetRequestModifierParams added in v0.3.28

type ExtractGetRequestModifierParams struct {
	AdditionalFilterParamNames lysset.Set[string]
	DbNames                    lysset.Set[string]
	GetOptions                 GetOptions
	JsonKeyDbNameMap           map[string]string
	SetFuncUrlParamNames       []string
}

type GetMetadata added in v0.1.8

type GetMetadata struct {
	Count                 int   `json:"count"`
	TotalCount            int64 `json:"total_count"`
	TotalCountIsEstimated bool  `json:"total_count_is_estimated"`
}

type GetOptions

type GetOptions struct {
	FieldsParamName  string // name of the param which limits the fields returned by a GET request, e.g. "xfields=name,age"
	FormatParamName  string // name of the param which determines the output format of a GET request, e.g. "xformat=csv"
	PageParamName    string // name of the param which defines the page offset returned by a paged GET request, e.g. "xpage=1"
	PerPageParamName string // name of the param which defines the number of records returned by a paged GET request, e.g. "xper_page=20"
	SortParamName    string // name of the param which sorts the records returned by a GET request, e.g. "xsort=name,-age"

	MultipleValueSeparator string // the string used by a GET request to separate values in a filter where each value should be returned, e.g. "|", usage: "name=Bill|Sam"
	MetadataSeparator      string // the string used to separate any extra data appended to a GET request query filter, e.g. "^", usage: "sales=>100^Last 7 days"

	DefaultPerPage int // default number of results returned by a paged GET request, e.g. 20
	MaxPerPage     int // max number of results returned per paged GET request, regardless of what the caller enters in the "PerPageParamName" param, e.g. 500

	MaxFileRecs  int  // max number of records contained in a file output
	CsvDelimiter rune // delimiter between values in CSV file output. 0 means not set, and the default will be used.
}

GetOptions contains the options used when processing GET requests, such as paging param names and default values. Since the json field names are used as filters, param names should be chosen which will never appear as json field names. This is the reason for the "x" prefix in the default param names.

func FillGetOptions

func FillGetOptions(input GetOptions) (ret GetOptions, err error)

FillGetOptions returns input GetOptions if they are passed, and sets any unset fields to a sensible default value

type GetOpts added in v0.3.36

type GetOpts[T any] struct {

	// AdditionalFilterParamNames are param names that are not in the store's db tags, but should be allowed anyway. Must be handled by the store's Select func.
	AdditionalFilterParamNames lysset.Set[string]

	// GetLastSyncAt gets the last synced timestamp for external data.
	GetLastSyncAt func(ctx context.Context) (lastSyncAt lystype.Datetime, err error)

	// SelectFunc, if passed, overrides the default store Select() func.
	SelectFunc func(ctx context.Context, params lyspg.SelectParams) (items []T, unpagedCount lyspg.TotalCount, err error)

	// SetFuncUrlParamNames are used if selecting from a setFunc rather than a view. They are the names of the url params that will be passed, in order, to the setFunc.
	// Don't use a Set: order must be preserved.
	SetFuncUrlParamNames []string
}

type GetReqModifiers

type GetReqModifiers struct {
	Format             string
	Fields             []string
	Conditions         []lyspg.Condition
	Page               int
	PerPage            int
	Sorts              []string
	SetFuncParamValues []any
}

GetReqModifiers contains data from a GET request's Url params which is used to modify a database SELECT statement

func ExtractGetRequestModifiers

func ExtractGetRequestModifiers(r *http.Request, params ExtractGetRequestModifierParams) (getReqModifiers GetReqModifiers, err error)

ExtractGetRequestModifiers reads the Url params of the supplied GET request and converts them into a GetReqModifiers

type ImportValueRepl added in v0.3.9

type ImportValueRepl struct {
	StringJsonName string
	Int64JsonName  string
	MapFunc        func(context.Context, *pgxpool.Pool) (map[string]int64, error)
}

ImportValueRepl is a struct that can be used when the input contains a foreign key. It allows the referenced table's string representation to be passed by the user. The string attribute gets replaced with the int64 attribute, and all the values are mapped using the supplied map. For example:

StringJsonName: "car_manufacturer"
Int64JsonName:  "car_manufacturer_fk"
MapFunc:        returns Ford = 1, Nissan = 2, etc

type PostOptions

type PostOptions struct {
	MaxBodySize   int64 // max bytes allowed in request body
	MaxImportRecs int   // max number of records allowed in Import
}

PostOptions contains the options used when processing POST or PUT requests

func FillPostOptions

func FillPostOptions(input PostOptions) (ret PostOptions)

FillPostOptions returns input PostOptions if they are passed, and sets any unset fields to a sensible default value

type RouteAdderFunc

type RouteAdderFunc func(r *mux.Router) *mux.Router

RouteAdderFunc is a function returning a subrouter

type StatusWriter added in v0.3.24

type StatusWriter struct {
	http.ResponseWriter
	Status int
	Bytes  int
}

StatusWriter is a wrapper around http.ResponseWriter that captures the status code and number of bytes written in the response. It implements http.Flusher and http.Hijacker so that it can also write websocket responses.

func (*StatusWriter) Flush added in v0.3.42

func (sw *StatusWriter) Flush()

func (*StatusWriter) Hijack added in v0.3.42

func (sw *StatusWriter) Hijack() (net.Conn, *bufio.ReadWriter, error)

func (*StatusWriter) Unwrap added in v0.3.42

func (sw *StatusWriter) Unwrap() http.ResponseWriter

func (*StatusWriter) Write added in v0.3.24

func (sw *StatusWriter) Write(b []byte) (int, error)

func (*StatusWriter) WriteHeader added in v0.3.24

func (sw *StatusWriter) WriteHeader(code int)

type StdResponse

type StdResponse struct {
	Status         string            `json:"status"`
	Data           any               `json:"data,omitempty"`
	GetMetadata    *GetMetadata      `json:"metadata,omitempty"`     // only used for GET many
	LastSyncAt     *lystype.Datetime `json:"last_sync_at,omitempty"` // if the data was synced from external source: the last sync timestamp
	ErrDescription string            `json:"err_description,omitempty"`
}

StdResponse is the return type of all API routes

type SubRoute

type SubRoute struct {
	Url        string
	RouteAdder RouteAdderFunc
}

SubRoute contains a Url path and the function returning the subrouter to process that path

Directories

Path Synopsis
internal
cmd
cmd/lyscli command
Package lysclient contains types and functions to help test a REST API which was created using lys functions.
Package lysclient contains types and functions to help test a REST API which was created using lys functions.
Package lyserr contains structs related to error handling used in lys and lyspg.
Package lyserr contains structs related to error handling used in lys and lyspg.
Package lysgen contains experimental functions to generate code from Postgres database tables.
Package lysgen contains experimental functions to generate code from Postgres database tables.
Package lysmeta contains functions that analyze structs.
Package lysmeta contains functions that analyze structs.
Package lyspg contains structs and functions providing generic CRUD operations on a Postgres database.
Package lyspg contains structs and functions providing generic CRUD operations on a Postgres database.
Package lyspgdb contains functions for creating and connecting to Postgres databases.
Package lyspgdb contains functions for creating and connecting to Postgres databases.
Package lyspgdb contains functions for monitoring Postgres databases.
Package lyspgdb contains functions for monitoring Postgres databases.
Package lysstring contains string functions.
Package lysstring contains string functions.
Package lystype contains date/time types used in lys and lyspg.
Package lystype contains date/time types used in lys and lyspg.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL