clickhousestats

Overview ¶

Package clickhousestats implements statistics logging to the ClickHouse database using async_insert queries.

Data schema ¶

This library supports any struct-based type or a pointer to it for representing a statistics record.

A special field named `_` or `_struct` can be used to specify the table name that will be used for logging records of that type. The type of this field is ignored, and the struct tag format is compatible with the clickhouse-gate-client library:

type StatsRecord struct {
    _ struct{} `clickhousegate:"table_name, everything other is ignored"`

Any other non-exported fields (i.e. starting with a lowercase letter) are ignored completely.

If no table name is specified using such a tag, then the type name converted to snake_case is used as the table name. The conversion is done using the github.com/iancoleman/strcase library:

type StatsRecord struct {
    // no tag: the table name is stats_record

If the struct type name can't be determined (e.g. when the struct type isn't a defined type), the ErrNoTable error will be returned.

All exported struct fields must correspond to similarly named table column names. The table might have more columns than the struct. To match names, both are converted to the lowercase, and _'s are removed. The common convention is CamelCase struct fields and snake_case table column names, but this isn't enforced. Currently, you can't customize a column name (e.g. using a struct tag).

The following types are fully supported and tested:

• All "simple" types like ints, floats, and string.
• time.Time for Date, DateTime, or DateTime64 column type. Subsecond resolution (3, 6, or 9 decimal places for fractional seconds) is preserved.
• UUID type (using both popular libraries: github.com/gofrs/uuid/v5 and github.com/google/uuid).
• github.com/shopspring/decimal.Decimal type (although you can use floats with a Decimal column, it is highly discouraged).
• Any type (including the `any` type, i.e. an empty interface) for a JSON column type.
• Nullable columns are supported using pointers.

JSON fields are marshaled using encoding/json from the standard library.

No special declaration is required for LowCardinality types or Enums.

clickhouse-gate-client compatibility ¶

This library aims to be compatible with clickhouse-gate-client but isn't an exact "drop-in replacement". Some points must be considered to achieve a compatibility:

• Both `_` and `_struct` fields are supported to specify the `clickhousegate` struct tag. The table name is used, and all the flags (like validate_names) are ignored.
• This library uses the table schema from the actual database and doesn't require a "category" schema in OnlineConf.
• The configuration is entirely different from that used by clickhouse-gate-client, despite being compatible with the clickhouse-gate configuration.
• This library doesn't provide some hack wrapper types like JSON or TimeMs. You can use any type for a JSON column, and time.Time for Date/DateTime/DateTime64 column types of any precision.
• Unlike clickhouse-gate-client, this library doesn't require any special field order.
• Some features (like a column name customization) aren't implemented intentionally to allow migration from and to clickhouse-gate-client.

Configuration ¶

The following configuration settings are used by this library:

• /clickhouse/native-host (required) - the ClickHouse native protocol endpoint.
• /clickhouse/base (default: "default")
• /clickhouse/user (default: no user)
• /clickhouse/pass (default: no password)
• /clickhouse/settings (default: empty) - a JSON or YAML map containing ClickHouse session settings: https://clickhouse.com/docs/operations/settings/settings. Settings with "async_insert" in the name are especially important for this library's performance.
• /clickhouse/debug (default: false) - github.com/ClickHouse/clickhouse-go/v2 debug mode
• /clickhouse/max-open-conns (default: 10) - maximum connections in a pool, also used as the worker goroutine count.
• /clickhouse/max-idle-conns (default: 3) - maximum idle connections in a pool.
• /busy-percentage (default: 70) - when the worker goroutine busy percentage is less than or equal to this value, the "wait" INSERT mode is used (the query status is checked and the record is logged as lost on INSERT failure). Otherwise, the "fire-and-forget" mode is used.
• /send-timeout (default: 50ms). It is used when all worker goroutines are busy (even the "fire-and-forget" ones). If this timeout is reached, the record is logged as lost. You can use negative values to wait forever, e.g. for cron jobs or other non-interactive cases, along with the pool size of 1.

When any configuration value whose key begins with /clickhouse/ gets changed, the database connection pool is re-initiated. Other values are checked dynamically. So you can change *any* configuration parameter on the fly without an application restart.

Although you can use any type implementing the OnlineConf interface, including Module, the typical configuration is:

# in onlineconf
/project/application/stats/clickhouse -> /infrastructure/clickhouse/project
/project/application/stats/send-timeout: 1m  # specified directly

# in your app's main function
stats, err := clickhousestats.Connect(ctx, oconfRoot.Subtree("/project/application/stats"))

Lost data ¶

In these cases, a data record is considered as "lost":

• when all the worker goroutines are busy and /send-timeout is reached,
• when the async_insert query has returned an error,
• when sending new data after calling Client.Close.

"Lost" records are logged using log/slog with the following keys:

• message key: "clickhousestats: lost record",
• "err" key - ErrShutdown, ErrBusy, or some clickhouse-go error,
• "record" key - the data itself.

TODO implement a package for recovering "lost" data from the logs.

Metrics ¶

Using the WithMetrics option with the Connect enables metrics collection with the supplied advmetricsset.Set. The following metrics are collected:

Connection pool metrics (labels: "addr" and "database"):

• clickhousestats_connections - current count of connections (gauge)
• clickhousestats_connection_duration_seconds - time spent while establishing new connections (histogram)
• clickhousestats_connection_errors - count of errors during establishing new connections (counter)

Record delivery metrics (label: "table")

• clickhousestats_records_total - total count of records sent (counter)
• clickhousestats_records_nowait - count of records sent in the "fire-and-forget" mode (i.e. when the /busy-percentage is exceeded) (counter)
• clickhousestats_records_lost - count of records that weren't delivered to the database and were logged with the "clickhousestats: lost record" message (counter)

Examples ¶

Schema declaration in the DB:

CREATE TABLE stats_data (
	id       UUID,
	name     String,
	optional Nullable(String),
	custom   JSON
)

Schema declaration in go:

type StatsData struct {
	_ struct{} `clickhousegate:"stats_data"`
	ID       uuid.UUID
	Name     string
	Optional *string
	Custom   JSONData
}

type JSONData struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
}