README
¶
go-benchmarks: a benchmark library with mean, median and arbitrary quantiles
Can be incorporated in tests, or in stand-alone programs.
Note: for more stability in tests, consider fixing the CPU frequency on the box being used. Typically, in a linux box this can be done with the tool
cpupower. For instancesudo cpupower frequency-set -r -d 3000000 -u 3000000will set the min and max frequency to 3Ghz.
Highlights:
- Pretty-printing of the duration of each execution (to avoid having to count digits to interpret scale of Go's default benchmark framework). Duration pretty-printing function can be arbitrarily configured.
- Includes also median, and arbitrary percentiles (default to 5 and 99).
Outputs will look like this:
Benchmarks: Mean Median 5%-tile 99%-tile Count(x10)
TestBenchArena/arena/1 263ns 262ns 259ns 272ns 334047
TestBenchArena/arena/5 285ns 284ns 280ns 296ns 311296
TestBenchArena/arena/10 322ns 321ns 316ns 366ns 279129
TestBenchArena/arena/100 968ns 963ns 948ns 1.1µs 99799
TestBenchArena/arenaPool/1 148ns 148ns 143ns 160ns 537448
TestBenchArena/arenaPool/5 173ns 172ns 168ns 185ns 477437
TestBenchArena/arenaPool/10 209ns 207ns 203ns 244ns 407099
TestBenchArena/arenaPool/100 877ns 851ns 826ns 1.1µs 109780
TestBenchArena/malloc/1 232ns 231ns 227ns 240ns 373248
TestBenchArena/malloc/5 890ns 887ns 881ns 1.0µs 108137
TestBenchArena/malloc/10 1.7µs 1.7µs 1.7µs 1.9µs 57526
TestBenchArena/malloc/100 16.4µs 16.3µs 16.2µs 16.7µs 6101
TestBenchArena/go+pinner/1 156ns 153ns 146ns 186ns 514560
TestBenchArena/go+pinner/5 557ns 545ns 535ns 703ns 168819
TestBenchArena/go+pinner/10 1.1µs 1.1µs 1.0µs 1.5µs 85634
TestBenchArena/go+pinner/100 12.3µs 11.8µs 11.5µs 34.4µs 8137
Examples
Simple Example
import (
benchmarks "github.com/janpfeifer/go-benchmarks"
)
…
testParams := []int{1, 2, 3, 5, 8}
testFns := make([]benchmarks.NamedFunc, len(testParams))
for ii, param := range testParams {
testFns.Name = fmt.Printf("Param=%d", param)
testFns.Func() = func() {
MyFunc(param)
}
}
benchmarks.New(testFns...).Done()
Benchmarking a minimal CGO call
For this we use the WithInnerRepeats option, to mitigate the time to call the function being tested itself
(repeatedCGO).
You can run it with something like: go test . -run=TestBenchCGO -test.v
import (
benchmarks "github.com/janpfeifer/go-benchmarks"
)
func TestBenchCGO(t *testing.T) {
plugin := must1(GetPlugin(*flagPluginName))
const repeats = 1000
repeatedCGO := func() {
for _ = range repeats {
dummyCGO(unsafe.Pointer(plugin.api))
}
}
benchmarks.New(benchmarks.NamedFunction{"CGOCall", repeatedCGO}).
WithInnerRepeats(repeats).
Done()
}
Outputs (in an intel 12K900 with the frequency limited to 3Ghz):
Benchmarks: Mean Median 5%-tile 99%-tile Runs(x1000)
CGOCall 67ns 66ns 65ns 72ns 14666
Benchmarking Local Allocations To Use As Arguments To CGO functions
func TestBenchArena(t *testing.T) {
plugin := must1(GetPlugin(*flagPluginName))
client := must1(plugin.NewClient(nil))
defer runtime.KeepAlive(client)
numAllocationsList := []int{1, 5, 10, 100}
allocations := make([]*int, 100)
testFns := make([]benchmarks.NamedFunction, 4*len(numAllocationsList))
const repeats = 10
idxFn := 0
for _, allocType := range []string{"arena", "arenaPool", "malloc", "go+pinner"} {
for _, numAllocations := range numAllocationsList {
testFns[idxFn].Name = fmt.Sprintf("%s/%s/%d", t.Name(), allocType, numAllocations)
var fn func()
switch allocType {
case "arena":
fn = func() {
for _ = range repeats {
arena := newArena(1024)
for idx := range numAllocations {
allocations[idx] = arenaAlloc[int](arena)
}
dummyCGO(unsafe.Pointer(allocations[numAllocations-1]))
arena.Free()
}
}
case "arenaPool":
fn = func() {
for _ = range repeats {
arena := getArenaFromPool()
for idx := range numAllocations {
allocations[idx] = arenaAlloc[int](arena)
}
dummyCGO(unsafe.Pointer(allocations[numAllocations-1]))
returnArenaToPool(arena)
}
}
case "malloc":
fn = func() {
for _ = range repeats {
for idx := range numAllocations {
allocations[idx] = cMalloc[int]()
}
dummyCGO(unsafe.Pointer(allocations[numAllocations-1]))
for idx := range numAllocations {
cFree(allocations[idx])
}
}
}
case "go+pinner":
fn = func() {
for _ = range repeats {
var pinner runtime.Pinner
for idx := range numAllocations {
v := idx
allocations[idx] = &v
pinner.Pin(allocations[idx])
}
dummyCGO(unsafe.Pointer(allocations[numAllocations-1]))
pinner.Unpin()
}
}
}
testFns[idxFn].Func = fn
idxFn++
}
}
benchmarks.New(testFns...).
WithInnerRepeats(repeats).
WithWarmUps(10).
Done()
}
Results:
Benchmarks: Mean Median 5%-tile 99%-tile Count(x10)
TestBenchArena/arena/1 263ns 262ns 259ns 272ns 334047
TestBenchArena/arena/5 285ns 284ns 280ns 296ns 311296
TestBenchArena/arena/10 322ns 321ns 316ns 366ns 279129
TestBenchArena/arena/100 968ns 963ns 948ns 1.1µs 99799
TestBenchArena/arenaPool/1 148ns 148ns 143ns 160ns 537448
TestBenchArena/arenaPool/5 173ns 172ns 168ns 185ns 477437
TestBenchArena/arenaPool/10 209ns 207ns 203ns 244ns 407099
TestBenchArena/arenaPool/100 877ns 851ns 826ns 1.1µs 109780
TestBenchArena/malloc/1 232ns 231ns 227ns 240ns 373248
TestBenchArena/malloc/5 890ns 887ns 881ns 1.0µs 108137
TestBenchArena/malloc/10 1.7µs 1.7µs 1.7µs 1.9µs 57526
TestBenchArena/malloc/100 16.4µs 16.3µs 16.2µs 16.7µs 6101
TestBenchArena/go+pinner/1 156ns 153ns 146ns 186ns 514560
TestBenchArena/go+pinner/5 557ns 545ns 535ns 703ns 168819
TestBenchArena/go+pinner/10 1.1µs 1.1µs 1.0µs 1.5µs 85634
TestBenchArena/go+pinner/100 12.3µs 11.8µs 11.5µs 34.4µs 8137
Documentation
¶
Overview ¶
Package benchmarks implements a benchmark library with mean, median and arbitrary quantiles. It includes a pretty-printing function of duration -- that can be arbitrarily configured.
Example 1: Simple example:
testParams := []int{1, 2, 3, 5, 8}
testFns := make([]benchmarks.NamedFunc, len(testParams))
for ii, param := range testParams {
testFns.Name = fmt.Printf("Param=%d", param)
testFns.Func() = func() {
MyFunc(param)
}
}
benchmarks.New(testFns...).Done()
Example 2: Measuring a CGO call -- we use inner-repeats because the time is very small.
const repeats = 1000
repeatedCGO := func() {
for _ = range repeats {
dummyCGO(unsafe.Pointer(plugin.api))
}
}
benchmarks.New(benchmarks.NamedFunction{"CGOCall", repeatedCGO}).
WithInnerRepeats(repeats).
Done()
Index ¶
- Variables
- func PrettyPrint(d time.Duration) string
- type NamedFunction
- type Options
- func (o *Options) Done()
- func (o *Options) WithColumnSize(columnSize int) *Options
- func (o *Options) WithDuration(duration time.Duration) *Options
- func (o *Options) WithHeader(printHeader bool) *Options
- func (o *Options) WithInnerRepeats(innerRepeats int) *Options
- func (o *Options) WithPrettyPrintFn(fn func(time.Duration) string) *Options
- func (o *Options) WithQuantiles(quantiles ...int) *Options
- func (o *Options) WithTolerance(tolerance float64) *Options
- func (o *Options) WithWarmUps(warmUps int) *Options
Constants ¶
This section is empty.
Variables ¶
var DefaultQuantiles = []int{5, 99}
DefaultQuantiles to use in benchmarking. It can be changed for a particular benchmark using Options.WithQuantiles.
Functions ¶
func PrettyPrint ¶
Types ¶
type NamedFunction ¶
type NamedFunction struct {
Name string
Func func()
}
NamedFunction holds a function to be benchmarked and its name.
type Options ¶
type Options struct {
// contains filtered or unexported fields
}
func New ¶
func New(fns ...NamedFunction) *Options
New sets up a benchmark for the list of named functions fns. You can further configure the returned Options, and call Done() when finished to execute the benchmark.
Example 1: Simple example:
testParams := []int{1, 2, 3, 5, 8}
testFns := make([]benchmarks.NamedFunc, len(testParams))
for ii, param := range testParams {
testFns.Name = fmt.Printf("Param=%d", param)
testFns.Func() = func() {
MyFunc(param)
}
}
benchmarks.New(testFns...).Done()
Example 2: Measuring a CGO call -- we use inner-repeats because the time is very small.
const repeats = 1000
repeatedCGO := func() {
for _ = range repeats {
dummyCGO(unsafe.Pointer(plugin.api))
}
}
benchmarks.New(benchmarks.NamedFunction{"CGOCall", repeatedCGO}).
WithInnerRepeats(repeats).
Done()
func (*Options) WithColumnSize ¶
WithColumnSize sets the size (in number of runes) for each column reported by benchmark. Handy if setting WithPrettyPrint to something that uses more or less space.
Default is 10. Notice that any value smaller than 9 may mis-align the header row.
func (*Options) WithDuration ¶
WithDuration sets the benchmark duration for each function for the options and returns the updated Options instance. When running the benchmark (Options.Done) it will run each function for at least this amount time, collecting statistics.
func (*Options) WithHeader ¶ added in v0.1.1
WithHeader sets whether to display a header. Default is true.
func (*Options) WithInnerRepeats ¶
WithInnerRepeats sets the expected number of inner repeats of the functions given for the benchmark and returns the updated Options instance.
This only informs the inner repetitions inside the functions given, this library won't repeat any calls. If passing a value > 1 here, you must repeat the piece of code you want to benchmark inside the functions given to New.
This is particularly important for things that run below a few microseconds, as it mitigates the time to call the functions passed. Default is 1.
Notice that reported measures are divided by this number. That means changing this number shouldn't affect the reported mean.
func (*Options) WithPrettyPrintFn ¶
WithPrettyPrintFn sets a custom pretty-print function for formatting durations and returns the updated Options instance.
func (*Options) WithQuantiles ¶
WithQuantiles sets the quantiles for the options and returns the updated Options instance. Default is given by DefaultQuantiles ({5, 99})
func (*Options) WithTolerance ¶
WithTolerance sets the tolerance in the approximate quantiles calculations. The smaller the tolerance the larger the amount of memory used in approximating the quantiles -- which may impact the running time due to GC (??)
Default is 0.001 which is good enough for most cases.
func (*Options) WithWarmUps ¶
WithWarmUps sets the number of warm-up iterations (before starting the benchmark) and returns the updated Options instance.
Source Files