rakyll.org

Stateful jobs

Mon, 21 Sep 2026 19:25:00 -0700

Almost every job scheduler we rely on today was built on the assumption that work is stateless. In fact, in the last 15 years we mostly designed stateless schedulers, or stateful executors for niche topics. In the stateless world, you have a pool of tasks, a pool of workers, some CPU and memory constraints, maybe a priority queue and dependencies, and your goal is to find an efficient placement. If a node dies, you reschedule the task elsewhere. If a task fails, you retry it. Scheduling is fundamentally an allocation and packing problem.

Compute

Wed, 29 Jul 2026 11:27:27 -0400

I said that it’s likely that AI layers commoditize like everything else has. The harness, the planning loop, tools use patterns, the memory files, the protocols, context management, etc. I have watched ideas in this layer go from novel to table stakes in a few months. Everyone follows the same patterns. And the ones that actually work get absorbed into the model in the next training run, at which point they stop being anyone’s differentiator at all. I cannot think of a layer with a shorter shelf life. The scaffolding is not where this gets decided.

Zero sum games

Sun, 19 Jul 2026 11:27:27 -0400

I’m relatively new to ML/AL. I spent most of my career in commodity layers claiming to do boring things, and I’m still learning this field’s history, its culture, and its assumptions. So take this as an outsider’s observation. There is one assumption I keep running into that contradicts what I have seen in the past: the belief that AI is a zero-sum game. It impacts projects, people, organizational dynamics, and everything you can imagine in this domain.

Thoughts on coding agents

Fri, 03 Jul 2026 11:27:27 -0400

My thoughts on coding agents are evolving but they significantly shifted in December 2025. The models are being trained to produce trajectories that are effective for people who already know where they want to go. The value is the model reducing the distance between a clear goal and its execution. Once you see it this way, the interesting question is “how cheap is it now to get from a decision to a working artifact?”

Generics facilitators in Go

Mon, 20 Dec 2021 11:27:27 -0400

Go 1.18 is going to be released with generics support. Adding generics to Go was a multi-year effort and was a difficult one. Go type system is not a traditional type system and it was not possible just to bring an existing generics implementation from other language and be done. The current proposal was accepted after years of user research, experiments and discussions. The proposal got iterated a few times during the implementation phase. I found the final result delightful.

Shardz

Fri, 03 Dec 2021 11:27:27 -0400

Shard coordination has been one of the bigger challenges to design sharded systems especially for engineers with little experience in the subject. Companies like Facebook have been using general purpose shard coordinators, e.g. Shard Manager, and suggesting that general purpose sharding policies have been widely successful. A general purpose sharding coordinator is not a solution to all advanced sharding needs, but a starting point for average use cases. It’s a framework to think about core abstractions in sharded systems and providing a protocol that orchestrate sharding decisions. In the last few weeks, I’ve been working on an extendible general purpose shard coordinator, Shardz. In this article, I will explain the main concepts and the future work.

Spanner's SQL support

Wed, 22 Jul 2020 11:27:27 -0400

Spanner is a distributed database Google initiated a while ago to build a highly available and highly consistent database for its own workloads. Spanner was initially built to be a key/value and was in a completely different shape than it is today and it had different goals. Since the beginning, it had transactional capabilities, external consistency and was able to failover transparently. Over time, Spanner adopted a strongly typed schema and some other relational database features. In the last years, it added SQL support*. Today we are improving both the SQL dialect and the relational database features simultaneously. Sometimes there is confusion whether Spanner supports SQL or not. The short answer is yes. The long answer is this article.

Notes on the Go generics proposal

Wed, 17 Jun 2020 11:27:27 -0400

Update: The proposal draft has been revisited to use brackets instead of parenthesis. This article will be updated with the new syntax soon.

Ian Lance Taylor and Robert Griesemer have been working on a generics proposal for Go for a while. Unlike other proposals, a highly significant language change as such generics will require experimentation and comprehensive feedback before it can be finalized and submitted as a formal language change proposal.

As an initial step, they have been working on an transitioning tool so the Go users can put their ideas into test and have working understanding of the proposal. go2go is a tool available if you install Go from the source. Then you can build, run and test .go2 files with generics.

Spanner's high availability writes

Tue, 19 May 2020 11:27:27 -0400

Spanner is a relational database with 99.999% availability which is roughly 5 mins a year. Spanner is a distributed system and can span multiple machines, multiple datacenters (and even geographical regions when configured). It splits the records automatically among its replicas and provides automatic failover. Unlike traditional failover models, Spanner doesn’t failover to a secondary cluster but can elect an available read-write replica as the new leader.

In relational databases, providing both high availability and high consistency in writes is a very hard problem. Spanner’s synchronous replication, the use of dedicated networking and Paxos voting provides high availability without compromising consistency.

Inlined defers in Go

Mon, 20 Jan 2020 11:27:27 -0400

Go’s defer keyword allows us to schedule a function to run before a function returns. Multiple functions can be deferred from a function. defer is often used to cleanup resources, finish function-scoped tasks, and similar. Deferring functions are great for maintability. By deferring, for example, we reduce the risk of forgetting to close the file in the rest of the program:

func main() {
    f, err := os.Open("hello.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    // The rest of the program...
}

Deferring helps us by delaying the execution of the Close method while allowing us to type it when we have the right context. This is how deferred functions also help the readability of the source code.

NUMA

Mon, 18 Nov 2019 11:28:27 -0400

Non-uniform memory access (NUMA) is an approach to optimize memory access time in multi-processor architectures. In NUMA architectures, processors can access to the memory chips near them instead of going to the physically distant ones.

In the distant past CPUs generally ran slower than the memory. Today, CPUs are quite faster than the memory they use. This became a problem because processors constantly started to wait for data to be retrieved from the memory. As a result of such data starvation problems, for example, CPU caches became a popular addition to modern computer architecture.

The Go type system for newcomers

Wed, 27 Dec 2017 11:27:27 -0400

It is real struggle to work with a new language, especially if the type doesn’t resemble what you have previously seen. I have been there with Go and lost my interest in the language when it first came out due to the reason I was pretending it is something I already knew.

Go is considered as an object-oriented language even though it lacks type hierarchy. It has an unconventional type system. It is expected to do the things differently in this language given the traditional paradigms are not always going to help the Go users. This article contains a few gotchas.

pprof user interface

Tue, 10 Oct 2017 11:27:27 -0400

pprof now is coming with a Web UI. In order to try it out, go get the pprof tool:

$ go get github.com/google/pprof

The tool launches a web UI if -http flag is provided. For example, in order to launch the UI with an existing profile data, run the following command:

$ pprof -http=:8080 profile.out

You can focus, ignore, hide, and show with regexp. As well as clicking on the boxes and using the refining menu also works:

The future of latency profiling in Go

Tue, 18 Jul 2017 11:27:27 -0400

Note: This article contains non-finalized ideas; we may end up not implementing any of this but ideally we should do work towards the direction explained here.

Go is the language to write servers, Go is the language to write microservices. Yet, we haven’t done much in the past for latency analysis and observability/diagnostics of request/RPC performance.

GopherCon 2017 was an opportunity for me to discuss our roadmap for latency analysis. I have talked to many whose main job is to provide instrumentation solutions to the ecosystem. A few common problems have been pointed out by pretty much everyone I talked to:

Go's work-stealing scheduler

Sun, 16 Jul 2017 11:27:27 -0400

Go scheduler’s job is to distribute runnable goroutines over multiple worker OS threads that runs on one or more processors. In multi-threaded computation, two paradigms have emerged in scheduling: work sharing and work stealing.

Work-sharing: When a processor generates new threads, it attempts to migrate some of them to the other processors with the hopes of them being utilized by the idle/underutilized processors.
Work-stealing: An underutilized processor actively looks for other processor’s threads and “steal” some.

The migration of threads occurs less frequently with work stealing than with work sharing. When all processors have work to run, no threads are being migrated. And as soon as there is an idle processor, migration is considered.

Profiler labels in Go

Mon, 03 Jul 2017 11:27:27 -0400

Go 1.9 is introducing profiler labels, a way to add arbitrary key-values to the samples collected by the CPU profiler. CPU profilers collect and output hot spots where the CPU spent most time in when executing. A typical CPU profiler output is primarily reports the location of these spots as function name, source file/line, etc. By looking at the data, you can also examine which parts of the code invoked these spots. You can also filter by invokers to have more granular understanding of certain execution paths.

Custom pprof profiles

Fri, 30 Jun 2017 11:27:27 -0400

Go provides several pprof profiles out of thet box to gather profiling data from Go programs.

The builtin profiles provided by the runtime/pprof package:

profile: CPU profile determines where a program spends its time while actively consuming CPU cycles (as opposed while sleeping or waiting for I/O).
heap: Heap profile reports the currently live allocations; used to monitor current memory usage or check for memory leaks.
threadcreate: Thread creation profile reports the sections of the program that lead the creation of new OS threads.
goroutine: Goroutine profile report the stack traces of all current goroutines.
block: Block profile show where goroutines block waiting on synchronization primitives (including timer channels). Block profile is not enabled by default; use runtime.SetBlockProfileRate to enable it.
mutex: Mutex profile reports the lock contentions. When you think your CPU is not fully utilized due to a mutex contention, use this profile. Mutex profile is not enabled by default, see runtime.SetMutexProfileFraction to enable.

Additional to the builtin profiles, runtime/pprof package allows you to export your custom profiles, and instrument your code to record execution stacks that contributes to this profile.

Debugging Go core dumps

Mon, 22 May 2017 11:27:27 -0400

Debugging is highly useful to examine the execution flow and to understand the current state of a program.

A core file is a file that contains the memory dump of a running process and its process status. It is primarily used for post-mortem debugging of a program, as well as to understand a program’s state while it is still running. These two cases make debugging of core dumps a good diagnostics aid to postmortem and analyze production services.

Automatic Stackdriver Tracing for gRPC

Wed, 22 Mar 2017 11:27:27 -0400

In monolithic systems, it is relatively easy to collect diagnostic data from the building blocks of a program. All modules live within one process and share common resources to report logs and errors.

Once you are distributing your system into microservices, it becomes harder to follow a call starting from the user’s entry point until a response is served. To address this problem, Google invented Dapper to instrument and analyze its production services. Dapper-like distributed tracing systems allow you to trace a user request from the entry point to the response.

Naming tests to self-document

Fri, 03 Feb 2017 11:27:27 -0400

Go doesn’t specifically enforce you how you choose your test names. Tests are a significant contributors for the maintainability of your code. Tests not just providing correctness checking but also are useful in self documenting your code and its usage. On top of that, tests are the single best source to read about responsibilities of a type, function, etc. This is where naming tests better can help the internal documentation and therefore the maintainability of your code.

Style guideline for Go packages

Sat, 14 Jan 2017 11:27:27 -0400

Go is about naming and organization as much as everything else in the language. Well-organized Go code is easy to discover, use and read. Well-organized code is as critical as well designed APIs. The location, name, and the structure of your packages are the first elements your users see and interact with.

This document’s goal is to guide you with common good practices not to set rules. You will always need to use your own judgement to pick the most elegant solution for your specific case.

Go 1.8 development stats

Wed, 11 Jan 2017 11:27:27 -0400

Go 1.8 is going to to launched in February 2017. There is a sizable list of new features and improvements on the release notes. While these notes is the best summary to see what has happened in the last 6 months, I will try to give you some stats to give you a sense of the size of the work. I have examined all the changes merged into the tree during the Go 1.8 window and will highlight some of the interesting and significant ones.

Mutex profile

Mon, 19 Dec 2016 11:27:27 -0400

Go 1.8 introduces a new profile, the contended mutex profile, that allows you to capture a fraction of the stack traces of goroutines with contended mutexes.

You need to set the sampling fraction by calling runtime.SetMutexProfileFraction to a value above zero to enable collection.

Consider the following program:

import _ "net/http/pprof"

var mu sync.Mutex
var items = make(map[int]struct{})

runtime.SetMutexProfileFraction(5)
for i := 0; i < 1000*1000; i++ {
    go func(i int) {
        mu.Lock()
        defer mu.Unlock()

        items[i] = struct{}{}
    }(i)
}

http.ListenAndServe(":8888", nil)

Run the program, http://localhost:8888/debug/pprof/mutex will serve the mutex profile. Then, you can use go tool pprof to examine the profile.

The default GOPATH

Wed, 14 Dec 2016 11:27:27 -0400

Go 1.8 will set a default GOPATH if the GOPATH env variable is not set.

The requirement of setting a GOPATH has been a major issue for Go users who installed the Go tools for the first time and got the “you have to set a GOPATH” error in their initial experience with the tools. Explaining the GOPATH is and instructing how to set this env variable were both distracting new users away from using Go. This was especially true for users who are not necessarily developing in Go but using go get to download commands.

HTTP/2 Server Push

Sat, 10 Dec 2016 11:27:27 -0400

Go 1.8 is going to feature support for HTTP/2 server push.

HTTP/2 has many features designed to make the Web faster. One of those features is the server push, the ability to send resources before the client asks for it. This feature enables websites to push assets like JavaScript and CSS files before waiting for the web page to be loaded and asking for those resources.

net/http package will support server push by exposing a Pusher API that will be supported by HTTP/2 ResponseWriters. This interface is only implemented when HTTP/2 is available.

Deprecation notices in Go

Thu, 08 Dec 2016 11:27:27 -0400

In Go, for a long time, we didn’t have a convention to label the deprecated APIs. In the past years, there is new convention emerged to add deprecation notices to the docs.

Today, standard library uses this specific format.

As an example, Go 1.8 deprecates sql/driver.Execer and adds a deprecation notice to its godoc.

// Execer is an optional interface that may be implemented by a Conn.
//
// If a Conn does not implement Execer, the sql package's DB.Exec will
// first prepare a query, execute the statement, and then close the
// statement.
//
// Exec may return ErrSkip.
//
// Deprecated: Drivers should implement ExecerContext instead (or additionally).
type Execer interface {
	Exec(query string, args []Value) (Result, error)
}

The deprecation notice should be in the godoc, begin with string “Deprecated: " and follow with a tip for replacement.

Using contexts to avoid leaking goroutines

Tue, 25 Oct 2016 11:27:27 -0400

The context package makes it possible to manage a chain of calls within the same call path by signaling context’s Done channel.

In this article, we will examine how to use the context package to avoid leaking goroutines.

Assume, you have a function that starts a goroutine internally. Once this function is called, the caller may not be able to terminate the goroutine started by the function.

// gen is a broken generator that will leak a goroutine.
func gen() <-chan int {
	ch := make(chan int)
	go func() {
		var n int
		for {
			ch <- n
			n++
		}
	}()
	return ch
}

The generator above starts a goroutine with an infinite loop, but the caller consumes the values until n is equal to 5.

Debugging code generation in Go

Sat, 15 Oct 2016 11:27:27 -0400

Last week, I was at dotGo. I gave a very short lightning talk about inspection of code generation with the tools already available in the toolchain. This post goes through the talk for those who didn’t have the privilege to be at the conference. Slides are also available at go-talks.

Throughout this article, we will use the following program:

package main

import "fmt"

func main() {
    sum := 1 + 1
    fmt.Printf("sum: %v\n", sum)
}

Toolchain

Go build command encapsulates a bunch of underlying tools such as the compiler and the linker. But, it also provides more details about the build process optionally. -x is a flag that makes go build output what is being invoked. If you want to see what the components of the toolchain are, which sequence they are invoked and which flags being used, use -x.

Keeping Go a human-first language

Tue, 11 Oct 2016 11:27:27 -0400

Disclaimer: I forked my opinions on this one from a barely readable Twitter thread and wanted to write it down how I feel about keeping the language internals away from the users, especially from the newcomers. This is not a skill-level concern, it is a core goal of Go to provide a high-level programming language that saves users from excessive mental overhead. Note that these are personal opinions and are not written on the behalf of a group.

Go tooling essentials

Sun, 25 Sep 2016 11:27:27 -0400

New to the Go tools? Or do you want to expand your knowledge? This article is about the flags for the Go tools everyone should know.

Disclaimer: This article might be slightly biased. This is a collection of flags I personally use and flags people around me having trouble finding references for. If you have more ideas, ping me on Twitter.

$ go build -x

-x lists all the commands go build invokes.

Examples coverage

Thu, 08 Sep 2016 11:27:27 -0400

Go programming language provides many unique good features to write and maintain examples for your packages backed by the testing tools.

As an addition to the test coverage and test coverage report, go test also can provide coverage for testable examples.

Use the following commands in your package to use the -run flag to only the match the example tests and view the results in your browser.

Parallelize your table-driven tests

Tue, 06 Sep 2016 11:27:27 -0400

With Go 1.7, testing package supports sub-tests that allows you to run multiple smaller tests from a test case. Each sub test is reported independently in the go test output. More information about these recent additions can be found at Marcel van Lohuizen’s recent talk from GolangUK 2016.

These additions to Go 1.7 enabled reporting and other testing.T functionality for subtests. One of the biggest contributions of the recent changes is to be able to use these features for table-driven tests.

Using Instruments to profile Go programs

Fri, 02 Sep 2016 11:27:27 -0400

Apple has a suite of instrumentation and tracing tools for performance analysis available as a part of their Xcode tooling set. In this article, we will use Instruments to record and analyze the CPU profile of a Go program.

Instruments also provide a large set of macOS-specific tracing and profiling if you have performance issues specifically on darwin.

Some of these specific profiles are:

System trace: Collects comprehensive information about system calls, scheduling, user-kernel space transitions. (Available only on OSX.)
System usage: Gives very detailed output about I/O system activity. (Available only on iOS.)
File Activity: Monitors file and directory activity such as open/close, permission modifications, creation, copying and moving.

Instruments provide a rich and very easy to use UI to display profiles. I highly recommend it as an addition to existing profiler UIs and visualizers.

Bidirectional gRPC streaming for Go

Tue, 30 Aug 2016 11:27:27 -0400

Disclaimer: This article is not about a core Go package or tool but gRPC.

gRPC provides support for implementing streaming endpoints as well as streaming support in their clients. Bidirectional streaming is useful if you want both server and client to be able to communicate to the other side independently in a full duplex fashion.

In this article, I will dig into how to use the streaming gRPC Go client to talk to a streaming API endpoint.

About

Sat, 27 Aug 2016 11:27:27 -0400

About the author

I work at Google working on our AI infrastructure. I’ve spent many years at Google where I worked on Google Drive, the API infrastructure, Go Programming Language, and our monitoring stack.

I live in San Francisco, CA and I’m a polyglot.

Contact

Email: jbd@google.com, rakyll@rakyll.org
Github: rakyll
Twitter: rakyll

Apply transformations to Go code with eg

Sat, 27 Aug 2016 11:27:27 -0400

If you are willing to make large scale refactoring in your Go programs, automating the refactoring tasks is more desirable than manual editing. eg is a program that allows you to perform transformations based on template Go files.

To install the tool, run the following:

$ go get golang.org/x/tools/cmd/eg

eg requires a template file to look for which transformation it should apply to your source code. What’s nice is that the template file is a Go file with little annotations.

Calling Go from Swift

Sat, 03 Oct 2015 11:27:27 -0400

Note: Swift bindings are highly experimental and subject to change. This work must currently be classified as preliminary work and we will be improving APIs in the long term.

As a part of the Go Mobile, we have announced tools and packages that make language bindings from Java to Go and Objective-C to Go available. A relatively new and less documented aspect of the bindings is the availability of the Swift to Go calls. This tutorial will explain you how to make your initial Swift to Go function invocation.

Go cross compilation

Tue, 08 Sep 2015 11:27:27 -0400

Note: This article extends Dave Cheney’s Go 1.5 cross compilers post.

Cross compilers got easier with Go 1.5. You don’t have to bootstrap the standard library and toolchain as you used to do prior to 1.5.

If cgo is not required

The go tool won’t require any bootstrapping if cgo is not required. That allows you to target the following program to any GOOS/GOARCH without requiring you to do any additional work. Invoke go build.

Concrete first, abstract later

Sat, 18 Oct 2014 11:27:27 -0400

If there is one standout language feature in Go, it is interfaces. Go’s internals combine useful ideas from various type systems, inevitably piquing curiosity. I recently surveyed globally available open source Go code bases for interface declarations, and the results indicate that Go developers often pollute their codebases with interfaces no one needs or will ever use.

Don’t export any interfaces until you have to.

Interfaces are great, but interface pollution is not. You are likely coming to Go from a language (often a static one) that generates a dispatch table during compilation, where the compiler requires you to explicitly declare the interfaces a type implements. That’s how the compiler generates a vtable with pointers to all available virtual functions. If your background is in C++ or Java, you might have a habit of starting your codebase with abstract types and writing the concrete implementation as a follow-up exercise.

rakyll.org

Stateful jobs

Compute

Zero sum games

Thoughts on coding agents

Generics facilitators in Go

Shardz

Spanner's SQL support

Notes on the Go generics proposal

Spanner's high availability writes

Inlined defers in Go

NUMA

The Go type system for newcomers

pprof user interface

The future of latency profiling in Go

Go's work-stealing scheduler

Profiler labels in Go

Custom pprof profiles

Debugging Go core dumps

Automatic Stackdriver Tracing for gRPC

Naming tests to self-document

Style guideline for Go packages

Go 1.8 development stats

Mutex profile

The default GOPATH

HTTP/2 Server Push

Deprecation notices in Go

Using contexts to avoid leaking goroutines

Debugging code generation in Go

Toolchain

Keeping Go a human-first language

Go tooling essentials

$ go build -x

Examples coverage

Parallelize your table-driven tests

Using Instruments to profile Go programs

Bidirectional gRPC streaming for Go

About

About the author

Contact

Apply transformations to Go code with eg

Calling Go from Swift

Go cross compilation

If cgo is not required

Concrete first, abstract later

Archive