Home · Articles


Go for Cloud

February 6, 2020

Over the last decade, Go became the default language for a large share of cloud infrastructure: containers, orchestrators, service meshes, proxies, and serverless backends. A lot of that adoption has little to do with syntax and everything to do with how Go binaries behave once they leave your laptop. Small static executables, fast startup, predictable memory behavior, and built-in diagnostics map directly onto the realities of running ephemeral workloads in the cloud.

Still, taking a Go service from go run to a production cloud environment comes with a handful of operational habits and subtle gotchas that are easy to miss at first. Below are the practices I keep coming back to when building, deploying, and operating Go services in the cloud.

Build small binaries. Go builds small binaries, which makes it well suited for containerized and serverless environments where image pull time directly affects cold starts and rollout speed. A typical Go service binary with all its runtime dependencies can be as small as 20-25 MB. If image size is tight, you can trim binaries further with -ldflags="-s -w" to strip symbol and DWARF tables, though keep an unstripped copy in your build artifacts if you rely on symbolized profiles or postmortem debugging.

Runtime initialization is fast. Go’s runtime initialization is fast. If you are writing autoscaling servers in Go, cold start is rarely bottlenecked by Go’s runtime initialization. Go libraries and frameworks also tend to avoid heavy reflection or classpath scanning at startup compared to ecosystems like the JVM, so the entire process reaches ready state quickly.

Build static binaries. Go programs compile into a single static binary, which simplifies delivery: CI/CD pipelines can produce one self-contained artifact and copy it straight into a minimal container or remote host. Watch out for CGO, though. When CGO is enabled, the binary dynamically links against the host’s libc, which can fail at startup in scratch or Alpine containers. Disable CGO explicitly when you want a truly static Linux binary:

$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o server .

Cross compile to 64-bit Linux. The Go compiler supports cross compilation out of the box. As long as you do not have CGO dependencies, you can build for your production OS and architecture from any developer machine:

$ GOOS=linux GOARCH=amd64 go build

Don’t ship your toolchain. You do not need the Go toolchain installed in production to run a Go binary. Avoid shipping production containers based on the full golang image. Instead, use a multi-stage container build to compile the binary and copy only the final executable (and CA certificates if you make outbound TLS calls) into a minimal base image such as distroless or scratch:

FROM golang:1.13 AS builder
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /out/server .

FROM gcr.io/distroless/static-debian10
COPY --from=builder /out/server /server
ENTRYPOINT ["/server"]

Rebuild and redeploy with Go releases. Go only supports the last two major versions. Just because Go runtime is compiled in the body, with each Go release, rebuild and redeploy your production services.

At Google, we use the release candidates to build production services as soon as there is an RC version. You can use the RC version for production services, or at least push to canary with the RC version. If you see an unexpected behavior, immediately file an issue.

The go tool can print the Go version used to build a binary:

$ go version <binary>
<binary>: go1.13.5

You can additionally use tools like gops to list and report the Go versions of the binaries currently running on your system.

Embed commit versions into binaries. Embed the revision numbers when you are building a Go binary. You can also embed the build constraint and other options used when building. debug.BuildInfo also provides information about the module as well as the dependencies. Alternatively, go command can report module information and the dependencies:

$ go version -m dlv
dlv: go1.13.5
	path	github.com/go-delve/delve/cmd/dlv
	mod	github.com/go-delve/delve	v1.3.2	h1:K8VjV+Q2YnBYlPq0ctjrvc9h7h03wXszlszzfGW5Tog=
	dep	github.com/cosiner/argv	v0.0.0-20170225145430-13bacc38a0a5	h1:rIXlvz2IWiupMFlC45cZCXZFvKX/ExBcSLrDy2G0Lp8=
	dep	github.com/mattn/go-isatty	v0.0.3	h1:ns/ykhmWi7G9O+8a448SecJU3nSMBXJfqQkl0upE1jI=
	dep	github.com/peterh/liner	v0.0.0-20170317030525-88609521dc4b	h1:8uaXtUkxiy+T/zdLWuxa/PG4so0TPZDZfafFNNSaptE=
	dep	github.com/sirupsen/logrus	v0.0.0-20180523074243-ea8897e79973	h1:3AJZYTzw3gm3TNTt30x0CCKD7GOn2sdd50Hn35fQkGY=
	dep	github.com/spf13/cobra	v0.0.0-20170417170307-b6cb39589372	h1:eRfW1vRS4th8IX2iQeyqQ8cOUNOySvAYJ0IUvTXGoYA=
	dep	github.com/spf13/pflag	v0.0.0-20170417173400-9e4c21054fa1	h1:7bozMfSdo41n2NOc0GsVTTVUiA+Ncaj6pXNpm4UHKys=
	dep	go.starlark.net	v0.0.0-20190702223751-32f345186213	h1:lkYv5AKwvvduv5XWP6szk/bvvgO6aDeUujhZQXIFTes=
	dep	golang.org/x/arch	v0.0.0-20171004143515-077ac972c2e4	h1:TP7YcWHbnFq4v8/3wM2JwgM0SRRtsYJ7Z6Oj0arz2bs=
	dep	golang.org/x/crypto	v0.0.0-20180614174826-fd5f17ee7299	h1:zxP+xTjjk4kD+M5IFPweL7/4851FUhYkzbDqbzkN1JE=
	dep	golang.org/x/sys	v0.0.0-20190626221950-04f50cda93cb	h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k=
	dep	gopkg.in/yaml.v2	v2.2.1	h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=

FaaS is Go binary as a service. Function-as-a-service products such as Google Cloud Functions or AWS Lambda serves Go functions. But in fact, they are building a user function into a binary and serve the binary. This means you have to organize and build packages acknowledging this fact. Because the final binary is not forked for every incoming request but is being reused:

Gracefully reject incoming requests. When an orchestrator scales down replicas or rolls out a new version, it sends SIGTERM before forcefully killing the process. Listen for termination signals and call http.Server’s Shutdown with a bounded context so your server stops accepting new connections while giving in-flight requests time to finish:

srv := &http.Server{Addr: ":8080", Handler: mux}
go srv.ListenAndServe()

quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
    log.Printf("graceful shutdown failed: %v", err)
}

Report the essential metrics. The Go runtime exposes a set of diagnostic metrics that are cheap to collect and catch common production issues early. Export runtime.NumGoroutine (to spot goroutine leaks), runtime.NumThreads, runtime.NumCGOCalls, and runtime.ReadMemStats (for live heap size, allocation rates, and GC pause times) to your monitoring system. See instrumentation guides such as Prometheus’ Go library as a reference on what to export.

Print scheduling and GC events. Go can print scheduler and garbage collector diagnostics directly to standard error via the GODEBUG environment variable, which is handy when debugging latency spikes or CPU starvation in containers.

The following command prints each GC cycle (heap size before/after collection and wall-clock pause times) along with scheduler snapshots (idle processors and runnable queue lengths) every 5000 ms:

$ GODEBUG=gctrace=1,schedtrace=5000 <binary>

Propagate the incoming context. Go allows propagating the context in the process via context.Context. You can also signal cancellation or timeout decisions to other goroutines using context. You can use context to propagate values such as trace/request IDs or other metadata relevant in the critical path. You can log with context key/values where it applies. If you have an incoming request context, keep propagating it. For example:

http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    // use r.Context() for the calls made here.
})

Continuously profile in production. Go uses pprof which is a lightweight profiling collection mechanism. It only adds a single-digit percentage overhead to the execution when enabled. You can utilize this strength by collecting profiles from production systems and understanding the fleet-wide hotspots to optimize. See Continuous Profiling of Go programs for more insights and a reference implementation of a continuous profiling product.

pprof can symbolize the profiling data by incorporating the binary. If you are collecting profiles from production, you’d like to store profiling data with symbols.
Even though there is no good standard library function for this task, there is an existing reference that can be adopted.

Dump debuggable postmortems. Go allows post-mortem debugging. When running Go in production, core dumps allow you to retrospectively investigate why binaries crash. If you have Go programs constantly crashing, you can retrieve their core dumps and understand why they crashed and which state they were in. You can also utilize core dumps to debug in production by taking a snapshot (a core dump) and using your debugger. See core dumps for more.


Home · Articles