Profiling Go with pprof

“This endpoint feels slow” is not a finding. Performance work starts with a repeatable measurement and ends with the three lines of code that actually consume the CPU. Go ships the profiler in the toolchain; nothing needs to be added to go.mod.

Before you start

You need Go 1.24 and go tool pprof, which are the same install. benchstat compares runs and is a separate binary:

go install golang.org/x/perf/cmd/benchstat@latest

Make sure $(go env GOPATH)/bin is on PATH, otherwise the shell cannot find benchstat.

Start with a benchmark you can rerun:

package parse

import (
	"strings"
	"testing"
)

func Parse(line string) (string, string) {
	key, value, ok := strings.Cut(line, "=")
	if !ok {
		return "", ""
	}
	return strings.TrimSpace(key), strings.TrimSpace(value)
}

func BenchmarkParse(b *testing.B) {
	line := "  user_id = 42  "
	b.ReportAllocs()

	for b.Loop() {
		Parse(line)
	}
}

b.Loop() (new in Go 1.24) keeps the call from being optimised away and handles timer start and stop for you. The older for i := 0; i < b.N; i++ form still works but needs a package-level sink to survive inlining.

go test -bench BenchmarkParse -benchmem ./parse

-benchmem adds B/op and allocs/op. Allocation problems usually show up there before they show up in a profile.

Capturing a profile

Benchmarks can emit profiles without touching application code:

go test -bench BenchmarkParse -benchmem \
  -cpuprofile cpu.out -memprofile mem.out ./parse

Two files come out: cpu.out is sampled CPU time, mem.out is a sampled heap. A few rules:

• The default sampling rate is enough to find hotspots. Do not set -memprofilerate=1 “for accuracy”; it slows the benchmark by an order of magnitude.
• A heap profile is a snapshot of the heap at that moment. Call runtime.GC() before capturing if you care about stable inuse_space numbers.
• One run represents one run. Make sure -benchtime is long enough; the 1s default usually is.

Reading pprof

go tool pprof cpu.out

Four commands cover most investigations:

(pprof) top
(pprof) top -cum
(pprof) list Parse
(pprof) web

• top sorts by flat time (the function itself, excluding callees) and points straight at the hot code.
• top -cum sorts by cumulative time and shows which call path is expensive overall.
• list Parse prints per-line cost, as long as the binary still has symbols.
• web renders a call graph and needs Graphviz’s dot.

Without Graphviz, use the built-in web UI:

go tool pprof -http=:8080 cpu.out

It serves Graph, Flame Graph and Source views on localhost:8080. In a flame graph the horizontal axis is merged stack width — wider means more samples — and the vertical axis is call depth. Look for frames that are wide and flat: those are functions spending time in themselves rather than in callees.

The common misread: a wide frame near the top is not automatically slow, it may simply be called very often. Cross-check with the flat ordering from top.

Memory and allocations

go tool pprof -sample_index=alloc_space mem.out

The default view is inuse_space (bytes held by live objects). The other indexes answer different questions:

-sample_index	Meaning
inuse_space	bytes held by live objects at capture time
inuse_objects	number of live objects
alloc_space	bytes allocated cumulatively, including freed
alloc_objects	objects allocated cumulatively

A heap that keeps growing calls for inuse_space. High GC CPU with a flat heap calls for alloc_space and allocs/op.

To compare two captures, diff them instead of eyeballing two top outputs:

go tool pprof -diff_base before.out after.out

Positive values mean the change added cost, negative means it removed cost.

Profiling a live server

net/http/pprof registers handlers on http.DefaultServeMux through a blank import:

package main

import (
	"log"
	"net/http"
	_ "net/http/pprof"
)

func main() {
	go func() {
		// Loopback only. Never 0.0.0.0.
		log.Println(http.ListenAndServe("127.0.0.1:6060", nil))
	}()

	// Application server
	_ = http.ListenAndServe(":8080", nil)
}

The endpoints worth remembering:

curl 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30' -o cpu.out
curl 'http://127.0.0.1:6060/debug/pprof/heap' -o heap.out
curl 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1'

seconds=30 is the CPU sampling window and does add overhead, so keep it short during peak traffic. goroutine?debug=1 dumps every stack and finds goroutine leaks faster than any profile.

Safety rules:

• Bind to 127.0.0.1 and reach it through an SSH tunnel or a private network. Never mount it on the public mux.
• If it must listen on a separate port, restrict it with a firewall allowlist.
• Binaries built with -trimpath -s -w may not resolve source lines in list; keep a symbol-rich build for debugging.

Block and mutex profiles are off by default and must be enabled in code:

runtime.SetBlockProfileRate(1)
runtime.SetMutexProfileFraction(1)

A rate of 1 records every event and is expensive. Turn it on while investigating, then turn it off. The endpoints are /debug/pprof/block and /debug/pprof/mutex.

Confirming a change with benchstat

A single benchmark run is mostly noise. Ten runs plus a comparison is the minimum:

go test -run '^$' -bench . -count=10 -benchmem ./parse > old.txt
# change the code
go test -run '^$' -bench . -count=10 -benchmem ./parse > new.txt
benchstat old.txt new.txt

-run '^$' skips ordinary tests and runs benchmarks only. benchstat reports medians, percentage change and a geomean, and marks whether the difference is statistically meaningful.

On Windows, watch the redirection encoding: PowerShell 5 writes UTF-16 by default, which benchstat cannot parse. Use cmd /c "go test ... > old.txt" or | Out-File -Encoding utf8 old.txt.

Decide the bar before measuring. A 20% improvement is worth merging; a 2% “win” inside the noise band is not.

Common mistakes

1. Benchmark results that are never used

The compiler removes calls without side effects and the benchmark measures nothing. Use b.Loop(), or assign to a package-level var Sink any.

2. Concluding from one run

CPU frequency, scheduling and neighbouring processes all move the number. -count=10 plus benchstat is the floor.

3. A -benchtime that is too short

Too few samples, and the profile is mostly noise. Start at the default 1s and raise it if the hotspot stays unclear.

4. Leaving CPU profiling on in production

Sampling costs CPU. Thirty seconds is a common ceiling; avoid peak hours.

5. Exposing /debug/pprof to the internet

It is a remote diagnostic backdoor that can reveal heap contents and call stacks. Loopback only.

6. Watching CPU and ignoring allocations

GC pressure is often the real latency source. Read -benchmem, alloc_space and GOGC together.

7. Comparing absolute numbers on shared CI runners

CPU quotas move between runs. Compare before/after on the same machine, or track trends only.

8. Optimising without a measurement

Intuition usually moves the cost somewhere else. Profile, change, then verify with benchstat.

Summary

• Get a repeatable benchmark first (b.Loop() plus -benchmem), then optimise.
• -cpuprofile and -memprofile produce profiles from tests; live servers use net/http/pprof bound to loopback.
• top finds flat hotspots, top -cum shows call paths, list maps cost to lines, the flame graph shows shape.
• Use -sample_index for memory: inuse_space for leaks, alloc_space for GC pressure.
• Let benchstat decide whether a change worked, with -count=10 as the starting point.

Next: profile the HTTP service from the practice chapter under real load, then diff the heap before and after the fix.