Free Hardware Performance Counters in OxCaml Runtime Events

At ICFP in Singapore last year I had lunch with Stephen Dolan and we got talking about hardware performance counters. Specifically, about how useful it would be to have them integrated into OxCaml's Runtime Events so you could ask questions like "which phases of my program have the lowest instructions-per-clock?" or "what is causing my TLB misses?". This post covers a prototype I've been working on since then.

OCaml's Runtime Events already gives you lightweight tracing of GC phases and user-defined spans in every OCaml binary. The prototype extends this to sample user-specified hardware counters at the start and end of each span, so you get performance data for free alongside the events you were already collecting. This works for built-in runtime events but also for custom events you can add around any logical section in your own code. Since spans can be continuously consumed, you can gather performance counter data on specific program or runtime phases across large numbers of long-running services. The overhead is indistinguishable from noise.

Why hardware performance counters?

Modern CPUs look like they execute instructions sequentially, but underneath they're parallel execution engines, reordering and speculatively executing instructions across deep pipelines. Multiple tiers of cache hide the fact that main memory is hundreds of cycles away and a single CPU might have dozens of cores that all need their caches kept coherent. To expose what's really going on, CPUs provide hardware performance counters. My 'Alder Lake' Intel i7-12700F has hundreds of them¹, covering everything from cache misses at each level to branch mispredictions to instructions retired.

On Linux the standard way to access these is through perf. You can run perf stat to get aggregate counter values for a whole program, or use perf record -e .. to do sampling based on counter overflow, which gives you a profile showing which lines of code are responsible for the event you're monitoring². These work well for standalone questions like finding the code causing the most L1 cache misses but they are unsuitable for continuous monitoring of long-running programs, and they can't easily correlate multiple counters with specific program phases, like asking which GC phases have the lowest instructions-per-clock.

Runtime Events already traces GC phases and user-defined spans in every binary. It also supports Custom Events, which let you emit spans and counters from your own code. If we extend it to capture hardware counters at span boundaries, we get continuous monitoring of both the runtime and user code, correlating hardware performance with logical program phases on real production traffic.

The implementation

The implementation has two sides: changes to the C runtime to set up and sample the counters, and changes to the OCaml API to expose the samples to consumers.

Runtime

We need to call perf_event_open for each counter we want to monitor on every thread and domain, and clean up when threads and domains exit. To handle this we add domain_start/stop, thread_start/stop hooks in the appropriate places in systhreads, covering both newly spawned threads and existing threads being registered.

When a Runtime Events span is started or stopped, we use rdpmc to sample the counters and append the values to the event data as it's written to the ring buffer. rdpmc takes around 20-40 cycles³, which is what makes the whole approach viable. That's very small compared to the useful work any span is doing.

The sampling is cheap but the configuration interface is less elegant. Counters to monitor are specified via the OCAML_RUNTIME_EVENTS_PERF_COUNTERS environment variable as a list of hex-encoded event codes and umasks. You need to look up the codes in the Intel Software Developer's Manual, from perfmon-events.intel.com or by running perf list and extracting the event code from perf stat -vv -e <counter name>.

For example, to count instructions, cpu-cycles, L2 cache misses (l2_request.miss) and L3 cache misses (longest_lat_cache.miss) on a Performance Core of a 12th Gen Intel Core i3-1220P: OCAML_RUNTIME_EVENTS_PERF_COUNTERS=r00c0,r003c,r3f24,r412e

OCaml API side

Consumers receive an additional local_ perf_sample array parameter on runtime_begin and runtime_end callbacks. This is an array of unboxed int64 pairs: the event code from the environment variable and the current counter value, in the same order as specified. The same parameter is available on add_user_event callbacks for custom events, and the samples are also accessible through the C API.

Taking it for a spin

The ahrefs benchmarks from Sandmark exercise code used heavily by their crawler and are known to be memory intensive, which is what we want for correlating GC behaviour with hardware performance. I had Claude port four of them (htmlStream, network, gzip, stre) to OxCaml, fixing up dependencies and using Anil's Oxmono to create a monorepo. You can find the benchmarks and the code to run them here: https://github.com/sadiqj/runtime-events-pmc-benchmarks.

GC phases by instructions-per-clock

Across all four benchmarks I monitored instructions and cpu-cycles for every runtime event span. The table below shows GC phases ordered by their total cycles, with the 5th, 50th and 95th percentile instructions-per-clock (IPC).

The p5-to-p95 spread is wide across most phases. major_sweep ranges from 0.10 to 4.32 IPC, which likely reflects variability in dense versus sparse shared heap pools. A sparse heap pool could be difficult to branch predict, leading to pipeline stalls and low IPC. The next step to investigate would be to add a runtime/custom events counter inside the sweeping code that emitted the occupancy of pools. This would tell you whether the issue is sparse or dense pools, or possibly even the number of pools swept within a major_sweep phase.

minor_local_roots and major_mark_roots have the lowest median IPC at 1.15 and 1.23 respectively. Root scanning involves chasing pointers through the stack and globals which is unfriendly to caches and branch predictors, so low IPC is expected here.

major_mark has the highest median IPC at 2.64, with a p95 of 5.03. Marking does more computation per memory access than sweeping or root scanning, so the CPU has more useful work to keep its pipelines busy. There was also significant optimisation work done by Stephen Dolan on prefetching during marking.

Since consumers get counter values at the start and end of every span, you aren't restricted to looking at aggregates. You can dump out individual traces and inspect them too:

Beyond the GC — profiling your own code

Performance counters are also emitted for Custom Events. Adding a custom event around a section of code is straightforward:

Runtime_events.User.write span_name Begin;
(* your code here *)
Runtime_events.User.write span_name End;

We added custom events around each benchmark phase in the ahrefs benchmarks. This time I monitored L2 and last-level cache (LLC) misses alongside cycles. The table below shows LLC misses and L2 misses per 1,000 instructions:

Most phases have very low LLC miss rates, well under 1.0 per 1,000 instructions. bench:complex_network_ops is the exception at 5.6 LLC misses per 1,000 instructions, roughly 10x the rate of most other phases. Its L2 miss rate of 12.7 is also the highest in the table, so if you were chasing memory access patterns this is where you'd start.

bench:nat_tables stands out too. Its LLC miss rate of 1.1 is modest but its L2 miss rate of 10.7 is among the highest, suggesting a working set that fits in LLC but not in L2.

With perf alone you would need to run separate profiling passes for each counter and then try to correlate them with specific sections of your code. Here you get it all in one continuous stream.

Like the runtime events, we can look at individual traces. Here are some operations from the gzip benchmark, showing user events alongside runtime ones. You can mouseover for more detail:

How expensive is it?

Each benchmark was run three ways: with runtime events off entirely, with runtime events on but no performance counters, and with runtime events on and four performance counters. Each configuration was run 20 times, pinned to a single P-core on my i3-1220p server, measuring wall-clock time.

The 5th and 95th percentiles are in square brackets. In every case the range crosses zero, which means the overhead is not distinguishable from noise. The numbers confirm what the cycle counts predict, that rdpmc adds around 30 cycles per counter per span. Even with four counters that's roughly 120 cycles at each span boundary, which is tiny compared to the work any GC phase is doing. The longest benchmark here runs for nearly 30 seconds.

This makes it practical to leave performance counters enabled on long-running services and collect hardware-level data continuously.

Trying it yourself

Note that this is currently x86-only and Linux-only since it relies on rdpmc and perf_event_open.

To install the prototype, create an opam switch from the branch:

opam switch create oxcaml-dev --empty
opam pin add -ny git+https://github.com/sadiqj/oxcaml#runtime_events_pmc
opam switch set-invariant -y --packages oxcaml-dev
eval $(opam env)

Build your program as normal. To specify which counters to monitor, set OCAML_RUNTIME_EVENTS_PERF_COUNTERS with the hex event codes as described above.

You'll also need a consumer to read the events. There's an example consumer in the benchmarks repo that connects to a running program given its executable directory and PID, and writes a CSV of all events with their durations and counter values.

If you try it out, let me know what you find.

Next steps

This is a prototype and isn't merged upstream to OxCaml yet. There are a few things I want to address before proposing it:

The counter configuration needs to be less clunky. Requiring hex event codes works but ideally you'd be able to specify counters by name, similar to how perf lets you write cache-misses instead of r412e.

The API could also be cleaner. Right now the performance samples are passed as an additional parameter on every callback, which makes the signature noisier even when you don't care about counters.

I'd like to explore the arm64 equivalents. arm64 has comparable facilities through the PMU.

Finally it would be worth extending Olly to support collecting the counters and pushing them through to traces, which would make this accessible without writing a custom consumer.

Longer term, the fact that the overhead is negligible means you could leave counters enabled across an entire fleet of services. Aggregating hardware counter data across hundreds of instances would let you ask questions that are impossible on a single machine, like which program phases have the worst cache behaviour on a specific microarchitecture, or whether certain workloads cause specific GC phases to degrade.