On Benchmarking
Why a throughput number is not an architecture decision.
A benchmark is useful only when you can explain what it measured, why it stopped scaling, and whether it resembles the workload you intend to run.
Your team tests a new lakehouse feature and gets an impressive result: 50,000 operations per second.
Do not choose an architecture yet. That number tells you only that this system completed 50,000 operations per second under some set of conditions. The decision depends on whether those conditions are the ones that matter.
Was the data already warm in cache? Did the client generate requests at the intended rate when the system slowed down? Were retries, commit conflicts, and errors counted? Did the test exercise a real data layout—or a convenient synthetic one?
A benchmark score is an observation, not a conclusion.
In his writing on active benchmarking, Brendan Gregg describes the fire-and-forget approach as passive benchmarking: run a test, collect its final numbers, and stop there. The number may be accurate for the narrow thing measured while still being a poor basis for an architecture decision.
The better approach is active benchmarking: use the workload to put the system under controlled pressure, then explain its behavior while the test is running. A score comes at the end of that investigation, not in place of it.
A benchmark needs a contract before it needs a cluster
Most misleading benchmarks are not fraudulent. They are underspecified.
Before starting a run, write a short benchmark contract:
The decision: What choice will this test inform—engine selection, table format, cluster size, or cost target?
The workload: Read/write mix, request sizes, partition distribution, dataset size, and concurrency pattern.
The system boundary: Which client, network, catalog, object store, metadata service, and compute layers are included?
The arrival model: Are requests issued at a fixed schedule, or does each client wait for a response before sending the next one?
The state: Is this a cold-cache run, a warm-cache run, a compaction-heavy period, or a steady-state stream?
The outcome: Throughput, error rate, cost, and latency percentiles—not one of them in isolation.
These conditions determine what the score means. Without them, a throughput chart is a screenshot, not a decision document.
For example, a data lake read test with a hot operating-system cache may correctly report excellent read performance. But it has measured the cache path, not object-store retrieval. That can still be a useful result—provided the chart says so.
Do not let your load generator hide the worst minutes
The most dangerous latency results often look reassuring.
Consider a closed-loop client: each worker sends a request, waits for a response, then sends the next request. When the service pauses for a lock, garbage collection, or an overloaded metadata service, the workers wait. During the pause, the client issues fewer requests, so it fails to record the queueing delay of work that would have arrived at the intended rate.
This is a coordinated omission. It can make a system appear more responsive precisely when it cannot keep up with the intended arrival rate.
The fix is not merely “use a better tool.” Make the arrival behavior explicit. If the real workload should offer 10,000 requests per second regardless of individual response times, use a load generator that can schedule that offered load independently of completed responses, or use a measurement method designed to correct for coordinated omission. Report both offered load and completed throughput, plus the percentile distribution and error rate.
Average latency still has value for capacity planning. It is not enough for a user-facing or pipeline SLO.
The fan-out problem, explored in The Tail at Scale, is why. If a request depends on 100 independent backend operations and each has a 1% chance of being slow, the chance that at least one is slow is:
1 − 0.99¹⁰⁰ ≈ 63%
The assumption of independence matters, and real systems often have correlated failures that are worse. Either way, a smooth mean can hide an unacceptable tail. Report p50, p95, and p99; add p99.9 only when the run has enough samples to make it meaningful. Always show timeouts and retries beside latency.
Use simple models as guardrails, not verdicts.
Performance models expose assumptions. They do not prove a system is correct, and they do not replace measurement.
Little’s Law exposes missing in-flight work
For a stable system:
L = λ × W
Here, L is the average number of in-flight requests, λ is effective throughput, and W is average time in the system.
If a service sustains 10,000 requests per second at 50 ms end-to-end latency, it needs roughly 500 requests in flight on average:
10,000 × 0.05 = 500
That means 500 in-flight operations, not necessarily 500 operating-system threads. An asynchronous client may achieve this with fewer threads and many connections; a synchronous client may need far more workers. The useful check is consistency: do observed throughput, latency, and in-flight work agree?
When they do not, investigate the arrival model, the measurement boundary, caching, batching, and client-side backpressure before trusting the result.
Amdahl’s Law makes serial work visible
For a fixed amount of work, Amdahl’s Law bounds the speedup available from parallelism:
S(N) = 1 ÷ [(1 − P) + (P ÷ N)]
If 5% of a workload is genuinely serial, the theoretical maximum speedup is 20x, even with unlimited parallel resources. In data systems, that serial portion may be catalog access, a commit path, a coordination barrier, or a single skewed partition.
Use this as a hypothesis, not a diagnosis. A plateau below the predicted ceiling may point to network, storage, skew, or client limitations. A plateau near it tells you where to investigate next.
The Universal Scalability Law turns a curve into questions
Neil Gunther’s Universal Scalability Law models the fact that added workers can introduce both contention and coordination overhead:
C(N) = N ÷ [1 + α(N − 1) + βN(N − 1)]
Here, C(N) is normalized capacity at scale factor N; α represents contention and β represents coordination or coherency cost.
The model is valuable because it asks the right question when a cluster stops scaling: what shared resource or coordination path grows with the cluster? Fit it to repeated measurements across enough scale points to see a trend. Do not expect three data points to predict an exact collapse point.
Test the states that production will actually visit
Uniform ramps to a clean peak are easy to graph. Production is rarely so polite.
Data platforms change behavior as caches warm, checkpoints accumulate, compactions begin, partitions skew, or write conflicts rise. A benchmark that measures only a fresh cluster at steady load can miss the states that create operational pain.
Add scenarios that deliberately exercise those transitions:
Compare cold-cache and warm-cache reads.
Run long enough to include compaction, checkpointing, or metadata maintenance.
Vary key and partition skew instead of using uniformly distributed data.
For optimistic concurrency control, generate both non-overlapping writes and writes that intentionally contend for the same conflict domain.
Step load up, hold it, then step it down. Measure whether latency and queues recover promptly.
Inject a realistic dependency slowdown or partial failure when the decision warrants it.
These experiments make feedback loops visible: a transient slowdown triggers retries, queue growth, timeouts, and more load until the system remains unhealthy after the original trigger has passed. Metastable failure modes are difficult to predict and easy to miss, which is why recovery behavior belongs in the test plan.
Treat the benchmark as an investigation
Active benchmarking means observing the full path while the workload is running. Start with a hypothesis—” the catalog is the limiter,” “small files are saturating metadata operations,” or “the client cannot maintain offered load”—then gather evidence from both the client and every layer in the system boundary.
Watch for:
client request rate, concurrency, connection pools, retries, and queue depth;
CPU utilization, run queues, context switches, memory pressure, and garbage collection;
network throughput, retransmissions, and connection saturation;
storage, object-store, and metadata-service latency and throttling;
engine-level task skew, shuffle behavior, commit latency, and failed operations.
Profilers and tracing tools, including eBPF-based tools where available, are useful because they turn a flat throughput line into an explanation. A flame graph may reveal CPU time in serialization; queue metrics may reveal a saturated metadata service; traces may show retries amplifying a minor slowdown.
The benchmark should answer more than “how fast?” It should answer:
What limited this workload at this scale, under these conditions—and what would need to change for it to go faster?
The benchmark review that earns trust
Before presenting a result, confirm that readers can find the answer to these questions:
What workload was tested, and how closely does it match production?
How was load offered, and can the client sustain that offer under pressure?
What cache state, data layout, configuration, and software versions were used?
What happened to throughput, errors, retries, and tail latency as load increased?
Where did the system bottleneck, and what evidence supports that conclusion?
Did the system recover when the load or fault was removed?
If those answers are absent, the throughput chart is not a decision document. It is a starting point for the next experiment.
Data engineering teams build systems that businesses depend on when the workload becomes messy: end-of-month spikes, backfills, hot partitions, expensive retries, and a storage service having a bad day. Benchmark for that reality.
Run the load. Watch the system. Explain the result. Then decide.
References and further reading
Brendan Gregg, Active Benchmarking
Gil Tene, HdrHistogram documentation on coordinated-omission correction
Jeffrey Dean and Luiz André Barroso, The Tail at Scale
John D. C. Little, A Proof for the Queuing Formula: L = λW
Gene M. Amdahl, Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities
Neil J. Gunther, A General Theory of Computational Scalability Based on Rational Functions
Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, and Timothy Zhu, Metastable Failures in Distributed Systems