Quantum programs should be benchmarked as experiments, not merely checked for successful execution. This guide provides a repeatable worksheet for comparing simulators, cloud hardware, optimization passes, and execution costs using circuit metrics, runtime, result quality, queue time, shot count, and reproducibility.
Overview
A quantum program can run successfully and still be unsuitable for production or further research. A simulator may return an ideal distribution quickly, while hardware introduces noise, queue delays, device-specific constraints, and usage charges. Conversely, a circuit that appears expensive on hardware may be the best option if it produces a useful result with fewer repetitions.
Quantum program benchmarking is the process of measuring these tradeoffs consistently. The goal is not to identify one universally “best” backend. It is to determine which implementation meets the needs of a particular task, such as debugging, algorithm development, hardware validation, or a hybrid quantum-classical workflow.
Use the same logical circuit, input data, measurement basis, shot budget, and success criteria whenever you compare alternatives. Record both technical and operational measures. A useful benchmark answers five questions:
- How large and complex is the circuit?
- How long does compilation and execution take?
- How many shots or repeated circuit evaluations are required?
- How close is the result to a defined reference?
- What does the run cost, and can another developer reproduce it?
For background on provider differences, see the guide to quantum cloud platforms. For implementation-level checks, pair benchmarking with the quantum circuit debugging guide.
How to estimate
Start by defining a benchmark record for every run. A spreadsheet, experiment-tracking system, or version-controlled JSON file is sufficient. The record should connect the source code and circuit version to the backend, compiler settings, inputs, and observed results.
Use the following calculation sequence:
- Freeze the experiment. Record the circuit, number of qubits, input data, measurement operations, SDK version, transpilation or compilation settings, and random seeds where applicable.
- Measure circuit structure. Capture total gate count, counts by gate type, circuit depth, two-qubit gate count, and the number of qubits. If the backend requires a native gate set, record the post-compilation values as well as the original logical values.
- Separate time categories. Track local preparation time, compilation time, queue or waiting time, execution time, and result-processing time. Combining them into one number hides useful bottlenecks.
- Set the shot budget. Record the number of shots per circuit and the number of circuit variants or parameter values. A simple workload estimate is
total shots = shots per circuit × number of circuit executions. - Define result quality. Compare the output with a reference distribution, known solution, target expectation value, or application-specific threshold. Do not describe a result as “accurate” without naming the comparison method.
- Calculate cost using current inputs. Apply the provider’s current pricing rules, account for any execution, task, or storage components that apply, and label unknown fields rather than filling them with assumptions.
- Repeat enough to assess variation. A single run can reveal a failure, but repeated runs are more useful for estimating variability, queue behavior, and stability.
For a practical comparison, use a table with one row per backend or circuit version:
| Category | Metric to record | Why it matters |
|---|---|---|
| Circuit | Qubits, depth, total gates, two-qubit gates | Shows structural workload and hardware sensitivity |
| Compilation | Compiler time, final depth, native gate count | Reveals transformation overhead |
| Execution | Shots, circuit repetitions, device runtime | Shows the direct experimental workload |
| Operations | Queue time, total elapsed time | Measures practical responsiveness |
| Quality | Error, fidelity proxy, success rate, or target value | Connects output to the use case |
| Cost | Recorded usage and estimated total charge | Supports planning and provider comparison |
| Reproducibility | Versions, settings, identifiers, and saved results | Allows later verification |
Inputs and assumptions
Benchmark conclusions are only as reliable as their inputs. Before comparing a quantum simulator with real hardware, document what each environment is expected to represent.
Simulator inputs
Record the simulator type, number of qubits, precision or noise model, available memory, parallelism, and whether compilation is included in the reported time. An ideal statevector simulation is useful as a reference for small circuits, but it does not by itself predict hardware output. A noisy simulation can provide a controlled comparison when its noise assumptions are explicitly recorded.
Hardware inputs
Record the selected device or backend, supported qubits, connectivity constraints, compilation settings, calibration context when exposed by the platform, shot count, and any mitigation or error-suppression method. Hardware metrics can change, so preserve the metadata returned with the run instead of relying only on a later dashboard view.
Quality assumptions
Choose a metric that matches the algorithm. Possible measures include distance between probability distributions, expectation-value error, optimization objective value, classification accuracy, or the percentage of runs meeting a defined condition. A circuit with a lower raw error is not automatically better if it takes substantially longer or costs more than an alternative that meets the application threshold.
Cost assumptions
Cost estimates should use a clearly dated pricing snapshot. Separate known charges from unknown or account-dependent components. A basic planning formula is:
estimated total cost = execution cost + compilation or task cost + storage or related usage cost
Use “not included” where a component cannot be verified. This is more useful than presenting a false level of precision. For enterprise planning, also record the engineering time needed to prepare circuits, inspect results, rerun failed jobs, and maintain integrations.
If error mitigation is part of the workflow, count every additional circuit evaluation and post-processing step. The quantum error mitigation guide can help frame the quality-versus-workload tradeoff.
Worked examples
Example 1: Comparing two circuit versions on a simulator
Suppose version A uses 12 qubits, has a compiled depth of 80, and requires 100 parameter evaluations. Version B uses the same qubit count, reduces compiled depth to 52, and requires 120 evaluations because the optimization loop has changed.
Version B is not automatically superior. Its depth reduction may lower per-circuit runtime, but the additional evaluations increase total workload. Estimate both sides using:
total evaluation work = per-evaluation runtime × number of evaluations
Then record measured runtime rather than inferring it from depth alone. Depth is a useful structural indicator, but simulator performance also depends on qubit count, simulation method, hardware resources, and implementation details.
Example 2: Estimating a hardware workload
Assume a variational workflow evaluates 40 parameterized circuits, with 2,000 shots per circuit, across 15 optimization iterations. The shot estimate is:
40 × 2,000 × 15 = 1,200,000 shots
This figure describes the planned measurement volume, not necessarily the provider’s billing unit. Record the number of submitted tasks as well, because a platform may organize batches and circuit executions differently. If an error-mitigation method requires three additional circuit variants for each evaluation, the planned circuit workload may increase substantially even though the logical algorithm is unchanged.
Example 3: Choosing between speed and result quality
Imagine two backend runs for the same circuit. Backend A completes sooner and has a lower recorded usage estimate, but its output is farther from the reference distribution. Backend B takes longer and uses more shots, yet meets the application’s defined error threshold.
The correct decision depends on the objective. For rapid debugging, Backend A may be adequate. For a reported experiment or a workflow requiring a quality threshold, Backend B may be the better choice. Put the decision rule in the benchmark record so that a future developer can understand why one result was accepted.
When to recalculate
Recalculate a benchmark whenever an input that can affect the conclusion changes. This includes provider pricing, backend availability, compiler or SDK versions, transpilation settings, simulator capabilities, noise-model assumptions, queue conditions, shot count, circuit depth, and error-mitigation strategy.
Re-run the benchmark after an optimization pass changes gate counts or connectivity, after moving from simulation to hardware, or after changing the objective function in a hybrid quantum-classical workflow. A benchmark should also be revisited when the application’s acceptance threshold changes.
For a repeatable developer workflow, keep a small benchmark suite under version control. Include a smoke test for correctness, a representative circuit for performance, and a larger workload for cost and scaling estimates. Store raw results, not only averages, and attach timestamps and environment metadata.
Before sharing a conclusion, use this checklist:
- Is the circuit version and compilation configuration recorded?
- Are simulator and hardware results being compared on the same logical task?
- Are queue, execution, and post-processing times separated?
- Is result quality measured against an explicit reference or threshold?
- Are shots, repetitions, mitigation overhead, and failed runs included?
- Are pricing fields dated and distinguished from estimates?
- Can another developer reproduce the run from the saved inputs?
Use this worksheet whenever you introduce a new backend, SDK, optimization pass, or circuit family. For broader tooling decisions, compare the workflow implications in the Qiskit, Cirq, and PennyLane SDK comparison, then benchmark the specific implementation rather than relying on general platform assumptions.