A quantum machine learning model is not competitive simply because it produces a strong score on one run. A useful comparison controls the data split, preprocessing, model budget, randomness, evaluation metrics, and execution environment so that a classical baseline and a quantum model answer the same question under comparable conditions. This guide presents a repeatable benchmarking workflow, a simple cost-and-outcome estimate, worked examples, and a results template you can reuse when evaluating a quantum neural network, reproducing a paper, or reviewing a new QML experiment.
Overview
The central question in a quantum machine learning benchmark is not “Is the quantum model accurate?” It is “What does the quantum model achieve compared with a credible classical alternative when both receive the same information and comparable experimental care?”
A defensible benchmark should compare at least:
- Predictive quality: performance on data that was not used for training or model selection.
- Reliability: variation across random seeds, folds, circuit initializations, and measurement shots.
- Resource use: training time, inference time, circuit evaluations, simulator workload, shots, and any hardware usage.
- Operational constraints: qubit count, circuit depth, connectivity assumptions, batching limits, queue time, and noise conditions.
- Statistical confidence: whether the observed difference is larger than ordinary experimental variation.
Quantum machine learning is often implemented as hybrid quantum-classical computing: a classical optimizer updates parameters while a simulator or quantum processor evaluates a circuit. Consequently, reporting only the number of trainable parameters or only the final accuracy gives an incomplete picture. A model may have a small circuit but require many circuit evaluations, or it may match a classical score while consuming substantially more execution time.
Before beginning, define the claim you want to test. Examples include “the quantum model improves balanced accuracy,” “the model reaches a target score with fewer labeled examples,” or “the circuit provides a useful feature representation under a fixed evaluation budget.” Each claim requires a different benchmark design.
How to estimate
Use a benchmark worksheet with one row per model and one column per controlled variable. The following workflow keeps the comparison reproducible.
1. Freeze the dataset protocol
Specify the dataset version, target definition, feature columns, missing-value treatment, normalization, encoding, and class-balance procedure. Split the data into training, validation, and test sets before tuning the models. If the dataset is small, use stratified cross-validation and reserve a final test set when feasible.
Preprocessing must be fitted on the training portion only. For example, calculate scaling parameters from the training data and apply them unchanged to validation and test data. The same transformed features should feed both the classical and quantum models. If a quantum model requires dimensionality reduction to fit a circuit, apply an equivalent, documented restriction to the baseline rather than giving the baseline access to extra information.
2. Select more than one classical baseline
Start with a simple reference model, such as a regularized linear classifier or a shallow tree-based model. Then add a stronger baseline appropriate to the data, such as a kernel method, ensemble, or small neural network. The point is not to make the classical model unnecessarily weak or expensive; it is to establish a credible range of performance.
Match the training information and tuning budget as closely as possible. Record the search space, number of trials, stopping rules, and random seeds. A highly tuned quantum model compared with an untuned classical model is not an informative quantum ML benchmark.
3. Define the quantum model and execution budget
Record the feature map, data-encoding method, variational ansatz, number of qubits, circuit depth, entangling pattern, parameter count, optimizer, learning-rate schedule, batch size, and maximum optimization steps. Also record whether the experiment uses exact expectation values, finite-shot simulation, or hardware measurements.
For a first estimate of execution workload, use:
Total circuit shots = circuit evaluations × shots per evaluation.
When a training step evaluates several parameter-shift gradients, multiply the number of objective evaluations by the number of gradient evaluations actually performed. If there are E evaluations per step, S shots per evaluation, and T optimization steps, a basic workload estimate is:
Estimated shots = E × S × T.
This is an estimate, not a universal cost model. Gradient techniques, grouping of observables, caching, batching, compilation, retries, and hardware scheduling can change the actual workload. Report measured wall-clock time separately from this theoretical count.
4. Repeat the experiment
Run each configuration across multiple predefined seeds or cross-validation folds. Store every test prediction, not just the best score. Report the mean and spread, such as standard deviation or an interval chosen for your evaluation design. Selecting the best seed after looking at test results turns the test set into a tuning set and makes the comparison optimistic.
Inputs and assumptions
A benchmark is only as useful as its assumptions. Put these inputs in version-controlled configuration files or an experiment tracker.
| Input | What to record | Why it matters |
|---|---|---|
| Dataset | Version, sample count, feature count, label balance, split seed | Prevents accidental changes in the task |
| Preprocessing | Scaling, feature selection, reduction, encoding | Ensures both model families receive comparable information |
| Baseline | Algorithm, parameters, tuning budget, software version | Defines what “competitive” means |
| Quantum circuit | Qubits, depth, gates, ansatz, encoding, trainable parameters | Links results to quantum resources |
| Measurement | Shots, observable grouping, noise model or hardware target | Quantifies sampling and execution effects |
| Compute | CPU/GPU, simulator mode, hardware queue time, elapsed time | Separates model quality from infrastructure cost |
| Statistics | Seeds, folds, confidence method, primary metric | Shows whether differences are stable |
Choose metrics based on the decision, not convenience. For balanced classification, accuracy can be useful, but it should be accompanied by precision, recall, F1 score, or area under a suitable curve when class errors have different consequences. For imbalanced data, balanced accuracy or macro-averaged metrics may be more informative than raw accuracy.
Include calibration when the model outputs probabilities used for ranking, triage, or threshold decisions. A model with slightly lower discrimination but better-calibrated probabilities may be more useful operationally. Track the calibration method as part of the pipeline, and fit it without leaking test labels.
For regression, report an error metric such as mean absolute error or root mean squared error, along with a scale-dependent interpretation. For every task, record peak memory, training duration, inference duration, and the number of model evaluations. If real hardware is involved, include shots, queue or submission overhead when relevant, failed jobs, retries, and whether error mitigation was applied.
Worked examples
Example 1: A small classification comparison
Suppose a team evaluates a binary classifier after reducing the input to a fixed number of features that both models can consume. The classical comparison includes a regularized linear model and a small multilayer perceptron. The quantum candidate uses a feature-encoding circuit followed by a variational circuit. All three models use the same training, validation, and test partitions and are evaluated over the same predefined seeds.
The team’s primary metric is balanced accuracy, with F1 score and calibration error as secondary metrics. The quantum model completes 40 optimization steps. Each step uses 12 objective evaluations, and each evaluation uses 500 shots. Its estimated shot workload is therefore:
12 × 500 × 40 = 240,000 shots.
This number should appear beside measured elapsed time and simulator or hardware details. If the quantum model scores 0.78 balanced accuracy and the classical models score 0.79 and 0.81, the result does not support a claim of superiority. It may still support a narrower conclusion: the quantum model reached a similar range under the chosen circuit and budget. If its score varies widely across seeds while the classical models remain stable, the next experiment should investigate initialization, optimizer behavior, shot noise, or circuit expressivity.
Example 2: A results table for an experiment report
| Model | Test metric mean ± spread | Secondary metric | Training time | Evaluation workload | Notes |
|---|---|---|---|---|---|
| Regularized linear | — | — | — | — | Fixed preprocessing and tuning budget |
| Classical neural network | — | — | — | — | Parameter count and stopping rule recorded |
| Quantum hybrid model | — | — | — | Shots, evaluations, and circuit depth | Simulator or hardware conditions recorded |
Do not fill this table with the best run only. Use the mean and spread across the planned repetitions, then add individual-run data to an appendix or experiment artifact. A useful conclusion distinguishes statistical similarity from practical usefulness. “No meaningful difference under this budget” is more accurate than “the approaches are equivalent,” unless the study was designed to test equivalence.
Example 3: Ablation before a larger claim
If the quantum model appears to outperform a baseline, run ablations before attributing the result to quantumness. Remove entangling gates, replace the quantum feature map with a classical transformation, vary circuit depth, and compare exact expectations with finite-shot measurements. Keep the data split and evaluation protocol fixed. These tests help determine whether the gain comes from the circuit structure, a preprocessing choice, parameter count, optimization, or random variation.
When to recalculate
Recalculate the benchmark whenever an input that can affect the decision changes. This includes the dataset version, label definition, train-test split, preprocessing pipeline, feature count, baseline tuning budget, circuit architecture, optimizer, shot count, simulator settings, hardware target, software dependencies, or error-mitigation procedure.
Pricing and platform conditions also belong in the worksheet when the experiment uses a paid simulator or quantum service. Do not copy an old estimate into a new report: verify the applicable pricing unit, minimum allocation, execution limit, and account assumptions at the time of the run. Keep measured usage separate from an estimate based on circuit count.
Revisit the methodology at least once during a yearly review and after major QML-library or backend changes. API updates can alter compilation, gradient evaluation, device support, or default behavior. A new backend may also change the meaning of wall-clock comparisons, so report the environment rather than treating old timings as permanent facts.
For the next benchmark, take these practical steps:
- Write the primary claim and metric before running models.
- Lock the data split, preprocessing, seeds, and tuning budgets.
- Train at least one simple and one strong classical baseline.
- Record quantum circuit resources and estimate total circuit shots.
- Repeat runs, preserve predictions, and report variation.
- Run ablations before connecting an observed improvement to the quantum component.
- Save code, configuration, dependency versions, logs, and raw results so the benchmark can be rerun when inputs change.
This workflow will not prove that a quantum model is universally better than a classical model. It will produce something more useful for engineering decisions: a transparent estimate of where the model performs well, what it costs to evaluate, how stable the result is, and which assumptions would need to change before the conclusion should be revisited.