Running the benchmarks#

benchmarks/ times complete runs of a small set of representative workloads. Each release adds a few lines per case to a history file kept per hardware set, so that regressions and improvements show from one release to the next. The design is SEP 38, which replaced SEP 30’s standard suite. The rules for a performance claim and the indicative results are in Benchmarks; this page is the manual.

uv run poe bench                      # the plan and its estimated time
uv run python -m benchmarks track     # the tracking run, about an hour
uv run python -m benchmarks history   # what earlier runs recorded
uv run python -m benchmarks ab --case rdw --baseline-python OLD/bin/python

Every command uses the native CPU backend unless --backend says otherwise, and needs it installed, as a plain uv sync does. Run on a quiet machine: a dedicated workstation, or an exclusive allocation on a cluster node.

The tracking set#

python -m benchmarks list prints the plan: one line per case, stepper and width, and the estimated time from the costs each case declares (complete runs on 16 cores of an AMD EPYC 7B13, about 33 minutes for the tracking set; other machines differ). With no selection it is the tracking set:

Case	Interface	Problem	Widths
`gnlse_raman_shock`	`prop_gnlse`	Dudley–Genty–Coen supercontinuum, 0.15 m	1, production
`gnlse_soliton`	`prop_gnlse`	N = 3 soliton, one soliton period	1, production
`rdw`	`prop_capillary`	He resonant dispersive wave, four modes, 1.5 m	1, production
`he_rdw_averaged`	`prop_capillary`	the same, mode-averaged	1, production
`argon_blueshift`	`prop_capillary`	Ar ionisation blueshift, 0.2 m	1, production
`argon_blueshift_averaged`	`prop_capillary`	the same, mode-averaged	1, production
`argon_blueshift_envelope`	`prop_capillary`	the same on the envelope grid	1, production
`h2_raman_averaged`	`prop_capillary`	H2 soliton self-frequency shift, 0.4 m	1, production
`ctc`	`prop_capillary`	H2 comb-to-continuum, three modes, 0.13 m	production
`radial_filament_256`	`prop_radial`	Ar filament, 256 radial nodes, 3 cm	1, production
`grin_nonlinear`	`prop_waveguide`	four-channel graded-index fibre, 2 cm	production
`kerr_beam_384`	`prop_free`	fused-silica Kerr beam, 384², four fixed RK4IP steps	production
`kerr_beam_192`	`prop_free`	the same at 192²	1

The definitions are in benchmarks/registry.py and benchmarks/builders.py. Every other registered case (the remaining modal, free-space and advanced-guide cases, and the backend exercises, and the sep47 set of short windows for SEP 47’s matrix) runs when named with --case, --set or --all; none is in the default selection. A case whose definition changes keeps its name and raises its revision, and the history compares rows only within one revision.

What is timed#

Each configuration (case, stepper, width) runs in its own worker process, whose thread pools are set in the environment before any scientific import. The worker:

builds the case, translates it and constructs the backend, timing these separately;
runs an untimed warm-up over the first 2 % of the guide, which builds FFT plans and fills caches;
times --repeats complete runs (default 3). Each restarts the backend from the translated launch and goes through solprop.driver.run.run_problem, the loop of Simulation.run, with a fresh boundary monitor as Simulation.run attaches one, so it includes the plane observations, the statistics and the boundary checks: what a user’s run costs.

The result is the median of the repeats, with the minimum and maximum beside it, the accepted and rejected steps, the source evaluations, the peak resident memory, and the integrator and tolerances the run actually used. Repeats of one configuration take the same steps; the summary says so when they do not.

Library defaults. A case is the interface call a user would write. It passes no tolerance or integrator, so it runs with its interface’s defaults (SEP 37). A changed default therefore shows in the history as a new integrator or tolerance on that row. Every noisy launch fixes the same seed, so its repeats and its releases are comparable. The N = 3 soliton is the one noiseless case: a soliton-recurrence study runs without noise, and SEP 37 measured a seed multiplying its steps by 65. The Kerr beams run their declared fixed RK4IP schedule.

Steppers. --stepper takes default, multistep, multistep-interaction or rk, or a comma-separated list to compare steppers in one run:

uv run python -m benchmarks track --case rdw --case ctc --stepper default,rk

An override replaces the integrator only; the tolerances stay the interface’s. A fixed-step case admits default only, and the plan says which it skipped.

Widths. Without --threads, each case runs at its declared widths: one thread and the production width, or one of them for the most expensive cases. --threads 1,8 names the widths for every selected case. The production width is the number of physical cores in the process’s CPU allocation unless --production gives it; on a cluster, give it explicitly.

Threading. --threading sets OpenMP’s placement and waiting in every worker: backend (the default since November 2026: unpinned, with the backend’s own waiting, none of the waiting variables set, so that a row measures what a user’s default run gets), passive (unpinned, idle threads sleep at once), passive-bound (the same, pinned to neighbouring cores), default (the runtime’s own short spin before sleeping), active (idle threads spin), bound (threads pinned to neighbouring cores) or bound-active. The policy is recorded with each row, and rows compare only within one policy. Binding also confines SciPy’s transforms on the host to the main thread’s core, so a free-space row’s edge reads cost more under a bound policy than under passive (the edge against 0.1.0).

On a GPU#

With a native GPU build installed (CUDA or HIP; see the native backend’s build guide), --device runs every selected case once on the device:

uv run python -m benchmarks track --device

The host still runs the production width of threads, but each row’s width is device. The hardware name is the GPU model, for example nvidia-a40 or nvidia-h200, plus the instance family on a cloud machine, and the host CPU model is recorded beside it. An AMD GPU whose rocm-smi series is the generic “AMD Radeon Graphics” is named from its PCI device or subsystem ID instead. Each row also records the device memory the run holds. The worker refuses a GPU build without --device and a CPU build with it. The case costs are CPU costs, so a device plan gives no time estimate.

Recording a run#

uv run python -m benchmarks track --append-history

A series measures a user’s default run, so it runs calibrated (SEP 46 section 6.13): before the first configuration, track runs solprop calibrate --thorough (--calibrate chooses another level, or none to keep the profiles there are), and its workers apply the profile (SOLPROP_CALIBRATION=use) without measuring inside a timed window. Each history row records the profile its run applied in the calibration column (built-in, or the level and generation, with +refinement), so a series that moves can be attributed to the code or to a recalibration. SOLPROP_CALIBRATION=off holds the built-in table, for experiments that must keep the settings fixed while the code changes; the first series recorded after SEP 46 starts a new segment of the history.

The run’s bundle goes to .benchmarks/track/<date>-<hardware>/ (git ignores it): the plan, each worker’s request, environment and logs, and results.json, which is rewritten after every configuration so that an interrupted run keeps what it measured. --archive DIR copies the bundle to a directory outside the repository.

--append-history adds one row per configuration to benchmarks/history/<hardware>.csv, which is committed. Each row names its backend, and rows compare only with rows of the same backend, so the numba_cpu rows recorded at 0.1.0, before SEP 43 retired that backend, keep their own series beside the native one. It records only the whole default plan with at least three repeats, and only when:

every configuration finished, and its repeats took the same steps and reached the same final field to rounding, with the native backend running the threads it was given;
the workers imported solprop from this checkout’s src/ (run from the checkout’s own environment, uv sync, not an installed copy), with, for a native run, a native build from a commit after which nothing under native/ changed but its documentation and tests;
nothing under src/, native/ or benchmarks/ is uncommitted (the history files aside, so one checkout can record several runs), unless --allow-dirty, whose rows are marked dirty;
SOLPROP_FFTW_PLAN is unset: a row does not record how it planned, and measured planning changes its construction time and its median.

The hardware name is the CPU model and the production width, plus the instance family on a cloud machine, for example apple-m5-max-18c, amd-epyc-7763-16c or arm-neoverse-v2-16c-aws-m8g; it never contains a host name. Give a cloud machine’s instance type with --instance-type where its DMI record does not name it (AWS’s does).

python -m benchmarks history prints, per hardware and width, one line per recorded run: its date, solprop version, the configurations it ran and how many started on a busy machine, and its change against the first and the previous recorded runs. A change is the geometric mean of median-time ratios (below one is faster) over the configurations both runs share, counted in brackets. Two rows are the same configuration only with the same number of memory (NUMA) domains spanned, threading policy, case revision, stepper, resolved integrator, tolerances and noise model. A changed library default therefore starts a new series rather than reading as a speed-up, and so does an allocation spread over more memory domains than usual: on a cluster, keep the cores together (--distribution=block:block under Slurm).

Recorded hardware#

The history starts at 0.1.0. Each release reruns the same hardware sets the same way, so that their rows stay one series:

Hardware file	Machine	Width	`--instance-type`
`apple-m5-max-18c`	Apple M5 Max, macOS	18	—
`intel-xeon-gold-6240-36c`	2× Intel Xeon Gold 6240, whole machine (exclusive Slurm job)	36	—
`amd-epyc-7763-16c`	2× AMD EPYC 7763 cluster node, exclusive, 16 cores in one NUMA domain	16	—
`amd-epyc-9b45-16c-gcp-c4d`	Google Cloud `c4d-highcpu-32`, SMT off, Ubuntu 24.04	16	`gcp c4d`
`amd-epyc-9b45-16c-gcp-c4d-windows`	the same shape, Windows Server	16	`gcp c4d windows`
`arm-neoverse-v2-16c-gcp-c4a`	Google Cloud `c4a-standard-16`, Ubuntu 24.04	16	`gcp c4a`
`nvidia-a40`	NVIDIA A40 cluster node, CUDA build (`AMPERE86`)	device	—
`nvidia-h200-do`	cloud `gpu-h200x1-141gb` instance, CUDA build (`HOPPER90`)	device	`do`
`amd-mi325x-do`	cloud `gpu-mi325x1-256gb` instance, HIP build (`AMD_GFX942`)	device	`do`

CPU rows time the release’s wheel. A clean checkout of the release tag, its environment made with uv sync --python 3.14 --frozen --no-install-package solprop-native, then uv pip install solprop-native==<version> from PyPI, and every command run as .venv/bin/python -m benchmarks …: a later uv sync or uv run builds native in the wheel’s place. The wheel records the tag’s commit, so the history accepts it.
numba rows exist only at 0.1.0: a second run in the same checkout with --backend numba_cpu, on the Apple M5 Max, the Xeon Gold 6240 and the EPYC 7763. A later release cannot add them.
GPU rows build the tag’s native backend with the cuda or hip preset’s options into a separate environment, out of tree (uv build native --wheel --config-setting build-dir=…), so that the checkout stays clean. ROCm 10 builds the HIP backend only with -DSOLPROP_NATIVE_WARNINGS_AS_ERRORS=OFF at 0.1.0 (fixed after it).
The A40’s memory domains. The 0.1.0 job held a whole node, so its host threads could run in all 8 NUMA domains and its rows record 8. At 0.2.0 every A40 node was partly busy: the job held 16 cores of a shared node, they spanned 3 domains, and those rows start a new series (their device times were 9 % faster than 0.1.0’s in geometric mean). An exclusive job returns to the 0.1.0 series.
The instance type separates the operating systems. The hardware name carries no operating system, so Linux and Windows on one cloud shape differ only by --instance-type. Google Cloud allowed a 32-vCPU C4D only in us-central1 at 0.1.0, so the two ran one after the other.
Windows ends an SSH session’s child processes with it: start the run through WMI (Invoke-CimMethod Win32_Process Create), and run one unrecorded case first, since the first worker after boot runs cold.
The Mac runs with Spotlight indexing paused (sudo mdutil -a -i off) and the controlling session idle. At 0.1.0 one run of gnlse_raman_shock at 18 threads took 31 s against 21 s in the recorded run and in three repeats, with the same steps: an occasional slow mode to watch for.
The AMD GPU name. ROCm 10 reports the cloud MI325X’s Card Series as “AMD Radeon Graphics”; the harness then names the part from its PCI IDs. The 0.1.0 rows, recorded before that fallback, had their slug and gpu_model set to it by hand; nothing else in them changed.

Comparing two builds while iterating#

ab times the same configurations with two prepared environments, for a change in progress:

uv run python -m benchmarks ab --case rdw --case argon_blueshift --threads 1 \
    --length 0.2 --baseline-python /path/to/baseline/bin/python \
    --candidate-python /path/to/candidate/bin/python

Each build is calibrated before the rounds, once per interpreter and with its arm’s variables (--calibrate). A development A/B defaults to the standard level, which takes a few seconds; thorough, track’s default, is for release benchmarks and for comparing what a heavy user’s calibration would give. --baseline-env and --candidate-env NAME=VALUE set an arm’s own variables over the threading preset, and an empty value unsets one. A worker otherwise runs at the row’s width, which keeps a profile’s team width out, and with it any setting fitted at another width. To compare the built-in table with this machine’s calibration as a default run sees them, give both arms no thread count of their own:

uv run python -m benchmarks ab --set tracking --threads 16 --production 16 \
    --calibrate none --length 0.2 --rounds 3 --repeats 1 \
    --baseline-python "$(which python)" \
    --baseline-env SOLPROP_THREADS= --baseline-env SOLPROP_NATIVE_THREADS= \
    --baseline-env SOLPROP_CALIBRATION=off \
    --candidate-env SOLPROP_THREADS= --candidate-env SOLPROP_NATIVE_THREADS= \
    --candidate-env SOLPROP_CALIBRATION=use

The same command with SOLPROP_CALIBRATION=off in both arms is its A/A control.

Selection works as for track. --length runs that fraction of each guide, from the entrance, for quick iteration. Each round runs every configuration once with each build, one worker each, and alternates which build goes first; --rounds sets the count (default 3) and --repeats the timed runs per worker (default 3). The harness is copied into the output directory first, so both builds run the same benchmark code.

The summary gives, per configuration, each build’s median time, the median over rounds of the candidate/baseline ratio of the per-round medians (below one is faster), the range of those ratios, whether both builds took the same steps, and the relative L2 difference of their final fields. Passing the same interpreter twice is an A/A control: its ratio range shows the noise to expect on that machine. A one-second reading before each worker labels a busy machine but never waits. ab gives no adoption verdict; the rules say how a claim is judged.

SEP 47’s matrix and the structural counters#

SEP 47 section 7.2 compares builds on a predeclared matrix of short windows, each holding the events it measures, with structural counters beside the timings (section 7.3). --set sep47 selects the windows added for it; each finishes in seconds at its full declared length, so run it without --length:

uv run python -m benchmarks ab --set sep47 --threads 1 --counters \
    --baseline-python /path/to/baseline/bin/python

Case	What one repetition is	Events it holds
`stats_period1_adams`, `stats_period1_rk`	16 mm of `gnlse_soliton`’s fibre under `Multistep()` or `RungeKutta()` (no stepper override), energy and peak power as periodic statistics with period 1	a row at every accepted step (about 190 and 40)
`snapshots_idle`, `snapshots_sparse`, `snapshots_every`	the same fibre to HDF5 with `step_planes=1` and a `SnapshotController` never asked, asked after every eighth accepted step, or asked at every accepted endpoint	a plane at every step; 0, 5 and 42 snapshots published
`with_input_repeat`	three runs of the fibre with other peak powers through `Simulation.with_input`	three preparations from one translation
`scan_local`	`run_scan` of four peak powers on `Local`, built with `with_input`, field-free	four points, no field copied to the host
`group_batch`	the same four points on `Batch(4)`, replaying ten 1.6 mm steps with the split-step collar	ten common steps of four members
`radial_filament_256_{saved,free}_{memory,hdf5}`	the radial filament’s first 3 mm, four planes, every field kept or none, to memory or to HDF5	four planes, transverse-edge reads and collar checks
`cartesian_filament_128_{saved,free}_{memory,hdf5}`	the Cartesian filament’s first 1 mm, likewise	the same on a 128² grid
`taper_flow_4`, `taper_flow_17`	40 µm of a He capillary tapering from 50 to 48 µm with taper coupling, 4 or 17 modes	a coupled flow on the native Adams march, and one wider than 16 channels, whose host passes hold each bin’s channel vectors in a row of their own (a device build’s passes hold none)

Today a snapshot controller needs step_planes=1 and HDF5 output, so the snapshot cases save a plane at every step; their variant without step planes becomes possible with SEP 47’s independent snapshot scheduling, and the cases will be extended then. Every case uses only public interfaces of the current code; one whose interface changes is updated with it, and the harness copied into each A/B run keeps both arms on one definition.

The rest of the matrix is existing cases, run by name with a short --length window where they are long: the cheap envelope and field steps (gnlse_soliton, he_rdw_averaged), the temporal and transverse collar edges (absorber_schemes, spatial_absorber), low-storage replay (kerr_beam_192), general z tables and varying guides (mixture_gradient, taper, taper_wall, moving_solid_guide) and the large fields at production sizes (radial_filament_512, cartesian_filament_256).

Counters. --counters (for track and ab) has each worker make one more repetition after the timed ones with the native adapter’s counters on, reported as the receipt’s counted and, in ab’s summary, per arm; counting therefore costs no timed repetition anything. With counting off, the adapter’s path is the one before the counters except for a single attribute test per native call (a march call, a plane, a statistics call, an edge read, a field crossing or load, a low-storage or group step, a native session’s advance or export): the per-attempt hooks are installed only while counting, and nothing else is computed, so no ledger entry was taken for them.

A native run is served by the native execution session (solprop_native.session.NativeSession), the only native executor since SEP 47 phase 5. The record keeps the keys the transitional bridge’s runs were counted under, so that an arm measured before phase 5 can be compared key for key with one after; the session’s own keys start session_. The record (solprop_native._counters, private to the harness) holds:

native_returns: every return to Python, by cause. For a native march called from Python (the backend’s own march, which no run takes since phase 5), a call’s reason (reached, monitor for a boundary-monitor check by distance, bound for its attempt or time limit, watch, capacity and the errors), with host_step split by the driver’s work due on its last step: statistics, step_planes, monitor (the check by steps), transverse (a collar’s per-step current) or transverse_read, joined with + when several fall due together; then host_attempt for each attempt solprop’s own loop steps, low_storage_step for the low-storage march. A native session returns once per advance, by its batch’s event: session_observation, session_snapshot (a requested snapshot serviced), session_warning, session_terminal, or session_progress with its cause, session_progress:buffer (the attempt or row buffer filled), session_progress:work_limit (the caller’s attempt bound) or session_progress:deadline (its time bound). Members of the native group session are native sessions and are counted so. A periodic row, a collar check, an edge read or a low-storage step is never a session return (SEP 47 section 7.3);
march_calls: native march calls by march, adams_march and rk_march from Python on the bridge, and the native session’s own calls inside its advance, read from its status, as session_adams, session_runge_kutta or session_low_storage;
attempts and accepted (native march, native session and host attempts together; a session’s from its delivered chunks), march_attempts, host_attempts, low_storage_steps and group_steps (a group’s common steps);
planes (calls of propagate_to on the bridge, observations after the launch in a session), statistics_rows (periodic rows, wherever taken) and statistics_evaluations (rows and plane statistics together), and transverse_reads (reads of the edge’s faces or a collar’s current made through Python: a native session reads its edge natively, so none);
field_crossings with field_bytes: fields copied to the host, by where. On the bridge at a plane (plane), at every accepted step of a host Adams engine (step) or at its load (load); in a session, each observation’s field exported into its destination in one native pass (export; a recipe asked again for the same batch is served from the first and not counted) and an Adams march’s state and history copied for a snapshot (snapshot);
field_loads with load_bytes: host fields loaded into the native state, by where: start (a run’s start or resume, on either path) and restore (the Adams march’s reload after a dense observation or at a method history’s restore). solprop’s host Adams engine reloads its masked state at every step, which is not counted; its step crossings mark that route.

The run’s own record adds the planes saved and, for a snapshot case, the snapshots published; the peak resident memory is the worker’s, as for every row. A host attempt makes several native calls, which are not counted one by one; a group run’s counts include each point’s resource probe, a native session that returns at its launch; and a device’s launches, transfers and allocations are not counted: they need a profiler. Counts explain a timing and do not replace it; check that both arms hold the same events before comparing their times (SEP 47 section 7.2).

A scan or group repetition builds its own backends, so its worker builds none beforehand and warms up with one whole untimed repetition. scan_local watches each point’s run to count its steps; Batch watches none, so group_batch’s steps are counted only by --counters, and ab reports its work as unchecked rather than same. Where a repetition keeps no final field (field-free planes, a scan), finiteness is read from its last plane’s statistics and the repeat difference is not defined; track treats either gap as unchecked, never as consistent.