Benchmarking is essential when researchers need to compare numerical solvers, machine learning models, sparse linear algebra libraries, or optimization software. A solver that performs well on one equation, matrix, tolerance, or hardware platform may behave very differently under another set of conditions.
A useful benchmark should provide repeatable problem definitions, consistent accuracy measurements, controlled computing environments, and transparent reporting. It should also match the type of software being evaluated. An ODE integrator, a neural operator, a sparse direct solver, and a PDE-constrained optimizer do not solve the same problem and should not be compared through the same test suite.
The scientific computing landscape includes several important benchmark resources. SciMLBenchmarks focuses on differential equation solver efficiency. PDEBench provides datasets and evaluation tasks for Scientific Machine Learning. HPCG represents sparse, memory-intensive workloads on high-performance computers. SuiteSparse supplies matrices from real applications. The ASU Mittelmann collection supports comparisons of optimization software, including PDE-constrained problems.
Why Solver Benchmarks Need Clear Categories
The word solver can describe several different types of software. A time integrator advances an ODE or semi-discretized PDE. A sparse linear solver calculates the solution of a matrix system. A neural operator approximates the mapping between PDE inputs and outputs. An optimization solver minimizes an objective while satisfying constraints.
These tools require different evaluation criteria. Time integrators are commonly compared through error, runtime, and function evaluations. Sparse solvers require matrix collections, memory measurements, iteration counts, and scaling results. Scientific ML models need training and inference metrics, generalization tests, and comparisons against numerical reference solutions.
A benchmark suite is useful only when its problem class matches the solver’s purpose.
Major Benchmark Families
| Benchmark Resource | Main Problem Class | Typical Evaluation | Best Use |
|---|---|---|---|
| SciMLBenchmarks | ODEs, SDEs, DAEs, PDE semi-discretizations, and related models | Work-precision diagrams, timing, error, and solver comparisons | Comparing differential equation algorithms |
| PDEBench | Time-dependent PDE datasets for Scientific ML | Prediction error, rollout quality, training, and inference performance | Comparing neural PDE surrogates and operator-learning models |
| HPCG | Structured sparse linear workload on HPC systems | System-level performance and scalability | Evaluating how HPC hardware handles representative sparse operations |
| SuiteSparse Matrix Collection | Real sparse matrices from many application areas | Runtime, memory, convergence, robustness, and factorization behavior | Testing sparse matrix algorithms and solver libraries |
| ASU Mittelmann | Continuous, discrete, and PDE-constrained optimization | Runtime, solved instances, optimality, and robustness | Comparing optimization software |
Clarifying Unverified Benchmark Names
Informal project notes and generated research drafts sometimes refer to suites called “HiMaS” or “SciBase.” These names should not be treated as established scientific solver benchmarks without a precise publication, repository, or institutional source.
When discussing sparse solver evaluation, it is safer to reference the actual resource being used, such as HPCG, SuiteSparse, or a documented comparison involving PETSc, Trilinos, hypre, or SuperLU.
PETSc, Trilinos, hypre, and SuperLU are solver libraries rather than one combined benchmark suite. Researchers can compare them on the same matrices or PDE workloads, but the test protocol must be defined separately.
SciMLBenchmarks
SciMLBenchmarks provides reproducible comparisons for differential equation algorithms. Its coverage includes non-stiff and stiff ODEs, stochastic differential equations, differential-algebraic equations, boundary value problems, and selected PDE systems after spatial discretization.
The central evaluation tool is the work-precision diagram. Each solver is run with several tolerance settings or numerical configurations. The measured error is plotted against computational cost, such as execution time, number of right-hand-side evaluations, or another work metric.
A solver that appears closer to the lower-left region of the diagram generally provides lower error at lower cost. No single point is sufficient because solver rankings can change as the requested accuracy changes.
What Work-Precision Diagrams Reveal
Suppose Solver A is faster at loose tolerances but becomes inefficient when high accuracy is required. Solver B may have greater setup cost but scale better as the tolerance becomes stricter. A single runtime comparison would hide this distinction.
Work-precision analysis makes the accuracy-versus-cost trade-off visible. It is particularly useful for adaptive time integrators because their step counts and function evaluations depend on the requested tolerance.
Cross-Language Comparisons
SciML benchmark collections may include algorithms accessed through different software ecosystems. Cross-language testing requires care because wrapper overhead, compilation, memory allocation, solver defaults, and error-control definitions can affect the result.
A fair comparison should use equivalent equations, initial conditions, tolerances, output times, Jacobian information, and stopping criteria.
PDEBench for Scientific Machine Learning
PDEBench is designed for machine learning research involving PDE-based physical systems. It provides code, datasets, baseline models, and evaluation tools for comparing learned surrogates across several equation families.
The associated datasets include problems such as advection, Burgers’ equation, diffusion-reaction systems, diffusion-sorption, compressible Navier-Stokes equations, Darcy flow, and shallow-water equations.
Researchers can use the supplied data to evaluate Fourier neural operators, convolutional networks, autoregressive models, and other Scientific ML approaches without independently generating every simulation dataset.
What PDEBench Measures
Evaluating a learned PDE model requires more than a single mean squared error. Useful measurements may include:
- Errors at individual time steps
- Long-horizon rollout stability
- Conservation-law violations
- Performance on unseen parameter values
- Errors near boundaries or sharp gradients
- Training and inference cost
- Memory consumption
The numerical simulator used to produce a training dataset and the learned model that approximates its output perform different tasks. Training cost, data-generation cost, and inference cost should therefore be reported separately.
Related SciML Data Resources
Benchmark data for PDE-based Scientific ML research is also available through the University of Stuttgart’s SciML benchmark collection on DaRUS.
Additional benchmark outputs and experimental comparisons may be found at benchmarks.sciml.ai. Researchers should record the exact dataset version, repository commit, and software environment used in each experiment.
HPCG for HPC System Evaluation
The High Performance Conjugate Gradients benchmark was created as a complement to High Performance LINPACK. HPL emphasizes dense linear algebra, while HPCG uses computational and memory-access patterns that are closer to many sparse scientific applications.
HPCG constructs a sparse system associated with a three-dimensional model problem and applies operations such as sparse matrix-vector multiplication, vector updates, dot products, and multigrid-preconditioned conjugate gradient iterations.
The benchmark exposes limits related to memory bandwidth, communication, synchronization, and data movement. These factors often dominate large sparse simulations even when a machine has very high theoretical floating-point performance.
What HPCG Does Not Measure
HPCG should not be treated as a complete ranking of all sparse linear solvers. It uses a specific benchmark algorithm and problem structure. A library designed for unsymmetric matrices, sparse direct factorization, indefinite systems, or application-specific preconditioning may not be represented fairly by one HPCG result.
HPCG is most useful for comparing computer systems or implementations of its prescribed workload. Broader solver evaluation requires multiple matrices and problem classes.
SuiteSparse Matrix Collection
The SuiteSparse Matrix Collection contains sparse matrices gathered from real applications. The collection was previously known as the University of Florida Sparse Matrix Collection.
Its matrices represent problems from computational fluid dynamics, structural analysis, circuit simulation, optimization, electromagnetics, graph analysis, and many other fields.
This diversity makes SuiteSparse useful for testing:
- Sparse direct factorization
- Krylov iterative methods
- Preconditioners
- Matrix reordering
- Graph partitioning
- GPU sparse kernels
- Memory and storage formats
Why Real Matrices Matter
A synthetic matrix may reproduce a selected size or sparsity level while missing the structural properties that make a real system difficult. Fill-in during factorization, diagonal dominance, symmetry, conditioning, block structure, and graph connectivity can all influence solver behavior.
A good SuiteSparse study should not select only matrices that favor one method. The test set should represent the target application and include clear inclusion and exclusion rules.
Solver Libraries and Benchmark Data
PETSc, Trilinos, hypre, and SuperLU are commonly used in scientific computing, but they provide different capabilities.
- PETSc supplies distributed vectors, matrices, Krylov methods, nonlinear solvers, time integrators, and interfaces to external packages.
- Trilinos contains packages for sparse algebra, iterative methods, preconditioning, direct solvers, nonlinear systems, and multiphysics workflows.
- hypre focuses on scalable iterative solvers and multigrid preconditioners for large parallel systems.
- SuperLU provides sparse LU factorization for sequential, multithreaded, and distributed environments.
A credible comparison should use equivalent matrix formats, stopping tolerances, preconditioner settings, precision, hardware allocation, and convergence definitions.
ASU Mittelmann Benchmarks
Hans Mittelmann’s ASU benchmark site collects test problems and performance comparisons for optimization software.
The broader site covers several optimization categories. Its PDE-constrained optimization collection includes fully discretized elliptic and parabolic control problems represented as AMPL models.
These problems can be submitted to nonlinear optimization solvers that support AMPL. They are useful for studying how optimizers handle large systems in which the decision variables and constraints arise from a discretized PDE.
What PDE-Constrained Optimization Measures
A PDE-constrained optimization benchmark may evaluate more than the speed of the PDE solution. It can also test:
- Objective-function reduction
- Constraint satisfaction
- First-order optimality
- Robustness to initial guesses
- Derivative and Hessian handling
- Memory use
- Scaling as discretization is refined
These tests are not substitutes for ordinary PDE or sparse linear solver benchmarks. They evaluate the complete optimization workflow.
How to Choose the Right Benchmark
| Evaluation Goal | Recommended Resource | Main Metrics |
|---|---|---|
| Compare ODE or time-integration algorithms | SciMLBenchmarks | Error, runtime, function evaluations, and work-precision |
| Compare learned PDE surrogate models | PDEBench | Prediction error, rollout stability, training, and inference cost |
| Compare HPC systems on sparse workloads | HPCG | Throughput, scaling, memory, and communication performance |
| Compare sparse solver algorithms | SuiteSparse | Runtime, memory, iterations, robustness, and residuals |
| Compare PDE-constrained optimizers | ASU Mittelmann | Runtime, solved problems, feasibility, and optimality |
Designing a Fair Solver Comparison
Every solver should receive the same mathematical problem. This includes the same domain, mesh, coefficients, initial conditions, boundary conditions, final time, and requested output.
When algorithms use different internal representations, the comparison should focus on a common external accuracy measure. A fast result is not useful if it fails to meet the target error.
The benchmark report should record:
- Solver and dependency versions
- Compiler and optimization settings
- CPU, GPU, memory, and network information
- Thread and process counts
- Precision and data type
- Tolerances and stopping conditions
- Preconditioners and matrix reorderings
- Warm-up and compilation treatment
- Number of repeated runs
- Error and failure definitions
Use Multiple Accuracy Levels
A benchmark performed at one tolerance can produce a misleading winner. Some methods have low overhead and perform well when accuracy requirements are loose. Others become more competitive at stricter tolerances.
For time integrators, several relative and absolute tolerance combinations should be tested. For iterative linear solvers, researchers should report residual definitions and examine whether algebraic residual reduction corresponds to actual solution error.
Work-precision diagrams, performance profiles, and convergence plots provide more information than a single timing table.
Measure Setup and Repeated-Solve Costs
Solver cost often contains several components:
- Matrix assembly
- Preconditioner construction
- Symbolic factorization
- Numerical factorization
- Individual solve time
- Data transfer
- Output and post-processing
A direct method may have high initial factorization cost but solve additional right-hand sides quickly. An iterative method may have lower setup cost but require many iterations for every solve.
Benchmark reports should separate one-time setup from repeated-solve performance.
Common Benchmarking Mistakes
One frequent mistake is comparing results generated on different meshes or with different error targets. A faster calculation on a coarser problem does not demonstrate that a solver is more efficient.
Other common mistakes include:
- Reporting runtime without reporting error
- Selecting only favorable matrices
- Using different stopping conditions
- Ignoring failed or non-convergent runs
- Combining compilation with execution for only one language
- Comparing CPU and GPU runs without documenting data-transfer costs
- Using HPCG to evaluate an unrelated neural PDE model
- Using PDEBench as a substitute for sparse matrix benchmarking
- Reporting only the fastest run instead of repeated measurements
- Failing to publish scripts and configurations
A Practical Benchmark Workflow
- Define the scientific problem and target use case.
- Select a benchmark resource that matches the solver class.
- Choose representative instances before running the solvers.
- Define common error, convergence, and failure criteria.
- Record software and hardware environments.
- Test several tolerances, matrix sizes, or mesh resolutions.
- Separate setup, solve, and data-transfer costs.
- Repeat runs and report statistical variation.
- Inspect accuracy, robustness, memory, and scalability together.
- Publish scripts, configuration files, and raw results.
Related Guides
- Performance Profiling and Optimization for Python PDE Solvers — Learn how to identify runtime and memory bottlenecks before comparing implementations.
- Mesh Quality and Convergence Studies: A Hands-On Guide — Review mesh refinement and convergence methods for fair PDE comparisons.
- Verification vs. Validation in Scientific Simulations — Understand how numerical correctness differs from agreement with physical observations.
Conclusion
Scientific solver benchmarks serve different purposes. SciMLBenchmarks compares differential equation algorithms through error and computational work. PDEBench provides standardized Scientific ML datasets and tasks. HPCG measures how HPC systems perform on a representative sparse workload. SuiteSparse supports broad testing of sparse matrix algorithms using real application data. ASU Mittelmann provides established optimization test problems, including PDE-constrained models.
These resources should not be treated as interchangeable. The correct benchmark depends on whether the target is a time integrator, learned surrogate, sparse linear solver, hardware platform, or optimization package.
A credible benchmark reports accuracy, runtime, memory, failures, solver settings, and hardware details. It uses identical problem instances and tests more than one tolerance or resolution. The goal is not simply to identify the fastest run, but to determine which method delivers the required accuracy and reliability for the intended scientific workload.