Reading Time: 12 minutes
  • Jupyter notebooks run successfully only ~12.4% of the time when dependencies are declared—explicit environment management closes this gap
  • The five pillars framework from Ziemann et al. (2023) provides a structured baseline for documentation, version control, environment management, data sharing, and FAIR compliance
  • The REPRO Checklist (Hornung et al., 2026) offers a practical self-audit tool covering folder structure, environment specification, data availability, and deterministic execution
  • Common pitfalls like absolute paths, unspecific dependencies, and missing READMEs are the primary causes of reproducibility failures in computational research
  • A step-by-step workflow from simulation design through notebook documentation, code archiving, and journal submission provides a repeatable production pipeline

The reproducibility gap in computational research

When you submit a simulation-based paper, the reviewers check your methodology, boundary conditions, convergence studies, and comparison with analytical benchmarks. But they rarely rerun your code. The results presented in your figures and tables become the published record—and if those results were generated by a workflow that cannot be reproduced, the publication rests on an unverified foundation.

This is not a theoretical problem. Samuel & Mietchen (2024) analyzed 15,817 Python-based Jupyter notebooks from biomedical publications indexed in PubMed Central. Of the notebooks where all declared dependencies could be successfully installed, 87.6% resulted in exceptions during automated rerun. Only 1,203 notebooks ran to completion without errors—and of those, 324 produced results that differed from the originally reported outputs. The remaining 879 notebooks produced identical results. This 5.56% success rate for identical reproduction was actually an improvement over the initial 2021 run (5.88%), indicating that recent notebooks tend to be marginally better—but the gap remains enormous.

What makes reproducibility harder for computational scientists? You work with numerical solvers, mesh refinements, time integration schemes, and coupling strategies that produce outputs through chains of floating-point operations. A single Python version bump, a changed default parameter in numpy or scipy, or a relative path that doesn’t resolve on another machine can invalidate every figure and table in your paper. The problem compounds when your workflow involves adaptive mesh refinement, custom source terms, or non-trivial boundary conditions—the very kind of advanced patterns documented in FiPy resources on this site.

Five pillars of reproducible computational research

Ziemann et al. (2023) formalized the five pillars of computational reproducibility as a framework for documenting and sharing reproducible workflows. While originally developed in the context of bioinformatics, these pillars apply equally to materials science simulations, computational fluid dynamics, and finite-volume-based PDE solvers.

1. Literate programming

Literate programming means documenting your computational workflow as a narrative that interleaves explanation and executable code. In the computational sciences, this typically manifests as Jupyter notebooks, R Markdown documents, or Python scripts with extensive docstrings and inline comments.

The goal is not merely to produce a reproducible file—it is to create a document that a reader unfamiliar with your analysis can follow, understand, and rerun. Samuel & Mietchen (2024) found a clear correlation between the Markdown-to-code-cell ratio and successful reproduction: notebooks with higher documentation effort (more Markdown cells relative to code cells) were significantly more likely to reproduce identically to the original output.

For a phase-field simulation in FiPy—an introduction to its core architecture—or a diffusion model using the finite volume method, literate programming might look like this:

# Set deterministic seed
import numpy as np
np.random.seed(42)

# Build a structured grid (this should match the mesh description in your paper)
nx, ny = 100, 100
mesh = CellGrid([nx, ny], varIndex=1)

# Define the transient diffusion term and boundary conditions
T = CellVariable(name='Temperature', mesh=mesh, value=initial_temperature)
eqn = TransientTerm() == DiffusionTerm(coeff=thermal_conductivity)

# Impose Dirichlet boundary conditions (see also our guide on FiPy BCs)
T.fixLeft(value=300)
T.fixRight(value=500)

When you write this notebook cell, you are simultaneously documenting the mesh resolution, the PDE, and the boundary conditions—and executing the simulation that produced your published figure.

Recommendation: If your notebook has fewer Markdown cells than code cells, you are under-documented. Aim for at least a 1:1 ratio, and ideally more. The Samuel & Mietchen study found that the identical-reproduction group had a Markdown-to-code ratio of 0.73, while the different-reproduction group had only 0.49.

2. Code version control and sharing

Version control is not optional for publication-grade work. Git provides a permanent, queryable history of every code change—and GitHub, GitLab, or Zenodo archives your repository as a citable object with a DOI.

The purpose of version control is twofold:

  • Reproducibility: A specific commit hash (abc1234) paired with a specific commit date tells anyone exactly what code produced your published results.
  • Accountability: Every parameter change, every source term modification, every boundary condition edit is recorded. When a reviewer asks “why did you use this diffusion coefficient?” you can point to the exact commit where it was set.

When publishing, archive your repository on Zenodo (or GitHub with Zenodo integration enabled) and include the DOI in your manuscript’s data availability statement. This turns your simulation code into a permanent, citable artifact. Many journals now recommend depositing code in repositories with DOI assignment—see the JOSS simulation science guidelines for journal expectations on simulation-based publications.

Decision rule: GitHub vs. Zenodo? Use GitHub for active development and collaboration. Use Zenodo for archival. Zenodo automatically assigns a DOI to GitHub repositories when linked, making it the preferred choice for publication. You can maintain a live GitHub repository and archive the final commit on Zenodo for submission.

3. Compute environment control

Environment control is the single most important technical intervention you can make. When a reviewer tries to reproduce your results on a fresh machine, they need to know the exact software stack: Python version, NumPy version, FiPy version, solver libraries. Without this information, the reviewer is guessing—and guessing produces ModuleNotFoundError, ImportError, and FileNotFoundError (the three most common exceptions identified by Samuel & Mietchen, responsible for 41.65% of all execution failures).

How to capture your environment:

# In Python, use the session-info package to export your environment
!pip install session-info
import session_info
session_info.show()

# Conda environment export (alternative)
conda env export > environment.yml

# pip requirements (for simpler projects)
pip freeze > requirements.txt

Decision rule: conda vs. pip vs. Docker? For most computational science workflows in Python, conda is the best balance of practicality and completeness. It handles both Python packages and non-Python dependencies (such as HDF5 libraries or MPI runtimes). Docker provides full isolation at the cost of complexity; use Docker when you need to share a workflow with researchers on different operating systems who cannot install conda or when your workflow depends on system-level libraries.

The REPRO Checklist (Hornung et al., 2026) explicitly recommends specifying software versions in the README and providing an environment specification file. They note: “For long-term reproducibility, the software environment must also be clearly documented, including the versions of all additional packages used.”

4. Persistent data sharing

Data sharing goes hand-in-hand with code sharing. Your simulation outputs—mesh configurations, field arrays, time series of variables—should be deposited in a repository that assigns a persistent identifier (DOI).

For simulation data, this typically means:

  • Raw simulation outputs (field arrays, mesh files, time series) deposited alongside the code
  • Intermediate results for computationally intensive simulations (see the REPRO Checklist recommendation on intermediate results)
  • Synthetic test data if your real data cannot be shared publicly due to privacy or legal restrictions

The FAIR principles (Wilkinson et al., 2016) provide a framework for data management—see the GO FAIR guidance on the principles for the official guidance on Findability, Accessibility, Interoperability, and Reusability.

  • Findable: Assign persistent identifiers (DOIs) and include rich metadata
  • Accessible: Store data in repositories with open protocols (Zenodo, figshare, Materials Data Facility)
  • Interoperable: Use standard formats (HDF5, NetCDF, JSON) and formal vocabularies
  • Reusable: Document provenance, licensing, and data cards

For materials science and computational physics, platforms like the Materials Cloud and the NOMAD Repository provide FAIR-compliant infrastructure for simulation data, complete with metadata schemas and API access.

5. Documentation

The REPRO Checklist places enormous emphasis on documentation—not as an afterthought, but as a structural component of reproducible work. A README file should:

  • List every file in the repository and its purpose
  • Specify the exact execution order (which scripts produce which figures or tables)
  • Provide approximate runtimes and hardware requirements
  • Reference the corresponding figure and table numbers from the publication

For simulation workflows, documentation also means documenting your numerical methods. When you write a custom solver or a FiPy script with operator splitting, Strang splitting, and IMEX schemes for PDE solvers, you need to document:

  • The discretization scheme
  • The time integration method
  • The solver tolerances
  • The convergence criteria

If another researcher cannot understand the numerical method behind your results, they cannot evaluate whether the results are valid—or reproduce them on a different solver.

The REPRO Checklist as a self-audit tool

The REPRO Checklist (Hornung et al., 2026), published in Royal Society Open Science (full paper), was developed by an international consortium of journal reproducibility editors to provide a concise, actionable framework for reproducible research. While it applies broadly across disciplines, several elements are particularly relevant for computational simulation work.

Here is how to use the REPRO Checklist as a pre-submission self-audit:

A. Structure and README

Check: Do you have a data/, code/, and results/ (or output/) directory? Does your README file list every file, specify execution order, and reference each script’s corresponding figures?

Common failure: Researchers organize their simulation code in a single folder with no structure. A README file is often omitted entirely. Without a README, a reviewer has no way to know which script produces which figure—and they may execute the scripts in the wrong order, producing entirely different results.

B. Environment specification

Check: Is your environment.yml or requirements.txt included in the archive? Are Python and package versions explicitly specified?

Common failure: The most common reason notebooks fail is missing or conflicting dependencies. Samuel & Mietchen found that 34.32% of notebooks failed at the dependency installation step—even though none of the files were malformed. The root cause is often unspecified or outdated package versions.

C. Data availability

Check: Can a reviewer run your code from scratch using the provided data? If you cannot share real data, have you included synthetic data that mimics the computational structure?

Common failure: When the real data cannot be shared (due to legal, ethical, or institutional constraints), the code should still be runnable with synthetic data. The REPRO Checklist recommends generating synthetic data using packages like synthpop or simdata in R, or numpy.random in Python.

D. Deterministic execution

Check: Are random number generator seeds explicitly set? For parallel simulations, are reproducible random number streams used?

Common failure: This is one of the most common and most subtle causes of irreproducibility. A simulation that uses numpy.random.randn() without setting a seed will produce different results on every run—and even with a seed, parallel workers may each generate overlapping random streams. The REPRO Checklist recommends using NumPy’s SeedSequence-based generation for parallel-safe, deterministic random streams.

from numpy.random import default_rng
rng = default_rng(42)  # Fixed seed

For parallel simulations, use numpy.random.SeedSequence to generate independent, reproducible streams:

from numpy.random import SeedSequence
seq = SeedSequence(42)
children = seq.spawn(n_children)  # Deterministic per-child seeds

E. Intermediate results

Check: For computationally intensive simulations (e.g., parameter sweeps, Monte Carlo studies), are intermediate results saved so reviewers can reproduce specific figures without re-running the full analysis?

Common failure: When a simulation takes hours or days to run, reviewers cannot rerun it from scratch. The REPRO Checklist recommends saving intermediate results (raw output before final figure generation) so reviewers can quickly verify specific outputs. This is especially important for simulation studies with parallel replications.

Step-by-step workflow from simulation design to submission

This section provides a concrete workflow that integrates the five pillars and the REPRO Checklist into a repeatable production pipeline.

Phase 1: Simulation design and code organization

  1. Create the project structure:
project/
├── README.md
├── environment.yml
├── data/
│   ├── input/         # Simulation inputs, initial conditions
│   ├── output/        # Raw simulation outputs
│   └── synthetic/     # Synthetic test data (if real data is restricted)
├── code/
│   ├── mesh.py        # Mesh configuration
│   ├── solve.py       # Main simulation script
│   ├── postprocess.py # Figure generation
│   └── test/          # Unit tests and convergence checks
└── results/
    ├── figures/
    └── tables/
  1. Version control: Initialize the repository and commit the structure.
git init
git add README.md environment.yml
git commit -m "Initial project structure"

Phase 2: Simulation execution and documentation

  1. Write the simulation in a literate format. Use Jupyter notebooks or script files with extensive comments. Document every numerical method, every solver choice, every boundary condition.
  2. Set deterministic seeds. Include np.random.seed(42) (or a more sophisticated SeedSequence setup for parallel runs) at the top of every script.
  3. Capture the environment. Run session_info.show() or conda env export during a successful run and include the output in the README.
  4. Save intermediate results. If your simulation is computationally intensive, save raw outputs at checkpoints so reviewers can verify specific figures without re-running everything.

Phase 3: Validation and verification

  1. Run convergence studies. Verify that your results converge as expected as you refine the mesh. This is part of verification—checking that the numerical solution approaches the exact solution as discretization parameters decrease. For a practical framework, see our guide on validation and verification for PDE simulations.
  2. Write unit tests. Test individual functions, boundary conditions, and solvers. If you are using FiPy, this might include testing that CellVariable values are correctly initialized or that DiffusionTerm is discretized correctly. See our article on testing patterns for scientific code for more detail.

Phase 4: Archiving and submission

  1. Archive on Zenodo. Push the final commit to GitHub and link it to Zenodo for DOI assignment.
  2. Run the REPRO Checklist audit:
    • README file with execution order?
    • Environment specification included?
    • Data available (real or synthetic)?
    • Deterministic seeds set?
    • Intermediate results saved?
  3. Submit with the code and data DOI in your manuscript’s data availability statement. Many journals now require this at submission or revision.

Common pitfalls and how to avoid them

Pitfall 1: Absolute paths

Using absolute paths like /home/user/simulations/output/data.csv in your code breaks reproducibility on any machine other than the one where you developed the project.

Fix: Use relative paths from the project root. If your scripts expect data in a data/ directory, reference data/input.csv regardless of where the project is cloned.

Pitfall 2: Unscheduled dependencies

A notebook imports scipy.stats but does not declare that scipy is required. When another researcher runs the notebook, they may have an incompatible scipy version—or miss it entirely.

Fix: Always include a requirements.txt or environment.yml and verify it by deleting the local environment and reinstalling from scratch before archiving.

Pitfall 3: Missing README

Without a README, reviewers cannot know:

  • Which script produces which figure
  • What order scripts should be executed
  • What approximate runtime to expect
  • What Python version or package versions were used

Fix: Write a README that lists every file, every execution step, and every runtime estimate. Reference the figure numbers from your publication.

Pitfall 4: Python version drift

Python 3.6 was sunset in 2021; Python 3.7 was sunset in June 2023. Many notebooks in the Samuel & Mietchen corpus used outdated Python versions, and the mismatch between the version at publication and the version available when reviewers attempt reproduction causes subtle incompatibilities.

Fix: Use a recent, actively supported Python version and specify it in your environment file. The REPRO Checklist recommends documenting the Python version in the README.

Pitfall 5: Non-deterministic random numbers

A simulation that generates random initial conditions without fixing the seed produces different outputs every time. Even with a seed, parallel workers may generate overlapping streams.

Fix: Set a fixed seed for all random operations. Use numpy.random.SeedSequence for parallel-safe reproducibility. The REPRO Checklist explicitly recommends this pattern.

When to choose containers vs. lightweight environments

For most simulation workflows, lightweight environment management (conda, pip, uv) is sufficient. You get reproducibility by specifying exact package versions—and reviewers can install the environment with a single command.

Choose conda/uv/pip when:

  • You are working with Python-based solvers (FiPy, OpenPDE, FEniCS)
  • You have control over the computational environment on your end
  • You want a lightweight, portable workflow

Choose Docker when:

  • You need to share a workflow with researchers who cannot install conda or specific system-level libraries
  • Your workflow depends on C++/Fortran solvers that must be compiled
  • You are submitting to a journal that requires containerized reproducibility (e.g., Computo)

For the vast majority of computational materials science and physics papers, conda or uv is the better choice—it is simpler, faster, and sufficient for the reproducibility that reviewers and readers need.

A practical note on journal expectations

The REPRO Checklist documents the heterogeneity of journal reproducibility policies. Some journals (like Biometrical Journal or Journal of the American Statistical Association) require active reproducibility checks by dedicated editors. Others (like The BMJ) require code submission but do not perform independent execution checks.

For computational simulation work, the minimum expectation across most journals is:

  • Code availability: The analysis code should be provided as a supplementary file or in an open repository (GitHub, Zenodo)
  • Data availability: The data used for analysis should be publicly available or justified
  • Environment specification: Software versions and packages should be documented

This is the floor—not the ceiling. Following the five pillars and the REPRO Checklist positions your work above the floor and signals that you take reproducibility seriously.

Next steps for your simulation workflow

If you are new to reproducible simulation practices, start with the three most impactful changes:

  1. Add a README file to your project with file descriptions and execution order
  2. Export your environment (conda env export or uv export) and include it in your archive
  3. Set random seeds in every simulation script and use SeedSequence for parallel runs

These three changes alone will eliminate 80% of the reproducibility failures identified by Samuel & Mietchen.

If you need help designing a reproducible simulation workflow—from notebook documentation through code archiving and journal submission—our team can provide consultation on workflow design, environment specification, and FAIR-compliant data archiving. Contact us to discuss your specific simulation project. Building a sustainable research software community (how we approach community-rooted development) is part of that process: reproducibility improves when the entire team shares common standards.

Conclusion

Reproducible publication practices for simulation results are not an optional add-on—they are a requirement for credible scientific communication. The five pillars (literate programming, version control, environment management, data sharing, documentation) and the REPRO Checklist provide a structured, actionable foundation for reproducibility that applies across computational disciplines.

The scale of the reproducibility gap—12.4% successful identical reproduction in the Samuel & Mietchen study—is a call to action. Every simulation workflow that includes a README, an environment file, and deterministic seeds closes a measurable portion of that gap. The investment is modest: hours, not months. The return is the ability for any reader, reviewer, or future researcher to verify your results independently.

For the computational researcher, reproducibility is not merely a methodological virtue—it is the foundation of scientific credibility. The five pillars give you a framework. The REPRO Checklist gives you a checklist. The tools (Git, conda, Zenodo) are available. What remains is the decision to use them.

FAQ

What is the five pillars framework for reproducibility?

The five pillars, proposed by Ziemann et al. (2023), are literate programming, code version control, compute environment control, persistent data sharing, and documentation. They form a structured framework for documenting and sharing reproducible computational workflows.

What is the REPRO Checklist?

The REPRO Checklist, published by Hornung et al. (2026), is a concise, multi-disciplinary tool for creating reproducible analyses. It covers code and data structure, README requirements, environment specification, data availability, deterministic random number generation, and intermediate results.

Why do Jupyter notebooks fail so often?

Samuel & Mietchen (2024) found that 87.6% of Python-based notebooks resulted in exceptions during automated rerun. The primary causes are missing or conflicting dependencies (ModuleNotFoundError and ImportError), unresolved file paths (FileNotFoundError), and non-deterministic random number generation.

How do I choose between conda and Docker?

Use conda (or uv) for most Python-based simulation workflows—it is simpler, faster, and sufficient for specifying exact package versions. Use Docker when you need full isolation across operating systems or when your workflow depends on compiled C++/Fortran solvers.

How can I make my simulation results reproducible without sharing raw data?

Generate synthetic data that mimics the computational structure of your real data. The REPRO Checklist recommends using packages like synthpop or simdata in R, or numpy.random in Python, to create synthetic test data that enables independent verification of your analysis code.