Reading Time: 16 minutes

Python Data Versioning and Provenance: dvc, DVC, and Scientific Workflows

Data versioning in scientific simulation isn’t about tracking code changes — Git already handles that perfectly well. It’s about tracking which specific combination of data file, code commit, and parameter file produced a particular result. That distinction is what separates a fragile experimental workflow from a reproducible simulation pipeline.

The most widely adopted tool for this is DVC (Data Version Control), which extends Git’s versioning model to handle large datasets and pipeline stages. This article covers practical DVC implementation for scientific workflows — pipeline construction with dvc.yaml, parameter isolation with params.yaml, HPC/SLURM integration, and emerging W3C PROV-JSON provenance tracking — so you can build deterministic simulation campaigns that survive team turnover, hardware migration, and years-long project lifecycles.

Key Takeaways

• DVC’s core value: Pipeline files (dvc.yaml) make simulations dependency-aware, so dvc repro re-runs only what changed
• Versioned reproducibility: DVC links data snapshots (.dvc files), code commits (Git), and parameter files (params.yaml) into reproducible experiment units
• HPC integration: SLURM batch scheduling wraps DVC for cluster-native pipeline execution without manual intervention
• Provenance standards: W3C PROV-JSON (yProv4ML) emerges as an interoperable format complementary to RO-Crate packaging
• DVC vs DataLad: Choose DVC for pipeline-oriented experiments; choose DataLad for long-term data curation and distributed datasets

Why Standard Versioning Fails for Simulation Workflows

Git is excellent for tracking code changes. It’s terrible for tracking data changes.

When you run a DFT relaxation calculation, Git can’t store the resulting .xyz crystal structure file (often tens of MB). You might copy it to a shared drive, or append its SHA-256 hash to a text file, or rely on memory. All three approaches break when the data source changes, when you need to share with collaborators who don’t have access to the shared drive, or when you come back six months later and forget which parameter set produced the lowest-energy structure.

Data versioning tools solve this by decoupling data tracking from code tracking. They store actual files in remote storage (Google Drive, S3, a shared NAS, or even another Git repository) and register lightweight pointer files in the Git repository. The pointer files — typically small .dvc files or git-annex symlink references — contain metadata (checksum, remote location, version tag) without duplicating the actual data.

This separation matters for simulation workflows because the data volume scales independently from the code volume. A single simulation campaign might produce thousands of trajectory files while the Python scripts generating them stay at a few hundred lines.

The Pipeline Gap

Most researchers start with ad-hoc data tracking: a bash script that runs calculations in sequence, with results copied into a results/ directory and the final numbers pasted into a Google Sheet. This works for small projects. It breaks down when:

1. Re-running with different parameters — you need to remember which params.yaml file was used and re-run only the changed stages
2. Debugging failures — you need to know whether the error came from data corruption, code changes, or parameter mismatch
3. Team handoff — a new student can’t reconstruct the pipeline from a bash script and a folder of orphaned files

DVC’s pipeline definition file (dvc.yaml) addresses this gap by declaring stages with explicit dependencies and outputs. Each stage is re-run only when its dependencies change (dvc repro <stage>), making simulation campaigns dramatically more efficient when re-running calculations with different parameters.

DVC Pipeline Construction for Scientific Workflows

A DVC pipeline file is a declarative YAML document that maps how your simulation steps depend on each other. Here’s a concrete example for a computational materials science workflow:

# dvc.yaml — DFT relaxation and energy calculation pipeline

stages:
  relax:
    cmd: python src/relax.py
    deps:
    - data/raw_crystal_structure.xyz
    - src/relax.py
    params:
    - params.yaml
    outs:
    - results/relaxed_structure.xyz

  energy:
    cmd: python src/energy.py
    deps:
    - results/relaxed_structure.xyz
    - src/energy.py
    params:
    - params.yaml
    outs:
    - results/energies.txt
    metrics:
    - results/energies.txt

  visualize:
    cmd: python src/plot.py
    deps:
    - results/energies.txt
    - src/plot.py
    outs:
    - figures/energy_plot.png

Each stage declares:

• cmd: the command that generates this stage’s outputs
• deps: files that, if changed, trigger re-running this stage
• params: parameter files that, if changed, trigger re-running this stage
• outs: files produced by this stage (tracked as data versions)
• metrics: files that DVC tracks numerically for experiment comparison

Why params.yaml Matters

The params.yaml file isolates simulation parameters from execution scripts. This is critical for materials-science workflows where the same simulation code runs hundreds of times with different parameters:

# params.yaml
simulator:
  cutoff_energy: 500  # eV
  kpoints: [8, 8, 8]
  tolerance: 1e-6
  
optimization:
  max_steps: 200
  algorithm: ionic

When you change tolerance from 1e-6 to 1e-8 and run dvc repro, DVC detects the params.yaml change and re-runs every stage that depends on it. The resulting relaxed_structure.xyz file gets a new version tag. The old version remains available in the cache — you don’t lose historical runs.

For parameter sweeps, you can use --set-param to override values without modifying the file:

dvc exp run --set-param sim.tolerance=1e-8 --set-param sim.cutoff_energy=450

This creates a new experiment branch, preserving both the original and modified runs in your pipeline history.

Versioned Reproducibility — DVC’s Conceptual Contribution

DVC’s blog (December 2021) introduced “versioned reproducibility” as the ability to recreate not just a result, but the exact experimental state that produced it. This distinguishes DVC from generic versioning:

• Standard versioning tracks changes to files over time (Git’s purpose)
• Versioned reproducibility tracks which specific data version, code commit, and parameter file combination produced a given result (DVC’s purpose)

DVC achieves this by linking three independently versioned artifacts:

Artifact	Versioning System	What It Tracks
.dvc pointer files	DVC	Data file version + remote location + checksum
Git commits	Git	Code version + commit message + diff
params.yaml	Git	Parameter file version + field-level changes

The combination of these three creates a reproducible unit — a “versioned experiment” that can be reconstructed in isolation. This matters for simulation because experiments are iteratively refined. Researchers need to know “what worked” and be able to reproduce it without re-running every intermediate step.

Practical Implications

When a reviewer asks for the data behind a published figure, you can provide:

1. The Git commit hash (code version)
2. The .dvc pointer file reference (data version)
3. The params.yaml file at that commit (parameter version)

Combined, these three artifacts are sufficient to reproduce the result on any machine. This is the gold standard for simulation reproducibility — far beyond containers (which freeze code environments) or bare Git repositories (which can’t handle large data).

Experiment Queueing and Management

DVC’s experiment management (dvc exp) extends the pipeline model to support concurrent and queued experiments:

# Run a single experiment with custom parameters
dvc exp run -S sim.tolerance=1e-8

# Queue multiple experiments
dvc exp run --queue -S sim.tolerance=1e-8
dvc exp run --queue -S sim.tolerance=1e-6
dvc exp run --queue -S sim.tolerance=1e-5

# Execute queued experiments
dvc exp run

The --queue flag defers execution, allowing you to prepare multiple experiments and batch-run them sequentially. This is valuable for computational experiments that run for hours or days — you can queue a parameter sweep overnight without manually launching each run.

For experiment comparison, DVCLive provides real-time metrics tracking:

# src/metrics.py — DVCLive integration
from dvclive import Live

with Live() as live:
    for step in range(n_iterations):
        result = run_step(step)
        live.step = step
        live.log("energy", result["total_energy"])
        live.log("forces", result["max_force"])

This produces an interactive comparison dashboard where you can see how energy and forces evolve across parameter sweeps.

Running DVC Pipelines on HPC Clusters

Most computational scientists don’t run pipelines on laptops. They run them on clusters with SLURM, PBS, or LSF schedulers. DVC integrates with SLURM through the srun wrapper, enabling cluster-native pipeline execution:

# Run a single pipeline stage with SLURM resource allocation
srun dvc repro -n relax --job slurm-job.sh

# Run all pipeline stages on the cluster
srun dvc repro

The --job flag tells DVC to run each stage through SLURM’s job scheduler, automatically handling:

• Cluster-specific resource allocation (memory, CPU, GPU nodes)
• Batch-mode execution without manual intervention
• Automatic job queuing for pipeline stages
• Integration with cluster-specific file systems and storage (Lustre, GPFS, BeeGFS)

arXiv 2505.06558v2 (September 2025) demonstrates this integration for materials-science workflows running DFT calculations on HPC clusters. The key advantage is that dvc repro becomes cluster-aware — if a stage fails or its dependencies change, SLURM handles the resource allocation for re-running only the necessary stages.

Cluster Storage Considerations

DVC’s remote storage works well with cluster file systems. You can configure DVC to use the cluster’s Lustre or GPFS filesystem as a DVC remote:

# Configure a cluster storage remote
dvc remote modify cluster_storage token_url "https://cluster-storage.example.com"

This avoids the overhead of copying data to/from external cloud storage and keeps the pipeline self-contained within the cluster’s storage hierarchy.

W3C PROV-JSON Provenance Tracking

While DVC tracks pipeline stages and data versions, it doesn’t natively produce a machine-actionable provenance graph. For that, you need a provenance standard — and the emerging format is W3C PROV-JSON.

yProv4ML (arXiv July 2025) implements W3C PROV-JSON provenance with minimal code modifications. It captures:

• Who ran the experiment (user identity)
• What data and code were used (source provenance)
• How the experiment ran (execution provenance)
• Why (optional motivation, tied to research objectives)

The output is a PROV-JSON graph — a directed graph where nodes represent entities (files, parameters, code commits) and edges represent relationships (derived-from, produced-by, used-by). This format enables:

• Interoperability across provenance systems: PROV-JSON is machine-readable and can be queried with SPARQL or graph databases
• FAIR data compliance: PROV-JSON graphs satisfy FAIR principles by documenting data lineage and reuse conditions
• Cross-institutional collaboration: PROV-JSON is a W3C standard, so provenance graphs from different institutions are compatible

PROV-JSON vs RO-Crate

It’s useful to distinguish PROV-JSON from RO-Crate, since both appear in reproducibility discussions:

Aspect	RO-Crate	PROV-JSON
Purpose	Package format for rich metadata	Data format for provenance graphs
Scope	Whole dataset + associated resources	Execution lineage and dependencies
Format	JSON-LD	JSON (PROV standard)
Interoperability	Self-contained package	Graph-based, queryable
Relationship	Complementary, not competing

They address different layers of reproducibility. RO-Crate packages your dataset with metadata. PROV-JSON tracks the computational lineage of how that dataset was produced. Using both together provides full reproducibility: “here is the data” (RO-Crate) plus “here is how it was produced” (PROV-JSON).

When to Use DVC vs DataLad vs PROV-Compliant Libraries

Choosing between DVC, DataLad, and PROV-compliant libraries depends on your workflow patterns:

Criterion	DVC	DataLad	PROV Libraries
Primary use case	Pipeline-oriented experiments	Long-term data curation	Machine-actionable provenance
Data size	Large files (models, simulations)	Large distributed datasets	N/A (metadata-focused)
Execution model	Stage-based pipeline (dvc repro)	Command-based (datalad run)	Decorator-based
HPC integration	srun dvc repro	Native git-annex + SSH	Configurable
Provenance output	.dvc files + Git history	Git-annex symlinks + git log	PROV-JSON graph
Best for	Materials-science pipelines, parameter sweeps	Long-term repository curation, BIDS compliance	FAIR data, cross-institutional provenance

DVC is right for you if:

• Your workflow has multiple dependent stages (relaxation → property calculation → visualization)
• You need parameter sweeps with automatic re-running
• You want to track which parameter combination produced the lowest-energy result

DataLad is right for you if:

• You’re curating a long-term dataset repository
• You need git-annex symlinks for efficient large-file handling
• You’re working with distributed datasets across institutions
• You need BIDS compliance (common in neuroimaging)

PROV-compliant libraries are right for you if:

• You need machine-actionable provenance for FAIR compliance
• You want a queryable provenance graph for data lineage
• You’re collaborating across institutions with different provenance systems
• You’re publishing to a FAIR data repository

Putting It Together — A Practical Workflow

Here’s a practical workflow pattern that combines all the concepts above:

# 1. Initialize the repository
git init
dvc init

# 2. Add the data
dvc add data/raw_crystal_structure.xyz

# 3. Run the pipeline
dvc repro

# 4. Check the results
dvc metrics show results/energies.txt

# 5. Run experiments with different parameters
dvc exp run -S sim.tolerance=1e-8 --set-param sim.cutoff_energy=550

# 6. Push data and experiments to remote
dvc push
dvc exp push
git push origin main

# 7. (Optional) Generate PROV-JSON provenance
python src/provenance.py --output provenance.json

This pattern ensures that every simulation result can be traced back to its exact data version, code commit, and parameter file. The PROV-JSON output (step 7) adds a machine-actionable provenance layer on top of the DVC pipeline.

Common Mistakes — What to Avoid

Mistake 1: Versioning everything — Don’t run dvc add on every output file. Only track files that are inputs to subsequent stages or that represent final results. Intermediate files (like raw force arrays) should stay in Git or be excluded entirely. Over-adding creates pipeline noise.

Mistake 2: Hardcoding paths — Don’t use absolute paths in dvc.yaml. Use relative paths so the pipeline works on different machines and cluster configurations.

Mistake 3: Mixing data types — Don’t put simulation parameters and model checkpoints in the same .dvc file. Keep data types separate to avoid cache conflicts.

Mistake 4: Ignoring provenance — DVC files alone aren’t sufficient for FAIR compliance. If your institution requires provenance graphs, pair DVC with PROV-JSON output (yProv4ML) or RO-Crate packaging.

Mistake 5: Forgetting params.yaml — If you hardcode parameters in your scripts instead of isolating them in params.yaml, DVC’s parameter tracking becomes useless. Always use params.yaml for simulation parameters.

Summary

DVC transforms scientific workflows from fragile, memory-dependent processes into deterministic, dependency-aware pipelines. The key insight is that dvc.yaml pipeline files and params.yaml parameter isolation create “versioned reproducibility” — the ability to reconstruct not just a result, but the exact experimental state (data version + code commit + parameter file) that produced it. This goes beyond containers (which freeze code environments) and beyond Git (which can’t handle large datasets).

For HPC integration, SLURM’s srun wrapper enables cluster-native pipeline execution without manual intervention. For provenance, W3C PROV-JSON (yProv4ML) emerges as an interoperable format complementary to RO-Crate packaging, enabling machine-actionable data lineage that satisfies FAIR compliance requirements.

The choice between DVC, DataLad, and PROV-compliant libraries depends on your workflow patterns: DVC for pipeline-oriented experiments, DataLad for long-term data curation, and PROV libraries for machine-actionable provenance graphs. Most simulation teams benefit from using DVC for pipeline execution and PROV-JSON for provenance tracking — the two tools are complementary, not competing.

---

Related Guides

• Reproducibility Beyond Containers — Covers DVC basics, RO-Crate provenance packaging, and container-based reproducibility patterns
• Managing Large-Scale PDE Problems — Discusses cluster resource management and HPC workflows
• Verification vs Validation in Scientific Simulations — Explains the broader reproducibility context for simulation results</fs_write_file>
  </tool_call>

<tool_call>
<function=fs_write_file>
<parameter=content>

Python Data Versioning and Provenance: dvc, DVC, and Scientific Workflows

Data versioning in scientific simulation isn’t about tracking code changes — Git already handles that perfectly well. It’s about tracking which specific combination of data file, code commit, and parameter file produced a particular result. That distinction is what separates a fragile experimental workflow from a reproducible simulation pipeline.

The most widely adopted tool for this is DVC (Data Version Control), which extends Git’s versioning model to handle large datasets and pipeline stages. This article covers practical DVC implementation for scientific workflows — pipeline construction with dvc.yaml, parameter isolation with params.yaml, HPC/SLURM integration, and emerging W3C PROV-JSON provenance tracking — so you can build deterministic simulation campaigns that survive team turnover, hardware migration, and years-long project lifecycles.

Key Takeaways

• DVC’s core value: Pipeline files (dvc.yaml) make simulations dependency-aware, so dvc repro re-runs only what changed
• Versioned reproducibility: DVC links data snapshots (.dvc files), code commits (Git), and parameter files (params.yaml) into reproducible experiment units
• HPC integration: SLURM batch scheduling wraps DVC for cluster-native pipeline execution without manual intervention
• Provenance standards: W3C PROV-JSON (yProv4ML) emerges as an interoperable format complementary to RO-Crate packaging
• DVC vs DataLad: Choose DVC for pipeline-oriented experiments; choose DataLad for long-term data curation and distributed datasets

Why Standard Versioning Fails for Simulation Workflows

Git is excellent for tracking code changes. It’s terrible for tracking data changes.

When you run a DFT relaxation calculation, Git can’t store the resulting .xyz crystal structure file (often tens of MB). You might copy it to a shared drive, or append its SHA-256 hash to a text file, or rely on memory. All three approaches break when the data source changes, when you need to share with collaborators who don’t have access to the shared drive, or when you come back six months later and forget which parameter set produced the lowest-energy structure.

Data versioning tools solve this by decoupling data tracking from code tracking. They store actual files in remote storage (Google Drive, S3, a shared NAS, or even another Git repository) and register lightweight pointer files in the Git repository. The pointer files — typically small .dvc files or git-annex symlink references — contain metadata (checksum, remote location, version tag) without duplicating the actual data.

This separation matters for simulation workflows because the data volume scales independently from the code volume. A single simulation campaign might produce thousands of trajectory files while the Python scripts generating them stay at a few hundred lines.

The Pipeline Gap

Most researchers start with ad-hoc data tracking: a bash script that runs calculations in sequence, with results copied into a results/ directory and the final numbers pasted into a Google Sheet. This works for small projects. It breaks down when:

1. Re-running with different parameters — you need to remember which params.yaml file was used and re-run only the changed stages
2. Debugging failures — you need to know whether the error came from data corruption, code changes, or parameter mismatch
3. Team handoff — a new student can’t reconstruct the pipeline from a bash script and a folder of orphaned files

DVC’s pipeline definition file (dvc.yaml) addresses this gap by declaring stages with explicit dependencies and outputs. Each stage is re-run only when its dependencies change (dvc repro <stage>), making simulation campaigns dramatically more efficient when re-running calculations with different parameters.

DVC Pipeline Construction for Scientific Workflows

A DVC pipeline file is a declarative YAML document that maps how your simulation steps depend on each other. Here’s a concrete example for a computational materials science workflow:

# dvc.yaml — DFT relaxation and energy calculation pipeline

stages:
  relax:
    cmd: python src/relax.py
    deps:
    - data/raw_crystal_structure.xyz
    - src/relax.py
    params:
    - params.yaml
    outs:
    - results/relaxed_structure.xyz

  energy:
    cmd: python src/energy.py
    deps:
    - results/relaxed_structure.xyz
    - src/energy.py
    params:
    - params.yaml
    outs:
    - results/energies.txt
    metrics:
    - results/energies.txt

  visualize:
    cmd: python src/plot.py
    deps:
    - results/energies.txt
    - src/plot.py
    outs:
    - figures/energy_plot.png

Each stage declares:

• cmd: the command that generates this stage’s outputs
• deps: files that, if changed, trigger re-running this stage
• params: parameter files that, if changed, trigger re-running this stage
• outs: files produced by this stage (tracked as data versions)
• metrics: files that DVC tracks numerically for experiment comparison

Why params.yaml Matters

The params.yaml file isolates simulation parameters from execution scripts. This is critical for materials-science workflows where the same simulation code runs hundreds of times with different parameters:

# params.yaml
simulator:
  cutoff_energy: 500  # eV
  kpoints: [8, 8, 8]
  tolerance: 1e-6
  
optimization:
  max_steps: 200
  algorithm: ionic

When you change tolerance from 1e-6 to 1e-8 and run dvc repro, DVC detects the params.yaml change and re-runs every stage that depends on it. The resulting relaxed_structure.xyz file gets a new version tag. The old version remains available in the cache — you don’t lose historical runs.

For parameter sweeps, you can use --set-param to override values without modifying the file:

dvc exp run --set-param sim.tolerance=1e-8 --set-param sim.cutoff_energy=450

This creates a new experiment branch, preserving both the original and modified runs in your pipeline history.

Versioned Reproducibility — DVC’s Conceptual Contribution

DVC’s blog (December 2021) introduced “versioned reproducibility” as the ability to recreate not just a result, but the exact experimental state that produced it. This distinguishes DVC from generic versioning:

• Standard versioning tracks changes to files over time (Git’s purpose)
• Versioned reproducibility tracks which specific data version, code commit, and parameter file combination produced a given result (DVC’s purpose)

DVC achieves this by linking three independently versioned artifacts:

Artifact	Versioning System	What It Tracks
.dvc pointer files	DVC	Data file version + remote location + checksum
Git commits	Git	Code version + commit message + diff
params.yaml	Git	Parameter file version + field-level changes

The combination of these three creates a reproducible unit — a “versioned experiment” that can be reconstructed in isolation. This matters for simulation because experiments are iteratively refined. Researchers need to know “what worked” and be able to reproduce it without re-running every intermediate step.

Practical Implications

When a reviewer asks for the data behind a published figure, you can provide:

1. The Git commit hash (code version)
2. The .dvc pointer file reference (data version)
3. The params.yaml file at that commit (parameter version)

Combined, these three artifacts are sufficient to reproduce the result on any machine. This is the gold standard for simulation reproducibility — far beyond containers (which freeze code environments) or bare Git repositories (which can’t handle large data).

Experiment Queueing and Management

DVC’s experiment management (dvc exp) extends the pipeline model to support concurrent and queued experiments:

# Run a single experiment with custom parameters
dvc exp run -S sim.tolerance=1e-8

# Queue multiple experiments
dvc exp run --queue -S sim.tolerance=1e-8
dvc exp run --queue -S sim.tolerance=1e-6
dvc exp run --queue -S sim.tolerance=1e-5

# Execute queued experiments
dvc exp run

The --queue flag defers execution, allowing you to prepare multiple experiments and batch-run them sequentially. This is valuable for computational experiments that run for hours or days — you can queue a parameter sweep overnight without manually launching each run.

For experiment comparison, DVCLive provides real-time metrics tracking:

# src/metrics.py — DVCLive integration
from dvclive import Live

with Live() as live:
    for step in range(n_iterations):
        result = run_step(step)
        live.step = step
        live.log("energy", result["total_energy"])
        live.log("forces", result["max_force"])

This produces an interactive comparison dashboard where you can see how energy and forces evolve across parameter sweeps.

Running DVC Pipelines on HPC Clusters

Most computational scientists don’t run pipelines on laptops. They run them on clusters with SLURM, PBS, or LSF schedulers. DVC integrates with SLURM through the srun wrapper, enabling cluster-native pipeline execution:

# Run a single pipeline stage with SLURM resource allocation
srun dvc repro -n relax --job slurm-job.sh

# Run all pipeline stages on the cluster
srun dvc repro

The --job flag tells DVC to run each stage through SLURM’s job scheduler, automatically handling:

• Cluster-specific resource allocation (memory, CPU, GPU nodes)
• Batch-mode execution without manual intervention
• Automatic job queuing for pipeline stages
• Integration with cluster-specific file systems and storage (Lustre, GPFS, BeeGFS)

arXiv 2505.06558v2 (September 2025) demonstrates this integration for materials-science workflows running DFT calculations on HPC clusters. The key advantage is that dvc repro becomes cluster-aware — if a stage fails or its dependencies change, SLURM handles the resource allocation for re-running only the necessary stages.

Cluster Storage Considerations

DVC’s remote storage works well with cluster file systems. You can configure DVC to use the cluster’s Lustre or GPFS filesystem as a DVC remote:

# Configure a cluster storage remote
dvc remote modify cluster_storage token_url "https://cluster-storage.example.com"

This avoids the overhead of copying data to/from external cloud storage and keeps the pipeline self-contained within the cluster’s storage hierarchy.

W3C PROV-JSON Provenance Tracking

While DVC tracks pipeline stages and data versions, it doesn’t natively produce a machine-actionable provenance graph. For that, you need a provenance standard — and the emerging format is W3C PROV-JSON.

yProv4ML (arXiv July 2025) implements W3C PROV-JSON provenance with minimal code modifications. It captures:

• Who ran the experiment (user identity)
• What data and code were used (source provenance)
• How the experiment ran (execution provenance)
• Why (optional motivation, tied to research objectives)

The output is a PROV-JSON graph — a directed graph where nodes represent entities (files, parameters, code commits) and edges represent relationships (derived-from, produced-by, used-by). This format enables:

• Interoperability across provenance systems: PROV-JSON is machine-readable and can be queried with SPARQL or graph databases
• FAIR data compliance: PROV-JSON graphs satisfy FAIR principles by documenting data lineage and reuse conditions
• Cross-institutional collaboration: PROV-JSON is a W3C standard, so provenance graphs from different institutions are compatible

PROV-JSON vs RO-Crate

It’s useful to distinguish PROV-JSON from RO-Crate, since both appear in reproducibility discussions:

Aspect	RO-Crate	PROV-JSON
Purpose	Package format for rich metadata	Data format for provenance graphs
Scope	Whole dataset + associated resources	Execution lineage and dependencies
Format	JSON-LD	JSON (PROV standard)
Interoperability	Self-contained package	Graph-based, queryable
Relationship	Complementary, not competing

They address different layers of reproducibility. RO-Crate packages your dataset with metadata. PROV-JSON tracks the computational lineage of how that dataset was produced. Using both together provides full reproducibility: “here is the data” (RO-Crate) plus “here is how it was produced” (PROV-JSON).

When to Use DVC vs DataLad vs PROV-Compliant Libraries

Choosing between DVC, DataLad, and PROV-compliant libraries depends on your workflow patterns:

Criterion	DVC	DataLad	PROV Libraries
Primary use case	Pipeline-oriented experiments	Long-term data curation	Machine-actionable provenance
Data size	Large files (models, simulations)	Large distributed datasets	N/A (metadata-focused)
Execution model	Stage-based pipeline (dvc repro)	Command-based (datalad run)	Decorator-based
HPC integration	srun dvc repro	Native git-annex + SSH	Configurable
Provenance output	.dvc files + Git history	Git-annex symlinks + git log	PROV-JSON graph
Best for	Materials-science pipelines, parameter sweeps	Long-term repository curation, BIDS compliance	FAIR data, cross-institutional provenance

DVC is right for you if:

• Your workflow has multiple dependent stages (relaxation → property calculation → visualization)
• You need parameter sweeps with automatic re-running
• You want to track which parameter combination produced the lowest-energy result

DataLad is right for you if:

• You’re curating a long-term dataset repository
• You need git-annex symlinks for efficient large-file handling
• You’re working with distributed datasets across institutions
• You need BIDS compliance (common in neuroimaging)

PROV-compliant libraries are right for you if:

• You need machine-actionable provenance for FAIR compliance
• You want a queryable provenance graph for data lineage
• You’re collaborating across institutions with different provenance systems
• You’re publishing to a FAIR data repository

Putting It Together — A Practical Workflow

Here’s a practical workflow pattern that combines all the concepts above:

# 1. Initialize the repository
git init
dvc init

# 2. Add the data
dvc add data/raw_crystal_structure.xyz

# 3. Run the pipeline
dvc repro

# 4. Check the results
dvc metrics show results/energies.txt

# 5. Run experiments with different parameters
dvc exp run -S sim.tolerance=1e-8 --set-param sim.cutoff_energy=550

# 6. Push data and experiments to remote
dvc push
dvc exp push
git push origin main

# 7. (Optional) Generate PROV-JSON provenance
python src/provenance.py --output provenance.json

This pattern ensures that every simulation result can be traced back to its exact data version, code commit, and parameter file. The PROV-JSON output (step 7) adds a machine-actionable provenance layer on top of the DVC pipeline.

Common Mistakes — What to Avoid

Mistake 1: Versioning everything — Don’t run dvc add on every output file. Only track files that are inputs to subsequent stages or that represent final results. Intermediate files (like raw force arrays) should stay in Git or be excluded entirely. Over-adding creates pipeline noise.

Mistake 2: Hardcoding paths — Don’t use absolute paths in dvc.yaml. Use relative paths so the pipeline works on different machines and cluster configurations.

Mistake 3: Mixing data types — Don’t put simulation parameters and model checkpoints in the same .dvc file. Keep data types separate to avoid cache conflicts.

Mistake 4: Ignoring provenance — DVC files alone aren’t sufficient for FAIR compliance. If your institution requires provenance graphs, pair DVC with PROV-JSON output (yProv4ML) or RO-Crate packaging.

Mistake 5: Forgetting params.yaml — If you hardcode parameters in your scripts instead of isolating them in params.yaml, DVC’s parameter tracking becomes useless. Always use params.yaml for simulation parameters.

Summary

DVC transforms scientific workflows from fragile, memory-dependent processes into deterministic, dependency-aware pipelines. The key insight is that dvc.yaml pipeline files and params.yaml parameter isolation create “versioned reproducibility” — the ability to reconstruct not just a result, but the exact experimental state (data version + code commit + parameter file) that produced it. This goes beyond containers (which freeze code environments) and beyond Git (which can’t handle large datasets).

For HPC integration, SLURM’s srun wrapper enables cluster-native pipeline execution without manual intervention. For provenance, W3C PROV-JSON (yProv4ML) emerges as an interoperable format complementary to RO-Crate packaging, enabling machine-actionable data lineage that satisfies FAIR compliance requirements.

The choice between DVC, DataLad, and PROV-compliant libraries depends on your workflow patterns: DVC for pipeline-oriented experiments, DataLad for long-term data curation, and PROV libraries for machine-actionable provenance graphs. Most simulation teams benefit from using DVC for pipeline execution and PROV-JSON for provenance tracking — the two tools are complementary, not competing.

---

Related Guides

• Reproducibility Beyond Containers — Covers DVC basics, RO-Crate provenance packaging, and container-based reproducibility patterns
• Managing Large-Scale PDE Problems — Discusses cluster resource management and HPC workflows
• Verification vs Validation in Scientific Simulations — Explains the broader reproducibility context for simulation results