Data Reproducibility

Data Reproducibility

Identity

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Data Reproducibility

Patterns

Environment Management

Name

Reproducible Computational Environments

Description

Ensure exact environment reproduction

When

Setting up any computational experiment

Pattern

Docker for complete environment isolation

FROM python:3.11.4-slim@sha256:abc123... # Pin digest

Pin all dependencies with hashes

COPY requirements.lock . RUN pip install --no-cache-dir -r requirements.lock

Set deterministic environment variables

ENV PYTHONHASHSEED=0 ENV CUBLAS_WORKSPACE_CONFIG=:4096:8

requirements.lock format:

numpy==1.24.3 --hash=sha256:abc...

pandas==2.0.1 --hash=sha256:def...

Conda alternative:

conda env export --from-history > environment.yml

conda-lock lock -f environment.yml

Seed Management

Name

Random Seed Management

Description

Control all sources of randomness

Pattern

import random import numpy as np import torch import os

def set_all_seeds(seed: int) -> dict: """Set ALL random seeds for reproducibility.""" random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) torch.cuda.manual_seed_all(seed) torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False os.environ['PYTHONHASHSEED'] = str(seed)

return {"seed": seed, "timestamp": datetime.utcnow().isoformat()}

Data Versioning

Name

Data Version Control

Description

Track data changes alongside code

Pattern

DVC (Data Version Control) setup

dvc init

dvc remote add -d storage s3://bucket/data

Track large files

dvc add data/training.csv

git add data/training.csv.dvc .gitignore

git commit -m "Add training data"

dvc push

.dvc file contains hash:

md5: abc123...

outs:

- md5: def456...

path: data/training.csv

To reproduce:

git checkout <commit>

dvc checkout

Experiment Manifest

Name

Experiment Manifest Creation

Description

Document everything needed to reproduce

Pattern

import hashlib import subprocess import json

def create_manifest(experiment_dir: str) -> dict: return { "timestamp": datetime.utcnow().isoformat(), "git_commit": subprocess.check_output( ["git", "rev-parse", "HEAD"] ).decode().strip(), "git_dirty": bool(subprocess.check_output( ["git", "status", "--porcelain"] )), "python_version": sys.version, "platform": platform.platform(), "seeds": {"numpy": 42, "torch": 42, "random": 42}, "data_hash": hash_directory(f"{experiment_dir}/data"), "config": yaml.safe_load(open(f"{experiment_dir}/config.yaml")), }

Save with results

results["_provenance"] = manifest json.dump(results, open("results.json", "w"))

Anti-Patterns

Hardcoded Paths

Name

Hardcoded File Paths

Problem

pd.read_csv('C:/Users/me/data.csv')

Solution

DATA_DIR = Path(os.environ.get('DATA_DIR', './data')) df = pd.read_csv(DATA_DIR / 'data.csv')

Missing Seeds

Name

Undocumented Random Seeds

Problem

Results change each run

Solution

Set and log all seeds before any random operations

Data Reproducibility - Sharp Edges

Floating Point Non-determinism Across Hardware

Id

floating-point-nondeterminism

Severity

critical

Summary

Same code, same seed, different results on different GPUs

Symptoms

• Results differ between local and cloud
• CI produces different numbers than laptop
• Model checkpoints don't reproduce exactly

Why

GPU operations often use non-deterministic algorithms for speed. Different hardware has different floating point precision. Order of operations affects floating point results.

Gotcha

Set seed everywhere

torch.manual_seed(42)

But still get different results!

GPU operations are non-deterministic by default

cudnn autotuning picks fastest (not reproducible) algorithm

Solution

torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False torch.use_deterministic_algorithms(True)

Note: Some ops don't have deterministic implementations

Implicit System Dependencies

Id

implicit-dependencies

Severity

critical

Summary

pip freeze misses system libraries your code depends on

Symptoms

• Works on your machine, fails on fresh install
• Cryptic import errors about missing .so files
• Different numerical results on different systems

Why

Python packages often wrap system libraries (BLAS, LAPACK, OpenSSL). pip/conda can't capture system-level dependencies. Different systems have different versions installed.

Solution

1. Use Docker for complete isolation 2. Document system requirements in README 3. Use conda for scientific packages (bundles system libs) 4. Test in clean environment before publishing

Data Changed But Nobody Noticed

Id

data-drift-unnoticed

Severity

high

Summary

Upstream data source changed, breaking reproducibility

Symptoms

• Model performance suddenly drops
• Can't reproduce old results with current data
• Results from paper don't match current code

Solution

1. Version data with DVC or similar 2. Hash all input data in manifests 3. Archive exact dataset used for publications 4. Never depend on mutable data sources for research

Timestamps Introduce Hidden Randomness

Id

timestamp-randomness

Severity

high

Summary

datetime.now() in features breaks reproducibility

Symptoms

• Different results when run at different times
• Time-based features change between runs

Solution

Bad

df['age'] = (datetime.now() - df['birth_date']).days / 365

Good: Use fixed reference date

REFERENCE_DATE = datetime(2024, 1, 1) df['age'] = (REFERENCE_DATE - df['birth_date']).days / 365

Python Dict/Set Ordering Was Random Before 3.7

Id

order-dependent-hashing

Severity

medium

Summary

Hash randomization affects iteration order

Symptoms

• Different feature order in older Python
• PYTHONHASHSEED not set

Solution

os.environ['PYTHONHASHSEED'] = '0'

Or use Python 3.7+ where dicts maintain insertion order

Data Reproducibility - Validations

Random Operations Without Seed

Id

no-seed-before-random

Severity

error

Type

regex

Pattern

• np\.random\.(rand|randn|choice)(?![\s\S]{0,200}seed)
• random\.(shuffle|choice)(?![\s\S]{0,200}seed)

Message

Set random seed before any random operations for reproducibility.

Fix Action

np.random.seed(42) or random.seed(42)

Applies To

• */.py

Hardcoded Absolute Paths

Id

hardcoded-absolute-path

Severity

warning

Type

regex

Pattern

• read_csv\(['"]C:|read_csv\(['"]D:
• read_csv\(['"]Users/|read_csv\(['"]/home/

Message

Hardcoded paths break reproducibility on other systems.

Fix Action

Use relative paths or environment variables

Applies To

• */.py

Unpinned Package Versions

Id

unpinned-dependencies

Severity

warning

Type

regex

Pattern

• ^[a-z][a-z0-9-]*$
• pip install (?!.*==)

Message

Pin exact versions for reproducibility: package==1.2.3

Applies To

• */requirements.txt
• */.sh

CUDA Without Deterministic Mode

Id

no-deterministic-cuda

Severity

warning

Type

regex

Pattern

• torch\.cuda(?![\s\S]{0,300}deterministic)

Message

Set torch.backends.cudnn.deterministic = True for reproducibility.

Applies To

• */.py

Missing Experiment Manifest

Id

no-experiment-manifest

Severity

info

Type

regex

Pattern

• def.*experiment(?![\s\S]{0,500}manifest|provenance|version)

Message

Log environment, seeds, and versions for reproducibility.

Applies To

• */.py

Current Datetime in Feature Engineering

Id

datetime-now-in-features

Severity

warning

Type

regex

Pattern

• datetime\.now\(\).feature|feature.datetime\.now

Message

Using current time in features breaks reproducibility.

Fix Action

Use a fixed reference date instead

Applies To

• */.py