Experimental Design

Experimental Design

Identity

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Experimental Design

Patterns

Factorial Design

Name

Factorial Experimental Design

Description

Test multiple factors efficiently

When

Need to study multiple variables and their interactions

Pattern

from itertools import product import pandas as pd from pyDOE2 import fullfact, ff2n, fracfact

Full factorial: All combinations of factors

def full_factorial_design(factors: dict) -> pd.DataFrame: """ Full factorial design - tests ALL combinations.

factors = { 'temperature': [100, 150, 200], 'pressure': [1, 2], 'catalyst': ['A', 'B'] }

Results in 3 x 2 x 2 = 12 experiments

""" names = list(factors.keys()) levels = [factors[name] for name in names] combinations = list(product(*levels))

return pd.DataFrame(combinations, columns=names)

2^k factorial (two levels per factor)

def two_level_factorial(n_factors: int) -> np.ndarray: """ 2^k factorial design with coded levels (-1, +1). Efficient for screening many factors. """ return ff2n(n_factors)

Fractional factorial for many factors

def fractional_factorial(design_string: str) -> np.ndarray: """ Fractional factorial - fewer runs, some aliasing.

'a b c ab' = 2^(3-1) design, 4 runs instead of 8 """ return fracfact(design_string)

Example: Screening 7 factors in 8 runs

Full factorial would need 2^7 = 128 runs

screening = fracfact('a b c d e f g') # Resolution III

Why

Factorials reveal interaction effects that one-at-a-time can't detect

Blocking Design

Name

Blocking to Control Confounds

Description

Account for known sources of variation

Pattern

import numpy as np from scipy.stats import f_oneway

def randomized_block_design( treatments: list, blocks: list, n_per_cell: int = 1 ) -> pd.DataFrame: """ Randomized Complete Block Design (RCBD).

Blocks account for known nuisance variation. Example: Days as blocks, treatments randomized within each day. """ design = [] for block in blocks: block_treatments = treatments.copy() np.random.shuffle(block_treatments)

for treatment in block_treatments: for _ in range(n_per_cell): design.append({ 'block': block, 'treatment': treatment })

return pd.DataFrame(design)

Latin Square: Block on two factors

def latin_square(n: int) -> np.ndarray: """ Latin Square design - block on row AND column. Each treatment appears once in each row and column. """ square = np.zeros((n, n), dtype=int) for i in range(n): for j in range(n): square[i, j] = (i + j) % n return square

Why

Blocking increases precision by removing known variation

Sample Size Design

Name

Sample Size Determination

Description

Calculate required samples before running experiment

Pattern

from statsmodels.stats.power import TTestIndPower, FTestAnovaPower

def required_sample_size( effect_size: float, alpha: float = 0.05, power: float = 0.80, design: str = "two_group" ) -> int: """ Calculate sample size for desired power.

effect_size: Cohen's d for t-test, f for ANOVA """ if design == "two_group": analysis = TTestIndPower() n = analysis.solve_power( effect_size=effect_size, alpha=alpha, power=power, ) elif design == "anova": analysis = FTestAnovaPower() n = analysis.solve_power( effect_size=effect_size, alpha=alpha, power=power, k_groups=3 # Number of groups ) return int(np.ceil(n))

Effect size conventions

EFFECT_SIZES = { 'small': 0.2, 'medium': 0.5, 'large': 0.8, }

Response Surface

Name

Response Surface Methodology

Description

Optimize continuous factors

When

Finding optimal settings for process parameters

Pattern

from pyDOE2 import ccdesign, bbdesign

def central_composite_design(n_factors: int) -> np.ndarray: """ Central Composite Design (CCD) for response surface. Combines factorial with star points and center points. """ return ccdesign(n_factors, center=(4, 4), face='circumscribed')

def box_behnken_design(n_factors: int) -> np.ndarray: """ Box-Behnken Design - 3 levels, no extreme corners. Good when extreme combinations are impractical. """ return bbdesign(n_factors)

Fit response surface model

def fit_response_surface(X, y): """Fit quadratic model with interactions.""" from sklearn.preprocessing import PolynomialFeatures from sklearn.linear_model import LinearRegression

poly = PolynomialFeatures(degree=2, include_bias=False) X_poly = poly.fit_transform(X) model = LinearRegression().fit(X_poly, y)

return model, poly

Why

RSM efficiently finds optimal conditions in fewer runs

Anti-Patterns

One At A Time

Name

One-Factor-at-a-Time (OFAT)

Problem

Change only one variable at a time

Miss ALL interaction effects

Requires many more runs for same information

Solution

Use factorial designs to capture interactions

No Randomization

Name

Running Experiments in Systematic Order

Problem

Run all treatment A first, then treatment B

Solution

Randomize run order to prevent time-based confounds

Experimental Design - Sharp Edges

One-Factor-At-A-Time Misses All Interactions

Id

ofat-misses-interactions

Severity

critical

Summary

OFAT requires more runs and can never detect interaction effects

Symptoms

• Changing one variable at a time
• Optimal found but doesn't work in practice
• Variables tested independently

Why

OFAT assumes factors are independent. In reality, factor A's effect often depends on factor B's level (interaction). Factorial designs detect these interactions.

OFAT also requires MORE total runs for the same information.

Gotcha

OFAT approach (BAD)

Vary temperature: 100, 150, 200 (best: 200)

Vary pressure at T=200: 1, 2, 3 (best: 2)

Conclude: T=200, P=2 is optimal

Reality: Optimal might be T=150, P=3

Interaction: High temp needs low pressure

OFAT could never find this!

Solution

Use factorial design: from pyDOE2 import ff2n design = ff2n(2) # 2 factors, 4 runs

Tests all combinations including interaction

Systematic Run Order Creates False Effects

Id

no-randomization-confound

Severity

critical

Summary

Running all treatment A first biases results with time effects

Symptoms

• Experiments run in convenient order
• All replicates of condition X run consecutively
• Unexpected 'treatment effects'

Why

Equipment drifts over time. Operators get fatigued. Materials age. If all treatment A runs are first, time effects get confounded with treatment effects.

Solution

np.random.shuffle(run_order)

Run experiments in random order

Or use blocking if time effects are expected

No Center Points to Check for Curvature

Id

missing-center-points

Severity

high

Summary

Linear model fit when relationship is actually curved

Symptoms

• 2-level factorial shows no effect
• Optimal settings don't work as expected
• Model has poor prediction accuracy

Solution

Add center points to 2-level factorial

design = np.vstack([ ff2n(2), # Corner points [[0, 0], [0, 0]] # Center points (replicates) ])

If center point response differs from corner average,

curvature exists → need higher-order model

Fractional Factorial Aliasing Confusion

Id

resolution-aliasing

Severity

high

Summary

In fractional designs, some effects are mathematically confounded

Symptoms

• Using fractional factorial without understanding resolution
• Main effect 'significant' but actually aliased with interaction

Why

Fractional factorials trade runs for information. Resolution III: Main effects aliased with 2-way interactions Resolution IV: Main effects clear, 2-way aliased with 2-way Resolution V: Main and 2-way clear, aliased with 3-way

Solution

Check resolution before interpreting

Use Resolution V or higher for main + 2-way interactions

Or use foldover to de-alias

No Replication = No Error Estimate

Id

inadequate-replication

Severity

medium

Summary

Without replicates, can't distinguish signal from noise

Symptoms

• One run per condition
• Every effect looks 'significant'
• No pure error estimate

Solution

Include replicates, especially at center points

design = full_factorial(factors) design = pd.concat([design, design]) # Duplicate for replicates

Or at minimum, replicate center points

Experimental Design - Validations

One-Factor-At-A-Time Pattern

Id

ofat-pattern

Severity

warning

Type

regex

Pattern

• for.in.values:\s\n.test.*single
• vary.one.at.*time

Message

Consider factorial design instead of OFAT for interaction effects.

Applies To

• */.py

Missing Run Order Randomization

Id

no-randomization

Severity

warning

Type

regex

Pattern

• for.treatment.in.*order(?![\s\S]{0,200}shuffle|random)

Message

Randomize run order to prevent time-based confounds.

Fix Action

np.random.shuffle(run_order)

Applies To

• */.py

Design Without Power Analysis

Id

no-power-analysis

Severity

warning

Type

regex

Pattern

• n.=.[1-9]\d?(?!\d).*experiment

Message

Calculate required sample size with power analysis before running.

Applies To

• */.py

Factorial Without Replication

Id

no-replication

Severity

info

Type

regex

Pattern

• factorial.n_rep.=.*1|fullfact(?![\s\S]{0,200}replicate)

Message

Include replicates to estimate pure error.

Applies To

• */.py

Potential Nuisance Variable Not Blocked

Id

missing-blocking

Severity

info

Type

regex

Pattern

• day|batch|operator(?![\s\S]{0,300}block)

Message

Consider blocking on known nuisance variables (day, batch, etc.).

Applies To

• */.py