Parameter Optimization

Parameter Optimization

Goal

Provide a workflow to design experiments, rank parameter influence, and select optimization strategies for materials simulation calibration.

Requirements

• Python 3.8+
• No external dependencies (uses Python standard library only)

Inputs to Gather

Before running any scripts, collect from the user:

Decision Guidance

Choosing a DOE Method

Is dimension <= 3 AND full coverage needed?
├── YES → Use factorial
└── NO → Is sensitivity analysis the goal?
    ├── YES → Use quasi-random (preferred; "sobol" is accepted but deprecated)
    └── NO → Use lhs (Latin Hypercube)

Choosing an Optimizer

Is dimension <= 5 AND budget <= 100?
├── YES → Bayesian Optimization
└── NO → Is dimension <= 20?
    ├── YES → CMA-ES
    └── NO → Random Search with screening

Script Outputs (JSON Fields)

Workflow

1. Generate DOE with scripts/doe_generator.py 2. Run simulations at DOE sample points (user's responsibility) 3. Summarize sensitivity with scripts/sensitivity_summary.py 4. Choose optimizer using scripts/optimizer_selector.py 5. (Optional) Fit surrogate with scripts/surrogate_builder.py

CLI Examples

# Generate 20 LHS samples for 3 parameters
python3 scripts/doe_generator.py --params 3 --budget 20 --method lhs --json

# Rank parameters by sensitivity scores
python3 scripts/sensitivity_summary.py --scores 0.2,0.5,0.3 --names kappa,mobility,W --json

# Get optimizer recommendation for 3D problem with 50 eval budget
python3 scripts/optimizer_selector.py --dim 3 --budget 50 --noise low --json

# Build surrogate model from simulation data
python3 scripts/surrogate_builder.py --x 0,1,2 --y 10,12,15 --model rbf --json

Conversational Workflow Example

User: I need to calibrate thermal conductivity and diffusivity for my FEM simulation. I can run about 30 simulations.

Agent workflow: 1. Identify 2 parameters → --params 2 2. Budget is 30 → --budget 30 3. Use LHS for general exploration:

   python3 scripts/doe_generator.py --params 2 --budget 30 --method lhs --json

4. After user runs simulations and provides outputs, summarize sensitivity:

   python3 scripts/sensitivity_summary.py --scores 0.7,0.3 --names conductivity,diffusivity --json

5. Recommend optimizer:

   python3 scripts/optimizer_selector.py --dim 2 --budget 30 --noise low --json

Error Handling

Limitations

• Not for real-time optimization: Scripts provide recommendations, not live optimization loops
• Surrogate is a placeholder: surrogate_builder.py computes basic metrics; replace with actual model for production
• No automatic simulation execution: User must run simulations externally and provide results

References

• references/doe_methods.md - Detailed DOE method comparison
• references/optimizer_selection.md - Optimizer algorithm details
• references/sensitivity_guidelines.md - Sensitivity analysis interpretation
• references/surrogate_guidelines.md - Surrogate model selection

Version History

• v1.1.0 (2024-12-24): Enhanced documentation, decision guidance, conversational examples
• v1.0.0: Initial release with core scripts

Design of Experiments (DOE) Methods

Overview

Design of Experiments (DOE) creates sample points in parameter space to efficiently explore simulation behavior. The goal is to maximize information gained per simulation run.

Method Comparison

---

Latin Hypercube Sampling (LHS)

How It Works

LHS divides each parameter range into n equal intervals, then places exactly one sample in each interval per dimension. This ensures no two samples share the same row or column in any 2D projection.

Example: 5 samples in 2D

    1.0 |     x     |     |     | x   |
    0.8 |  x  |     |     |     |     |
    0.6 |     |     |     | x   |     |
    0.4 |     |     | x   |     |     |
    0.2 |     | x   |     |     |     |
        +-----+-----+-----+-----+-----+
          0.2   0.4   0.6   0.8   1.0

When to Use

• General parameter exploration
• Moderate dimensions (3-20 parameters)
• Unknown response surface shape
• Limited simulation budget

When to Avoid

• Need exact corner/edge coverage
• Very low dimensions where factorial is feasible
• Reproducibility required (use fixed seed)

Sample Size Recommendations

---

Sobol Sequences (Quasi-Random)

How It Works

Sobol sequences are low-discrepancy sequences that fill space more uniformly than random sampling. Points are generated deterministically using bit operations on direction numbers.

Example: Sobol vs Random in 2D (16 points)

Sobol (uniform fill)        Random (clusters/gaps)
+---+---+---+---+          +---+---+---+---+
| x |   | x |   |          |   | x |   | x |
+---+---+---+---+          +---+---+---+---+
|   | x |   | x |          | x |   |   |   |
+---+---+---+---+          +---+---+---+---+
| x |   | x |   |          |   | x | x | x |
+---+---+---+---+          +---+---+---+---+
|   | x |   | x |          |   |   | x |   |
+---+---+---+---+          +---+---+---+---+

When to Use

• Sensitivity analysis (Sobol indices)
• Need uniform coverage guarantees
• Reproducible experiments
• Sequential sampling (can add points incrementally)

When to Avoid

• Very high dimensions (>15, curse of dimensionality)
• Need stratified random sampling

Sample Size Recommendations

For Sobol sensitivity analysis, use N * (d + 2) samples where:

• N = base sample size (64, 128, 256, 512, 1024)
• d = number of parameters

---

Full Factorial Design

How It Works

Factorial designs test all combinations of discrete parameter levels. For k levels across d dimensions, this produces k^d samples.

Example: 3 levels, 2 dimensions = 9 samples

    High  | x   x   x |
          |           |
    Med   | x   x   x |
          |           |
    Low   | x   x   x |
          +-----------+
            L   M   H

When to Use

• Low dimensions (1-3 parameters)
• Need exact corner coverage
• Testing parameter interactions
• Screening designs

When to Avoid

• High dimensions (exponential growth)
• Continuous parameters with smooth response
• Limited budget

Sample Count Growth

---

Decision Flowchart

START
  |
  v
Is d <= 3 AND need corner coverage?
  |
  +-- YES --> FACTORIAL
  |
  +-- NO --> Is sensitivity analysis the goal?
               |
               +-- YES --> SOBOL
               |
               +-- NO --> LHS

Implementation Notes

The doe_generator.py script in this skill:

• Uses standard library only (no scipy/numpy required)
• LHS: True Latin Hypercube with random permutations
• Sobol: Simplified quasi-random (for full Sobol, use scipy.stats.qmc)
• Factorial: Full grid with level interpolation

Optimizer Selection Guide

Overview

Choosing the right optimization algorithm depends on problem characteristics: dimensionality, evaluation budget, noise level, and whether gradients are available.

Algorithm Comparison

---

Bayesian Optimization (BO)

How It Works

BO builds a probabilistic surrogate model (typically Gaussian Process) of the objective function and uses an acquisition function to balance exploration vs exploitation.

Iteration loop:
1. Fit GP to observed (x, y) pairs
2. Compute acquisition function (EI, UCB, PI)
3. Find x_next = argmax(acquisition)
4. Evaluate y_next = f(x_next)
5. Add (x_next, y_next) to dataset
6. Repeat until budget exhausted

When to Use

• Expensive simulations (minutes to hours per evaluation)
• Low to moderate dimensions (d <= 10)
• Smooth or moderately noisy objectives
• Small evaluation budgets (20-100)

When to Avoid

• High dimensions (d > 15) - GP scales as O(n^3)
• Very noisy objectives without noise model
• Fast evaluations where random search suffices

Key Hyperparameters

Libraries

• botorch (PyTorch-based, production-ready)
• scikit-optimize (sklearn-compatible)
• GPyOpt (flexible but less maintained)

---

CMA-ES (Covariance Matrix Adaptation)

How It Works

CMA-ES is an evolutionary strategy that adapts the search distribution covariance matrix based on successful mutations.

Iteration loop:
1. Sample lambda offspring from N(m, sigma^2 * C)
2. Evaluate and rank offspring
3. Update mean m toward better solutions
4. Update covariance C and step-size sigma
5. Repeat until convergence

When to Use

• Moderate to high dimensions (5-100)
• Non-convex, multimodal landscapes
• No gradients available
• Budget of 100+ evaluations

When to Avoid

• Very small budgets (< 50)
• Low dimensions where BO is more efficient
• Highly noisy objectives

Key Hyperparameters

Libraries

• cma (official Python package)
• pycma (pure Python)
• nevergrad (includes CMA-ES)

---

Gradient-Based Methods

When to Use

• Gradients available (adjoint methods, autodiff)
• Smooth objectives with single optimum
• Very large dimensions

Algorithms

Convergence Criteria

Stop when ANY condition is met:
- ||grad|| < tol (gradient norm)
- |f_k - f_{k-1}| < ftol (function change)
- ||x_k - x_{k-1}|| < xtol (step size)
- k > max_iter (iteration limit)

---

Random Search

How It Works

Sample uniformly at random from the parameter space and keep the best result.

When to Use

• Very high dimensions (d > 50)
• Highly noisy objectives
• Large budgets (1000+ evaluations)
• Baseline comparison

Efficiency Note

Random search with n samples finds a point in the top 1/n fraction with probability 1 - (1 - 1/n)^n ≈ 0.63.

---

Decision Flowchart

START
  |
  v
Are gradients available AND noise is low?
  |
  +-- YES --> GRADIENT-BASED (L-BFGS-B)
  |
  +-- NO --> Is dimension <= 10 AND budget <= 100?
               |
               +-- YES --> BAYESIAN OPTIMIZATION
               |
               +-- NO --> Is dimension <= 100?
                            |
                            +-- YES --> CMA-ES
                            |
                            +-- NO --> RANDOM SEARCH

Handling Noise

Handling Constraints

Expected Evaluation Counts

Implementation Notes

The optimizer_selector.py script in this skill provides recommendations based on:

• Dimension
• Budget
• Noise level
• Presence of constraints

It returns the most suitable algorithm(s) with expected evaluation counts and notes about configuration.

Sensitivity Analysis Guidelines

Overview

Sensitivity analysis quantifies how input parameter variations affect simulation outputs. It helps identify the most influential parameters for calibration and uncertainty reduction.

Method Categories

---

Local Sensitivity (One-At-a-Time)

How It Works

Vary one parameter while holding others fixed at nominal values. Compute partial derivatives:

S_i = (dy/dx_i) * (x_i / y)  [normalized]

When to Use

• Quick screening
• Nearly linear response
• Local behavior near operating point

Limitations

• Misses parameter interactions
• Depends on chosen nominal point
• Invalid for nonlinear responses

---

Morris Method (Elementary Effects)

How It Works

Compute elementary effects by stepping through parameter space along random trajectories:

EE_i = [y(x + delta*e_i) - y(x)] / delta

Statistics computed:

• mu* (mean absolute EE): Overall importance
• sigma (std of EE): Nonlinearity/interactions

Interpretation

Sample Requirements

• Trajectories r: 10-50 (typically 20)
• Levels p: 4-8
• Total evaluations: r * (d + 1)

When to Use

• Moderate budgets
• Screening before detailed analysis
• Want to detect interactions

---

Sobol Indices (Variance-Based)

How It Works

Decompose output variance into contributions from each parameter and their interactions:

V(Y) = sum(V_i) + sum(V_ij) + ... + V_12...d

Indices:

• S_i (first-order): Main effect of parameter i
• S_Ti (total): Main + all interactions involving i

Interpretation

Sample Requirements

For Saltelli estimator:

• Base samples N: 512, 1024, 2048
• Total evaluations: N * (d + 2)

When to Use

• Quantitative variance attribution needed
• Sufficient budget for global analysis
• Nonlinear, interacting models

---

Interpreting Rankings

Score Thresholds

When Rankings Are Close

If top parameters have similar scores: 1. Check for interactions (S_Ti >> S_i) 2. Consider fixing less important parameters 3. Use higher sample sizes for better precision

Red Flags

---

Visualization Recommendations

Bar Charts

Plot parameters sorted by sensitivity score with confidence intervals.

kappa       ████████████████████ 0.52
mobility    ███████████ 0.28
W           ██████ 0.15
rho         ██ 0.05

Interaction Heatmaps

For second-order indices S_ij, use heatmap with parameters on both axes.

Scatter Plots

Plot output vs each input to visually confirm sensitivity rankings.

---

Common Pitfalls

---

Implementation Notes

The sensitivity_summary.py script in this skill:

• Takes pre-computed sensitivity scores as input
• Ranks parameters from most to least influential
• Flags if all sensitivities are very low
• Returns structured JSON for downstream use

For computing Sobol indices, use external tools:

• SALib (Python, comprehensive)
• sensitivity (R package)
• UQlab (MATLAB)

Surrogate Model Guidelines

Overview

Surrogate models (metamodels, emulators) approximate expensive simulations with fast-to-evaluate functions. They enable rapid optimization, sensitivity analysis, and uncertainty quantification.

Model Comparison

---

Polynomial Response Surface

How It Works

Fit polynomial of specified order to simulation data:

y = b0 + sum(b_i * x_i) + sum(b_ij * x_i * x_j) + ...

Orders:

• Linear: 1 + d terms
• Quadratic: 1 + d + d*(d+1)/2 terms

When to Use

• Very limited data (< 20 points)
• Smooth, nearly quadratic response
• Need interpretable coefficients
• Quick approximation

When to Avoid

• Highly nonlinear responses
• Extrapolation required
• High dimensions (coefficient explosion)

Sample Requirements

---

Radial Basis Functions (RBF)

How It Works

Interpolate using weighted sum of radial basis functions centered at data points:

y(x) = sum(w_i * phi(||x - x_i||))

Common kernels:

• Gaussian: exp(-r^2 / epsilon^2)
• Multiquadric: sqrt(1 + (epsilon*r)^2)
• Thin-plate spline: r^2 * log(r)

When to Use

• Exact interpolation required
• Moderate dimensions (< 20)
• No noise in data
• Complex response surfaces

When to Avoid

• Noisy data (will fit noise)
• Very large datasets (O(n^3) fitting)
• Need uncertainty estimates

Key Hyperparameters

---

Gaussian Process (Kriging)

How It Works

Model output as realization of Gaussian Process with specified mean and covariance (kernel):

y(x) ~ GP(m(x), k(x, x'))

Provides:

• Mean prediction: E[y(x)]
• Uncertainty: Var[y(x)]

When to Use

• Need uncertainty quantification
• Expensive simulations (BO framework)
• Smooth responses
• Sequential/adaptive sampling

When to Avoid

• Very large datasets (> 1000 points)
• High dimensions (> 15-20)
• Very fast simulations

Kernel Selection

Libraries

• GPyTorch (PyTorch, scalable)
• GPy (numpy, flexible)
• scikit-learn (simple API)

---

Neural Networks

When to Use

• Very large datasets (> 10000 points)
• High dimensions
• Complex nonlinear patterns
• Have GPU resources

When to Avoid

• Small datasets (< 100 points)
• Need uncertainty (without extra work)
• Interpretability required
• Training time is limited

---

Validation Strategies

Cross-Validation

Metrics

Red Flags

---

Adaptive Sampling

When initial surrogate is poor, add samples strategically:

---

Decision Flowchart

START
  |
  v
Is data noisy?
  |
  +-- YES --> Need uncertainty?
  |             |
  |             +-- YES --> GAUSSIAN PROCESS
  |             |
  |             +-- NO --> NEURAL NETWORK (if large data)
  |
  +-- NO --> Is response smooth and low-dim?
               |
               +-- YES --> Is data < 20 points?
               |             |
               |             +-- YES --> POLYNOMIAL
               |             |
               |             +-- NO --> RBF or GP
               |
               +-- NO --> RBF or GP with Matern kernel

---

Implementation Notes

The surrogate_builder.py script in this skill:

• Is a placeholder for demonstration
• Computes basic MSE metric only
• Does not fit actual models

For real surrogate modeling, use:

• scikit-learn (polynomial, GP, neural networks)
• scipy.interpolate (RBF)
• GPyTorch (scalable GPs)
• SMT (Surrogate Modeling Toolbox)