Ml Systems Engineer Rl Engineering

Machine Learning Systems Engineer, RL Engineering

When to Use

• Design RL training platform — controllers, workers, resource scheduling
• Implement rollout collection — vectorized envs, async actors, trajectory buffers
• Operate distributed training — data parallel, parameter servers, gradient sync patterns
• Manage replay buffers — prioritization, storage, sampling at scale
• Wire checkpointing — policy/value nets, optimizer state, resume after preemption
• Integrate experiment tracking — seeds, configs, metric schemas, artifact lineage
• Connect simulators — Gymnasium-style APIs, custom env servers, batch stepping
• Export policies for batch eval or downstream inference path
• Debug training instability — NaNs, reward scale, worker desync, straggler GPUs
• Plan GPU/memory layout for actor vs learner processes

When NOT to Use

• Churn models, A/B tests, classical supervised pipelines → data-scientist
• Production LLM features, agents, RAG → ai-engineer
• Safeguard/moderation inference gateways → ml-infrastructure-engineer-safeguards
• Safety classifier research → ml-research-engineer-safeguards
• CI/CD and generic K8s ops → devops, cluster-deployment-engineer
• DC-wide GPU supply programs → data-center-compute-supply-efficiency
• HTTP API p99 without RL training context → performance-engineer
• RL algorithm theory only (no systems) → ai-researcher for literature; stay systems-focused here

Related skills

Core Workflows

1. RL systems framing

Env contract, on/off-policy, scale targets.

See `references/rl_systems_framing.md`.

2. Training platform architecture

Controllers, workers, scheduling.

See `references/training_platform_architecture.md`.

3. Environments and rollouts

Vectorization, trajectory format.

See `references/environments_rollouts.md`.

4. Replay, checkpoints, experiments

Buffers, resume, tracking.

See `references/replay_checkpoints_experiments.md`.

5. Evaluation and policy export

Eval harness, deployment handoff.

See `references/evaluation_policy_export.md`.

6. Reliability and observability

Stability, metrics, incident debug.

See `references/reliability_observability_rl.md`.

Outputs

• Architecture doc — actor/learner topology, data flow, failure domains
• Env API spec — observation, action, reward, reset, seed semantics
• Runbook — launch, resume, preempted job recovery, scale-out
• Config template — hyperparameters + infra knobs versioned together
• Metric dashboard spec — reward, length, KL, GPU, steps/sec, queue depth
• Policy export package — weights, normalization stats, eval report

Principles

• Reproducibility — seed envs, log config hash, pin sim versions
• Separate rollout from learn — scale collectors and learners independently
• Deterministic resume — checkpoint includes optimizer and buffer cursor when needed
• Observe the MDP — log reward components, not only scalar return
• Fail fast on desync — version mismatch between workers is a top incident class

Environments and rollouts

Table of contents

1. Env API contract 2. Vectorization 3. Trajectory schema 4. Common failures

Env API contract

Standardize:

• reset(seed) → obs, info
• step(action) → obs, reward, terminated, truncated, info
• action_space / observation_space metadata exposed to controller
• Determinism — which ops are stochastic; how seeds propagate

Version env binary alongside training config — mismatch causes silent metric drift.

Vectorization

Target steps/sec per core in benchmarks before full training run.

Trajectory schema

Store per step (minimum):

• obs, action, reward, done, log_prob (if on-policy)
• value, advantage (if computed on actor)
• env_id, episode_id, global_step
• Optional: info keys for reward decomposition

Fixed schema enables replay ingestion and debugging tools.

Common failures

Log episode statistics — length, terminal reason, reward histogram.

Evaluation and policy export

Table of contents

1. Eval harness 2. Deterministic eval 3. Policy export 4. Handoff to serving

Eval harness

Separate training from eval jobs:

• Fixed seed suite; frozen env version
• Deterministic policy (eval mode, no exploration noise)
• Report mean/std return over N episodes per seed
• Compare checkpoints on same harness — learning curves

Schedule eval on checkpoint events, not only end of run.

Deterministic eval

• Fix seeds list in config
• Disable domain randomization if used in training
• Same observation normalization as training checkpoint
• Log videos/traces sparingly — storage cost

Regression gate: eval return must not drop > X% vs prior champion.

Policy export

Artifacts:

• TorchScript / ONNX / native weights (document format)
• Normalization constants
• Action post-processing (clip, discrete mapping)
• Expected obs shape and dtype

Include latency benchmark on reference hardware for downstream teams.

Handoff to serving

Document exploration off in production; epsilon schedule not applied.

For HTTP/gateway deployment → coordinate with inference infra skills; this skill owns export correctness.

Reliability and observability (RL training)

Table of contents

1. Health signals 2. Stability incidents 3. Debugging playbook 4. Cost controls

Health signals

Dashboard: one pane training, one pane infra.

Stability incidents

Post-incident: save last good checkpoint before corrupt state.

Debugging playbook

1. Reproduce on single actor + single learner locally 2. Log one full episode with info reward terms 3. Compare policy weights hash actor vs learner 4. Plot reward vs global step for subset of seeds 5. Verify env version in container image digest

Escalate algorithm change to research; fix infra desync here.

Cost controls

• Autoscale actors down after target steps
• Kill jobs over GPU-hour budget without checkpoint progress
• Use spot with aggressive checkpointing
• Profile sim — often cheaper to optimize env than add GPUs

Align long-range GPU needs with data-center-compute-supply-efficiency when capacity-bound.

Replay, checkpoints, and experiments

Table of contents

1. Replay buffer 2. Checkpoints 3. Experiment tracking 4. Config management

Replay buffer

Monitor buffer fill rate and sample age (staleness for off-policy).

Checkpoints

Include for resume:

• Policy (+ target nets if applicable)
• Optimizer state
• Global step / epoch
• Observation normalizer stats
• RNG states (python, numpy, torch, cuda)
• Replay cursor or on-policy batch offset

Store in versioned object store; manifest JSON with git SHA and env version.

Test resume continues learning curve — not flat restart.

Experiment tracking

Log at minimum:

• episode_return, length (mean, std, percentiles)
• policy_loss, value_loss, entropy, kl (algorithm-specific)
• steps_per_sec, actor count, learner GPU util
• grad_norm, NaN flags

Tag runs: team, algorithm, env version, config hash.

Config management

Single source:

algorithm: ...
env: ...
infra:
  num_actors: ...
  learner_gpus: ...
  checkpoint_every_steps: ...

Never split infra knobs from algorithm hparams across unlinked files.

RL systems framing

Table of contents

1. Problem shape 2. On-policy vs off-policy 3. Scale dimensions 4. Non-goals

Problem shape

Capture before building infra:

Document observation normalization — running stats owned by trainer or env.

On-policy vs off-policy

Wrong pattern → wasted GPUs (replay-bound vs rollout-bound).

Scale dimensions

Estimate:

• Steps/sec target across fleet
• Actor count vs learner GPUs
• Observation size × batch → network bandwidth
• Checkpoint frequency vs storage cost
• Preemption rate on spot nodes → resume SLA

Non-goals

Route elsewhere:

• Reward function design (research/product) — systems expose hooks only
• Hyperparameter search orchestration — may share platform but not core RL path
• Supervised pretraining — separate pipeline unless warm-start checkpoint

Training platform architecture

Table of contents

1. Topology patterns 2. Components 3. Scheduling 4. Networking

Topology patterns

Actors (CPU/GPU-light) → Trajectory stream → Learner (GPU-heavy)
                              ↓
                         Checkpoint store

Components

Use frameworks (Ray RLlib, CleanRL distributed, custom) but document ownership of each box.

Scheduling

• GPU affinity — learners on GPU nodes; actors on CPU unless inference-heavy
• Gang scheduling — avoid partial allocation leaving job stuck
• Spot/preemptible — checkpoint every N minutes; graceful SIGTERM handler
• Queue fairness — team quotas on GPU pool

Coordinate cluster primitives with cluster-deployment-engineer.

Networking

• Compress observations if wide (JPEG, fp16, zstd)
• gRPC/IPC for local; Redis/pub-sub only if measured need
• Watch NCCL for learner multi-GPU vs actor traffic on same NIC
• Set timeouts on stale actors — do not block learner indefinitely