Sensor Fusion Engineer

Sensor Fusion Engineer

When to Use

• Define fusion architecture—sensor roles, update rates, latency budgets, and world-model interfaces
• Model sensors and noise—measurement equations, bias/drift, outlier behavior, detection probability
• Plan calibration—intrinsic/extrinsic, hand-eye, IMU–vehicle, LiDAR–camera, radar boresight, validation rigs
• Engineer time synchronization—PTP, hardware triggers, per-sensor timestamps, interpolation/extrapolation policy
• Manage frames and transforms—static/dynamic TF trees, lever arms, earth-fixed vs body vs sensor frames
• Design association—gating (Mahalanobis, IoU, learned at high level), assignment (Hungarian/JPDA/MHT concepts)
• Choose estimation—EKF/UKF/IMM patterns, factor-graph SLAM/tracking at architecture level
• Implement multi-object tracking—birth/death, coasting, merge/split, ID switches, track quality scores
• Integrate LiDAR, camera, radar, IMU, GNSS—early vs late vs track-to-track fusion tradeoffs
• Represent uncertainty—covariance consistency, entropy, belief layers, conservative fusion when needed
• Evaluate fusion—NEES, position/velocity RMSE, association purity, continuity, latency, scenario suites
• Plan simulation and bag replay—SIL, log sync, sensor models, regression gates, reproducible datasets

When NOT to Use

• Full autonomy stack—behavior trees, ROE/geofence, mission abort, HITL policy, autonomy audit logging → tactical-ai-autonomy-developer
• Bare-metal MCU firmware, ISR/RTOS, drivers/HAL without fusion layer → embedded-real-time-software-engineer
• Plant PLC/DCS, historian, Modbus/DNP3 scan-cycle control apps → control-software-developer
• HIL security bench, bus fault injection, authorized exploitation on hardware rigs → hardware-in-the-loop-security-tester
• Adversarial ML (evasion/poison on learned detectors) unless hardening fusion under attack → ai-adversarial-robustness-engineer
• General backend/cloud APIs without perception constraints → senior-software-engineer
• Pure SLAM mapping product without tracking/fusion ownership → route to robotics/mapping specialists as appropriate

Related skills

Core Workflows

1. Scope and fusion boundaries

Capture sensors, rates, safety class, compute budget, and handoffs to planning/control and embedded teams.

See `references/sensor_fusion_scope.md`.

2. Calibration, sync, and frames

Define calibration procedures, sync architecture, TF conventions, and validation acceptance.

See `references/calibration_sync_and_frames.md`.

3. Estimation, state, and uncertainty

Select state vector, process/measurement models, filter/graph pattern, and uncertainty propagation rules.

See `references/estimation_filters_and_state.md`.

4. Association and tracking

Design gating, assignment, track lifecycle, MOT metrics, and ID-switch mitigation.

See `references/multi_sensor_association_tracking.md`.

5. Multi-sensor fusion patterns

Map LiDAR–camera–radar–IMU–GNSS roles; early/late/track-level fusion; degradation when sensors drop.

See `references/lidar_camera_radar_fusion.md`.

6. Evaluation, simulation, and metrics

Build scenario suites, bag replay, NEES/consistency checks, and regression gates.

See `references/evaluation_simulation_and_metrics.md`.

Outputs

• Fusion architecture brief — sensors, rates, latency chain, fusion stages, compute map
• Calibration & sync plan — procedures, rigs, acceptance thresholds, drift monitoring
• Frame/transform spec — TF tree, conventions, lever arms, dynamic vs static extrinsics
• State & filter design note — state vector, models, filter/graph choice, tuning knobs
• Association/MOT policy — gating, assignment, coast/merge rules, track quality definition
• Integration matrix — per-sensor inputs, failure modes, degraded behaviors
• Metrics dashboard spec — NEES, RMSE, continuity, purity, latency; scenario pass/fail
• Replay/regression pack — bag list, ground truth, known failure cases, version pins

Principles

• Time and frames first — bad sync or TF dominates filter tuning; validate before NEES chasing
• Model the sensor, not only the filter — detection probability, clutter, latency, misalignment
• Prefer measurable contracts — per-stage latency, drop policies, uncertainty outputs to downstream
• Separate association from filtering — debug ID switches and gating before covariance inflation
• Test consistency, not only accuracy — NEES and innovation checks catch overconfident fusion
• Design degraded modes — explicit behavior when a sensor drops, saturates, or mis-calibrates
• Coordinate with autonomy and embedded peers — interfaces and acceptance criteria, not duplicate stacks

When to load references

Calibration, synchronization, and frames

Table of contents

1. Calibration types 2. Calibration workflow 3. Time synchronization 4. Coordinate frames 5. Validation checks

Calibration types

Record temperature and vibration sensitivity where relevant; plan recalibration triggers.

Calibration workflow

1. Define acceptance metrics — reprojection error (px), plane fit residual (m), yaw error (deg), velocity residual 2. Collect calibration sequences — coverage of FOV, range, motion excitation for IMU 3. Estimate offline — store parameters with version, date, rig ID, operator 4. Validate on hold-out — separate logs/scenes not used in solve 5. Monitor in field — online checks (reprojection drift, gravity direction, ground plane)

Maintain calibration artifact store: YAML/URDF snippets, not only binary blobs.

Time synchronization

Policies to document:

• Interpolation vs extrapolation for measurements arriving between filter updates
• Maximum time offset before reject measurement
• Out-of-order message handling in replay vs live

Coordinate frames

Standard conventions (adapt to program, document explicitly):

TF tree rules:

• Single authority for each transform; avoid duplicate publishers
• Static extrinsics in URDF; dynamic via state estimator where needed
• Lever arms from rotation center to each sensor phase center

Validation checks

Flag rolling shutter and motion distortion on cameras/LiDAR before blaming the filter.

Estimation, filters, and state

Table of contents

1. State vector patterns 2. Process and measurement models 3. Kalman-family filters 4. Factor graphs (high level) 5. Tuning and consistency

State vector patterns

Keep state minimal for the motion model in use; avoid unobservable states without constraints.

Process and measurement models

Process model: how state evolves between measurements (constant velocity, coordinated turn, IMU propagation).

Measurement model: \(h(x)\) mapping state to sensor space (range-bearing, pixel bbox, point cloud plane).

Document for each sensor:

• Expected nonlinearity (strong → UKF or iterative EKF)
• Clutter and false alarm rate (affects gating, not only R matrix)
• Detection probability \(P_d\) vs range/occlusion

Kalman-family filters

Update pattern:

1. Predict with process noise Q (and IMU propagation if applicable) 2. Gate measurements (see association reference) 3. Update with innovation \(y = z - h(\hat{x})\) 4. Monitor innovation statistics

Factor graphs (high level)

Use when batch or sliding-window smoothing helps:

• Visual–inertial odometry, LiDAR odometry + loop closure
• Offline calibration refinement
• Joint optimization of poses and landmarks

Concepts: nodes (variables), factors (constraints), sparse solvers, marginalization for real-time windows.

Do not duplicate a full SLAM product scope unless explicitly in charter—interface poses/tracks to mapping nodes.

Tuning and consistency

Consistency checks:

• NEES (Normalized Estimation Error Squared) for Gaussian filters
• NIS per sensor stream
• Innovation whiteness (rough check in field)

If NEES ≫ dof: filter overconfident (inflate R, fix calibration). If NEES ≪ dof: over-inflated uncertainty or wrong model.

Evaluation, simulation, and metrics

Table of contents

1. Metric categories 2. Estimation consistency 3. Tracking and fusion quality 4. Simulation and SIL 5. Bag replay and regression

Metric categories

Report confidence intervals over scenarios, not single-number hero runs.

Estimation consistency

NEES (Normalized Estimation Error Squared):

\[ \text{NEES}_k = (x_k - \hat{x}_k)^T P_k^{-1} (x_k - \hat{x}_k) \]

For well-tuned Gaussian filter, NEES ≈ \(\chi^2\) with dof = state dimension (per update, use with care).

Innovation tests: whiteness, magnitude per sensor channel.

Tracking and fusion quality

Define success criteria per ODD (operational design domain) before tuning.

Simulation and SIL

Practices:

• Seed RNG for reproducibility
• Model latency and drop not only ideal measurements
• Version sensor models with fusion software in regression manifest
• Cross-check sim vs bag replay on same scenario

Coordinate with autonomy sim (tactical-ai-autonomy-developer) on scenario ownership.

Bag replay and regression

Bag replay checklist:

1. Fix clock and TF to recorded or recomputed calibration version 2. Pin software commit, config, and calibration hash in report 3. Run deterministic mode where middleware allows 4. Compare outputs to baseline and labeled ground truth

Regression suite structure:

Artifacts: metric JSON, plots (NEES, RMSE vs time), failure case list with bag timestamps.

Do not claim safety certification from metrics alone—engineering evidence for internal review.

LiDAR, camera, radar, IMU, and GNSS fusion

Table of contents

1. Fusion taxonomy 2. Modality strengths 3. Integration patterns 4. Degraded modes 5. Common failure modes

Fusion taxonomy

Choose based on sync quality, calibration stability, and team ownership per sensor stack.

Modality strengths

Integration patterns

LiDAR + camera

• Project LiDAR points to image for semantic labeling or frustum filtering
• Lift 2D detections with depth from LiDAR/radar/stereo
• BEV fusion networks (concept)—shared grid in bird's-eye view
• Validate time offset between rolling shutter and spinning LiDAR

Radar + camera / LiDAR

• Radar range-rate anchors velocity; camera/LiDAR refine lateral position
• Guard against radar multipath ghosts in association gates
• Doppler ambiguity policies at high speed

IMU + GNSS (+ wheel)

• INS/GNSS EKF for ego; wheel odometry as pseudo-measurement
• GNSS denial: switch covariances, use vision/LiDAR odometry constraints
• Document lever arms and antenna phase center

Multi-LiDAR / multi-camera

• Unified frame via extrinsics; overlap FOV for redundancy
• Per-sensor health flags before fusion weighting

Degraded modes

Publish degradation enum to autonomy stack (tactical-ai-autonomy-developer) for behavior changes.

Common failure modes

• Stale TF or wrong extrinsic after shock/vibration
• Clock skew causing systematic association bias
• Overconfident fusion when one sensor is miscalibrated
• ID switches at occlusions—fix association before tuning Q/R
• Double counting same object from radar + camera late fusion
• Latency mismatch—planner uses old tracks marked as fresh

Multi-sensor association and tracking

Table of contents

1. Data association 2. Gating 3. Assignment algorithms 4. Track lifecycle 5. MOT metrics

Data association

Problem: match measurements (detections, clusters, radar returns) to existing tracks or new births.

For multi-sensor fusion, clarify association level:

• Per-sensor then fuse tracks (track-to-track)
• Fuse detections in common frame then associate (early fusion)
• Hybrid: camera 2D → 3D lift then LiDAR association

Gating

Reject unlikely pairs before assignment.

Document gate thresholds per class and range; tune with labeled data where possible.

Clutter: uniform or Poisson clutter models inform expected false associations; adjust birth threshold.

Assignment algorithms

Cost matrix examples: negative log likelihood, Mahalanobis distance, weighted sum of position + appearance (high level).

Track lifecycle

Rules to specify:

• M-of-N confirmation for radar/low \(P_d\) sensors
• Coast limit (time and distance) before delete
• Duplicate track suppression when two tracks lock same object
• Track quality score for downstream (planner consumes only Q > τ)

MOT metrics

Report per scenario class (urban, highway, pedestrian-heavy) not only aggregate.

Pair with fusion latency and track age at handoff to planning.

Sensor fusion scope

Table of contents

1. Role boundary 2. Architecture capture 3. Sensor inventory template 4. Latency and compute budget 5. What good looks like

Role boundary

Architecture capture

Document before detailed design:

Sensor inventory template

Include detection pipeline ownership (learned detector vs geometric) and where fusion attaches (detections vs features vs tracks).

Latency and compute budget

Chain end-to-end:

capture → driver → preprocess → detect → associate → filter → publish

For each stage record: typical, p95, max; drop policy when over budget; whether to hold last estimate or coast tracks.

What good looks like

• Clear fusion boundary vs autonomy behavior and embedded firmware
• Sensor table with rates, frames, and explicit drop/degrade rules
• Latency budget with measured p95, not only nominal rates
• Downstream contracts: message types, uncertainty fields, max age of data
• Open risks: uncalibrated extrinsics, unsynced cameras, untested GNSS denial