Embedded Real Time Software Engineer

Embedded Real-Time Software Engineer

When to Use

• Select or compare MCUs/MPUs for real-time, memory, peripheral, safety, and toolchain fit
• Choose bare-metal vs RTOS and define task model, priorities, periods, and deadlines
• Design ISR → deferred work paths (bottom halves, work queues, DMA completion chains)
• Set memory policy—static allocation, stack sizing, heap ban/limit, MPU regions where used
• Analyze timing—schedulability sketches, jitter budgets, measurement hooks, WCET reasoning
• Implement concurrency—mutex/Semaphore choice, priority inheritance, lock-free only with proof
• Layer drivers and HAL—register access, DMA, error recovery, test doubles for host tests
• Plan bring-up and debug—JTAG/SWD, trace, logic analyzer, assert/fault hooks, post-mortem dumps
• Design power modes—wake sources, clock gating, peripheral retention, RTOS tickless tradeoffs
• Apply coding discipline—MISRA C awareness, defensive checks, watchdog strategy, update strategy
• Frame safety-aware design for automotive/medical/industrial (hazards, FMEA hooks) without cert claims

When NOT to Use

• HIL security assessment, bus fault injection, or bench penetration on real targets → hardware-in-the-loop-security-tester
• General backend, APIs, microservices, or cloud runtime without MCU constraints → senior-software-engineer
• SCADA/ICS plant operations, Purdue model, OT network monitoring → scada-ics-cyber-security-specialist
• Service-level profiling, load tests, p99 on servers or browsers → performance-engineer
• FPGA/RTL-only design, timing closure, synthesis—unless co-designing firmware for SoC/FPGA fabric
• Pre-merge plan/design gates across domains without embedded implementation → build-validator
• Criticality tiering, RTO/RPO, enterprise continuity without firmware architecture → mission-critical
• Enterprise security program, IAM, or SOC operations → cybersecurity, information-security-engineer
• Production incident command on live fleets → incident-responder

Related skills

Core Workflows

1. Scope, constraints, and platform choice

Capture hard real-time vs soft real-time, safety class, power budget, toolchain, and certification boundaries (inform only—do not claim compliance).

See `references/embedded_rt_scope_and_constraints.md`.

2. Scheduling, RTOS, and deadlines

Define tasks, priorities, periods, deadlines, synchronization, and jitter acceptance; sketch schedulability and worst-case paths.

See `references/scheduling_rtos_and_deadlines.md`.

3. Interrupts, drivers, and HAL

Partition ISR work, driver state machines, DMA paths, and hardware abstraction with testability.

See `references/interrupts_drivers_and_hal.md`.

4. Memory, concurrency, and safety-aware design

Stack/heap policy, MPU usage, locking rules, priority inversion mitigation, watchdogs, and hazard-aware patterns.

See `references/memory_concurrency_and_safety.md`.

5. Timing analysis and debugging

Measure latency, build WCET arguments, use trace and analyzers, capture field diagnostics.

See `references/timing_analysis_and_debugging.md`.

6. Power, boot, and deployment

Reset/boot chain, clock trees, low-power modes, OTA/update constraints, and manufacturing hooks.

See `references/power_boot_and_deployment.md`.

Outputs

• Platform decision record — MCU, RTOS/bare-metal, memory map, toolchain, open risks
• Task/scheduling table — name, priority, period, WCET budget, shared resources, blocking rules
• ISR/deferred-work map — latency budget per IRQ, bottom-half mechanism, re-entrancy notes
• Memory budget — per-task stacks, globals, DMA buffers, heap policy (if any)
• Driver/HAL interface sheet — init/teardown, error codes, thread/ISR context rules
• Timing evidence pack — measurements, trace captures, WCET assumptions and gaps
• Power mode matrix — states, wake sources, peripheral retention, transition times
• Review checklist — MISRA-oriented items, watchdog, safe defaults, update/rollback hooks

Principles

• Measure timing; do not guess — instrument before optimizing; document measurement setup
• Keep ISRs minimal — defer protocol and heavy work; respect IRQ latency budgets
• Prefer static allocation — prove stack depth; ban unbounded heap in safety paths
• Make priority inversion visible — inheritance, ceiling mutexes, or lock-free with formal sketch
• Layer for testability — HAL behind interfaces; host tests for logic; HIL for integration
• Separate safety claims from engineering — hazard IDs and mitigations yes; certification no
• Pair with security and ops peers — HIL security, mission-critical tiering, build validation as needed

When to load references

Embedded RT scope and constraints

Table of contents

1. Role boundary 2. Constraint capture 3. MCU selection tradeoffs 4. Bare-metal vs RTOS 5. What good looks like

Role boundary

Constraint capture

Document before design:

MCU selection tradeoffs

Produce a short decision matrix with scored options and explicit disqualifiers.

Bare-metal vs RTOS

At pattern level: FreeRTOS (wide BSP), Zephyr (device tree, networking)—compare tick rate, tickless idle, and driver model fit; do not prescribe vendor-specific APIs unless user context requires it.

What good looks like

• Constraints written and signed by product/safety stakeholders where applicable
• Platform choice recorded with measurable timing and memory budgets
• Clear boundary: firmware engineering vs security bench vs enterprise tiering
• Open risks listed (unmeasured WCET, shared bus contention, heap use)

Interrupts, drivers, and HAL

Table of contents

1. ISR design rules 2. Deferred work patterns 3. Driver architecture 4. HAL layering 5. DMA and buses

ISR design rules

• Minimize latency: clear flags, capture timestamps, enqueue pointer—return
• No blocking: no printf, malloc, mutex lock (unless documented ISR-safe API)
• Re-entrancy: same IRQ can nest on some cores—use per-channel state or disable briefly
• Shared IRQ lines: demux in ISR; defer demuxed handling per device
• Measure: GPIO toggle or trace marker for IRQ duration histogram

Document per-IRQ maximum time and stack use if nested.

Deferred work patterns

Choose one pattern per peripheral class; avoid mixing without state diagram.

Driver architecture

Typical layers:

Application / middleware
        ↓
Driver API (init, read, write, ioctl, async callback)
        ↓
HAL (register macros, clock enable, pin mux)
        ↓
Hardware

Driver state machine:

Return typed errors (timeout, NACK, CRC, bus fault); implement bounded retry.

HAL layering

• HAL: chip-specific registers; no business logic
• PAL (optional): board pin mapping, external transceiver enable
• Mock HAL: host unit tests for logic above register layer
• Compile-time vs runtime configuration—prefer static tables for cert-friendly builds

Keep register access in one module; forbid scatter #define magic across app code.

DMA and buses

For multi-master buses (CAN, I2C), serialize access with mutex at driver layer; never from ISR except ISR-safe lock.

Memory, concurrency, and safety

Table of contents

1. Memory policy 2. Stack and heap 3. MPU and protection 4. Concurrency hazards 5. Safety-aware practices 6. MISRA C awareness

Memory policy

Default posture for hard real-time / safety paths:

• Static allocation for all runtime objects
• No malloc in control loops; if heap exists, isolate to init phase or non-RT tasks
• Const data in flash; avoid RAM duplication
• DMA buffers statically placed with correct section attributes
• Zero-init assumptions documented for BSS

Stack and heap

Run stack profiling under stress tests before release; record high-water per task.

MPU and protection

When MPU/MMU available:

• Separate privileged drivers from application tasks
• Read-only code; no execute from RAM unless required and audited
• Guard regions below stacks
• Peripheral registers accessible only from driver task/ISR context

Document fault handler behavior: log, safe state, reset tier.

Concurrency hazards

Use lock-free only with memory ordering diagram and review; prefer mutex with bounded hold time.

Safety-aware practices

Without claiming certification:

• Map functions to hazards (unintended motion, energy release, data corruption)
• Define safe states on fault: de-energize, limp mode, last-known-good output
• Watchdog hierarchy: task pet, logical flow checks, external windowed WDT
• Input validation at boundaries; range checks before actuation
• Diversity or monitor tasks for critical outputs where standard requires
• Change impact: every patch traces to hazard analysis ID when in regulated program

Escalate formal safety case work to qualified safety engineers; provide evidence packs they request.

MISRA C awareness

High-level discipline (not a full rules audit):

• No implicit conversions that widen signedness risk
• Explicit widths (uint32_t); avoid plain int for hardware
• Check every return value; no empty default in switch
• Limit function complexity; single exit where team standard requires
• Restrict function-like macros and pointer arithmetic
• Use const and static to enforce linkage intent
• Pair with static analysis tool in CI when available

Document deviations with rationale in project matrix.

Power, boot, and deployment

Table of contents

1. Reset and boot chain 2. Clocks and power modes 3. Low-power design 4. RTOS tickless and wake 5. Update and deployment 6. Manufacturing hooks

Reset and boot chain

Document sequence:

Rules:

• Init order respects dependencies (clocks → pins → peripherals → tasks)
• Fail closed on verify errors; defined recovery (retry bank, service mode)
• Boot time budget for products with fast wake requirements

Clocks and power modes

For each mode list: wake sources, RAM retained, peripheral state, transition time, RTOS compatibility.

Low-power design

• Gate clocks to unused blocks; disable peripherals explicitly
• Choose polling vs interrupt for rare events—energy vs latency
• Batch sensor samples; align radio TX with wake windows
• Avoid busy-wait except documented short spins
• Profile energy per use-case not just average current

RTOS tickless and wake

• Tickless idle reduces timer interrupts but adds wake latency
• Verify timeout accuracy for protocol stacks after tickless
• Document maximum idle duration before missed deadlines
• Test wake from every IRQ path used in production

Update and deployment

Coordinate with security peers for trust model; implement firmware-side state machine only.

Manufacturing hooks

• Test firmware or BIST modes behind GPIO/straps
• JTAG lock policy documented
• Serial number / calibration storage layout
• EOL tests: RAM march, flash CRC, communication loopback
• Provisioning keys in HSM/OTP—never log secrets

Use build-validator for release readiness across firmware + CI + ops when scope exceeds implementation.

Scheduling, RTOS, and deadlines

Table of contents

1. Task model 2. Priorities and policies 3. Periods, deadlines, jitter 4. Synchronization 5. Schedulability sketch

Task model

For each task/thread document:

Keep ISR work out of task tables—reference IRQ budget separately.

Priorities and policies

• Assign rate-monotonic or deadline-monotonic baselines for periodic tasks; justify exceptions
• Reserve highest priority for shortest-deadline control loops only when measured necessary
• Avoid priority inversion by design—document every cross-priority lock
• Use time-slicing sparingly on hard RT paths; prefer cooperative or dedicated cores
• On SMP: pin tasks, isolate shared caches, or partition peripherals per core

RTOS pattern notes (not API-specific):

• FreeRTOS: priority inheritance on mutexes; config configMAX_PRIORITIES, tickless idle impacts jitter
• Zephyr: preemptive threads, work queues, k_mutex priority ceiling; device init levels affect boot order

Periods, deadlines, jitter

Capture end-to-end chains (sensor → filter → actuate) with budget per segment.

For soft RT: define degraded mode when slack exhausted (drop frames, reduce rate).

Synchronization

Never block inside ISR except documented RTOS FromISR APIs with bounded time.

Schedulability sketch

For periodic task set (simplified RMS check):

1. List tasks sorted by period ascending 2. Assign priorities accordingly unless safety overrides 3. For each task \(i\), compute utilization \(U_i = C_i / T_i\) 4. Sum \(U = \sum U_i\); compare to RMS bound or use response-time analysis for shared resources 5. Add interrupt load and driver DMA as equivalent utilization or blocking terms

Document unverified segments explicitly; schedule measurement before claiming hard guarantees.

Timing analysis and debugging

Table of contents

1. Measurement first 2. WCET concepts 3. Jitter and latency chains 4. Debug and trace 5. Field diagnostics

Measurement first

Before optimizing:

Record configuration: clock, cache state, compiler flags, build ID.

WCET concepts

WCET = worst-case execution time under stated assumptions.

Build argument from:

1. Path analysis — longest path through code; no "typical" paths for hard deadlines 2. Hardware effects — wait states, flash prefetch, cache misses, bus contention 3. Blocking — mutex wait, DMA completion, interrupt preemption stack 4. Tool support — aiT, measurement + margin, hybrid when available

Document assumptions and residual risk when WCET is estimated not proven.

Jitter and latency chains

For chain \(A \rightarrow B \rightarrow C\):

• Budget per segment including queueing delay
• Account for tick granularity if RTOS delays wake
• Include interrupt storms and lower-priority task blocking windows
• Report p99 vs max—hard RT needs max unless statistical argument accepted by safety process

Debug and trace

Production constraints: disable invasive debug in release; gate semihosting; protect keys.

Field diagnostics

• Structured fault records (reason, PC, LR, stack, version, uptime)
• Assert policy: development verbose; release minimal codes
• Counters for restarts, watchdog events, stack high-water
• Safe logging rate-limited; circular buffer flush on fault
• Correlate with build fingerprint and config CRC

Pair with hardware-in-the-loop-security-tester when reproducing issues requires bench stimulus—not for everyday bring-up.