Ia Rust Systems

Rust Systems & Services

Covers modern application-layer Rust (edition 2024): CLIs, web services, libraries. Not no_std/embedded.

Tooling

• Pin rust-toolchain.toml per repo so every contributor and CI uses the same compiler.
• cargo update -p <crate> for single-package upgrades. cargo update rewrites everything — avoid in PR diffs.
• Cargo.lock goes in version control for binaries and libraries (modern guidance; reproducibility wins).

Workspaces

Multi-crate projects use a workspace with layered crates. Dependencies point inward only.

Cargo.toml                  # [workspace] members + [workspace.dependencies]
crates/
  protocol/    # Shared types, no deps on other workspace crates
  storage/     # Persistence, depends on protocol
  service/    # Business logic, depends on protocol + storage
  cli/        # Binary, depends on everything

• Centralize versions in [workspace.dependencies], reference as foo = { workspace = true } in members.
• Keep the leaf-most crate (protocol / types) dependency-free so every other crate can depend on it without cycles.
• Feature flags belong on the crate that introduces the dependency, not re-exported through the workspace root.
• Library crates expose one stable facade: a thin lib.rs with a //! module doc comment stating purpose, followed by pub use re-exports of the public surface. Consumers learn one import path per concept; internal module layout can be reorganized without breaking callers.
• Feature gates must error, never silently degrade. If runtime config requests a capability the binary wasn't compiled with (e.g. device = "gpu" on a non-CUDA build), fail at startup with a clear error. Silent fallback produces different behavior from what the operator configured, often without anyone noticing.
• Centralize lints at the workspace root with [workspace.lints.*]. Every member crate inherits the same ruleset — no drift between crates, no per-crate #![deny(...)] stacks. Example:

  [workspace.lints.rust]
  unsafe_code = "warn"
  missing_docs = "warn"

  [workspace.lints.clippy]
  all = { level = "warn", priority = -1 }
  pedantic = { level = "warn", priority = -1 }
  nursery = { level = "warn", priority = -1 }
  module_name_repetitions = "allow"
  must_use_candidate = "allow"

Each member crate opts in with [lints] workspace = true in its own Cargo.toml. Changing a lint in one place updates every crate.

Build Profiles

When tuning Cargo build profiles (release LTO, release-dbg symbols, release-min for distributable binaries) or adding dev-machine speedups (mold linker, target-cpu=native, share-generics), load build-profiles.md.

Error Handling

Split by crate role:

• Libraries / lower crates: define typed errors with thiserror. Consumers can pattern-match.
• Binaries / top-level crates: use anyhow::Result with .context("what was being attempted"). Human-readable error chains.
• Never return Box<dyn Error> from library APIs — it erases variant information.
• Use ? liberally. Never .unwrap() or .expect() outside tests and main. An expect("...") is acceptable only when the invariant is provably upheld and the message explains why.
• Convert at boundaries: #[from] on thiserror variants for auto-conversion; .map_err(MyError::from) when explicit.
• bail!("...") / ensure!(cond, "...") in application code for early exits.
• Prefer Result<T, E> over panics for any recoverable error. Panics are for programmer bugs (broken invariants), not runtime failures.
• `#[must_use]` on fallible APIs: annotate functions returning Result or newtype-wrapped results that callers frequently ignore. Catches let _ = validate(x); at compile time instead of shipping a silently-dropped error.
• Make illegal call-sequences unrepresentable rather than returning a runtime error — the type-state pattern. When an API has a mandatory call order (configure → connect → use), encode each stage as a distinct type (Client<Uninitialized> → Client<Initialized> → Client<Connected>) carrying PhantomData<State>; a method only exists on the state that permits it. Calling send_request before connect then fails to compile instead of panicking at runtime — there is no error variant to handle because the bad sequence cannot be written.

Ownership Discipline

• Take &str over &String, &[T] over &Vec<T> in function signatures — accepts more call sites for free.
• Return owned (String, Vec<T>) from constructors and public APIs. Borrow in hot paths where lifetimes are obvious.
• Reach for Arc<T> only when sharing across threads. Single-threaded sharing uses Rc<T> or references.
• Cow<'_, str> when a function sometimes allocates and sometimes borrows (e.g. normalization).
• Lifetime elision handles 90% of cases. If you're writing 'a in more than one signature, reconsider whether that type should own its data instead.
• `bytes::Bytes` for zero-copy slicing of shared immutable buffers — network parsers, frame decoders, protocol handlers. BytesMut for building buffers that split_to / split_off into Bytes without reallocation. Prefer Bytes over Arc<Vec<u8>> when slicing is the dominant access pattern.
• Reduce hot-path heap allocations with stack-or-inline collections when the typical size is small and known:
• smallvec::SmallVec<[T; N]> — inline for ≤N items, spills to heap beyond. Good for "usually 1-8 items" cases like parsed tag lists, lookup keys, small event batches.
• arrayvec::ArrayVec<T, CAP> — fixed capacity, never heap-allocates. Returns an error when full. Good for bounded message buffers or per-request scratch space.
• String interning for repeatedly-seen strings (enum-like values parsed from config, tenant IDs, route keys): dashmap::DashMap<String, &'static str> with Box::leak on miss gives &'static str comparisons without per-call allocations.

These are optimizations — profile first. Vec/String on a cold path isn't the bottleneck.

Async with Tokio

• Default runtime: #[tokio::main] with features = ["full"] for apps; features = ["rt", "macros", "sync"] for libraries that need to stay slim.
• tokio::spawn for independent tasks. JoinSet for a dynamic group you'll await together with cancellation.
• tokio::select! for racing futures (timeouts, cancellation, first-wins).
• Never block the runtime: tokio::task::spawn_blocking for sync CPU work or blocking I/O libs.
• tokio::sync::Mutex only when the guard must be held across .await. Otherwise std::sync::Mutex is faster.
• `tokio::sync::RwLock` when reads dominate writes (config snapshots, route tables, hot caches). Many readers proceed in parallel; Mutex serializes them. For snapshot-swap semantics (rarely-updated config), arc-swap::ArcSwap is faster still — no lock on the read path.
• Cancellation: CancellationToken (from tokio-util) propagates shutdown. Long-running tasks must check it.
• Backpressure via bounded mpsc channels — unbounded channels hide memory growth until OOM.
• `Semaphore` for hard concurrency limits on spawn paths that don't fit a channel model (e.g. "at most 50 concurrent outbound HTTP calls"). let _permit = sem.acquire().await?; inside the task; dropping the permit releases the slot. Pair with Arc<Semaphore> shared across spawners.
• Don't mix async runtimes. Pick tokio and stick with it; async-std and smol don't interop cleanly.
• Vectored writes (write_vectored + std::io::IoSlice) coalesce many buffers — interleaved headers and payloads — into a single syscall when flushing a batch of messages to a socket; the kernel does the gather. An optimization for measured syscall-bound flush paths — profile first; a single write_all is fine elsewhere.

CLI Tools (clap)

• Use the derive API: #[derive(Parser)] + #[derive(Subcommand)]. Less boilerplate, types drive the help text.
• One enum Commands variant per subcommand; flatten shared flags into a #[command(flatten)] struct CommonArgs.
• --json flag on query commands for agent/pipe consumption. Emit via serde_json::to_string(&value)?.
• Exit codes: 0 success, 1 for errors main returned, 2 for argparse (clap handles this), reserve 3+ for domain meanings documented in --help.
• Provide --version automatically via #[command(version)].

See cli-tools.md for config layering, logging setup, progress reporting, and shell completions.

HTTP Services (axum)

• Framework default: axum (tokio-native, tower middleware, extractor-based handlers). Pick actix-web only if an existing codebase uses it.
• Handlers return Result<impl IntoResponse, AppError>. Implement IntoResponse for AppError to centralize error → status mapping.
• Validate input at the boundary: axum::extract::Json<T> where T: Deserialize + Validate (use validator crate). Internal services trust input was validated.
• Share state via State<Arc<AppState>> — not globals, not lazy_static.
• Middleware via tower::ServiceBuilder: tracing → timeout → auth → CORS → handler. Order matters.
• Resilience layer stack (outbound HTTP clients and shared services): ServiceBuilder::new().layer(TimeoutLayer).layer(RateLimitLayer).layer(ConcurrencyLimitLayer).layer(LoadShedLayer).layer(RetryLayer).service(client). Name each layer explicitly — LoadShedLayer sheds excess load, ConcurrencyLimitLayer caps in-flight requests, RateLimitLayer bounds request rate, RetryLayer retries classified transient errors. Combining LoadShedLayer + ConcurrencyLimitLayer produces proper backpressure instead of unbounded queueing.

See axum-service.md for project layout, extractors, error types, graceful shutdown, and OpenAPI generation.

Concurrency

rayon and tokio coexist — use tokio::task::spawn_blocking to call a rayon pool from async code. Never call .block_on() from inside a tokio task; it deadlocks the runtime.

Testing

• Built-in #[test]. Prefer cargo nextest run --workspace over cargo test — it runs tests in parallel processes with proper isolation.
• Unit tests live in mod tests { ... } at the bottom of the file (access to private items).
• Integration tests in tests/ directory. One file per public surface area.
• #[tokio::test] for async tests. Add flavor = "multi_thread" when the code under test spawns tasks.
• rstest for parametrized tests and fixtures. proptest / quickcheck for property-based tests on pure logic.
• insta for snapshot testing CLI output, serialization, large structs. Review diffs with cargo insta review.
• assert_cmd + predicates for CLI integration tests (invokes the binary, asserts on stdout/stderr/exit code).
• Assert on error variants with `matches!`: assert!(matches!(result.unwrap_err(), MyError::Validation(_))). Cleaner than match arms when the test only cares whether the error is the right kind, and doesn't force updates when unrelated variants are added.
• Coverage: cargo llvm-cov --workspace --html. Target 70%+ on application code, higher on library crates.
• Fuzzing for parsers: cargo fuzz + libfuzzer-sys on any code that parses untrusted input (file formats, protocols, query languages). A short nightly fuzz run surfaces the panics and UB that unit tests miss.

For generic test discipline (anti-patterns, mock rules, rationalization resistance), see the ia-writing-tests skill.

Unsafe Discipline

• Default: no unsafe. If clippy flags it, don't #[allow] it — refactor.
• Every unsafe block gets a // SAFETY: comment above it explaining why each invariant holds. No comment = reviewer rejects.
• Keep unsafe blocks minimal — wrap in a safe abstraction at module boundary, mark the module pub(crate).
• Use miri (cargo +nightly miri test) on any crate containing unsafe or raw pointer arithmetic — catches UB that optimizers mask.
• Prefer bytemuck, zerocopy, bytes over hand-rolled transmutes for zero-copy patterns.
• `std::env::set_var` and `remove_var` are `unsafe` under edition 2024. Concurrent getenv from another thread is UB at the libc level; the unsafety can't be wrapped away by OnceLock::call_once or std::sync::Once — they ensure the closure runs once, not that it runs while no other thread is reading the environment. Pin every env-var write to single-threaded startup, before tokio::main or any std::thread::spawn. Common offender: native-library discovery paths (LD_LIBRARY_PATH, ORT_DYLIB_PATH, LIBTORCH, plugin loader paths) set lazily on first use — compute and set them in main (or a static initializer that runs before the runtime) so they're written before any concurrent reader exists.

Production Resilience

When productionizing a service (config validation, /health + /ready endpoints, graceful shutdown, retries/timeouts/jitter, connection pools, diagnostic secret redaction), load production-resilience.md.

Observability

For logging (tracing + tracing-subscriber with init recipe), #[instrument] spans, correlation IDs, metrics, and distributed tracing patterns, load observability.md. Never use println! or log:: in new code.

CI

General CI design lives with the ia-infrastructure-engineer agent. For Rust-specific callouts (rustsec/audit-check, cargo-llvm-cov, Swatinem/rust-cache, taiki-e/install-action, matrix coverage guidance, doc-test step), load ci-pipeline.md.

Discipline

• Simplicity first — every change as simple as possible, impact minimal code.
• Only touch what's necessary — avoid unrelated changes in a PR.
• No #[allow(clippy::...)] as a shortcut — fix the underlying issue. Document exceptions with a rationale.
• Before adding a trait or generic, verify it's used in 3+ places. Otherwise a concrete type is clearer.
• Verify: see Verify section — pass all checks with zero warnings before declaring done.

Verify

• cargo fmt --all -- --check passes with zero diffs
• cargo clippy --workspace --all-targets --all-features -- -D warnings passes
• cargo nextest run --workspace (or cargo test --workspace) passes with zero failures
• cargo deny check passes (licenses, advisories, duplicates) for any crate going to production
• No new unsafe without // SAFETY: comment

References

• cli-tools.md — clap patterns, config layering, tracing setup, progress, shell completions
• axum-service.md — project layout, extractors, error types, graceful shutdown, testing
• build-profiles.md — release/release-dbg/release-min profiles, mold linker, dev compile speedups
• ci-pipeline.md — Rust-specific CI steps (cargo audit, llvm-cov, rust-cache, matrix strategy, doc tests)
• production-resilience.md — fail-fast config, health/ready endpoints, graceful shutdown, retries, timeouts, connection pools
• observability.md — tracing init recipe, span instrumentation, correlation IDs, metrics, distributed tracing

Axum HTTP Services

Patterns for building production HTTP services with axum + tokio + tower.

Project Layout

src/
  main.rs           # Entrypoint: config load, tracing init, server bind, graceful shutdown
  app.rs            # Router assembly: `pub fn router(state: AppState) -> Router`
  state.rs          # AppState struct (pools, clients, config)
  error.rs          # AppError enum + IntoResponse impl
  routes/
    mod.rs
    users.rs        # One module per resource
    health.rs
  services/         # Business logic, no HTTP types
  repo/             # Data access (sqlx), no HTTP types
  config.rs
  telemetry.rs      # tracing + metrics setup
tests/
  api.rs           # Integration tests hitting the router directly

Rules mirror the layered architecture from ia-nodejs-backend:

• Routes parse + call services + format response. No business logic.
• Services never import from axum or http. No HTTP status codes leak in.
• Repos never construct AppError variants that map to HTTP — they return typed storage errors that services convert.

AppState

#[derive(Clone)]
pub struct AppState {
    pub db: sqlx::PgPool,
    pub http: reqwest::Client,
    pub config: Arc<Config>,
}

Clone is cheap because the expensive members are Arc inside. Inject with State<AppState> extractor — don't use globals or OnceCell.

Error Type

use axum::{http::StatusCode, response::{IntoResponse, Response}, Json};
use serde_json::json;
use thiserror::Error;

#[derive(Debug, Error)]
pub enum AppError {
    #[error("not found")]
    NotFound,
    #[error("invalid input: {0}")]
    Validation(String),
    #[error("unauthorized")]
    Unauthorized,
    #[error(transparent)]
    Sqlx(#[from] sqlx::Error),
    #[error(transparent)]
    Other(#[from] anyhow::Error),
}

impl IntoResponse for AppError {
    fn into_response(self) -> Response {
        let (status, code) = match &self {
            AppError::NotFound => (StatusCode::NOT_FOUND, "not_found"),
            AppError::Validation(_) => (StatusCode::BAD_REQUEST, "validation"),
            AppError::Unauthorized => (StatusCode::UNAUTHORIZED, "unauthorized"),
            AppError::Sqlx(_) | AppError::Other(_) => {
                tracing::error!(error = ?self, "internal error");
                (StatusCode::INTERNAL_SERVER_ERROR, "internal")
            }
        };
        let body = Json(json!({
            "error": { "code": code, "message": self.to_string() }
        }));
        (status, body).into_response()
    }
}

• One error envelope shape across every handler. Callers parse .error.code once.
• Log the full error with ?self for INTERNAL_SERVER_ERROR paths; never leak internal messages to the client.
• Use ? in handlers freely — From impls convert sqlx::Error, anyhow::Error into AppError.

Handlers

#[tracing::instrument(skip(state), fields(user_id))]
pub async fn get_user(
    State(state): State<AppState>,
    Path(id): Path<Uuid>,
) -> Result<Json<UserResponse>, AppError> {
    tracing::Span::current().record("user_id", %id);
    let user = state.users.find(id).await?.ok_or(AppError::NotFound)?;
    Ok(Json(user.into()))
}

• Handlers take extractors first, body last. Extractors run in order; the body extractor must be last (it consumes the request).
• Return Result<Json<T>, AppError> for JSON endpoints, Result<impl IntoResponse, AppError> for flexible responses.
• Use #[tracing::instrument(skip(state))] on every handler — skip the state so secrets don't land in logs.

Validation

use validator::Validate;

#[derive(Deserialize, Validate)]
pub struct CreateUser {
    #[validate(length(min = 1, max = 100))]
    pub name: String,
    #[validate(email)]
    pub email: String,
}

pub async fn create_user(
    State(state): State<AppState>,
    Json(body): Json<CreateUser>,
) -> Result<(StatusCode, Json<UserResponse>), AppError> {
    body.validate().map_err(|e| AppError::Validation(e.to_string()))?;
    let user = state.users.create(body.into()).await?;
    Ok((StatusCode::CREATED, Json(user.into())))
}

Or centralize via a ValidatedJson<T> extractor so every handler calls .validate() without repetition.

Middleware

use tower::ServiceBuilder;
use tower_http::{trace::TraceLayer, timeout::TimeoutLayer, cors::CorsLayer};

pub fn router(state: AppState) -> Router {
    Router::new()
        .route("/health", get(health::shallow))
        .route("/ready",  get(health::deep))
        .nest("/users", users::routes())
        .layer(
            ServiceBuilder::new()
                .layer(TraceLayer::new_for_http())
                .layer(TimeoutLayer::new(Duration::from_secs(30)))
                .layer(CorsLayer::permissive())
                .into_inner(),
        )
        .with_state(state)
}

Middleware order: tracing → timeout → rate limit → auth → CORS → handler. Tracing outside the timeout so you see timed-out requests.

Graceful Shutdown

async fn shutdown_signal() {
    let ctrl_c = async {
        tokio::signal::ctrl_c().await.expect("install ctrl+c handler");
    };
    #[cfg(unix)]
    let terminate = async {
        tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
            .expect("install sigterm handler")
            .recv().await;
    };
    tokio::select! { _ = ctrl_c => {}, _ = terminate => {} }
    tracing::info!("shutdown signal received");
}

// main.rs
let listener = tokio::net::TcpListener::bind(&config.addr).await?;
axum::serve(listener, app::router(state))
    .with_graceful_shutdown(shutdown_signal())
    .await?;

• Drain in-flight requests up to a budget (30s typical), then hard-exit.
• Close DB pools and flush telemetry after the server returns, before main exits.

Testing

use axum::body::Body;
use axum::http::{Request, StatusCode};
use tower::ServiceExt; // for `.oneshot()`

#[tokio::test]
async fn get_user_returns_404_for_unknown_id() {
    let state = test_state().await;
    let app = app::router(state);
    let response = app
        .oneshot(
            Request::builder()
                .uri(format!("/users/{}", Uuid::new_v4()))
                .body(Body::empty())
                .unwrap(),
        )
        .await
        .unwrap();
    assert_eq!(response.status(), StatusCode::NOT_FOUND);
}

• Exercise the router directly with tower::ServiceExt::oneshot — no TCP bind needed, fast and deterministic.
• sqlx::test macro gives each test an isolated database via transactions or template DBs.
• For full end-to-end, use reqwest against a real bound port with TcpListener::bind("127.0.0.1:0") to pick a free port.

Database (sqlx)

Prefer the compile-time-checked macros over the runtime builder:

// Schema drift caught at `cargo build`, not at 3am in prod.
let user = sqlx::query_as!(
    User,
    "SELECT id, email, created_at FROM users WHERE id = $1",
    id,
)
.fetch_optional(&state.db)
.await?;

• sqlx::query_as! / sqlx::query! verify SQL against the real database schema at compile time. Requires DATABASE_URL in the environment or a checked-in .sqlx/ offline query cache (cargo sqlx prepare).
• Use .fetch_optional() for "by id" lookups — returns Option<T>, maps cleanly to AppError::NotFound.
• .fetch_all() only when you've bounded the result set with LIMIT. No unbounded selects from request handlers.
• Transactions: let mut tx = state.db.begin().await?; → do work → tx.commit().await?;. Dropping without commit rolls back.
• Prefer query_as! + explicit struct over FromRow derive when the struct and SQL columns are 1:1; derive when you want reuse across multiple queries.

Ship .sqlx/ in the repo so CI builds don't need a live database. Regenerate with cargo sqlx prepare --workspace when queries change.

OpenAPI

utoipa generates OpenAPI from derive macros on handlers + types. aide is an alternative with better runtime integration. Either way: the schema is derived from code, not maintained separately.

#[derive(OpenApi)]
#[openapi(paths(get_user, create_user), components(schemas(UserResponse, CreateUser)))]
struct ApiDoc;

Serve the spec at /openapi.json and Swagger UI at /docs in non-prod environments.

Common Traps

• Don't put an Arc<Mutex<T>> in AppState for things that should be behind a DB. Shared mutable state across requests is almost always a design smell.
• Don't use tokio::sync::RwLock where arc-swap::ArcSwap fits — config reloads, rarely-changing snapshots.
• Don't forget with_state at the end of router assembly. The compiler error is confusing (method not found on Router<AppState>).
• Don't bind to 0.0.0.0 in local dev unless you mean it. Use 127.0.0.1 by default; make the bind address configurable.

Build Profiles

Load this reference when setting up or tuning a Rust project's Cargo build profiles. Tune profiles for the shape of the binary — defaults ship fast debug builds and modest-optimization release builds, but application Rust benefits from more aggressive profiles.

Profile definitions (Cargo.toml)

# Production release: maximum optimization, minimum binary
[profile.release]
lto = "fat"           # Link-time optimization across all crates
codegen-units = 1     # Single codegen unit trades compile time for runtime perf
strip = true          # Strip symbols from the final binary
panic = "abort"       # No unwinding tables — smaller binary, faster panics

# Release with symbols kept for profiling (perf, flamegraph, pprof)
[profile.release-dbg]
inherits = "release"
strip = false
debug = true

# Size-minimized release for distributable CLIs
[profile.release-min]
inherits = "release"
opt-level = "z"       # Optimize for size over speed

panic = "abort" breaks catch_unwind-based recovery — skip it for libraries others will link against, or for binaries that rely on panic hooks (some web frameworks do). For most CLIs and backend services, it's pure win.

Dev-machine compile speedups (.cargo/config.toml)

Cut PR compile time on Linux with mold:

[build]
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

[target.x86_64-unknown-linux-gnu]
rustflags = [
    "-C", "link-arg=-fuse-ld=mold",
    "-C", "target-cpu=native",   # dev machines only — bakes in CPU features
    "-Z", "share-generics=y",    # share monomorphizations across crates (nightly)
]

[alias]
t = "nextest run"

Mold is Linux-only (lld on other platforms). target-cpu=native is a developer-machine convenience — remove for reproducible CI and distributable binaries.

Rust CI Pipeline — language-specific callouts

Load this reference when setting up or reviewing a CI pipeline for a Rust project. General CI design (matrix strategy, caching, deployment gating) lives with the ia-infrastructure-engineer agent — this file covers only the Rust-specific pieces.

• `rustsec/audit-check` — runs cargo audit against the RustSec Advisory DB. Catches published CVEs in your dependency graph. Wire it as a required PR check, not a nightly job; advisory hits on main are already too late.
• Coverage via `cargo-llvm-cov` — cargo llvm-cov --workspace --lcov --output-path lcov.info produces codecov-compatible output without the instrumentation overhead of tarpaulin. Prefer it for any new Rust project.
• `Swatinem/rust-cache` — caches ~/.cargo/registry, ~/.cargo/git, and target/ keyed on Cargo.lock. Cuts cold-CI time from ~5min to ~90s on typical workspaces. Install before cargo commands.
• `taiki-e/install-action` — fast binary installer for cargo tools (nextest, llvm-cov, audit). Faster than cargo install on CI by orders of magnitude.
• Matrix coverage: at minimum stable on Linux. Add beta + nightly on Linux and stable on Windows + macOS if the crate is a library others consume; skip the full OS matrix for internal services.
• Doc tests: cargo test --doc --all-features as a separate step. Doc tests are easy to break with a refactor and easy to miss with nextest run alone.

Rust CLI Tools

Patterns for building agent-friendly, scriptable CLIs with clap.

Project Layout

src/
  main.rs       # CLI parse + dispatch only, no business logic
  cli.rs        # clap structs and enums
  commands/
    mod.rs
    index.rs    # one handler per subcommand
    search.rs
  config.rs     # TOML/env loading, validation
  output.rs    # JSON / human formatters
Cargo.toml

Keep main.rs under 20 lines. Every subcommand is a free function that takes parsed args + shared state and returns Result<()>. This makes each command independently testable.

Clap Derive Patterns

use clap::{Parser, Subcommand, Args};

#[derive(Parser)]
#[command(name = "myapp", version, about)]
struct Cli {
    #[command(flatten)]
    global: GlobalOpts,

    #[command(subcommand)]
    command: Commands,
}

#[derive(Args)]
struct GlobalOpts {
    /// Increase logging verbosity (-v, -vv, -vvv)
    #[arg(short, long, global = true, action = clap::ArgAction::Count)]
    verbose: u8,

    /// Emit JSON instead of human output
    #[arg(long, global = true)]
    json: bool,
}

#[derive(Subcommand)]
enum Commands {
    /// Index the current project
    Index(IndexArgs),
    /// Search the index
    Search(SearchArgs),
}

• global = true flags apply to every subcommand without repeating.
• Use value_enum for typed string flags:

  #[arg(long, value_enum, default_value_t = Format::Md)]
  format: Format,

• #[arg(value_parser = validate_path)] to reject bad values before the handler runs.

Layered Parsing for Non-Trivial CLIs

Once flag count crosses ~10 or commands start sharing complex validation, split parsing into two stages:

1. Low stage — LowArgs mirrors the raw CLI surface 1:1. Clap populates it. No cross-field validation, no domain types. 2. High stage — HiArgs (or Config) is what the rest of the program consumes. Constructing HiArgs::from(low) runs all semantic validation: mutual exclusions, path existence, glob compilation, range constraints, regex validity. Fails with one clear error message.

let low = Cli::parse();
let hi = HiArgs::try_from(low).context("invalid arguments")?;
run(hi).await

Downstream code accepts &HiArgs (or specific typed fields from it) and never re-checks. This keeps validation in one place and makes it impossible for a handler to receive an invalid combination. Pattern comes from ripgrep; worth it as soon as flag interactions become non-trivial.

For simple CLIs (one subcommand, a handful of flags), skip this — one clap-derive struct is enough.

Config Layering

Priority (lowest to highest): built-in defaults → ~/.config/myapp/config.toml → project .myapp/config.toml → env vars (MYAPP_*) → CLI flags.

Use figment or hand-roll with serde + toml:

#[derive(Deserialize, Debug)]
pub struct Config {
    #[serde(default = "default_limit")]
    pub limit: usize,
    pub api_key: Option<String>,
}

impl Config {
    pub fn load(project_root: &Path) -> Result<Self> {
        let mut cfg: Config = toml::from_str(
            &std::fs::read_to_string(project_root.join(".myapp/config.toml"))
                .unwrap_or_default(),
        )?;
        if let Ok(key) = std::env::var("MYAPP_API_KEY") {
            cfg.api_key = Some(key);
        }
        cfg.validate()?;
        Ok(cfg)
    }
}

Validation runs in load() — never defer it to the first call site.

Logging

use tracing_subscriber::{EnvFilter, fmt};

fn init_logging(verbose: u8) {
    let level = match verbose {
        0 => "warn",
        1 => "info",
        2 => "debug",
        _ => "trace",
    };
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new(format!("myapp={level}")));
    fmt().with_env_filter(filter).with_writer(std::io::stderr).init();
}

• Logs go to stderr. Only command results go to stdout. This keeps pipes clean.
• Respect RUST_LOG / MYAPP_LOG env var if set — it overrides -v.
• Never log at info! inside hot loops; budget is roughly one log line per user-visible action.

Output

Every query command supports two modes:

if args.json {
    println!("{}", serde_json::to_string(&results)?);
} else {
    render_human(&results);
}

• JSON output must be a single line or a valid JSON document — no mixed human + JSON in the same stream.
• Exit with non-zero on failure even when --json is set; don't emit {"error": "..."} with exit 0.

Progress

• Interactive terminals (check std::io::IsTerminal::is_terminal(&stderr)): indicatif progress bars on stderr.
• Non-interactive (pipes, CI, agents): plain periodic log lines. Never emit control codes to a non-tty.
• --quiet flag suppresses both.

Shell Completions

use clap_complete::{generate, Shell};

Commands::Completions { shell } => {
    let mut cmd = Cli::command();
    generate(shell, &mut cmd, "myapp", &mut std::io::stdout());
}

Ship completions via the completions subcommand rather than pre-generated files — keeps them in sync with the actual flag set.

Testing CLIs

use assert_cmd::Command;
use predicates::prelude::*;

#[test]
fn search_returns_json() {
    Command::cargo_bin("myapp").unwrap()
        .args(["search", "foo", "--json"])
        .assert()
        .success()
        .stdout(predicate::str::starts_with("["));
}

• tempfile::TempDir for isolated project roots.
• Snapshot stdout with insta::assert_snapshot! for human-readable output that changes rarely.
• Test exit codes explicitly — they're part of the CLI contract for scripts.

Common Traps

• Don't print to stdout from library crates. Return structured data, let the binary format it.
• Don't swallow SIGPIPE. On Unix, when the reader closes a pipe early, the default is to die — let it. If you install a tokio::signal handler, re-raise or exit cleanly on pipe errors.
• Don't ship a CLI that panics on bad input. Map every user-facing error to a clean anyhow chain with .context().

Observability for Rust Services

Load this reference when adding logging, tracing, metrics, or distributed tracing to a Rust service. println! and log:: are forbidden in new code — use tracing + tracing-subscriber.

Logging

• tracing + tracing-subscriber with json() formatter in production, fmt().pretty() in dev.
• Init recipe: build subscriber layers and register once at main entry. Respect RUST_LOG for runtime filter override, include thread IDs for concurrent contexts, gate OpenTelemetry behind a feature flag so dev builds don't pull the whole OTEL SDK:

  pub fn init_tracing() {
      let fmt_layer = tracing_subscriber::fmt::layer()
          .with_target(false)
          .with_thread_ids(true);
      let filter_layer = tracing_subscriber::EnvFilter::try_from_default_env()
          .unwrap_or_else(|_| "info".into());
      tracing_subscriber::registry()
          .with(filter_layer)
          .with(fmt_layer)
          .init();
  }

Structured Spans

• #[tracing::instrument(skip(large_arg), fields(user_id = %user.id))] on service methods — automatic span creation, structured fields.
• Skip large args to keep spans lightweight; prefer named fields over stringified args.

Correlation IDs

Extract or generate at ingress middleware, attach to the root span, propagate via traceparent header to downstream calls. Required for any multi-service system.

Metrics

metrics crate with metrics-exporter-prometheus. Counter for traffic/errors, Histogram for latency, Gauge for saturation. Label cardinality bounded — no user IDs, no unbounded dimensions.

Distributed Tracing

tracing-opentelemetry exports spans to Jaeger/Tempo/Honeycomb/Datadog. Gate the OpenTelemetry subscriber behind a feature flag to keep dev/test builds fast.

Production Resilience

Load this reference when productionizing a Rust service — adding config validation, health endpoints, graceful shutdown, retry/timeout discipline, or connection pools. Not needed for CLI tools or dev-only code.

• Fail-fast config: parse and validate all config at startup with serde + a Config::load() -> Result<Self> that returns errors for missing/invalid values. Crash before binding the listen port, not on the first request.
• Health endpoints: /health (shallow liveness, returns 200 if the process responds) and /ready (deep readiness, verifies DB, cache, and downstream services). Load balancers route on /ready; orchestrators restart on /health. Diagnostic endpoints must redact secrets (tokens, passwords, API keys, PII) before returning.
• Graceful shutdown: install a tokio::signal handler, trigger a CancellationToken, drain in-flight requests with a timeout, then exit. Axum: .with_graceful_shutdown(shutdown_signal).
• Retries: use backon or tokio-retry with exponential backoff + jitter. Retry only transient errors (connection reset, 429, 502/503/504). Never retry 4xx.
• Timeouts on every network call — no defaults. tokio::time::timeout(dur, fut) or reqwest::Client::builder().timeout(dur).
• Connection pools: sqlx::PgPool, reqwest::Client — build once, clone (cheap, Arc inside), share via State.

ia-rust-systems Specification

Intent

ia-rust-systems is a language-class skill (stack-specific patterns and idioms). Rust patterns for CLI tools, backend services, and general application code. Use when working with Rust, Cargo workspaces, axum/tokio services, clap CLIs, async concurrency, or configuring clippy, rustfmt, cargo-nextest, or Cargo.toml.

Scope

In scope:

• Behaviors described in SKILL.md and routed via the should_trigger phrasings in distillery/tests/fixtures/triggers/ia-rust-systems.jsonl.
• Updates to runtime behavior, structure, trigger precision, references, and validation.

Out of scope:

• Acting as the runtime instructions themselves (those live in SKILL.md).
• Trigger phrasings already covered by adjacent ia-* skills (validate-plugin flags >70% description overlap as DUPLICATE_TRIGGER).
• <!-- to fill in: domain-specific exclusions when the skill drifts -->

Trigger Context

• Class: language
• Hook regex: plugins/whetstone/hooks/skill-patterns.sh -> SKILL_PATTERNS[ia-rust-systems]
• Common requests (from fixture should_trigger):
• "write a rust CLI tool using clap derive"
• "build an axum service with tokio"
• "set up a cargo workspace with multiple crates"
• Should not trigger for (from fixture should_not_trigger):
• "write a FastAPI endpoint for user registration"
• "add a Laravel queue job for emails"
• "write a React component for the navbar"

Source And Evidence Model

Authoritative sources:

• SKILL.md -- runtime instructions and reference routing.
• references/*.md -- bundled supplementary content (6 file(s)).
• distillery/tests/fixtures/triggers/ia-rust-systems.jsonl -- positive and negative trigger phrasings under regression test.
• plugins/whetstone/hooks/skill-patterns.sh -- regex pattern that fires this skill.
• distillery/.eval-data/ia-rust-systems/ -- harvested session examples (when present).

Data that must not be stored in this skill or its references:

• Secrets, credentials, tokens.
• Machine-specific filesystem paths (/home/..., /Users/..., ~/ai/...). The validator (MACHINE_PATH_LEAK) flags these as HIGH.
• Private URLs, customer data, or unredacted personal information.

Coverage matrix

Evaluation

Lightweight (run on every change):

python3 distillery/scripts/distiller.py validate-plugin --component ia-rust-systems
python3 distillery/scripts/distiller.py test-triggers --skill ia-rust-systems

Deeper (when behavior risk warrants):

python3 distillery/scripts/distiller.py dspy-eval ia-rust-systems
python3 distillery/scripts/distiller.py diagnose-negatives ia-rust-systems

Acceptance gates:

• validate-plugin --component ia-rust-systems returns 0 HIGH findings.
• test-triggers --skill ia-rust-systems returns F1 = 1.0 with floors of 5 should_trigger and 5 should_not_trigger.
• For dspy-eval, the composite score does not regress against the most recent saved baseline (see distillery/.eval-data/ia-rust-systems/history.json).

Known Limitations

<!-- to fill in over time as drift surfaces. Default rule: any time diagnose-negatives surfaces a recurring failure pattern, document it here so future maintainers understand the trade-off the current implementation accepts. -->

Maintenance Notes

• Update SKILL.md when the runtime workflow, branch conditions, or output contract changes.
• Update this SPEC.md when intent, scope, evidence model, evaluation gates, or maintenance expectations change.
• Update the trigger fixture when adding new positive phrasings, removing stale ones, or expanding scope (the 5/5 floor is a hard validator gate).
• Update the hook regex in skill-patterns.sh whenever fixture positives expose a missed phrasing; verify F1 = 1.0 with eval-triggers before committing.
• Run the full release pipeline via /release -- never bump versions or update CHANGELOG.md from a per-skill edit.