Asupersync Mega Skill

Asupersync Mega Skill

Asupersync is a spec-first, cancel-correct, capability-secure async runtime for Rust. Not a Tokio wrapper -- a complete replacement with stronger guarantees around structured concurrency, obligation tracking, deterministic testing, and capability security.

This skill is primarily for agents integrating Asupersync into other projects or extracting maximum architectural leverage from it in greenfield systems. It also covers repo-internal work when that is the actual task.

For codebase orientation, types, module map, and workspace layout see SOURCE-MAP.md.

Quick Orient

Minimal bootstrap:

use asupersync::runtime::RuntimeBuilder;

fn main() -> Result<(), asupersync::Error> {
    let rt = RuntimeBuilder::current_thread().build()?;
    rt.block_on(async {
        let cx = asupersync::Cx::for_request();
        asupersync::proc_macros::scope!(cx, {
            cx.trace("running");
            asupersync::Outcome::ok(())
        });
    });
    Ok(())
}

This is the smallest runnable seam, not the recommended production architecture. Do not build serious services around Cx::for_request() plus block_on(...) alone; prefer runtime-managed contexts, request/call regions at service boundaries, and graduate to AppSpec + supervision when the topology becomes long-lived.

Where to focus first:

• Lead with core runtime, Cx/Scope, cancellation, obligations, channels, sync, time, lab/DPOR, and observability
• For ordinary services, build next on native service, web, grpc, database, and supervision surfaces
• Treat Browser Edition, QUIC/H3, messaging, remote/distributed, and RaptorQ as requirement-driven lanes, not default starting points

Default recommendation order for most real projects:

• core runtime + Cx + Scope
• native service / web / grpc boundaries
• native database and actor/supervision surfaces as needed
• deterministic tests and diagnostics from the start

Do not lead with Browser Edition, QUIC/H3, messaging, remote/distributed, or RaptorQ unless the target project explicitly needs those capabilities.

Full surface guidance: STACK-SURFACES.md.

Start Here

Choose one lane before touching code:

1. Native greenfield Build directly on RuntimeBuilder, Cx, Scope, LabRuntime, and optional AppSpec. 2. Brownfield native migration Rewrite your app's async seams around &Cx, region-owned tasks, cancel-aware primitives, and deterministic tests. 3. Boundary interop Use asupersync-tokio-compat only for crates you cannot remove yet. Keep Tokio out of core business logic.

Default rule:

• prefer native Asupersync surfaces,
• use compat only as a quarantine boundary,
• plan to remove compat once the stubborn dependency is gone.

Non-Negotiables

• Do not treat Asupersync as an executor swap.
• Put &Cx first in async APIs you control.
• Use Scope and child regions for owned work. Avoid detached background tasks.
• Add cx.checkpoint() in loops, retry bodies, long handlers, and shutdown-sensitive code.
• Prefer cancel-aware primitives and two-phase effects.
• Use deterministic tests as part of normal development, not as optional polish.
• Treat Cx::for_testing() as test-only. Cx::for_request() is a convenience seam, not your whole architecture.
• Keep Tokio and Tokio-only crates behind explicit adapter modules if you must keep them at all.

Leverage, Not Just Migration

If the target system is doing real work, do not stop after "the code compiles on Asupersync."

• Budget, Outcome, and capability narrowing are part of the application's semantic contract, not optional polish. See BUDGET-OUTCOME-CAPABILITIES.md.
• Runtime controls are part of the architecture. See RUNTIME-CONTROLS.md.
• Long-lived state belongs in supervised structures. See SUPERVISION-OTP.md.
• Treat the lab runtime and operator diagnostics as part of the normal development loop. See OBSERVABILITY-FORENSICS.md.
• Prefer native combinators over ad hoc select!-style orchestration. See ADVANCED-FEATURES.md.
• Primitive choice and scheduler cooperation materially affect leverage. See PRIMITIVES-AND-ORCHESTRATION-CHOOSER.md and PERFORMANCE-AND-SCHEDULING.md.

Canonical Spine

• Bootstrap: runtime::RuntimeBuilder, Runtime, RuntimeHandle
• App code: Cx, Scope
• Tests: test_utils::{run_test, run_test_with_cx}, LabRuntime, LabConfig
• Service boundaries: web::request_region::{RequestRegion, RequestContext}, grpc::CallContext::with_cx(...)
• Higher-level apps: app::AppSpec, actor, gen_server, supervision, spork

Start with RuntimeBuilder + Cx + Scope. Graduate to AppSpec + supervision when you need restart policy, named workers, or explicit application topology.

Macro guidance: scope! is useful. Manual APIs are still the safest authoritative path. Do not assume proc-macro surfaces are automatically the best default path for every task.

Standard Workflow

• Inventory all tokio::*, tokio-util, hyper, axum, tonic, reqwest, sqlx, quinn, h3, rdkafka, and related dependencies.
• Classify each dependency as: native replacement, compat holdout, or deliberate workaround.
• Replace runtime bootstrap first.
• Thread &Cx through your own async APIs.
• Replace detached spawning with region-owned work.
• Replace sync/time/net/io/channel/web/db/messaging surfaces domain by domain.
• Add deterministic tests while migrating, not after.
• Remove compat boundaries as soon as the underlying dependency no longer needs them.

Reference Index

Quick Router: Start Here For Your Task

All References

Integration and Migration

• Leverage playbook: LEVERAGE-PLAYBOOK.md
• Budgets, outcomes, capabilities: BUDGET-OUTCOME-CAPABILITIES.md
• Native greenfield: NATIVE-GREENFIELD.md
• Greenfield patterns: GREENFIELD-PATTERNS.md
• Brownfield migration: BROWNFIELD-MIGRATION.md
• Tokio mapping: TOKIO-MAPPING.md
• Tokio replacement matrix: TOKIO-REPLACEMENT-MATRIX.md
• Compat boundary rules: COMPAT-BOUNDARY.md
• Compat bridge recipes: COMPAT-BRIDGE.md
• Adoption lanes: ADOPTION-LANES.md
• Anti-patterns: ANTI-PATTERNS.md

Architecture and Primitives

• Primitive and orchestration chooser: PRIMITIVES-AND-ORCHESTRATION-CHOOSER.md
• Channel and sync internals: CHANNELS-SYNC-INTERNALS.md
• Performance and scheduling: PERFORMANCE-AND-SCHEDULING.md
• Scheduler internals: SCHEDULER-INTERNALS.md
• Lock ordering: LOCK-ORDERING.md
• Advanced features: ADVANCED-FEATURES.md
• Runtime controls and diagnostics: RUNTIME-CONTROLS.md
• Supervision and OTP: SUPERVISION-OTP.md

Networking and Services

• Networking and protocol stack: NETWORKING-PROTOCOL-STACK.md
• Web and gRPC: WEB-GRPC-HTTP.md
• Database, messaging, fs, process: DB-MESSAGING-FS-PROCESS.md
• Distributed execution: DISTRIBUTED-AND-RIGOR.md
• RaptorQ and distributed snapshots: RAPTORQ-DISTRIBUTED.md

Testing and Diagnostics

• Testing and forensics: TESTING-FORENSICS.md
• Lab runtime, DPOR, traces: LAB-TRACE-DPOR.md
• Mathematical foundations: MATHEMATICAL-FOUNDATIONS.md
• Observability and forensics: OBSERVABILITY-FORENSICS.md
• Error taxonomy: ERROR-TAXONOMY.md
• Troubleshooting: TROUBLESHOOTING.md

Codebase Navigation

• Source map, module map, types, workspace: SOURCE-MAP.md
• Stack surface guidance: STACK-SURFACES.md
• Browser / WASM: BROWSER-WASM.md
• Browser / React / Next: BROWSER-FRAMEWORKS.md
• Repo contributor guide: REPO-CONTRIBUTOR-GUIDE.md

Validation

When changing code:

• run the host project's normal formatter, compiler, lint, and test suite,
• add deterministic integration tests for the migrated path,
• verify cancellation, shutdown, and resource-release behavior,
• verify that no core domain code still depends on Tokio if the goal is full native adoption.

If working inside the Asupersync repo itself, see REPO-CONTRIBUTOR-GUIDE.md for mandatory compiler checks and testing discipline.

Operating Rules

• When forced to choose between "minimal code churn" and "native Asupersync semantics", choose the latter unless the task explicitly calls for a temporary boundary bridge.
• Forbidden crates in core: tokio, hyper, reqwest, axum, async-std, smol.
• Inside the Asupersync repo: follow AGENTS.md. Never delete files without permission. Branch is main, never master.

Adoption Lanes

Use this file to choose the right integration strategy before editing code.

Lane 1: Native Greenfield

Choose this when:

• you are starting a new Rust service, library, daemon, or CLI,
• you control most async boundaries,
• you want structured concurrency, deterministic testing, and explicit capability threading from day one.

Default moves:

• bootstrap with RuntimeBuilder,
• design async APIs around &Cx,
• use Scope / child regions for owned work,
• use LabRuntime and run_test_with_cx for tests,
• choose native Asupersync web/grpc/net/db/messaging surfaces instead of Tokio-ecosystem crates.

Exit criteria:

• no Tokio dependency in core code,
• no hidden ambient runtime assumptions,
• cancellation and shutdown are explicit in code and tests.

Lane 2: Brownfield Native Migration

Choose this when:

• the project already uses Tokio heavily,
• you can change function signatures,
• you want to move to full native Asupersync rather than sit on a compat layer forever.

Default moves:

• replace runtime bootstrap first,
• inventory every tokio::* and Tokio-ecosystem dependency,
• thread &Cx through code you control,
• replace tokio::spawn with region-owned work,
• migrate primitives domain by domain,
• add deterministic tests as each slice lands.

Exit criteria:

• Tokio is removed from core modules,
• migrated domains use native Asupersync surfaces,
• any remaining Tokio-only crate is isolated behind a dedicated adapter boundary.

Lane 3: Boundary Interop

Choose this when:

• a dependency still requires tokio::runtime::Handle, tokio::io, hyper runtime traits, or similar,
• replacing that crate immediately would cost too much,
• you need an incremental migration path.

Default moves:

• use asupersync-tokio-compat,
• keep the bridge in one adapter module or crate,
• pass Cx explicitly into the boundary,
• prefer strict cancellation modes,
• never let Tokio become the primary runtime for the app.

Exit criteria:

• the dependency is either removed or fully caged behind one small interop layer,
• business logic no longer knows about Tokio,
• the compat surface has an explicit removal plan.

Wrong Choices

Pick a different lane if you catch yourself doing any of these:

• "I will keep all current APIs and only swap the executor."
• "I will let Tokio and Asupersync both spawn freely in core code."
• "I will use compat everywhere because it is easier."
• "I will postpone deterministic tests until after the migration."

If you need native guarantees, your architecture has to move, not just the dependency graph.

Advanced Features Worth Exploiting

Once the basic migration is native, the next gains come from using Asupersync as more than a runtime replacement.

Start With The Three High-Leverage Deep Dives

• runtime shaping and operator controls,
• supervised/stateful application design,
• diagnostics, metrics, and failure forensics.

Those are where most "we switched runtimes but still think like Tokio" gaps show up.

Read:

• LEVERAGE-PLAYBOOK.md
• RUNTIME-CONTROLS-DIAGNOSTICS.md
• GREENFIELD-PATTERNS.md

Supervision, AppSpec, And Spork

The highest-value advanced app-model story is:

• AppSpec for application topology,
• supervision for restart policy and deterministic child ordering,
• actor / GenServer / Spork for stateful internal services,
• registry capability plus name leases for named components.

This is the right promotion path when a system has always-on workers, caches, control loops, subscription pumps, or restart domains. It is usually cleaner than trying to fake those concerns with detached tasks and ad hoc channels.

Resilience Combinators And Plan Rewrites

Do not reimplement resilience policy with open-coded loops and ad hoc select! logic if the system needs real orchestration.

High-value native surfaces include:

• quorum
• hedge
• adaptive_hedge
• bulkhead
• rate_limit
• retry
• bracket
• pipeline
• map_reduce
• circuit_breaker

Why they matter:

• they are already cancel-aware,
• loser-drain semantics are part of the design,
• budget and outcome behavior are explicit,
• the plan rewrite engine can optimize combinator DAGs without silently breaking cancel/drain/quiescence invariants.

Use these when building:

• gateways,
• parallel fan-out request paths,
• consensus/quorum workflows,
• data pipelines,
• rate-limited external integrations.

Relevant sources:

• src/combinator/
• src/combinator/laws.rs
• src/plan/rewrite.rs
• src/plan/analysis.rs

Also remember:

• keep Outcome::Cancelled and Outcome::Panicked distinct at policy boundaries,
• use tighter budgets for hedges, cleanup, and adapters,
• prefer lawful orchestration surfaces over open-coded select forests.

Remote / Distributed Surfaces

Asupersync has more than local task orchestration.

Important advanced surfaces include:

• named remote spawn instead of closure shipping,
• obligation-backed leases,
• idempotency store for retry-safe remote execution,
• session-typed protocol state machines,
• logical-time envelopes for causal correlation,
• saga compensation flow,
• distribution with quorum and optional hedging,
• RaptorQ-backed snapshot/distribution machinery.

Use these only when the target system actually has distributed semantics. They are not decorative features.

Relevant sources:

• src/remote.rs
• src/distributed/
• src/raptorq/

Advanced Service Edge Design

At the service edge, the high-value move is not just "port the router." It is:

• request-as-region isolation,
• least-privilege Cx narrowing,
• service/combinator composition instead of tower-first thinking,
• deadlines and cancellation made visible at the boundary.

Relevant references live in the skill entrypoint:

• web and gRPC patterns,
• runtime controls.

This is also where capability security becomes real instead of rhetorical:

• narrowed Cx for handlers,
• read-only contexts where appropriate,
• no hidden runtime globals or service locators,
• background components promoted into supervised app structure rather than booted from handlers.

Protocol Breadth And Maturity

Broadly strong native surfaces:

• HTTP/1.1
• HTTP/2
• TLS
• WebSocket
• database clients
• service/middleware composition

Surfaces to validate before promising downstream parity:

• QUIC / HTTP3
• some messaging integrations
• some browser/wasm adapters
• niche distributed paths

That means the skill should steer users toward native breadth confidently, but still verify the exact advanced path they need.

Guidance For Ambitious Systems

• Use child-region budgets and root-region limits deliberately.
• Promote long-lived state into supervised structures instead of background task soup.
• Use diagnostics and lab forensics before issues become production folklore.
• Prefer native combinators and service layers over carrying tower/Tokio-era abstractions forever.
• Measure with real metrics or benches when tuning runtime knobs.
• Treat obligation-tracked channels, reply obligations, and lease cleanup as application-design tools, not only runtime internals.

Asupersync Anti-Patterns

These are the fastest ways to sabotage a migration.

Architecture Mistakes

• Treating Asupersync as a drop-in executor swap.
• Keeping Tokio as a silent co-runtime in core code.
• Hiding Cx in globals, thread-locals, or hidden framework state.
• Building new features on compat because it is easier than going native.
• Treating RuntimeBuilder + block_on as the final architecture for a long-lived service that really wants AppSpec / supervision.
• Recreating a global process registry or service locator instead of using capability-scoped naming.

Concurrency Mistakes

• Leaving tokio::spawn or detached equivalents inside handlers and services.
• Starting request-local or task-local work with no owning region.
• Using race/select patterns that abandon losers without proving cleanup.
• Forgetting checkpoints in loops, retries, or long handlers.
• Holding wide cancellation masks around normal business logic instead of short cleanup-critical sections.

Resource / Cleanup Mistakes

• Holding permits, locks, or leases across indefinite waits.
• Assuming drop-based cleanup is good enough.
• Failing to verify quiescence and leak behavior after migration.
• Dropping AppHandle, named-server lease handles, or other obligation-like lifecycle handles without explicit resolution.
• Using plain channels where reply obligations or typed protocol edges should be explicit.

Testing Mistakes

• Converting runtime code but leaving #[tokio::test] patterns untouched.
• Using wall clock or ambient randomness in deterministic tests.
• Accepting non-deterministic flakes as normal after adopting Asupersync.
• Only testing happy-path completion and never testing cancel/drain/finalize behavior.
• Ignoring replay artifacts, futurelock warnings, or leak oracles because "the test usually passes."

API / Ergonomics Mistakes

• Assuming proc macros are more authoritative than manual APIs.
• Overusing Cx::for_testing() or Cx::for_request() instead of designing the real ownership flow.
• Passing full-capability Cx everywhere instead of narrowing at boundaries.
• Flattening Outcome::Cancelled and Outcome::Panicked into generic Err too early.
• Using Budget::INFINITE everywhere because budget design feels inconvenient.

Status / Capability Mistakes

• Assuming every feature documented in the repo is equally mature.
• Ignoring partial or unsupported classifications for QUIC/H3, SQLx compile-time macros, Kafka advanced consumers, Windows signals, or PTY support.

Recovery Rule

If you notice any of the above, stop optimizing for low churn. Rework the design around:

• explicit Cx,
• region-owned work,
• native replacements,
• deterministic validation,
• explicit boundary bridges only where unavoidable.

Brownfield Migration To Native Asupersync

This is the default migration path when you want full replacement, not permanent coexistence.

Order Of Work

1. Inventory runtime entrypoints, spawns, select/race logic, timers, channels, networking, web stack, database stack, tests, and any Tokio-locked third-party crates. 2. Replace the runtime bootstrap. 3. Introduce &Cx into the APIs you control. 4. Replace detached/background task patterns with region-owned work. 5. Migrate primitives by domain. 6. Add deterministic tests for each migrated slice. 7. Isolate any unavoidable holdouts behind compat. 8. Remove compat as the final step.

Replace Bootstrap First

Typical transformations:

• #[tokio::main] -> explicit RuntimeBuilder + block_on
• #[tokio::test] -> #[test] + run_test(...) or run_test_with_cx(...)
• implicit runtime handles -> explicit RuntimeHandle or Cx-scoped spawn paths

Thread &Cx Early

Do not wait until the end.

Refactor your own async APIs like this:

// before
async fn fetch_user(id: UserId) -> Result<User, Error>

// after
async fn fetch_user(cx: &Cx, id: UserId) -> Result<User, Error>

Benefits:

• cancellation becomes explicit,
• time/budget/randomness/tracing stop being ambient,
• testing becomes deterministic and easier to wire.

Replace Task Ownership Semantics

Look for:

• tokio::spawn
• JoinHandle used as detached background work
• handler-local tasks with unclear cleanup
• select! branches that abandon losing futures

Preferred outcomes:

• tasks become region-owned,
• handler/request work is scoped,
• losers are cancelled and drained where semantics require it,
• shutdown flows close to quiescence instead of "best effort."

Migrate By Domain

Do not migrate randomly. Use slices:

• sync and channels
• time and retries
• io and networking
• web/grpc
• database/messaging
• fs/process/signal
• advanced protocol surfaces

The detailed mapping is in TOKIO-MAPPING.md.

Compat During Migration

Use compat only when one of these is true:

• the dependency still demands a Tokio handle,
• it requires Tokio I/O traits or hyper runtime traits,
• removing it would force a much larger redesign than the current task allows.

Rules:

• keep compat in a dedicated boundary module,
• pass Cx into the boundary,
• never let Tokio leak back into business logic,
• plan the boundary's removal.

Brownfield Checklist

• runtime bootstrap replaced,
• &Cx threaded through owned code,
• Tokio spawns removed from handlers/services/core,
• native primitives adopted by domain,
• deterministic tests added,
• holdouts isolated,
• remaining partial/unsupported surfaces explicitly documented.

Browser And Framework Integration

Use this reference when the target includes browser execution, React, or Next.js. The browser lane is real, but it is not "run the entire native runtime everywhere JavaScript exists."

The First Decision: Direct Runtime Or Bridge-Only?

Do not blur this boundary.

If the environment is not a supported direct-runtime lane, keep runtime execution in a browser boundary and communicate over explicit RPC/API seams.

Profile Selection Is Mandatory

Choose exactly one wasm browser profile:

• wasm-browser-minimal
• wasm-browser-dev
• wasm-browser-prod
• wasm-browser-deterministic

Rules:

• exactly one canonical profile on wasm32
• native-only features are compile-time rejected
• browser onboarding should validate profile closure before framework work

Use profile intent correctly:

• minimal for contract/ABI checks
• dev for local development and diagnostics
• prod for production-lean envelope
• deterministic for replay-oriented validation

Vanilla Browser Pattern

Good direct-runtime posture:

• initialize in a real browser entrypoint
• keep capability boundaries explicit
• verify quiescence, cancellation, and security policy early

What to validate first:

• browser-ready handoff
• nested cancel cascade reaches quiescence
• browser fetch security/default-deny policy

Do not start with framework glue before the vanilla browser lane is green.

React Pattern

The repo's React guidance is more specific than "use an effect."

Canonical patterns:

• task groups with explicit cancellation UX
• bounded retry after transient failure
• bulkhead isolation between independent work groups
• tracing-hook transitions with deterministic scenario ids

Practical rules:

• component lifecycle should map cleanly onto scope ownership
• user cancel actions should drive explicit cancellation, not silent abandonment
• retries should stay bounded and observable
• sibling feature areas that can overload independently should use bulkhead

thinking rather than one shared failure domain

React anti-patterns:

• detached async work that outlives component lifecycle
• retries with no total budget
• effect cleanup that does not actually drain outstanding work
• unstructured logs that cannot be replay-correlated

Next.js Pattern

The important mental model is phase-based:

• ServerRendered -> Hydrating -> Hydrated -> RuntimeReady

Use that model explicitly.

Rules:

• runtime init belongs in client-hydrated code, not server or edge phases
• hard navigation and cache revalidation should be treated as explicit runtime

scope invalidations

• re-init should be deterministic and logged as such
• App Router boundaries are real lifecycle boundaries, not incidental framework

details

Good posture:

• keep browser runtime creation in client components or browser-only modules
• treat ServerRendered/ClientSsr runtime init failures as misuse, not a

flaky environment problem

• make rebootstrap on navigation or invalidation explicit

Browser Scheduler Semantics Matter

The browser adapter is not allowed to throw away the native scheduler model.

Important semantics from the repo docs:

• lane order still matters: cancel > timed > ready
• cancel fairness must remain bounded
• scheduler pump must be non-reentrant
• wake dedup must survive host-turn boundaries
• yield_now() must cooperate without monopolizing the same turn
• deterministic metadata should exist for parity and replay

Practical implication for downstream code:

• do not build UI/runtime glue that assumes unlimited same-turn microtask churn
• do not inline-poll on timer callbacks or wake callbacks
• treat main-thread starvation as a semantic bug, not just a UX bug

Worker Offload Is Policy-Governed

If browser runtime work moves into Web Workers, treat it as a policy boundary:

• ownership remains attached to the originating region/task
• cancellation must cross the worker boundary explicitly
• replay metadata must follow the job
• offload should not be used to hide scheduler bugs or unbounded main-thread

work

Unsupported Runtime Failures Are Useful

The browser stack deliberately throws unsupported-runtime diagnostics for bad contexts. Treat them as guidance, not noise.

Representative codes from the repo docs:

• ASUPERSYNC_BROWSER_UNSUPPORTED_RUNTIME
• ASUPERSYNC_REACT_UNSUPPORTED_RUNTIME
• ASUPERSYNC_NEXT_UNSUPPORTED_RUNTIME

Typical causes:

• attempted init in Node or SSR
• missing browser DOM/WebAssembly/fetch/runtime prerequisites
• direct runtime usage in server or edge paths

Correct response:

• move runtime creation into a supported client/browser boundary
• keep server/edge paths on bridge-only adapters

Evidence Contract For Browser Adoption

Browser work should produce artifacts, not just console impressions.

Capture:

• scenario id
• profile flags
• command bundle used
• pass/fail per step
• artifact paths
• failure excerpts and remediation hints

This matters because the browser lane has explicit policy, closure, redaction, and replay contracts.

Browser Troubleshooting Ladder

1. verify onboarding scenario bundle 2. verify dependency/profile policy 3. verify log-quality and redaction contracts 4. run targeted lifecycle/security/parity tests 5. escalate only with artifacts in hand

Treat missing artifacts as workflow failure.

High-Value Adoption Advice

• start with the vanilla/browser core lane before React or Next
• validate profile closure before fighting framework behavior
• keep runtime state in client-controlled lifecycle boundaries
• make cache invalidation and hard navigation explicit rebootstrap events
• use deterministic scenario ids and structured logs from the beginning

Anti-Patterns

• trying to run Browser Edition directly in Node, SSR, or edge by default
• mixing multiple canonical browser profiles in one wasm build
• assuming browser support means native DB/TLS/process/fs/server surfaces exist
• hiding lifecycle bugs behind retries or generic "hydration issue" language
• treating unsupported-runtime diagnostics as optional warnings

Read Next

• BROWSER-WASM.md
• TESTING-FORENSICS.md
• OBSERVABILITY-FORENSICS.md
• TROUBLESHOOTING.md

Browser And WASM Guidance

Use this only when the target actually includes browser or WASM deployment.

Current Support Posture

The repo's browser story is explicit and fail-closed:

• direct runtime support is for the browser main thread,
• SSR / server / edge / Node-only contexts are bridge-only or unsupported for direct browser runtime execution,
• canonical browser profiles are selected by feature flags.

Use this file as the lane chooser and posture summary.

For the detailed framework patterns and failure modes, read BROWSER-FRAMEWORKS.md.

Canonical Browser Profiles

The repo documents four canonical wasm browser profiles:

• wasm-browser-minimal
• wasm-browser-dev
• wasm-browser-prod
• wasm-browser-deterministic

Exactly one canonical browser profile should be selected for wasm builds.

Recommended posture:

• minimal for closure/contract checks
• dev for local diagnostics
• prod for production-lean browser envelope
• deterministic for replay-oriented validation

Important Constraints

Direct browser runtime does not mean "everything from native Asupersync works in the browser."

Expect browser-path exclusions around:

• native TLS
• native database features
• Kafka
• native filesystem/process/signal/server surfaces

Framework Guidance

Browser Edition docs define explicit boundaries for:

• browser-only modules,
• React client trees,
• Next.js client components,
• bridge-only server or edge paths.

Do not create runtime state in unsupported server or edge contexts and hope it will degrade gracefully. The repo explicitly rejects that posture.

Additional framework-specific guidance exists for:

• React task groups, retry, bulkhead isolation, and tracing hooks
• Next.js hydration/runtime phase boundaries and rebootstrap
• browser scheduler semantics and worker-offload policy
• unsupported-runtime diagnostics and evidence capture

When To Use This Lane

Only use Browser Edition directly when:

• you actually target browser execution,
• you can keep runtime creation in supported client-side environments,
• you can respect the direct-runtime vs bridge-only boundary.

Otherwise stay on native server-side Asupersync or use an explicit bridge architecture.

Browser Adoption Rules That Matter

• Validate profile closure before writing framework adapters.
• Get the vanilla browser path green before React or Next.
• Treat unsupported-runtime errors as useful guidance, not optional warnings.
• Keep runtime initialization inside supported client/browser boundaries.
• Capture artifacts for onboarding, replay, and policy failures instead of relying on console impressions.

Read Next

• BROWSER-FRAMEWORKS.md
• TESTING-FORENSICS.md
• OBSERVABILITY-FORENSICS.md

Budget, Outcome, And Capability Security

These three concepts are not side details in Asupersync. They are the control plane for application semantics.

If a migration keeps Cx but still thinks in terms of "plain Result, ambient authority, and best-effort cleanup", it has not really adopted Asupersync yet.

Outcome Discipline

Outcome<T, E> is deliberately four-valued:

• Ok(T)
• Err(E)
• Cancelled(CancelReason)
• Panicked(PanicPayload)

The repo treats this as a severity lattice:

• Ok < Err < Cancelled < Panicked

Practical downstream rule:

• preserve all four states as long as you can,
• collapse them only at a real policy boundary such as HTTP, CLI, RPC, queue

ack, or supervision policy.

Why this matters:

• Cancelled is not "just another error". It changes retry, shutdown,

observability, and drain behavior.

• Panicked is not a recoverable domain error. It is a stronger failure that

should usually page supervision, emit heavier evidence, or map to a hard service failure.

• outcome severity composes across joins, races, retries, and supervision in a

way a flattened Result<T, anyhow::Error> cannot.

Good policy boundary examples:

• HTTP: Cancelled -> 499, Panicked -> 500
• gRPC/service edge: map Cancelled to caller-aborted/deadline semantics,

not generic internal failure

• worker/queue loop: distinguish retryable application error from shutdown or

sibling fail-fast cancellation

Bad pattern:

• converting everything to Err(String) at the first adapter boundary

Budget Discipline

Budget is not a timeout convenience; it is the explicit statement of how much work a failure domain may consume.

Key fields in the repo's model:

• deadline
• poll quota
• cost quota
• priority

The important algebraic rule is meet():

• outer budget and inner budget combine by taking the tighter constraint,
• child work should usually inherit a stricter effective budget than the caller,
• budget propagation is part of correctness, not only performance tuning.

Practical downstream rules:

• give cleanup a bounded budget,
• give hedged or speculative work a tighter budget than the main request,
• give backoff/retry loops a total budget, not just per-attempt sleeps,
• do not use Budget::INFINITE for every request path just because it is easy.

Use budget deliberately by surface:

Good pattern:

• parent request gets the SLA budget,
• DB fallback or hedge gets a smaller child budget,
• shutdown/finalizer path gets a short masked cleanup budget.

Bad patterns:

• Budget::INFINITE everywhere
• using timeout wrappers without reasoning about the cleanup budget of the

cancelled work

• unbounded retry loops that ignore budget exhaustion

Cancellation Severity Matters

CancelReason is structured, not decorative. Examples in the repo include:

• User
• Timeout
• FailFast
• RaceLost
• ParentCancelled
• Shutdown

Use that structure.

Practical policy advice:

• RaceLost usually means "loser must drain quietly", not "error the request"
• Timeout often means retry or degrade
• Shutdown means stop acquiring new work and prioritize bounded cleanup
• FailFast often means sibling topology or supervision policy is in charge,

not local recovery

Capabilities Are The Security Model

The capability row is type-level and compile-time enforced:

• [SPAWN, TIME, RANDOM, IO, REMOTE]

The core repo model in src/cx/cap.rs matters for downstream design:

• capability rows are zero-cost marker types,
• SubsetOf encodes monotone narrowing,
• widening is compile-time rejected,
• marker traits are sealed to prevent external capability forgery.

That means least privilege is not just documentation. It can be part of the Rust type system.

What this buys downstream:

• handlers can get only the effects they actually need,
• framework wrappers can offer cx_readonly() or narrowly scoped cx_narrow(),
• application services can prevent accidental spawning, I/O, randomness, or

remote execution from the wrong layer,

• ambient service-locator style design becomes structurally harder.

Common Capability Shapes

Representative patterns visible in repo docs and wrappers:

Do not default to full All capabilities in every layer.

Framework Boundary Rule

The best Asupersync wrappers do this:

1. receive a real runtime-managed Cx, 2. wrap it in a framework-specific context, 3. narrow capability exposure at the boundary, 4. let deeper layers accept only the narrowed Cx they actually need.

Good examples to model:

• HTTP request regions via web::request_region
• gRPC call wrappers via grpc::CallContext::with_cx(...)

Masking Rule

mask() is for bounded release/finalize sections, not general control flow.

Use masking only when all of these are true:

• the code is in a narrow cleanup/reply/finalize section,
• the work has a bounded budget,
• you understand exactly what invariant must be preserved before cancellation

becomes observable again.

Do not use masking to hide sloppy cancellation handling.

Design Patterns That Actually Pay Off

Pattern 1: Preserve Outcome To The Edge

• internal services return Outcome or preserve equivalent information
• transport adapter decides how to map it
• retry/supervision policy sees the real failure class

Pattern 2: Budget The Whole Flow

• request gets a budget
• retry/hedge branches inherit tighter child budgets
• finalizers get separate bounded cleanup budgets
• tests assert exhaustion and cleanup behavior explicitly

Pattern 3: Narrow Authority Early

• handlers do not receive global singletons
• boundary wrapper exposes limited Cx
• pure domain code remains pure

Anti-Patterns

• passing a full-power Cx through the whole program "for convenience"
• flattening Cancelled and Panicked into ordinary error early
• using timeout as policy while ignoring loser-drain and finalization cost
• making Budget::INFINITE the default for all request paths
• masking wide sections of business logic instead of fixing protocol edges

When To Read Next

• For concrete request/call boundary patterns: WEB-GRPC-HTTP.md
• For runtime knobs and diagnostics: RUNTIME-CONTROLS-DIAGNOSTICS.md
• For supervision and long-lived apps: SUPERVISION-OTP.md
• For migration mistakes: ANTI-PATTERNS.md

Channel and Sync Primitive Internals

Two-Phase Channel Pattern

The core cancel-safety mechanism. All channels use reserve/commit:

// Phase 1: Reserve (cancel-safe, nothing committed)
let permit = tx.reserve(cx).await?;
// Phase 2: Commit (linear, must happen or abort)
permit.send(message);

Dropping a permit aborts cleanly. Message never partially sent.

MPSC Channel

Source: src/channel/mpsc.rs

Multi-producer, single-consumer with two-phase send.

let (tx, mut rx) = mpsc::channel::<T>(capacity);

// Send side (cancel-safe)
let permit = tx.reserve(&cx).await?;  // wait for capacity
permit.send(value);                    // cannot fail

// Receive side
match rx.recv(&cx).await {
    Ok(value) => { /* got value */ }
    Err(RecvError::Closed) => { /* all senders dropped */ }
}

• SendWaiter uses Arc<AtomicBool> for waker dedup
• Bounded capacity with backpressure
• try_send() for non-blocking attempts

Oneshot Channel

Source: src/channel/oneshot.rs

Single send, single receive with two-phase send.

let (tx, rx) = oneshot::channel::<T>();
let permit = tx.reserve(&cx)?;
permit.send(value);
let result = rx.recv(&cx).await?;

Broadcast Channel

Source: src/channel/broadcast.rs

Fan-out to multiple subscribers with waiter cleanup on drop.

let (tx, _) = broadcast::channel::<T>(capacity);
let mut rx1 = tx.subscribe();
let mut rx2 = tx.subscribe();

let permit = tx.reserve(&cx).await?;
permit.send(value);

// Lagging receivers get RecvError::Lagged(n)

Watch Channel

Source: src/channel/watch.rs

Last-value multicast. Always-current read.

let (tx, rx) = watch::channel(initial_value);
tx.send(new_value);

rx.changed(&cx).await?;  // wait for change
let val = rx.borrow_and_clone();

WatchWaiter uses Arc<AtomicBool> for waker dedup.

Session Channel

Source: src/channel/session.rs

Typed RPC with reply obligation. Reply is a linear resource.

Sync Primitives

All primitives are cancel-safe and deterministic under lab runtime.

Mutex

Source: src/sync/mutex.rs

let mutex = Mutex::new(42);
let mut guard = mutex.lock(&cx).await?;  // takes &Cx, returns Result
*guard += 1;
// guard drop releases lock

• Fair, cancel-safe, tracks contention
• Two-phase: Phase 1 (wait for availability) is cancel-safe, Phase 2 (acquire) cannot fail
• Each guard tracked as an obligation
• Uses parking_lot::Mutex internally for waiter queue
• Poison on panic

RwLock

Source: src/sync/rwlock.rs

let rw = RwLock::new(data);
let read = rw.read(&cx).await?;   // shared access
let write = rw.write(&cx).await?; // exclusive access

Writer-preference with reader batching.

Semaphore

Source: src/sync/semaphore.rs

let sem = Semaphore::new(permits);
let permit = sem.acquire(&cx, count).await?;
// permit is an obligation released on drop

Counting semaphore with permit-as-obligation model.

Barrier

Source: src/sync/barrier.rs

let barrier = Barrier::new(n);
let result = barrier.wait(&cx).await?;
if result.is_leader() { /* elected leader */ }

N-way synchronization with leader election.

Notify

Source: src/sync/notify.rs

let notify = Notify::new();
notify.notified().await;       // wait for notification
notify.notify_one();           // wake one waiter
notify.notify_waiters();       // wake all waiters

OnceLock (OnceCell)

Source: src/sync/once_cell.rs

let cell = OnceCell::new();
let val = cell.get_or_init(async { compute().await }).await;

Cancel-safe: failed init lets next caller retry.

Pool

Source: src/sync/pool.rs

Object pool with per-thread caches. Uses #[allow(unsafe_code)] for unsafe impl Send.

ContendedMutex

Source: src/sync/contended_mutex.rs

parking_lot::Mutex wrapper with optional lock-metrics instrumentation (wait/hold time tracking).

Cancel Safety Summary

Waker Dedup Pattern

Used across channels and sync primitives:

• Waker::will_wake() checks skip redundant clones
• Refresh only when executor context actually changes
• Reduces allocation and contention on wake paths

Waiter Registration Race Prevention

Sink and transport channels re-check capacity after waiter registration. This closes the lost-wakeup race between capacity check and registration.

Tokio Compat Boundary

asupersync-tokio-compat is real and useful, but it is not the preferred architecture.

When To Use It

Use compat only when a dependency still requires one of these:

• tokio::runtime::Handle::current()
• Tokio I/O traits
• hyper runtime traits
• a Tokio-hosted future that cannot be removed yet

Typical examples:

• reqwest
• axum
• tonic
• sqlx
• other crates that still assume Tokio is present

Hard Rules

• the main asupersync crate must not depend on compat,
• Tokio must never become the primary executor for the application,
• Cx must cross the boundary explicitly,
• adapter-spawned work must still be region-owned and cancellation-aware.

What Compat Actually Provides

• runtime bridge: with_tokio_context(...)
• sync context bridge for construction paths that need a Tokio handle
• Tokio <-> Asupersync IO adapters
• hyper executor/timer/body bridges
• tower bridge
• cancellation policies for wrapped Tokio futures

Recommended Boundary Shape

Keep the whole thing in one module or crate.

Pattern:

• core domain code exposes native Asupersync interfaces,
• adapter module owns the Tokio-specific client/service,
• adapter functions accept &Cx,
• compat is the only place where Tokio types appear.

Cancellation Policy Guidance

Compat exposes cancellation modes because Tokio-originated futures may not respect Asupersync semantics.

Prefer:

• strict handling when correctness matters,
• explicit timeout fallback only when you understand the operational tradeoff,
• best-effort only for low-risk glue where native semantics are impossible.

Removal Plan

Compat is successful only if it shrinks over time.

Good end state:

• domain and service code are fully native,
• one or two boundary modules remain for genuinely unavoidable third-party crates,
• or the compat layer is gone entirely.

Bad end state:

• compat spreads across the codebase,
• Tokio types leak into business logic,
• new features keep being built on the bridge instead of on native surfaces.

Compat Bridge

What It Is

asupersync-tokio-compat is a separate workspace crate for running Tokio-locked dependencies inside an Asupersync-centered application.

Important boundary rules from the repo docs:

• the main asupersync crate must remain Tokio-free
• the compat layer lives in its own crate
• Cx still crosses adapter boundaries explicitly
• region ownership and cancellation still matter

Feature Gates

When To Use It

Use compat when all of these are true:

1. A needed dependency is genuinely Tokio-locked. 2. Native replacement this cycle would blow the scope. 3. You can keep the boundary narrow and explicit. 4. You have a removal plan.

Good Uses

• Keep reqwest temporarily while migrating toward native HTTP clients.
• Run axum or tonic workloads through a bounded bridge while replacing vertical slices.
• Keep SQLx-adjacent pieces during a staged database migration.

Bad Uses

• "We want Asupersync branding but no real migration."
• Running separate uncoordinated Tokio and Asupersync thread pools.
• Mixing tokio::spawn and region-owned Asupersync work without a single owner model.

Documented Failure Modes

Best-Practice Policy

Prefer this order:

1. native surface 2. explicit compat bridge 3. removal of compat once the blocker is gone

Source Truth

• /data/projects/asupersync/asupersync-tokio-compat/Cargo.toml
• /data/projects/asupersync/asupersync-tokio-compat/src/lib.rs
• /data/projects/asupersync/docs/tokio_adapter_boundary_architecture.md
• /data/projects/asupersync/docs/tokio_interop_support_matrix.md
• /data/projects/asupersync/docs/tokio_migration_cookbooks.md

Database, Messaging, Filesystem, Process, Signal

This file covers the broad system-integration surfaces beyond the core runtime.

Database

Native database surfaces exist behind features:

• database::sqlite
• database::postgres
• database::mysql

Migration guidance:

• prefer native clients when doing full replacement,
• pass &Cx through transactional and query code,
• treat cancellation and deadlines as part of the API contract,
• make connection ownership and pooling explicit.

Important limitation:

• SQLx compile-time query! style macros are explicitly unsupported in the repo's migration matrix.
• If you depend on them today, either keep SQLx behind compat temporarily or redesign around native query paths.

Messaging

Native messaging surfaces exist, but some areas remain partial.

Be conservative with:

• Kafka advanced consumers,
• Redis cluster failover,
• NATS JetStream.

If your workload depends on a feature that the repo classifies as partial, either:

• validate it carefully before adopting it,
• or keep that slice behind a boundary bridge until the native surface is sufficient.

Filesystem

Prefer fs::* over tokio::fs.

Migration checklist:

• replace file reads/writes/metadata/path ops,
• test cleanup and rename semantics,
• validate any niche behavior such as symlink handling or platform quirks,
• keep deterministic or isolated fixtures in tests.

Process

Prefer process::* over tokio::process.

Migration checklist:

• replace command spawning,
• handle structured exit and shutdown,
• verify stdio flows,
• test cancellation and reaping behavior.

Known caveat:

• PTY-oriented workflows are explicitly unsupported and need an external crate or a kept boundary.

Signal

Prefer signal::* over tokio::signal.

Important caveat:

• Unix coverage is the strongest path,
• Windows signal coverage is still partial in the repo's matrix.

If Windows signal semantics are central to the app, validate them explicitly before claiming full native parity.

Distributed Execution And Rigor Stack

Asupersync's distributed story is not "ship closures to workers and hope timeouts clean things up."

It is built around the same principles as local execution:

• explicit ownership,
• explicit cancellation,
• explicit obligations,
• deterministic evidence.

Remote Execution Model

The repo's remote surface is centered on named computations, not arbitrary closure shipping.

Key properties:

• remote spawn executes a named computation with explicit input
• leases are obligation-backed
• retries are deduplicated by an idempotency store
• protocol transitions are session-typed and explicit
• logical clock metadata travels with protocol messages
• saga compensation is a first-class rollback model

This is a stronger model than "spawn some closure on another node".

What That Means For Downstream Integrators

Design remote work like this:

1. give work a stable name and explicit serialized input, 2. give retries an idempotency key, 3. model ownership with leases and lifetimes, 4. define compensation for side effects that may outlive partial failure, 5. preserve evidence so replay and diagnosis stay possible.

Good fits:

• workflow steps with explicit identities
• bounded remote compute jobs
• replicated state distribution
• orchestrated sagas with compensations

Bad fits:

• arbitrary closure capture
• implicit global mutable state on both sides
• retries without dedupe or lease semantics

Lease-Backed Naming Is A Design Tool

Names are not free strings in Asupersync; they can be modeled as lease-backed resources.

This matters for:

• supervised named services,
• service discovery inside one runtime tree,
• distributed role assignment,
• avoiding stale registrations during cancellation or restart.

If "who currently owns this name?" matters, use a lease-backed model instead of best-effort registration cleanup.

Distributed Protocol Design Rules

Use these rules when building on the remote/distributed surfaces:

• every step needs a stable identity
• every non-local effect needs a cancellation/compensation story
• every retryable message needs idempotency semantics
• every ownership transfer needs an explicit authority boundary
• every cross-node timeline should carry causal metadata

If you cannot explain the lease, idempotency, and compensation story, the design is not done yet.

Logical Clocks Are Not Academic Decoration

The runtime can use Lamport, Vector, or Hybrid logical clocks.

Use them deliberately when:

• you need causal explanations across tasks or nodes,
• traces must be correlated across distributed components,
• a race or partial-order bug cannot be explained by wall clock alone.

Do not force distributed diagnosis to depend purely on wall-clock timestamps.

Sagas, Not Hidden Rollbacks

Asupersync's distributed model expects forward work and compensations to be explicit.

Good posture:

• reserve external resources explicitly,
• commit only when the protocol says the effect is owned,
• define compensations for partial completion,
• test the compensation path deterministically.

Bad posture:

• "if anything fails, we will figure it out from logs"

RaptorQ And Snapshot Distribution

RaptorQ is not a random side module. It gives the runtime a deterministic, policy-driven way to distribute and recover state snapshots.

Practical downstream takeaway:

• if you need resilient snapshot or artifact distribution, this stack may be

more appropriate than ad hoc "send all bytes to every replica" approaches,

• recovery can be proof- and artifact-backed instead of opaque.

Use it when:

• snapshot fan-out is expensive,
• partial replica availability is normal,
• deterministic recovery and evidence matter.

The Rigor Stack: What It Buys You

Asupersync includes a lot of formal and statistical machinery. Do not treat it as decoration; translate it into operational advantage.

When To Pay The Rigor Tax

Use more of the rigor stack when the system has:

• high concurrency with nontrivial races
• costly failures during shutdown or fail-fast
• distributed ownership / saga complexity
• hard-to-reproduce incidents
• operator workflows that need evidence instead of anecdotes

You do not need every formal surface for every app. But you should know they exist and design so they remain usable.

Evidence-Led Design

The skill's recommended posture is:

• keep ids stable,
• keep cancellation explicit,
• keep outcomes distinct,
• keep traces and artifacts reproducible,
• keep protocol transitions typed and auditable.

That is what allows downstream teams to use replay, crashpacks, spectral warnings, progress certificates, and model-checking export meaningfully.

Anti-Patterns

• remote execution with opaque closures and no idempotency story
• distributed retries that can double-apply effects
• using wall clock alone for causal explanation
• treating saga compensation as an incident-response task instead of a protocol
• assuming the rigor tooling is only for the Asupersync maintainers

Read Next

• SUPERVISION-OTP.md
• OBSERVABILITY-FORENSICS.md
• TESTING-FORENSICS.md
• ADVANCED-FEATURES.md

Error Taxonomy and Diagnostics

Error Types

Source: src/error.rs, src/error/

Core Error

pub struct Error {
    kind: ErrorKind,
    category: ErrorCategory,
    recoverability: Recoverability,
    // ...
}

ErrorKind Variants

Recoverability

pub enum Recoverability {
    Recoverable,     // Retry may succeed
    NonRecoverable,  // Retry will not help
    Unknown,         // Caller must decide
}

RecoveryAction

pub enum RecoveryAction {
    Retry,
    RetryWithBackoff(BackoffHint),
    Abort,
    Escalate,
}

Common Runtime Errors

"ObligationLeak detected"

Cause: Task completed while holding an obligation (permit, ack, lease).

// WRONG: permit dropped without send/abort
let permit = tx.reserve(cx).await?;
return Outcome::ok(());  // Leak!

// RIGHT: always resolve obligations
let permit = tx.reserve(cx).await?;
permit.send(message);  // Resolved

Policy: Configurable via ObligationLeakResponse:

• Panic -- fail fast (good for lab/CI)
• Log -- practical production starting point
• Recover -- abort the leaked path, continue
• Silent -- rare, intentional only

Threshold-based escalation: LeakEscalation in runtime config.

If a leak is detected during thread unwinding, Panic downgrades to Log to avoid double-panic aborts.

"RegionCloseTimeout"

Cause: Region stuck waiting for children that won't complete.

// Fix: add checkpoints in loops
loop {
    cx.checkpoint()?;  // Allows cancellation
    // ... work ...
}

"FuturelockViolation"

Cause: Task holding obligations but not making progress.

// WRONG: await while holding permit
let permit = tx.reserve(cx).await?;
other_thing.await;  // If blocks forever -> futurelock
permit.send(msg);

// RIGHT: minimize hold duration
let msg = other_thing.await;
let permit = tx.reserve(cx).await?;
permit.send(msg);

Deterministic Test Drift

Symptom: Same seed produces different results.

Check for:

• std::time::Instant::now() (use cx.now())
• rand::random() (use cx.random_u64())
• HashMap/HashSet (use DetHashMap/DetHashSet)
• Non-deterministic I/O (use VirtualTcp)

Channel Errors

Outcome Handling

match outcome {
    Outcome::Ok(val) => { /* success */ }
    Outcome::Err(e) => { /* application error */ }
    Outcome::Cancelled(reason) => {
        // Structured: reason.kind tells you why
        match reason.kind {
            CancelKind::User => { /* explicit cancel */ }
            CancelKind::Timeout => { /* deadline exceeded */ }
            CancelKind::FailFast => { /* sibling failed */ }
            CancelKind::RaceLost => { /* lost a race */ }
            CancelKind::ParentCancelled => { /* parent region cancelled */ }
            CancelKind::Shutdown => { /* runtime shutdown */ }
        }
    }
    Outcome::Panicked(payload) => { /* task panicked */ }
}

HTTP mapping: Ok -> 200, Err -> 4xx/5xx, Cancelled -> 499, Panicked -> 500.

Diagnostics Surfaces

TaskInspector

Source: src/observability/task_inspector.rs

Introspects live task state: blocked reasons, obligation holdings, budget usage, cancellation status.

CancellationExplanation

Source: src/observability/diagnostics.rs

Traces full cancel propagation chain: who requested cancellation, why, and what was affected.

TaskBlockedExplanation

Identifies what a task is waiting on: lock, channel receive, semaphore, another task, etc.

ObligationLeak Diagnostics

Pinpoints which obligation was not resolved, who held it, and when.

Spectral Health Monitor

Source: src/observability/spectral_health.rs

Early-warning severity model over live wait graph: none / watch / warning / critical.

Progress Certificates

Source: src/cancel/progress_certificate.rs

Drain phase: warmup, rapid_drain, slow_tail, stalled, quiescent. With Freedman/Azuma confidence bounds.

Debugging Workflow

1. Reproduce under LabRuntime with fixed seed 2. Enable trace capture and futurelock detection 3. Check oracle failures (quiescence, obligation leak, loser drain) 4. Use TaskInspector for live task state 5. Use CancellationExplanation for cancel chain 6. Preserve crashpack and replay artifacts 7. Use evidence ledger for subtle failures

Greenfield Patterns

Golden Rules

1. Every effectful async function that matters should accept &Cx. 2. Concurrency belongs in regions/scopes, not detached executors. 3. Cancellation checkpoints belong in loops and long-running work. 4. Message and resource lifecycles should resolve obligations explicitly. 5. Deterministic tests are part of the design, not a later add-on. 6. Budgets belong to failure domains; do not make everything Budget::INFINITE. 7. Preserve Outcome::Cancelled and Outcome::Panicked until a real policy boundary.

Choose The Right Level

Do not make every greenfield app a pile of naked spawned tasks.

If the system has named workers, restart policy, or explicit startup/shutdown topology, graduate to AppSpec early instead of bolting those concerns onto raw task spawning later.

Minimal Bootstrap

Documented bootstrap pattern from docs/integration.md:

```rust,ignore use asupersync::{Cx, Outcome}; use asupersync::proc_macros::scope; use asupersync::runtime::RuntimeBuilder;

fn main() -> Result<(), asupersync::Error> { let rt = RuntimeBuilder::current_thread().build()?;

rt.block_on(async { let cx = Cx::for_request(); scope!(cx, { cx.trace("worker running"); Outcome::ok(()) }); });

Ok(()) }

Use this as an orientation example, not as a license to stop at request-scoped toy code.

## Long-Lived Service Skeleton

If the process has real always-on topology, graduate quickly from `block_on(...)`
to `AppSpec`:

use asupersync::app::AppSpec; use asupersync::supervision::RestartPolicy;

let app = AppSpec::new("api") .with_budget(app_budget) .with_registry(registry_cap) .with_restart_policy(RestartPolicy::OneForOne) .child(http_child()) .child(replication_child()) .start(&mut state, &cx, parent_region)?;

// later: stop / join explicitly

Important guidance:

- `AppSpec` is the right unit for long-lived service trees.
- `AppHandle` is a real lifecycle handle; resolve it explicitly.
- Put background loops and internal services under the app tree instead of
  smuggling them out through detached tasks.

## Runtime Shape Is Part Of The Design

Choose runtime preset and knobs intentionally.

- `current_thread()` for simple services, CLIs, or deterministic-first builds
- `low_latency()` for request/response systems
- `high_throughput()` for queue-heavy or batch-heavy servers

Then tune only what the workload actually needs:

- blocking pool bounds,
- deadline monitoring,
- root-region limits,
- observability/metrics,
- cancel-streak and governor controls if cancellation pressure matters.

## Native Function Shape

Preferred shape for effectful operations:

async fn do_work(cx: &Cx, input: Input) -> Result<Output, Error> { cx.checkpoint()?; // effectful logic here Ok(output) }

If the function is pure, keep `Cx` out of it.

## Capability-Narrowed Edge Pattern

At framework or handler boundaries, narrow `Cx` instead of passing full authority everywhere.

async fn handler(ctx: &RequestContext<'_>) -> Response { let cx = ctx.cx_narrow::<RequestCaps>(); cx.checkpoint()?; // handler logic with only the capabilities it actually needs }

This is the practical shape behind capability security in downstream apps.

## Owned Concurrency Pattern

scope!(cx, { let a = spawn!(async { worker_a(cx).await }); let b = spawn!(async { worker_b(cx).await }); let (ra, rb) = join!(a, b); (ra, rb) });

If macro caveats get in the way, fall back to explicit `Scope` APIs.

## Cancellation-Safe Send Pattern

let permit = tx.reserve(cx).await?; permit.send(message);

Do not reserve and then await unrelated work while holding the permit unless
you fully understand the failure mode.

## Orchestration Pattern

When the system needs retries, quorums, hedging, bulkheads, or structured cleanup, prefer native combinators over open-coded orchestration.

Good fit:

- fan-out request paths,
- external API integrations,
- consensus-ish flows,
- multi-stage processing pipelines.

Why:

- loser drain is explicit,
- budget behavior is explicit,
- the plan rewrite layer can optimize while preserving invariants.

## Web-App Pattern

Native high-level web API from `src/web/mod.rs`:

use asupersync::web::{Router, Json, State, get, post};

async fn list_users(State(db): State<Db>) -> Json<Vec<User>> { Json(db.list_users().await) }

async fn create_user(State(db): State<Db>, Json(input): Json<CreateUser>) -> StatusCode { db.insert(input).await; StatusCode::CREATED }

let app = Router::new() .route("/users", get(list_users).post(create_user)) .with_state(db);

For framework authors, prefer `Cx` wrappers instead of exposing the whole effect
surface to handlers.

## Actor / Supervision Pattern

If the app wants OTP-style components:

- use `actor.rs` for bounded mailbox actors
- use `gen_server.rs` for request/reply servers
- use `supervision.rs` for restart topology
- inspect `examples/spork_minimal_supervised_app.rs`

## Pick The Right Surface

| Need | Prefer |
|------|--------|
| local fork/join work | `Scope` + child regions |
| single-owner mailbox state | `actor` |
| typed request/reply state machine | `GenServer` |
| restartable service topology | `AppSpec` + `supervision` |
| protocol edge with linear reply/resource semantics | session / tracked channels |

Do not force all concurrency through one pattern.

## Budget And Outcome Discipline

Good default posture:

- give adapters, hedges, and cleanup phases tighter budgets than core request handling,
- keep `Cancelled` distinct from ordinary error for shutdown and retry policy,
- keep `Panicked` distinct from recoverable error,
- use masked cleanup sparingly and only for bounded release/finalize sections.

This is where Asupersync becomes structurally different from ad hoc async code.

## Capability-Boundary Pattern

Prefer boundaries that narrow authority:

- request region + narrowed `Cx` for HTTP,
- call context + narrowed `Cx` for gRPC,
- registry capability injection for named internal services,
- no ambient singleton service locators.

Read next:

- `WEB-GRPC-HTTP.md`
- `LEVERAGE-PLAYBOOK.md`

## Resilience Composition

Do not hand-write every timeout/retry/select pattern.

Reach for:

- `service::ServiceBuilder` for timeout, load shedding, concurrency limit, rate limit, and retry,
- combinators like `hedge`, `quorum`, and `bracket` when orchestration itself is part of the design,
- plan/rewrite surfaces when the orchestration graph becomes large enough to justify lawful rewriting.

If loser cleanup matters, prefer surfaces that make drain behavior explicit and testable.

If the app has multiple long-lived children or named services, prefer `AppSpec` as the root of the application instead of manual boot code.

## Greenfield Defaults By App Type

| App Type | Default Stack |
|----------|---------------|
| Internal HTTP API | `RuntimeBuilder` + `web` + `service` + `http` + `database` |
| gRPC service | `RuntimeBuilder` + `grpc` + `service` + `database` |
| Agent / worker system | `RuntimeBuilder` + `channel` + `sync` + `actor` / `GenServer` / `spork` |
| Protocol server | `RuntimeBuilder` + `io` + `net` + `codec` |
| Deterministic test harness | `LabRuntime` + targeted channels/sync/combinators |
| Browser runtime | Browser Edition lane only; use the browser / wasm reference in this skill |

## Greenfield Upgrade Triggers

Move from "basic native runtime" to the richer Asupersync stack when you see any of these:

- background tasks that really want ownership and restart policy,
- request handlers spawning child work that must drain cleanly,
- need for named workers or registry leases,
- repeated retry/timeout/select logic that wants combinators,
- operator need to explain stuck work or stalled shutdown,
- distributed or quorum-aware coordination requirements.

Lab Runtime, DPOR, and Trace Infrastructure

LabRuntime

Source: src/lab/runtime.rs, src/lab/config.rs

Deterministic runtime for testing. Same seed = same execution = reproducible bugs.

Configuration

let lab = LabRuntime::new(
    LabConfig::new(42)               // seed for deterministic scheduling
        .max_steps(100_000)          // prevent infinite loops
        .panic_on_leak(true)         // obligation leaks fail fast
        .futurelock_max_idle_steps(1000) // detect stuck tasks
        .panic_on_futurelock(true)
        .capture_trace(true),        // enable trace replay
);

lab.run(|cx| async {
    cx.region(|scope| async {
        scope.spawn(task_under_test);
    }).await
});

// Oracle checks
assert!(lab.obligation_leak_oracle().is_ok());
assert!(lab.quiescence_oracle().is_ok());

Futurelock Detection

Not time-based -- obligation-based. Detects tasks that:

• Still hold pending obligations
• Are not making poll progress
• Have crossed futurelock_max_idle_steps threshold

Emits TraceEventKind::FuturelockDetected with task, region, and held-obligation details. Can panic immediately via panic_on_futurelock.

Chaos Injection

Source: src/lab/chaos.rs

Deterministic and seed-bound. Pre-poll and post-poll injection points:

• Cancellation injection
• Delay injection
• Budget exhaustion
• Wakeup storms

Presets: with_light_chaos(), with_heavy_chaos(), with_chaos(...) for focused campaigns.

Snapshots

Source: src/lab/snapshot_restore.rs

Restorable snapshots with deterministic content hashes. Structural validation checks:

• Reference validity
• Region-tree acyclicity
• Closed-region quiescence
• Timestamp consistency

Crashpacks

Source: src/trace/crashpack.rs

Deterministic crashpack linkage: stable id/path/fingerprint plus replay command metadata. Auto-attached on failing lab runs.

Oracle Suite

Source: src/lab/oracle/

Available Oracles

• Quiescence oracle: verifies region close implies no live children
• Obligation leak oracle: verifies all obligations resolved
• Loser drain oracle: verifies race losers fully drained
• Cancellation protocol oracle: verifies request -> drain -> finalize sequence

E-Process Monitoring

Source: src/lab/oracle/eprocess.rs

Anytime-valid monitoring using supermartingale-based testing. Can peek after every scheduling step with controlled type-I error (Ville's inequality).

Evidence Ledger

Source: src/lab/oracle/evidence.rs

Structured evidence with Bayes factors and log-likelihood contributions for subtle failures.

Conformal Calibration

Source: src/lab/conformal.rs

Split conformal prediction for oracle anomaly thresholds. Distribution-free, finite-sample coverage guarantees under exchangeability.

Virtual Time Wheel

Source: src/lab/virtual_time_wheel.rs

Deterministic virtual time with explicit tie-breaking. Sleeps complete instantly; time is controlled by the lab scheduler.

DPOR Schedule Explorer

Source: src/lab/explorer.rs

DPOR-style schedule exploration treating executions as Mazurkiewicz traces:

• Track coverage by equivalence class fingerprints
• Prioritize exploration based on trace topology
• Deterministic, replayable concurrency debugging with coverage semantics

Trace Infrastructure

Canonicalization

Source: src/trace/canonicalize.rs

Mazurkiewicz trace monoid: two traces differing only by swapping adjacent independent events are equivalent. Canonicalized to Foata normal form for stable fingerprints.

Geodesic Normalization

Source: src/trace/geodesic.rs

Constructs valid linear extensions minimizing owner switches via A* solver. Smaller, more canonical traces for diff/replay/minimize.

Race Detection

Source: src/trace/dpor.rs, src/trace/independence.rs

Vector clocks per task plus resource-footprint conflicts. Backtracking point extraction for systematic interleaving exploration.

Persistent Homology

Source: src/trace/boundary.rs, src/trace/gf2.rs, src/trace/scoring.rs

Topological signals from commuting diamond complexes. Betti numbers quantify scheduling freedom. GF(2) bitset algebra.

Sheaf Consistency

Source: src/trace/distributed/sheaf.rs

Detects global inconsistency in distributed obligation tracking that evades pairwise checks.

TLA+ Export

Source: src/trace/tla_export.rs

Export traces as TLA+ behaviors for bounded TLC model checking of core invariants.

Vector Clocks

Source: src/trace/distributed/vclock.rs

Causal ordering for distributed tracing. Lamport, Vector, and Hybrid logical clock modes.

Scenario-Based Testing

Source: src/lab/scenario.rs, src/lab/scenario_runner.rs

Reusable scenario YAML for: heavy chaos, partitions, host crash/restart, clock skew/lease behavior, cancellation campaigns.

# examples/scenarios/partition_heal.yaml
# examples/scenarios/clock_skew_lease.yaml

Test Artifact Outputs

When ASUPERSYNC_TEST_ARTIFACTS_DIR is set:

• event_log.txt
• failed_assertions.json
• repro_manifest.json
• JSON summaries for replay automation

Practical Test Shapes

Minimal Deterministic Test

#[test]
fn test_cancel_safety() {
    let lab = LabRuntime::new(
        LabConfig::new(42)
            .panic_on_leak(true)
            .capture_trace(true),
    );
    lab.run(|cx| async { /* test logic */ });
    assert!(lab.obligation_leak_oracle().is_ok());
    assert!(lab.quiescence_oracle().is_ok());
}

Using Test Helpers

test_utils::run_test(async { /* simple async test */ });
test_utils::run_test_with_cx(|cx| async move { /* test with Cx */ });

Determinism Rules

• Never use std::time::Instant::now() -- use cx.now()
• Never use ambient RNG -- use cx.random_u64()
• Prefer util::DetHashMap/DetHashSet over std::collections::HashMap/HashSet
• Use VirtualTcp for network tests instead of real sockets

Leverage Playbook

Most of Asupersync's value appears only after you stop thinking in terms of "what replaces tokio::spawn?" and start thinking in terms of ownership, budgets, obligations, and replayable lifecycle control.

1. Treat Budget And Outcome As Design Inputs

Do not treat Budget and Outcome<T, E> as decorative metadata.

• Use tighter child-region budgets for risky or secondary work: hedges, retries, adapter bridges, cleanup, background replication.
• Keep Outcome::Cancelled and Outcome::Panicked visible at orchestration boundaries. Flattening them into Err throws away shutdown, retry, and diagnostic meaning.
• Use cancellation masking only for short, cleanup-critical sections. A wide masked region is usually a design smell.
• Prefer explicit shutdown policies over "best effort" drop behavior.

Relevant paths:

• README.md
• src/types/
• src/cancel/
• src/cx/cx.rs

2. Choose The Right Concurrency Surface

Do not force everything into one abstraction.

Rule of thumb:

• use plain Scope when work is local and tree-shaped,
• use actor when state ownership is the main issue,
• use GenServer when reply semantics and lifecycle discipline matter,
• use AppSpec when the program has a real application topology.

Relevant paths:

• src/app.rs
• src/actor.rs
• src/gen_server.rs
• src/supervision.rs
• examples/spork_minimal_supervised_app.rs

2.5 Promotion Triggers: When To Upgrade Your Design

Agents often underuse Asupersync by staying on the smallest surface too long. Use these symptom-to-upgrade rules.

3. Model Long-Lived Apps Explicitly

For long-lived services, RuntimeBuilder + block_on is not enough by itself.

AppSpec buys you:

• a region-owned supervision tree,
• deterministic child start order,
• root-budget propagation,
• registry capability injection,
• explicit AppHandle lifecycle (stop / join) instead of silent leaks.

Design advice:

• Put always-on workers, replication loops, sidecar observers, and control-plane services under AppSpec.
• Treat AppHandle and named-server handles as obligation-like lifecycle handles: resolve them explicitly, do not casually drop them.
• Use restart policy and supervision strategy to encode failure domains instead of rebuilding custom watchdog threads.

Relevant paths:

• src/app.rs
• src/supervision.rs
• docs/spork_glossary_invariants.md

4. Use Capability-Scoped Boundaries

The service boundary pattern is:

• request/call gets its own region,
• handler receives metadata plus Cx,
• handler narrows capability set,
• spawned child work stays owned by that boundary region.

Use:

• web::request_region::{RequestRegion, RequestContext}
• RequestContext::cx_narrow::<...>()
• RequestContext::cx_readonly()
• grpc::CallContext::with_cx(...)
• CallContextWithCx::cx_narrow::<...>()

Important guidance:

• Cx::for_request() is a convenience seam, not the center of a production architecture.
• Do not pass full-capability Cx through every handler if most handlers only need trace/time/spawn.
• Do not rebuild ambient registries, global service locators, or hidden runtime handles.

Relevant paths:

• docs/integration.md
• src/web/request_region.rs
• src/grpc/server.rs

5. Prefer Native Orchestration Over Hand-Rolled select! Forests

Asupersync has a richer orchestration story than "manually race futures."

Use service layers for boundary resilience:

• ServiceBuilder::timeout(...)
• ServiceBuilder::load_shed()
• ServiceBuilder::concurrency_limit(...)
• ServiceBuilder::rate_limit(...)
• ServiceBuilder::retry(...)

Use combinators when the orchestration itself is the design:

• hedge for latency tails,
• quorum for M-of-N workflows,
• bracket for acquire/use/release correctness,
• join / race / timeout where loser-drain behavior is part of the contract.

Use the plan/rewrite layer when orchestration becomes a real DAG and you want lawful rewrites instead of hand-maintained nesting.

Important guidance:

• Only permit aggressive rewrites when branches are independent and cancel-safe.
• Losers must drain before the combinator returns when semantics require it.
• Prefer native streams over carrying tokio-stream forward out of habit.

Relevant paths:

• src/service/builder.rs
• src/combinator/
• src/plan/rewrite.rs
• src/combinator/laws.rs
• src/stream/

6. Use Obligation-Tracked Protocol Edges

If a protocol edge has "must send", "must reply", "must release", or "must unregister" semantics, use the tracked surface instead of pretending a plain channel send is sufficient.

Examples:

• reserve/commit sends for cancel-safe messaging,
• session channels for typed request/reply flows,
• GenServer calls for reply-obligation semantics,
• name leases for named worker registration,
• permit-backed semaphores and pools where release discipline matters.

Important guidance:

• Do not hold permits or leases across unrelated awaits.
• Prefer call over cast when the protocol requires acknowledgement or reply ownership.
• Use CastOverflowPolicy deliberately. Mailbox overflow policy is part of system semantics, not a default you should ignore.

Relevant paths:

• src/channel/mpsc.rs
• src/channel/session.rs
• src/gen_server.rs
• src/cx/registry.rs
• README.md

7. Understand The Distributed Model Correctly

The distributed story is not "ship arbitrary futures to another machine."

It is:

• named computation,
• serialized input,
• explicit remote capability,
• lease- and idempotency-backed lifecycle,
• saga/compensation-aware workflow recovery,
• logical-clock-aware tracing.

Design advice:

• Build a registry of remote computations instead of attempting closure shipping.
• Treat remote work as an obligation-backed child of local structured concurrency.
• Model compensations and idempotency keys up front for multi-step workflows.
• Test partition, heal, retry, and lease-expiry behavior under deterministic harnesses.

Relevant paths:

• src/remote.rs
• src/distributed/
• tests/calm_saga_integration.rs
• examples/scenarios/partition_heal.yaml
• examples/scenarios/clock_skew_lease.yaml

8. Build For Replay, Not Just Success

Asupersync is strongest when you lean into replayability and diagnostics.

Use:

• fixed seeds,
• trace capture,
• quiescence and obligation-leak oracles,
• futurelock detection,
• deterministic chaos injection,
• crashpacks and replay manifests,
• task inspector and structured explanations,
• evidence ledgers when failures are subtle.

Do this early:

• add deterministic tests for each migrated slice,
• keep artifact pointers and seeds for failures,
• treat "can replay the bad run" as a quality bar.

9. Do Not Overshoot Into Advanced Surfaces

Some Asupersync surfaces are real but should not be the default starting point for ordinary service work.

Do not lead with these unless the target requirements justify them:

• Browser Edition
• QUIC / HTTP3
• messaging integrations
• remote / distributed execution
• RaptorQ snapshot distribution

For most projects, the highest-leverage path is still:

• RuntimeBuilder + Cx + Scope
• request/call regions
• native service / web / grpc
• native channels/sync/combinators
• deterministic tests and diagnostics from the start

Relevant paths:

• src/lab/
• src/trace/crashpack.rs
• src/observability/diagnostics.rs
• src/observability/task_inspector.rs
• src/lab/oracle/evidence.rs

Lock Ordering and Concurrency Discipline

Canonical Lock Order

When acquiring multiple locks, the strict order is:

E(Config) -> D(Instrumentation) -> B(Regions) -> A(Tasks) -> C(Obligations)

Violating this order causes deadlocks. This is enforced by:

• ShardGuard variants with label system
• Debug checks that verify acquisition order
• 23 dedicated tests for lock ordering correctness

Source: src/runtime/sharded_state.rs

ShardedState

Runtime state split into independently locked shards:

Why Independent Shards?

Hot-path polling proceeds without serializing every region or obligation mutation. Each shard can be locked independently when only one table is needed.

Multi-Shard Operations

Use ShardGuard to acquire multiple shards in canonical order. The guard variants enforce ordering at compile time (type system) and runtime (debug assertions).

ContendedMutex

Source: src/sync/contended_mutex.rs

Wrapper around parking_lot::Mutex with optional contention metrics (feature: lock-metrics):

• Wait time tracking
• Hold time tracking
• Contention event counting

Use for all shard locks in ShardedState.

Channel Waker Dedup

Pattern used across the codebase: Arc<AtomicBool> on:

• mpsc SendWaiter
• broadcast receivers
• watch WatchWaiter

Prevents duplicate wakeups and reduces contention on the wake path.

Worker Wake Coordination

• Idle -> Polling -> Notified state machine for centralized wake dedup
• Scheduling paths route through wake_state.notify()
• Wakes during poll are coalesced (no double-enqueueing)
• Waker::will_wake guards skip redundant clones on waiter registration

Lost-Wakeup Prevention

Multiple strategies used:

• Permit-style Parker with queue rechecks after wakeup
• Capacity re-checks after waiter registration (closes capacity-check/registration race)
• Both send and receive waiters woken on channel close

Intrusive Queue Links

Source: src/runtime/scheduler/intrusive.rs

• Links stored directly in TaskRecord
• Queue-tag membership checks (O(1) pop without allocation)
• Owner pop and thief steal stay O(1)

Atomic Counter Discipline

Source: src/runtime/scheduler/global_injector.rs

• Timed counters incremented before heap insert
• Saturating decrements on pop
• Cached earliest-deadline fast path
• Workers skip timed-lane mutex when no deadline work exists

Steal-Path Locality

Source: src/runtime/scheduler/local_queue.rs

• Local queues track pinned local tasks
• When none present, stealers take no-branch non-local path
• When locals exist, skipped/restored with SmallVec (allocation-free common path)

Migration to parking_lot

Runtime, scheduler, I/O, lab, networking, and transport internals all use parking_lot primitives where it improves lock-path cost. This was a deliberate, measured migration.

Rules

1. Always acquire in canonical order: E -> D -> B -> A -> C 2. Never hold a shard lock across an await point 3. Use ContendedMutex for shard locks (enables metrics) 4. Use ShardGuard for multi-shard operations 5. Prefer atomic operations over locks on hot paths 6. Use Waker::will_wake to skip redundant clone operations

Mathematical Foundations and Alien-Artifact Algorithms

Asupersync uses mathematically rigorous machinery where it buys real correctness, determinism, and debuggability. These are implemented, not aspirational.

Core Mathematical Framework

Formal Semantics

Small-step operational semantics in asupersync_v4_formal_semantics.md with Lean mechanization scaffold (formal/lean/Asupersync.lean).

Budget composition is semiring-like:

combine(b1, b2) =
  deadline   := min(b1.deadline,   b2.deadline)
  pollQuota  := min(b1.pollQuota,  b2.pollQuota)
  costQuota  := min(b1.costQuota,  b2.costQuota)
  priority   := max(b1.priority,   b2.priority)

Regret-Bounded Adaptive Cancel Preemption (EXP3/Hedge)

Source: src/runtime/scheduler/three_lane.rs

Deterministic EXP3/Hedge over candidate cancel-streak limits {4, 8, 16, 32}:

p_t(a) = (1 - gamma) * w_t(a) / sum_b w_t(b) + gamma / K
w_{t+1}(a) = w_t(a) * exp((gamma / K) * r_hat_t(a))

Importance-weighted reward: r_hat_t(a_t) = r_t / p_t(a_t).

Adapts to workload regime shifts while preserving deterministic replay and bounded starvation.

Variance-Adaptive Drain Certificates (Freedman + Azuma)

Source: src/cancel/progress_certificate.rs

Cancellation drain modeled as stochastic progress process:

P(M_t - M_0 >= x) <= exp(-x^2 / (2(V_t + c*x/3)))

Where V_t is predictable variation and c bounds one-step increments.

Phase classification: warmup, rapid_drain, slow_tail, stalled, quiescent.

Freedman provides tighter variance-aware bound; Azuma is conservative baseline.

Spectral Wait-Graph Early Warning

Source: src/observability/spectral_health.rs

Treats task wait-for graph as dynamic signal. Tracks:

• Fiedler trajectory (algebraic connectivity)
• Spectral gap/radius
• Nonparametric indicator stack: autocorrelation, variance ratio, flicker, skewness, Kendall tau, Spearman rho, Hoeffding's D, distance correlation
• Split conformal bounds for next-step prediction
• Anytime-valid deterioration e-process

Severity: none / watch / warning / critical.

Mazurkiewicz Trace Monoid + Foata Normal Form

Source: src/trace/canonicalize.rs

Two traces differing only by swapping adjacent independent events are equivalent. Canonicalized to unique Foata normal form:

M(Sigma, I) = Sigma* / equiv_I

Provides canonical fingerprints for schedule exploration and stable replay.

Geodesic Schedule Normalization

Source: src/trace/geodesic.rs, src/trace/event_structure.rs

Given dependency DAG (trace poset), constructs valid linear extension minimizing "owner switches" (context-switch entropy proxy) using deterministic heuristics and bounded A* solver.

DPOR Race Detection + Happens-Before

Source: src/trace/dpor.rs, src/trace/independence.rs

DPOR-style race detection using minimal happens-before relation (vector clocks per task) plus resource-footprint conflicts. Systematic interleaving exploration targeting truly different behaviors.

Persistent Homology of Trace Commutation Complexes

Source: src/trace/boundary.rs, src/trace/gf2.rs, src/trace/scoring.rs

Square cell complex from commuting diamonds. Betti numbers/persistence quantify "non-trivial scheduling freedom." Deterministic GF(2) bitset linear algebra and boundary-matrix reduction.

Prioritizes exploration toward rare concurrency behaviors.

Sheaf-Theoretic Consistency Checks

Source: src/trace/distributed/sheaf.rs

For distributed obligation tracking: detects obstructions where no global assignment explains all local observations. Catches split-brain saga states that evade pairwise checks.

Anytime-Valid Monitoring (E-Processes)

Source: src/lab/oracle/eprocess.rs, src/obligation/eprocess.rs

Ville's inequality: P_H0(exists t : E_t >= 1/alpha) <= alpha

Continuously monitor invariants without invalidating significance. Supports optional stopping -- peek after every scheduling step with controlled type-I error.

Conformal Calibration

Source: src/lab/conformal.rs

Split conformal prediction for oracle anomaly thresholds:

P(Y in C(X)) >= 1 - alpha

Finite-sample, distribution-free coverage under exchangeability across deterministic seeds.

Algebraic Law Sheets + Rewrite Engine

Source: src/combinator/laws.rs, src/plan/rewrite.rs, src/plan/analysis.rs

Explicit law sheet for combinators (severity lattices, budget semirings, race/join laws). Rewrite engine guarded by conservative static analyses:

• Obligation-safety lattice
• Cancel-safety lattice
• Deadline min-plus reasoning

TLA+ Export

Source: src/trace/tla_export.rs

Traces exported as TLA+ behaviors with spec skeletons for bounded TLC model checking.

Explainable Evidence Ledgers

Source: src/lab/oracle/evidence.rs

Structured evidence using Bayes factors and log-likelihood contributions. Agent-friendly debugging with equations, substitutions, and one-line intuitions.

Native Greenfield Asupersync

This is the preferred path when you control the architecture.

Build Around The Real Core

• RuntimeBuilder owns runtime bootstrap and process-level configuration.
• Cx is the capability token that carries cancellation, tracing, time, randomness, budget, and scoped authority.
• Scope owns spawned work and child regions.
• LabRuntime gives deterministic execution, replay, and invariant checks.
• AppSpec / supervision / actors / spork are higher-level composition layers when you need long-lived supervised systems.

Process Bootstrap

Pattern:

use asupersync::runtime::RuntimeBuilder;

fn main() -> Result<(), asupersync::Error> {
    let rt = RuntimeBuilder::current_thread().build()?;
    rt.block_on(async {
        // prefer runtime-managed contexts in production,
        // use Cx::for_request() only as a convenience seam
    });
    Ok(())
}

Useful runtime builder levers:

• worker count
• blocking pool bounds
• observability hooks
• deadline monitoring
• env or config-file overrides
• logical clock mode
• root-region limits

API Design Rules

• Put &Cx first in async APIs you own.
• Use Scope or child-region APIs for owned concurrency.
• Add checkpoints in loops, long retries, and handler bodies.
• Surface cancellation, panic, or cleanup semantics at orchestration boundaries.
• Narrow Cx capabilities at framework boundaries instead of passing full power everywhere.

Request / Service Shape

Good pattern:

• per-request region or per-call region,
• wrap request metadata and Cx together,
• narrow capabilities for handlers,
• let handler-spawned work live inside the request region.

Relevant repo patterns:

• web::request_region::{RequestRegion, RequestContext}
• grpc::CallContext::with_cx(...)

Concurrency Guidance

Prefer:

• Scope::spawn
• child regions with tighter budgets
• explicit race/join semantics that preserve loser draining where needed
• native channel and sync primitives

Be careful with:

• proc macros beyond scope!
• low-level Cx::race* variants that may drop losers instead of proving they drained

If loser drain matters, use the manual scope/task APIs that preserve the stronger semantics.

Supervision / OTP-Style Systems

Reach for:

• app::AppSpec
• actor
• gen_server
• supervision
• spork

Use these when your system has:

• long-lived workers,
• named processes or registries,
• restart strategies,
• explicit application startup/shutdown trees.

The best repo example is examples/spork_minimal_supervised_app.rs.

Greenfield Default Stack

For a fully native app, prefer this stack:

• runtime: RuntimeBuilder
• app/task model: Cx, Scope, child regions
• channels/sync: channel::*, sync::*
• time: time::*
• networking: net::*, tls::*, websocket::*
• web: web::*, service::*
• grpc: grpc::*
• database: database::*
• testing: test_utils, LabRuntime
• observability: observability::*

Greenfield Validation Checklist

• all owned async APIs accept &Cx,
• no detached tasks,
• checkpoints exist in long-running loops,
• service boundaries narrow capabilities,
• tests use deterministic helpers,
• no Tokio dependency is present in core code.

Networking and Protocol Stack

Asupersync ships a cancel-safe networking stack from raw sockets through application protocols. Every layer participates in structured concurrency.

Reactor and I/O

Reactor Backends

Source: src/runtime/reactor/

I/O Driver

Source: src/runtime/io_driver.rs

• Registrations are RAII-backed; deregistration treats NotFound as already-cleaned
• Token slabs are generation-tagged (blocks stale-token wakeups after reuse)
• Unknown tokens logged instead of panic (diagnostics under fault conditions)
• Oneshot waker semantics: reactor disarms interest after each readiness event, stream re-arms explicitly

epoll Specifics

• Edge-triggered and edge-oneshot modes
• Explicit PRIORITY/HUP/ERROR propagation
• Stale fd/token cleanup on ENOENT and closed-fd conditions (including fd-reuse edge cases)

io_uring Specifics

• Timeout expiry (ETIME) handled as timeout, not failure
• Stale completions for deregistered tokens ignored

TCP

Source: src/net/tcp/

• TcpStream, TcpListener, split reader/writer halves
• Registered with I/O reactor, oneshot waker semantics
• VirtualTcp (src/net/tcp/virtual_tcp.rs): fully in-memory TCP for lab tests, same API, deterministic

UDP

Source: src/net/udp.rs

Async UDP with send/receive and cancellation safety.

Unix Sockets

Source: src/net/unix/

Unix domain sockets with stream and datagram support.

DNS

Source: src/net/dns/

Async DNS resolution with address-family selection.

WebSocket

Source: src/net/websocket/

RFC 6455: handshake, binary/text frames, ping/pong, close frames with status codes. Split reader/writer for concurrent send/receive within same region.

HTTP/1.1

Source: src/http/h1/

• Chunked transfer encoding
• Connection keep-alive
• Streaming request/response bodies
• Integration with connection pool

HTTP/2

Source: src/http/h2/

• Frame parsing
• HPACK header compression
• Flow control
• Stream multiplexing over single connection
• Integration with connection pool

Connection Pooling

Source: src/http/pool.rs

Shared connection pool for HTTP/1.1 and HTTP/2 with keep-alive management.

Response Compression

Source: src/http/compress.rs

Optional response compression middleware.

TLS

Source: src/tls/

Wraps rustls for TLS 1.2/1.3:

QUIC and HTTP/3

Source: src/net/quic_core/, src/net/quic_native/, src/http/h3_native.rs

In progress. Native feature surfaces exposed via quic/http3 features.

Transport Layer

Source: src/transport/

Low-level delivery behavior above raw sockets and below protocol clients:

Shared channel close paths wake both send and receive waiters (no stranded operations).

Bytes

Source: src/bytes/

Zero-copy buffer types: Bytes, BytesMut, Buf, BufMut. Compatible API surface for buffer management across the stack.

Codec

Source: src/codec/

Encoding/decoding primitives and framing layer. Used by HTTP, WebSocket, gRPC, and database wire protocols.

gRPC

Source: src/grpc/

Native gRPC client/server with health checks. CallContext::with_cx(...) for capability-scoped handlers.

Web Framework

Source: src/web/

Router, extractors, middleware, request-region isolation. Request-as-region pattern for structured concurrency per request.

Service Layer

Source: src/service/

ServiceBuilder with middleware: timeout, load_shed, concurrency_limit, rate_limit, retry. Optional Tower adapter via tower feature.

Cancel Safety Across the Stack

All networking layers respect:

• Region budgets for reads/writes
• Cancellation drains connections cleanly
• Lab runtime substitutes virtual TCP for deterministic network testing
• Two-phase semantics where applicable (send permits on channels)

Observability And Failure Forensics

Asupersync's operator story is stronger than "export a few traces." Use it.

Three Layers To Understand

Do not collapse these into one vague "logging" concept.

Runtime Observability Surfaces

High-value user-facing surfaces include:

• ObservabilityConfig
• LogCollector
• metrics exporters / OTLP integration
• TaskInspector
• Diagnostics
• CancellationExplanation
• TaskBlockedExplanation
• ObligationLeak

Use them when the question is "what is the system doing right now?" rather than "can I replay this exact failure?"

Relevant paths:

• src/observability/mod.rs
• src/observability/diagnostics.rs
• src/observability/task_inspector.rs

Progress Certificates And Drain Phases

Asupersync does not reduce shutdown to "wait and hope."

The runtime tracks cancellation drain progress with explicit phase labels such as:

• warmup
• rapid_drain
• slow_tail
• stalled
• quiescent

Use this to distinguish:

• expected cleanup tail,
• true shutdown wedge,
• causal chain depth problems,
• resource/obligation leaks.

Relevant paths:

• README.md
• src/cancel/progress_certificate.rs

Task And Wait-Graph Diagnostics

Before adding more logs, ask the runtime:

• which task is blocked,
• what it is waiting on,
• which obligations it still holds,
• whether cancellation has propagated,
• whether the wait graph is degrading structurally.

This is what TaskInspector, TaskBlockedExplanation, CancellationExplanation, and spectral health diagnostics are for.

Relevant paths:

• src/observability/task_inspector.rs
• src/observability/diagnostics.rs
• src/observability/spectral_health.rs

Futurelock, Crashpacks, And Replay

Use this when a concurrency failure matters:

1. keep the seed, 2. keep the trace fingerprint, 3. keep the crashpack / replay pointer, 4. keep the oracle failures, 5. keep the reproduction command.

This turns "it wedged once in CI" into a reusable debugging asset.

Relevant paths:

• src/lab/runtime.rs
• src/trace/crashpack.rs
• TESTING.md

Evidence Ledger

Some failures are subtle enough that raw traces are not enough. Asupersync can also produce structured evidence-ledger output for invariant failures.

Use it when:

• the failure is probabilistic-looking but deterministic under replay,
• there are competing explanations for a leak or stall,
• you need machine- and human-readable justification for why the runtime thinks an invariant failed.

Relevant path:

• src/lab/oracle/evidence.rs

Practical Posture

For a serious service:

• enable structured observability,
• preserve replay artifacts for concurrency failures,
• use task inspector and diagnostics before speculative debug printing,
• use progress certificates to interpret drain behavior,
• treat futurelock and obligation-leak signals as design bugs, not random noise.

Performance And Scheduling Mental Model

Treat Asupersync as a runtime you can reason about, not a black-box executor.

The highest leverage performance gains usually come from making your workload cooperate with the runtime model:

• lane-aware scheduling,
• structured ownership,
• bounded cancellation pressure,
• low-contention state design,
• explicit blocking boundaries.

The Core Scheduler Model

The runtime uses a three-lane scheduler:

• cancel lane
• timed lane
• ready lane

Priority is explicit:

• cancel work outranks timed work,
• timed work outranks ordinary ready work,
• fairness is bounded rather than wishful.

What this means for downstream code:

• cancellation-heavy systems can remain responsive if code checkpoints and

cleanup is bounded,

• deadline-driven work benefits from explicit time/budget discipline,
• "ready" work should not assume it can monopolize the worker.

What Cooperates With The Runtime

Good performance/correctness shapes:

• long loops call cx.checkpoint()
• CPU-heavy loops chunk work into bounded pieces
• blocking work is isolated to blocking pools or other explicit boundaries
• !Send local tasks stay truly local and short
• speculative or hedged branches get tight budgets
• hot shared state is sharded or single-owned instead of globally locked

Good question to ask:

• if this task were cancelled or deprioritized here, would it release pressure

quickly and predictably?

What Fights The Runtime

Bad shapes:

• huge compute loops with no checkpoints
• wide masked sections
• many tiny tasks contending on one hot global mutex
• open-coded fire-and-forget background work
• blocking syscalls in core async paths
• treating cancellation as rare and therefore not performance-sensitive

These problems show up as fairness drift, deadline misses, lock contention, stalled drains, and bad shutdown tails.

Runtime Presets Are Architectural Choices

Choose the starting preset based on workload:

Then tune only if measurements justify it.

The repo already exposes knobs for:

• worker and blocking pool sizing
• poll budget / scheduling batch controls
• cancel-streak behavior and adaptive governance
• root-region limits
• deadline monitoring
• logical clock mode
• observability and leak response

Use those knobs after you understand the workload shape, not before.

Locking And Sharded State

Asupersync's own runtime state is sharded for a reason. Copy that lesson.

Canonical shard order in the runtime:

• E(Config) -> D(Instrumentation) -> B(Regions) -> A(Tasks) -> C(Obligations)

What downstream integrators should learn from that:

• separate hot-path mutation domains when possible,
• keep lock acquisition order deterministic when multiple locks are needed,
• do not introduce a "god mutex" around unrelated state,
• use ContendedMutex where you need evidence about wait/hold time.

If you must lock multiple structures, define an order and document it.

Locality Matters

The runtime distinguishes local !Send work and stealable Send work.

Downstream implication:

• pin truly local stateful work to a local owner when that helps locality,
• do not force everything into cross-worker sharing,
• do not create fake locality for work that is actually parallel and migratable.

The point is to align ownership and movement cost.

Cancellation Pressure Is Part Of Performance

Asupersync treats cancellation/drain as hot-path behavior, not rare cleanup.

Practical implications:

• short cleanup paths matter for tail latency,
• loser-drain semantics in races are not optional bookkeeping,
• supervision and shutdown behavior affect scheduler pressure directly,
• speculative work should be budgeted so cancel storms remain bounded.

If shutdown or fail-fast is important in your service, benchmark and test the drain path, not only the success path.

Blocking Pool Discipline

Use explicit blocking boundaries for:

• file-heavy native work
• legacy drivers
• CPU-bound sync libraries
• niche system integration that is not yet exposed natively

Rules:

• keep the blocking surface narrow,
• own the handoff explicitly,
• measure whether blocking threads expand or retire sensibly,
• do not casually let blocking work seep into request code.

Diagnostics To Use While Tuning

High-value operator surfaces:

• deadline monitor
• TaskInspector
• blocked-task explanations
• obligation leak diagnostics
• lock metrics via ContendedMutex
• progress certificates and drain phase labels
• fairness counters such as yield/cancel streak telemetry

Use these before guessing.

Tuning Strategy

1. Pick the right ownership model. 2. Remove hot shared-state bottlenecks. 3. Add checkpoints and bound cleanup. 4. Isolate blocking work. 5. Only then tune runtime knobs.

If the architecture is wrong, builder tuning will not save it.

Practical Workload Heuristics

HTTP/gRPC server

• prefer low_latency()
• keep request handlers narrow-capability and short
• push long-lived subsystems into AppSpec or actors
• use ServiceBuilder for request-path backpressure instead of bespoke control

Queue/worker pipeline

• prefer high_throughput()
• batch where possible
• use bulkhead, rate_limit, and pools explicitly
• keep cancellation checkpoints in long loops

Deterministic test or forensic harness

• prefer current_thread() or LabRuntime
• turn on strict diagnostics
• make trace/replay artifacts part of normal investigation

Anti-Patterns

• tuning cancel streaks before adding checkpoints
• creating thousands of tiny tasks around one global lock
• running CPU-bound code inline because "it is still async"
• using masked sections to suppress cancellation churn instead of fixing cleanup
• treating fairness issues as mysterious when counters and diagnostics exist

Read Next

• RUNTIME-CONTROLS-DIAGNOSTICS.md
• OBSERVABILITY-FORENSICS.md
• TESTING-FORENSICS.md
• PRIMITIVES-AND-ORCHESTRATION-CHOOSER.md

Primitive And Orchestration Chooser

One of the biggest ways to underuse Asupersync is to treat every problem as "spawn a task, stick a mutex around state, and maybe add a timeout."

Asupersync gives you more precise tools. Use them precisely.

First Choose The Ownership Model

Before choosing a channel or lock, decide who owns the state and lifecycle.

If state already has one natural owner, do not turn it into shared-state-plus-locks just because that is what Tokio code often did.

Channel Chooser

Critical Asupersync distinction:

• mpsc and oneshot are two-phase send surfaces,
• reserve/commit exists to keep cancellation from half-sending work,
• session reply handles are linear resources and should be treated that way.

Good uses:

• watch for config snapshot / current status
• broadcast for event fan-out
• session for typed internal RPC where "forgot to reply" must become visible

Bad uses:

• watch as a durable event stream
• broadcast for linear reply protocols
• oneshot chains as a substitute for a real protocol

Sync Primitive Chooser

Practical rule:

• if the invariant is "exactly N concurrent uses", think Semaphore
• if the invariant is "single mutable state cell", think Mutex
• if the invariant is "resource checkout must resolve cleanly", think Pool
• if the invariant is "someone must answer this request", think session or

GenServer, not raw locks

Service Layer Vs Combinator Vs Actor

These are different tools, not substitutes.

Use ServiceBuilder when you want layered request semantics.

Use combinators when the graph itself matters:

• quorum writes
• hedged reads
• structured retries
• staged pipelines
• bulkhead isolation

Use actors or GenServer when there is one natural state owner and mailbox semantics matter more than middleware layering.

Combinator Chooser

Practical Selection Rules

Use GenServer instead of raw channels when:

• callers need typed call and cast semantics,
• reply obligations must never be forgotten,
• mailbox policy, stop semantics, or restart behavior matter.

Use session channels instead of mpsc + oneshot bundles when:

• you want the protocol itself to be linear and visible,
• reply resolution should participate in obligation accounting,
• cancellation behavior must be testable end to end.

Use Pool instead of ad hoc resource vectors when:

• checkout/release semantics matter,
• resources are expensive,
• cancellation during checkout/use must stay correct.

Use ContendedMutex on suspected hot locks when:

• you need evidence about wait/hold time,
• you are tuning sharded state or cache hot spots,
• you want lock metrics rather than intuition.

Primitive Choice By Common Migration Problem

Anti-Patterns

• using Mutex because ownership was not designed explicitly
• stuffing long-lived service topology into naked spawned tasks
• hand-writing select!-style spaghetti for timeout/retry/race logic
• using watch to represent must-process event history
• using broadcast when consumers only need the latest snapshot
• building internal RPC with loose mpsc messages and no reply obligation
• choosing primitives by familiarity instead of protocol semantics

Read Next

• GREENFIELD-PATTERNS.md
• SUPERVISION-OTP.md
• WEB-GRPC-HTTP.md
• ADVANCED-FEATURES.md

RaptorQ Fountain Coding and Distributed Systems

RaptorQ Overview

Source: src/raptorq/

RFC 6330 systematic RaptorQ codes: any K-of-N encoded symbols suffice to recover original K source symbols. Underpins distributed snapshot distribution.

Decoder Policy Selection

Runtime policy can choose:

• Conservative baseline
• High-support-first
• Block-Schur low-rank hard-regime plans

Based on extracted matrix features. Hard-regime transitions recorded with reason labels.

Dense-Factor Caching

Bounded capacity with hit/miss/eviction telemetry in decode stats.

GF(256) Kernel Selection

Deterministic per-process selection. Policy snapshots for dual-lane fused operations. Optional SIMD acceleration via simd-intrinsics feature (AVX2/NEON).

Validation

# Fast smoke
NO_PREFLIGHT=1 ./scripts/run_raptorq_e2e.sh --profile fast --bundle

# Full profile
NO_PREFLIGHT=1 ./scripts/run_raptorq_e2e.sh --profile full --bundle

# Forensics (includes repair_campaign perf smoke)
NO_PREFLIGHT=1 ./scripts/run_raptorq_e2e.sh --profile forensics --bundle

Outputs: summary.json, scenarios.ndjson, validation_stages.ndjson.

Distributed Primitives

Source: src/remote.rs, src/distributed/

Named Remote Spawn

Not closure shipping. Named computations with serialized input:

spawn_remote(cx, RemoteCap::new(), ComputationName("my_task"), input)

Lease Obligations

Leases are obligation-backed, participate in region close/quiescence.

Idempotency Store

Deduplicates spawn retries with TTL-bounded records and conflict detection.

Session-Typed Protocol

Origin/remote state machines validate legal spawn/ack/cancel/result/renewal transitions.

Saga Compensations

Forward steps and compensations tracked as structured rollback flow.

let saga = Saga::new("transfer")
    .step("debit", debit_fn, compensate_debit)
    .step("credit", credit_fn, compensate_credit);

Logical-Time Envelopes

Protocol messages carry logical clock metadata for causal correlation.

Consistent Hashing

Source: src/distributed/consistent_hash.rs

Deterministic consistent hashing for stable assignment. No iteration-order landmines.

Used for assigning encoded symbols to replicas in snapshot distribution.

Distributed Snapshots

Region state encoded via RaptorQ, symbols assigned via consistent hashing, recovery requires quorum of symbols from surviving nodes.

Security Layer

Source: src/security/

Per-symbol authentication tags prevent Byzantine symbol injection. Integrates with RaptorQ pipeline.

Testing Distributed Logic

• Test quorum loss, recovery, and cancellation explicitly
• Use VirtualTcp for deterministic network behavior
• Use lab scenarios: examples/scenarios/partition_heal.yaml, examples/scenarios/clock_skew_lease.yaml
• Test idempotency and lease expiry under chaos
• Verify saga compensations fire correctly
• Use src/lab/scenario.rs for repeatable validation

Distributed Model Summary

Stack Surface Guidance

Practical Inventory

Web / Service / gRPC Detail

web

High-level router surface:

• Router
• get, post, put, patch, delete
• Path, Query, Json, State, Cookie, CookieJar
• Json, Html, Redirect, Response, StatusCode

service

Middleware / service surfaces:

• Service, Layer, ServiceBuilder
• timeout
• concurrency limit
• rate limit
• retry
• buffer
• hedge
• load shed
• load balancing
• reconnect
• optional Tower adapter

grpc

Exports include:

• GrpcClient
• Server, ServerBuilder
• Channel, ChannelBuilder
• request/response/streaming types
• interceptors
• health checking
• reflection
• gRPC-web

Database Detail

Native database surfaces

• SQLite: blocking-pool bridge
• Postgres: async TCP wire protocol
• MySQL: async TCP wire protocol

Pool surfaces:

• DbPool
• AsyncDbPool
• transaction helpers in src/database/transaction.rs

Important caveat:

• SQLx compile-time query checking remains a notable gap in native replacement docs.

Actor / Spork Detail

Use these when the target system is naturally stateful or supervision-driven:

• src/actor.rs
• src/gen_server.rs
• src/supervision.rs
• examples/spork_minimal_supervised_app.rs

Recommendation Order

Default recommendation order:

1. Core runtime, cancellation, lab runtime 2. channels/sync/time 3. io/net/http/service/web 4. gRPC and database 5. actors/spork 6. browser or compat bridge 7. QUIC/H3, messaging, remote/distributed, RaptorQ only when explicitly needed