Migrate

Migrate to AgentControl

You're using a skill that will guide you through migrating an application from hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation. Your job is to run the migration in five stages, stopping at each stage for the user to confirm:

1. Audit the code — read-only scan that produces a structured list of everything hardcoded (prompt, model, parameters, tools, app-scoped knobs). 2. Wrap the call — install the SDK, create the config in LaunchDarkly with a fallback that mirrors the hardcoded values, and rewrite the call site to fetch the config fresh on every request. 3. Move the tools — extract each tool's JSON schema, attach it to the config, and swap every call site that references the old tool list. 4. Add tracking — wire the per-request tracker (duration, tokens, success/error) around the provider call. 5. Attach evaluators — either offline evals via the Playground + Datasets, or online judges that score sampled traffic automatically.

⚠️ Three first-run failure modes to avoid.

>

1. Tracker in the wrong scope. For an agent with a loop, mint create_tracker() once per user turn in a setup_run entry node — not inside call_model. Per-iteration factory calls produce N runIds and trip the at-most-once guards. See agent-mode-frameworks.md § Custom `StateGraph`.

2. `load_chat_model` wrapper reuse. Templates like langchain-ai/react-agent ship a load_chat_model(f"{provider}/{name}") helper that wraps init_chat_model(...) and silently drops every variation parameter. Delete it (don't just avoid using it) and replace call sites with create_langchain_model(ai_config).

3. Fallthrough not flipped after `/configs-create`. A freshly-created config's fallthrough points at an auto-generated disabled variation, so the SDK returns enabled=False until /configs-targeting runs. Flip it before Stage 2 verification.

Coverage — which shapes are well-trodden vs require extrapolation

The skill is optimized for Python and Node.js / TypeScript; other languages are install-only. Within Python and Node the coverage tiers are:

Reading the key: ✅ = follow the skill verbatim; ⚠️ = the architecture applies but you'll translate idioms or cross-reference another skill; ℹ️ = skill doesn't go past the install step.

If the target app is in the ⚠️ column, start by reading agent-mode-frameworks.md § Framework-agnostic invariants — those three rules (one agent_config per turn, one tracker per turn, at-most-once methods fire once at turn end) apply regardless of framework, and every code snippet in this skill is an instantiation of them. Translate the Python example's shape onto the target framework's primitives.

Prerequisites

This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment, and an application that already calls an LLM provider with hardcoded model, prompt, and parameter values.

Required environment:

• LD_SDK_KEY — server-side SDK key (starts with sdk-) from the target LaunchDarkly project

MCP tools used directly by this skill: none — every LaunchDarkly write happens in a focused sibling skill.

Check the SDK CHANGELOG before applying any pattern. The API surface described throughout this skill targets the SDK behavior at the time of the skill's last update; SDK releases can rename, remove, or split methods after that. Before you start, fetch the latest CHANGELOG for the SDK(s) you'll target and skim for anything that contradicts the pattern you're about to apply:

• Python: https://github.com/launchdarkly/python-server-sdk-ai/blob/main/packages/sdk/server-ai/CHANGELOG.md (and per-provider CHANGELOGs under packages/ai-providers/server-ai-{openai,langchain}/CHANGELOG.md)
• Node: https://github.com/launchdarkly/js-core/blob/main/packages/sdk/server-ai/CHANGELOG.md (and per-provider CHANGELOGs under packages/ai-providers/server-ai-{openai,langchain,vercel}/CHANGELOG.md)

If a CHANGELOG entry post-dates this skill and changes an API you're about to use, the CHANGELOG wins — and the skill should be updated.

Hand-off model. This skill does not auto-invoke other skills. At each stage that needs a LaunchDarkly write, this skill prepares the inputs (config key, mode, model, prompt, tool schemas, judge keys) and then tells the user to run the next slash-command themselves. After the user finishes that sibling skill, return to the next step here. Treat the "Delegate" lines below as next-step instructions, not auto-handoffs.

Sibling skills the user runs at each stage:

• projects — pre-Stage 2, only if no project exists yet
• configs-create — Stage 2 (creates the config and first variation)
• tools — Stage 3 (creates tool definitions and attaches them)
• configs-targeting — between Stage 2 and Stage 4 (promotes the new variation to fallthrough so the SDK actually serves it)
• online-evals — Stage 5 (attaches judges, creates custom judges)

Core Principles

1. Inspect before you mutate. Every stage begins with a read-only audit. Do not touch code until Step 1 is confirmed by the user. 2. Replace config, not business logic. The SDK call is a drop-in for the place where the model, parameters, and prompt are defined — not for the provider call itself. OpenAI/Anthropic/Bedrock calls stay where they are. 3. Fallback mirrors current behavior. The fallback passed to completion_config / agent_config must preserve the hardcoded values you removed, so the app is unchanged if LaunchDarkly is unreachable. 4. Stages are ordered. Wrap before you add tools. Add tools before you track. Track before you add evals. Skipping ahead produces configs without traffic, metrics without context, and judges with nothing to score. 5. Hand off to focused skills, manually. Each stage that needs a LaunchDarkly write tells the user to run a sibling slash-command (/configs-create, /tools, /configs-targeting, /online-evals) and waits for them to come back. This skill does not auto-invoke other skills.

Workflow

Minimum viable migration

Stages 1–4 (audit, wrap, tools, tracker) are independently shippable. A migration that stops after Stage 4 is complete, production-ready, and delivers the core value — externalized prompts and model config, targeting, variation A/B testing, and Monitoring-tab metrics. Stage 5 (evaluators) is a quality-of-life addition, not a gate. Do not block a Stage-4 rollout on evaluators; ship the run-scoped tracker path, verify metrics flow, then come back for Stage 5 when the team has time to curate a dataset.

That said, do not skip Stage 4. A migration without the tracker gives you externalized prompts but no visibility, which is most of the payoff left on the floor.

Step 1: Audit the codebase (Stage 1)

This is the first stage. It is read-only — no code writes, no LaunchDarkly resources created. The goal is to scan the repo and produce a structured manifest of every hardcoded value that needs to move, then hand the manifest back to the user for confirmation before any code is touched in Stage 2.

Use phase-1-analysis-checklist.md to scan:

1. Language and package manager — Python (pip/poetry/uv), TypeScript/JavaScript (npm/pnpm/yarn), Go, Ruby, .NET 2. LLM provider — OpenAI, Anthropic, Bedrock, Gemini, LangChain, LangGraph, CrewAI, Strands 3. Existing LaunchDarkly usage — any pre-existing LDClient or ldclient initialization to reuse 4. Hardcoded model configs — model name string literals, temperature / max_tokens / top_p, system prompts, instruction strings 5. Template placeholders in prompts — .format() calls, f-strings in prompt constants, JS/TS template literals, %(var)s, hand-rolled str.replace("__VAR__", ...). Flag each placeholder name and its runtime-value source; all get rewritten to Mustache {{ variable }} in Stage 2. 6. Externalized prompt files — scan YAML / JSON / TOML / Markdown / .prompt / .j2 files and prompt-template registries (langchain.hub.pull(...), LangSmith client.pull_prompt(...)) for prompts loaded at runtime. Common shapes: CrewAI agents.yaml / tasks.yaml, LangChain Promptfiles, k8s ConfigMap overlays, Pydantic Settings classes with prompt_* fields. Same Mustache rewrite (sub-step 5 of Stage 2) applies if the placeholder syntax differs. See phase-1-analysis-checklist.md § 4. 7. Hardcoded app-scoped knobs — search-result limits, retry budgets, tool-timeout overrides, feature toggles, any config-dataclass field that isn't a prompt or model parameter but still governs agent behavior. These belong in model.custom on the variation (not model.parameters, which is forwarded to the provider SDK and will crash on unknown kwargs). 8. Mode decision — completion mode (chat messages array) or agent mode (single instructions string). Completion mode is the default and the only mode that supports judges attached in the UI.

For each hardcoded target the audit finds, record:

• File path and line range
• Current value (model name, full prompt text, parameter dict)
• Target config field (model.name, model.parameters.temperature, messages[].content, instructions)
• Whether the surrounding call uses function calling / tools (drives Stage 3)
• Whether the surrounding call has retry logic (affects where Stage 4 tracker calls go)

This manifest is the contract for the next four stages.

Stage 1 output (return to user as a structured summary):

Language: Python 3.12
Package manager: uv
LLM provider: OpenAI
Existing LD SDK: none
Target mode: completion
Hardcoded targets:
  - src/chat.py:42   model="gpt-4o"
  - src/chat.py:43   temperature=0.7, max_tokens=2000
  - src/chat.py:45   system="You are a helpful assistant..."
Externalized prompt files: none (or e.g. "prompts/agents.yaml — CrewAI role/goal/backstory")
Prompt-template registries: none (or e.g. langchain.hub.pull("rlm/rag-prompt") at app.py:14)
Coverage totals: 3 hardcoded code targets · 0 externalized prompt files · 0 registry pulls
Proposed plan: single config key `chat-assistant`, mirror fallback, Stage 3 (tools) skipped (no function calling), Stage 4 (tracking) inline, Stage 5 (evals) attach built-in accuracy judge.

STOP. Present this summary, state the coverage totals out loud (e.g. "I found N hardcoded code targets and M externalized prompt files — does that match what you expected?"), and wait for the user to reply with one of four explicit forms:

• `confirm` — proceed to Stage 2.
• `add: <files or paths>` — re-run the audit with the new locations and present an updated summary.
• `fix: <correction>` — update a target in the list (provider, mode, prompt content, etc.) and ask again.
• `stop` — pause the migration here.

Do not interpret any other word — including skip, next, go, ok, proceed — as confirmation; ask the user to pick one of the four forms. This is the most important checkpoint in the workflow — if the audit is wrong, every stage after this will be wrong. The user should cross-check the hardcoded-targets list against what they know is in the code before giving the go-ahead.

Step 2: Wrap the call in the AI SDK (Stage 2)

This is the first stage that writes code. It has nine sub-steps.

1. Delete any hand-rolled model / tool wrappers the audit flagged. Do this before installing the new SDK so the replacement lands in a repo without confusing fallback imports. The two shapes the Stage 1 audit should have surfaced:

• `load_chat_model(f"{provider}/{name}")` or any `init_chat_model(...)` wrapper. Ships with langchain-ai/react-agent and many derivative repos. Delete the function and its module; the replacement is create_langchain_model(ai_config) (installed in the next sub-step). Leaving the wrapper in place means the next edit in this repo will import the familiar helper and silently drop variation parameters.
• Hand-rolled `resolve_tools` / `TOOL_REGISTRY` / `ALL_TOOLS` helpers that hard-code a static tool list. Delete them; ldai_langchain.langchain_helper.build_structured_tools(ai_config, TOOL_REGISTRY_DICT) is the canonical replacement and gets wired in Stage 3. If you leave the hand-rolled version, both shapes will live side-by-side and the next contributor will pick the familiar one.

Commit the deletion separately from the SDK install if the repo's review process benefits from it — otherwise bundle with sub-step 2.

2. Install the AI SDK. Detect the package manager from Step 1, then install:

• Python: launchdarkly-server-sdk + launchdarkly-server-sdk-ai>=0.20.0
• Node.js/TypeScript: @launchdarkly/node-server-sdk + @launchdarkly/server-sdk-ai@^0.20.0
• Go: github.com/launchdarkly/go-server-sdk/v7 + github.com/launchdarkly/go-server-sdk/ldai

Tier-2 provider packages (install in Stage 4, only if you're using the matching provider):

• OpenAI: launchdarkly-server-sdk-ai-openai>=0.4.0 (Python) / @launchdarkly/server-sdk-ai-openai@^0.5.5 (Node)
• LangChain / LangGraph: launchdarkly-server-sdk-ai-langchain>=0.5.0 (Python) / @launchdarkly/server-sdk-ai-langchain@^0.5.5 (Node)
• Vercel AI SDK (Node only): @launchdarkly/server-sdk-ai-vercel@^0.5.5
• Anthropic, Gemini, Bedrock — no provider package published; use Tier-3 custom extractor (see built-in-metrics)

3. Initialize `LDAIClient` once at startup. Reuse any existing LDClient — do not create a second base client. Place the initialization in the same module that owns existing app config.

Python:

   import os
   import ldclient
   from ldclient.config import Config
   from ldai.client import LDAIClient

   # Order matters: ldclient.get() raises if called before ldclient.set_config().
   # The set_config call is what initializes the singleton; .get() just returns it.
   sdk_key = os.environ.get("LD_SDK_KEY")
   if sdk_key:
       ldclient.set_config(Config(sdk_key))
   else:
       # Missing key: init in offline mode so the app still starts and the fallback
       # path runs on every call. Never raise at import time for a missing env var —
       # that turns a config gap into a boot failure.
       import logging
       logging.getLogger(__name__).warning(
           "LD_SDK_KEY not set; configs will use fallback values only."
       )
       ldclient.set_config(Config("", offline=True))

   ai_client = LDAIClient(ldclient.get())

Node.js/TypeScript:

   import { init } from '@launchdarkly/node-server-sdk';
   import { initAi } from '@launchdarkly/server-sdk-ai';

   // The Node SDK does not have an explicit offline mode — a missing or invalid
   // key fails fast during waitForInitialization, and every agent_config /
   // completion_config call returns the fallback. Log a warning; do not throw.
   if (!process.env.LD_SDK_KEY) {
     console.warn('LD_SDK_KEY not set; configs will use fallback values only.');
   }
   const ldClient = init(process.env.LD_SDK_KEY ?? 'sdk-offline');
   await ldClient.waitForInitialization({ timeout: 10 }).catch(() => {
     // Swallow init failures in offline mode; fallback path runs.
   });
   const aiClient = initAi(ldClient);

4. Hand off to `configs-create`. Print the extracted model, prompt/instructions, parameters, and mode from the Stage 1 manifest, then tell the user: "Run `/configs-create` with these inputs, then come back here." Supply the config key you want the code to call (e.g. chat-assistant). Do not attempt to auto-invoke the sibling skill — wait for the user to finish it before continuing.

After `configs-create` finishes, the user must also run `/configs-targeting` to promote the new variation to fallthrough. A freshly created variation returns enabled=False to every consumer until targeting is updated. Skip this and Stage 2 verification (sub-step 9 below) will silently take the fallback path on every request.

5. Rewrite template placeholders to Mustache syntax. If the hardcoded prompt interpolates runtime values with Python .format(), f-strings, JS template literals, or any other non-Mustache syntax (e.g. {system_time}, ${userName}, %(topic)s), rewrite every placeholder to {{ variable }} Mustache form. Do this in both the file you're about to send to /configs-create and the fallback string you'll write in sub-step 6. The AI SDK interpolates variables through a Mustache renderer on the LD-served path and the fallback path using the fourth-argument variables dict to completion_config(...) / completionConfig(...). Leaving a Python-style {system_time} literal in the fallback ships a silent regression when LaunchDarkly is unreachable — the renderer won't match the single-brace form and the literal {system_time} goes to the provider as part of the prompt.

Before:

   SYSTEM_PROMPT = "You are a helpful assistant. The time is {system_time}."
   prompt = SYSTEM_PROMPT.format(system_time=datetime.now().isoformat())

After (in source):

   SYSTEM_PROMPT = "You are a helpful assistant. The time is {{ system_time }}."
   # .format() is removed at the call site — the SDK interpolates via `variables`
   config = ai_client.completion_config(
       CONFIG_KEY,
       context,
       fallback,
       variables={"system_time": datetime.now().isoformat()},
   )

Common shapes to rewrite:

• Python "{var}" / "{var!s}" / "%(var)s" → "{{ var }}"
• JS/TS ` ${var} template literals inside prompt strings → "{{ var }}"`
• Any hand-rolled str.replace("__VAR__", value) scheme → "{{ var }}"

See fallback-defaults-pattern.md § Template placeholders for the fallback-specific variant.

6. Build the fallback. Mirror the hardcoded values you extracted. Use AICompletionConfigDefault / AIAgentConfigDefault in Python, plain object literals in Node. See fallback-defaults-pattern.md for inline, file-backed, and bootstrap-generated patterns.

Python fallback (completion mode):

   from ldai.client import AICompletionConfigDefault, ModelConfig, ProviderConfig, LDMessage

   fallback = AICompletionConfigDefault(
       enabled=True,
       model=ModelConfig(name="gpt-4o", parameters={"temperature": 0.7, "max_tokens": 2000}),
       provider=ProviderConfig(name="openai"),
       messages=[LDMessage(role="system", content="You are a helpful assistant...")],
   )

7. Replace the hardcoded call site. Swap the hardcoded model/prompt/params for a completion_config / completionConfig (or agent_config / agentConfig) call, then read the returned fields into the existing provider call. Keep the provider call intact.

Python — before:

   response = openai_client.chat.completions.create(
       model="gpt-4o",
       temperature=0.7,
       max_tokens=2000,
       messages=[
           {"role": "system", "content": "You are a helpful assistant..."},
           {"role": "user", "content": user_input},
       ],
   )

Python — after:

   context = Context.builder(user_id).set("email", user.email).build()
   config = ai_client.completion_config("chat-assistant", context, fallback)

   if not config.enabled:
       return disabled_response()

   params = config.model.parameters or {}
   response = openai_client.chat.completions.create(
       model=config.model.name,
       temperature=params.get("temperature"),
       max_tokens=params.get("max_tokens"),
       messages=[m.to_dict() for m in (config.messages or [])] + [
           {"role": "user", "content": user_input},
       ],
   )

Python — after (agent mode) — for LangGraph, CrewAI, or any framework that takes a goal/instructions string:

   context = Context.builder(user_id).kind("user").build()
   config = ai_client.agent_config("support-agent", context, FALLBACK)

   if not config.enabled:
       return disabled_response()

   # config is a single AIAgentConfig object — NOT a (config, tracker) tuple.
   # Obtain the tracker once per execution via the factory: tracker = config.create_tracker()
   model_name = f"{config.provider.name}/{config.model.name}"
   instructions = config.instructions
   params = config.model.parameters or {}

   # Pass model_name + instructions into your framework's agent constructor.
   # Example: LangGraph prebuilt agent (Python — `from langchain.agents import create_agent`;
   # this replaces `langgraph.prebuilt.create_react_agent`, deprecated in LangGraph 1.0
   # and removed in 2.0. Same return shape; `prompt=` was renamed to `system_prompt=`.)
   # agent = create_agent(
   #     create_langchain_model(config),  # forwards every variation parameter
   #     TOOLS,                            # Stage 3 will replace this with a config.tools loader
   #     system_prompt=instructions,
   # )

See before-after-examples.md for full Python OpenAI, Node Anthropic, and LangGraph agent-mode paired snippets.

8. Check `config.enabled`. If it returns False, handle the disabled path without crashing and without calling the provider. The check is required — not optional.

9. Verify. Run the app with a valid LD_SDK_KEY; confirm the call succeeds and the response matches pre-migration output. Then temporarily set LD_SDK_KEY=sdk-invalid (or unset it) and confirm the fallback path runs without error. Both paths must work before moving to Stage 3.

Delegate: `configs-create` (sub-step 4).

Step 3: Move tools into the config (Stage 3)

Skip this step if the audited app has no function calling / tools. Otherwise:

1. Enumerate the tools currently registered. Common shapes to look for:

• openai.chat.completions.create(tools=[...]) — OpenAI direct
• anthropic.messages.create(tools=[...]) — Anthropic direct
• create_agent(llm, tools=[...], system_prompt=...) — LangGraph prebuilt (Python, langchain.agents; replaces deprecated langgraph.prebuilt.create_react_agent)
• createReactAgent({ llm, tools: [...] }) — LangGraph.js prebuilt (Node, @langchain/langgraph/prebuilt)
• Agent(tools=[...]) — CrewAI
• Agent(tools=[...]) — Strands (Python @tool-decorated callables passed through the constructor; TS SDK uses Zod-schema tools)
• Custom `StateGraph` — module-level TOOLS = [...] list referenced in both model.bind_tools(TOOLS) and ToolNode(TOOLS). This is the langchain-ai/react-agent template shape; the list is usually in a tools.py module. Grep for bind_tools( and ToolNode( together — they will point at the same list.

Record each tool's name, description, and JSON schema.

For LangChain/LangGraph tools defined with @tool, extract the schema via tool.args_schema.model_json_schema() (or the equivalent Pydantic model_json_schema() call). For plain async callables used as tools (common in custom StateGraph shapes), LangChain infers the schema from the function signature at bind time — extract it via StructuredTool.from_function(fn).args_schema.model_json_schema(). Do not hand-write the schema.

2. Hand off to `tools`. Print the extracted tool names, descriptions, and schemas, then tell the user: "Run `/tools` with these tools and the variation key, then come back here." The sibling skill creates tool definitions (create-ai-tool) and attaches them to the variation (update-ai-config-variation). Wait for the user to finish before proceeding to sub-step 3. Do not auto-invoke.

3. Replace the hardcoded tools array at the call site with a read from config.tools (or the SDK equivalent for your language). Load the actual implementation functions dynamically from the tool names — see agent-mode-frameworks.md for the dynamic-tool-factory pattern from the devrel agents tutorial.

For custom `StateGraph` shapes, you must update both call sites: .bind_tools(TOOLS) and ToolNode(TOOLS) must both read from the same config.tools-derived list. Forgetting one leaves the LLM seeing the new tools but the executor still running the old ones, or vice versa.

4. Verify. Run the app; confirm the tool flows still execute correctly. get-ai-config (via the delegate) confirms the tools are attached server-side.

Delegate: `tools` (sub-step 2).

Step 4: Instrument the tracker (Stage 4)

Delegate: `built-in-metrics` wires the per-request tracker.track_* calls (duration, tokens, success/error, feedback) around the provider call. Use `custom-metrics` alongside it if the app needs business metrics beyond the built-in agent ones. Note: do not confuse this with launchdarkly-metric-instrument, which is for ldClient.track() feature metrics — a different API. See sdk-ai-tracker-patterns.md for the full per-method Python + Node matrix that the delegate skill draws on.

Hand off: print the config key, variation key, provider, and whether the call is streaming, then tell the user: "Run `/built-in-metrics` with these inputs, then come back here." Do not auto-invoke. Return here for sub-step 5 (verify) once they're done.

1. Create the tracker. Obtain a per-execution tracker via the factory on the config returned in Stage 2: tracker = config.create_tracker() (Python) or const tracker = aiConfig.createTracker(); (Node). Call the factory once per user turn and reuse the returned tracker for every tracking call in that turn — each call mints a fresh runId that tags every event emitted from the turn so they can be correlated via exported events or downstream queries. (The Monitoring tab aggregates today; run-level grouping is a downstream concern — but the runId is also what the SDK's at-most-once guards are keyed on, so minting a new one mid-turn breaks the guard semantics regardless of where the events end up.)

Where to call the factory depends on the call shape:

• Completion mode / one-shot provider call: mint the tracker right after completion_config(...) returns, in the same function that handles the request.
• Agent mode with a ReAct loop (LangGraph, LangChain, custom): mint the tracker in a dedicated setup_run entry node that executes once before the loop, stash it on graph state, and read it from state in call_model / tool handlers / a terminal finalize node. Emitting track_duration / track_tokens / track_success inside the loop body will trip the at-most-once guards. See agent-mode-frameworks.md § Custom `StateGraph` (run-scoped architecture) for the full setup_run + call_model + finalize pattern.
• Managed runner (Tier 1): skip this step entirely. ManagedModel mints the tracker internally per run() / invoke(). Move to sub-step 4 if that's what the app uses.

2. Pick a tier from the four-tier ladder. See sdk-ai-tracker-patterns.md § Tier decision table for the full table (chat loop → Tier 1; provider-package call → Tier 2; custom extractor → Tier 3; streaming/manual → Tier 4).

3. Wire the chosen tier. The delegate skill has full Python + Node examples for each tier plus per-provider files. A condensed Tier 2/3 example for reference — OpenAI via the provider package:

Python:

   from ldai_openai import get_ai_metrics_from_response
   import openai

   client = openai.OpenAI()

   tracker = config.create_tracker()

   def call_openai():
       return client.chat.completions.create(
           model=config.model.name,
           messages=[{"role": "system", "content": config.messages[0].content},
                     {"role": "user", "content": user_prompt}],
       )

   # Exceptions are tracked automatically — track_metrics_of catches
   # exceptions, records tracker.track_error(), and re-raises. Wrap your
   # own try/except only for local handling (logging, fallback).
   response = tracker.track_metrics_of(get_ai_metrics_from_response, call_openai)

Node:

   import { getAIMetricsFromResponse } from '@launchdarkly/server-sdk-ai-openai';

   const tracker = aiConfig.createTracker();
   // Exceptions are tracked automatically — trackMetricsOf catches
   // exceptions, records tracker.trackError(), and re-throws.
   const response = await tracker.trackMetricsOf(
     getAIMetricsFromResponse,
     () => openaiClient.chat.completions.create({
       model: aiConfig.model!.name,
       messages: [...aiConfig.messages, { role: 'user', content: userPrompt }],
     }),
   );

For Anthropic direct, Bedrock (no provider package), Gemini, and custom HTTP, write a small extractor returning LDAIMetrics — see the delegate skill's anthropic-tracking.md, bedrock-tracking.md, and gemini-tracking.md. LangChain single-node and LangGraph go through the launchdarkly-server-sdk-ai-langchain / @launchdarkly/server-sdk-ai-langchain provider package. Build the model with create_langchain_model(config) (Python) / createLangChainModel(config) (Node) — both forward all variation parameters — and track with get_ai_metrics_from_response / getAIMetricsFromResponse. See langchain-tracking.md.

4. Wire feedback tracking if the app has thumbs-up/down UI. Both SDKs expose trackFeedback with a {kind} argument.

Python:

   from ldai.tracker import FeedbackKind
   tracker.track_feedback({"kind": FeedbackKind.Positive})

Node:

   import { LDFeedbackKind } from '@launchdarkly/server-sdk-ai';
   tracker.trackFeedback({ kind: LDFeedbackKind.Positive });

Deferred feedback across processes. If the thumbs-up UI fires in a different process than the one that produced the response, do not call create_tracker() again in the consumer — that mints a new runId. Persist the tracker's resumption token (tracker.resumption_token in Python, tracker.resumptionToken in Node) alongside the message, then rehydrate the tracker with LDAIConfigTracker.from_resumption_token(...) (Python) or aiClient.createTracker(token, context) (Node) in the feedback handler.

5. Verify. Hit the wrapped endpoint in staging, then open the config in LaunchDarkly → Monitoring tab. Duration, token, and generation counts should appear within 1–2 minutes. If nothing shows up, walk the checklist in sdk-ai-tracker-patterns.md under "Troubleshooting."

Step 5: Attach evaluations (Stage 5)

1. Decide between three evaluation paths. This is the most commonly misunderstood stage — there are three paths, not two, and the right default for a migration context is often the one people skip.

Most migration users should start with offline eval, then add programmatic direct-judge only if they need continuous live scoring after the rollout is stable.

2. For agent-mode migrations, default to offline eval. UI-attached auto judges are completion-mode only today. The documented path for agent mode is either (a) offline regression via the LD Playground + Datasets (works for all modes), or (b) programmatic direct-judge wired into the call site. Generate a starter dataset CSV from the audit manifest (one representative input per row) and point the user at the Offline Evals guide for the Playground walkthrough. Only wire programmatic direct-judge into production code if the user explicitly asks for continuous live scoring.

Recommended offline-eval shape for a migration:

• Run the default variation (or whichever variation mirrors the pre-migration hardcoded behavior) against the dataset first — this is the baseline.
• Clone it into a second variation pointing at a different model family (e.g., if the baseline is anthropic/claude-sonnet-4-5, clone to openai/gpt-4o or openai/gpt-4o-mini). The comparison is most informative across families, not across siblings.
• Attach the built-in Accuracy judge with a pass threshold of 0.85, and run both variations against the same dataset.
• Promote the winner to fallthrough via /configs-targeting only if it beats the baseline on Accuracy and does not regress on Relevance or Toxicity.

Write this shape into the project's datasets/README.md (or equivalent) so the comparison pattern is reproducible after the migration ships.

3. Hand off to `online-evals` — only for UI-attached judges (completion mode) or to create custom judge configs that will be referenced by the programmatic path. Tell the user: "Run `/online-evals` with these inputs, then come back here." Do not auto-invoke. Pass:

• The parent config key and variation key
• A list of built-in judges (Accuracy, Relevance, Toxicity) or custom judge keys to create/attach
• Target environment

The delegate handles creating custom judge configs, attaching them via the variation PATCH endpoint, and setting fallthrough on each judge config. Offline eval does not go through this delegate — it's a Playground workflow, not an API write.

4. For programmatic direct-judge: wire `create_judge` + `evaluate` + `track_judge_result`. This is the only path at Stage 5 that writes code. The Python shape:

   from ldai.client import AIJudgeConfigDefault

   judge = ai_client.create_judge(
       judge_key,                               # judge config key in LD
       ld_context,
       AIJudgeConfigDefault(enabled=False),     # fallback: skip eval on SDK miss
   )

   if judge and judge.enabled:
       result = await judge.evaluate(
           input_text,
           output_text,
           sampling_rate=0.25,                  # optional; default 1.0 (always eval)
       )
       if result.sampled:
           tracker.track_judge_result(result)

Four rules:

• `create_judge` returns `Optional[Judge]`. Always guard with if judge and judge.enabled: — it returns None if the judge config is disabled for the context or the provider is missing. A direct .evaluate() on a None return will raise AttributeError.
• Pass `AIJudgeConfigDefault`, not AICompletionConfigDefault. The create_judge default parameter is typed Optional[AIJudgeConfigDefault]; passing the completion type will not type-check and is a doc-level bug in some older examples.
• `sampling_rate` is a parameter on `evaluate()`, not on create_judge. It defaults to 1.0 (evaluate every call). For live paths, pass something lower (0.1–0.25) to control cost.
• `evaluate()` returns a `JudgeResult` (never None). Check result.sampled to know whether the evaluation actually ran, and call track_judge_result(result). Node uses trackJudgeResult(result) and LDJudgeResult with the same sampled field.

Ask the user which judge config key to use. LaunchDarkly ships three built-in judges — Accuracy, Relevance, Toxicity — but the actual config keys for the built-ins are not canonical SDK constants and aren't documented. Have the user open AgentControl > Library in the LD UI and copy the key of the judge they want to reference, or create a custom judge config via configs-create first.

5. Verify.

• UI-attached auto judges: trigger a request in staging, open the Monitoring tab → "Evaluator metrics" dropdown. Scores appear within 1–2 minutes at the configured sampling rate.
• Programmatic direct-judge: hit the wrapped endpoint and confirm track_judge_result lands on the parent config's Monitoring tab.
• Offline eval: run the dataset through the LD Playground, compare baseline vs new-variation scores side by side. No runtime wiring required.

Delegate: `online-evals` (sub-step 3, optional — only for UI-attached judges or custom-judge creation; offline eval doesn't delegate).

Edge Cases

What NOT to Do

These are ordered by how likely they are to show up as a first-run failure. The first three rules — about tracker and config lifetime — account for most of the "migration looks done but the Monitoring tab is fragmented / wrong" reports.

Tracker and config lifetime (most common failure mode)

• Don't call `create_tracker()` / `createTracker()` more than once per user turn. One turn = the full request/response cycle including every ReAct iteration, tool call, and retry. See Stage 4 Step 1 for the canonical placement in each app shape (completion / agent loop / managed runner).
• Don't call `track_duration` / `track_tokens` / `track_success` / `track_error` / `track_time_to_first_token` inside a loop body. These are at-most-once per tracker; second calls are dropped. Accumulate inside the loop, emit once in a terminal/finalize node. Per-event methods (track_tool_call, track_tool_calls, track_feedback, track_judge_result) are safe to call repeatedly. Full matrix: sdk-ai-tracker-patterns.md § At-most-once guards.
• Don't call `agent_config()` / `completion_config()` more than once per user turn. Each call is a flag evaluation and emits a $ld:ai:agent:config event. Re-fetching inside a loop step or a tool body inflates agent-config counts on the Monitoring tab and lets a mid-turn targeting change swap the variation between LLM calls in a single turn. Resolve once at the top, stash on state, and have every subsequent consumer read from state. Tools that need variation-scoped knobs should use the tool-factory pattern (make_search(ai_config) that closes over the knob at setup time) — see agent-mode-frameworks.md § Getting knobs into tools.
• Don't cache the config object across requests — resolve once per turn, yes, but still resolve once per turn. Caching at module scope defeats the targeting-change mechanism entirely.
• Don't delete the fallback once LaunchDarkly is wired up. It is required for the enabled=False and SDK-unreachable paths.
• Don't tuple-unpack the return of completion_config / agent_config / completionConfig / agentConfig. They return a single config object (e.g. AIAgentConfig, AICompletionConfig), not (config, tracker). Obtain the tracker by calling config.create_tracker() / aiConfig.createTracker(). LLMs hallucinate both the tuple shape and a config.tracker property — the actual API is a factory.

LangChain / LangGraph patterns (second most common failure mode)

• If the repo already contains a `load_chat_model(f"{provider}/{name}")` helper, delete it — don't just avoid using it. This exact shape ships with langchain-ai/react-agent and is copied into dozens of derivative repos; look for utils.load_chat_model, utils.build_model, or any one-arg init_chat_model wrapper that splits a "provider/model" string. Re-using it is the first-run failure mode: every variation parameter (temperature, max_tokens, top_p, stop sequences) silently drops on the floor because init_chat_model only receives the name and provider. create_langchain_model(ai_config) is a one-for-one replacement that forwards the whole model.parameters dict. Replace every call site, then delete the wrapper file-side so the next reader can't reach for it.
• Same rule applies to hand-rolled `resolve_tools` / `TOOL_REGISTRY` / `ALL_TOOLS` helpers. If the template already has a resolve_tools(tool_keys) or an ALL_TOOLS module-level list, import build_structured_tools from ldai_langchain.langchain_helper and delete the hand-rolled version. build_structured_tools(ai_config, TOOL_REGISTRY_DICT) reads ai_config.model.parameters.tools and wraps the matching callables as LangChain StructuredTools with the LD tool key as the StructuredTool.name — so ToolNode lookup works without a second mapping. Don't leave both in the repo.
• Don't put app-scoped knobs directly in model.parameters. create_langchain_model forwards every key in parameters to the provider SDK via init_chat_model, so a max_search_results / retry_budget / feature_toggle entry will crash the provider with an unexpected-keyword-argument error. The correct home is model.custom, which the provider helpers ignore and the app reads via ai_config.model.get_custom("key"). The MCP update-ai-config-variation tool does not currently expose top-level custom, so pick one of two paths: (a) PATCH the variation via the REST API to set model.custom directly, or (b) set it via MCP inside parameters.custom (as a nested dict) and use a defensive accessor that reads both locations. Full walk-through with code samples in langchain-tracking.md § MCP caveat.
• Don't re-encode tool schemas inside the fallback. When LaunchDarkly is unreachable the fallback should run without tools (or with whatever minimal provider-bound parameters the app needs to keep operating). Building a _FALLBACK_TOOLS array that duplicates the config's tool schema re-introduces the hardcoded config the migration was supposed to move out of code.
• Don't import LaunchDarklyCallbackHandler from ldai.langchain — neither the class nor the dotted module path exists. The Python LangChain helper package is ldai_langchain (top-level module, underscore). Use create_langchain_model(config) + track_metrics_of_async(get_ai_metrics_from_response, lambda: llm.ainvoke(messages)) as the canonical pattern.

Stage / handoff discipline

• Don't skip Step 1 even when the user says "just wrap it." Without the audit, the fallback will drift from the hardcoded behavior.
• Don't delegate to configs-create before extracting the prompt and model — the delegate needs them as inputs.
• Don't try to attach tools during initial setup-ai-config. Tool attachment is a separate step owned by tools.
• Don't claim you "delegated to configs-create" or any other sibling skill. This skill does not auto-invoke. At each handoff, print the inputs and tell the user to run the sibling slash-command, then wait. Anything else misleads the user about what just happened.
• Don't skip the /configs-targeting step between Stage 2 and Stage 4. A freshly created variation returns enabled=False until targeting promotes it to fallthrough — Stage 2 verification will silently take the fallback path on every request.
• Don't attempt a multi-agent graph migration in one pass. Migrate a single agent first; use agent-graph-reference.md as the next-step read.

Stage 5 evaluations

• Don't wire evals before the tracker is in place. Judges score traffic; without Stage 4 traffic, there is nothing to judge.
• Don't frame Stage 5 as "either UI or programmatic." There are three paths: offline eval (recommended default for migration), UI-attached auto judges (completion-mode only), and programmatic direct-judge. Offline eval is the one most people skip and usually the right starting point.
• Don't pass sampling_rate to create_judge — it's a parameter on Judge.evaluate(), not create_judge().
• Don't hardcode judge config keys ("accuracy-judge", "relevance-judge", etc). The built-in keys are not canonical SDK constants; ask the user to look them up in AgentControl > Library in the LD UI.
• Don't forget the if judge and judge.enabled: guard after create_judge. It returns Optional[Judge] and returns None when the judge config is disabled for the context.

API surface gotchas

• Don't use launchdarkly-metric-instrument for Stage 4 (tracking). That skill is for ldClient.track() feature metrics, not agent tracker.track_* calls — they are different APIs.
• Don't use track_request() in Python — it does not exist in launchdarkly-server-sdk-ai. Use track_metrics_of with a provider-package or custom extractor, or drop to explicit track_duration + track_tokens + track_success / track_error if you're on the streaming path.
• Don't pass graph_key=... to tracker.track_*() methods in Python — it is not an accepted argument. Trackers obtained inside a graph traversal are automatically configured with the correct graph key.

Related Skills

• configs-create — called by Stage 2 to create the config
• tools — called by Stage 3 to create and attach tool definitions
• online-evals — called by Stage 5 to attach judges
• configs-variations — add variations for A/B testing after migration is complete
• configs-targeting — roll out new variations to users after migration is complete
• configs-update — modify config properties as your app evolves
• launchdarkly-metric-instrument — for ldClient.track() feature metrics (NOT for agent tracker calls)

References

• phase-1-analysis-checklist.md — Step 1 audit checklist, grep patterns, SDK routing table, mode decision tree
• before-after-examples.md — Paired hardcoded-to-wrapped snippets for Python OpenAI, Node Anthropic, Python LangGraph
• sdk-ai-tracker-patterns.md — Every tracker.track_* method in Python and Node side by side, auto-helper matrix, and common gotchas
• agent-mode-frameworks.md — How to wire agent_config into LangGraph, CrewAI, and custom react loops; dynamic tool loading pattern
• fallback-defaults-pattern.md — Three fallback patterns (inline, file-backed, bootstrap-generated) and when to use each
• agent-graph-reference.md — Out-of-scope pointer doc for multi-agent migrations

LaunchDarkly Config Migrate Skill

An Agent Skill for migrating an application with hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation in five stages: extract, wrap, tools, tracking, evals.

Overview

This skill orchestrates the full migration journey from hardcoded openai.chat.completions.create(model="gpt-4o", ...) (or equivalent in any provider SDK) to a managed config with tools, tracking, and judges. It delegates each stage to a focused skill and covers the tracker wiring inline — since no existing skill owns tracker.track_* calls.

The five stages:

1. Extract hardcoded model names, prompts, and parameters (read-only) 2. Wrap the call site in completion_config / completionConfig with a safe fallback — delegates the config creation to configs-create 3. Tools — move function-calling schemas into LaunchDarkly — delegates to tools 4. Tracking — wire track_duration, track_tokens, track_success/track_error, optional track_feedback — inline, with a reference doc covering every SDK method in Python and Node side by side 5. Evals — attach judges for LLM-as-a-judge scoring — delegates to online-evals

Installation (Local)

Copy skills/agentcontrol/migrate/ into your agent client's skills path.

Prerequisites

• Remotely hosted LaunchDarkly MCP server
• LD_SDK_KEY environment variable (server-side SDK key, starts with sdk-)
• An application with hardcoded LLM calls (OpenAI, Anthropic, Bedrock, Gemini, LangChain, LangGraph, CrewAI, or Strands)

Usage

Migrate our chat service from hardcoded OpenAI prompts to LaunchDarkly AgentControl

Our LangGraph agent has its model and instructions baked in — walk me through wrapping it in a config

Wire up the agent tracker and attach accuracy + relevance judges to our existing config

Structure

migrate/
├── SKILL.md
├── README.md
└── references/
    ├── phase-1-analysis-checklist.md
    ├── before-after-examples.md
    ├── sdk-ai-tracker-patterns.md
    ├── agent-mode-frameworks.md
    ├── fallback-defaults-pattern.md
    └── agent-graph-reference.md

Related

• config Create: Delegated to by Stage 2 (wrap)
• config Tools: Delegated to by Stage 3 (tools)
• config Online Evals: Delegated to by Stage 5 (evals)
• config Variations: Next step after migration for A/B testing
• config Targeting: Next step after migration for rollout control
• LaunchDarkly AgentControl Docs

License

Apache-2.0

Agent Graph Reference

Out of scope for the main migration workflow. Read this only after a single-agent migration works end-to-end. The main SKILL.md workflow stops at single-agent because multi-agent orchestration is a meaningful jump in complexity and is still evolving in the SDK.

Python is still the richer surface. launchdarkly-server-sdk-ai (Python) has the fully-documented graph API used in the traversal pattern below. @launchdarkly/server-sdk-ai (Node) exposes Agent Graph Definitions and graph metric tracking — consult the js-core source for the current Node API shape before wiring Node graph code; the Python pattern in this doc is canonical.

What an agent graph is

An agent graph is a directed graph where each node is its own config (with its own instructions, model, parameters, and tools) and each edge carries routing metadata for handoffs. A supervisor node routes incoming requests to worker nodes based on the supervisor's output; worker nodes may themselves route to other workers or terminate. The graph lives in LaunchDarkly — both its topology and each node's config are managed as versioned resources and can be changed at runtime without redeploying.

Why use it:

• Add or remove an agent by editing LaunchDarkly — no redeploy
• A/B test routing strategies — target different graph topologies to different users
• Roll out a new worker node to a percentage of traffic
• Guardrail a specific path — attach a judge at a terminal node

SDK surface (Python, from main)

The current Python API (verified against launchdarkly-server-sdk-ai main branch) exposes these methods on LDAIClient:

def agent_graph(self, key: str, context: Context) -> AgentGraphDefinition:
    """Retrieve an agent graph by key."""

async def create_agent_graph(
    self,
    key: str,
    context: Context,
    tools: Optional[ToolRegistry] = None,
    default_ai_provider: Optional[str] = None,
) -> Optional[ManagedAgentGraph]:
    """Experimental — not production-ready. Returns a managed graph that can be invoked directly."""

Use `agent_graph` for read-only traversal (you drive the loop). `create_agent_graph` + `ManagedAgentGraph.run` is experimental and carries explicit production-not-ready warnings in the SDK source. Stick with agent_graph for now.

AgentGraphDefinition

graph_def: AgentGraphDefinition = ai_client.agent_graph("support-flow", context)

graph_def.is_enabled() -> bool
graph_def.root() -> Optional[AgentGraphNode]
graph_def.traverse(fn, execution_context=None)            # callback over nodes from root
graph_def.reverse_traverse(fn, execution_context=None)    # callback over nodes from terminals
graph_def.get_node(key: str) -> Optional[AgentGraphNode]
graph_def.get_child_nodes(node_key: str) -> List[AgentGraphNode]
graph_def.get_parent_nodes(node_key: str) -> List[AgentGraphNode]
graph_def.terminal_nodes() -> List[AgentGraphNode]
graph_def.get_tracker() -> Optional[AIGraphTracker]

AgentGraphNode

node.get_key() -> str
node.get_config() -> AIAgentConfig            # the same shape as agent_config() returns
node.get_edges() -> List[Edge]
node.is_terminal() -> bool

Edge

@dataclass
class Edge:
    key: str
    source_config: str
    target_config: str
    handoff: Optional[dict]                    # arbitrary dict; typically has a 'route' key

AIGraphTracker

tracker.track_invocation_success() -> None
tracker.track_invocation_failure() -> None
tracker.track_duration(duration: int) -> None          # milliseconds, graph-level total
tracker.track_total_tokens(tokens: TokenUsage) -> None
tracker.track_path(path: List[str]) -> None            # e.g. ["supervisor", "security", "support"]
tracker.track_redirect(source_key: str, redirected_target: str) -> None
tracker.track_handoff_success(source_key: str, target_key: str) -> None
tracker.track_handoff_failure(source_key: str, target_key: str) -> None

Things that are NOT on the graph tracker:

• track_node_invocation — not a public method. Use track_path(execution_path) at the end of traversal instead.
• track_tool_call(node_key, tool_name) — graph-level tool-call tracking does not exist. Track per-node tool calls via node_tracker.track_tool_call(tool_name) on each node's tracker (obtained via node.get_config().create_tracker()). Trackers returned via a graph traversal are automatically bound to the right graph key — do not pass graph_key as a keyword.
• track_judge_response — does not exist on AIGraphTracker. Record judge results at the config level via LDAIConfigTracker.track_judge_result(result) instead.
• No track_request(), no track_duration() per call — use track_duration(total_ms) once per traversal.

If you see older devrel-agents-tutorial code that calls track_node_invocation, track_tool_call, or pokes graph_tracker._ld_client.track(...) directly, that code targets an earlier API shape and needs updating. A PR is in flight against launchdarkly-labs/devrel-agents-tutorial to align the tutorial with the current SDK.

Canonical traversal pattern

from ldai.client import LDAIClient
from ldai.tracker import TokenUsage

async def execute_graph(ai_client: LDAIClient, graph_key: str, context, user_input: str):
    graph = ai_client.agent_graph(graph_key, context)

    if not graph.is_enabled():
        raise ValueError(f"Agent graph '{graph_key}' is not enabled")

    # Build a lookup from node key to AgentGraphNode so we can follow edges.
    nodes: dict[str, object] = {}
    graph.reverse_traverse(lambda node, _: nodes.update({node.get_key(): node}), {})

    graph_tracker = graph.get_tracker()
    start = time.time()
    execution_path = []
    shared_ctx = {"user_input": user_input, "final_response": "", "tool_calls": [],
                  "total_input_tokens": 0, "total_output_tokens": 0}

    current_node = graph.root()
    if not current_node:
        raise ValueError("Graph has no root node")

    prev_node_key = None
    visited = set()
    MAX_HOPS = 10
    hop_count = 0

    try:
        while current_node:
            node_key = current_node.get_key()
            if node_key in visited:
                raise ValueError(f"Cycle detected at {node_key}")
            visited.add(node_key)
            hop_count += 1
            if hop_count > MAX_HOPS:
                raise ValueError(f"Max hops exceeded: {hop_count}")

            config = current_node.get_config()
            execution_path.append(node_key)

            # Track handoff into this node
            if graph_tracker and prev_node_key:
                graph_tracker.track_handoff_success(prev_node_key, node_key)

            # Compute valid routes from outgoing edges
            edges = current_node.get_edges()
            valid_routes = [
                (edge.handoff or {}).get("route")
                for edge in edges
                if (edge.handoff or {}).get("route")
            ]

            # Execute this node — uses your existing agent-mode wiring
            result = await run_node(config, shared_ctx, valid_routes=valid_routes)

            # Per-node tool-call tracking lives on the node's config tracker.
            # Create one tracker per node execution (fresh runId) and reuse it
            # for every tracking call inside that node.
            if result.get("tool_calls"):
                node_tracker = config.create_tracker()
                for tool_name in result["tool_calls"]:
                    node_tracker.track_tool_call(tool_name)

            # Merge node result into shared context
            update_shared_ctx(shared_ctx, result)

            # Terminal?
            if not edges or current_node.is_terminal():
                break

            # Pick next node by matching the node's routing_decision to an edge handoff
            next_node = select_next_node(edges, result, nodes, graph_tracker, source_key=node_key)
            prev_node_key = node_key
            current_node = next_node

        # Graph-level metrics
        if graph_tracker:
            graph_tracker.track_path(execution_path)
            graph_tracker.track_duration(int((time.time() - start) * 1000))
            if shared_ctx["total_input_tokens"] or shared_ctx["total_output_tokens"]:
                graph_tracker.track_total_tokens(TokenUsage(
                    input=shared_ctx["total_input_tokens"],
                    output=shared_ctx["total_output_tokens"],
                    total=shared_ctx["total_input_tokens"] + shared_ctx["total_output_tokens"],
                ))
            graph_tracker.track_invocation_success()

    except Exception:
        if graph_tracker:
            graph_tracker.track_invocation_failure()
        raise

    return shared_ctx


def select_next_node(edges, result, nodes, graph_tracker, source_key: str):
    routing = result.get("routing_decision", "").lower().strip() if result.get("routing_decision") else None
    route_map = {
        (edge.handoff or {}).get("route", "").lower().strip(): edge.target_config
        for edge in edges
        if (edge.handoff or {}).get("route")
    }

    if routing and routing in route_map:
        return nodes.get(route_map[routing])
    if routing:
        # Unrecognized route — signal failure with source + attempted target
        if graph_tracker:
            graph_tracker.track_handoff_failure(source_key, routing)
    # Fallback: first edge
    if edges:
        return nodes.get(edges[0].target_config)
    return None

Migrating a multi-agent app to graphs

Do this in phases, not one big bang:

1. Pick one worker node to migrate first. Use the single-agent skill workflow on that worker in isolation — extract, wrap, tools, tracking, evals. Leave the rest of the multi-agent app hardcoded. 2. Confirm the wrapped worker runs in production for the traffic it serves today, with metrics flowing in the Monitoring tab. 3. Migrate the supervisor the same way — single-agent workflow — but keep its routing logic hardcoded initially (a big if/elif over the other workers). 4. Create the agent graph in LaunchDarkly via the UI. Define nodes (one per worker + supervisor) and edges (with handoff.route metadata). 5. Replace the hardcoded router with the traversal pattern above. Call ai_client.agent_graph(...) instead of assembling the pipeline by hand. 6. Verify the Monitoring tab shows the graph-level metrics (track_path, track_duration, track_total_tokens, handoff success/failure counts) in addition to the per-node metrics. 7. Only then start moving routing decisions into LaunchDarkly edges and using targeting to change the graph topology per user segment.

Each phase is reversible. If something breaks at phase 5, the supervisor can fall back to the hardcoded router while the graph issue is fixed.

Limitations to know about

• Python has the canonical surface. The Python traversal pattern above is what this doc covers in full. For Node graphs, consult the @launchdarkly/server-sdk-ai source for the current API.
• `create_agent_graph` is experimental. Do not build production features on ManagedAgentGraph.run. Use the traversal pattern above.
• Graph tracker is less granular than the config tracker. If you want per-node duration or per-node token breakdowns, obtain a per-node tracker via node.get_config().create_tracker() — the graph tracker handles totals only.
• Cycles must be caught in your code. The SDK does not stop cycle traversal automatically; track visited and hop_count yourself.
• Fallback shape. There is no AIAgentGraphDefault. Each node's AIAgentConfig still takes an AIAgentConfigDefault, but the graph itself has no aggregate fallback. If agent_graph fails, handle it at the app level — typically by falling back to the hardcoded pre-migration pipeline.

Resources

• Python SDK source: https://github.com/launchdarkly/python-server-sdk-ai
• packages/sdk/server-ai/src/ldai/agent_graph/__init__.py — AgentGraphDefinition and AgentGraphNode
• packages/sdk/server-ai/src/ldai/tracker.py — AIGraphTracker (near the bottom of the file)
• packages/sdk/server-ai/src/ldai/client.py — LDAIClient.agent_graph and create_agent_graph
• Node SDK source: https://github.com/launchdarkly/js-core/tree/main/packages/sdk/server-ai
• SDK CHANGELOGs (for per-release breaking changes and the version each method landed in):
• Python: https://github.com/launchdarkly/python-server-sdk-ai/blob/main/packages/sdk/server-ai/CHANGELOG.md
• Node: https://github.com/launchdarkly/js-core/blob/main/packages/sdk/server-ai/CHANGELOG.md
• Devrel reference implementation (Python, after PR alignment): https://github.com/launchdarkly-labs/devrel-agents-tutorial on the tutorial/agent-graphs branch

Agent-Mode Frameworks

How to wire a config in agent mode into the frameworks that take a goal/instructions string: LangGraph, CrewAI, Strands, and custom ReAct loops. Also covers the dynamic tool loading pattern from the devrel-agents-tutorial — how to extract tool names from config.tools at runtime and instantiate the actual tool implementations without hardcoding.

When to pick agent mode

Completion mode is the default and covers direct provider calls (OpenAI, Anthropic, Bedrock) where the app assembles a messages array. Pick agent mode when:

Agent mode returns an instructions string. Completion mode returns a messages array. Both modes support tools, parameters, and the same tracker — the only difference is the input shape the SDK returns to you.

Caveat: judges cannot be attached to agent-mode variations via the LaunchDarkly UI. Agent mode evaluations must go through the programmatic judge API (create_judge(...).evaluate(input, output)). See online-evals for the programmatic path.

Model construction for LangChain / LangGraph. When the framework runs on top of LangChain (which includes LangGraph's prebuilt agent and most custom graphs), build the chat model with create_langchain_model(ai_config) (Python) or createLangChainModel(aiConfig) (Node). These helpers forward every variation parameter (temperature, max_tokens, top_p, …) and handle LaunchDarkly→LangChain provider-name mapping internally. Do not hand-roll init_chat_model(model=..., model_provider=...) — it silently drops every variation parameter. See langchain-tracking.md for the canonical single-model and LangGraph patterns, including the SDK helpers sum_token_usage_from_messages / get_tool_calls_from_response (Python, ldai_langchain) used inside the track_metrics_of_async / trackMetricsOf extractor.

Framework-agnostic invariants for the run-scoped pattern

The concrete examples below use specific frameworks (LangGraph, CrewAI, Strands) and specific node names (setup_run, call_model, finalize). Treat those as incidentals. The three invariants below apply to any agent framework — DSPy, AutoGen, Pydantic AI, Haystack, LlamaIndex agents, or a hand-rolled tool loop in pure Python/TypeScript. If the framework has its own idioms, translate these three rules onto them:

1. Resolve `agent_config()` / `agentConfig()` once per user turn. Every call is a flag evaluation and emits a $ld:ai:agent:config event. Re-fetching inside a loop step or a tool body amplifies the event count per turn and lets a mid-turn targeting change swap the variation between LLM calls. Do the resolve at the highest scope that corresponds to "one user-input-to-final-response cycle" — a handler function, a LangGraph entry node, a CrewAI kickoff, whatever the framework exposes. 2. Mint one tracker via `create_tracker()` / `createTracker()` per user turn. Same scope as the agent_config call. The runId ties every event from one turn together; per-step factory calls fragment the correlation and reset the SDK's at-most-once guards. If the framework has a multi-turn session (chat thread), each turn inside the session still gets its own fresh tracker — sessions share a thread_id, not a runId. 3. Emit the five at-most-once methods once at the end of the turn. track_duration / track_tokens / track_success / track_error / track_time_to_first_token each fire at most once per tracker. Accumulate inside the loop body (sum token usage across steps, stash a perf_counter_ns timer up top), emit once after the loop exits or in a dedicated finalize node. track_tool_calls / track_feedback / track_judge_result are per-event — call them as many times as the agent does those things.

What "one user turn" means differs by app shape:

If you can answer "what's the smallest unit at which I'd want to see a single 'execution' in the Monitoring tab?" — that's the turn. Mint exactly one tracker there.

Tool-scoped and app-scoped knobs go in model.custom

If the Stage 1 audit identified configuration that isn't a native model parameter — the kind of thing a provider SDK will reject with unexpected keyword argument if you forward it — these fields belong in ModelConfig(custom={...}), not ModelConfig(parameters={...}). Typical examples:

• max_search_results (tool behavior — how many hits the search tool returns)
• chunk_size / chunk_overlap (RAG preprocessing knobs)
• retry_budget / retry_backoff (app-level retry policy)
• enable_reranking, use_cache, any boolean feature toggle the agent consumes
• any value that governs tool behavior or app behavior rather than model behavior

create_langchain_model (Python) / createLangChainModel (Node) forwards every key in parameters wholesale to the provider SDK. Anthropic, OpenAI, and Gemini all raise on unknown kwargs — a max_search_results entry in parameters crashes the request with AsyncMessages.create() got an unexpected keyword argument 'max_search_results'. Put the same field in custom and the helper leaves it alone; the app reads it where it's needed.

# Fallback: mirror the hardcoded knob shape using custom
FALLBACK = AIAgentConfigDefault(
    enabled=True,
    model=ModelConfig(
        name="claude-sonnet-4-5-20250929",
        parameters={"temperature": 0.3, "max_tokens": 2000},  # provider-bound
        custom={"max_search_results": 10},                    # app-scoped
    ),
    provider=ProviderConfig(name="anthropic"),
    instructions="You are a helpful assistant.",
)

# In the tool or app code that needs the knob:
def search(query: str) -> dict:
    ai_config = get_current_agent_config()
    max_results = ai_config.model.get_custom("max_search_results") or 10
    return TavilySearch(max_results=max_results).invoke({"query": query})

Mirror the same shape on the LaunchDarkly variation. MCP caveat. The update-ai-config-variation MCP tool does not currently expose the custom field — to populate model.custom on an existing variation, PATCH it through the REST API directly:

curl -X PATCH \
  "https://app.launchdarkly.com/api/v2/projects/$PROJECT/ai-configs/$CONFIG_KEY/variations/$VARIATION_ID" \
  -H "Authorization: $LD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"patch":[{"op":"add","path":"/model/custom","value":{"max_search_results":10}}]}'

Getting knobs into tools: tool factories that close over per-run config (default)

Tools need read access to model.custom at call time. The correct pattern is a tool factory that takes the per-run ai_config resolved in setup_run (or at the top of a custom ReAct loop) and returns a tool callable that closes over whatever knobs it needs:

# tools.py
def make_search(ai_config) -> Callable[..., Any]:
    max_results = ai_config.model.get_custom("max_search_results") or 10

    async def search(query: str) -> dict:
        """Search the web for current information on a given topic."""
        return await TavilySearch(max_results=max_results).ainvoke({"query": query})

    return search

TOOL_FACTORIES = {"search": make_search}

# graph.py setup_run
built = {name: fn(ai_config) for name, fn in TOOL_FACTORIES.items()}
tools = build_structured_tools(ai_config, built)

This is the default because it has three properties nothing else preserves all at once:

1. Turn-level atomicity. A mid-turn flag change doesn't swap max_search_results between the first tool call and the second — the factory captured it once, at setup_run. 2. No extra LD evaluations. agent_config() is a flag evaluation and an emitted event. Calling it from a tool once per user turn is fine; calling it per tool invocation on a chatty agent inflates $ld:ai:agent:config counts by a factor of however many tool calls run per turn. 3. Tools don't take a dependency on LD. The tool function is a plain callable. Testing is substitution, not monkeypatching get_agent_config.

Do not call `get_agent_config()` (or `ai_client.agent_config(...)`) from inside a tool body. The alternatives — re-resolving from inside the tool, or reaching into runtime.context from inside the tool — either break turn atomicity or require plumbing LangGraph's context through every tool signature. Tool factories sidestep both problems and are also how ldai_langchain.langchain_helper.build_structured_tools expects its registry to be shaped (a dict of {name: Callable} ready to bind).

The only legitimate reason to re-fetch inside a tool is if the tool's behavior needs to follow a flag change within a single turn. That's vanishingly rare; default to factories and treat re-fetch as an exception that requires a concrete reason.

Wiring agent_config into each framework

LangGraph prebuilt agent (Python — langchain.agents.create_agent)

API note. Use from langchain.agents import create_agent. The earlier from langgraph.prebuilt import create_react_agent is deprecated in LangGraph 1.0 and removed in 2.0. Same return shape; the only call-site rename is prompt= → system_prompt=. Node.js still uses createReactAgent from @langchain/langgraph/prebuilt.

from langchain.agents import create_agent
from ldai_langchain import create_langchain_model
from ldai.client import LDAIClient, AIAgentConfigDefault, ModelConfig, ProviderConfig

FALLBACK = AIAgentConfigDefault(
    enabled=True,
    model=ModelConfig(name="gpt-4o", parameters={"temperature": 0.3}),
    provider=ProviderConfig(name="openai"),
    instructions="You are a helpful assistant.",
)

def build_agent(ai_client: LDAIClient, user_id: str, tools: list):
    context = Context.builder(user_id).kind("user").build()
    config = ai_client.agent_config("support-agent", context, FALLBACK)

    if not config.enabled:
        return None, None

    # create_langchain_model forwards every variation parameter — do NOT hand-roll
    # ChatOpenAI(model=..., temperature=...). It silently drops unnamed parameters.
    llm = create_langchain_model(config)

    agent = create_agent(
        llm,
        tools,
        system_prompt=config.instructions,
    )
    return agent, config.create_tracker()

Key points:

• system_prompt=config.instructions — the instructions string replaces the hardcoded prompt
• Model + parameters come from create_langchain_model(config) — forwards the whole model.parameters dict
• A fresh tracker is minted via config.create_tracker() and returned alongside the agent so the caller can wire Stage 4 tracking around agent.invoke(...). Each call to create_tracker() produces a new runId; the caller should treat the returned tracker as owning the execution.

CrewAI Agent

from crewai import Agent
from ldai.client import LDAIClient, AIAgentConfigDefault

def build_crew_agent(ai_client, user_id: str):
    context = Context.builder(user_id).kind("user").build()
    config = ai_client.agent_config("researcher-agent", context, FALLBACK)

    if not config.enabled:
        return None

    # CrewAI expects role/goal/backstory — split the instructions or store them
    # in the Config as three variables and pipe them in at runtime.
    return Agent(
        role="Research Analyst",
        goal="Produce a summary of recent config adoption patterns.",
        backstory=config.instructions,
        llm=config.model.name,  # CrewAI accepts a string or a LangChain model
    )

Pattern note: CrewAI's Agent takes three separate fields. If you want to drive all three from LaunchDarkly, either:

• Use prompt variables on the config ({{role}}, {{goal}}, {{backstory}}) and pass them as the variables argument to agent_config(...)
• Or store a structured JSON blob in instructions and parse it in the app

Prompt variables are cleaner and keep the config human-readable in the UI.

Strands Agent

Strands is a provider-agnostic, async-first agent SDK. The same Agent class runs against Anthropic, OpenAI, and Bedrock by swapping the model argument; tools are plain @tool-decorated Python functions passed through the constructor; and SlidingWindowConversationManager keeps short-term memory across invoke_async turns without external state. Agent-mode instructions maps directly to Agent(system_prompt=...).

Strands does not ship a first-party LaunchDarkly provider package. To serve multiple providers from a single config key, dispatch on agent_config.provider.name and construct the matching Strands model class.

Provider dispatcher. Drop parameters.tools before passing params into the Strands model class — LaunchDarkly surfaces attached tools via a flat parameters.tools shape in the variation payload, but Strands receives tools via the Agent constructor. Passing tools through a second time via model params is an error.

from strands.models.anthropic import AnthropicModel
from strands.models.openai import OpenAIModel


def create_strands_model(agent_config):
    """Map an LDAIAgentConfig to the matching Strands model class by provider."""
    provider = (agent_config.provider.name if agent_config.provider else "").lower()
    model_id = agent_config.model.name
    params = dict(agent_config.model.to_dict().get("parameters") or {})
    # LD surfaces attached tools via parameters.tools; Strands takes tools via
    # Agent(tools=[...]). Drop the key before passing params to the model class.
    params.pop("tools", None)

    if provider == "anthropic":
        # AnthropicModel requires max_tokens as a kwarg, not inside params.
        max_tokens = int(
            params.pop("max_tokens", None) or params.pop("maxTokens", None) or 1024
        )
        return AnthropicModel(model_id=model_id, max_tokens=max_tokens, params=params or None)
    if provider == "openai":
        # Pass parameters through as-is — gpt-5 wants max_completion_tokens,
        # gpt-4o wants max_tokens. Keep that choice in the LD variation.
        return OpenAIModel(model_id=model_id, params=params)
    raise ValueError(f"Unsupported provider for Strands: {provider!r}")

Call site. Build the agent once per request, pull the tracker off the config, and wrap invoke_async with track_duration_of — Strands is Tier 3 (custom extractor) because there is no provider package.

from strands import Agent, tool
from strands.agent.conversation_manager.sliding_window_conversation_manager import (
    SlidingWindowConversationManager,
)
from ldai.client import AIAgentConfigDefault, ModelConfig, ProviderConfig
from ldai.tracker import TokenUsage
from ldclient import Context


@tool
def get_order_status(order_id: str) -> str:
    """Look up the status of a customer order by order ID."""
    ...


FALLBACK = AIAgentConfigDefault(
    enabled=True,
    model=ModelConfig(name="gpt-5", parameters={"max_completion_tokens": 2000}),
    provider=ProviderConfig(name="openai"),
    instructions="You are a helpful assistant.",
)


def track_strands_metrics(tracker, result):
    """Record token usage from a Strands AgentResult on the LD tracker.

    accumulated_usage aggregates tokens across every provider call in the turn,
    including tool-calling round trips — unlike the single-response shape from
    Anthropic or OpenAI direct.
    """
    usage = getattr(result.metrics, "accumulated_usage", {}) or {}
    input_tokens = usage.get("inputTokens", 0)
    output_tokens = usage.get("outputTokens", 0)
    total = usage.get("totalTokens", 0) or (input_tokens + output_tokens)
    if total > 0:
        tracker.track_tokens(TokenUsage(input=input_tokens, output=output_tokens, total=total))


async def run_turn(ai_client, user_id: str, user_input: str):
    context = Context.builder(user_id).kind("user").build()
    agent_config = ai_client.agent_config("strands-agent", context, FALLBACK)

    if not agent_config.enabled:
        return disabled_response()

    agent = Agent(
        name="order-assistant",
        model=create_strands_model(agent_config),
        system_prompt=agent_config.instructions,
        tools=[get_order_status],
        conversation_manager=SlidingWindowConversationManager(window_size=40),
    )
    tracker = agent_config.create_tracker()

    try:
        result = await tracker.track_duration_of(lambda: agent.invoke_async(user_input))
        tracker.track_success()
        track_strands_metrics(tracker, result)
        return result.message["content"][0]["text"]
    except Exception:
        tracker.track_error()
        raise

Key points:

• system_prompt=agent_config.instructions — the instructions string replaces the hardcoded system prompt.
• create_strands_model(agent_config) is the provider-dispatch seam. Add a branch per provider the variation can serve.
• The tracker is Tier 3: tracker.track_duration_of(...) + an explicit track_tokens call fed by track_strands_metrics. See strands-tracking.md for the single-call track_metrics_of_async variant and the per-field breakdown of accumulated_usage.
• Always ldclient.get().flush() before process exit in short-lived scripts — trailing events can otherwise be lost.

TypeScript caveat. The Strands TypeScript SDK ships BedrockModel and OpenAIModel only — it cannot run Anthropic-backed variations. If the app needs to serve both OpenAI and Anthropic from a single config, use the Python SDK.

Custom StateGraph (bind_tools + ToolNode)

The most common LangGraph pattern in the wild is not the prebuilt agent — it's a custom StateGraph with a call_model node that does model.bind_tools(TOOLS), a separate "tools" node that runs ToolNode(TOOLS), and a conditional edge between them. This is the shape of the langchain-ai/react-agent template.

Two things make it different from the prebuilt create_agent:

1. Tools appear in two places — bind_tools(TOOLS) (so the LLM knows which tools exist) and ToolNode(TOOLS) (so the executor knows how to run them). Both must read from the same source. 2. The system prompt is injected manually in the call_model node body (usually as the first message in the ainvoke([{"role": "system", ...}, *state.messages]) call), not passed as a constructor argument.

Before — the typical template shape (src/react_agent/graph.py, src/react_agent/tools.py, src/react_agent/context.py):

# tools.py
TOOLS: List[Callable[..., Any]] = [search]

# context.py
@dataclass(kw_only=True)
class Context:
    system_prompt: str = field(default=prompts.SYSTEM_PROMPT)
    model: str = field(default="anthropic/claude-sonnet-4-5-20250929")

# graph.py
from .tools import TOOLS

async def call_model(state: State, runtime: Runtime[Context]):
    model = load_chat_model(runtime.context.model).bind_tools(TOOLS)
    system_message = runtime.context.system_prompt.format(system_time=now_iso())
    response = await model.ainvoke(
        [{"role": "system", "content": system_message}, *state.messages]
    )
    return {"messages": [response]}

builder = StateGraph(State, context_schema=Context)
builder.add_node(call_model)
builder.add_node("tools", ToolNode(TOOLS))
builder.add_edge("__start__", "call_model")
builder.add_conditional_edges("call_model", route_model_output)
builder.add_edge("tools", "call_model")
graph = builder.compile()

After — run-scoped architecture. The critical shape for the tracker factory is that one user turn = one `runId` = one tracker, not one LLM call = one tracker. A ReAct loop that calls call_model three times in a single turn must not mint three trackers, or billing and the Monitoring tab will treat the turn as three separate executions. The fix is to resolve the config and mint the tracker once, in a dedicated entry node, and thread both through graph state for every subsequent node.

Three nodes, in order:

1. `setup_run` (entry) — resolves agent_config, mints the tracker with create_tracker(), builds model with create_langchain_model(ai_config), builds tools via the factory pattern below, starts a perf_counter_ns() timer, and stashes all of it on State. Runs exactly once per turn. 2. `call_model` — reads model / tools / tracker / accumulator from State, runs model.ainvoke(...), accumulates token usage, calls tracker.track_tool_calls([...]) per step. Does not call track_metrics_of_async here — that wrapper records duration + success on every invocation and would fire once per iteration. On exception: call tracker.track_duration + tracker.track_error and re-raise (the finalize node will not run). 3. `finalize` (terminal) — runs exactly once at the end of the turn on the success path. Calls tracker.track_duration(elapsed_ms) + tracker.track_tokens(accumulated) + tracker.track_success(). Each of these fires exactly once per run, which is what the at-most-once guards enforce.

# tools.py — tool factories close over per-run config, so tools never re-fetch
from typing import Any, Callable, Dict
from langchain_core.tools import StructuredTool
from langchain_tavily import TavilySearch

def make_search(ai_config) -> Callable[..., Any]:
    """Closure: capture `max_search_results` once at setup_run time."""
    max_results = ai_config.model.get_custom("max_search_results") or 10

    async def search(query: str) -> dict:
        """Search the web for current information on a given topic."""
        return await TavilySearch(max_results=max_results).ainvoke({"query": query})

    return search

# Registry of factories keyed by LD tool name. Each factory takes the
# per-run ai_config and returns a ready-to-bind callable. This decouples
# the Config (tool metadata) from the app (implementations) and
# means tools never call get_agent_config() themselves.
TOOL_FACTORIES: Dict[str, Callable[[Any], Callable[..., Any]]] = {
    "search": make_search,
}


# graph.py
import time
from typing import Any, Dict, List, TypedDict
from uuid import uuid4
from ldai_langchain import create_langchain_model, get_ai_metrics_from_response
from ldai_langchain.langchain_helper import build_structured_tools
from ldai.tracker import TokenUsage
from ldclient import Context as LDContext
from langchain_core.runnables import RunnableConfig
from .tools import TOOL_FACTORIES


# Run-scoped State. Fields below messages[] are non-serializable and only
# meaningful inside one turn — if you add a LangGraph checkpointer, exclude
# them from the checkpoint.
class State(TypedDict, total=False):
    messages: List[Any]                 # standard LangGraph messages reducer applies
    tracker: Any                        # LDAIConfigTracker, minted in setup_run
    model: Any                          # Bound chat model with per-run tools
    tools: List[Any]                    # StructuredTool list for this turn
    start_perf_ns: int                  # time.perf_counter_ns() at setup_run
    token_accumulator: TokenUsage       # Sum of usage_metadata across loop iterations
    disabled_message: str               # Set iff ai_config.enabled is False

# If the repo's State is a @dataclass (e.g., langchain-ai/react-agent),
# attribute access works the same — `state.tools` instead of `state["tools"]`.
# Same field set, same reducers (wrap the messages field with
# `Annotated[List[Any], add_messages]`). Don't convert during migration.


async def setup_run(state: State, config: RunnableConfig) -> Dict[str, Any]:
    """Runs once per user turn. Resolves the config, mints the tracker,
    builds the model and tools, and stashes everything on state.

    Node signature takes `config: RunnableConfig` rather than
    `runtime: Runtime[Context]` so the example works with an empty Context
    dataclass and reads `thread_id` from LangGraph's standard
    `config["configurable"]["thread_id"]` plumbing. If your app has a typed
    Context schema with fields beyond thread_id, use the Runtime[Context]
    signature instead and keep those fields on Context.
    """
    configurable = config.get("configurable") or {}
    ld_key = configurable.get("thread_id") or f"anon-{uuid4()}"
    ld_context = LDContext.builder(ld_key).kind("user").build()
    ai_config = get_ai_client().agent_config(
        "react-agent",
        ld_context,
        FALLBACK,
    )

    if not ai_config.enabled:
        # Return BOTH a flag the router can short-circuit on AND an AIMessage
        # appended to state["messages"]. Downstream consumers (UI, tests, the
        # caller that invoked the graph) read the last message — setting only
        # `disabled_message` without touching `messages` produces a graph
        # whose last-message shape depends on state entry, which surprises
        # readers. Every node return should leave `messages` in a valid shape.
        return {
            "disabled_message": "Feature is currently unavailable.",
            "messages": [AIMessage(content="Feature is currently unavailable.")],
        }

    # Three-tier tool-registry contract. Don't conflate these:
    #   1. TOOL_FACTORIES          — {name: factory}   at module scope (static, never changes)
    #   2. built_callables         — {name: callable}  per-run (factories applied to ai_config)
    #   3. tools (StructuredTool)  — per-run list, ready for bind_tools + ToolNode dispatch
    # build_structured_tools reads ai_config.model.parameters.tools to decide
    # which entries from built_callables to wrap — the LLM only sees the subset
    # the variation attached, even if the registry has more callables.
    built_callables = {name: fn(ai_config) for name, fn in TOOL_FACTORIES.items()}
    tools = build_structured_tools(ai_config, built_callables)

    return {
        "tracker": ai_config.create_tracker(),
        "model": create_langchain_model(ai_config).bind_tools(tools),
        "tools": tools,
        "start_perf_ns": time.perf_counter_ns(),
        "token_accumulator": TokenUsage(input=0, output=0, total=0),
        # Cache instructions so call_model doesn't touch ai_config again.
        "instructions": ai_config.instructions or "",
    }


async def call_model(state: State) -> Dict[str, Any]:
    """Reads model / tracker / tools from state. No LD access, no config fetch."""
    tracker = state["tracker"]
    model = state["model"]
    messages = [{"role": "system", "content": state["instructions"]}, *state["messages"]]

    try:
        response = cast(AIMessage, await model.ainvoke(messages))
    except Exception:
        elapsed_ms = (time.perf_counter_ns() - state["start_perf_ns"]) // 1_000_000
        tracker.track_duration(elapsed_ms)
        tracker.track_error()
        raise

    # Accumulate token usage across loop iterations; finalize emits the sum.
    acc = state["token_accumulator"]
    if getattr(response, "usage_metadata", None):
        um = response.usage_metadata
        state["token_accumulator"] = TokenUsage(
            input=acc.input + um.get("input_tokens", 0),
            output=acc.output + um.get("output_tokens", 0),
            total=acc.total + um.get("total_tokens", 0),
        )

    if response.tool_calls:
        tracker.track_tool_calls([call["name"] for call in response.tool_calls])

    return {"messages": [response]}


async def finalize(state: State) -> Dict[str, Any]:
    """Runs exactly once at the end of a successful turn. Emits the three
    once-per-run tracker events: duration, tokens, success."""
    tracker = state["tracker"]
    elapsed_ms = (time.perf_counter_ns() - state["start_perf_ns"]) // 1_000_000
    tracker.track_duration(elapsed_ms)
    acc = state["token_accumulator"]
    if acc.total > 0:
        tracker.track_tokens(acc)
    tracker.track_success()
    return {}


def route_after_setup(state: State) -> Literal["call_model", "__end__"]:
    return "__end__" if "disabled_message" in state else "call_model"


def route_model_output(state: State) -> Literal["tools", "finalize"]:
    last = state["messages"][-1]
    return "tools" if getattr(last, "tool_calls", None) else "finalize"


async def tools_node(state: State) -> Dict[str, Any]:
    """Dynamic ToolNode wrapper — rebuilds dispatch per invocation.

    `ToolNode` builds its `{name: callable}` dispatch dict at construction
    time, so `ToolNode([])` cannot execute anything (dispatch returns
    "Error: <tool> is not a valid tool, try one of []." verbatim). Because
    our tool callables close over per-run `ai_config` (see TOOL_FACTORIES),
    we cannot pre-build the ToolNode at compile time — the concrete
    callables differ each turn. Construct a fresh ToolNode from
    state["tools"] per invocation.

    Two things to get right in the body:
    - Use `ainvoke`, not `invoke` — the enclosing nodes in this example
      are async and LangGraph will raise if a sync node is awaited.
    - Pass an explicit `{"messages": [...]}` payload rather than the full
      State dict — ToolNode's contract is that input shape, and passing
      extra fields has surprised users on some LangGraph versions.
    """
    return await ToolNode(list(state["tools"])).ainvoke(
        {"messages": list(state["messages"])}
    )


# No context_schema=Context here — Context can be empty or dropped entirely.
# thread_id and any other per-invocation plumbing flows through RunnableConfig
# via config["configurable"] (see setup_run). Add context_schema=Context only
# if your app truly has typed per-request context fields beyond thread_id.
builder = StateGraph(State)
builder.add_node(setup_run)
builder.add_node(call_model)
builder.add_node("tools", tools_node)  # NOT ToolNode([]) — see tools_node docstring
builder.add_node(finalize)
builder.add_edge("__start__", "setup_run")
builder.add_conditional_edges("setup_run", route_after_setup)
builder.add_conditional_edges("call_model", route_model_output)
builder.add_edge("tools", "call_model")
builder.add_edge("finalize", "__end__")
graph = builder.compile()

Why this shape:

• One `create_tracker()` per turn — setup_run is the only caller. Each factory call mints a fresh runId; per-iteration calls fragment the run downstream and reset the at-most-once guards.
• One `agent_config(...)` per turn — setup_run resolves once. Re-fetching in a loop step inflates agent-config event counts and lets a mid-turn targeting change swap variations between LLM calls.
• `track_duration` / `track_tokens` / `track_success` fire in `finalize`, not `call_model` — these are at-most-once; per-iteration calls are silently dropped.
• `track_tool_calls` in `call_model` is fine — it's per-event metadata, not at-most-once.
• Tools close over per-run config — make_search(ai_config) captures max_search_results at setup time. A mid-turn variation change doesn't affect the in-flight turn; the next turn picks up the new value.
• `tools` node uses the `tools_node` wrapper, not `ToolNode([])` — ToolNode builds its dispatch dict at construction, so an empty list produces empty dispatch and every tool call fails with is not a valid tool. The wrapper rebuilds per invocation from state.
• Fallback instructions use Mustache ("...{{ system_time }}"), not .format() — the SDK's Mustache renderer runs on both the LD-served path and the fallback path; single-brace placeholders ship a stale literal.

Gotchas:

• Never call `track_metrics_of_async` in a loop node. trackMetricsOf is designed for single-call shapes (one provider call, one success event). In a ReAct loop it would re-fire track_success per iteration and trip the at-most-once guard. Use manual track_tool_calls per step and explicit track_duration + track_tokens + track_success in finalize.
• Lazy-init the `ai_client`. Avoid ai_client = LDAIClient(...) at module import time — it couples test collection to LD initialization and makes the per-turn runId story harder to mock. Wrap with def get_ai_client(): ... and cache on first use.
• Per-run `LDContext` keys — and their MAU cost. A shared literal like "anonymous" collapses every request into one targeting/billing segment, which breaks experimentation and per-run segmentation. The obvious fix is to use the LangGraph thread_id if present and fall back to uuid4() per setup_run. Before doing this in production, check the MAU impact: LaunchDarkly bills by distinct context keys per month, so a production agent serving 100k anonymous runs/day will register 3M+ MAU even though there's no real user. Three shapes, pick the one that fits:

1. Known user identity (preferred). If the caller knows who the user is (session cookie, auth token, SSO ID), use that as the context key. No anonymous keys at all. Targeting, segmentation, and MAU all work correctly. 2. Session-scoped key. Use the LangGraph thread_id, or the chat-session ID, or whatever the longest-lived identifier is below "real user." MAU scales with session count, not turn count. 3. Per-turn `uuid4()`. Only in demos, or if you genuinely need per-run isolation for experiment targeting and are willing to pay the MAU. Document the decision in the repo README so the next migrator doesn't swap it out without reading the tradeoff.

• State field serialization. If you add a LangGraph checkpointer, exclude the run-scoped fields (tracker, model, tools, start_perf_ns) from the checkpoint — they're non-serializable and only meaningful inside one turn anyway.

Node.js — LangGraph.js createReactAgent (prebuilt)

For apps built on @langchain/langgraph's prebuilt createReactAgent, the loop happens inside one agent.invoke(...) call — you don't own the nodes. That makes the run-scoped pattern simpler than the Python custom-StateGraph shape above: resolve agentConfig once, mint the tracker once, wrap the whole agent.invoke in one tracker.trackMetricsOf(...) call per user turn.

import { init } from '@launchdarkly/node-server-sdk';
import { initAi, type LDAIAgentConfig, type LDAIMetrics } from '@launchdarkly/server-sdk-ai';
import { createLangChainModel } from '@launchdarkly/server-sdk-ai-langchain';
import { createReactAgent } from '@langchain/langgraph/prebuilt';
import { MemorySaver } from '@langchain/langgraph';

const ldClient = init(process.env.LD_SDK_KEY!);
await ldClient.waitForInitialization({ timeout: 10 });
const aiClient = initAi(ldClient);

// Sum token usage across every message the agent produced in one turn.
// Multiple fallback field names cover the provider variation LangChain
// normalizes over (OpenAI, Anthropic, Bedrock, Gemini, …).
function langgraphMetrics(result: any): LDAIMetrics {
  let input = 0, output = 0, total = 0;
  for (const msg of result.messages ?? []) {
    const usage = msg.response_metadata?.token_usage ?? msg.usage_metadata;
    if (!usage) continue;
    input += usage.input_tokens ?? usage.prompt_tokens ?? usage.promptTokens ?? 0;
    output += usage.output_tokens ?? usage.completion_tokens ?? usage.completionTokens ?? 0;
    total += usage.total_tokens ?? usage.totalTokens ?? 0;
  }
  if (total === 0) total = input + output;
  return { success: true, tokens: total > 0 ? { input, output, total } : undefined };
}

async function runTurn(userInput: string, threadId: string): Promise<string | null> {
  const context = { kind: 'user' as const, key: threadId };
  const agentConfig: LDAIAgentConfig = await aiClient.agentConfig(
    'react-agent',
    context,
    FALLBACK,                            // LDAIAgentConfigDefault literal
  );

  if (!agentConfig.enabled) return null;

  // Build everything once per user turn.
  const llm = await createLangChainModel(agentConfig);
  const agent = createReactAgent({
    llm,
    tools: buildTools(agentConfig),      // factory pattern — see below
    prompt: agentConfig.instructions,
    checkpointer: new MemorySaver(),     // reuse across turns if you want chat memory
  });

  // One tracker per user turn. Fresh runId. At-most-once guards reset.
  const tracker = agentConfig.createTracker();

  // Exceptions are tracked automatically — trackMetricsOf catches
  // exceptions, records tracker.trackError(), and re-throws.
  const result = await tracker.trackMetricsOf(
    langgraphMetrics,
    () => agent.invoke(
      { messages: [{ role: 'user', content: userInput }] },
      { configurable: { thread_id: threadId } },
    ),
  );
  const messages = result.messages ?? [];
  return messages.length ? String(messages[messages.length - 1].content) : null;
}

Tool factories in TypeScript. The same pattern as Python — a record of (agentConfig) => tool factories, applied per-turn so each tool closes over the live variation's model.custom knobs:

import { tool } from '@langchain/core/tools';
import { z } from 'zod';

type ToolFactory = (agentConfig: LDAIAgentConfig) => ReturnType<typeof tool>;

function makeSearch(agentConfig: LDAIAgentConfig) {
  const maxResults =
    (agentConfig.model?.custom?.max_search_results as number | undefined) ?? 10;
  return tool(
    async ({ query }: { query: string }) => {
      const res = await fetch(`https://api.tavily.com/search?q=${encodeURIComponent(query)}&n=${maxResults}`);
      return await res.json();
    },
    {
      name: 'search',
      description: 'Search the web for current information on a given topic.',
      schema: z.object({ query: z.string() }),
    },
  );
}

const TOOL_FACTORIES: Record<string, ToolFactory> = {
  search: makeSearch,
};

function buildTools(agentConfig: LDAIAgentConfig) {
  const attached = ((agentConfig.model?.parameters?.tools as Array<{ name: string }>) ?? [])
    .map((t) => t.name);
  return attached
    .filter((name) => name in TOOL_FACTORIES)
    .map((name) => TOOL_FACTORIES[name](agentConfig));
}

Key differences from Python custom-`StateGraph`:

• Because LangGraph.js's createReactAgent is prebuilt, you don't write a setup_run / call_model / finalize graph — agent.invoke(...) is the single call that represents the turn, so wrapping it in one trackMetricsOf call is sufficient. The at-most-once guards are naturally satisfied by the single wrapping call.
• No dynamic ToolNode wrapper needed. createReactAgent takes a tool list at construction; construct the agent per-turn (inside runTurn) so the tools can close over the current agentConfig.
• Multi-turn chat: if you want runTurn to be called three times in a row for one session (shared threadId / MemorySaver), each call still mints its own tracker inside runTurn. The threadId threads conversation memory through checkpointer; the runId identifies each turn independently.

Custom `StateGraph` in Node.js. If the app uses a hand-rolled StateGraph with its own call_model / tool nodes (uncommon in Node compared to Python, but possible), the same run-scoped architecture from the Python section above applies — setup_run as an entry node, tools_node wrapper around new ToolNode(state.tools).invoke(...), finalize as a terminal node. The TypeScript syntax differs but the node responsibilities are identical. Prefer createReactAgent unless the app genuinely needs graph-level control.

Custom ReAct loop

The same run-scoped shape applies: one config fetch + one create_tracker() at the top of the turn, accumulate tokens across the loop, emit track_duration / track_tokens / track_success exactly once after the loop exits.

import time
from ldai.tracker import TokenUsage

def run_turn(ai_client, user_id: str, user_input: str):
    context = Context.builder(user_id).kind("user").build()
    config = ai_client.agent_config("custom-react", context, FALLBACK)

    if not config.enabled:
        return disabled_response()

    # Resolve everything needed for the turn once, up top.
    system_prompt = config.instructions
    model_name = config.model.name
    tracker = config.create_tracker()
    tool_callables = build_tools_from_config(config)  # closes over config
    start_ns = time.perf_counter_ns()
    acc = TokenUsage(input=0, output=0, total=0)

    try:
        history = [{"role": "user", "content": user_input}]
        for step in range(MAX_TURNS):
            response = my_provider.complete(
                model=model_name,
                system=system_prompt,
                messages=history,
                tools=config.model.get_parameter("tools") or [],
            )

            # Accumulate token usage across the loop; finalize emits the sum.
            usage = extract_tokens(response)  # returns TokenUsage
            acc = TokenUsage(
                input=acc.input + usage.input,
                output=acc.output + usage.output,
                total=acc.total + usage.total,
            )

            tool_calls = extract_tool_calls(response)
            if tool_calls:
                tracker.track_tool_calls([c["name"] for c in tool_calls])
                history.extend(run_tools(tool_calls, tool_callables))
                continue

            # Done — fall through to the success emit block.
            break

        elapsed_ms = (time.perf_counter_ns() - start_ns) // 1_000_000
        tracker.track_duration(elapsed_ms)
        if acc.total > 0:
            tracker.track_tokens(acc)
        tracker.track_success()
        return response
    except Exception:
        elapsed_ms = (time.perf_counter_ns() - start_ns) // 1_000_000
        tracker.track_duration(elapsed_ms)
        tracker.track_error()
        raise

The call site stays in your control; the config just delivers instructions, model.name, model.parameters, and tools. Everything that's stable across the turn (model name, instructions, tool bindings, tracker) is hoisted out of the loop body — the loop itself only does message passing and tool dispatch.

Do not call track_duration / track_tokens / track_success inside the for body. The at-most-once guards will warn and drop the second-and-later calls on the same tracker, so per-step tracker calls will silently lose data. Accumulate inside the loop, emit once after.

Dynamic tool loading — the "tools factory" pattern

The devrel-agents-tutorial uses a dynamic tool factory that reads tool names from config.tools and instantiates the actual tool implementations at runtime. This decouples the config (which holds tool metadata) from the application (which holds the executable code).

The pattern

# tools_impl/dynamic_tool_factory.py — adapted from devrel-agents-tutorial

def extract_tool_names(config) -> list[str]:
    """Read the list of tool names from the config."""
    if not hasattr(config, "tools") or not config.tools:
        return []
    return [tool.name if hasattr(tool, "name") else tool.get("name") for tool in config.tools]


def create_dynamic_tools_from_launchdarkly(config) -> list:
    """Instantiate tool implementations for every tool name on the config."""
    tool_names = extract_tool_names(config)
    instances = []
    for name in tool_names:
        tool = _create_tool_instance(name)
        if tool is not None:
            instances.append(tool)
    return instances


def _create_tool_instance(tool_name: str):
    """Map a tool name to an actual implementation. Add one branch per tool."""
    if tool_name == "search_kb":
        from my_tools.search import SearchKBTool
        return SearchKBTool()
    elif tool_name == "calculator":
        from my_tools.calc import CalculatorTool
        return CalculatorTool()
    # ... etc
    return None

Then at the call site:

config = ai_client.agent_config("support-agent", context, FALLBACK)
tools = create_dynamic_tools_from_launchdarkly(config)

agent = create_agent(
    build_llm(config),
    tools,
    system_prompt=config.instructions,
)

What this gives you:

• Toggle a tool on/off by editing the config in LaunchDarkly — no redeploy needed to remove a tool from production
• Roll out a new tool to 5% of users by editing targeting rules (combined with configs-targeting)
• Keep the actual tool implementation code in the repo; only metadata lives in LaunchDarkly

Extracting schemas from existing hardcoded tools

If your tools are defined with LangChain @tool or Pydantic BaseModel schemas, extract the JSON schema programmatically to pass to tools during Stage 3:

from my_tools import search_kb  # a @tool-decorated function

schema = search_kb.args_schema.model_json_schema()
# schema is a dict ready to pass as the tool's parameters field

Do not hand-write the schema — LangChain already generated it from the function signature, and Pydantic will keep it in sync. The tools delegate accepts raw JSON Schema for the parameters field.

Dynamic schemas from LaunchDarkly

The devrel tutorial also shows the reverse: reading a JSON schema from the config and constructing a Pydantic model at runtime:

from pydantic import BaseModel, Field, create_model

def _create_dynamic_tool_input(tool_config: dict) -> type[BaseModel]:
    """Build a Pydantic input schema from a config tool's parameters."""
    properties = tool_config.get("properties", {})
    fields = {}
    for name, cfg in properties.items():
        py_type = {"string": str, "number": float, "integer": int, "boolean": bool}.get(
            cfg.get("type"), str
        )
        default = ... if name in tool_config.get("required", []) else None
        fields[name] = (py_type, Field(default=default, description=cfg.get("description", "")))
    return create_model("DynamicToolInput", **fields)

This lets LaunchDarkly change a tool's parameter schema without redeploying the app. Use it when the tool implementation is generic enough to accept any parameter shape (e.g. a proxy that forwards requests to an external API). For most tools, a static Pydantic schema in the repo is simpler.

Routing / multi-node hint

If the framework needs to pick between multiple downstream agents (e.g. a supervisor that routes to a security agent or a support agent based on the user input), do not roll your own routing. Use LaunchDarkly agent graphs — the graph's edges carry the routing contract, and the supervisor's instructions can be auto-injected with the valid routes.

The devrel tutorial's generic_agent.py shows a minimal version of this:

# If this node has outgoing edges with routes, inject them into instructions
if self.valid_routes:
    route_instruction = (
        f"\n\nYou must select one of these routes: {self.valid_routes}. "
        f'Return your choice in JSON format: {{"route": "<selected_route>"}}'
    )
    instructions = instructions + route_instruction

For the full graph pattern, read agent-graph-reference.md — but again, single-agent migration comes first, and agent graphs are currently Python-only in the SDK.

Keep the provider call in the repo

One rule that applies across all three frameworks: the provider SDK call (OpenAI, Anthropic, Bedrock) stays in your code. The config only changes the inputs to that call — model name, instructions, parameters, tool list. It does not replace the provider SDK. That means:

• You keep full control of error handling, retries, timeouts, custom headers
• You keep full control of streaming logic and backpressure
• You keep full control of authentication (API keys, IAM roles, Bedrock Converse sessions)
• The config is additive — removing it gets you back to the original hardcoded app, provided the fallback mirrors the old values