Rl Env From Description

RL Env From Description

Convert a plain-English description of an RL training environment into runnable code across OpenEnv, OpenReward (ORS), Verifiers, and NeMo Gym. Two other framework variants (SkyRL Gym, GEM) are secondary and only relevant for text-action-with-tag-parsing envs — produce them only if the user asks.

When to use

• A user describes an env in plain English (a goal, an action surface, or a reward shape) and wants code.
• A user asks to "build an env for X", "scaffold an RL env", "port this env to OpenEnv/ORS/Verifiers/NeMo Gym".
• A user already has a runnable env in one framework and wants the same env in others.
• A user asks "what's the right way to design my reward / state / tool surface for this task" — start with the interview below, then implement.

Do not use for: training runs (TRL/GRPO config), evaluation harness work, or general agent-design questions that don't end with new env code.

Recommended layout (suggest, don't impose)

A clean shape that scales well — but the user gets to pick the actual paths:

<env_dir>/                          # whatever the user names it
├── <domain>.py                     # SHARED pure logic (e.g. game.py)
├── tasks.py                        # SHARED list of task dicts (optional)
├── openenv/                        # OpenEnv variant (HTTP, MCP)
├── ors/                            # ORS variant (HTTP, REST + SSE)
├── verifiers/                      # Verifiers variant (in-process)
└── nemo_gym/                       # NeMo Gym variant (HTTP, REST + cookies)

Inside each framework folder, the public contract is:

• pyproject.toml — framework-specific deps
• __init__.py
• One implementation file (server.py for ORS, server/<env>_environment.py for OpenEnv, env.py for Verifiers, server.py for NeMo Gym)
• rollout.py — runs an LLM against the env end-to-end
• README.md — one-page consumption guide

Always ask the user where they want files written. If they don't have a preference, propose the layout above. Don't force it.

The four-step flow

Step 1 — Interview the user (focused, not exhaustive)

Ask only the questions that determine architecture. The full bank lives in references/interview.md; the must-cover set is:

1. What does the agent DO? One sentence describing the goal and the loop. 2. Action surface — structured tool calls (most cases) or free text with tag parsing (rare; only when the model has no tool-calling support). 3. State — does anything persist across turns? Per-session sandbox? In-memory dict? Nothing? 4. Reward — when does it fire (per-step, on terminate, post-episode)? What's the success criterion? 5. External backends — sandbox (E2B), web service, none? 6. Termination — fixed turn cap, model-emits-terminate, or a derived condition. 7. Where should the files live? Project-relative path; never assume.

If the user has already given enough signal in their description (e.g. they cited an existing env they want to mirror), skip questions whose answers are obvious. Don't make people repeat themselves.

When in doubt about an architectural choice, propose a default with a one-line rationale and let the user veto.

Step 2 — Pick the closest archetype

Match the user's description to one of these archetypes; tell them which archetype you're using and why:

Step 3 — Implement in dependency order

The shared module first, then per-framework variants. Order doesn't matter between frameworks.

1. `<env_dir>/<domain>.py` + `<env_dir>/tasks.py` — the only file that contains domain logic. Frameworks just wrap it. Keep it pure-Python; no framework imports. 2. OpenEnv — read references/openenv.md (planner-level) or trigger generate-openenv-env skill (full workflow). Use MCPEnvironment + @mcp.tool + create_app(...) in server/app.py. 3. ORS — read references/ors.md or trigger generate-ors-env. Use Environment + @tool methods + ToolOutput(blocks=[...], reward=..., finished=...). Per-tool-call reward is the framework's defining feature. 4. Verifiers — read references/verifiers.md or trigger generate-verifiers-env. Plain Python tool functions on a toolkit class; vf.ToolEnv + vf.Rubric for native consumption. 5. NeMo Gym — read references/nemo_gym.md or trigger generate-nemo-gym-env. SimpleResourcesServer with one app.post("/<tool>") per tool; cookie sessions; post-episode /verify reward.

Step 4 — Validate end-to-end before declaring done

Each framework folder gets ONE smoke rollout against a small LLM (Qwen via HF Router by default, or OpenAI if OPENAI_API_KEY is set). The rollout must:

• Discover the tools the env exposes (don't hardcode names — except for NeMo Gym, which has no list_tools()).
• Drive a 3–5 turn loop and print every tool call + result.
• Fail loudly if a tool call errors. (MAX_TURNS=3 for the smoke check.)

If the env needs an external backend (E2B, etc.), check for the relevant secret in .env and stop with a clear error if it's missing.

What success looks like

A user typing "make me an env where the agent plays connect-four at path/to/connect_four/" should end with:

• path/to/connect_four/game.py (the shared engine), tasks.py (a few starting positions)
• path/to/connect_four/openenv/, .../ors/, .../verifiers/, .../nemo_gym/ all runnable
• 4 green rollout smoke tests (NeMo Gym tested via deployed Space if local Ray init fails on shared nodes)
• Whatever README convention the project uses, updated

…in one continuous flow, with the user only answering 5–7 questions along the way.

Reference docs

• references/interview.md — full question bank with example answers
• references/openenv.md — OpenEnv-specific implementation notes (planner-level; defers to generate-openenv-env)
• references/ors.md — ORS planner-level (defers to generate-ors-env)
• references/verifiers.md — Verifiers planner-level (defers to generate-verifiers-env)
• references/nemo_gym.md — NeMo Gym planner-level (defers to generate-nemo-gym-env)

When the user wants only one framework variant, trigger the framework-specific skill directly: generate-openenv-env, generate-ors-env, generate-verifiers-env, or generate-nemo-gym-env.

Hard guardrails

• Don't impose a folder layout. Suggest the recommended one once; respect the user's choice if they want different paths.
• Don't skip the shared domain module. Cross-framework consistency is impossible without it. Every framework variant must wrap the same <domain>.py — never duplicate logic.
• Don't run training. This skill ends with rollouts. Training/eval is a separate concern.
• Don't invent APIs. When unsure about a framework's actual call shape, read its references/architecture.md (in the framework-specific skill) before writing code.
• Coordinate spaces matter for vision envs. Declare the convention (pixel vs normalized 0–1000) in the prompt. Qwen2.5-VL emits 0–1000 normalized; the rollout adapter must rescale.

Official documentation

• OpenEnv: meta-pytorch/OpenEnv · docs
• OpenReward (ORS): openrewardstandard.io · docs.openreward.ai · openreward on PyPI
• Verifiers: PrimeIntellect-ai/verifiers · docs
• NeMo Gym: NVIDIA-NeMo/Gym · docs

Interview question bank

Use this when the user's description is too thin to start coding. Don't run the full bank — pick what's missing from their prompt.

1. The loop in plain English

Ask: "In one or two sentences, what is the agent doing each turn?"

Listen for:

• The trigger ("the user asks…", "a board state arrives…")
• The agent's choice ("…the agent picks an action…")
• The feedback ("…and sees the result of that action")

If you can't draft a 5-step bullet trace from their answer, ask again.

2. Action surface

Ask: "What can the agent do, concretely?" — accept any of:

• A list of tool names with arguments → structured tool calls, the default. Targets: OpenEnv (@mcp.tool), ORS (@tool on Environment), Verifiers (plain functions).
• "It just types a guess word" → single-tool, like Wordle.
• "It writes code blocks / XML tags" → text-action with parsing. Only do this if the model has no native tool calling. Targets: SkyRL Gym / GEM.
• "It clicks and types on a screen" → vision / computer-use, 19-tool action surface modelled on Anthropic's computer_20251124 schema (broadest superset across Claude / OpenAI Operator / Qwen3-VL).

If you genuinely can't tell, propose structured tool calls — they work with every modern model.

3. State across turns

Ask: "Between two tool calls in the same episode, does anything need to remember the previous call?"

State that survives across episodes is rare and usually a bug — flag it.

4. Reward

Ask: "How do we know the agent did well?" — the answer pins the reward style:

5. External backends

Ask: "Does the env need anything outside the Python process?"

• E2B sandbox? → Need E2B_API_KEY in .env. Verifiers / SkyRL / GEM run the sandbox in-process; OpenEnv / ORS / NeMo Gym run it server-side.
• Web service? → Probably belongs as a tool that uses requests / httpx.
• Database? → Treat as a tool too. State lives there, not in the env class.
• Nothing? → Easiest case.

6. Termination

Ask: "When does an episode end?"

• "After N turns" → fixed turn cap, set in env config.
• "When the model says it's done" → expose a terminate(status) tool, watch for finished=True.
• "When a condition is met" (e.g. won == True) → check inside the tool that mutated state, return finished=True from there.
• "Whichever comes first" → both. The rollout caps MAX_TURNS and the env signals finished early.

7. The two-question shortcut

If you only have time for two questions, ask:

1. "What does the agent do?" (covers loop + action surface) 2. "What's the reward signal?" (covers reward + termination)

Everything else can be defaulted reasonably.

NeMo Gym quick reference (for the umbrella skill)

For implementation, defer to generate-nemo-gym-env. Planner-level summary here.

What NeMo Gym is

NVIDIA's RL gym layer for LLM agents. Built on Ray. Python package nemo_gym (install via pip install git+https://github.com/NVIDIA-NeMo/Gym). Targets the NeMo training stack but works fine with TRL/GRPO.

Core types

Reward model

Post-episode via /verify. The trainer sends the full trajectory (body.response.output) plus body.ground_truth; you return BaseVerifyResponse(**body.model_dump(), reward=...). No per-step rewards.

When NeMo Gym beats OpenEnv / ORS / Verifiers

• You want post-hoc grading from a Ray-orchestrated job (e.g. unit-test execution, LLM-as-judge).
• You're already in NVIDIA's NeMo stack.
• You need cookie-based sessions for tool isolation across concurrent rollouts.
• Aggregate metrics across episodes (/verify is the natural seam).

When NeMo Gym loses

• The user wants per-tool-call reward → ORS.
• The user is on a shared SLURM / HF cluster node where Ray init can't bind → OpenEnv or ORS.
• The user wants tool-discovery via list_tools() — NeMo Gym has none; tool schemas are hardcoded in the rollout.

Common confusions

• Ray init fails on shared cluster nodes (gcs_server can't bind). Local python server.py doesn't work in those environments — only deployed Space does.
• No client SDK. The rollout speaks raw requests with a requests.Session() for cookie persistence.
• Tool schemas are hardcoded in the rollout. When the server's Pydantic body changes, manually update the rollout's tool definition list.
• Dataset format requires `responses_create_params` (JSON-stringified) and `ground_truth` (list of dicts). The ground_truth[0] shape is your call — typically {"expected_output": "..."}.

OpenEnv quick reference (for the umbrella skill)

The umbrella skill should read this when planning the OpenEnv variant. For the full implementation flow, defer to the generate-openenv-env skill.

What OpenEnv is

HTTP server exposing tools via the MCP (Model Context Protocol) shape. The runtime is FastAPI; tools are FastMCP-decorated functions. The client (MCPToolClient) discovers tools via list_tools() and calls them via call_tool(name, **args).

Core types

Reward model

External. The env doesn't return a reward per tool call; the trainer or rollout computes it from the trajectory. Pair with TRL's GRPOTrainer reward function, or compute inline in the rollout.

When OpenEnv beats ORS / Verifiers

• You want MCP ecosystem compatibility (Claude Desktop, MCP Inspector, etc.)
• You need a full Gradio UI bundled with the env
• You're targeting Meta's OpenEnv ecosystem / HF Spaces deployment
• The reward depends on the whole trajectory (not per-call)

When OpenEnv loses

• You want per-call reward — pick ORS instead.
• You don't need HTTP at all — pick Verifiers.
• The tool schemas need fancy validation that FastMCP's pydantic introspection can't handle (rare).

ORS (OpenReward) quick reference (for the umbrella skill)

For implementation, defer to generate-ors-env. This is the planner-level summary.

What ORS is

HTTP REST + Server-Sent Events protocol from openrewardstandard.io. Official Python SDK is `openreward` on PyPI (the ors-sdk name does not exist — common mistake). Servers expose:

• GET /list_environments
• GET /<env_name>/tools
• POST /<env_name>/sessions (with task_spec body)
• POST /<env_name>/sessions/<sid>/tool (call a tool)
• GET /<env_name>/splits and POST /<env_name>/tasks

The Python server does this for you when you subclass Environment and decorate @tool.

Core types

Reward model

Per tool call. Every ToolOutput carries a reward field. Use None for "no reward this step" and a float for "scored". The session ends when finished=True.

When ORS beats OpenEnv / Verifiers

• You want per-step rewards (env-side, not trainer-side).
• You want declarative task_spec + train/val/test splits without writing your own dataset code.
• You want to deploy to OpenReward.ai's hosted infrastructure.

When ORS loses

• The MCP ecosystem matters more than reward ergonomics → OpenEnv.
• No HTTP server desired → Verifiers.

Common confusions

• ors-sdk is not on PyPI. Always use openreward.
• Endpoint name is the lowercased class name: WordleORS → wordleors.
• OpenReward(base_url=URL) mangles URL with subdomain prefixes — use EnvironmentsAPI direct for HF Space targets.
• Task lives in openreward.api.environments.types, but you usually return plain dicts from list_tasks() — ORS auto-wraps them.

Verifiers quick reference (for the umbrella skill)

For implementation, defer to generate-verifiers-env. Planner-level summary here.

What Verifiers is

PrimeIntellect Verifiers — in-process tool-calling RL env framework. No HTTP, no Docker. The trainer or rollout imports tool functions directly. Designed for fast iteration and clean handoff to TRL GRPOTrainer.

Core types

Reward model

Rubric-based, post-hoc. Each func in the rubric is async def correctness(completion, answer, **kwargs) -> float. They run after the rollout completes and the floats are aggregated (averaged or weighted).

When Verifiers beats OpenEnv / ORS

• You want zero deployment friction.
• The env is pure Python or runs an in-process sandbox per episode.
• You're iterating on reward design and want to redeploy graders without restarting a server.
• You're going straight to TRL training — Verifiers' adapter pattern is the cleanest.

When Verifiers loses

• The env needs to run on different infra than the trainer (GPU pool vs CPU pool).
• You want HTTP for cross-language consumers.
• You want the agent's screen / terminal output to render in a separate UI for human inspection (Verifiers has no UI).

Common confusions

• Two consumption paths exist (toolkit class for TRL adapter, free functions for vf.ToolEnv). Always provide both — they share state via a module-level shared toolkit.
• **kwargs in tool signatures is forbidden by some downstream trainers (vLLM-based) — JSON schema introspection fails. Use explicit params.
• TypedDicts are common in verifiers data structures — access by key, not attribute.