Generate Ors Env

generate-ors-env

Build the ORS variant of an env using the official `openreward >= 0.1.33` package (the ors-sdk name is a common mistake — it does not exist on PyPI).

Concept

ORS is the Open Reward Standard (openrewardstandard.io) — an HTTP REST + Server-Sent Events protocol for agent envs. Reward arrives inline with every ToolOutput, which is the framework's defining feature compared to OpenEnv (external/post-hoc reward) and NeMo Gym (post-episode /verify).

When the user has a shared domain module (<domain>.py) and wants an ORS variant, never duplicate domain logic into the framework folder — wrap it.

Archetypes

Imports — exactly these

Server side:

from openreward.environments import (
    Environment, Server, tool, ToolOutput, TextBlock, Split, ImageBlock,
)

Client side (rollouts):

from openreward import EnvironmentsAPI
api = EnvironmentsAPI(base_url=URL, api_key="")
env = api.get(ENV_NAME)

Don't use `OpenReward(api_key=..., base_url=...)` even though it's the high-level client. It prepends matrix. / api. / construct. subdomains to the base URL — that breaks HF Space URLs. EnvironmentsAPI talks to base_url verbatim.

Architecture

<env_dir>/ors/
├── pyproject.toml         # openreward>=0.1.33 + e2b-* (if needed) + pydantic
├── __init__.py
├── Dockerfile             # local dev image
├── Dockerfile.spaces      # HF Space (port 7860, single-stage pip install)
├── README.spaces.md       # HF Space frontmatter
├── server.py              # the Environment subclass + main()
├── tasks.py               # list of dicts (task_spec for each task)
├── rollout.py             # or rollout_openai.py + rollout_qwen.py
└── README.md              # one-page dev README

Implementation order

1. Tasks file — tasks.py

A list of plain dicts. Each dict becomes a task_spec per session. ORS auto-wraps these into Task objects on list_tasks().

TASKS = [
    {"answer": "apple", "task": "Guess the 5-letter word."},
    # ...
]

2. The Environment subclass — server.py

from pydantic import BaseModel
from openreward.environments import Environment, Server, tool, ToolOutput, TextBlock, Split

class GuessInput(BaseModel):
    word: str

class WordleORS(Environment):
    def __init__(self, task_spec=None, secrets=None, **kw):
        super().__init__(task_spec=task_spec or {}, secrets=secrets or {})
        self._game = None

    def setup(self):                      # called on first tool invocation
        self._game = WordleGame(self.task_spec.get("answer"))

    def teardown(self):                   # called on session delete
        self._game = None

    @classmethod
    def list_splits(cls): return [Split(name="train", type="train")]

    @classmethod
    def list_tasks(cls, split): return TASKS

    def get_prompt(self):
        return [TextBlock(text="Play Wordle. Guess the 5-letter word.")]

    @tool
    def guess(self, params: GuessInput) -> ToolOutput:
        feedback = self._game.guess(params.word)
        return ToolOutput(
            blocks=[TextBlock(text=feedback)],
            reward=self._game.reward,
            finished=self._game.done,
        )

Key contracts:

• Tools take a `params: PydanticModel` as the second arg. ORS uses the model's JSON schema as the tool's input_schema.
• Empty inputs still need a Pydantic model (class _Empty(BaseModel): pass). Don't omit the param.
• `ToolOutput.blocks` is [TextBlock | ImageBlock]. For images: ImageBlock(data=<base64>, mimeType="image/png"). Vision models actually see this.
• `reward` is float | None. None means "no reward this step"; 0.0 means "stepped, scored zero". For pure terminal reward, return None everywhere except in the last ToolOutput.
• `finished=True` ends the session. Pair with reward=1.0 (or whatever) to give the rollout a clean stop.
• `task_spec` is a dict you read from self.task_spec — no schema validation. If you want validation, do it in setup().

3. Server entry point — server.py main

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--port", type=int, default=8080)
    parser.add_argument("--host", type=str, default="0.0.0.0")
    args = parser.parse_args()
    Server([WordleORS]).run(host=args.host, port=args.port)

The endpoint name is auto-derived from the class name lowercased — WordleORS → wordleors. Tell the user this so they know what ENV_NAME to pass.

4. Rollout

Always discover tools and tasks from the env. Don't hardcode names:

api = EnvironmentsAPI(base_url=ENV_URL, api_key="")
env = api.get("wordleors")
tasks = env.list_tasks("train")
tools = env.list_tools(format="openai")     # built-in OpenAI tool-schema converter
with env.session(task=tasks[0]) as session:
    prompt = session.get_prompt()
    result = session.call_tool("guess", {"word": "crane"})
    # result.blocks, result.reward, result.finished

For vision envs, the screenshot tool returns an ImageBlock — read it as b.data (already base64). Pass that into the model's image content.

5. Dockerfiles

Dockerfile.spaces is the HF Space deploy image. Keep it minimal:

FROM python:3.11-slim
RUN useradd -m -u 1000 user
RUN pip install --no-cache-dir openreward pydantic <other-deps>
USER user
ENV HOME=/home/user PATH=/home/user/.local/bin:$PATH
WORKDIR $HOME/app
COPY --chown=user . $HOME/app
EXPOSE 7860
CMD ["python", "server.py", "--host", "0.0.0.0", "--port", "7860"]

README.spaces.md:

---
title: My Env ORS
emoji: 🎯
colorFrom: pink
colorTo: indigo
sdk: docker
app_port: 7860
tags: [ors, openreward]
---

Pushing to HF Spaces

Create a Space named <owner>/<env_name>-ors. Set E2B_API_KEY (and any other secrets) as Space secrets, not environment variables — they survive rebuilds. The local .env file should not be uploaded.

api.add_space_secret(repo_id="<owner>/<env>-ors", key="E2B_API_KEY", value="...")
api.upload_file(path_or_fileobj="Dockerfile.spaces", path_in_repo="Dockerfile", repo_id=...)
api.upload_file(path_or_fileobj="README.spaces.md", path_in_repo="README.md", repo_id=...)
# upload server.py, tasks.py, __init__.py, pyproject.toml

Validation gates

1. Local server — uv run python server.py --port 8772 then curl http://localhost:8772/list_environments returns ["<envname>"]. 2. Tool discovery — curl http://localhost:8772/<envname>/tools | jq '.tools | length' matches the number of @tool methods. 3. End-to-end — MAX_TURNS=3 uv run python rollout.py drives the model through at least one tool call without errors.

Gotchas (from real-world ORS work)

• `from openreward.environments.types import Task` — wrong; Task is in openreward.api.environments.types and you usually don't import it. list_tasks can return plain dicts; ORS wraps them.
• `OpenReward(base_url=URL)` rewrites the URL — prepends matrix. / api. / construct. subdomains. For HF Spaces, use EnvironmentsAPI(base_url=URL, api_key="") directly.
• `e2b-desktop` without `e2b` — e2b-desktop imports from e2b, but doesn't pin it. Add both to dependencies.
• Endpoint name is the lowercased class name — MyEnvORS becomes myenvors. Tell users this explicitly so their ENV_NAME env var is right.

Reference

• references/architecture.md — protocol shape + Server / Environment / Session lifecycle

Official documentation

• openrewardstandard.io — protocol specification
• docs.openreward.ai — Python SDK + platform docs
• openreward on PyPI — current package (latest 0.1.81+)
• Talc-AI/OpenReward on GitHub — source

ORS architecture (deep)

Wire protocol (REST + SSE)

Endpoint name = <EnvironmentClassName>.lower(). So class WordleORS(Environment) → /wordleors/....

The EnvironmentsAPI Python client wraps this via aiohttp. OpenReward wraps EnvironmentsAPI and rewrites base_url to matrix.<host> — that's why HF Space targets need EnvironmentsAPI direct.

Environment lifecycle

1. `__init__(task_spec, secrets, kw)** — called once per session. Store task_spec (already on self.task_spec after super().__init__); validate or stub. 2. **setup()** — called on first tool invocation. Allocate sandbox, init game state. 3. **teardown()** — called on DELETE /sessions/<sid>. Kill the sandbox, free resources. 4. **get_prompt() -> [TextBlock | ImageBlock]** — called when the client asks for the prompt. Reads from self.task_spec. 5. **@classmethod list_splits()** and **@classmethod list_tasks(split)** — class-level (no self). Splits are static metadata. Tasks can be plain dicts; ORS wraps them. 6. **Tool methods** — decorated @tool, signature (self, params: BaseModel) -> ToolOutput`.

ToolOutput shape

ToolOutput(
    blocks=[TextBlock(text="..."), ImageBlock(data=<b64>, mimeType="image/png")],
    metadata={"any": "json"},
    reward=0.5,                  # float | None
    finished=False,              # bool — True ends the session
)

Per-step rewards add up across the trajectory; finished=True is the only way for the env to signal terminal state.

Splits & tasks

Split.type is "train" | "validation" | "test". The split name and type can differ. List N tasks per split, each a dict that becomes the per-session task_spec.

A common pattern:

TASKS = [{"answer": w, "task": "Guess the word"} for w in WORDS[:50]]

@classmethod
def list_tasks(cls, split): return TASKS

ORS wraps each dict into a Task(server_name=cls.__name__, environment_name=..., task_spec=dict) automatically.

Secrets

Environment.__init__ takes secrets. The client passes secrets per-session in the POST /sessions body. Use this for per-rollout API keys (E2B sandbox, etc.) — they don't appear in the server's environment.

Sync vs async clients

Both EnvironmentsAPI and AsyncEnvironmentsAPI exist. The sync wrapper runs an async loop under the hood. For multi-rollout scenarios prefer async:

async with AsyncOpenReward(api_key="").environments as api:
    env = api.get(name, base_url=URL)
    async with env.session(task=task) as session:
        ...

Image handling

ImageBlock.data is base64, not raw bytes. The server-side ImageBlock(data=base64.b64encode(png).decode(), mimeType="image/png") matches the client-side b.data (still base64). Don't re-encode.

Canonical helper:

def _shot_block(sandbox) -> ImageBlock:
    data = sandbox.screenshot()
    return ImageBlock(data=base64.b64encode(data).decode("utf-8"), mimeType="image/png")

Production deployment

The wordle and desktop ORS variants both deploy to HF Spaces using a minimal Dockerfile.spaces:

FROM python:3.11-slim
RUN useradd -m -u 1000 user
RUN pip install --no-cache-dir openreward pydantic <env-specific>
USER user
WORKDIR /home/user/app
COPY --chown=user . .
EXPOSE 7860
CMD ["python", "server.py", "--host", "0.0.0.0", "--port", "7860"]

Plus README.spaces.md with the HF frontmatter. Push via:

api.add_space_secret(repo_id, "E2B_API_KEY", value)
api.upload_file(path_or_fileobj="Dockerfile.spaces", path_in_repo="Dockerfile", ...)
api.upload_file(path_or_fileobj="README.spaces.md", path_in_repo="README.md", ...)

For OpenReward.ai deployment (the platform), the same files work — see docs.openreward.ai for the GitHub-integration flow.

What can go wrong

• pip install ors-sdk — package doesn't exist on PyPI.
• Endpoint name mismatch — class MyEnv is served at /myenv, not /my_env.
• OpenReward(base_url="https://X.hf.space") ends up calling https://matrix.X.hf.space — broken DNS. Use EnvironmentsAPI.
• setup() not called — happens silently when an __init__ raises before tools are registered. Check the server log.