Building Chat Widgets

Building Chat Widgets

Create interactive widgets for AI chat with actions and entity tagging.

Quick Start

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      if (action.type === "view_details") {
        navigate(`/details/${action.payload.id}`);
      }
    },
  },
});

---

Action Handler Types

---

Widget Lifecycle

1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state

---

Core Patterns

1. Widget Templates

Define reusable widget layouts with dynamic data:

{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "item-1",
      "onClickAction": {
        "type": "item.select",
        "handler": "client",
        "payload": { "itemId": "item-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Item title", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}

2. Client-Handled Actions

Actions that update local state, navigate, or send follow-up messages:

Widget Definition:

{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}

Frontend Handler:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      switch (action.type) {
        case "open_article":
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});

3. Server-Handled Actions

Actions that mutate data, update widgets, or require backend processing:

Widget Definition:

{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}

Backend Handler:

from chatkit.types import (
    Action, WidgetItem, ThreadItemReplacedEvent,
    ThreadItemDoneEvent, AssistantMessageItem, ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: RequestContext,  # Note: Already RequestContext, not dict
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]  # Use .payload, not .arguments

            # 1. Update widget with selection
            updated_widget = build_selector_widget(selected=line_id)
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}"}],
                )
            )

            # 3. Trigger client effect
            yield ClientEffectEvent(
                name="selection_changed",
                data={"lineId": line_id},
            )

4. Entity Tagging (@mentions)

Allow users to @mention entities in messages:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: { type: item.type, article_id: item.id },
      }));
    },

    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },
  },
});

5. Composer Tools (Mode Selection)

Let users select different AI modes from the composer:

const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});

Backend Routing:

async def respond(self, thread, item, context):
    tool_choice = context.metadata.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    else:
        agent = self.general_agent

    result = Runner.run_streamed(agent, input_items)
    async for event in stream_agent_response(context, result):
        yield event

---

Widget Component Reference

Layout Components

Content Components

Interactive Components

---

Critical Implementation Details

Action Object Structure

IMPORTANT: Use action.payload, NOT action.arguments:

# WRONG - Will cause AttributeError
action.arguments

# CORRECT
action.payload

Context Parameter

The context parameter is RequestContext, not dict:

# WRONG - Tries to wrap RequestContext
request_context = RequestContext(metadata=context)

# CORRECT - Use directly
user_id = context.user_id

UserMessageItem Required Fields

When creating synthetic user messages:

from chatkit.types import UserMessageItem, UserMessageTextContent

# Include ALL required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)

---

Anti-Patterns

1. Mixing handlers - Don't handle same action in both client and server 2. Missing payload - Always include data in action payload 3. Using action.arguments - Use action.payload 4. Wrapping RequestContext - Context is already RequestContext 5. Missing UserMessageItem fields - Include id, thread_id, created_at 6. Wrong content type - Use type="input_text" for user messages

---

Verification

Run: python3 scripts/verify.py

Expected: ✓ building-chat-widgets skill ready

If Verification Fails

1. Check: references/ folder has widget-patterns.md 2. Stop and report if still failing

References

• references/widget-patterns.md - Complete widget patterns
• references/server-action-handler.md - Backend action handling

Server Action Handler Reference

Complete backend pattern for handling widget actions.

Full Action Handler Implementation

from chatkit.server import ChatKitServer
from chatkit.types import (
    Action,
    WidgetItem,
    ThreadMetadata,
    ThreadItemReplacedEvent,
    ThreadItemDoneEvent,
    AssistantMessageItem,
    HiddenContextItem,
    ClientEffectEvent,
    UserMessageItem,
    UserMessageTextContent,
)
from datetime import datetime
from typing import Any, AsyncIterator

class MyServer(ChatKitServer[RequestContext]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: RequestContext,  # Already RequestContext, not dict!
    ) -> AsyncIterator[ThreadStreamEvent]:
        """Handle widget actions.

        CRITICAL NOTES:
        - context is RequestContext, not dict (type hint is misleading)
        - Use action.payload, not action.arguments
        - Include all required fields in UserMessageItem
        """

        if action.type == "item.select":
            item_id = action.payload["id"]

            # 1. Update widget with selection state
            updated_widget = build_selector_widget(
                items=self.items,
                selected=item_id,
            )
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Add hidden context for future agent input
            await self.store.add_thread_item(
                thread.id,
                HiddenContextItem(
                    id=self.store.generate_item_id("ctx", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=f"<SELECTED>{item_id}</SELECTED>",
                ),
                context=context,
            )

            # 3. Stream assistant response
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {item_id}. What next?"}],
                )
            )

            # 4. Trigger client effect
            yield ClientEffectEvent(
                name="selection_mode",
                data={"itemId": item_id},
            )

        elif action.type == "form.submit":
            form_data = action.payload

            # Process form submission
            result = await process_form(form_data)

            # Create synthetic user message to trigger agent
            synthetic_message = UserMessageItem(
                id=self.store.generate_item_id("message", thread, context),
                thread_id=thread.id,
                created_at=datetime.now(),
                content=[
                    UserMessageTextContent(
                        type="input_text",  # Must be "input_text", not "text"
                        text=f"Form submitted: {result.summary}"
                    )
                ],
                inference_options={},  # Required field
            )

            # Run agent with synthetic message
            async for event in self.respond(thread, synthetic_message, context):
                yield event

        elif action.type == "confirm.accept":
            # Handle confirmation action
            item_id = action.payload["item_id"]

            # Update database
            await self.db.confirm_item(item_id)

            # Replace widget with success state
            success_widget = build_success_widget(item_id)
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": success_widget})
            )

            # Notify client
            yield ClientEffectEvent(
                name="item_confirmed",
                data={"itemId": item_id},
            )

Action Type Definition

from chatkit.types import Action

# Action[str, Any] has these fields:
action.type      # str - action identifier (e.g., "task.start")
action.payload   # dict[str, Any] - action data
action.handler   # "client" | "server" - where processed

Common Patterns

Selection Actions

if action.type.endswith(".select"):
    entity_id = action.payload["id"]

    # Update widget
    yield ThreadItemReplacedEvent(
        item=sender.model_copy(update={
            "widget": build_widget(selected=entity_id)
        })
    )

Form Actions

if action.type == "form.submit":
    # Create synthetic message
    message = UserMessageItem(
        id=self.store.generate_item_id("message", thread, context),
        thread_id=thread.id,
        created_at=datetime.now(),
        content=[UserMessageTextContent(type="input_text", text="...")],
        inference_options={},
    )

    # Run agent
    async for event in self.respond(thread, message, context):
        yield event

Confirmation Actions

if action.type == "confirm":
    # Perform action
    await self.do_action(action.payload)

    # Replace widget
    yield ThreadItemReplacedEvent(
        item=sender.model_copy(update={
            "widget": build_success_widget()
        })
    )

Error Handling

async def action(self, thread, action, sender, context):
    try:
        if action.type == "dangerous.action":
            # Validate
            if not self.can_perform(action, context):
                yield ThreadItemDoneEvent(
                    item=AssistantMessageItem(
                        id=self.store.generate_item_id("msg", thread, context),
                        thread_id=thread.id,
                        created_at=datetime.now(),
                        content=[{"text": "Permission denied."}],
                    )
                )
                return

            # Proceed
            ...

    except Exception as e:
        # Log error
        logger.error(f"Action failed: {e}")

        # Notify user
        yield ThreadItemDoneEvent(
            item=AssistantMessageItem(
                id=self.store.generate_item_id("msg", thread, context),
                thread_id=thread.id,
                created_at=datetime.now(),
                content=[{"text": f"Action failed: {str(e)}"}],
            )
        )

Widget Patterns Reference

Complete patterns for ChatKit widget implementation.

Widget Template Format

{
  "version": "1.0",
  "name": "task_list",
  "template": "{\"type\":\"ListView\",\"children\":[...jinja template...]}",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "tasks": { "type": "array", "items": {...} }
    }
  }
}

Loading Templates in Python

from chatkit.widgets import WidgetTemplate, WidgetRoot

# Load template from file
task_list_template = WidgetTemplate.from_file("task_list.widget")

def build_task_list_widget(tasks: list[Task]) -> WidgetRoot:
    return task_list_template.build(
        data={
            "tasks": [task.model_dump() for task in tasks],
            "selected": None,
        }
    )

Client-to-Server Action Forwarding

When client handles action locally then notifies server:

widgets: {
  onAction: async (action, widgetItem) => {
    if (action.type === "select_name") {
      // Forward to server for processing
      await chatkit.sendCustomAction(action, widgetItem.id);

      // Refresh local state after server processes
      const data = await refreshStatus();
      if (data) handleStatusUpdate(data);
    }
  },
}

Thread Item Actions (Feedback/Retry/Share)

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  threadItemActions: {
    feedback: true,   // Thumbs up/down
    retry: true,      // Regenerate response
    share: true,      // Share message
  },

  onLog: ({ name, data }) => {
    if (name === "message.feedback") {
      trackFeedback(data);
    }
  },
});

Widget Streaming from Tools

When agent tool generates a widget:

from chatkit.types import WidgetItem, ThreadItemDoneEvent
from agents import function_tool

@function_tool
async def show_article_list(ctx: AgentContext, query: str) -> str:
    """Show a list of articles matching the query."""

    articles = await article_store.search(query)

    # Build widget
    widget = build_article_list_widget(articles)

    # Create widget item
    widget_item = WidgetItem(
        id=ctx.store.generate_item_id("widget", ctx.thread, ctx.request_context),
        thread_id=ctx.thread.id,
        created_at=datetime.now(),
        widget=widget,
    )

    # Save to store
    await ctx.store.add_thread_item(ctx.thread.id, widget_item, ctx.request_context)

    # Yield as event
    yield ThreadItemDoneEvent(item=widget_item)

    return f"Showing {len(articles)} articles"

Local Tool Wrappers for Widget Streaming

Problem: MCP tools alone don't stream widgets - need local wrappers.

from agents import function_tool

@function_tool
async def show_task_form(
    ctx: RunContextWrapper[AgentContext],
) -> str:
    """Show interactive task creation form widget."""

    agent_ctx = ctx.context
    mcp_url = agent_ctx.mcp_server_url

    # Call MCP tool via HTTP
    result = await _call_mcp_tool(
        mcp_url,
        "show_task_form",
        arguments={"params": {"user_id": agent_ctx.user_id}},
        access_token=agent_ctx.access_token,
    )

    # Return result - RunHooks will stream widget
    return json.dumps(result)

# In RunHooks.on_tool_end()
async def on_tool_end(self, output: str | None, tool_name: str) -> None:
    if tool_name == "show_task_form":
        result = json.loads(output)
        if result.get("action") == "show_form":
            widget = build_task_form_widget()
            yield WidgetItem(...)

Entity Conversion in Backend

Convert @mentions to model-readable markers:

class EntityAwareConverter(BasicThreadItemConverter):
    async def to_agent_input(self, items: list[ThreadItem]) -> list:
        result = []
        for item in items:
            if isinstance(item, UserMessageItem):
                content = item.content
                for entity in item.entities or []:
                    if entity.type == "article":
                        content = content.replace(
                            f"@{entity.title}",
                            f"<ARTICLE id='{entity.id}'>{entity.title}</ARTICLE>"
                        )
                result.append({"role": "user", "content": content})
        return result

Common Validation Errors

Error 1: 'Action' object has no attribute 'arguments'

AttributeError: 'Action[str, Any]' object has no attribute 'arguments'

Fix: Use action.payload instead of action.arguments

Error 2: UserMessageTextContent type mismatch

ValidationError: Input should be 'input_text' [type=literal_error, input_value='text']

Fix: Use type="input_text" for user input, not type="text"

Error 3: UserMessageItem missing required fields

4 validation errors for UserMessageItem
- id: Field required
- thread_id: Field required
- created_at: Field required
- inference_options: Field required

Fix: Include all required fields when creating UserMessageItem

Error 4: RequestContext wrapping issue

2 validation errors for RequestContext
metadata: Input should be a valid dictionary [input_value=RequestContext(...)]

Fix: Don't wrap context - it's already a RequestContext object

Widget Action Testing Checklist

• [ ] Widget renders with correct data
• [ ] All buttons have clear labels
• [ ] Client actions navigate/update UI correctly
• [ ] Server actions call backend successfully
• [ ] Action payload contains all required data
• [ ] Widget updates after server action completes
• [ ] No AttributeError on action.payload access
• [ ] No ValidationError on UserMessageItem creation
• [ ] Local tool wrappers trigger widget streaming

Evidence Sources

Patterns derived from:

• cat-lounge/backend/app/widgets/cat_name_suggestions.widget
• cat-lounge/frontend/src/components/ChatKitPanel.tsx
• metro-map/backend/app/server.py
• metro-map/frontend/src/components/ChatKitPanel.tsx
• news-guide/frontend/src/components/ChatKitPanel.tsx
• news-guide/backend/app/agents/news_agent.py