Project Builder

Phase 0: SKILL DISCOVERY & REQUIRED READING

⚠️ CRITICAL — UI Design Quality Gate: If the project produces ANY visual HTML output (dashboard, web app, landing page, portfolio, any page the user will see), you MUST read_file the ui-design skill's SKILL.md and follow it BEFORE writing any HTML/CSS. This is not optional. project-builder handles engineering; ui-design handles visual quality (and tells you when to reach for a component library like shadcn/ui, HeroUI, or coss ui instead of hand-writing). Skipping ui-design produces generic AI slop.

A. Pick the skills. Gather every data source the project needs. For each one, prefer a skill: check <available_skills>, and if nothing fits, try search_skills(query) for official + marketplace coverage. Skills are the most reliable layer — they ship tested clients, auth, and rate-limit handling. Web search is a last resort. Only write raw HTTP / SDK code when no skill can cover the source.

B. Read the platform rules for what the project touches. These rules live in references (not in your system prompt) so you must read_file them before writing code. Skipping this is the #1 cause of 401s, broken paths, and "worked locally, fails in preview" bugs.

---

Phase 1: DESIGN

Translate vague requests into concrete specs. If intent is ambiguous, ask ONE question.

Architecture decision tree:

Periodic alerts/reports?  → Scheduled Task
Live visual interface?    → Preview Server (dashboard)
One-time analysis?        → Inline (no build needed)
Reusable tool?            → Script in workspace

For medium+ projects, present to user BEFORE writing code: 1. Data flow — sources → processing → output 2. Architecture choice and why 3. Cost estimate — (cost/run) × frequency × 30 = monthly 4. Known limitations

UI Design Gate (required, blocking — for visual projects): If the architecture choice is Preview Server or any project that outputs HTML the user will see: 1. read_file the ui-design skill's SKILL.md now (if you haven't already in this session) and pick a track (hand-built vs component library). 2. For hand-built UI, run the Design Dials (in ui-design's references/design-process.md) to determine Surface, Accent, Typography, and Aesthetic Family. 3. Include the Design Dials output line in your phase plan below. If you skip this step, the UI will look like generic AI output. This gate is blocking — do not proceed to Phase 2 without completing it.

Design Gate (required, blocking): After Phase 1, STOP and present a short phase plan (milestones for DESIGN/BUILD/DEBUG). Ask explicitly: "Approve this plan and proceed to Phase 2 BUILD?" Match the user's language when phrasing the question — never inject a hardcoded non-English string.

• If user confirms: proceed to Phase 2.
• If user requests changes: revise design and re-confirm.
• If no confirmation: do not write/modify code.

---

Phase 1.5: SCAFFOLD (mandatory for shareable projects)

After design is confirmed, before writing any code, scaffold the project under the standard layout. This makes the project shareable via community-publish skill from day one — no migration later.

Standard project location: output/projects/{slug}/

output/projects/{slug}/
├── project.yaml          # name, version (start 0.1.0), type, description, license, entry, env_required
├── PROJECT.md            # 4 required sections: What / Required env / How to start / Outputs / Troubleshooting
├── .env.example          # every env var the code reads, with placeholder values
├── .gitignore            # at minimum: .env, *.key, *.pem, __pycache__, node_modules
└── src/                  # all code lives here, NOT scattered
    ├── run.py            # type=task — first line MUST be: # -*- task-system: v3 -*-
    ├── server.py         # type=service
    ├── main.py           # type=script
    └── index.html / app.py + frontend  # type=preview

Project type → entry mapping:

Skip scaffold only when:

• Pure inline analysis with no persistent code
• Modifying an existing output/projects/... project (keep its layout)
• User explicitly says "just throw a script in /tmp" or similar

During Phase 2 BUILD, maintain the scaffold:

• Every new env var read by code → add to .env.example in same edit
• Every behavioral change → update PROJECT.md
• Never write code outside src/ (configs, fixtures: project root or src/data/)

Why this matters: Projects already in standard layout publish in one command. Projects scattered across tasks/, output/scripts/, dashboards/, etc. need tidy_project() migration before they can be shared, and the user often doesn't want to rebuild PROJECT.md from memory.

For existing scattered code: call community-publish skill → tidy_project(any_dir) to reorganize before publishing.

---

API cost & rate limits: All external API calls go through sc-proxy, which bills per request and enforces rate limits. Before designing, read `config/context/references/sc-proxy.md` for pricing table and limits.

• Estimate cost: credits_per_request × requests_per_run × runs_per_day × 30
• Respect rate limits: e.g. CoinGecko 60 req/min — a task polling 10 coins every minute is fine; 100 coins is not
• Prefer batch endpoints over N single calls (e.g. coin_price with multiple ids vs N separate calls)
• Pure script tasks (no API): ~0 credits/run
• LLM cost warning: high-end models can exceed $0.10 per single call. Pricing varies dramatically by model tier; expensive models can be 100x+ the cost of budget models for the same workflow.
• Model-aware estimate required: break LLM cost down by model (model_price_per_call × expected_calls_per_run × runs_per_day × 30) instead of using a single generic number.
• Dashboard auto-refresh costs credits — default to manual refresh unless user asks otherwise
• Spending protection: if projected monthly LLM cost is high, explicitly ask whether to enforce per-caller limits before implementation.
• Per-caller tracking (required): every proxied request must include SC-CALLER-ID (e.g. job:{JOB_ID}, preview:{preview_id}, chat:{thread_id}) so usage can be traced and capped. Details in config/context/references/sc-proxy.md § Caller Credit Limit

Data reliability: Native tools > proxied APIs > direct requests > web scraping > LLM numbers (never). Iron rule: Scripts fetch data. LLMs analyze text. Final output = script variables + LLM prose.

Task scripts can import skill functions directly:

from core.skill_tools import coingecko, coinglass  # auto-discovers skills/*/exports.py
prices = coingecko.coin_price(coin_ids=["bitcoin"], timestamps=["now"])

Tool names = SKILL.md frontmatter tools: list. See build-patterns.md § Using Skill Functions.

---

Phase 2: BUILD

Every piece follows this cycle:

Build one small piece → Run it → Verify output → ✅ Next piece / ❌ Fix first

Verification layering:

• Critical (must pass before preview/activate): data correctness, core logic, no crashes
• Informational (can fix after delivery): styling, edge case messages, minor UX polish

Anti-patterns:

• ❌ "Done!" without running anything
• ❌ Writing 200+ lines then testing for the first time
• ❌ "It should work"

→ Detailed patterns: read `references/build-patterns.md`

Code Practices

• read_file before edit_file — understand what's there
• edit_file > write_file for modifications
• Check ls before write_file — avoid duplicating existing files
• Large files (>300 lines): split into multiple files, or skeleton-first + bash inject
• Env vars: os.environ["KEY"], persist installs to setup.sh

Dashboard UX Defaults (type=preview)

Decide sensible defaults yourself and render real data on first load. Treat filters as optional refinements users can adjust later — never as prerequisites that gate the initial view. Auto-refresh on a sensible interval. No "Click to load" / "Enter address" / "Select symbol" before anything appears.

Visual design quality (MANDATORY for all HTML output): If the ui-design skill is installed, you MUST read_file its SKILL.md and follow it before writing any HTML/CSS. project-builder owns the engineering workflow; ui-design owns the visual quality. Using project-builder alone produces functional but visually generic output.

---

Platform Rules

• Agent tools are tool calls only — not importable in scripts
• Preview paths must be relative (./path not /path)
• Hardcode the preview port in code, do not read from env. Each preview runs in its own pod and the env-port contract is not reliable across pods. Pick any free port (e.g. 8765), write it directly into the app, and pass the same number to preview(action="serve", port=...). The two must match exactly.
• Concurrent previews need different IDs. If two previews share the same dir, the newer one auto-kills the older one (same-dir replacement rule). When iterating, reuse the same id rather than inventing variants, or use distinct dirs.
• Fullstack = one port (backend serves API + static files)
• Cron times are UTC — convert from user timezone
• Preview serving & publishing → read platform reference config/context/references/preview-guide.md
• localhost APIs → read config/context/references/localhost-api.md
• Task scripts decide WHEN to invoke the agent, WHAT data/context to pass, WHICH model to use
• Pattern: script fetches data → evaluates if noteworthy → calls LLM only when needed → prints result
• LLM in scripts — two options (details in references/build-patterns.md):
• OpenRouter (via sc-proxy): lightweight, for summarize/translate/format text. Direct API call, no agent overhead.
• localhost /chat/stream: full agent with tools. Use only when LLM needs tool access.
• Data template rule: Script owns the numbers, LLM owns the words. Final output assembles data from script variables + analysis from LLM. Never let LLM output be the sole source of numbers the user sees.
• API costs & rate limits → read platform reference config/context/references/sc-proxy.md

---

Phase 3: DEBUG

CHECK LOGS → REPRODUCE → ISOLATE → DIAGNOSE → FIX → VERIFY → REGRESS

• CHECK LOGS first — task logs, preview diagnostics, stderr. If logs reveal a clear cause, skip to FIX.
• REPRODUCE only when logs are insufficient — see the failure yourself
• ISOLATE which layer is broken (data? logic? LLM? output? frontend? backend?)
• FIX the root cause, then VERIFY with the same repro steps. Don't just fix — fix and confirm.

Three-Strike Rule: Same approach fails twice → STOP → rethink → explain to user → different approach.

→ Full debug procedures: read `references/debug-handbook.md`

---

Quick Checklists

Kickoff: ☐ Clarified intent ☐ Proposed architecture ☐ Estimated cost ☐ User confirmed (required before Phase 2)

Build: ☐ Each component tested ☐ Numbers match source ☐ Errors handled ☐ Preview healthy (web)

Debug: ☐ Logs checked ☐ Reproduced (or skipped — logs sufficient) ☐ Isolated layer ☐ Root cause found ☐ Fix verified ☐ Regressions checked

Build Patterns — Detailed Reference

Pattern A: Scheduled Task (Cron Job)

Use when: User wants recurring monitoring, alerts, or reports.

Build order (each step verified before the next):

Step 1 — Data fetching (verify: real data comes back)

Preferred: Use `core.skill_tools` to call skill functions directly:

# skill_tools auto-discovers skills/*/exports.py — no manual path setup needed
from core.skill_tools import coingecko, coinglass

# Same function names as agent tools, namespaced by skill
prices = coingecko.coin_price(coin_ids=["bitcoin", "ethereum"], timestamps=["now"])
btc_price = prices[0]['price']

fr = coinglass.funding_rate(symbol="BTC")
binance_rate = [e for e in fr['exchanges_data'] if e['exchangeName'] == 'Binance'][0]['rate']

Fallback: Direct API call (for skills without exports.py):

import requests, os, json

def fetch_prices():
    resp = requests.get("https://api.twelvedata.com/price", params={
        "symbol": "CL1,BZ1",
        "apikey": os.environ.get("TWELVEDATA_API_KEY", "")
    })
    return resp.json()

if __name__ == "__main__":
    result = fetch_prices()
    print(json.dumps(result, indent=2))

Run it: python3 tasks/{id}/run.py ✅ Pass = real prices printed, recent timestamps ❌ Fail = fix before proceeding

Step 2 — Data validation (verify: bad data gets caught)

def validate_price(price_str, symbol):
    """Returns (float, error). Error is None if valid."""
    try:
        price = float(price_str)
    except (ValueError, TypeError):
        return None, f"{symbol}: invalid price '{price_str}'"
    if price <= 0 or price > 500:  # reasonable range for crude oil
        return None, f"{symbol}: price {price} out of range"
    return price, None

Test with both good and bad inputs.

Step 3 — Logic/analysis (verify: output format is correct)

def analyze(prices, thresholds):
    alerts = []
    for symbol, data in prices.items():
        change_pct = data["change_pct"]
        if abs(change_pct) > thresholds["alert_pct"]:
            alerts.append(f"⚠️ {symbol}: {change_pct:+.1f}%")
    return alerts

Test with edge cases: exactly at threshold, just above, just below.

Step 4 — Output formatting (verify: readable, correct numbers)

# Print only when there's something actionable
output = format_report(prices, alerts)
if output.strip():
    print(output)  # Non-empty stdout → auto-pushed to user
# Empty stdout → silent, no push, no cost

Run full pipeline, manually verify every number in output matches raw data.

Step 5 — Register and activate

register_task(title="WTI Monitor", schedule="0 * * * *")
# Write verified run.py to tasks/{id}/
# One final dry-run
bash("python3 tasks/{id}/run.py")
# Only then:
activate_task(job_id)

Using Skill Functions in Task Scripts (core.skill_tools)

Skills with exports.py expose their tool functions for direct use in task scripts. No manual path setup needed — core.skill_tools auto-discovers all exports.

from core.skill_tools import coingecko, coinglass, birdeye

# Same names as agent tools, namespaced by skill
prices = coingecko.coin_price(coin_ids=["bitcoin", "ethereum"], timestamps=["now"])
btc_price = prices[0]['price']

fr = coinglass.funding_rate(symbol="BTC")
binance = [e for e in fr['exchanges_data'] if e['exchangeName'] == 'Binance']
btc_funding = binance[0]['rate'] if binance else None

etf = coinglass.cg_btc_etf_flows()

How to know what's available:

• Tool names = SKILL.md frontmatter tools: list
• Import pattern: from core.skill_tools import {skill_name} → {skill_name}.{tool_name}(...)
• Functions are sync (safe for task scripts). Return types match the underlying API response.

When skill_tools is NOT available:

• Skills without exports.py (e.g. hyperliquid — needs wallet context)
• Async-only tools (twelvedata uses aiohttp)
• For these, use direct API calls via requests / core.http_client

Data Integrity for Tasks with LLM Analysis

Core principle: Script owns the numbers, LLM owns the words.

The script fetches data, the LLM analyzes/formats text. Numbers in the final output must come from script variables, NOT from LLM output. This eliminates hallucinated data entirely.

The template pattern (recommended):

# ✅ BEST: Script builds the output, LLM only provides analysis text
prices = fetch_prices()

# Ask LLM for qualitative analysis only — no numbers in its output
analysis = llm_call(f"""Given these facts:
- WTI crude is at ${prices['wti']:.2f}, changed {prices['wti_change']:+.1f}% today
- Brent crude is at ${prices['brent']:.2f}, changed {prices['brent_change']:+.1f}% today
Write a 2-sentence market analysis. Do NOT repeat the exact prices — I will insert them myself.""")

# Script assembles final output — numbers from variables, words from LLM
output = f"""🛢️ Oil Market Update
WTI: ${prices['wti']:.2f} ({prices['wti_change']:+.1f}%)
Brent: ${prices['brent']:.2f} ({prices['brent_change']:+.1f}%)

{analysis}"""
print(output)

Why this works: Even if the LLM hallucinates, the numbers the user sees (${prices['wti']:.2f}) come directly from the API response variable, not from LLM text. The LLM output is sandwiched between script-controlled data.

Fallback — when LLM must reference numbers:

If the analysis inherently needs to mention numbers (e.g. "WTI is above Brent by $X"), use post-validation:

# ✅ OK: LLM references numbers, but script validates afterward
analysis = llm_call(f"""Analyze these prices:
WTI: ${prices['wti']:.2f} | Brent: ${prices['brent']:.2f}
Use ONLY these exact numbers. Do NOT round or estimate.""")

# Post-check: verify key numbers survive
for key, val in [("WTI", prices['wti']), ("Brent", prices['brent'])]:
    if f"{val:.2f}" not in analysis:
        analysis = analysis.replace(f"${val:.0f}", f"${val:.2f}")  # fix rounding
        print(f"[WARNING] LLM may have altered {key} price", file=sys.stderr)

# ❌ WRONG: asking LLM to fetch/know prices
prompt = "What's the current WTI oil price and analyze its trend?"
# LLM will hallucinate a number — guaranteed

# ❌ WRONG: trusting LLM output for the final number display
response = call_agent("Summarize BTC price and trends")
print(response)  # Numbers in here are unverified — user sees hallucinated data

LLM Integration: Two Options

Pick the right weight class for the LLM task:

Rule of thumb: If the task just needs "read this text, produce that text" → OpenRouter. If it needs to call tools, search the web, or use agent capabilities → /chat/stream.

Option 1: OpenRouter via sc-proxy (lightweight LLM)

For simple text tasks (summarize news, translate, format reports). Uses sc-proxy, no auth needed.

import os, base64, tempfile, requests, json

def setup_proxy():
    """One-time proxy setup. Call once at script start."""
    ca_pem = base64.b64decode(os.environ["STARCHILD_API_PROXY_CA_BASE64"])
    with tempfile.NamedTemporaryFile(delete=False, suffix=".crt") as f:
        f.write(ca_pem)
        ca_path = f.name
    proxy_url = f"http://[{os.environ['STARCHILD_API_PROXY_HOST']}]:{os.environ['STARCHILD_API_PROXY_PORT']}"
    return {"http": proxy_url, "https": proxy_url}, ca_path

PROXIES, CA_BUNDLE = setup_proxy()

def llm_call(prompt, model="google/gemini-2.5-flash-lite", max_tokens=1000):
    """Lightweight LLM call via OpenRouter. No agent tools, just text in → text out."""
    resp = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        proxies=PROXIES, verify=CA_BUNDLE, timeout=30,
        headers={"Content-Type": "application/json"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens},
    )
    data = resp.json()
    if "choices" not in data:
        raise RuntimeError(f"OpenRouter error: {data}")
    return data["choices"][0]["message"]["content"]

Model selection for OpenRouter:

• Cheapest (summaries, formatting): google/gemini-2.5-flash-lite ($0.10/M)
• Balanced (analysis, longer text): google/gemini-2.5-flash ($0.30/M)
• Quality (nuanced writing): anthropic/claude-sonnet-4 (~$3/M)
• Browse all models: GET https://openrouter.ai/api/v1/models (filter by pricing)

Option 2: localhost /chat/stream (full agent)

For tasks that need agent capabilities (tool calls, web search, memory, complex reasoning):

import requests, json, sys, os

JOB_ID = os.path.basename(os.path.dirname(os.path.abspath(__file__)))

def call_agent(message):
    resp = requests.post("http://localhost:8000/chat/stream", json={
        "message": message,
        "call_source": "task",
        "internal_options": {"job_id": JOB_ID},
    }, stream=True, timeout=30)
    reply = ""
    for line in resp.iter_lines():
        if not line or not line.startswith(b"data: "): continue
        event = json.loads(line[6:])
        if event.get("type") == "agent_complete":
            reply = (event["data"].get("reply") or "").strip(); break
        elif event.get("type") == "agent_error":
            print(event["data"].get("error"), file=sys.stderr); sys.exit(1)
    return reply

---

Pattern B: Dashboard / Web UI

Use when: User wants a visual interface to view data or interact with something.

Build order:

Step 1 — Backend data endpoint (verify: returns correct JSON)

# app.py — start with just one endpoint
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
import requests, os

app = FastAPI()

@app.get("/api/data")
def get_data():
    resp = requests.get("https://api.example.com/data", ...)
    return resp.json()

Test: python3 app.py & then curl localhost:8000/api/data ✅ Pass = valid JSON response

Step 2 — Minimal frontend (verify: renders in preview)

<!-- index.html — just enough to prove it connects -->
<div id="app">Loading...</div>
<script>
fetch('api/data')
  .then(r => r.json())
  .then(data => {
    document.getElementById('app').textContent = JSON.stringify(data);
  });
</script>

Mount in FastAPI: app.mount("/", StaticFiles(directory="static", html=True)) preview_serve(dir="project", command="python3 app.py", port=8000) ✅ Pass = health_check.ok is true AND data shows in Preview panel

Step 3 — Polish frontend (verify: looks right, data correct) Add styling, charts, layout. Verify after each significant change.

Step 4 — Error handling and edge cases What happens when API is down? Empty data? Add graceful fallbacks.

Dashboard Rules

• Frontend calls backend at relative paths: fetch('api/data') not fetch('/api/data')
• External APIs go through backend proxy endpoints, never called from frontend JS (CORS blocks it)
• Auto-refresh costs credits — default to manual refresh, let user opt into auto-refresh
• Listen on 127.0.0.1 only, never 0.0.0.0
• Fullstack = one port: backend serves API + static files

Visualization Library Selection

Simple charts (line/bar/pie)?  → Chart.js (default choice, lightweight, easy)
Beautiful real-time dashboard? → ApexCharts (smooth animations, better aesthetics)
Custom/unique visualizations?  → D3.js (maximum flexibility, steep learning curve)
Unsure?                        → Chart.js

Real-time Data Strategy

Server → client only (metrics, live data)?  → SSE (simpler, auto-reconnects)
Bidirectional (chat, collaborative)?        → WebSocket
Simple, low frequency, max compatibility?   → Polling (setInterval + fetch)

• SSE is the default for dashboards — simpler than WebSocket, auto-reconnects, works over HTTP/3
• Polling is the fallback when SSE/WebSocket aren't available
• Always implement fallback: SSE → polling on error

Error Handling & Loading States

Every dashboard should handle: loading, success, error, and empty states.

// Pattern: fetch with loading/error states
async function loadData() {
  showLoading();
  try {
    const resp = await fetch('api/data');
    if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
    const data = await resp.json();
    if (!data || data.length === 0) {
      showEmpty("No data available");
    } else {
      renderDashboard(data);
    }
  } catch (err) {
    showError(`Failed to load: ${err.message}`);
  }
}

Dashboard Design Quick Reference

• Visual hierarchy: critical data top-left → secondary top-right → charts center → tables bottom
• Responsive: mobile-first (320px → 768px → 1024px), use CSS Grid or Flexbox
• Dark mode: use CSS variables + prefers-color-scheme media query
• Accessibility: color-blind safe palette, ARIA labels, keyboard nav, min 44×44px touch targets
• Performance: debounce resize, limit data points (decimateData), lazy load off-screen charts

For complete code examples (Chart.js, ApexCharts, D3.js, SSE/WebSocket/Polling, responsive layouts, dark mode, accessibility patterns, multi-source dashboards, caching): → Read references/dashboard-examples.md

---

Pattern C: One-off Script / Tool

Build order: 1. Write the script 2. Run it with test inputs 3. Verify output 4. Deliver to user

Simple scripts = single build loop cycle. Just don't skip "run and verify."

---

Pattern D: Complex Multi-Component System

For projects with 3+ interacting components:

1. Draw the data flow — write it down as a numbered list, show the user 2. Build each component in isolation — test with hardcoded inputs 3. Connect components one at a time — verify each connection 4. End-to-end test — run the whole system, verify final output

Never build all components at once. The debugging cost of finding a bug in a 5-component system is 5× higher than in a 1-component system.

Dashboard Examples — Code Reference

Read this file when building dashboards and you need concrete code templates.

Chart.js — Basic Usage

<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<canvas id="myChart"></canvas>
<script>
const chart = new Chart(document.getElementById('myChart').getContext('2d'), {
  type: 'line',  // line, bar, pie, doughnut, radar, polarArea
  data: {
    labels: ['Jan', 'Feb', 'Mar', 'Apr', 'May'],
    datasets: [{
      label: 'Revenue',
      data: [12, 19, 3, 5, 2],
      borderColor: 'rgb(75, 192, 192)',
      backgroundColor: 'rgba(75, 192, 192, 0.1)',
      tension: 0.1,
      fill: true
    }]
  },
  options: {
    responsive: true,
    maintainAspectRatio: false,
    plugins: { legend: { display: false } },
    scales: { y: { beginAtZero: true } }
  }
});

// Update dynamically
function updateChart(newData) {
  chart.data.datasets[0].data = newData;
  chart.update();
}
</script>

ApexCharts — Real-time Area Chart

<script src="https://cdn.jsdelivr.net/npm/apexcharts"></script>
<div id="chart"></div>
<script>
const options = {
  chart: {
    type: 'area', height: 350,
    animations: { enabled: true, dynamicAnimation: { speed: 1000 } }
  },
  series: [{ name: 'Value', data: [31, 40, 28, 51, 42, 109, 100] }],
  xaxis: { categories: ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul'] }
};
const chart = new ApexCharts(document.querySelector("#chart"), options);
chart.render();

// Append data in real-time
function addDataPoint(value) {
  chart.appendData([{ data: [value] }]);
}
</script>

D3.js — Custom Bar Chart

<script src="https://d3js.org/d3.v7.min.js"></script>
<svg id="chart" width="600" height="400"></svg>
<script>
const data = [30, 86, 168, 281, 303, 365];
const svg = d3.select("#chart");
const margin = {top: 20, right: 20, bottom: 30, left: 40};
const width = 600 - margin.left - margin.right;
const height = 400 - margin.top - margin.bottom;

const x = d3.scaleBand().domain(d3.range(data.length)).range([0, width]).padding(0.1);
const y = d3.scaleLinear().domain([0, d3.max(data)]).range([height, 0]);

const g = svg.append("g").attr("transform", `translate(${margin.left},${margin.top})`);
g.selectAll(".bar").data(data).join("rect")
  .attr("x", (d, i) => x(i)).attr("y", d => y(d))
  .attr("width", x.bandwidth()).attr("height", d => height - y(d))
  .attr("fill", "steelblue");
</script>

SSE (Server-Sent Events) — Client + Server

Client:

const eventSource = new EventSource('api/stream');
eventSource.addEventListener('update', (event) => {
  const data = JSON.parse(event.data);
  updateDashboard(data);
});
eventSource.onerror = () => {
  console.log('SSE failed, falling back to polling');
  eventSource.close();
  setInterval(() => fetch('api/data').then(r => r.json()).then(updateDashboard), 10000);
};

Server (FastAPI):

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio, json

app = FastAPI()

async def event_stream():
    while True:
        data = get_metrics()  # your data source
        yield f"event: update\ndata: {json.dumps(data)}\n\n"
        await asyncio.sleep(5)

@app.get("/api/stream")
async def stream():
    return StreamingResponse(event_stream(), media_type="text/event-stream")

WebSocket — Client

const ws = new WebSocket('wss://api.example.com/ws');
ws.onopen = () => ws.send(JSON.stringify({ type: 'subscribe', channel: 'metrics' }));
ws.onmessage = (event) => updateDashboard(JSON.parse(event.data));
ws.onclose = () => setTimeout(connectWebSocket, 5000);  // auto-reconnect

Responsive Grid Layout

<div class="grid">
  <!-- Top: Key metrics row -->
  <div class="card full">
    <div class="metrics-row">
      <div class="metric">
        <span class="metric-label">Total Revenue</span>
        <span class="metric-value">$142,592</span>
        <span class="metric-change positive">+12.5%</span>
      </div>
    </div>
  </div>
  <!-- Middle: Charts -->
  <div class="card wide"><canvas id="revenueChart"></canvas></div>
  <div class="card"><canvas id="usersChart"></canvas></div>
  <!-- Bottom: Table -->
  <div class="card full"><table class="data-table">...</table></div>
</div>

/* Mobile-first responsive */
.metrics-row { display: flex; flex-direction: column; gap: 1rem; }
@media (min-width: 768px) {
  .metrics-row { flex-direction: row; flex-wrap: wrap; }
  .metric { flex: 1 1 calc(50% - 0.5rem); }
}
@media (min-width: 1024px) {
  .metric { flex: 1 1 calc(25% - 0.75rem); }
}
.button { min-height: 44px; min-width: 44px; }  /* Touch-friendly */

Dark Mode

:root {
  --bg-primary: #ffffff; --bg-secondary: #f8f9fa;
  --text-primary: #212529; --text-secondary: #6c757d;
  --border-color: #dee2e6;
}
@media (prefers-color-scheme: dark) {
  :root {
    --bg-primary: #1a1a1a; --bg-secondary: #2d2d2d;
    --text-primary: #e9ecef; --text-secondary: #adb5bd;
    --border-color: #495057;
  }
}
body { background: var(--bg-primary); color: var(--text-primary); }
.card { background: var(--bg-secondary); border: 1px solid var(--border-color); }

Performance Patterns

// Debounce resize
let t; window.addEventListener('resize', () => { clearTimeout(t); t = setTimeout(() => chart.resize(), 250); });

// Limit data points
function decimateData(data, max = 100) {
  if (data.length <= max) return data;
  const step = Math.ceil(data.length / max);
  return data.filter((_, i) => i % step === 0);
}

// Lazy load charts (Intersection Observer)
const observer = new IntersectionObserver((entries) => {
  entries.forEach(e => { if (e.isIntersecting) { initChart(e.target); observer.unobserve(e.target); } });
});
document.querySelectorAll('.chart-container').forEach(el => observer.observe(el));

Accessibility

<div class="chart-container" role="img" aria-label="Revenue trend showing 12% growth">
  <canvas id="chart"></canvas>
  <table class="sr-only"><caption>Revenue Data</caption>
    <thead><tr><th>Month</th><th>Revenue</th></tr></thead>
    <tbody><tr><td>January</td><td>$12,000</td></tr></tbody>
  </table>
</div>

.sr-only { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0,0,0,0); }

Color-blind safe palette: #0066CC (blue), #28A745 (green), #FFC107 (amber), #DC3545 (red), #17A2B8 (cyan)

Multi-source Data Fetching

async function fetchAllData() {
  const [crypto, stocks, weather] = await Promise.all([
    fetch('api/crypto').then(r => r.json()),
    fetch('api/stocks').then(r => r.json()),
    fetch('api/weather').then(r => r.json()),
  ]);
  return { crypto, stocks, weather, timestamp: new Date().toISOString() };
}

Caching Pattern

const cache = new Map();
async function cachedFetch(url, ttlMs = 60000) {
  const cached = cache.get(url);
  if (cached && Date.now() - cached.time < ttlMs) return cached.data;
  const data = await fetch(url).then(r => r.json());
  cache.set(url, { data, time: Date.now() });
  return data;
}

Debug Handbook — Systematic Diagnosis

When something goes wrong, follow a system. Don't guess.

The Debug Protocol

1. CHECK LOGS → Read existing logs/errors first — they often reveal the cause directly
2. REPRODUCE  → See the actual failure (not just the user's description)
3. ISOLATE    → Which layer/component is broken?
4. DIAGNOSE   → What specifically is wrong?
5. FIX        → Targeted fix for the root cause
6. VERIFY     → Prove the fix works with the same reproduction steps
7. REGRESS    → Make sure the fix didn't break something else

Step 1 — Check logs first:

• Tasks: get_scheduled_task_log(job_id) — full stdout + stderr
• Previews: preview_check(preview_id) + check process output
• Scripts: read stderr from last run
• If logs reveal a clear root cause (e.g. traceback, API 401, missing env var), skip straight to FIX — no need to reproduce what you can already see.

Rules:

• Always check logs before attempting reproduction.
• Never fix without diagnosing. "I added some error handling" is not a fix.
• Never declare fixed without step 6. "Should work now" is not verification.
• If the same fix fails twice, STOP. The diagnosis is wrong — go back to step 3.

---

Debugging Scheduled Tasks

Step 1: See what actually happened

get_scheduled_task_log(job_id="xxx")    # Full stdout + stderr of last run
list_scheduled_tasks()                   # Status, next run, last result

Step 2: Isolate the layer

Run the script manually with diagnostic output:

python3 tasks/{id}/run.py 2>&1

The task pipeline has these layers — check them in order:

Layer 1: DATA FETCH   → Did the API call succeed? Did it return valid data?
Layer 2: VALIDATION   → Did validation catch bad data, or let it through?
Layer 3: PROCESSING   → Did analysis/formatting produce correct output?
Layer 4: LLM (if any) → Did the LLM hallucinate numbers or ignore injected data?
Layer 5: OUTPUT       → Is the final print correct? Formatting issues?

Layer-by-layer diagnosis:

# Add temporary debug prints to isolate:
data = fetch_prices()
print(f"[DEBUG L1] Raw API response: {json.dumps(data)}", file=sys.stderr)

validated = validate(data)
print(f"[DEBUG L2] After validation: {validated}", file=sys.stderr)

result = process(validated)
print(f"[DEBUG L3] After processing: {result}", file=sys.stderr)

# If LLM is involved:
llm_response = call_agent(prompt)
print(f"[DEBUG L4] LLM response: {llm_response[:200]}", file=sys.stderr)
# Compare numbers in llm_response with validated data

output = format_output(result)
print(f"[DEBUG L5] Final output: {output}", file=sys.stderr)

Common task issues:

---

Debugging Previews / Dashboards

Step 1: Check the preview service

cat /data/previews.json                        # Is it running?
curl -s -o /dev/null -w "%{http_code}" http://localhost:{port}  # Is it responding?
preview_check(preview_id="xxx")                # Detailed diagnostics

Step 2: Isolate frontend vs backend

# Backend OK?
curl -s http://localhost:{port}/api/data | head -50

# Frontend OK?
curl -s http://localhost:{port}/ | head -30

Common preview issues:

Fix in place: Use edit_file to fix the specific bug. Don't create new files. Then preview_stop + preview_serve to restart.

---

Debugging Scripts

# Run with full error output
python3 scripts/my_script.py 2>&1

# Module issue?
pip list | grep <package>

# Env var issue?
echo $RELEVANT_VAR

# Network issue?
curl -v "https://api.example.com/endpoint" 2>&1 | tail -20

---

The Three-Strike Rule

If the same approach fails twice with similar errors:

1. STOP editing the same code 2. Re-read the error — the full traceback, not just the last line 3. Print the actual API/data response — don't assume what it returns 4. Try a fundamentally different approach — different API, different logic, different architecture 5. Tell the user what you've tried, what failed, and what you're trying next

❌ Bad:  Fix → fail → tweak same fix → fail → tweak again → fail
✅ Good: Fix → fail → fix again → fail → STOP → rethink → explain to user → new approach

---

When the User Reports a Problem

Always: 1. Reproduce the issue yourself first (run the script, curl the endpoint, check the preview) 2. Show the user what you found: "I ran the script and got X — here's what's wrong" 3. Fix with evidence, not hope 4. Verify by running the same reproduction after the fix

Never:

• "I've fixed it!" (without running the fix)
• "Try again now" (without verifying yourself first)
• "That shouldn't happen" (something happened — investigate)
• Blame the user's input/environment before checking your own code