Modelwatch
Wire continuous behavioral checks into your agent workflow so you notice when a provider silently changes model behavior in production.
Overview
ModelWatch is a MCP server for the Operate phase that continuously monitors LLM app behavior for drift when providers change models silently.
What is this MCP server?
- Detects silent provider model updates that change app behavior without version bumps
- Free tier includes 5 specs and 500 runs per month on modelwatch.app
- stdio npm package modelwatch-mcp with MODELWATCH_API_KEY and optional self-hosted API base
- Continuous behavioral drift monitoring aimed at LLM-powered products
- MCP integration your coding agent calls—not a standalone dashboard-only workflow
- Free tier: 5 specs and 500 runs per month
- Server version 0.1.1 via npm registry
- Optional MODELWATCH_API_BASE for self-hosted API
What problem does it solve?
You shipped an LLM feature that worked yesterday, but today answers drift with no deploy—and you have no automated way to prove the model itself changed.
Who is it for?
Indie builders running Claude Code or Cursor against a live LLM product who want MCP-triggered drift runs on a free tier (5 specs, 500 runs/mo).
Skip if: Teams with no production LLM surface yet, or anyone who only needs generic server uptime rather than output-behavior baselines.
What do I get? / Deliverables
After you register specs and runs through the MCP, you get repeatable drift checks tied to your agent so regressions surface before users flood support.
- MCP-accessible drift monitoring runs against your LLM app specs
- Ongoing signal when provider-side model behavior diverges from baselines
Recommended MCP Servers
Journey fit
How it compares
LLM behavioral drift MCP—not a generic APM skill or prompt-writing assistant.
Common Questions / FAQ
Who is ModelWatch for?
Solo and small teams shipping LLM-powered apps who need continuous checks that provider model changes did not break expected behavior.
When should I use ModelWatch?
Use it in Operate when you have baseline specs for outputs and want your agent to schedule or invoke drift monitoring after deploys or on a cadence.
How do I add ModelWatch to my agent?
Install the modelwatch-mcp npm stdio package, set MODELWATCH_API_KEY from https://modelwatch.app, add the server to Claude Code or Cursor MCP config, and restart the client.