Modelwatch
- Updated July 27, 2026
- bch1212/modelwatch
ModelWatch is a MCP server that continuously monitors LLM app behavior for drift when providers change models silently.
About
ModelWatch is an MCP server that connects your AI coding agent to ModelWatch’s drift monitoring API so you can define behavioral specs and run checks against LLM app outputs. developers shipping agents or SaaS with model providers often discover breaking changes only from user complaints; this tool targets that gap by catching behavioral drift when providers update models quietly. Install via npm as a stdio MCP server, set MODELWATCH_API_KEY from modelwatch.app, and optionally point MODELWATCH_API_BASE at a self-hosted deployment. It fits teams who already have prompts or eval fixtures and want ongoing monitoring without building a custom regression harness. Complexity is intermediate because you need an LLM app worth monitoring and a clear definition of expected behavior. It is not a general uptime or infra monitor—it is specialized for LLM behavioral consistency.
- Detects silent provider model updates that change app behavior without version bumps
- Free tier includes 5 specs and 500 runs per month on modelwatch.app
- stdio npm package modelwatch-mcp with MODELWATCH_API_KEY and optional self-hosted API base
- Continuous behavioral drift monitoring aimed at LLM-powered products
- MCP integration your coding agent calls—not a standalone dashboard-only workflow
Modelwatch by the numbers
- Data as of Jul 28, 2026 (Skillselion catalog sync)
claude mcp add --env MODELWATCH_API_KEY=YOUR_MODELWATCH_API_KEY --env MODELWATCH_API_BASE=YOUR_MODELWATCH_API_BASE modelwatch-mcp -- npx -y modelwatch-mcpAdd your badge
Show developers this MCP server is listed on Skillselion. Paste this into your README.
| Package | modelwatch-mcp |
|---|---|
| Transport | STDIO |
| Auth | Required |
| Last updated | July 27, 2026 |
| Repository | bch1212/modelwatch ↗ |
What it does
Wire continuous behavioral checks into your agent workflow so you notice when a provider silently changes model behavior in production.
Who is it for?
Best when you're running Claude Code or Cursor against a live LLM product and want MCP-triggered drift runs on a free tier (5 specs, 500 runs/mo).
Skip if: Skip if you have no production LLM surface yet, or anyone and only needs generic server uptime rather than output-behavior baselines.
What you get
After you register specs and runs through the MCP, you get repeatable drift checks tied to your agent so regressions surface before users flood support.
- MCP-accessible drift monitoring runs against your LLM app specs
- Ongoing signal when provider-side model behavior diverges from baselines
By the numbers
- Free tier: 5 specs and 500 runs per month
- Server version 0.1.1 via npm registry
- Optional MODELWATCH_API_BASE for self-hosted API
Recommended MCP Servers
Agent Audit McpImmutable audit trail for agent-to-agent interactions2
Agentguard47Read-only MCP server for coding-agent traces, alerts, costs, usage, and budget health.4
Agentic Observability MCPAgent tracing, cost tracking, anomaly detection for LLM agents
Agent Incident Relay McpAgent Incident Relay MCP — Article 73 5-clock broadcaster.
Agent Infra Mcp18 tools: agent memory, API drift detection, and HMAC-signed trust receipts
Agentmonitorrelay McpAI agent run monitoring with incident replay and SLA receipts.How it compares
LLM behavioral drift MCP—not a generic APM skill or prompt-writing assistant.
FAQ
Who is ModelWatch for?
and small teams shipping LLM-powered apps who need continuous checks that provider model changes did not break expected behavior.
When should I use ModelWatch?
Use it in Operate when you have baseline specs for outputs and want your agent to schedule or invoke drift monitoring after deploys or on a cadence.
How do I add ModelWatch to my agent?
Install the modelwatch-mcp npm stdio package, set MODELWATCH_API_KEY from https://modelwatch.app, add the server to Claude Code or Cursor MCP config, and restart the client.