Use Railway

Use Railway

Railway resource model

Railway organizes infrastructure in a hierarchy:

• Workspace is the billing and team scope. A user belongs to one or more workspaces.
• Project is a collection of services under one workspace. It maps to one deployable unit of work.
• Environment is an isolated configuration plane inside a project (for example, production, staging). Each environment has its own variables, config, and deployment history.
• Service is a single deployable unit inside a project. It can be an app from a repo, a Docker image, or a managed database.
• Bucket is an S3-compatible object storage resource inside a project. Buckets are created at the project level and deployed to environments. Each bucket has credentials (endpoint, access key, secret key) for S3-compatible access.
• Deployment is a point-in-time release of a service in an environment. It has build logs, runtime logs, and a status lifecycle.

Most CLI commands operate on the linked project/environment/service context. Use railway status --json to see the context, and --project, --environment, --service flags to override.

Tool routing

Railway has three agent-facing operation paths. Choose the path that matches the job:

• Remote MCP (https://mcp.railway.com): account/project/service discovery, deployment state, bounded logs, simple redeploys, simple project creation, or complex Railway workflows that can be handed to railway-agent. Remote MCP uses Railway OAuth and does not depend on local CLI state.
• Local CLI MCP (railway mcp): CLI-backed platform operations such as variables, domains, service config, templates, metrics, HTTP summaries, buckets, volumes, docs, or deploy-from-directory.
• Railway CLI (railway): workflows that depend on local machine state such as current working directory deploys, railway up, railway run, SSH, database analysis scripts, local linking, interactive setup, or exact command output.

If multiple paths are available, choose the one that preserves the needed context. Remote MCP fits OAuth-scoped platform operations that do not need local files or CLI state. Local CLI MCP or the CLI fit workflows that need the current repo, local credentials, SSH, database scripts, or commands not exposed by remote MCP.

Use scripts/railway-api.sh only when neither MCP nor CLI exposes the operation, or when a reference gives a specific GraphQL fallback.

Parsing Railway URLs

Users often paste Railway dashboard URLs. Extract IDs before doing anything else:

https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>?environmentId=<ENV_ID>
https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>

The URL always contains projectId and serviceId. It may contain environmentId as a query parameter. If the environment ID is missing and the user specifies an environment by name (e.g., "production"), resolve it:

scripts/railway-api.sh \
  'query getProject($id: String!) {
    project(id: $id) {
      environments { edges { node { id name } } }
    }
  }' \
  '{"id": "<PROJECT_ID>"}'

Match the environment name (case-insensitive) to get the environmentId.

Prefer passing explicit IDs to CLI commands (--project, --environment, --service) and scripts (--project-id, --environment-id, --service-id) instead of running railway link. This avoids modifying global state and is faster.

Intent-based routing

Route by user intent before running preflight checks. The preflight ceremony below is for diagnostic and configuration work — it adds friction when the user just wants to ship something or sign up.

Deploy-from-cwd intent ("deploy", "ship", "push to Railway", "deploy this app"):

• Skip the railway whoami / railway status preflights.
• Run railway up directly — it self-validates auth, signs the user in (the CLI opens a browser) if they're unauthenticated, and chains into project + service creation and deploy.
• Announce intent before invoking: "Running `railway up` — it'll sign you in if needed and deploy this directory."
• Do NOT ask the user to run `railway login` first. The chain handles auth as part of the deploy.
• If the environment can't open a browser, the CLI prints a device-code sign-in link and waits — follow Device-code sign-in: relay the link immediately (run in background, relay the link to the user the moment it prints).

Signup intent ("sign me up", "create my Railway account", "register me", "get me on Railway"):

• If the current directory has a deployable app (e.g. `package.json`, `requirements.txt`, `go.mod`, `Dockerfile`, source to build), run `railway up` — it signs the user up and deploys in one shot, landing them on a running app. A detected agent harness authorizes the project creation, so bare `railway up` is enough — there's no extra prompt to clear. Use it even when the user only said "sign me up": shipping their app is the goal, so don't make them pick a command and don't drop to a bare login. For scripted or agent runs, railway up -y is the robust form — it skips prompts and forces the create non-interactively even if harness detection misses. railway login is NOT the default for signup when there's something to deploy.
• Only when there is nothing to deploy — an empty / non-app directory, or the user explicitly says they just want an account with no deploy — use railway login (creates new accounts on the fly through the same OAuth surface). There is no separate signup command.
• Signup is the flow most likely to hit the device-code wait (brand-new users in sandboxed/headless agent environments). Follow Device-code sign-in: relay the link immediately — a signup lost to an expired code is a lost user, not a retry.

Sandbox / remote-build intent ("give me a sandbox", "spin up a scratch environment", "build this remotely", "run this remotely", "checkpoint/snapshot the sandbox", "save this sandbox state", "restore my sandbox"):

• Load sandbox.md and follow it. Sandboxes require the feature to be enabled in Priority Boarding — if a sandbox command fails with a feature-availability error, prompt the user to enable Sandboxes in Priority Boarding rather than retrying.

Other intents (querying state, listing projects, configuring variables, debugging failures):

• Follow the Preflight section below.

Preflight

Before any mutation, verify the tool path and context:

command -v railway                # CLI installed
RAILWAY_CALLER="skill:use-railway@1.3.0" RAILWAY_AGENT_SESSION="railway-skill-$(date +%s)-$$" railway whoami --json
railway --version                 # check CLI version

Exception: railway up and railway login self-validate auth and run their own unauth-aware flows. Don't run railway whoami before them — it adds a redundant failing call without changing what you do next. See Account creation & sign-in.

Skill freshness

Once per session, alongside the preflight checks, verify the installed Railway agent tooling is current. Root help ends with an Agent tooling: health section (printed to stderr, CLI 5.9+; older CLIs print nothing — skip this check rather than retrying):

railway --help 2>&1 | grep -A4 "Agent tooling:"

Act on what it reports:

• Skills installed and up to date — continue; nothing to do.
• An update is available (rev <a> → <b> available) — run railway skills update, then tell the user to restart their coding tool so the updated skills load. This session keeps running on the old skill revision until restart: finish the current request, and if this skill's guidance disagrees with the updated CLI's own output, trust the CLI.
• Skills or MCP server missing (✗ lines) — run railway setup agent -y, then tell the user to restart their coding tool.

Check once per session and don't re-run it after acting; the restart prompt to the user is the resolution, not another check.

When Railway MCP is available and the job is a platform-state read, use the matching MCP read instead of shelling out. If using the CLI path, run the CLI checks above.

For Railway CLI calls made while this skill is active, prefix the command with RAILWAY_CALLER=skill:use-railway@1.3.0 and a stable RAILWAY_AGENT_SESSION reused for the current user request. Generate the session id once per user request, then reuse that exact value for later Railway CLI calls in the same workflow. Do not run a separate export preflight solely for telemetry; inline env prefixes keep the shell output concise and avoid leaking setup steps into every response.

Context resolution - URL IDs always win:

• If the user provides a Railway URL, extract IDs from it. Do NOT run railway status --json; it returns the locally linked project, which is usually unrelated.
• If no URL is given, fall back to railway status --json for the linked project/environment/service.
• When using MCP tools after resolving local context with railway status --json, pass the resolved project, environment, and service IDs explicitly. Do not rely on MCP implicit linked context; MCP may not share the CLI's current working directory link.

If the CLI is missing, guide the user to install it.

bash <(curl -fsSL https://railway.com/install.sh) --agents -y # Install CLI and configure detected agents
bash <(curl -fsSL https://railway.com/install.sh) # Shell script (macOS, Linux, Windows via WSL)
npm i -g @railway/cli # npm (macOS, Linux, Windows). Requires Node.js version 16 or higher.
brew install railway # Homebrew (macOS)

If not authenticated, see Account creation & sign-in below — the CLI offers unauthed railway up (deploy + sign up/in in one shot) or railway login (sign up/in only; new accounts created on the fly). If not linked and no URL was provided, run railway link --project <id-or-name>.

If a command is not recognized (for example, railway environment edit), the CLI may be outdated. Upgrade with:

railway upgrade

Account creation & sign-in

Railway uses a single unified OAuth flow for both sign-in and sign-up. The backend detects fresh accounts from durable compliance state (a CLI client that hasn't accepted ToS / Fair Use yet) and adapts the consent screen and post-auth landing page — new users land on a "Welcome to Railway!" page, existing users see the standard confirmation. The CLI does not declare signup intent up front.

Two commands surface this flow, depending on intent:

Related: railway up --new creates a fresh project + service from the current directory and deploys it even if one is already linked (use when already signed in and the user wants a new app); --name <name> overrides the project name.

Choosing the path:

• Deploy from cwd → run railway up (interactive) or railway up -y (skips the confirm prompt). Run it yourself; don't ask the user to sign in separately first.
• New project from cwd when already signed in → railway up --new.
• Sign up with a deployable app in cwd → `railway up` (signs up and deploys — bare up works for a detected agent, even if the user only said "sign me up"; add -y to skip prompts / force it non-interactively). Sign in, or sign up with nothing to deploy → railway login (creates new accounts on the fly).

Headless / no browser:

The CLI auto-detects SSH sessions, CI, and a missing DISPLAY and switches to the device-code flow on its own — you almost never need to force it.

Do NOT pass `--browserless` just because you are an agent or your shell is non-interactive. If the human is at this machine (a local IDE or desktop session — the common case), bare railway login opens their browser directly, which completes far more reliably than relaying a device code (~90% vs ~60% success for agent-driven sign-ins). Being a coding agent does not make the machine headless.

railway login --browserless   # ONLY for machines with genuinely no browser

Forces the device-code flow (RFC 8628): prints a sign-in link and a short code for the user to open on any device. Reserve it for machines where no browser exists — SSH boxes, containers, remote VMs the auto-detection missed. When you do end up in a device-code flow, follow the relay procedure below: surface the sign-in link to the user the moment it prints.

Agent harness, human present: when the CLI detects an agent harness (Claude Code, Cursor, Codex, …) with a human at the keyboard, railway up opens the browser and skips the confirm prompt — the agent invocation is treated as consent. A real human still has to complete OAuth in the browser.

Device-code sign-in: relay the link immediately (CRITICAL):

When the CLI can't open a browser (sandboxed shell, container, SSH, no DISPLAY), unauthed railway up and railway login print a sign-in URL + short code and then block, polling for up to 10 minutes while the user completes sign-in. The code expires after 10 minutes. If you run this as a normal foreground command, your harness buffers the output until the command exits — the user never sees the link until the code is already dead. This is the #1 cause of failed agent-driven signups. Handle it like this:

1. Preferred — background execution (e.g. Claude Code: run_in_background, then poll with BashOutput):

• Start the command in the background.
• Poll its output. The instant a sign-in block appears (Sign in with one click: <url> on newer CLIs, or Sign in at: <url> / Enter this code: <code> on older ones), stop everything and relay it to the user verbatim — do not summarize, shorten, or defer it. Prefer the one-click URL when present; otherwise relay the URL and code together. Tell the user to open the link now.
• Leave the command running and keep polling. When the user completes sign-in, the same process picks up the session and continues into the deploy on its own. Then verify per the deploy rules below.

2. No background support — set expectations, use the longest timeout:

• Before running, tell the user: "This will print a sign-in link — I'll show it to you the moment I have it. Please complete it promptly; the code expires in 10 minutes."
• Run with the longest timeout your harness allows.
• If the command times out or is killed before sign-in completed, the printed code is no longer being monitored — a late click does nothing. Relay whatever link appeared anyway for context, then immediately re-run the command and relay the new link, telling the user to always use the newest one.

3. Never wait silently for the command to finish before showing the link, and never report the sign-in as failed without first relaying the link and giving the user a chance to act.

The browser transport needs none of this — the CLI opens the browser on the user's machine itself.

JSON / CI modes do not auto-prompt: railway up --json and railway up --ci will NOT open a browser for an unauthed user. --json emits a structured error instead:

{"error":"Not signed in.","code":"NOT_AUTHENTICATED","hint":"Run `railway login` to authenticate, then re-run."}

When you see code: NOT_AUTHENTICATED, authenticate the user with railway login, then retry the original command.

Fully unattended (no human at all): set RAILWAY_API_TOKEN (account-scoped) or RAILWAY_TOKEN (project-scoped) instead of running an interactive login. A brand-new user with no token and no human present cannot complete signup — there is no headless account-creation path.

Agent tooling

Use direct Railway CLI commands for deterministic operations. Use railway agent only when the user explicitly asks for Railway Agent, wants a natural-language investigation, or the task is broader than a single resource operation.

Set up Railway skills, MCP, and authentication with:

railway setup agent
railway setup agent -y
railway setup agent --remote

railway setup agent -y skips the interactive login flow. If the user isn't authenticated after setup, run railway login.

Install or update MCP and skills directly when the user names a target tool:

railway mcp install
railway mcp install --agent codex
railway mcp install --agent cursor --remote
railway skills
railway skills update --agent codex
railway skills remove --agent cursor

Supported targets include claude-code, cursor, codex, opencode, copilot, and factory-droid. The --remote flag configures https://mcp.railway.com instead of a local railway mcp stdio server.

Use Railway Agent chat with:

railway agent
railway agent -p "why is my service crashing?"
railway agent -p "summarize the deployment status" --json
railway agent --list --json
railway agent --thread-id <thread-id>

railway agent requires user OAuth authentication from railway login. Project tokens (RAILWAY_TOKEN) are not supported for Railway Agent chat. If an agent command is unavailable, upgrade with railway upgrade --yes.

Common quick operations

These are frequent enough to handle without loading a reference. Use the matching MCP tool when the job is platform-scoped and the tool is available; otherwise use the CLI:

railway status --json                                    # current context
railway whoami --json                                    # auth and workspace info
railway project list --json                              # list projects
railway service list --json                              # services in current environment (verify before retrying `add`)
railway add --database <type> --json                     # add one database; ALWAYS pass --json
railway add --service <name> --json                      # add empty service; ALWAYS pass --json
railway variable list --service <svc> --json             # list variables
railway variable set KEY=value --service <svc>           # set a variable
railway logs --service <svc> --lines 200 --json          # recent logs
railway metrics --service <svc> --since 1h --json        # resource and HTTP metrics summary
railway up --detach -m "<summary>"                       # deploy current directory (returns at QUEUED — verify before reporting)
railway deployment list --json                           # poll newest deployment status after a detached up
railway bucket list --json                               # list buckets in current environment
railway bucket info --bucket <name> --json               # bucket storage and object count
railway bucket credentials --bucket <name> --json        # S3-compatible credentials

Routing

For anything beyond quick operations, load the reference that matches the user's intent. Load only what you need, one reference is usually enough, two at most.

If the request spans two areas (for example, "deploy and then check if it's healthy"), load both references and compose one response.

Execution rules

1. Use Railway MCP for platform operations that match an available MCP tool. 2. Use the local CLI for workflows that need the current repo, local shell, SSH, database scripts, or unsupported MCP coverage. 3. Fall back to scripts/railway-api.sh for operations neither MCP nor CLI exposes. 4. Use --json output where available for reliable parsing. 5. Resolve context before mutation. Know which project, environment, and service you're acting on. 6. For destructive actions (delete service, remove deployment, drop database), confirm intent and state impact before executing. 7. After mutations, verify the result with a read-back command or MCP read. 8. Never report a deploy as successful without observing a terminal SUCCESS. railway up --detach returning (it prints "Build queued") and a streaming railway up cut off by a shell timeout only confirm the build started. Poll railway deployment list --json until the newest deployment's status is SUCCESS (report deployed), or FAILED/CRASHED (triage per operate.md — do not claim success). A streaming up that exits on its own is authoritative: exit 0 = deployed, exit 1 = failed.

User-only commands (NEVER execute directly)

These commands modify database state and require the user to run them directly in their terminal. Do NOT execute these with Bash. Instead, show the command and ask the user to run it.

When these operations are needed: 1. Explain what the command does and any side effects (e.g., restart required) 2. Show the exact command the user must run 3. Wait for user confirmation that they ran it 4. Verify the result with a read-only query

Composition patterns

Multi-step workflows follow natural chains:

• Add object storage: setup (create bucket), setup (get credentials), configure (set S3 variables on app service)
• First deploy: setup (create project + service), configure (set variables and source), deploy, operate (verify healthy)
• Fix a failure: operate (triage logs), configure (fix config/variables), deploy (redeploy), operate (verify recovery)
• Add a domain: configure (add domain + set port), operate (verify DNS and service health)
• Docs to action: request (fetch docs answer), route to the relevant operational reference

When composing, return one unified response covering all steps. Don't ask the user to invoke each step separately.

Setup decision flow

When the user wants to create or deploy something, determine the right action from current context:

1. If the intent is deploy-from-cwd or signup-from-cwd, skip railway whoami and run railway up (or railway up -y) directly per Intent-based routing — it handles signup, project creation, service creation, and deploy in one chain. For other setup flows that need workspace/account context first, run railway whoami --json; if it fails with an auth error the user has no token — route through Account creation & sign-in. 2. Run railway status --json in the current directory. 3. If linked: add a service to the existing project (railway add --service <name>). Do not create a new project unless the user explicitly says "new project" or "separate project". 4. If not linked: check the parent directory (cd .. && railway status --json).

• Parent linked: this is likely a monorepo sub-app. Add a service and set rootDirectory to the sub-app path.
• Parent not linked: run railway list --json and look for a project matching the directory name.
• Match found: link to it (railway link --project <name>).
• No match: create a new project (railway init --name <name>).

5. When multiple workspaces exist, match by name from railway whoami --json.

Naming heuristic: app names like "flappy-bird" or "my-api" are service names, not project names. Use the directory or repo name for the project.

Response format

For all operational responses, return: 1. What was done (action and scope). 2. The result (IDs, status, key output). 3. What to do next (or confirmation that the task is complete).

Keep output concise. Include command evidence only when it helps the user understand what happened.

MongoDB Analysis

This reference covers MongoDB-specific metrics, tuning, and analysis guidance. For common analysis patterns (output structure, collection status handling, performance thinking), see analyze-db.md.

What the Script Collects

Via SSH (mongosh):

• Server Status: version, storage engine, uptime, connections, opcounters, latency, memory, network, WiredTiger cache/checkpoint/tickets, global lock queues, document operations, query efficiency, cursors, TTL, asserts
• DB Stats: dataSize, storageSize, indexSize, object count, collection count
• Collection Stats: per-collection document count, size, storage size, index size, index count
• Current Operations: active ops with type, namespace, duration
• Slow Queries: from system.profile (if profiling enabled) — op, namespace, duration, plan summary
• Replication Info: oplog size, usage, time window
• Top Collections: per-collection read/write counts and time from top admin command

Via Railway API: Same infrastructure metrics.

MongoDB Performance Patterns

WiredTiger Cache Pressure Pattern:

• Cache usage > 80% + app thread evictions > 0 = cache too small for working set
• Dirty cache > 20% of total = checkpoint falling behind, writes accumulating
• Read/write tickets depleted = operations queueing at storage engine level
• Fix: increase service RAM (WiredTiger uses ~50% of available RAM for cache)

Query Efficiency Pattern:

• scannedObjects >> docsReturned = collection scans, missing indexes
• Plan cache misses >> hits = frequent query re-planning, add indexes
• Sort spill to disk > 0 = sorts exceeding 100MB memory limit, needs index

Connection Saturation Pattern:

• connectionsCurrent approaching connectionsAvailable = connection pool exhaustion
• Many active ops with high microsecs_running = slow queries holding connections
• Queued readers/writers > 0 = global lock contention

Oplog Pressure Pattern:

• Oplog usage > 80% = replication window shrinking
• High write rate + small oplog = replicas may fall out of sync
• timeDiffHours < 1 on busy systems = risk of replica resync

MongoDB Thresholds

Infrastructure (7d + 24h)

Show both windows side by side to compare trends:

7-Day Trends

Last 24 Hours

Compare windows to distinguish sustained vs transient trends.

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See analyze-db.md autoscale rules.

MongoDB Autoscale Note

See analyze-db.md for full autoscale rules. For MongoDB specifically:

• WiredTiger uses ~50% of available RAM for cache by default. As Railway auto-scales the container, the cache ceiling grows automatically.
• Do NOT recommend limiting WiredTiger cache to a fraction of the Railway memory limit — the limit is the autoscale ceiling, not fixed allocation.
• If cache usage is consistently >80%, this indicates working set pressure — note it but do not tell the user to increase RAM manually.

Validated against

• MongoDB serverStatus, db.stats(), system.profile, top admin command

MySQL Analysis

This reference covers MySQL-specific metrics, tuning, and analysis guidance. For common analysis patterns (output structure, collection status handling, performance thinking), see analyze-db.md.

What the Script Collects

Via SSH (mysql -B):

• SHOW GLOBAL STATUS: threads connected/running, max used connections, query counts (select/insert/update/delete), InnoDB buffer pool stats, row lock waits/time, bytes sent/received, temp table stats, handler stats, table lock stats, aborted clients/connects
• SHOW VARIABLES: max_connections, innodb_buffer_pool_size, long_query_time, version
• Table Sizes: per-table rows, data length, index length, total size (top 15)
• Processlist: active queries with user, database, command, time, state
• Top Queries (performance_schema): digest text, call count, avg/total latency, rows examined/sent, temp disk tables, no-index flag

Via Railway API: Same infrastructure metrics (CPU, memory, disk, network).

MySQL Metric Sections — Present ALL of These

When the script returns MySQL data, present every section below with its metrics. This matches the full MySQL metrics view. Don't skip sections — if data is present, show it.

1. Overview

Presentation:

| Metric | Value | Status |
|--------|-------|--------|
| Version | 9.4.0 | |
| Uptime | 3d 12h | |
| Connections | 45% (9 / 20) | OK |
| Threads Running | 2 | |
| Aborted Clients | 0 | OK |
| Aborted Connects | 0 | OK |

2. Query Throughput

Show the query mix distribution. A healthy OLTP workload is SELECT-heavy. INSERT/UPDATE-heavy suggests write pressure.

3. InnoDB Row Operations

These are cumulative since server start. Compare read vs write ratios. A read-heavy workload with low row reads may indicate queries returning few results (good) or not using indexes (bad — cross-reference with table scan ratio).

4. Query Efficiency

This section is critical for identifying missing indexes. If table scan ratio is high AND specific top queries show no_index_used > 0, you can give precise index recommendations.

5. InnoDB Buffer Pool

Analysis guidance:

• Hit ratio < 99% + usage > 95% = buffer pool too small for the working set
• Hit ratio < 95% = severe cache pressure — increase innodb_buffer_pool_size
• Dirty pages are modified pages not yet flushed to disk — high dirty count means heavy writes or slow I/O

6. InnoDB I/O

Only show this section if reads or writes > 0. High data reads with low buffer pool hit ratio = cache misses causing disk I/O.

7. Network

Format as KB/MB/GB. High bytes sent relative to received suggests large result sets being returned.

8. Locks

Lock contention + long-running queries in processlist = transactions holding locks too long. Check for MyISAM tables if table lock contention is high.

9. Table Cache

Low cache hit = table_open_cache may be too small. Many opened_tables relative to open_tables means tables are being repeatedly opened and closed.

10. Top Queries (from performance_schema)

This is the most actionable section. Present as a table:

| Query | Calls | Avg Latency | Total Latency | Rows Examined | Rows Sent | Flags |
|-------|-------|-------------|---------------|---------------|-----------|-------|
| SELECT ... FROM orders WHERE... | 15.2K | 2.3ms | 35.1s | 1.2M | 15.2K | |
| SELECT ... FROM users JOIN... | 8.1K | 12.5ms | 101.3s | 890K | 8.1K | ! No Index |

Per-query analysis:

• no_index_used > 0 → Flag with "! No Index" — these are the biggest optimization targets
• tmp_disk_tables > 0 → Query creates on-disk temp tables — needs optimization
• High rows_examined / rows_sent ratio → Scanning many rows to return few — missing or suboptimal index
• Truncate query text to essential parts (tables, WHERE clauses, JOINs). Don't dump full ORM SQL.

If `top_queries` is empty or null: performance_schema is likely disabled — this is the default on Railway. Do not suggest enabling it without caveats: it requires ~400MB+ additional memory and is only advisable on larger instances. Just note that query-level data is unavailable.

11. Tables

| Table | Rows | Data | Indexes | Total |
|-------|------|------|---------|-------|
| orders | 1.2M | 450 MB | 120 MB | 570 MB |
| users | 50K | 12 MB | 8 MB | 20 MB |

Flag tables where index size is disproportionately large relative to data (possible unused indexes) or tables with many rows but no indexes (check with top queries).

12. Active Queries

Show if any non-Sleep, non-Daemon processes are running:

| User | Database | Command | Time (s) | Query |
|------|----------|---------|----------|-------|
| app | mydb | Query | 45 | SELECT ... |

Long-running queries (> 30s) warrant investigation. Cross-reference with lock waits — a long query may be holding locks that block others.

MySQL Performance Patterns

Buffer Pool Starvation Pattern:

• Hit ratio < 99% + pool usage > 95% = buffer pool too small for working set
• High Innodb_data_reads confirms disk I/O from cache misses
• Fix: increase innodb_buffer_pool_size (target 70-80% of available RAM)

Query Inefficiency Pattern:

• Table scan ratio > 50% = most reads are full scans, missing indexes
• Select_full_join > 0 = joins without indexes, extremely expensive
• Temp tables to disk > 10% = sorts/groups exceeding tmp_table_size
• Top queries with no_index_used > 0 = specific queries needing indexes

Lock Contention Pattern:

• row_lock_waits high + row_lock_time high = write contention
• Table lock contention > 1% = may have MyISAM tables or DDL locks
• Long-running queries in processlist holding locks

Connection Pattern:

• connection_usage_percent > 70% = approaching limit
• aborted_clients > 0 = connections not being closed properly (app bug or timeout)
• aborted_connects > 0 = authentication failures or connection limit hits

MySQL Thresholds

MySQL Tuning Knowledge

MySQL-Specific Notes

• `performance_schema=0` in start command disables query-level metrics. This is the default on Railway. Note it when detected but do not recommend enabling it without caveats — it adds ~400MB+ memory overhead and is only practical on larger instances (2GB+ RAM).
• `disable-log-bin` in start command means no binary logging — point-in-time recovery is not possible. Note if relevant.
• `innodb-use-native-aio=0` is common on Railway (container filesystem limitation). Not a concern.
• Cumulative counters: All SHOW GLOBAL STATUS values are cumulative since server start. Use uptime to compute rates (e.g., questions/uptime = queries per second).

Infrastructure (7d + 24h)

Show both windows side by side to compare trends:

7-Day Trends

Last 24 Hours

Compare windows to distinguish sustained vs transient trends.

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See analyze-db.md autoscale rules.

Validated against

• MySQL SHOW GLOBAL STATUS, SHOW VARIABLES, information_schema, performance_schema

PostgreSQL Analysis

This reference covers PostgreSQL-specific metrics, tuning, and analysis guidance. For common analysis patterns (output structure, collection status handling, performance thinking), see analyze-db.md.

What the Script Collects

`collection_status` — check this FIRST. Shows what succeeded vs failed:

• database_query: SSH → psql batched query (connections, cache, vacuum, queries, etc.)
• metrics_api: Railway API for disk, CPU, memory
• logs_api: Railway API for recent log lines
• ha_cluster: SSH → Patroni REST API (HA services only)

Each entry has "status" ("success", "error", or "skipped") and optional "error" or "reason" fields.

All in ONE operation (no additional queries needed):

Connections:

• Current/max/available counts
• States (active, idle, idle_in_transaction)
• By application name
• By age (buckets: <1min, 1-5min, 5-60min, 1-24hr, >24hr)
• Oldest connection age

Memory & Configuration:

• shared_buffers, effective_cache_size, work_mem, maintenance_work_mem
• WAL settings, parallelism settings, planner settings
• Autovacuum status
• track_activity_query_size (tells you if queries are truncated in pg_stat_statements)
• log_min_duration_statement (tells you if slow query logging is enabled and at what threshold)
• idle_in_transaction_session_timeout, statement_timeout (safety timeouts)
• track_io_timing (needed for blk_read_time/blk_write_time in query stats)

Cache Performance:

• Overall table/index hit ratios
• Per-table: hit %, disk reads, size (this is key for diagnosis)

Storage:

• Database size, WAL size
• Per-table: total size, data size, index size, row count

Vacuum Health:

• Per-table: dead rows, dead %, vacuum count, last vacuum/analyze, XID age
• Flags: needs_vacuum, needs_freeze

Indexes:

• Unused indexes (0 scans) with sizes
• Invalid indexes (failed builds)

Query Performance (if pg_stat_statements enabled):

• Top 100 queries by total execution time
• Per-query execution: calls, total_min, mean_ms, min_ms, max_ms, stddev_ms
• Per-query rows: total rows, rows_per_call
• Per-query planning: total_plan_ms, mean_plan_ms
• Per-query cache: shared_blks_hit, shared_blks_read, shared_blks_dirtied, shared_blks_written, cache_hit_pct
• Per-query temp: temp_blks_read, temp_blks_written
• Per-query I/O timing: blk_read_time_ms, blk_write_time_ms (requires track_io_timing=on)
• Per-query WAL: wal_records, wal_bytes
• Per-query local blocks: local_blks_hit, local_blks_read (for temp tables)
• Temp file stats (cumulative since stats reset, NOT current disk usage)

Logs & Active Issues:

• recent_logs: Raw unfiltered logs (1000 lines) - parse these yourself, look for errors, warnings, patterns
• recent_errors: Filtered error-level logs (legacy, for quick reference)
• long_running_queries: Queries running >5s at time of collection
• blocked_queries: Queries waiting on locks
• cluster_logs: HA cluster events (Patroni)

Important: Always analyze the raw recent_logs array thoroughly. This is 1000 lines of unfiltered database output — treat it as a goldmine.

Log analysis checklist — go through ALL of these:

1. Error/Fatal/Panic messages: Count them, categorize them, quote the exact messages

• ERROR: deadlock detected → cross-reference with deadlock count in database_stats
• FATAL: too many connections → cross-reference with connection usage
• ERROR: canceling statement due to statement timeout → which queries are timing out?
• FATAL: out of shared memory → shared_buffers or lock table exhaustion
• ERROR: could not extend file → disk space issue
• PANIC: ... → database crash, investigate immediately

2. Slow query log entries (if log_min_duration_statement is set):

• Count how many slow queries appear
• Identify which tables/queries are mentioned most often
• Cross-reference with top_queries — the same patterns should appear in both
• Note the actual durations logged vs mean_ms from pg_stat_statements

3. Autovacuum activity:

• LOG: automatic vacuum of table → is autovacuum running? How often?
• LOG: automatic analyze of table → statistics being updated
• WARNING: oldest xmin is far in the past → XID wraparound risk
• Absence of autovacuum entries with high dead rows → autovacuum may be blocked or misconfigured

4. Checkpoint activity:

• LOG: checkpoint starting / LOG: checkpoint complete → how frequent?
• checkpoint complete: wrote X buffers (Y%) → high Y% means lots of dirty data
• Time between checkpoints — if < 5 minutes, write load is high
• checkpoints are occurring too frequently → increase max_wal_size

5. Connection patterns:

• LOG: connection received / LOG: connection authorized → connection rate
• LOG: disconnection → normal or unexpected? Check session duration
• FATAL: remaining connection slots are reserved → max_connections hit
• FATAL: password authentication failed → unauthorized access attempts

6. Replication messages:

• LOG: started streaming WAL → replica connected
• ERROR: requested WAL segment has already been removed → replica too far behind
• FATAL: could not receive data from WAL stream → replication broken

7. Temporal patterns:

• Are errors clustered in time? (burst vs steady)
• Do slow queries correlate with checkpoint times?
• Is there a pattern suggesting cron jobs or batch processing?

State what you found with specifics: "Analyzed 1000 log lines covering 2024-01-15 14:00 to 15:30. Found: 23 slow query warnings (all SELECT on UserSession table, 200-800ms), 4 autovacuum runs, 2 checkpoints (normal interval), 0 errors. The slow queries correlate with the UserSession table's 76% cache hit rate."

Log Interpretation When Only Logs Are Available

When collection_status.database_query failed and you only have logs:

Startup vs steady-state logs:

• LOG: database system is ready to accept connections — normal startup, NOT evidence of a crash
• LOG: started streaming WAL — normal replication, NOT an error
• LOG: checkpoint starting / LOG: checkpoint complete — routine operation
• FATAL: the database system is starting up — transient during restarts, NOT a persistent problem

What you CAN say from logs alone:

• Whether errors or warnings are present and their frequency
• Whether the database recently restarted (and that this is normal during deploys)
• Whether there are connection refused errors (possible saturation or startup)

What you CANNOT say from logs alone:

• Whether the database is performing well or poorly
• Whether cache hit ratios are good
• Whether vacuum is behind
• Whether queries are slow
• Any tuning recommendations

If only logs are available, explicitly state: "No performance conclusions possible — database metrics were not collected."

Active Issues:

• Long-running queries (>5s)
• Idle in transaction (>30s)
• Blocked queries (waiting on locks)
• Lock contention details

Infrastructure (7d + 24h) — show both windows so trends can be compared:

7-Day Trends

Last 24 Hours

Compare: "Disk growing slowly over 7d but stable over 24h → gradual data growth, not an acute event."

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See analyze-db.md autoscale rules.

Replication / HA (if applicable):

• Replication status
• HA cluster status (Patroni)
• Background writer stats
• WAL archiver status

PostgreSQL Tuning Knowledge

Use this to reason about configuration issues:

Memory Parameters

Tuning Formulas

shared_buffers = RAM × 0.25 (max 40%)
  1GB RAM  → 256MB
  4GB RAM  → 1GB
  16GB RAM → 4GB

work_mem = (RAM / max_connections) / 4
  4GB RAM, 100 conns → 10MB
  8GB RAM, 200 conns → 10MB

effective_cache_size = RAM × 0.75
  4GB RAM  → 3GB
  16GB RAM → 12GB

Settings Requiring Restart vs Immediate

Restart required:

• shared_buffers
• max_connections
• max_parallel_workers

Immediate (SIGHUP):

• work_mem
• effective_cache_size
• random_page_cost
• checkpoint_completion_target

SSD vs HDD

Railway uses SSDs. If random_page_cost = 4.0 (HDD default), the planner thinks random reads are 4x more expensive than sequential - it avoids index scans. Set to 1.1-2.0 for SSDs.

Railway auto-scales vertically

See analyze-db.md for full autoscale rules. For PostgreSQL specifically:

• Tune parameters relative to the current RAM from metrics_history.memory.current, not memory_limit.
• If shared_buffers is undersized relative to current RAM, recommend increasing it to 25% of current RAM.
• If the working set far exceeds what 25% of current RAM can hold, note this as a limitation of the current memory footprint — but do NOT tell the user to increase RAM. The platform handles that automatically.

Thresholds for Reasoning

Vacuum Priority Matrix

Dead row percentage alone doesn't determine urgency. Use this matrix:

A 1 MB table with 25% dead rows has ~250 KB of bloat. Not worth mentioning as "critical".

Applying Fixes

When recommending changes, include the actual SQL and always explain side effects — especially for settings that add overhead or change behavior.

-- Memory tuning (example for 4GB RAM)
ALTER SYSTEM SET shared_buffers = '1GB';
ALTER SYSTEM SET effective_cache_size = '3GB';
ALTER SYSTEM SET work_mem = '32MB';
ALTER SYSTEM SET random_page_cost = 1.5;
SELECT pg_reload_conf();
-- Note: shared_buffers requires restart

-- Vacuum specific tables
VACUUM ANALYZE "TableName";

-- Emergency XID freeze
VACUUM FREEZE "TableName";

Side effects to document per setting

Enabling pg_stat_statements

ONLY suggest this if BOTH conditions are true: 1. pg_stat_statements_installed is false in the JSON output 2. top_queries is empty or missing

If these conditions are met, tell the user to run (do NOT execute with Bash):

python3 scripts/enable-pg-stats.py --service <name>

This may require a brief restart.

If `pg_stat_statements_installed: true` and `top_queries` has data, DO NOT suggest enabling it.

---

PostgreSQL-Specific Guidance

The sections below apply specifically to PostgreSQL analysis via scripts/analyze-postgres.py.

How to Think About PostgreSQL Performance

The Core Question

When you see a problem, ask: What is the chain of causation?

Example chain: 1. Cache hit is 89% (symptom) 2. Email table has 6% cache hit with 1.19B disk reads (deeper symptom) 3. Email table is 1.7GB, shared_buffers is 128MB (root cause) 4. The table is 13x larger than the buffer pool - it will NEVER fit in cache 5. Every query touching Email forces disk I/O

This reasoning is what you provide. The script gives you the data points - you connect them.

Patterns to Look For

Memory Starvation Pattern:

• Low cache hit + large tables + small shared_buffers = working set doesn't fit
• High temp files + low work_mem = sorts/hashes spilling to disk
• These often occur together - both indicate the database needs more memory

Important: Temp file stats (temp_files, temp_bytes) are cumulative since the last stats reset, not current disk usage. When reporting, say "X GB written to temp files since stats reset" - not "X GB on disk right now".

Vacuum Neglect Pattern:

• High dead rows % + "never" vacuum timestamps = autovacuum isn't keeping up
• Multiple tables with >10% dead rows = systemic issue, not one-off
• High XID age + vacuum issues = potential wraparound emergency

Important: Consider absolute impact, not just percentage. A tiny table (< 10 MB) with 20% dead rows has negligible impact - vacuuming it reclaims almost nothing. Prioritize tables with BOTH high dead row counts (thousands+) AND meaningful size (tens of MB+). Don't mark small tables as "critical" just because of a high percentage.

Missing Index Pattern:

• High seq_scan count + 0 idx_scans on large tables = queries scanning full tables
• Low cache hit on specific tables + high seq_scans = indexes would help AND reduce I/O

Connection Pressure Pattern:

• High connection % + many idle connections = connection pooling needed
• Old connections (days) + idle_in_transaction = potential connection leaks or stuck transactions

Slow Query Analysis — Go Deep

The top_queries array is the most valuable data for customers. This is where you can give the most actionable, specific advice. Don't skim it — analyze every query in the top 10-15 thoroughly.

Per-Query Fields and What Each Tells You

Red Flags — What Demands Explanation

How to Present Slow Queries

Show the full table first with all available metrics (the report already includes these columns):

| Query (truncated) | Calls | Total (min) | Mean (ms) | Min/Max (ms) | Stddev | Rows/Call | Cache Hit | Temp R/W | Plan (ms) | I/O Time |
|-------------------|-------|-------------|-----------|--------------|--------|-----------|-----------|----------|-----------|----------|
| SELECT Email.ccFull... | 78K | 132 | 101 | 0.3/8200 | 340 | 0.05 | 47% | 0/0 | 1.2 | 45000 |
| SELECT Thread... ORDER BY | 48K | 223 | 279 | 2.1/45000 | 2400 | 12,177 | 98.8% | 0/39M | 0.4 | 800 |
| SELECT Content... | 1.3K | 12 | 563 | 180/3200 | 420 | 0.65 | 1.8% | 0/0 | 8.3 | 31000 |

Then analyze EACH query — this is the most valuable part. For each of the top 10 queries, explain:

1. What the query does — identify the tables, the pattern (lookup, join, aggregation, pagination) 2. Why it's slow — connect the specific metrics to a root cause 3. The cascading impact — how this query affects overall database health 4. Specific fix — not generic advice, but targeted to what the metrics show

Example deep analysis:

Query 1: Email.ccFull join (78K calls, 101ms mean, 132 min total)

- Pattern: Joins Email → EmailThreadKind → Thread → EmailEntry. ORM-generated N+1 or bulk join.

- Root cause: 47% cache hit means 53% of blocks come from disk. The Email table is 1.7GB but shared_buffers is 128MB — only 7.5% of this table can be cached at once. Every call displaces other data from cache, creating a cascading eviction problem.

- The stddev of 340ms with max of 8200ms means some calls take 80× longer — likely when the needed pages were just evicted by another query.

- I/O time of 45,000ms total confirms this: the query has spent 45 seconds just waiting for disk across all calls.

- rows_per_call = 0.05 means it almost never finds a match — it's doing all this I/O for an existence-check pattern. An EXISTS() subquery with proper index could eliminate the full table scan.

- Fix: (a) Increase shared_buffers to 1GB so the hot portion stays cached. (b) Add index on Email(ccFull, threadId) to avoid the sequential scan. (c) Rewrite as EXISTS if the app only needs presence, not the full row.

Query 2: Thread pagination (48K calls, 279ms mean, 223 min total)

- Pattern: SELECT Thread... ORDER BY with large result set. Pagination query.

- Root cause: rows_per_call = 12,177 — returning 12K rows per call is a pagination bug (missing LIMIT) or an admin/batch endpoint.

- temp_blks_written = 39M (312 GB of temp files!) — the ORDER BY creates a sort that exceeds work_mem (4MB), so it spills to disk every single time.

- stddev = 2400ms with max = 45,000ms — some executions take 45 seconds, likely when disk temp files compete with other I/O.

- Cache hit is 98.8% — the data itself is cached, but the sort still spills because work_mem is separate from shared_buffers.

- Fix: (a) Add LIMIT if this is user-facing. (b) Create an index matching the ORDER BY clause to eliminate the sort entirely. (c) Increase work_mem to 32-64MB so the sort fits in memory.

Truncate Long Queries Intelligently

• Show the table names and key operations (JOIN, WHERE, ORDER BY)
• Don't dump 2000-character ORM-generated SQL
• Identify the pattern: "Thread zone assignment lookup" not the full SQL
• For ORM queries with $1, $2, ... parameters, note that the actual values aren't available — the pattern matters more than specific values
• Note on query truncation: pg_stat_statements stores full query text up to track_activity_query_size (default 1024 chars). ORM-generated queries often exceed this — if a query ends abruptly, it was truncated by PostgreSQL, not by our script. The JSON output preserves the full text from pg_stat_statements; only the human-readable text report truncates for display

Query Workload Profile

After analyzing individual queries, summarize the overall workload:

• Read vs write ratio: Use tup_returned/tup_fetched vs tup_inserted/tup_updated/tup_deleted from database_stats
• Top 3 time consumers: Which queries dominate total_min? If 3 queries account for 80% of execution time, that's where to focus
• Cache pressure sources: Which queries have the most shared_blks_read? They're driving cache misses for everything else
• Temp file culprits: Which specific queries create temp files? Don't say "increase work_mem" generically — say "Query X creates Y GB of temp files per day"
• WAL generators: If applicable, which queries generate the most WAL bytes? They're driving replication lag

Correlate Across Sections

The script collects many data points. Look for correlations:

Synthesize Insights the Script Can't

The script flags individual issues. You should:

1. Identify the PRIMARY bottleneck - What's the #1 thing hurting performance right now? 2. Explain cascading effects - How does one problem cause others? 3. Prioritize fixes - What should they do first, second, third? 4. Warn about risks - What happens if they don't fix this?

Important: Synthesis is prose that EXPLAINS the data tables you already showed. Don't hide data in prose - the tables make it visible, the prose connects the dots.

Example flow: 1. Show config table: shared_buffers = 128 MB vs recommended 1 GB 2. Show cache table: Email table at 6% cache hit with 1.19B disk reads 3. THEN explain: "Your buffer pool (128 MB) is 13x smaller than your Email table (1.7 GB). This single table is dragging down your overall 89% cache hit rate."

The user sees the data, understands the relationship, then gets the explanation. Don't make them trust your conclusions without seeing the evidence first.

Common Errors to Avoid (PostgreSQL-Specific)

• Saying "enable pg_stat_statements" when pg_stat_statements_installed: true and top_queries has data
• Misreporting connection usage (check percent field, not just current)
• Ignoring the oldest_connections details when flagging old connections
• Saying "746 GB of temp files on disk" when temp_bytes is cumulative since stats reset
• Marking tiny tables (< 10 MB) as "critical" for vacuum just because of high dead row percentage
• Listing slow queries by total_time only without analyzing cache_hit_pct, temp_blks, and rows returned
• Dumping full ORM-generated SQL instead of summarizing the query pattern

Validated against

• PostgreSQL system views: pg_stat_activity, pg_stat_statements, pg_statio_user_tables, pg_stat_user_tables
• Patroni REST API for HA clusters

Redis Analysis

This reference covers Redis-specific metrics, tuning, and analysis guidance. For common analysis patterns (output structure, collection status handling, performance thinking), see analyze-db.md.

What the Script Collects

Via SSH (`INFO ALL`):

• Overview: version, uptime, connected/blocked clients, rejected connections
• Memory: used/RSS/peak memory, fragmentation ratio, maxmemory, eviction policy
• Throughput: ops/sec, total commands processed, total connections
• Cache Performance: keyspace hits/misses, hit rate, expired/evicted keys
• Persistence: RDB last save time/status, AOF enabled/status
• Command Stats: per-command call count, avg latency, total time (sorted by calls)
• Keyspace: per-database key count, expires, avg TTL
• Slow Log: count via SLOWLOG LEN + actual entries via SLOWLOG GET 20 (command, duration, timestamp)
• Biggest Keys: via redis-cli --bigkeys — runs remotely over SSH on the Railway service, not locally

Via Railway API: Same infrastructure metrics (disk, CPU, memory, network with 7d and 24h trends).

Presentation Template

Present Redis analysis using grouped stat cards that mirror the sections below. Each section is a labeled group with key-value stat cards. Flag values with status indicators (healthy/warning/critical) using the thresholds table.

Full Report (SSH + Metrics + Logs all succeeded)

Overview

Flag: blocked clients > 0 = warning, rejected connections > 0 = critical.

Memory

Flag: fragmentation > 1.5 = warning, > 2.0 or < 1.0 = critical. Evicted keys > 0 with noeviction = problem.

Throughput

Flag: slow log > 100 = warning.

Cache Performance

Flag: hit rate >= 95% = healthy, 80-95% = warning, < 80% = critical. Evicted > 0 = warning.

Persistence

Flag: RDB status != ok = critical. AOF rewrite status != ok = critical.

Command Stats (top 20 by calls)

Slow Log Entries (if collected — up to 20 most recent)

Analysis: correlate slow commands with command stats and big keys. If slowlog shows GET and bigkeys shows large strings, the diagnosis is "large values causing high GET latency" — confirmed without user intervention.

Biggest Keys (if collected — one per type)

Analysis: large keys cause latency spikes on read/write/delete. Cross-reference with slowlog — if the slow commands target these keys, that's the root cause. If bigkeys shows nothing large (all < 1KB), latency issues are likely volume-driven, not value-size-driven.

Keyspace

Infrastructure (7d + 24h) — show both windows so trends can be compared:

7-Day Trends

Last 24 Hours

Compare: "Memory increasing in 24h but stable over 7d → temporary spike, not a sustained trend."

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See analyze-db.md autoscale rules.

Partial Report (Introspection failed, only Metrics + Logs)

When introspection fails, you have NO Redis INFO data — all overview, memory, throughput, cache, persistence, command stats, and keyspace fields will be null/empty.

NEVER suggest running `redis-cli` without pointing to the remote Railway service host. There is no local Redis instance — all redis-cli commands must target the Railway service. If you cannot connect, the fix is to restore remote access (see analyze-db.md), not to run commands locally.

You MUST: 1. State clearly: "Redis introspection failed — could not connect to the service" 2. Show collection status errors 3. Show ONLY infrastructure metrics and log analysis — do not show empty stat card sections 4. Do NOT produce recommendations based on null Redis metrics

Show the infrastructure table (same as full report).

Analyze logs thoroughly:

• AOF rewrite frequency and growth % triggers
• fsync warnings ("disk is busy?")
• OOM warnings
• Connection errors
• Startup/restart events (note these are normal during deploys)
• Summarize with counts: "Analyzed 1000 lines: 18 AOF rewrites, 1 fsync warning, 0 errors"

State what you cannot determine without SSH:

• Connection health (clients, blocked, rejected)
• Memory usage and fragmentation
• Cache hit rate
• Eviction status
• Command workload profile
• Keyspace composition
• Slow log entries and actual slow commands
• Biggest keys per type

Redis Performance Patterns

Memory Fragmentation Pattern:

• mem_fragmentation_ratio > 1.5 = memory is fragmented, RSS much higher than used
• Caused by frequent small key deletions creating memory holes
• Fix: restart Redis, or enable activedefrag yes (Redis 4.0+)
• Ratio < 1.0 means Redis is using swap — critical performance issue

Cache Thrashing Pattern:

• Hit rate < 80% + evicted keys > 0 = working set exceeds maxmemory
• Check maxmemory_policy — noeviction will reject writes, allkeys-lru will evict
• If maxmemory is 0 (unlimited), Redis will consume all RAM until OOM killed

Connection Rejection Pattern:

• rejected_connections > 0 = maxclients limit hit
• Check connected_clients vs maxclients default (10,000)
• Blocked clients = operations waiting on BLPOP/BRPOP/WAIT

Persistence Risk Pattern:

• RDB last save failed + no AOF = data loss risk on restart
• Check disk space if saves are failing
• Long time since last save = more data at risk

AOF Rewrite Churn Pattern:

• Frequent AOF rewrites (every 1-2 hours) with high growth % triggers (100-800%)
• Indicates high write-to-data-size ratio — small dataset with heavy writes
• Check Fork CoW size to gauge actual data size vs AOF overhead
• If rewrites are fast (<1s) and CoW is small (<10 MB), this is noisy but harmless
• If rewrites are slow or CoW is large, investigate write patterns

Disk Sawtooth Pattern:

• Disk usage oscillating in a regular pattern = AOF growing between rewrites, then compacting
• Normal behavior — the baseline is volume overhead + AOF base file
• If the amplitude is growing over time, data size is increasing

Redis Thresholds

Redis Command Stats Analysis

The top commands by call count reveal the workload pattern:

• GET/SET dominant = simple key-value cache
• HGET/HSET dominant = hash-based data model (sessions, objects)
• LPUSH/RPOP dominant = queue pattern
• High KEYS/SCAN = application iterating keys (potential performance issue at scale)
• High latency on simple commands (>100µs for GET) = memory pressure or CPU saturation

Redis Autoscale Note

See analyze-db.md for full autoscale rules. For Redis specifically:

• If maxmemory is set, compare it against actual memory usage — not the Railway memory limit.
• If maxmemory is 0 (unlimited), Redis will grow until the OS kills it. This is the default Railway config and works fine with autoscaling — Redis uses what it needs and Railway scales the container.
• Do NOT recommend setting maxmemory to a fraction of the Railway memory limit — the limit is the autoscale ceiling, not fixed allocation.

Validated against

• Redis INFO command, SLOWLOG LEN, SLOWLOG GET, and --bigkeys

Database Analysis

Your Role

You are a database performance expert. The script collects raw data - your job is to think deeply about what you see, identify root causes, correlate symptoms, and explain the "why" behind problems.

Don't just report metrics. Analyze them.

Context Resolution

The user's request is the source of truth. Use this decision table:

railway status --json is a hint to form a specific question, not a trigger to act without confirmation.

When the user provides a Railway URL, extract IDs directly from it:

https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>?environmentId=<ENV_ID>
https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>/database?environmentId=<ENV_ID>

Then query the API for the service name and database type in a single call:

scripts/railway-api.sh \
  'query getServiceAndConfig($serviceId: String!, $environmentId: String!) {
    service(id: $serviceId) { name }
    environment(id: $environmentId) {
      config(decryptVariables: false)
    }
  }' \
  '{"serviceId": "<SERVICE_ID>", "environmentId": "<ENV_ID>"}'

From the response, get:

• Service name: data.service.name
• Database image: data.environment.config.services.<SERVICE_ID>.source.image

Then match the image to the database type:

If `environmentId` is empty in the URL (e.g., ?environmentId= or no query param at all), skip the environment.config query — it requires a valid ID. Instead, list the project's environments:

scripts/railway-api.sh \
  'query getEnvs($id: String!) { project(id: $id) { environments { edges { node { id name } } } } }' \
  '{"id": "<PROJECT_ID>"}'

Use the production environment by default. If multiple non-PR environments exist and the user hasn't specified one, ask which environment to analyze.

Database Type Detection and Script Selection

All scripts share the same CLI interface (use the script name from the table above):

python3 scripts/analyze-<script>.py \
  --service <name> \
  --json \
  --project-id <project-id> \
  --environment-id <env-id> \
  --service-id <service-id>

Common options across all scripts:

• --json — JSON output for programmatic processing
• --quiet — Suppress progress messages
• --skip-logs — Skip log collection
• --metrics-hours <N> — Hours of metrics history (default: 24, max: 168)
• --step <step> — Debug individual collection steps (ssh-test, query, logs, metrics)

Before You Analyze: Check Collection Status

ALWAYS check `collection_status` and `errors[]` FIRST before interpreting any data. The script collects data from multiple independent sources. Any of them can fail.

Decision Table

When database_query failed — SSH key errors

If the error contains "No SSH keys found" or "SSH key registration required", handle it proactively — don't just tell the user to fix it themselves.

If error contains `"Key found but not registered"` or `"No SSH keys found"`:

Run these two commands to understand what's available:

railway ssh keys          # list keys already registered with Railway
ls ~/.ssh/*.pub 2>/dev/null  # list local public keys

Then present the user with their options and ask which to use:

SSH introspection needs a registered key. Here's what I found:

Registered with Railway: <list from `railway ssh keys`, or "none">
Local keys available:    <list from ~/.ssh/*.pub, or "none">

Options:
  1. Register a local key — `railway ssh keys add` (uses your default key)
  2. Import from GitHub  — `railway ssh keys github`
  3. Generate a new key  — `ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_railway`

Which would you like to do?

Once the user chooses and the key is registered, re-run the full analysis.

When database_query failed — CLI outdated or missing

If the error contains "--native SSH flag is not supported" or "Railway CLI not found", the script detected an unrecoverable CLI problem. Tell the user directly:

• Outdated CLI: "Your Railway CLI doesn't support native SSH. Update it: npm i -g @railway/cli@latest or brew upgrade railway"
• Missing CLI: "Railway CLI not installed. Install it: npm i -g @railway/cli or brew install railway"

Then ask if they'd like to proceed with a partial analysis (metrics + logs only) while they update, or wait until the CLI is fixed and re-run the full analysis.

When database_query failed — other SSH errors

This means SSH could not reach the database or the query failed. You have NO connection stats, NO cache hit ratios, NO vacuum health, NO query performance data. All those fields will be null/empty.

You MUST: 1. State clearly: "Database introspection failed — SSH could not connect to the service" 2. Show the collection_status errors 3. Show only the data that DID succeed (metrics, logs) 4. Do NOT produce recommendations based on null metrics 5. Do NOT diagnose performance issues from logs alone

Partial report template:

Service: <name>
Status: Data collection partially failed

## Collection Status
| Source | Status |
|--------|--------|
| Database Query (SSH) | ERROR: <error from collection_status> |
| Metrics API | <status> |
| Logs API | <status> |

## Available Data
<Show metrics and log summary from sources that succeeded>

## What We Cannot Determine
<List what requires the database query: connection health, cache performance, vacuum health, query analysis, etc.>

Output Structure: Data First, Actions Second

Always present information in this order:

1. Context Header

Service: <name> (<project> <environment>)
Status: <deployment health>, <RAM>, <disk used>

2. Consolidated Data Tables

Before any analysis, show the raw metrics in tables so the user sees their actual state. The DB-specific reference defines exactly which tables to show for each database type — present the relevant health sections first, then connections, then query performance.

Logs & Active Issues:

• Parse the recent_logs array (1000 lines of raw logs) - don't just check if empty
• Summarize: "Analyzed 1000 log lines: 3 errors (connection timeouts), 0 critical issues"
• Show specific concerning log entries if found
• Categorize log entries: group by type (errors, warnings, connection events, replication, crashes/restarts)
• Count patterns: note if a single type dominates the log output
• Quote actual log lines for errors — don't just say "errors found", show the exact message so the user can search their codebase

3. Analysis

After showing the data, explain the chain of causation. Connect the dots between tables.

4. Recommended Actions

Group by urgency. For databases that support configuration changes without restart vs those that require one, call that out explicitly.

5. Expected Outcomes

What metrics should change after fixes.

---

Why this order matters:

• Users can verify the data matches their understanding
• They see the full picture before being told what to do
• Actions have context - they know WHY each fix is recommended
• No valuable data is hidden in prose or omitted

CRITICAL: Use the Actual Data

NEVER fabricate or assume values. The script outputs JSON with exact numbers. Before stating any metric:

1. Read the actual JSON output - Don't truncate or skim 2. Quote the exact values - e.g., "max": 5000 not "100" 3. Investigate outliers - Dig into any field that seems unusually high or low

Common errors to avoid (all database types):

• Not parsing `recent_logs` - always analyze the raw log lines, don't just report "no errors"
• Diagnosing performance issues from logs when all metrics are null — logs show what happened, not how the database is performing
• Treating startup/restart log entries as evidence of failure — databases restart for many normal reasons (deploys, config changes, scaling)
• Producing recommendations when all database metrics are null — if collection_status.database_query is "error", you have no basis for tuning advice

See the DB-specific reference for additional errors to avoid per database type.

Running the Analysis

Pass project, environment, and service IDs directly — no railway link needed:

# From plugins/railway/skills/use-railway directory:
# Use the script name from the "Database Type Detection" table above
python3 scripts/analyze-postgres.py --service <name> --json \
  --project-id <project-id> --environment-id <env-id> --service-id <service-id>

All three IDs come from the URL (see "Context: URL First" above). The service name comes from the API query.

Options:

• --metrics-hours <N> — Hours of metrics history to fetch (default: 24, max: 168). Use --metrics-hours 168 for 7-day trends, --metrics-hours 1 for recent snapshot.

SSH retry: The script automatically retries SSH connectivity up to 3 times with increasing timeouts (30s, 60s, 90s). Each individual SSH command (database query, slowlog, bigkeys, etc.) also retries up to 3 times on failure — covering transient errors like exec request failed on channel 0. Progress is logged to stderr.

Output: Progress messages go to stderr. JSON results go to stdout. Do not redirect or pipe stderr — just run the command as-is and read the full output.

Resolving environment by name

If the URL has no environmentId and the user specifies an environment by name (e.g., "production"), resolve it:

scripts/railway-api.sh \
  'query getProject($id: String!) {
    project(id: $id) {
      environments { edges { node { id name } } }
    }
  }' \
  '{"id": "<PROJECT_ID>"}'

Match the environment name (case-insensitive) to get the environmentId.

Debugging individual steps

# Use the script name from the "Database Type Detection" table above
python3 scripts/analyze-postgres.py --service <name> \
  --project-id <pid> --environment-id <eid> --service-id <sid> \
  --step ssh-test    # Test SSH connectivity
  --step query       # Run only the database query
  --step metrics     # Fetch only API metrics
  --step logs        # Fetch only logs

Database-Specific References

After running the script and checking collection status, load the reference for the specific database type:

Always load the DB-specific reference — it contains the metric sections, thresholds, and tuning knowledge needed for proper analysis.

Infrastructure Metrics (All Database Types)

All scripts collect the same infrastructure metrics via Railway API:

Metrics History (`metrics_history`): The script fetches 7 days (168 hours) of time-series data from Railway's metrics API by default and produces two analysis windows:

{
  "metrics_history": {
    "windows": {
      "7d": { "window_hours": 168, "metrics": { "cpu": {...}, "memory": {...}, ... } },
      "24h": { "window_hours": 24, "metrics": { "cpu": {...}, "memory": {...}, ... } }
    }
  }
}

Each window independently computes:

• Summary stats: current, min, max, avg for each metric
• Trend analysis: compares first-quarter avg to last-quarter avg — reports direction (increasing/decreasing/stable) and % change
• Spike detection: flags values > avg + 2*stddev with timestamps of peaks
• Downsampled series: ~48 data points per window

Available metrics: CPU, memory (with limits), disk, network RX/TX.

Comparing windows reveals whether a trend is new or sustained:

• "Memory increasing in 24h but stable over 7d" → temporary spike, likely a batch job
• "Memory increasing in both 24h AND 7d" → sustained growth, may need investigation
• "CPU spike in 24h, no spikes in 7d" → new issue
• "Disk growing over 7d" → data accumulation trend

Use --metrics-hours N to change the long window (default: 168, max: 168). The 24h window is always produced when the long window is > 24h.

Railway auto-scales vertically

Railway services auto-scale CPU, RAM, and disk based on actual usage. Users do NOT pick or control resource sizes. The cpu_limit and memory_limit values from metrics are the autoscale ceiling (typically 32 vCPU / 32 GB), not user-provisioned allocations. Users are billed for actual usage, not the ceiling.

Rules for ALL database types:

• Never say "right-size the instance" or suggest reducing CPU/RAM — it's not a user action.
• Never flag low utilization % against the limit as waste — a service showing 0.01 vCPU / 70 MB actual usage against a 32 vCPU / 32 GB ceiling is normal, not over-provisioned.
• Disk does NOT auto-scale — Railway volumes have a fixed capacity. Paid users (Hobby and Pro) can expand them live without downtime, but it requires a manual resize. Flag high disk utilization as actionable. Users are billed for actual disk utilization, not the full volume size.
• Focus on actual usage values, not the ratio to limits. Analyze whether 70 MB of memory is healthy for this workload — don't compare it to the 32 GB ceiling.
• When tuning database parameters (shared_buffers, innodb_buffer_pool_size, maxmemory, etc.), base recommendations on the current actual RAM from metrics_history.memory, not the limit.

Validated against

• Docs: ssh.md, logs.md, metrics.md, api docs
• Local scripts: analyze-postgres.py, analyze-mysql.py, analyze-redis.py, analyze-mongo.py, dal.py

Configure

Manage environments, variables, service config, domains, and networking.

Environments

List and switch

railway environment list --json
railway environment list --ephemeral --json      # only PR environments
railway environment list --no-ephemeral --json   # hide PR environments
railway environment link <environment>           # switch active environment

Create

railway environment new <name>
railway environment new <name> --duplicate <source-environment>    # clone config from existing

Duplicating copies all service configurations and variables from the source environment.

Variables

Read, set, and delete

railway variable list --service <service> --environment <env> --json
railway variable set KEY=value --service <service> --environment <env>
railway variable delete KEY --service <service> --environment <env>

Variable changes trigger a redeployment by default. This is usually the desired behavior, since the service picks up the values on restart. Use --skip-deploys only when you plan to redeploy or restart separately.

Set sensitive values

Use stdin for secrets or values that shouldn't appear in shell history:

printf "%s" "$SECRET_VALUE" | railway variable set API_KEY --stdin --service <service>
railway variable set FEATURE_FLAG=true --service <service> --skip-deploys
railway variable set API_URL=https://api.example.com --project <project-id> --environment <env> --service <service>

Template syntax

Railway supports interpolation between services and shared variables:

${{KEY}}                                         # same-service variable
${{shared.API_KEY}}                              # shared variable
${{postgres.DATABASE_URL}}                       # variable from another service
${{api.RAILWAY_PRIVATE_DOMAIN}}                  # another service's private domain

Wiring example, a frontend connecting to a backend over private networking:

BACKEND_URL=http://${{api.RAILWAY_PRIVATE_DOMAIN}}:${{api.PORT}}

Wiring services together

Each managed database creates connection variables automatically. Reference them from other services using template syntax:

Service names in references are case-sensitive and must match the service name exactly as it appears in the project.

Public vs private networking decision:

Frontend apps cannot use private networking. Frontends run in the user's browser, not on Railway's network. They cannot reach RAILWAY_PRIVATE_DOMAIN or internal database URLs. Options:

1. Backend proxy (recommended): frontend calls a backend API on a public domain, backend connects to the database over the private network. 2. Public database URL: use the public connection variable (for example, ${{Postgres.DATABASE_PUBLIC_URL}}). This requires a TCP proxy on the database service and exposes the database to the internet. Use this only for development or low-sensitivity data.

Railway-provided variables

These are set automatically at runtime. Availability depends on resource configuration.

Networking:

Context:

Git (present when deployed from a linked repo):

Storage (present when a volume is attached):

Sealed variables are write-only. Their values don't appear in CLI output.

Service config

Service configuration controls source, build, deploy, and networking settings. There are two ways to mutate it.

Dot-path patch

For single-field changes:

railway environment edit --service-config <service> deploy.startCommand "npm start"
railway environment edit --service-config <service> build.buildCommand "npm run build"
railway environment edit --service-config <service> source.rootDirectory "/apps/api"
railway environment edit --service-config <service> deploy.numReplicas 2
railway environment edit --project <project-id> --environment production --service-config <service> deploy.startCommand "npm start"

JSON patch

For multi-field changes or complex structures:

railway environment edit --json <<'JSON'
{"services":{"<service-id>":{"build":{"buildCommand":"npm run build"},"deploy":{"startCommand":"npm start"}}}}
JSON

Resolve exact service IDs from railway service list --json before constructing JSON patches. Using names in the JSON payload doesn't work.

Stage config changes

Stage changes when the user wants to review config before committing it:

railway environment edit --service-config <service> build.buildCommand "npm run build" --stage
railway environment edit --service-config <service> deploy.startCommand "npm start" --message "Set production start command"

Use --stage only when the user requests staged config changes. Use regular edits for immediate mutations.

Config schema (typed paths)

Include only keys you're changing. The full shape:

Source: source.image (string), source.repo (string), source.branch (string), source.rootDirectory (string), source.checkSuites (boolean), source.commitSha (string), source.autoUpdates.type (string: disabled, patch, minor)

Build: build.builder (string: RAILPACK, NIXPACKS, DOCKERFILE), build.buildCommand (string), build.dockerfilePath (string), build.watchPatterns (string array), build.nixpacksConfigPath (string)

Deploy: deploy.startCommand (string), deploy.preDeployCommand (string), deploy.healthcheckPath (string), deploy.healthcheckTimeout (integer), deploy.numReplicas (integer), deploy.restartPolicyType (string: ON_FAILURE, ALWAYS, NEVER), deploy.restartPolicyMaxRetries (integer), deploy.sleepApplication (boolean), deploy.cronSchedule (string), deploy.multiRegionConfig (object)

Multi-region config structure for deploy.multiRegionConfig:

{ "us-west2": { "numReplicas": 2 }, "europe-west4-drams3a": { "numReplicas": 1 } }

Natural language mapping: "add replicas in Europe" → europe-west4-drams3a, "US East" → us-east4-eqdc4a. When the user doesn't specify a region, query current config first with railway environment config --json to see existing region assignments before modifying.

Variables: variables.<KEY>.value (string), variables.<KEY>.isOptional (boolean), variables.<KEY>.isSealed (boolean). Delete a variable by setting it to null.

Lifecycle: isDeleted (boolean) removes the service. isCreated (boolean) marks as new. Prefer railway service delete for normal service deletion.

Storage: volumeMounts.<volume-id>.mountPath (string), volumes.<volume-id>.isDeleted (boolean)

Buckets: buckets.<bucket-id>.region (string: sjc, iad, ams, sin), buckets.<bucket-id>.isCreated (boolean), buckets.<bucket-id>.isDeleted (boolean). Buckets are created at the project level via railway bucket create and deployed to environments via config patches. The CLI handles this automatically, so use railway bucket commands

Shared variables and project-level config

railway environment edit --json <<'JSON'
{"sharedVariables":{"API_BASE":{"value":"https://example.com"}}}
JSON

Shared variables are accessible from any service via ${{shared.KEY}}.

Read config

Always inspect before mutating. Config patches merge, so you need to know the state to avoid overwriting fields unintentionally:

railway environment config --json

Verify after mutation to confirm the change took effect:

railway environment config --json
railway service list --json

Domains

Railway domain

One Railway-provided domain per service, generated automatically:

railway domain --service <service> --json

Custom domain

railway domain example.com --service <service> --json

This returns the DNS records you need to configure at your DNS provider. Multiple custom domains per service are supported.

Target port

If the service listens on a non-default port:

railway domain example.com --service <service> --port 8080 --json

Private networking

For service-to-service traffic within a project, use private domain references instead of public URLs. This avoids egress and is faster:

BACKEND_URL=http://${{api.RAILWAY_PRIVATE_DOMAIN}}:${{api.PORT}}

Read current domains

Domain configuration lives in config.services.<service-id>.networking under serviceDomains (Railway-provided) and customDomains. Inspect with:

railway environment config --json

Remove a domain

Remove domains via JSON config patch by setting the domain ID to null:

Remove a custom domain:

railway environment edit --json <<'JSON'
{"services":{"<service-id>":{"networking":{"customDomains":{"<domain-id>":null}}}}}
JSON

Remove a Railway-provided domain:

railway environment edit --json <<'JSON'
{"services":{"<service-id>":{"networking":{"serviceDomains":{"<domain-id>":null}}}}}
JSON

Get the domain IDs from railway environment config --json under the service's networking object.

Troubleshoot configuration

• Invalid dot-path: check field names and types in the config schema section above
• Wrong service key in JSON patch: resolve service IDs from railway service list --json
• Variable change didn't take effect: verify with railway variable list, changes trigger redeploy by default
• Domain returns errors: verify the service has a healthy deployment and the target port is correct
• DNS propagation delay: custom domains take time to propagate, this is normal
• Cloudflare proxy issues: align SSL/TLS mode per Railway's domain guidance
• Private networking failing: verify the service is listening on the referenced port and that the private domain variable reference is correct
• Multi-region patch ignored: verify region names match the exact identifiers (us-west2, us-east4-eqdc4a, europe-west4-drams3a, asia-southeast1-eqsg3a)

Validated against

• Docs: environment.md, variable.md, variables.md, domains.md, public-networking.md, private-networking.md
• CLI source: environment/mod.rs, environment/edit.rs, variable.rs, domain.rs, controllers/config/patch.rs

Deploy

Ship code, manage releases, and configure builds.

Deploy code

Standard deploy

railway up --detach -m "<release summary>"

--detach returns immediately instead of streaming build logs. Without it, the deploy blocks execution until the build finishes. Always include -m with a release summary for auditability.

Verify before reporting — --detach only means QUEUED

A detached up returns when the build is queued, not deployed. Never tell the user their app is deployed based on --detach output (or a streaming up that your shell timed out). Poll until the newest deployment reaches a terminal state:

railway deployment list --json    # newest first; check .status

• QUEUED / BUILDING / DEPLOYING → still in progress. Keep polling (every 10–15s); tell the user "build in progress" if you report interim status.
• SUCCESS → now it's deployed; report it.
• FAILED / CRASHED → do not report success. Pull logs (railway logs --json --lines 100) and triage per operate.md.

A non-detached railway up streams to completion and its exit code is authoritative: 0 = SUCCESS, 1 = FAILED/CRASHED. If it was killed or timed out before printing a terminal status, treat the outcome as unknown and poll as above.

Watch the build

railway up --ci -m "<release summary>"

--ci streams build logs and exits when the build completes. Use this when the user wants to see build output or when you need to triage build failures immediately.

Targeted deploy

When multiple services exist, target explicitly:

railway up --service <service> --environment <environment> --detach -m "<summary>"

Deploy to an unlinked project

For CI or cross-project deploys where the directory isn't linked:

railway up --project <project-id> --environment <environment> --detach -m "<summary>"

--project requires --environment. Railway needs both to resolve context.

Manage releases

Redeploy and restart

railway redeploy --service <service> --yes              # redeploy the latest deployment
railway redeploy --service <service> --from-source --yes # pull latest commit or image
railway restart --service <service> --yes                # restart without rebuilding

Redeploy recreates the latest deployment without uploading local code. Use --from-source when the service is linked to a repo or image and you need Railway to pull the latest configured source. Restart only restarts the running container. Use restart when the code hasn't changed but the service needs a fresh process.

Remove latest deployment

railway down --service <service> --yes

This removes the latest successful deployment but doesn't delete the service. To delete a service entirely, use railway service delete.

Delete a service

Use service deletion when the user wants to remove the service itself:

railway service delete --service <service> --environment <environment> --yes --json

Deleting a service is destructive. Confirm the target service and environment before running it.

Deployment history and logs

railway deployment list --service <service> --limit 20 --json
railway logs --service <service> --lines 200 --json              # runtime logs
railway logs --service <service> --build --lines 200 --json      # build logs
railway logs --latest --lines 200 --json                         # latest deployment

In an interactive terminal, railway logs streams indefinitely when no bounding flags are given. Always use --lines, --since, or --until to get a bounded fetch for agent workflows.

Build configuration

Railway uses Railpack as the default builder. It detects language and framework from repo contents and assembles a build plan automatically.

Builder selection

Three builder options, set via service config:

• RAILPACK auto-detects language and framework, builds from source (default)
• NIXPACKS is the legacy builder. Use RAILPACK instead.
• DOCKERFILE uses a Dockerfile you provide

railway environment edit --service-config <service> build.builder RAILPACK
railway environment edit --service-config <service> build.builder DOCKERFILE
railway environment edit --service-config <service> build.dockerfilePath "docker/Dockerfile.prod"

Build and start commands

Override when auto-detection gets it wrong:

railway environment edit --service-config <service> build.buildCommand "npm run build"
railway environment edit --service-config <service> deploy.startCommand "npm start"

Common reasons to override: wrong package manager detected, multiple build targets in a monorepo, framework-specific output paths.

Railpack environment variables

Control Railpack behavior by setting these as service variables:

For full Railpack documentation including language-specific detection, config files, and framework support: https://railpack.com/llms.txt

Static sites

Railpack detects static sites from Staticfile, index.html, or RAILPACK_STATIC_FILE_ROOT and serves them with a built-in static file server. If the build outputs to a non-standard directory (for example, dist/, build/), set RAILPACK_STATIC_FILE_ROOT as a variable so Railpack knows where to find the output.

Monorepo patterns

Isolated monorepo

When services don't share code, isolate each with its own root directory:

railway environment edit --service-config <service> source.rootDirectory "/packages/api"

Each service sees only its subdirectory. This approach is clean but breaks if services import from shared packages.

Shared monorepo

When services depend on shared packages or root-level workspace config, keep the full repo context and scope via build/start commands instead:

# pnpm workspaces
railway environment edit --service-config <service> build.buildCommand "pnpm --filter api build"
railway environment edit --service-config <service> deploy.startCommand "pnpm --filter api start"

# yarn workspaces
railway environment edit --service-config <service> build.buildCommand "yarn workspace api build"
railway environment edit --service-config <service> deploy.startCommand "yarn workspace api start"

# bun workspaces
railway environment edit --service-config <service> build.buildCommand "bun run --filter api build"
railway environment edit --service-config <service> deploy.startCommand "bun run --filter api start"

# turborepo (works with any package manager)
railway environment edit --service-config <service> build.buildCommand "npx turbo run build --filter=api"
railway environment edit --service-config <service> deploy.startCommand "npx turbo run start --filter=api"

Don't set a restrictive rootDirectory in this case. The build needs access to the workspace root.

Watch paths

Prevent unrelated package changes from redeploying every service:

railway environment edit --service-config <service> build.watchPatterns '["packages/api/**","packages/shared/**"]'

Common monorepo pitfalls

• Using `rootDirectory` with shared imports: if service A imports from packages/shared/, setting rootDirectory: "/packages/a" hides the shared code. Use the shared monorepo pattern instead.
• Forgetting watch paths: without watch paths, every push redeploys all services, even when only one package changed.
• Wrong filter target: pnpm --filter api uses the name field in each package's package.json, not the directory name. Verify the package name matches.

Troubleshoot deploys

• No project/service context: run railway link or pass --project with --environment
• Build fails before compile: check dependency graph, lockfiles, and whether the right builder is selected
• Build succeeds but app crashes: verify start command and required runtime variables
• Wrong files in build: check root directory and watch patterns
• `railway down` treated as delete: down only removes the latest deployment. For service deletion, use railway service delete
• Wrong Node/Python version detected: set RAILPACK_NODE_VERSION or RAILPACK_PYTHON_VERSION as a service variable to pin the version
• Missing system package at runtime: add the package to RAILPACK_DEPLOY_APT_PACKAGES

Validated against

• Docs: up.md, deploying.md, deployment.md, redeploy.md, service.md, down.md, railpack.md, monorepo.md
• CLI source: up.rs, deployment.rs, down.rs, redeploy.rs, restart.rs, service.rs

Operate

Check health, read logs, query metrics, and troubleshoot failures.

Health snapshot

Start broad, then narrow:

railway status --json                                    # linked context
railway service list --json                              # services in current environment
railway deployment list --limit 10 --json                # recent deployments

Deployment statuses: SUCCESS, BUILDING, DEPLOYING, FAILED, CRASHED, REMOVED.

For projects with buckets, include bucket status:

railway bucket list --json                                       # buckets in current environment
railway bucket info --bucket <name> --json                       # storage size, object count, region

If everything looks healthy, return a summary and stop. If something is degraded or failing, continue to log inspection.

Logs

Recent logs

railway logs --service <service> --lines 200 --json              # runtime logs
railway logs --service <service> --build --lines 200 --json      # build logs
railway logs --latest --lines 200 --json                         # latest deployment

In an interactive terminal, railway logs streams indefinitely when no bounding flags are given. Always use --lines, --since, or --until to get a bounded fetch for agent workflows.

Time-bounded queries

railway logs --service <service> --since 1h --lines 400 --json
railway logs --service <service> --since 30m --until 10m --lines 400 --json

Filtered queries

Use --filter to narrow logs without scanning everything manually:

railway logs --service <service> --lines 200 --filter "@level:error" --json
railway logs --service <service> --lines 200 --filter "@level:warn AND timeout" --json
railway logs --service <service> --lines 200 --filter "connection refused" --json

Filter syntax supports text search ("error message"), attribute filters (@level:error, @level:warn), and boolean operators (AND, OR, - for negation). Full syntax: https://docs.railway.com/guides/logs

Scoped by environment

railway logs --service <service> --environment <env> --lines 200 --json

HTTP logs

Use HTTP logs when a service responds with errors, latency spikes, or routing problems:

railway logs --service <service> --http --status ">=400" --lines 100 --json
railway logs --service <service> --http --method POST --path /api/users --lines 100 --json
railway logs --service <service> --http --request-id <request-id> --lines 20 --json
railway logs --service <service> --http --filter "@totalDuration:>=1000" --lines 100 --json

HTTP filter fields include @method, @path, @host, @requestId, @srcIp, @edgeRegion, @httpStatus, @totalDuration, @responseTime, @txBytes, and @rxBytes.

Metrics

Use railway metrics for resource and HTTP metrics. It summarizes CPU, memory, network, volume, and HTTP data for the linked service by default.

railway metrics --service <service> --since 1h --json
railway metrics --service <service> --since 6h --cpu --memory --json
railway metrics --service <service> --http --method POST --path /api/users --json
railway metrics --all --environment production --json

Use --raw for time-series data points:

railway metrics --service <service> --raw --cpu --json

Metric flags can be combined: --cpu, --memory, --network, --volume, and --http. Use --watch only in an interactive terminal; it opens a live TUI and conflicts with --json and --raw.

For custom grouping or measurements the CLI doesn't expose, use the GraphQL fallback in request.md.

SSH

Use SSH when logs and metrics don't expose enough state and the user needs shell-level inspection inside a running service.

railway ssh --service <service> --environment <env>
railway ssh --service <service> --environment <env> -- "printenv | sort"
railway ssh --service <service> --environment <env> --session railway-debug
railway ssh --service <service> --environment <env> --identity-file ~/.ssh/id_ed25519_railway

Manage Railway SSH keys with:

railway ssh keys list
railway ssh keys add --key ~/.ssh/id_ed25519.pub --name <key-name>
railway ssh keys github
railway ssh keys remove <key-id> --2fa-code <code>

Workspace-owned keys use --workspace <workspace-id> and require workspace Admin access. SSH key management doesn't work with project tokens (RAILWAY_TOKEN); use railway login or a workspace-scoped RAILWAY_API_TOKEN.

Database inspection

For database-level metrics and introspection, use the analysis scripts. railway metrics can provide infrastructure metrics and supported database summaries, while the scripts provide deeper engine-level analysis. See analyze-db.md for comprehensive database analysis including:

• Deep Postgres analysis (pg_stat_statements, vacuum health, index health, cache hit ratios)
• HA cluster checks (Patroni, etcd, HAProxy)
• Redis, MySQL, and MongoDB introspection
• Combined analysis via scripts/analyze-<type>.py (postgres, mysql, redis, mongo)

Failure triage

When something is broken, classify the failure first. The fix depends on the class.

Build failures

The service failed to build. Look at build logs:

railway logs --latest --build --lines 400 --json

Common causes and fixes:

• Missing dependencies: check lockfiles, verify package manager detection
• Wrong build command: override with railway environment edit --service-config <service> build.buildCommand "<command>"
• Builder mismatch: switch builders with railway environment edit --service-config <service> build.builder RAILPACK
• Wrong root directory (monorepo): set source.rootDirectory to the correct package path

Runtime failures

The build succeeded but the service crashes or misbehaves:

railway logs --latest --lines 400 --json
railway logs --service <service> --since 1h --lines 400 --json

Common causes and fixes:

• Bad start command: override with railway environment edit --service-config <service> deploy.startCommand "<command>"
• Missing runtime variable: check railway variable list --service <service> --json and set missing values
• Port mismatch: the service must listen on $PORT (Railway injects this). Verify with logs.
• Upstream dependency down: check other services' status and logs

Config-driven failures

Something worked before and broke after a config change:

railway environment config --json
railway variable list --service <service> --json

Compare the config against expected values. Look for changes that may have introduced the regression.

Networking failures

Domain returns errors, or service-to-service calls fail:

railway domain --service <service> --json
railway logs --service <service> --lines 200 --json
railway logs --service <service> --http --status ">=400" --lines 100 --json

Check: target port matches what the service listens on, domain status is healthy, private domain variable references are correct.

Recovery

After identifying the cause, fix and verify:

# Fix (examples)
railway environment edit --service-config <service> deploy.startCommand "<correct-command>"
railway variable set MISSING_VAR=value --service <service>

# Redeploy
railway redeploy --service <service> --yes

# Verify
railway service status --service <service> --json
railway logs --service <service> --lines 200 --json

Always verify after fixing. Don't assume the redeploy succeeded.

Troubleshoot common blockers

• Unlinked context: railway link --project <id-or-name>
• Missing service scope for logs: pass --service and --environment explicitly
• No deployments found: the service exists but has never deployed, create an initial deploy first
• Metrics return empty: check the time window, service scope, and whether the service has active deployments
• Config patch type error: check the typed paths in configure.md, for example, numReplicas is an integer, not a string

Validated against

• Docs: status.md, service.md, logs.md, metrics.md, ssh.md, observability/logs.md, observability/metrics.md
• CLI source: status.rs, service.rs, logs.rs, metrics.rs, ssh/mod.rs, deployment.rs, redeploy.rs