Ops Rollback

ops-rollback

The reverse-the-deploy skill. Reads the rollback contract that ops-deploy recorded, snapshots failed-state evidence, executes the reversion, verifies recovery, updates the deploy log. Distinct from ops-incident (broader incident process — rollback may be one of several tools an incident uses) and from ops-deploy (forward action with gates — this skill is its explicit reverse).

Trigger phrases (when to load this skill)

Load when the conversation matches any of:

• Direct verb: operator says "rollback", "revert the deploy", "go back to last good", "undo the release", "back out the change", "deploy was bad — revert".
• Within an incident: ops-incident Operation 1 identified that the active deploy is the cause; rollback is the chosen restoration mechanism.
• Post-deploy regret: deploy succeeded smoke tests but is now showing problems in metrics — the operator wants to revert before the problem grows.

Do NOT load when:

• Multiple things are broken with unclear cause — load ops-incident first; rollback is one option there.
• The current deploy is the right code; rollback would just regress to a different problem — load ops-incident to find a forward fix.
• Operator wants to revert a single commit (not a deploy) — that's a git revert workflow, not this skill.
• No prior deploy was recorded — there's nothing to roll back TO; load ops-incident for restoration via other means.
• The "rollback" is a config change (profile switch, env var revert) — load config-deploy for surgical config reversion.

Operations

This skill has 4 named operations. Execute in strict order — preservation of evidence comes BEFORE reversion.

Operation 1: Identify the rollback target + verify it's good

Trigger: skill loaded; operator named the environment to roll back (or single-environment deployment is implied).

Process:

1. Find the rollback contract recorded by the prior ops-deploy:

   ls /tmp/predeploy-*-<env>.txt 2>/dev/null
   tail -5 docs/DEPLOY-LOG.md   # last successful deploys

Capture: previous commit SHA, previous image tags, previous profile name. 2. If no contract exists (skill being run cold without a prior ops-deploy):

• Look for the last successful deploy in docs/DEPLOY-LOG.md to identify the SHA.
• If no log either: identify last-known-good by git log + operator confirmation. Don't pick blindly.

3. Verify the rollback target is ACTUALLY good (don't roll back to another broken state):

   git log -1 --format="%H %s %ci" <target-sha>
   # confirm: this SHA was the deployed-and-smoked state, not a half-deployed step

If the target's deploy log entry says status=ok, it's verified. If status=failed or no entry, escalate — operator must pick a different target. 4. Compute what specifically will change on rollback:

   git diff --stat <target-sha>..HEAD       # files that will revert
   git log --oneline <target-sha>..HEAD     # commits that will be undone

Surface this to operator: "rolling back N commits across M files; the changes you're undoing are: [list]". Get explicit confirmation before proceeding. 5. Identify migration / state implications — the data class of "irreversible":

• Any DB schema migration in the diff? (AICP has no SQL; fleet projects might.) If yes, rollback may strand data. Surface and pause.
• Any config that changed file paths, retention windows, encryption keys? Rolling back doesn't unmove files or undelete state.
• Any LocalAI Collection re-indexing? Rolling back doesn't undo that work but may make the new index incompatible.

Quality bar (Operation 1 done when):

• [ ] Rollback target SHA + image tags + profile name explicitly identified.
• [ ] Target verified as last-known-good (deploy log entry or operator confirmation).
• [ ] Diff/log preview shown to operator with commit list + file count.
• [ ] Operator explicitly confirmed (not implied) — rollback is destructive of forward work.
• [ ] State-irreversibility implications surfaced: migrations, file moves, re-indexes.

Operation 2: Preserve failed-state evidence

Trigger: Operation 1 confirmed; rollback is about to mutate state.

Process:

1. Snapshot the failed-state — same shape as ops-incident Op1 (this skill is reusing that pattern):

   ts=$(date -u +%FT%TZ); mkdir -p /tmp/rollback-$ts
   docker compose ps > /tmp/rollback-$ts/compose-ps.txt
   docker compose logs --tail 500 > /tmp/rollback-$ts/logs.txt 2>&1
   .venv/bin/aicp --check > /tmp/rollback-$ts/aicp-check.txt 2>&1
   .venv/bin/aicp --dlq-status > /tmp/rollback-$ts/dlq.txt 2>&1
   git log --oneline -10 > /tmp/rollback-$ts/recent-commits.txt
   git diff <target-sha>..HEAD > /tmp/rollback-$ts/diff-being-undone.diff

2. Capture the FAILING commit SHA (current HEAD before rollback) — needed for ops-incident to diagnose root cause AFTER service is restored:

   git rev-parse HEAD > /tmp/rollback-$ts/failed-sha.txt

3. If runtime state is at risk (e.g., DLQ entries, in-flight tasks): copy them to the snapshot dir before any restart:

   cp -r ~/.aicp/dlq/$(date +%Y-%m-%d).jsonl /tmp/rollback-$ts/ 2>/dev/null
   cp .aicp/state.yaml /tmp/rollback-$ts/ 2>/dev/null

Quality bar (Operation 2 done when):

• [ ] /tmp/rollback-<ts>/ exists with logs + ps + check + dlq + recent-commits + diff.
• [ ] Failed SHA captured to file (preserves identity through rollback).
• [ ] Runtime state copied if at risk.
• [ ] Snapshot completed BEFORE any rollback mutation.

Operation 3: Execute the rollback

Trigger: Operation 2 evidence preserved.

Process:

1. Move the code:

   git checkout <target-sha>
   # or for branch-tip rollback:
   git reset --hard <target-sha>   # only if operator explicitly authorized destructive (rare; checkout is preferred)

Prefer git checkout (detached HEAD) for safety; the operator can later branch off it. reset --hard only on explicit authorization — it can lose unpushed commits in environments where this checkout could lose work. 2. Move the profile if rollback contract included one:

   make profile-use PROFILE=<previous-profile>

3. Rebuild + restart services:

   docker compose build 2>&1 | tail -10
   docker compose up -d 2>&1 | tail -10

Use up -d (not --no-deps) — env vars and shared config likely changed alongside the rollback. 4. Watch logs for first 30 seconds — same pattern as ops-deploy Op2:

   docker compose logs -f --tail 50 --since 30s &
   sleep 30; kill $!

5. If rollback itself fails (rolled-back code doesn't boot): this is an emergency. Either roll FURTHER back (to an even earlier known-good), or invoke ops-incident because the system has no known-good.

Quality bar (Operation 3 done when):

• [ ] Code moved to target SHA (verified by git rev-parse HEAD).
• [ ] Profile activated if part of rollback contract.
• [ ] Compose services rebuilt and up.
• [ ] No service in Restarting state.
• [ ] No FATAL or "Exited" entries in first-30s logs.

Operation 4: Smoke + log + handoff

Trigger: Operation 3 reverted state running.

Process:

1. Smoke test the rolled-back system — same shape as ops-deploy Op3:

   docker compose ps --format json | jq -r '.[] | "\(.Name): \(.State) (\(.Health // "no-healthcheck"))"'
   .venv/bin/aicp --check 2>&1 | head -20
   .venv/bin/aicp --backend local --prompt "say ok" 2>&1 | tail -5

All services healthy + golden-path inference responds = rollback succeeded. 2. Update docs/DEPLOY-LOG.md with the rollback entry:

   <ts> deploy=<env> sha=<target-sha> status=rollback by=<op> reverted-from=<failed-sha> reason="<one-line>"

3. Notify operator surfaces:

• ntfy: ROLLBACK <env> reverted <failed-sha> → <target-sha> dur=<minutes>
• Mission Control / standing-orders update if fleet was affected.

4. Hand off to ops-incident for diagnosis:

• The failed-state snapshot in /tmp/rollback-<ts>/ is the evidence.
• The diff in diff-being-undone.diff is the change set to investigate.
• Tell operator explicitly: "service restored via rollback; load ops-incident to identify root cause and author durable fix before re-deploying."

Quality bar (Operation 4 done when):

• [ ] All services smoke-test green at the rolled-back state.
• [ ] docs/DEPLOY-LOG.md updated with rollback entry naming both target and failed SHAs.
• [ ] Notification sent (ntfy / MC / explicit "no notification surface configured").
• [ ] ops-incident handoff explicitly stated to operator with snapshot path.
• [ ] No half-rolled-back state remaining (no service still on the failed image, no profile still on failed config).

Gotchas (known failure modes — read before doing)

Gotcha 1: Rolling back to another broken state

Operator says "rollback to last week's deploy". Skill picks the SHA from a week ago — but that deploy was ALSO rolled back at the time, and the status=ok entry is the one from 9 days ago. Skill rolls back to a known-broken target; service stays broken; operator now has TWO incidents to debug.

The rule: Operation 1 step 3 verifies the target by reading docs/DEPLOY-LOG.md for status=ok. If the most recent entry for the target SHA is status=failed or status=rollback, that SHA is NOT a valid rollback target — keep walking back the log until a confirmed-good is found, or escalate to operator.

Gotcha 2: Rollback wipes runtime state

Failed deploy added a new field to a config; operator's runtime now references entries with that field. Rollback reverts the config; on restart, the runtime crashes parsing entries that the rolled-back code doesn't understand. Rollback "succeeded" code-wise but the system is now broken in a new way.

The rule: Operation 1 step 5 surfaces state-irreversibility implications BEFORE rollback. If runtime state references the new code's shape, rollback alone isn't sufficient — either migrate state back or accept data loss. Don't rollback in the dark.

Gotcha 3: Mutating before snapshotting

Same anti-pattern as ops-incident Gotcha 1: skill jumps to git checkout <target> without first capturing the failed-state. Logs that lived in the running container are gone after restart; the failed-state diff is unrecoverable; root cause is now a guessing game.

The rule: Operation 2 is non-negotiable. Snapshot to /tmp/rollback-<ts>/ BEFORE any mutation in Operation 3. The five-second cost preserves the entire evidence base for ops-incident to use.

Gotcha 4: git reset --hard loses unpushed forward work

Skill defaults to git reset --hard <target-sha>. But the operator had two un-pushed commits on the failed branch (a half-finished hotfix attempt that hadn't been committed cleanly). Reset destroys them; the operator's last 30 minutes of work are gone with no recovery.

The rule: Operation 3 step 1 prefers git checkout <sha> (detached HEAD). reset --hard is reserved for explicit operator authorization and is never the default. If the operator wants the branch tip moved, they say so AFTER the system is restored — restoration first, history surgery second.

Gotcha 5: Rollback declared successful, root cause unidentified

Service is back; deploy log shows the rollback; operator moves on. But nobody loaded ops-incident — root cause is unknown, and the same broken deploy will be re-attempted next week with the same failure mode.

The rule: Operation 4 step 4 makes the handoff EXPLICIT. The rollback skill's job ends with "service restored AND ops-incident invoked-or-explicitly-deferred". If operator defers ("we'll diagnose later"), the deferral is logged in docs/DEPLOY-LOG.md (status=rollback diag-deferred), so the next deploy attempt has a record that the cause is unknown.

Reference exemplars

The Extension Standards reference exemplars for skills are the second brain's model-builder and wiki-agent skills — see ~/devops-solutions-research-wiki/skills/. This skill follows their structure: trigger phrases, multiple named operations, per-operation Process + Quality bar, Gotchas with detection + rule + reasoning.

Domain context

This skill operates in the backend-ai-platform-python domain. AICP rollback is mostly straightforward — code+config+profile, no schema migrations to worry about — because there's no SQL database. Sister fleet projects (openfleet, dspd, nnrt) may have schema migrations; this skill flags them in Op1 step 5 and stops if rollback would strand data. The rollback contract relies on ops-deploy having recorded predeploy state correctly; if the skill is run cold (no contract), docs/DEPLOY-LOG.md is the fallback source of truth. AICP profile YAMLs are part of the rollback contract — a deploy that activated default -> personal is incomplete to roll back without also reverting the profile.

Related skills