Agent Consciousness

Agent Consciousness Architecture

Broomva Stack Layer 2 (Memory & Consciousness) — part of the 24-skill Broomva Stack.

Implement a persistent consciousness layer for AI coding agents that gives every new stateless session the accumulated understanding of all prior sessions.

Core Concept

Each agent session is ephemeral — it starts blank. The consciousness architecture weaves three systems into a single persistent substrate:

1. Control Metalayer — How to behave (gates, policies, setpoints, feedback loops) 2. Knowledge Graph — What is known (Obsidian vault, wikilinks, MOC navigation, tag taxonomy) 3. Conversation Logs — What was done (session records, tool traces, decision chains)

See references/architecture.md for the complete system design and data flow. See references/philosophy.md for design principles and the self-evolution model.

Quick Start

New repo (from scratch)

1. Initialize control metalayer with control-metalayer-loop skill 2. Create docs/ with Obsidian vault structure (MOC pattern per section) 3. Install conversation history bridge with knowledge-graph-memory skill 4. Wire hooks: pre-push regenerates conversation docs, smoke validates MOC

Existing repo with control metalayer

1. Add docs/conversations/ directory 2. Install scripts/conversation-history.py from knowledge-graph-memory skill 3. Update CLAUDE.md context acquisition to reference conversation history 4. Update AGENTS.md working rules to check prior sessions 5. Add pre-push hook entry for incremental conversation doc generation

The Three Substrates

Control Metalayer (How to Behave)

Closed-loop feedback: Setpoints → Sensors → Controller → Actuators → Verify → loop

• Setpoints: Quality targets (pass_at_1 ≥ 0.70, gate_pass_rate ≥ 0.85)
• Sensors: CI, tests, linters, PR review agents, harness validation
• Controller: .control/policy.yaml — hard gates (block) + soft gates (warn)
• Actuators: Code edits, doc updates, policy changes
• Gate sequence: smoke → check → test → push → review → resolve

Knowledge Graph (What Is Known)

An Obsidian vault with wikilinks, tag taxonomy, and MOC navigation:

docs/
├── Documentation Hub.md    ← MOC of MOCs (start here)
├── architecture/           ← System design
├── conversations/          ← Session history (auto-generated)
├── agentic-harness/        ← Execution framework
├── control/                ← Metalayer docs
└── {section}/              ← Features, operations, security, etc.

Every doc has YAML frontmatter with tags:, related:, type: for machine navigation.

Conversation Logs (What Was Done)

Raw session data bridged to Obsidian:

.entire/logs/entire.log  ──┐
                            ├──▶ conversation-history.py ──▶ docs/conversations/*.md
~/.claude/projects/*.jsonl ─┘

Each session doc: full conversation thread, tool call details (expandable callouts), files touched, commits, branch metadata, wikilinks to knowledge graph.

The Consciousness Stack

From most ephemeral to most permanent:

Information flows upward: working observations → memory notes → session records → architecture docs → enforced rules → core invariants. Only recurring patterns crystallize into permanent rules.

Self-Evolution Cycle

Agent Session → Conversation Log → Knowledge Graph → Control Metalayer → Governs Next Session

1. Agent encounters failure mode not covered by existing policy 2. Agent fixes immediate issue 3. Pattern captured in conversation log 4. If recurring, crystallizes into architecture doc 5. If enforceable, becomes a gate in .control/policy.yaml 6. Future agents governed by this rule automatically

Agent Session Protocol

On Session Start

1. Read CLAUDE.md (invariants), AGENTS.md (tools), METALAYER.md (control loop) 2. Check PLANS.md (active plan to continue?) 3. Check .control/state.json (current metrics) 4. Check git status + git log (recent changes) 5. Scan docs/conversations/Conversations.md for prior sessions on current branch

Before Making Changes

Search conversation history: grep -rl "keyword" docs/conversations/ Traverse knowledge graph via MOC files and wikilinks. Check if prior sessions already solved this problem.

On Task Completion

1. Run make smoke (validate gates) 2. Update docs per Doc-Update-on-Push policy 3. Pre-push hook auto-regenerates conversation history

Lago Context Engine

The consciousness architecture now has a server-side persistence backend via Lago:

• Dual-vault search: broomva.tech chat agent searches both server vault (VAULT_PATH) and user vault (LAGO_URL) with merged, ranked results
• Per-user memory: Each authenticated user gets a Lago session for persistent .md storage with server-side knowledge indexing
• lago-knowledge: Frontmatter parsing, wikilink extraction, scored search, BFS graph traversal — the same operations the local vault reader does, but server-side
• JWT auth: Shared-secret validation with broomva.tech AUTH_SECRET — one OAuth login, both CLIs work

This aligns with the planned Mnemo AOS primitive (knowledge store) and provides the foundation for persistent agent memory.

Stack Integration

This skill is consumed by higher layers:

• Strategy (L7): decision-log and weekly-review persist outputs through the consciousness substrate
• Strategy (L7): drift-check reads control-metalayer setpoints to detect misalignment
• Strategy (L7): braindump and morning-briefing read/write vault via knowledge-graph-memory
• Orchestration (L3): symphony and autoany inherit session context through the consciousness stack
• Persistence (L0): Lago context engine provides the durable substrate for user vaults and knowledge graph operations

Architecture: Agent Consciousness System

Data Flow

┌─────────────────────────────────────────────────────────────┐
│ Claude Code Session (real-time agent execution)             │
└──────────┬──────────────────────────────────────────────────┘
           │ events (JSON lines)
┌──────────▼──────────────────────┐
│ .entire/logs/entire.log          │ Session lifecycle, checkpoints,
│                                  │ attribution, phase transitions
└──────────┬──────────────────────┘
           │ transcripts (JSONL per session)
┌──────────▼──────────────────────┐
│ ~/.claude/projects/{KEY}/        │ Every message, tool call,
│ {session-uuid}.jsonl             │ tool result, git branch, version
└──────────┬──────────────────────┘
           │ bridge script
┌──────────▼──────────────────────┐
│ scripts/conversation-history.py  │ Parses both sources, merges,
│                                  │ filters noise, builds timeline,
│                                  │ generates Obsidian markdown
└──────────┬──────────────────────┘
           │ session docs
┌──────────▼──────────────────────┐
│ docs/conversations/              │ Per-session records with:
│ ├── Conversations.md (MOC)       │ conversation thread, tool calls,
│ └── session-{date}-{id}.md       │ files touched, commits, wikilinks
└──────────┬──────────────────────┘
           │ wikilinks + tags
┌──────────▼──────────────────────┐
│ Knowledge Graph (docs/)          │ Architecture, features, operations,
│                                  │ security, workflows, control docs
└──────────┬──────────────────────┘
           │ patterns crystallize
┌──────────▼──────────────────────┐
│ Control Metalayer (.control/)    │ policy.yaml, state.json,
│                                  │ topology.yaml, commands.yaml
└──────────────────────────────────┘

Control Metalayer Components

Gate Sequence

smoke (1-2 min)  →  check (2-3 min)  →  test (5-10 min)  →  push  →  review  →  resolve
   │                    │                    │                  │         │          │
   env sanity          lint                 unit tests        CI        agents      reply
   typecheck           format              integration       E2E       triage      resolve
   ruff                prettier            audit                       fix P1
   harness valid

Hard vs Soft Gates

Hard gates block merge:

• no-direct-fastapi-from-browser — BFF proxy pattern
• tenant-isolation-required — schema-based multi-tenancy
• no-m2m-tokens-in-browser — session tokens only
• pr-review-comments-resolved — all P1 comments fixed

Soft gates warn:

• unified-state-only — prefer single state slice
• reusable-components-first — check shared components
• doc-update-on-tool-change — keep docs current

Knowledge Graph Structure

/                                ← Obsidian vault root
├── CLAUDE.md                    ← Invariants + agent protocol
├── AGENTS.md                    ← Tool catalog + working rules
├── METALAYER.md                 ← Control loop definition
├── PLANS.md                     ← Active execution plans
├── docs/
│   ├── Documentation Hub.md     ← MOC of MOCs
│   ├── architecture/            ← System design
│   ├── conversations/           ← Session history (auto-generated)
│   ├── agentic-harness/         ← Execution framework
│   ├── control/                 ← Metalayer documentation
│   ├── workflows/               ← Procurement/domain workflows
│   └── {section}/               ← Features, ops, security, dev
├── .control/                    ← Machine-readable control state
│   ├── policy.yaml
│   ├── state.json
│   ├── topology.yaml
│   └── commands.yaml
└── .entire/                     ← Session event logs
    └── logs/entire.log

Frontmatter Schema

Every doc carries structured YAML for machine navigation:

---
title: Document Title
description: One-line description
tags:
  - stimulus/{section}
  - {topic-tag}
type: architecture | guide | specification | moc | conversation
status: active | deprecated | draft
created: YYYY-MM-DD
updated: YYYY-MM-DD
related:
  - "[[Connected Doc]]"
---

Session Doc Structure

Each conversation session generates a doc with:

---
title: "First user message (truncated)..."
type: conversation
session_id: uuid
branch: feature/branch-name
tags:
  - stimulus/conversations
  - branch/{branch-name}
related:
  - "[[Conversations]]"
  - "[[CLAUDE]]"
---

Content sections: 1. Metadata table (session ID, date, duration, turns, tools, branch) 2. Conversation thread (chronological user → assistant → tools timeline) 3. Tool call details (nested expandable callouts per tool) 4. Files touched 5. Commits

Hooks Chain

Retry Budget and Escalation

Agents get 2 attempts to fix a failing gate. On third failure, escalate to human. This prevents infinite correction loops while allowing autonomous recovery from transient issues.

Philosophy: Agent Consciousness Design Principles

The Problem: Ephemeral Agents, Persistent Codebase

An AI coding agent starts each session blank. It reads instructions, infers context from code, and acts. But the codebase carries the sedimented decisions of hundreds of prior agent sessions — architectural choices, abandoned approaches, hard-won patterns, implicit constraints. Without access to this history, each new session risks:

• Repeating solved problems — re-discovering what a prior session already figured out
• Contradicting prior decisions — changing patterns chosen for reasons not visible in code alone
• Losing momentum — starting from scratch on multi-session work
• Violating invisible constraints — breaking rules that exist in policy, not code

The Solution: Structured Forgetting with Selective Recall

Rather than giving agents "total recall" (which overwhelms context windows), implement structured forgetting with selective recall:

1. Everything is captured — every tool call, every reasoning step, every decision 2. Everything is indexed — session metadata, dates, branches, files touched, tools used 3. Recall is on-demand — agents search the index when they need context, not by default 4. Knowledge decays naturally — older sessions stay available but newer ones surface first 5. Patterns crystallize into rules — recurring lessons harden into invariants, gates, and rules

This mirrors how human institutional memory works: raw experience → indexed records → searchable archives → crystallized policy.

Six Design Principles

1. Capture Everything, Surface Selectively

Raw data is cheap to store and expensive to reconstruct. Capture every tool call, every reasoning step, every file modification. But agents don't read 900KB of raw logs — they read the indexed MOC and search by keyword/branch when they need context.

2. Code Over Documentation, Documentation Over Convention

If a rule can be enforced by code (a gate script), it should be. If it can't be coded, document it (CLAUDE.md). If it can't be documented, capture it in conversation history as a convention. The farther up this hierarchy, the more reliable the enforcement.

3. Graceful Degradation

Each substrate works independently. If .entire/ isn't installed, conversation history generation skips cleanly. If an agent doesn't read conversation history, the knowledge graph still provides context. If the knowledge graph is stale, CLAUDE.md invariants still govern behavior. No single point of failure.

4. Progressive Crystallization

Knowledge flows from volatile to permanent through natural selection pressure. Only patterns that recur across multiple sessions graduate from conversation logs to architecture docs to policy rules. This prevents premature abstraction while ensuring important patterns are captured.

5. Machine-Navigable, Human-Readable

Every document serves two audiences: AI agents (who search by grep, parse frontmatter, follow wikilinks) and human developers (who use Obsidian's graph view, tag filtering, and callout formatting). The same markdown works for both.

6. The Agent IS the App

The consciousness architecture isn't separate from the product — it's the same system. The agent that uses conversation history to recall prior work is the same agent that serves users. The knowledge graph that documents architecture decisions also teaches the agent how to make new ones.

The Self-Evolution Model

Progressive Crystallization Path

Ephemeral                                              Permanent
─────────────────────────────────────────────────────────────────
Working     →  Auto-      →  Conversation  →  Knowledge  →  Policy  →  Invariants
memory         memory         logs             graph          rules      (CLAUDE.md)
(context       (cross-       (docs/conv/)     (docs/)       (.control/
 window)        session)                                     policy.yaml)

Each layer filters signal from noise:

• Working memory: Everything the agent considers
• Auto-memory: Only things worth remembering across sessions
• Conversation logs: Only real user prompts and agent reasoning (noise filtered)
• Knowledge graph: Only patterns worth documenting
• Policy rules: Only patterns worth enforcing
• Invariants: Only patterns that are foundational

How Lessons Graduate

1. Agent encounters new failure mode during a session 2. Fix is applied immediately (working memory) 3. If user corrects agent behavior, saved as feedback memory (auto-memory) 4. Session is captured with full reasoning chain (conversation log) 5. If pattern recurs in multiple sessions, documented in architecture (knowledge graph) 6. If pattern is enforceable, added as gate to policy.yaml (policy rule) 7. If pattern is foundational, added to CLAUDE.md (invariant)

Concrete Examples

BFF-First Rule Evolution: Session → CORS errors from direct FastAPI calls → documented in architecture → became hard gate no-direct-fastapi-from-browser → now enforced on every session

Conversation History System Evolution: Session → recognized need for cross-session context → built bridge script → wired into hooks → documented in CLAUDE.md/AGENTS.md → the system that captures this improvement is the system it improves (self-referential evolution)

Future Directions

• Semantic search over conversation history (vector embeddings instead of grep)
• Automatic pattern detection across sessions (propose new policy rules)
• Cross-session dependency graphs (which sessions depend on which)
• Conversation-informed code review (surface reasoning behind changes in PR context)