Indexion Refactor

indexion refactor — Codebase Refactoring

Detect and eliminate duplication at three levels — textual, structural, and conceptual — using indexion's analysis commands, then verify SoT is enforced.

When to Use

• After adding a new abstraction (type, module, API layer)
• After introducing a new file format or I/O boundary
• When a fix required touching 3+ files for the same reason
• When a "guard" or "skip" was added to work around a structural problem
• When opendir, ENOENT, or similar filesystem errors appear from unexpected paths
• When extracting shared code across packages
• When cleaning up after a refactor (removing trivial wrapper functions)
• Periodic SoT health check on a codebase

Three Levels of Duplication

Textual duplication is easy to find and fix. Conceptual duplication is the hardest and most dangerous — it produces no copy-paste matches but means changing one concept requires updating every scattered implementation.

Workflow

Phase 1: Clear textual duplication (plan refactor)

Start with high-confidence matches and work down.

# Step 1: Find 90%+ duplicates (high confidence)
indexion plan refactor --threshold=0.9 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  cmd/indexion/

indexion plan refactor --threshold=0.9 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  src/

Read the output in three sections:

Same-file duplicates (functions within one file at 90%+) are the highest-value targets — easiest to fix, clearest wins. Example: get_global_data_dir and get_global_cache_dir share 95% structure, extracted into resolve_os_dir.

# Step 2: Use grep to trace references before consolidating
indexion grep "TypeIdent:TfidfEmbeddingProvider" src/
indexion grep --semantic=name:is_whitespace src/

# Step 3: Fix, then re-run to confirm duplicates are gone
indexion plan refactor --threshold=0.9 --include='*.mbt' ...

# Step 4: Lower threshold and iterate
indexion plan refactor --threshold=0.85 --include='*.mbt' ...

`plan refactor` options:

What remains after cleanup (stop signals):

• Platform stubs (native.mbt / stub.mbt) — intentional platform branching
• Type method similarity (to_string on different types) — different types, same pattern
• CLI command boilerplate (command() functions) — @argparse API pattern, not duplication
• Semantic-but-different functions (is_disqualifying_keyword vs is_skip_token) — different purpose

Phase 2: Extract cross-package shared code (plan solid)

After cleaning within each directory, find code that should be shared across packages.

# Find overlap between two packages
indexion plan solid --from=src/a,src/b

# Specify extraction target
indexion plan solid --from=src/a,src/b --to=src/common

# Use tree edit distance for precise function-level matching
indexion plan solid --from=src/a,src/b --strategy=apted

# Higher threshold for stricter matching
indexion plan solid --from=src/a,src/b --threshold=0.95

# Filter files
indexion plan solid --from=src/a,src/b --include='*.mbt' --exclude='*_test.mbt'

plan solid differs from plan refactor:

`plan solid` options:

Workflow:

1. Run plan refactor on each directory individually first to clean internal duplication 2. Run plan solid --from=dirA,dirB to find cross-directory extraction candidates 3. Extract shared code following the plan's recommendations 4. Use indexion grep "TypeIdent:SharedType" to verify all references are updated

Phase 3: Remove unnecessary wrappers (plan unwrap)

After consolidation, clean up trivial delegation functions that add indirection without value.

# Step 1: Quick check
indexion grep --semantic=proxy src/

# Step 2: Detailed report
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Step 3: Preview changes (safe — no files modified)
indexion plan unwrap --dry-run --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Step 4: Apply fixes
indexion plan unwrap --fix --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Step 5: Run tests
moon test --target native

What gets detected: Functions whose body is a single function call with all arguments forwarded as simple identifiers — no control flow, no transforms.

// Detected (default) — trivial delegation
fn matches_pattern(text : String, pat : String) -> Bool {
  @glob.glob_match(text, pat)
}

// Excluded by default (use --all to include)
fn length(self : MyList) -> Int {
  self.items.length()    // self-delegation (encapsulation)
}
fn emit(value : String) -> Action {
  Emit(value)            // bare constructor
}

`plan unwrap` modes:

`plan unwrap` options:

Review before removing:

• Platform wrappers (FFI, @osenv_path) are abstraction layers, not accidental indirection
• Public API wrappers used by external packages — removing them is a breaking change
• Always `--dry-run` first

Phase 4: Detect concept-level duplication (explore + analysis)

This is the hardest level. Textual and structural tools won't find it because the code is different — but the concept is the same.

# Find which files share vocabulary (= work in the same concept domain)
indexion explore --threshold=0.4 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  src/ cmd/

Files at 40-60% similarity without structural duplication are concept neighbors — they use the same terms because they deal with the same domain.

For each high-similarity pair, ask: "What concept do they share, and who owns it?"

# Inspect shared vocabulary with tree structure comparison
indexion explore file_a.mbt file_b.mbt --threshold=0 --strategy=apted

Common patterns of concept leakage:

Phase 5: Consolidate into SoT

The module that defines the concept should be the only one that implements the logic.

Rules: 1. One concept, one module, one function. If "extract text from archive" appears in vfs.mbt, discover.mbt, and args.mbt, it belongs in vfs.mbt only. 2. Callers receive results, not ingredients. Don't export is_archive_spec + ArchiveSpec::from_spec + expand_archive separately. Export try_extract_archive_text(path, spec) -> String?. 3. Guards are symptoms, not fixes. if is_virtual_path(x) { skip } means virtual paths shouldn't reach here at all. Fix the source, not the sink. 4. Re-reading from disk what's already in memory is a concept leak. If SupportedFile.content holds the text, no downstream code should call @fs.read_file_to_string(file.path).

Phase 6: Verify

# Confirm textual duplication is gone
indexion plan refactor --threshold=0.9 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  src/ cmd/indexion/

# Confirm concept similarity is reduced
indexion explore file_a.mbt file_b.mbt --threshold=0

# Confirm wrappers are cleaned up
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Run tests
moon test --target native

After SoT consolidation:

• Textual similarity between the concept owner and its callers drops
• Callers become shorter (one API call instead of multi-step logic)
• The concept owner may grow, but it's the single place to change

Phase 7: Prove non-recurrence with tests

Write a test that structurally prevents the old pattern from recurring:

test "SoT: SupportedFile.path is always a real filesystem path" {
  // Create an archive, run load_supported_file_info
  // Assert: no path contains "!/"
  // Assert: every path passes @fs.path_exists
}

The test doesn't check behavior — it checks the SoT invariant.

Red Flags

"I need to add a guard here"

If you're adding if is_special_case(x) { skip } to a function that shouldn't receive special cases, the problem is upstream. The function's caller should never pass that value.

"It works but prints errors to stderr"

Stderr messages from C runtime (opendir: No such file or directory) mean invalid data reached a system call. catch absorbs the error, but perror() already printed. The only fix is preventing invalid data from reaching the call.

"I'll fix it in each command separately"

If the same fix is needed in explore, search, grep, reconcile, plan documentation... the fix belongs in the shared pipeline, not in each command.

"The similarity is just shared vocabulary, not real duplication"

40-60% TF-IDF similarity between modules that aren't supposed to share concepts is a warning. The vocabulary match IS the signal.

Quick Reference: Which Command When