Cmux Testing

cmux Testing

Regression test commit policy

When adding a regression test for a bug fix, use a two-commit structure so CI proves the test catches the bug:

1. Commit 1: Add the failing test only (no fix). CI should go red. 2. Commit 2: Add the fix. CI should go green.

This makes it visible in the GitHub PR UI that the test genuinely fails without the fix.

Test quality policy

• Do not add tests that only verify source code text, method signatures, AST fragments, or grep-style patterns.
• Do not add tests that read checked-in metadata or project files such as Resources/Info.plist, project.pbxproj, .xcconfig, or source files only to assert that a key, string, plist entry, or snippet exists.
• Tests must verify observable runtime behavior through executable paths (unit/integration/e2e/CLI), not implementation shape.
• For metadata changes, prefer verifying the built app bundle or the runtime behavior that depends on that metadata, not the checked-in source file.
• If a behavior cannot be exercised end-to-end yet, add a small runtime seam or harness first, then test through that seam.
• If no meaningful behavioral or artifact-level test is practical, skip the fake regression test and state that explicitly.

Test framework

Swift Testing is the current Apple-supported primitive for tests on this codebase (shipped with Swift 6 / Xcode 16, supported on the macOS versions we target). Use it for everything that is not a UI test.

• Default to Swift Testing for all unit and integration tests. import Testing, annotate tests with @Test, group with @Suite, assert with #expect(...) and try #require(...). Do not write new tests with import XCTest unless they are UI tests.
• UI tests stay on XCTest / XCUITest. Swift Testing does not support UI testing (no XCUIApplication integration). Files under cmuxUITests/ continue to use XCTestCase + XCUIApplication. Do not migrate them and do not try to bridge Swift Testing into UI tests.
• New test targets start on Swift Testing. Every new Swift package's Tests/<Name>Tests/ directory (e.g. Packages/macOS/CmuxSettings/Tests/CmuxSettingsTests/) should ship with Swift Testing from the first commit. Xcode 16 auto-detects the framework based on the import Testing statement; no extra Package.swift configuration is required.
• Migration guide when touching an existing XCTest test. Convert in place: XCTestCase subclass becomes a @Suite struct (or final class if you need a reference type); each func testFoo() becomes @Test func foo(); XCTAssertEqual(a, b) becomes #expect(a == b); XCTAssertTrue(cond) becomes #expect(cond); XCTUnwrap(x) becomes try #require(x); XCTFail("msg") becomes Issue.record("msg"). setUp() becomes init() on the suite; tearDown() becomes deinit. Async setup is async init(). Do not bulk-rewrite untouched tests; migrate incrementally as a side effect of editing the file.
• Parameterized tests use @Test(arguments: [...]). Prefer this over duplicate test methods.
• Parallelization and shared state. Swift Testing runs tests in parallel by default, including across suites. If a suite genuinely needs ordering or guards shared mutable state, annotate it with .serialized instead of adding locks or sleeps.
• Tags with @Test(.tags(.something)) (or on a @Suite) let CI and local runs filter selectively.

Test target validation

reload.sh does not compile the test target. It builds only the cmux scheme, so a green reload.sh says nothing about whether cmuxTests/cmuxUITests still compile. A symbol that is moved or renamed can keep the cmux app building while breaking the test target (real case: a write(to:atomically:) typo and a removed TabManager.CommandResult only surfaced in the tests job). Before pushing package/refactor changes, build the cmux-unit scheme (with -derivedDataPath /tmp/cmux-<tag> and, for cmuxApp/AppDelegate churn, the GlobalISel workaround flag) or let the tests CI job gate it — never treat reload.sh alone as proof the tests build.

Detailed references

• Read references/swift-testing-migration.md when converting XCTest unit tests to Swift Testing or adding new package tests.
• Read references/regression-and-quality.md when adding a regression test, deciding whether a test is behavioral enough, or checking Xcode project test wiring.
• Read references/local-vs-ci-validation.md when choosing between reload.sh, cmux-unit, GitHub Actions, E2E/UI tests, and Python socket tests.

Local vs CI Validation

reload.sh

reload.sh builds the Debug app for a tag. It does not compile the test target.

A successful reload proves the app target built. It does not prove:

• cmuxTests compile
• cmuxUITests compile
• package test targets compile
• test-only imports still resolve

For package/refactor work, treat reload as insufficient by itself.

Unit test target

xcodebuild -scheme cmux-unit is safe because it does not launch the app. Prefer CI when practical, but use cmux-unit when package/refactor changes can break tests while the app target still builds.

Use a tagged derived data path:

xcodebuild -project cmux.xcodeproj -scheme cmux-unit -configuration Debug -destination 'platform=macOS' -derivedDataPath /tmp/cmux-<tag> build

For cmuxApp or AppDelegate churn, include the repo's known GlobalISel workaround flag if required by current project instructions.

E2E and UI tests

E2E and UI tests run via GitHub Actions or on the VM. Trigger E2E/UI through:

gh workflow run test-e2e.yml

Do not launch an untagged app locally to satisfy socket/UI tests.

Python socket tests

Python socket tests under tests_v2/ connect to a running cmux instance socket. If they must be run locally, use a tagged build socket:

CMUX_SOCKET_PATH=/tmp/cmux-debug-<tag>.sock

Never launch or target an untagged cmux DEV.app for these tests. It can conflict with the user's running debug instance.

Regression and Test Quality

Regression commit policy

When adding a regression test for a bug fix, use a two-commit structure so CI proves the test catches the bug:

1. Add the failing test only. 2. Add the fix.

This makes it visible in GitHub that the test fails without the fix and passes with it.

Behavioral tests

Tests should verify observable runtime behavior through executable paths:

• unit
• integration
• E2E
• CLI
• artifact-level behavior of a built product

Avoid tests that only verify:

• source code text
• method signatures
• AST fragments
• grep-style patterns
• checked-in plist/project/config snippets

For metadata changes, prefer testing the built app bundle or the runtime behavior that depends on the metadata. If no meaningful behavioral or artifact-level test is practical, skip the fake regression test and say so.

Test wiring

Test files in cmuxTests/ must be wired into cmux.xcodeproj/project.pbxproj.

A .swift file added to cmuxTests/ without matching project entries can be silently ignored by Xcode. Both targeted xcodebuild test -only-testing:cmuxTests/<TestClass> and bot reviews can pass with "Executed 0 tests".

The workflow-guard-tests job runs:

./scripts/lint-pbxproj-test-wiring.sh

When hand-editing wiring, use a wired sibling like TabManagerUnitTests.swift as the template.

When tests missed a bug

When the user says tests missed a bug, add or adjust behavior-level coverage around the exact repro path before claiming the fix is complete.

Do not add a broad implementation-shape test that would have passed while the user-visible bug remained.

Swift Testing Migration

Use Swift Testing for unit and integration tests. XCTest remains for UI tests.

New tests

New unit and integration tests should:

import Testing

@Suite
struct ExampleTests {
    @Test
    func computesValue() {
        #expect(1 + 1 == 2)
    }
}

Use try #require(...) when a value must be unwrapped before continuing.

XCTest conversion

When touching an existing XCTest unit test, convert in place if the edit naturally crosses that code.

Mapping:

• XCTestCase subclass -> @Suite struct or @Suite final class
• func testFoo() -> @Test func foo()
• XCTAssertEqual(a, b) -> #expect(a == b)
• XCTAssertTrue(condition) -> #expect(condition)
• XCTUnwrap(value) -> try #require(value)
• XCTFail("message") -> Issue.record("message")
• setUp() -> init()
• tearDown() -> deinit
• async setup -> async init()

Do not bulk-rewrite untouched tests just to migrate them.

Parameterized tests

Prefer:

@Test(arguments: [
    ("input-a", "output-a"),
    ("input-b", "output-b"),
])
func formats(input: String, expected: String) {
    #expect(format(input) == expected)
}

This is clearer than duplicating test methods with copy/paste assertions.

Parallel execution

Swift Testing runs tests in parallel by default, including across suites. If a suite genuinely needs ordering or guards shared mutable state, use .serialized:

@Suite(.serialized)
struct FileBackedTests { ... }

Prefer isolated temp directories and injected dependencies over serialization when practical.

UI tests

Files under cmuxUITests/ stay on XCTest/XCUITest. Swift Testing does not support XCUIApplication UI testing.