
Implement Conformance Testing Script
- 40 installs
- 54 repo stars
- Updated July 22, 2026
- codeplain-ai/plain-forge
Generate a Bash or PowerShell conformance-test runner script for a new language in a ***plain project, following the reference pattern.
About
Generates a language-agnostic conformance-test runner script (Bash or PowerShell) for a ***plain build folder. A developer uses it to add a conformance-test runner for a new language to a ***plain project.
- Generates a Bash or PowerShell conformance-test runner per language
- Install-inline or activate-only variant depending on prepare-environment
Implement Conformance Testing Script by the numbers
- 40 all-time installs (skills.sh)
- Ranked #1,285 of 2,153 Testing & QA skills by installs in the Skillselion catalog
- Data as of Aug 1, 2026 (Skillselion catalog sync)
npx skills add https://github.com/codeplain-ai/plain-forge --skill implement-conformance-testing-scriptAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 40 |
|---|---|
| repo stars | ★ 54 |
| Last updated | July 22, 2026 |
| Repository | codeplain-ai/plain-forge ↗ |
What it does
Generate a Bash or PowerShell conformance-test runner script for a new language in a ***plain project, following the reference pattern.
Files
Implement Conformance Testing Script
This skill produces a single executable script that runs the conformance tests for a generated build folder, following a consistent, language-agnostic pattern.
The reference implementations are:
- assets/run_conformance_tests_java.sh — Java, install-inline variant.
- assets/run_conformance_tests_python.sh — Python, install-inline variant.
- assets/run_conformance_tests_<lang>.ps1 — Windows PowerShell equivalents.
Read both before writing anything — every script you produce must be a faithful translation of the same pattern into the target language's tooling and the user's shell environment.
How conformance scripts differ from unit-test scripts
A conformance script is structurally very close to a unit-test script (see the sibling skill `implement-unit-testing-script`) but with two important differences:
1. Two positional arguments instead of one. A conformance script takes both the build folder (source under test) and a separate conformance tests folder (the tests to execute against that build). 2. Tests are loaded from outside the working folder. The build is staged into .tmp/<lang>_<arg> and the script cds into it, but the test command is pointed at the original $current_dir/<conformance_tests_folder>. Tests are never copied into the staging area.
Everything else — toolchain check, build staging, dependency isolation, exit codes — is the same.
Variant decision: install-inline vs. activate-only
Before writing anything, decide which variant to emit. Both variants share toolchain check, arg validation, cwd capture, test execution, and exit-code handling — they differ only in the middle (steps 4–7 of the pattern below).
| Look for an existing | Emit |
|---|---|
prepare_environment_<lang>.sh / .ps1 in the project's test_scripts/ folder (or wherever config.yaml's prepare-environment-script: key points) | Activate-only variant. Verifies the prepared env, activates it, and runs tests. Does not stage the build or install deps — prepare already did. |
| Nothing — no prepare script | Install-inline variant. Stages the build, installs deps, and runs tests in one shot. |
Why this split exists
The conformance runner is invoked once per functional spec by the renderer. Each functional spec in a module has its own conformance_tests/<module>/<spec>/ folder, and after the renderer finishes generating code for a new spec, it runs the conformance tests of every previous spec in the same module to detect regressions. For a module with N functional specs, this script is called on the order of N times per render — not once per render.
That per-spec invocation pattern is what makes the install step expensive. A naive runner that does pip install / npm ci / mvn install -DskipTests / cargo build on every invocation pays the install cost N times per render. For anything beyond a toy project, that cost dominates wall-clock time.
The two variants are a direct response to this:
- Install-inline is correct only when N is small (a few specs) or dependencies are cheap. It is self-contained: stage, install, run, repeat from scratch every invocation.
- Activate-only is the production answer. `prepare_environment_<lang>` runs once per render and pays the install cost a single time, populating
.tmp/<lang>_<arg>/with the warmed environment. Each of the N conformance invocations then just attaches to that working folder and runs the tests — no install, no compile, just activate-and-go.
Why picking the right variant matters: if you emit the install-inline variant alongside an existing prepare script, prepare's work is wiped (by the script's rm -rf .tmp/<lang>_$1) or duplicated (by re-running install) on every run — defeating prepare's whole purpose. Conversely, emitting activate-only without a prepare script means the "verify prepared environment" check fails on every run because nothing has populated the working folder. See Anti-Patterns.
Pick the Shell First
Before writing anything, decide which shell flavor the script must target — it depends on the user's environment, not on the language:
- Bash (`.sh`) — macOS, Linux, WSL, CI runners on Linux. Default unless the user is on native Windows.
- PowerShell (`.ps1`) — native Windows / PowerShell-only environments.
If you can't tell from the project (no obvious OS hints, no existing scripts), ask the user.
The same pattern applies to both. Only the syntax changes.
The Pattern
Steps 1–3 and step 8 are identical in both variants. Steps 4–7 differ — pick the subsection below that matches the variant you decided on.
Common steps (both variants)
1. Toolchain check. Verify that the required language runtime / build tool (and the required version, if any) is installed. If not, print an error and exit with code 69. 2. Argument validation. Require two positional arguments: <build_folder> and <conformance_tests_folder>. If either is missing, print usage and exit with code 69. 3. Capture original cwd. Store pwd in a variable (current_dir / $PWD) before changing directories — the test command in step 8 needs it to resolve the conformance tests folder.
Steps 4–7 — install-inline variant (no prepare script)
4. Working directory setup. Define a working folder at .tmp/<lang>_<arg1>. Wipe it (rm -rf / Remove-Item -Recurse -Force) and recreate it. This folder — and only this folder — is where every subsequent write must land. 5. Copy the build. Recursively copy everything from <build_folder> ($1) into the working folder. Do not copy the conformance tests — they stay where they are. After this step both $1 (build folder) and $2 (conformance tests folder) are treated as read-only for the rest of the script. 6. Enter the working directory. cd / Set-Location into .tmp/<lang>_<arg1>. If that fails, exit with code 69. All remaining steps run from inside the working folder; they must never write back to $1 or $2. 7. Install dependencies into an isolated environment inside `.tmp/<lang>_<arg1>`. Set up a per-working-folder dependency location (a Python venv at ./.venv, a local ./node_modules, a project-scoped Maven repo at ./.m2, etc.) and install/resolve all dependencies into it. Never install into the source build folder ($1), the conformance tests folder ($2), the user's global cache (~/.m2, system-wide pip, ~/.cargo, ~/.npm, ...), or anywhere outside .tmp/<lang>_<arg1>. If the install command fails, propagate its exit code immediately and do not proceed to step 8. See Dependency isolation (install-inline).
Steps 4–7 — activate-only variant (prepare script exists)
4. Verify the prepared environment. Both:
- Check that the working folder
.tmp/<lang>_<arg1>exists. - Check that the language's isolation location inside it exists (e.g.
.venv/bin/activatefor Python,.m2/for Java,node_modules/for Node,.gocache/for Go,.cargo/for Rust).
If either check fails, print a helpful error ("Error: prepared environment missing — did you run prepare_environment_<lang>.<sh|ps1> first?") and exit 69. Do not silently fall back to creating it inline — that would mask a real misconfiguration and turn this script into the install-inline variant in disguise. After this step both $1 and $2 are treated as read-only for the rest of the script. 5. Enter the working directory. cd / Set-Location into .tmp/<lang>_<arg1>. If that fails, exit 69. All remaining steps run from inside the working folder; they must never write back to $1 or $2. 6. Activate the prepared dependency environment. Per-language:
- Python:
source .venv/bin/activate(must succeed; exit69on failure). - Java: set
MAVEN_LOCAL_REPO="$(pwd)/.m2"so it can be passed as-Dmaven.repo.local="$MAVEN_LOCAL_REPO"tomvnin step 8. - Node.js / Go / Rust: nothing to activate explicitly — the test command in step 8 just needs to receive the same isolation flag/env var that prepare used (
./node_modulesis found by default; passGOMODCACHE/CARGO_HOME).
Activation is always relative to the working folder, never to $1 or $2 — prepare populated .tmp/<lang>_<arg1>/..., and that is the only place to attach to. 7. (There is no step 7 in this variant — install was prepare's job. Skip straight to step 8.)
Common step 8 (both variants)
8. Run the conformance tests. Invoke the language's standard test command, pointed at `$current_dir/<conformance_tests_folder>` (the original cwd from step 3 + the second arg). The script's final exit code is whatever the test command returns — except for the "no tests discovered" case below.
The test command is read-only with respect to $current_dir/$2. It loads test files from there, but any artifacts the runner produces (caches, JUnit XML, coverage reports, compiled test classes, etc.) must land inside .tmp/<lang>_<arg1>, not next to the test files. If your chosen runner defaults to writing output beside the tests, pass an explicit output-directory flag pointing inside the working folder (e.g. pytest --basetemp=./.pytest_tmp, jest --cacheDirectory=./.jest_cache, Maven target/ under .tmp via mvn -f "$current_dir/$2/pom.xml" -Dproject.build.directory="$(pwd)/target" test).
Read-only inputs — hard rule
A conformance script has two read-only inputs: the source build folder ($1) and the conformance tests folder ($2). Neither one may be written to under any circumstances. The script must never:
- install dependencies into
$1or$2(nopip installinside$1/$2, nonpm installinside them, nomvn installwriting into them, no Cargo build artifacts ending up under them), - write a virtualenv /
node_modules/.m2/.gocache/.cargodirectory inside$1or$2, - run the test command with its
cwdset to$1or$2(every test command runs from inside.tmp/<lang>_<arg1>after thecdin step 6 / activate-only step 5), - create logs, caches, build outputs, JUnit XML, coverage reports, compiled test classes, or temp files inside
$1or$2.
Why each input is read-only:
- `$1` (build folder) is shared with the renderer (
plain_modules/...by default) and downstream tooling. Writing into it corrupts the renderer's view of "what was generated" and breaks subsequent renders. The whole point of staging into.tmp/is so the source folder stays a clean, reproducible artifact of the render. - `$2` (conformance tests folder) is the user's authored test source — typically checked into version control. Writing into it pollutes the working tree, churns git status, and (with frameworks that auto-discover) can make subsequent runs pick up generated files as if they were tests.
If you find yourself about to issue any command whose cwd is $1 or $2, or whose target path starts with $1/ or $2/, stop. Either move the operation into .tmp/<lang>_<arg1>, or you're doing something the script must not do.
"No tests discovered" detection
The Python reference script grep's the test runner output for "Ran 0 tests in" and exits 1 if no tests ran. Replicate the equivalent check for the target language wherever that language's test runner silently passes when given an empty test set:
- Python
unittest:"Ran 0 tests in" - Node.js
jest:"No tests found" - Go
go test:"no test files"/"no tests to run" - Rust
cargo test:"running 0 tests" - Java
mvn test: usually fails loudly already; no extra check needed.
A silently-passing zero-test run is the most dangerous failure mode of a conformance runner — always guard against it. This applies to both variants.
Conventions
Shared across both shell flavors and both variants:
- Exit codes:
69— unrecoverable invocation error: missing argument, missing toolchain, can't enter working folder, can't create venv (install-inline), or prepared environment missing/broken (activate-only). Matches the reference scripts'UNRECOVERABLE_ERROR_EXIT_CODE.1— "no tests discovered" guard tripped (see above).- Any other non-zero code — propagated from the underlying test command.
- Working folder naming:
.tmp/<lang>_<arg1>where<lang>is a short identifier for the language (java,python,node,go,rust, ...). Use the first argument (the build folder) in the path, never the conformance tests folder. All dependency installs, build outputs, caches, test runner artifacts, and the test invocation itself live inside this folder. Nothing the script does should touch$1after step 5 (install-inline) / step 4 (activate-only), or$2at any point. - Logging: print short progress lines (
"Preparing <lang> build subfolder: ...","Activating prepared virtual environment...","Running <lang> conformance tests...") so failures are easy to triage. Wrap noisy "preparing" lines in aVERBOSEcheck if matching the Python reference. - Capture `current_dir` before `cd`. This is the single most common bug in hand-written conformance scripts: forgetting that the conformance tests folder argument is relative to the invocation directory, not the working folder.
Dependency isolation (install-inline)
This section applies to install-inline scripts only. For activate-only scripts, the isolation location is set up by prepare; you just need to point the test command at it — see Activating a prepared environment.
The dependency environment must live inside $WORKING_FOLDER so the test run can't be polluted by — or pollute — the user's global caches. Pick the most idiomatic isolation mechanism for the language:
| Language | Isolation mechanism | Install command (run inside $WORKING_FOLDER) | Test command (point at $current_dir/$2) |
|---|---|---|---|
| Python | venv at ./.venv | python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt | python -m unittest discover -b -s "$current_dir/$2" (or pytest "$current_dir/$2") |
| Node.js | local ./node_modules (default) | npm ci (preferred) or npm install | npx jest --rootDir "$current_dir/$2" |
| Java | project-scoped Maven repo at ./.m2 | mvn -Dmaven.repo.local=./.m2 install -DskipTests (build + install artifact so the test pom can resolve it) | mvn -f "$current_dir/$2/pom.xml" -Dmaven.repo.local="$(pwd)/.m2" test |
| Go | module cache at ./.gocache | GOMODCACHE="$PWD/.gocache" go mod download (optional pre-warm) | GOMODCACHE="$PWD/.gocache" go test "$current_dir/$2/..." |
| Rust | cargo home at ./.cargo | CARGO_HOME="$PWD/.cargo" cargo fetch (optional pre-warm) | CARGO_HOME="$PWD/.cargo" cargo test --manifest-path "$current_dir/$2/Cargo.toml" |
Notes:
- Every path in the install command and test command is relative to `.tmp/<lang>_<arg1>`. That's why the script
cds into the working folder in step 6 — from that point on,./.venv,./node_modules,./.m2, etc. all resolve under.tmp/<lang>_<arg1>, never under$1or$2. - Always pass the isolation flag/env var to both the install command and the test command. They must agree on where deps live, otherwise the test command will silently fall back to the global cache or (worse) write into
$1/$2. - Python is the only ecosystem where the venv is mandatory to satisfy "into a virtual environment" literally. The others use language-native equivalents that achieve the same isolation.
- Propagate the install exit code immediately. In Bash:
<install cmd> || exit $?. In PowerShell: check$LASTEXITCODEandexit $LASTEXITCODEif non-zero. - Time the dependency setup with
date +%s.%N(Bash) /Get-Date(PowerShell) and print"Requirements setup completed in X.XX seconds". If this number is large, that's the signal to add aprepare_environment_<lang>script (and switch this script to the activate-only variant).
Activating a prepared environment (activate-only)
This section applies to activate-only scripts only. The isolation location was created by prepare; conformance just needs to attach to it and pass the right flags to the test command.
| Language | Verify exists in step 4 | Activate in step 6 | Test command in step 8 (point at $current_dir/$2) |
|---|---|---|---|
| Python | .tmp/<lang>_$1/.venv/bin/activate | source .venv/bin/activate (after cd-ing into the working folder) | python -m unittest discover -b -s "$current_dir/$2" |
| Node.js | .tmp/<lang>_$1/node_modules/ | (nothing) | npx jest --rootDir "$current_dir/$2" |
| Java | .tmp/<lang>_$1/.m2/ | MAVEN_LOCAL_REPO="$(pwd)/.m2" | mvn -f "$current_dir/$2/pom.xml" -Dmaven.repo.local="$MAVEN_LOCAL_REPO" test |
| Go | .tmp/<lang>_$1/.gocache/ | export GOMODCACHE="$(pwd)/.gocache" | go test "$current_dir/$2/..." |
| Rust | .tmp/<lang>_$1/.cargo/ | export CARGO_HOME="$(pwd)/.cargo" | cargo test --manifest-path "$current_dir/$2/Cargo.toml" |
Notes:
- Verify, don't recreate. If
.venvis missing, exit69with a clear "did you run prepare_environment first?" message — do not silently fall back to creating it inline. That would silently degrade a misconfigured project into the install-inline path and mask the real problem. - Match prepare's isolation paths exactly. If prepare puts the venv at
.venvand you look for it atvenv, the verify step will always fail. Read `implement-prepare-environment-script` for the canonical paths. - Don't time anything in this variant. The slow phase is prepare; conformance just runs the tests. Adding a duration log here is misleading — it makes the script look like it's doing the install when it isn't.
Bash specifics
- Shebang:
#!/bin/bash. - File naming:
run_conformance_tests_<lang>.sh, placed inassets/(skill reference) ortest_scripts/(target project). - Arguments:
$1= build folder,$2= conformance tests folder. - Make it executable:
chmod +xthe produced script. - `cd` failure check: the reference scripts use the
cd ... 2>/dev/null+[ $? -ne 0 ]pattern. Keep it.
PowerShell specifics
- No shebang. Use a
param([Parameter(Mandatory=$true)][string]$BuildFolder, [Parameter(Mandatory=$true)][string]$ConformanceTestsFolder)block at the top instead. - File naming:
run_conformance_tests_<lang>.ps1. - Exit codes: use
exit 69etc. (PowerShell honors them just like Bash). - Toolchain check: prefer
Get-Command <tool> -ErrorAction SilentlyContinueand, where a specific version is needed, parse the tool's--versionoutput. - Filesystem: use
Test-Path,Remove-Item -Recurse -Force,New-Item -ItemType Directory,Copy-Item -Recurse,Set-Location. Quote paths to handle spaces. - Capture original cwd:
$currentDir = (Get-Location).Pathbefore anySet-Locationcall. - No `chmod` step needed. If execution policy is likely to block the script, mention
Set-ExecutionPolicy -Scope CurrentUser RemoteSignedto the user — don't bake it into the script.
Workflow
1. Decide the variant. Look in the project for prepare_environment_<lang>.sh / .ps1 (check test_scripts/, then any prepare-environment-script: key in config.yaml). If present → emit activate-only. If absent → emit install-inline. See Variant decision. 2. Confirm the target language, shell flavor (Bash or PowerShell), and dependency manifest (pom.xml, requirements.txt / pyproject.toml, package.json, go.mod, Cargo.toml, ...). Ask if any is unclear. 3. Read assets/run_conformance_tests_java.sh and assets/run_conformance_tests_python.sh to refresh the exact structure. Both are install-inline references — for activate-only, follow steps 4–7 of the activate-only variant and the Activating a prepared environment table. 4. Translate each step into the equivalent commands for the target language and shell. The toolchain check, dependency install/activate, and test invocation are the language-specific parts; the rest is mechanical translation between Bash and PowerShell syntax. 5. Pick the right per-language row:
- Install-inline: Dependency isolation (install-inline) table — use the same flag/env var in steps 7 and 8.
- Activate-only: Activating a prepared environment table — use the matching verify, activate, and test-command columns in steps 4, 6, and 8.
6. Add the language-appropriate "no tests discovered" guard from No tests discovered detection. 7. Save the new script. For Bash, chmod +x it. 8. For activate-only scripts only: smoke-test by running prepare_environment_<lang>.<sh|ps1> <build> && run_conformance_tests_<lang>.<sh|ps1> <build> <tests>. If the conformance script errors with "prepared environment missing" right after a successful prepare, the two scripts disagree on either the working-folder path or the isolation location — fix that before declaring done.
Anti-Patterns
- (Hard mistake) Don't install into, build into, or otherwise write to the source build folder (`$1`) or the conformance tests folder (`$2`). Both arguments are read-only input. Every install, cache, build artifact, log, JUnit XML, coverage report, compiled test class, and temp file must land in
.tmp/<lang>_<arg1>. This includes never runningpip install,npm install,mvn install, orcargo buildwith$1or$2as theircwdor target; never letting a venv /node_modules/.m2/.gocache/.cargodirectory appear inside$1or$2; and never running the test command from inside either folder. The whole point of staging into.tmp/is so the build folder remains a clean artifact of the render and the conformance tests folder remains a clean tree under the user's version control — writing to either one corrupts those guarantees. - Don't emit the install-inline variant when a `prepare_environment_<lang>` script already exists. The conformance script's
rm -rf .tmp/<lang>_$1will wipe everything prepare did, and the inline install will redo it from scratch on every run. Always run the Variant decision check first. - Don't emit the activate-only variant when no prepare script exists. The "verify prepared environment" check will fail on every run because nothing has populated the working folder.
- Don't silently fall back from activate-only to install-inline when the prepared environment is missing. Exit
69with a clear error so the misconfiguration is visible. Silent fallback hides the real bug and produces inconsistent behavior between runs. - Don't copy the conformance tests folder into `.tmp/`. Only the build folder is staged (and only in install-inline). The test folder is read in place from
$current_dir/$2. - Don't compute the test path after `cd`. Capture
current_dirfirst; otherwise$2will be resolved relative to the working folder and silently miss the tests. - Don't skip the "no tests discovered" check. A conformance suite that finds zero tests and exits
0is the worst possible failure mode — it looks like success in CI. - Don't skip the toolchain check, even when "everyone has it installed" — exit code
69is what the calling system relies on to detect a missing runtime. - Don't reuse the source folder in place (install-inline). Always copy into
.tmp/<lang>_<arg1>first; the renderer relies on this isolation. - Don't change the exit-code contract. Other parts of the system branch on
69and1specifically — and these codes must be identical between the Bash and PowerShell variants. - Don't write a cross-shell hybrid (e.g. a
.shthat detects PowerShell, or vice versa). Ship one script per shell, named with the appropriate extension. - Don't install dependencies into the user's global location (
~/.m2, system-widepip,~/.cargo, etc.) in the install-inline variant. Always isolate inside$WORKING_FOLDERso concurrent runs and other projects can't interfere. - Don't run the test command without first verifying the install / activation succeeded. A failed install (or missing prepared env) followed by a "test" run produces misleading errors that look like test failures.
#!/usr/bin/env pwsh
$ErrorActionPreference = 'Stop'
$UNRECOVERABLE_ERROR_EXIT_CODE = 69
$NPM_INSTALL_OUTPUT_FILTER = "up to date in|added [0-9]* packages, removed [0-9]* packages, and changed [0-9]* packages in|removed [0-9]* packages, and changed [0-9]* packages in|added [0-9]* packages in|removed [0-9]* packages in"
# Function to check and kill any Node process running on port 3000 (React development server)
function Check-AndKillNodeServer {
try {
$connections = Get-NetTCPConnection -LocalPort 3000 -ErrorAction SilentlyContinue
if ($connections) {
foreach ($conn in $connections) {
$proc = Get-Process -Id $conn.OwningProcess -ErrorAction SilentlyContinue
if ($proc -and $proc.ProcessName -eq "node") {
Write-Host "Found Node server running on port 3000. Killing it..."
Stop-Process -Id $proc.Id -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
if ($env:VERBOSE -eq "1") {
Write-Host "Node server terminated."
}
}
}
}
} catch {
# Silently ignore errors (port may not be in use)
}
}
# Function to get all child processes of a given PID recursively
function Get-ChildProcesses {
param([int]$ParentPid)
$children = Get-CimInstance Win32_Process -Filter "ParentProcessId = $ParentPid" -ErrorAction SilentlyContinue
$result = @()
foreach ($child in $children) {
$result += $child.ProcessId
$result += Get-ChildProcesses -ParentPid $child.ProcessId
}
return $result
}
# Cleanup function to ensure all processes are terminated
function Cleanup {
# Kill any running npm processes started by this script
if ($script:NPM_PID) {
Stop-Process -Id $script:NPM_PID -Force -ErrorAction SilentlyContinue
}
# Kill React app and its children if they exist
if ($script:REACT_APP_PID) {
$processesToKill = Get-ChildProcesses -ParentPid $script:REACT_APP_PID
# Kill the main process
Stop-Process -Id $script:REACT_APP_PID -Force -ErrorAction SilentlyContinue
# Kill all the subprocesses
foreach ($pid in $processesToKill) {
Stop-Process -Id $pid -Force -ErrorAction SilentlyContinue
}
if ($env:VERBOSE -eq "1") {
Write-Host "React app is terminated!"
}
}
# Remove temporary files if they exist
if ($script:build_output -and (Test-Path $script:build_output)) {
Remove-Item $script:build_output -Force -ErrorAction SilentlyContinue
}
}
# Check for and kill any existing Node server from previous runs
Check-AndKillNodeServer
# Check if build folder name is provided
if (-not $args[0]) {
Write-Host "Error: No build folder name provided."
Write-Host "Usage: $($MyInvocation.MyCommand.Name) <build_folder_name> <conformance_tests_folder>"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
# Check if conformance tests folder name is provided
if (-not $args[1]) {
Write-Host "Error: No conformance tests folder name provided."
Write-Host "Usage: $($MyInvocation.MyCommand.Name) <build_folder_name> <conformance_tests_folder>"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
$BuildFolder = $args[0]
$ConformanceTestsFolder = $args[1]
if ($args[2] -eq "-v" -or $args[2] -eq "--verbose") {
$env:VERBOSE = "1"
}
$current_dir = Get-Location
try {
# Define the path to the subfolder
$NODE_SUBFOLDER = "node_$BuildFolder"
# Running React application
Write-Host "### Step 1: Starting the React application in folder $NODE_SUBFOLDER..."
if ($env:VERBOSE -eq "1") {
Write-Host "Preparing Node subfolder: $NODE_SUBFOLDER"
}
# Check if the node subfolder exists
if (Test-Path $NODE_SUBFOLDER) {
# Delete all files and folders except "node_modules", "plain_modules", and "package-lock.json"
Get-ChildItem -Path $NODE_SUBFOLDER -Force |
Where-Object {
$_.Name -ne "node_modules" -and
$_.Name -ne "plain_modules" -and
$_.Name -ne "package-lock.json"
} | Remove-Item -Recurse -Force
if ($env:VERBOSE -eq "1") {
Write-Host "Cleanup completed, keeping 'node_modules' and 'package-lock.json'."
}
} else {
if ($env:VERBOSE -eq "1") {
Write-Host "Subfolder does not exist. Creating it..."
}
New-Item -ItemType Directory -Path $NODE_SUBFOLDER -Force | Out-Null
}
Copy-Item -Path "$BuildFolder/*" -Destination $NODE_SUBFOLDER -Recurse -Force
# Move to the subfolder
if (-not (Test-Path $NODE_SUBFOLDER)) {
Write-Host "Error: Node build folder '$NODE_SUBFOLDER' does not exist."
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
Push-Location $NODE_SUBFOLDER
# Temporarily allow stderr output without throwing (npm may write warnings to stderr)
# ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting
$ErrorActionPreference = 'Continue'
$npmInstallOutput = npm install --prefer-offline --no-audit --no-fund --loglevel error 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String
$ErrorActionPreference = 'Stop'
# Filter out noisy npm install lines
$npmInstallOutput -split "`n" | Where-Object { $_ -notmatch $NPM_INSTALL_OUTPUT_FILTER } | ForEach-Object {
if ($_.Trim()) { Write-Host $_ }
}
if ($LASTEXITCODE -ne 0) {
Write-Host "Error: Installing Node modules."
exit 2
}
if ($env:VERBOSE -eq "1") {
Write-Host "Building the application..."
}
$script:build_output = [System.IO.Path]::GetTempFileName()
# Temporarily allow stderr output without throwing (build tools may write to stderr)
$ErrorActionPreference = 'Continue'
npm run build > $script:build_output 2>&1
$ErrorActionPreference = 'Stop'
if ($LASTEXITCODE -ne 0) {
Write-Host "Error: Building application."
Get-Content $script:build_output
Remove-Item $script:build_output -Force -ErrorAction SilentlyContinue
exit 2
}
Remove-Item $script:build_output -Force -ErrorAction SilentlyContinue
if ($env:VERBOSE -eq "1") {
Write-Host "Starting the application..."
}
# Start the React app in the background and redirect output to a log file
$env:BROWSER = "none"
$reactProcess = Start-Process -FilePath "npm" -ArgumentList "start", "--", "--no-open" `
-NoNewWindow -PassThru -RedirectStandardOutput "app.log" -RedirectStandardError "app_err.log"
if ($env:VERBOSE -eq "1") {
Write-Host "Application is starting..."
}
# Capture the process ID
$script:REACT_APP_PID = $reactProcess.Id
# Try to find the child npm process
Start-Sleep -Milliseconds 500
$npmChild = Get-CimInstance Win32_Process -Filter "ParentProcessId = $($script:REACT_APP_PID)" -ErrorAction SilentlyContinue |
Where-Object { $_.Name -match "npm" } | Select-Object -First 1
if ($npmChild) {
$script:NPM_PID = $npmChild.ProcessId
}
# Wait for the "compiled successfully!" or Vite ready message in the log file
while ($true) {
if (Test-Path "app.log") {
$logContent = Get-Content "app.log" -Raw -ErrorAction SilentlyContinue
if ($logContent) {
if ($logContent -match "(?i)compiled successfully|compiled with warnings") {
break
}
if ($logContent -match "(?i)VITE v[0-9]+\.[0-9]+\.[0-9]+\s+ready in") {
break
}
}
}
# Also check if localhost:3000 responds
try {
$response = Invoke-WebRequest -Uri "http://localhost:3000" -UseBasicParsing -TimeoutSec 1 -ErrorAction SilentlyContinue
if ($response) { break }
} catch {
# Not ready yet
}
# Check if the React app process is still running
$proc = Get-Process -Id $script:REACT_APP_PID -ErrorAction SilentlyContinue
if (-not $proc -or $proc.HasExited) {
Write-Host "Error in :ImplementationCode: (React app process (PID: $($script:REACT_APP_PID)) has terminated unexpectedly)."
if (Test-Path "app.log") { Get-Content "app.log" }
if (Test-Path "app_err.log") { Get-Content "app_err.log" }
exit 2
}
Start-Sleep -Milliseconds 100
}
# At this point, the React app is up and running in the background
if ($env:VERBOSE -eq "1") {
Write-Host "React app is up and running!"
}
Pop-Location
# Execute all Cypress conformance tests in the build folder
Write-Host "### Step 2: Running Cypress conformance tests $ConformanceTestsFolder..."
# Move back to the original directory
Set-Location $current_dir
# Define the path to the conformance tests subfolder
$NODE_CONFORMANCE_TESTS_SUBFOLDER = "node_$ConformanceTestsFolder"
if ($env:VERBOSE -eq "1") {
Write-Host "Preparing conformance tests Node subfolder: $NODE_CONFORMANCE_TESTS_SUBFOLDER"
}
# Check if the conformance tests node subfolder exists
if (Test-Path $NODE_CONFORMANCE_TESTS_SUBFOLDER) {
# Delete all files and folders except "node_modules", "plain_modules", and "package-lock.json"
Get-ChildItem -Path $NODE_CONFORMANCE_TESTS_SUBFOLDER -Force |
Where-Object {
$_.Name -ne "node_modules" -and
$_.Name -ne "plain_modules" -and
$_.Name -ne "package-lock.json"
} | Remove-Item -Recurse -Force
if ($env:VERBOSE -eq "1") {
Write-Host "Cleanup completed, keeping 'node_modules' and 'package-lock.json'."
}
} else {
if ($env:VERBOSE -eq "1") {
Write-Host "Subfolder does not exist. Creating it..."
}
New-Item -ItemType Directory -Path $NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | Out-Null
}
Copy-Item -Path "$ConformanceTestsFolder/*" -Destination $NODE_CONFORMANCE_TESTS_SUBFOLDER -Recurse -Force
# Move to the subfolder with Cypress tests
if (-not (Test-Path $NODE_CONFORMANCE_TESTS_SUBFOLDER)) {
Write-Host "Error: conformance tests Node folder '$NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist."
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
Push-Location $NODE_CONFORMANCE_TESTS_SUBFOLDER
# Temporarily allow stderr output without throwing (npm may write warnings to stderr)
# ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting
$ErrorActionPreference = 'Continue'
$npmInstallOutput = npm install cypress --save-dev --prefer-offline --no-audit --no-fund --loglevel error 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String
$ErrorActionPreference = 'Stop'
$npmInstallOutput -split "`n" | Where-Object { $_ -notmatch $NPM_INSTALL_OUTPUT_FILTER } | ForEach-Object {
if ($_.Trim()) { Write-Host $_ }
}
if ($env:VERBOSE -eq "1") {
Write-Host "Running Cypress conformance tests..."
}
$ErrorActionPreference = 'Continue'
$cypress_info_output = npx cypress info 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String
$ErrorActionPreference = 'Stop'
$CYPRESS_BROWSER_FLAG = ""
if ($cypress_info_output -match "(?i)chrome") {
$CYPRESS_BROWSER_FLAG = "--browser=chrome"
}
Write-Host "CYPRESS_BROWSER_FLAG: $(if ($CYPRESS_BROWSER_FLAG) { $CYPRESS_BROWSER_FLAG } else { 'none' })"
$env:BROWSERSLIST_IGNORE_OLD_DATA = "1"
if ($CYPRESS_BROWSER_FLAG) {
npx cypress run $CYPRESS_BROWSER_FLAG --config video=false 2>$null
} else {
npx cypress run --config video=false 2>$null
}
$cypress_run_result = $LASTEXITCODE
if ($cypress_run_result -ne 0) {
if ($env:VERBOSE -eq "1") {
Write-Host "Error: Cypress conformance tests have failed."
}
exit 1
}
} finally {
Cleanup
}
#!/usr/bin/env pwsh
$ErrorActionPreference = 'Stop'
$UNRECOVERABLE_ERROR_EXIT_CODE = 69
# Check if build folder name is provided
if (-not $args[0]) {
Write-Host "Error: No build folder name provided."
Write-Host "Usage: $($MyInvocation.MyCommand.Name) <build_folder_name> <conformance_tests_folder>"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
# Check if conformance tests folder name is provided
if (-not $args[1]) {
Write-Host "Error: No conformance tests folder name provided."
Write-Host "Usage: $($MyInvocation.MyCommand.Name) <build_folder_name> <conformance_tests_folder>"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
$BuildFolder = $args[0]
$ConformanceTestsFolder = $args[1]
$current_dir = Get-Location
$GO_BUILD_SUBFOLDER = "go_$BuildFolder"
if ($env:VERBOSE -eq "1") {
Write-Host "Preparing Go build subfolder: $GO_BUILD_SUBFOLDER"
}
# Check if the go build subfolder exists
if (Test-Path $GO_BUILD_SUBFOLDER) {
# Delete all files and folders inside
Get-ChildItem -Path $GO_BUILD_SUBFOLDER -Force | Remove-Item -Recurse -Force
if ($env:VERBOSE -eq "1") {
Write-Host "Cleanup completed."
}
} else {
if ($env:VERBOSE -eq "1") {
Write-Host "Subfolder does not exist. Creating it..."
}
New-Item -ItemType Directory -Path $GO_BUILD_SUBFOLDER -Force | Out-Null
}
Copy-Item -Path "$BuildFolder/*" -Destination $GO_BUILD_SUBFOLDER -Recurse -Force
# Move to the subfolder
if (-not (Test-Path $GO_BUILD_SUBFOLDER)) {
Write-Host "Error: Go build folder '$GO_BUILD_SUBFOLDER' does not exist."
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
Push-Location $GO_BUILD_SUBFOLDER
try {
Write-Host "Runinng go get in the build folder..."
go get
# Move to conformance tests folder
Set-Location "$current_dir/$ConformanceTestsFolder"
if ($LASTEXITCODE -and $LASTEXITCODE -ne 0) {
Write-Host "Error: Conformance tests folder '$current_dir/$ConformanceTestsFolder' does not exist."
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
Write-Host "Checking for go.mod in conformance test directory..."
if (Test-Path "go.mod") {
Write-Host "Running go get in conformance test directory..."
go get
} else {
Write-Host "No go.mod found in conformance test directory, skipping go get"
}
# Move back to build directory
Set-Location "$current_dir/$GO_BUILD_SUBFOLDER"
# Execute Go lang conformance tests
Write-Host "Running Golang conformance tests...`n"
# Temporarily allow stderr output without throwing (Go may write to stderr)
# ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting
$ErrorActionPreference = 'Continue'
$output = go run "$current_dir/$ConformanceTestsFolder/conformance_tests.go" 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String
$exit_code = $LASTEXITCODE
$ErrorActionPreference = 'Stop'
# If there was an error, print the output and exit with the error code
if ($exit_code -ne 0) {
Write-Host $output
exit $exit_code
}
# Exit with the exit code of the test command
exit $exit_code
} finally {
Pop-Location
}
#!/bin/bash
export JAVA_HOME=$(/usr/libexec/java_home -v 21)
java --version
# Check if build folder name is provided
if [ -z "$1" ]; then
printf "Error: No build folder name provided.\n"
printf "Usage: $0 <build_folder_name> <conformance_tests_folder>\n"
exit 1
fi
# Check if conformance tests folder name is provided
if [ -z "$2" ]; then
printf "Error: No conformance tests folder name provided.\n"
printf "Usage: $0 <build_folder_name> <conformance_tests_folder>\n"
exit 1
fi
current_dir=$(pwd)
printf "Current directory: $current_dir\n"
tree $2
JAVA_BUILD_SUBFOLDER=,tmp/$1
if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then
printf "Preparing Java build subfolder: $JAVA_BUILD_SUBFOLDER\n"
fi
rm -rf $JAVA_BUILD_SUBFOLDER
mkdir -p $JAVA_BUILD_SUBFOLDER
cp -R $1/* $JAVA_BUILD_SUBFOLDER
printf "Copied from $1 to $JAVA_BUILD_SUBFOLDER...\n"
# Move to the subfolder
cd "$JAVA_BUILD_SUBFOLDER" 2>/dev/null
printf "Moved to $JAVA_BUILD_SUBFOLDER...\n"
if [ $? -ne 0 ]; then
printf "Error: Java build folder '$JAVA_BUILD_SUBFOLDER' does not exist.\n"
exit 2
fi
echo "Runinng maven install in $(pwd)..."
output=$(mvn clean install -DskipTests 2>&1)
exit_code=$?
# If there was an error, print the output and exit with the error code
if [ $exit_code -ne 0 ]; then
echo "Command failed: mvn clean install -DskipTests"
echo "Error: Running maven build failed with exit code $exit_code"
echo "Output: $output"
exit $exit_code
fi
CONFORMANCE_TESTS_FOLDER=.tmp/java_conformance
cd "$current_dir" 2>/dev/null
printf "Moved to $current_dir...\n"
printf "Preparing Java conformance tests subfolder: $CONFORMANCE_TESTS_FOLDER\n"
rm -rf $CONFORMANCE_TESTS_FOLDER
mkdir -p $CONFORMANCE_TESTS_FOLDER
cp -R $2/* $CONFORMANCE_TESTS_FOLDER
printf "Copied from $2 to $CONFORMANCE_TESTS_FOLDER...\n"
# Move to the subfolder
cd "$CONFORMANCE_TESTS_FOLDER" 2>/dev/null
printf "Moved to $CONFORMANCE_TESTS_FOLDER...\n"
if [ $? -ne 0 ]; then
printf "Error: Java conformance tests folder '$CONFORMANCE_TESTS_FOLDER' does not exist.\n"
exit 2
fi
echo "Runinng maven install in $(pwd)..."
output=$(mvn clean install -DskipTests 2>&1)
exit_code=$?
# If there was an error, print the output and exit with the error code
if [ $exit_code -ne 0 ]; then
echo "Command failed: mvn clean install -DskipTests"
echo "Error: Running maven build failed with exit code $exit_code"
echo "Output: $output"
exit $exit_code
fi
# Execute all Java unittests in the subfolder
echo "Running Java unittests in $(pwd)..."
output=$(mvn test 2>&1)
exit_code=$?
# If there was an error, print the output and exit with the error code
if [ $exit_code -ne 0 ]; then
echo "$output"
exit $exit_code
fi
# Echo the original exit code of the unittest command
exit $exit_code#!/usr/bin/env pwsh
$ErrorActionPreference = 'Stop'
$UNRECOVERABLE_ERROR_EXIT_CODE = 69
# Check if build folder name is provided
if (-not $args[0]) {
Write-Host "Error: No build folder name provided."
Write-Host "Usage: $($MyInvocation.MyCommand.Name) <build_folder_name> <conformance_tests_folder>"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
# Check if conformance tests folder name is provided
if (-not $args[1]) {
Write-Host "Error: No conformance tests folder name provided."
Write-Host "Usage: $($MyInvocation.MyCommand.Name) <build_folder_name> <conformance_tests_folder>"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
$BuildFolder = $args[0]
$ConformanceTestsFolder = $args[1]
# Try to find Python interpreter (python3 first, then python)
if (Get-Command python3 -ErrorAction SilentlyContinue) {
$PYTHON_CMD = "python3"
} elseif (Get-Command python -ErrorAction SilentlyContinue) {
$PYTHON_CMD = "python"
} else {
Write-Host "Error: Python interpreter not found. Please install Python."
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
$current_dir = Get-Location
$PYTHON_BUILD_SUBFOLDER = "python_$BuildFolder"
if ($env:VERBOSE -eq "1") {
Write-Host "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER"
}
# Check if the Python build subfolder exists
if (Test-Path $PYTHON_BUILD_SUBFOLDER) {
# Delete all files and folders inside
Get-ChildItem -Path $PYTHON_BUILD_SUBFOLDER -Force | Remove-Item -Recurse -Force
if ($env:VERBOSE -eq "1") {
Write-Host "Cleanup completed."
}
} else {
if ($env:VERBOSE -eq "1") {
Write-Host "Subfolder does not exist. Creating it..."
}
New-Item -ItemType Directory -Path $PYTHON_BUILD_SUBFOLDER -Force | Out-Null
}
Copy-Item -Path "$BuildFolder/*" -Destination $PYTHON_BUILD_SUBFOLDER -Recurse -Force
# Move to the subfolder
if (-not (Test-Path $PYTHON_BUILD_SUBFOLDER)) {
Write-Host "Error: Python build folder '$PYTHON_BUILD_SUBFOLDER' does not exist."
exit $UNRECOVERABLE_ERROR_EXIT_CODE
}
Push-Location $PYTHON_BUILD_SUBFOLDER
try {
# Execute all Python conformance tests in the build folder
Write-Host "Running Python conformance tests...`n"
# Temporarily allow stderr output without throwing (Python unittest writes progress to stderr)
# ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting
$ErrorActionPreference = 'Continue'
$output = & $PYTHON_CMD -m unittest discover -b -s "$current_dir/$ConformanceTestsFolder" 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String
$exit_code = $LASTEXITCODE
$ErrorActionPreference = 'Stop'
# Echo the original output
Write-Host $output
# Check if no tests were discovered
if ($output -match "Ran 0 tests in") {
Write-Host "`nError: No unittests discovered."
exit 1
}
# Exit with the exit code of the unittest command
exit $exit_code
} finally {
Pop-Location
}
#!/bin/bash
UNRECOVERABLE_ERROR_EXIT_CODE=69
# Check if build folder name is provided
if [ -z "$1" ]; then
printf "Error: No build folder name provided.\n"
printf "Usage: $0 <build_folder_name> <conformance_tests_folder>\n"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
fi
# Check if conformance tests folder name is provided
if [ -z "$2" ]; then
printf "Error: No conformance tests folder name provided.\n"
printf "Usage: $0 <build_folder_name> <conformance_tests_folder>\n"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
fi
# Try to find Python interpreter (python3 first, then python)
if command -v python3 &> /dev/null; then
PYTHON_CMD="python3"
elif command -v python &> /dev/null; then
PYTHON_CMD="python"
else
printf "Error: Python interpreter not found. Please install Python.\n"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
fi
current_dir=$(pwd)
PYTHON_BUILD_SUBFOLDER=".tmp/$1"
if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then
printf "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER\n"
fi
rm -rf $PYTHON_BUILD_SUBFOLDER
mkdir -p $PYTHON_BUILD_SUBFOLDER
cp -R $1/* $PYTHON_BUILD_SUBFOLDER
# Move to the subfolder
cd "$PYTHON_BUILD_SUBFOLDER" 2>/dev/null
if [ $? -ne 0 ]; then
printf "Error: Python build folder '$PYTHON_BUILD_SUBFOLDER' does not exist.\n"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
fi
printf "Creating and activating virtual environment...\n"
# Time the virtual environment creation and activation
start_time=$(date +%s.%N)
VENV_DIR=".venv"
if ! $PYTHON_CMD -m venv "$VENV_DIR"; then
printf "Error: Failed to create virtual environment in '$VENV_DIR'.\n"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
fi
# shellcheck disable=SC1091
source "$VENV_DIR/bin/activate"
if [ $? -ne 0 ]; then
printf "Error: Failed to activate virtual environment at '$VENV_DIR/bin/activate'.\n"
exit $UNRECOVERABLE_ERROR_EXIT_CODE
fi
# Install requirements if requirements.txt exists
if [ -f "requirements.txt" ]; then
pip install --upgrade pip
pip install -r requirements.txt
else
echo "Warning: requirements.txt not found. Cannot proceed with setting up requirements. The requirements may also already be installed"
fi
end_time=$(date +%s.%N)
# Calculate and display the time taken
duration=$(echo "$end_time - $start_time" | bc)
printf "Requirements setup completed in %.2f seconds\n\n" "$duration"
# Execute all Python conformance tests in the build folder
printf "Running Python conformance tests...\n\n"
output=$($PYTHON_CMD -m unittest discover -b -s "$current_dir/$2" 2>&1)
exit_code=$?
# Echo the original output
echo "$output"
# Check if no tests were discovered
if echo "$output" | grep -q "Ran 0 tests in"; then
printf "\nError: No unittests discovered.\n"
exit 1
fi
# Echo the original exit code of the unittest command
exit $exit_code