Oro K6 Testing

Oro Commerce k6 Performance Testing

Overview

Oro ships a stock k6 test harness (from oro/performance-tests) that exercises the storefront from the outside — unauthenticated browsing, login, product listing/search/detail, shopping list creation, and checkout. Scripts are plain JavaScript, k6 is a single Go binary (or grafana/k6 Docker image), and every run emits named custom metrics you can gate with per-metric thresholds.

The Three-Script Pipeline

Scripts live in performance/scripts/ and run in this fixed order against a fresh or cleared cache. The order matters because step 1 primes caches that steps 2 and 3 measure — running them out of order inflates the numbers.

1. `warmingUpTheApp.js` — 1 VU for 60s, THRESHOLD_95=3000. Primes OPcache, Doctrine metadata, layout cache, and search index warm-up paths. Skipping this makes the load runs look slower than reality. 2. `storefrontTests.js` — full browse flow (home, listing, search, product detail, login). Typical 1 VU / 600s / THRESHOLD_95=850. 3. `checkoutTest.js` — authenticated checkout from an existing shopping list. Requires SL_ID, SHIPPING_METHOD, PAYMENT_METHOD.

Do not reuse the load-run threshold (850 ms) for the warm-up — first-touch is always slower and the run will fail on cold caches.

Environment Variables

All passed with -e KEY=VALUE:

Hero Run Command (Docker)

docker run --rm --network host -u "$(id -u):$(id -g)" \
  -v "${PWD}/performance:/home/k6/performance" -w /home/k6/performance \
  grafana/k6:latest run \
  -e BASE_URL="https://oro.docker.local" \
  -e USERNAME="AmandaRCole@example.org" \
  -e PASSWORD="AmandaRCole@example.org" \
  -e VU=1 -e DURATION=60s -e THRESHOLD_95=3000 \
  scripts/warmingUpTheApp.js

--network host lets k6 reach localhost and compose services; -u $(id -u):$(id -g) keeps summary.html owned by your user instead of root. See references/docker-invocation.md for every flag.

Custom Oro Metrics

Oro's scripts emit 14 named metrics on top of the k6 defaults — load_product_listing_page_logged_in_user, create_sl_request_post_request, authentication_post_request, etc. You can set per-metric thresholds in options.thresholds instead of a single global 95th percentile. Full list and threshold examples in references/custom-metrics.md.

Reports

handleSummary() in the stock scripts writes summary.html via k6-reporter. Open it locally — it contains per-group timing breakdowns, VU curves, and threshold pass/fail badges. A stdout text summary is also emitted.

Key Pitfalls

1. Running load tests without warming up first — cold-cache results inflate mean/p95 for everything. Always run warmingUpTheApp.js with THRESHOLD_95=3000 before any load run. 2. Docker run without `-u $(id -u):$(id -g)` — summary.html lands root-owned inside the mounted volume; subsequent runs fail to overwrite it and your user can't delete it without sudo. 3. Single global `THRESHOLD_95` — a slow product listing hides behind a fast login page in the aggregate. Target specific named metrics (e.g. load_product_listing_page_logged_in_user) in options.thresholds. 4. Default `--network bridge` against localhost — the container can't reach host compose services. Use --network host or attach the container to the compose network explicitly.

See Also

• references/custom-metrics.md — 14 named Oro metrics, descriptions, per-metric threshold examples
• references/docker-invocation.md — full grafana/k6 Docker run, flag-by-flag, network modes, file ownership
• references/default-metrics.md — the 9 built-in k6 metrics and derived KPIs (avg/peak RT, error rate, CRPS, concurrent users)
• references/install.md — Debian/Ubuntu, macOS, Windows, Docker install
• references/v6.1.md — v6.1 script layout, env var reference, warm-up rationale
• references/v7.0.md — 7.1-dev notes (placeholder)
• Oro k6 docs
• k6 documentation

Custom Oro k6 Metrics

Oro's stock k6 scripts emit 14 named metrics in addition to k6's built-ins. Each is a Trend metric (duration in ms) unless noted. Set per-metric thresholds in options.thresholds instead of relying on a single global p95 — a slow product listing will otherwise hide behind fast login requests in the aggregate.

The 14 Metrics

Guest vs. logged-in pairs exist because authenticated browsing triggers price-list resolution, customer-scope ACL, and cart/shopping-list side panels — those are measurably heavier than anonymous requests and deserve separate thresholds.

Per-Metric Thresholds

Instead of one global http_req_duration threshold, gate each named metric. Example options block inside a script:

export const options = {
  vus: Number(__ENV.VU),
  duration: __ENV.DURATION,
  thresholds: {
    // Guest pages — primed cache, should be fast
    'load_home_page_guest_user':           ['p(95)<600'],
    'load_product_listing_page_guest_user':['p(95)<850'],
    'load_product_detail_page_guest_user': ['p(95)<900'],
    'load_product_search_page_guest_user': ['p(95)<1200'],

    // Logged-in pages — heavier, price-list + ACL
    'load_home_page_logged_in_user':             ['p(95)<900'],
    'load_product_listing_page_logged_in_user':  ['p(95)<1200'],
    'load_product_detail_page_logged_in_user':   ['p(95)<1400'],
    'load_product_search_page_logged_in_user':   ['p(95)<1600'],

    // Auth + shopping list creation
    'authentication_post_request':    ['p(95)<1000'],
    'create_sl_request_post_request': ['p(95)<1200'],
    'create_sl_widget':               ['p(95)<1500'],

    // Failure rate gate
    'check_failure_rate': ['rate<0.01'],
  },
};

Warm-Up Overrides

During warmingUpTheApp.js the same metric names are emitted, but the cold-cache first-touch is 3–4× slower. Either set THRESHOLD_95=3000 as a single global gate (current Oro default) or multiply the per-metric thresholds above by ~3 in the warm-up script. Do not run the warm-up with load-run thresholds — it will fail on legitimate cold-start cost.

Reading Metric Values Out

Every run's stdout summary lists each named metric with avg, min, med, max, p(90), p(95). The HTML report (summary.html) renders the same table plus sparkline trends. For CI gating, use the JSON output (--summary-export=summary.json) and parse metrics.<name>.values.p(95).

Default k6 Metrics and KPIs

k6 emits nine built-in metrics automatically for every HTTP request, in addition to any custom metrics the script defines. These are always present — set global thresholds on them in options.thresholds.

The 9 Built-in HTTP Metrics

http_req_duration is the primary gate. Split it by group_duration when you need per-flow attribution without defining custom metrics.

Derived KPIs

These aren't separate metrics but are computed from the built-ins and surfaced in Oro's reports and the k6-reporter HTML:

Setting Global Thresholds

export const options = {
  thresholds: {
    http_req_duration:    ['p(95)<850', 'p(99)<2000'],
    http_req_failed:      ['rate<0.01'],
    iteration_duration:   ['p(95)<5000'],
  },
};

A threshold failure marks the run as failed (exit code 99 in k6 >= 0.50, or 1 in older versions) — useful for CI gating.

Running k6 via the grafana/k6 Docker Image

The grafana/k6 image is a tiny Alpine container shipping the k6 binary at /usr/bin/k6 as entrypoint. Any k6 subcommand (run, archive, inspect, cloud) works as the first argument.

Canonical Invocation

docker run --rm --network host -u "$(id -u):$(id -g)" \
  -v "${PWD}/performance:/home/k6/performance" -w /home/k6/performance \
  grafana/k6:latest run \
  -e BASE_URL="https://oro.docker.local" \
  -e USERNAME="AmandaRCole@example.org" \
  -e PASSWORD="AmandaRCole@example.org" \
  -e VU=1 -e DURATION=60s -e THRESHOLD_95=3000 \
  scripts/warmingUpTheApp.js

Flags Explained

Network Mode Alternatives

• Host network (shown above) — simplest when the target is on localhost or in /etc/hosts. Only works on Linux; on macOS/Windows Docker Desktop, --network host is a no-op.
• Attached to a compose network — use when the target lives in a compose stack:

  docker run --rm --network oro_default \
    -v "${PWD}/performance:/home/k6/performance" -w /home/k6/performance \
    -u "$(id -u):$(id -g)" grafana/k6:latest run \
    -e BASE_URL="http://nginx" ... scripts/storefrontTests.js

The hostname (nginx) must be a compose service on oro_default.

• `host.docker.internal` — on macOS/Windows Docker Desktop, use BASE_URL="http://host.docker.internal" instead of localhost with default bridge network. On Linux, add --add-host=host.docker.internal:host-gateway.

File Ownership Pitfall

Without -u, k6 runs as root inside the container. summary.html written through the bind mount keeps root ownership on the host:

-rw-r--r-- 1 root root 412K Apr 11 14:22 summary.html

The next run (as your user, with -u added) gets EACCES on write. Fix by sudo rm summary.html once, then always include -u "$(id -u):$(id -g)". There is no reason to ever run k6 as root.

TLS Verification

Oro dev environments usually have self-signed certs. k6 fails TLS verification by default. Add --insecure-skip-tls-verify to the run args or set insecureSkipTLSVerify: true in options.

Installing k6

Four install paths. Pick one.

Debian / Ubuntu (APT)

Imports Grafana's signing key into a dedicated keyring and adds the stable APT source:

sudo gpg -k
sudo gpg --no-default-keyring \
  --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
  --keyserver hkp://keyserver.ubuntu.com:80 \
  --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install -y k6

Verify with k6 version. The binary lives at /usr/bin/k6.

macOS (Homebrew)

brew install k6

Installs from the core formula (/opt/homebrew/bin/k6 on Apple Silicon, /usr/local/bin/k6 on Intel).

Windows (winget)

winget install k6 --source winget

Alternative: choco install k6 if you prefer Chocolatey. The binary lands on PATH automatically.

Docker

docker pull grafana/k6:latest

No local install — the binary runs inside the container. Pin a concrete tag (e.g. grafana/k6:0.52.0) for CI. See docker-invocation.md for the full docker run pattern including network mode and user flags.

Version Check

Regardless of install path:

k6 version
# k0.52.0 ((devel), go1.22.5, linux/amd64)

Oro's stock scripts target k6 >= 0.50. Older k6 versions may lack handleSummary() or recent options.thresholds features.

v6.1 Notes — Oro k6 Performance Tests

Stock Script Layout

On Oro 6.1, the k6 test harness sits in the oro/performance-tests package (or under performance/ at the app root when copied into a project):

performance/
├── scripts/
│   ├── warmingUpTheApp.js   # Cache priming run, 1 VU / 60s
│   ├── storefrontTests.js   # Full storefront browse flow
│   └── checkoutTest.js      # Checkout from an existing shopping list
└── summary.html             # Generated each run by k6-reporter's handleSummary()

All three scripts read configuration exclusively from __ENV.* — there is no config file. Arguments flow through docker run ... -e KEY=VALUE ... run scripts/X.js.

Environment Variable Reference

Warm-Up Rationale

k6 measures wall-clock durations. On a cold Oro stack the first request hits:

• OPcache miss (recompile all touched PHP)
• Doctrine metadata hydration (cache miss on class metadata + proxies)
• Layout cache miss (Oro Layout block tree rebuild)
• Search index cold-read (first elastic/ES query warms OS page cache)
• Fragment/ESI cache miss

The first 5–20 requests after cache:clear can be 3–4× slower than steady state. Running storefrontTests.js directly against a cold stack produces a p95 dominated by these one-time costs, not representative throughput.

warmingUpTheApp.js uses 1 VU / 60s / THRESHOLD_95=3000 precisely so those cold hits don't fail the run. It's a setup step, not a benchmark — its own numbers are not meaningful. What matters is that the subsequent storefrontTests.js run with THRESHOLD_95=850 reflects primed-cache performance.

Rule of thumb: after any cache:clear, platform update, or container restart, warm up before measuring.

Known Quirks

• summary.html is written by handleSummary() using the k6-reporter package imported from https://raw.githubusercontent.com/... — the first run downloads it from the network. Offline environments need to vendor it locally and patch the import.
• TLS verification defaults to strict. Dev stacks with self-signed certs need --insecure-skip-tls-verify on the k6 run command.

v7.0 / 7.1-dev Notes — Oro k6 Performance Tests

Status: placeholder. A fetch of the current Oro 7.1-dev master showed no material changes to the k6 performance test harness compared with 6.1. The script layout (performance/scripts/{warmingUpTheApp,storefrontTests,checkoutTest}.js), the 14 custom metric names, and the THRESHOLD_95 / VU / DURATION env var contract are all unchanged.

Follow the patterns in v6.1.md until this page is updated with verified 7.0+ differences.

Things To Recheck When 7.0 GAs

• Whether the k6-reporter HTML summary dependency is vendored vs. fetched from GitHub on each run.
• Whether any custom metric was renamed or added (e.g. new storefront pages, new checkout steps).
• Whether stock thresholds in options.thresholds shifted with any 7.0 storefront performance work.
• Whether SHIPPING_METHOD / PAYMENT_METHOD identifier formats changed (e.g. checkout workflow refactor).