Oro E2e Testing

OroCommerce v6.1 End-to-End Behat Testing

Overview

E2e tests drive a real browser against a deployed Oro application. Unlike functional/integration Behat (local, service-isolated, DB rolled back per scenario), e2e tests disable all isolators and interact with the app exclusively through Mink + ChromeDriver. The target can be staging, a dedicated QA instance, or a throwaway production clone.

The load-bearing distinction is the isolation flag:

• --skip-isolators — pure e2e. Disables DB, cache, and service-container isolation entirely. Tests only talk to the browser. No fixture loading. No direct service calls.
• --skip-isolators-but-load-fixtures — hybrid. Keeps the fixture-loader isolator active, disables everything else. Requires ORO_DB_DSN pointing at the remote DB and local source code whose migrations exactly match the deployed version. Mismatch produces silent garbage data.

If a step definition calls a service (not the browser), it will fail on a pure e2e run unless that service is reachable via env vars.

Hero behat.yml

imports:
  - ./behat.yml.dist

default: &default
  extensions: &default_extensions
    Behat\MinkExtension:
      base_url: "https://staging.example.com"
    Oro\Bundle\TestFrameworkBundle\Behat\ServiceContainer\OroTestFrameworkExtension:
      artifacts:
        handlers:
          local:
            directory: '%paths.base%/public/media/behat'
            base_url: ~
            auto_clear: false

Start ChromeDriver with Oro's expected endpoint (not selenium-standalone):

chromedriver --url-base=wd/hub --port=4444

Running Tests

# Pure e2e against the remote app
php bin/behat --skip-isolators -- path/to/feature.feature

# E2e that also loads Alice fixtures into the remote DB
php bin/behat --skip-isolators-but-load-fixtures -- path/to/feature.feature

# Install Oro's pre-built e2e scenarios and a .behat-secrets.yml.dist template
composer require oro/e2e-tests --dev -n
php bin/behat --skip-isolators -- vendor/oro/e2e-tests/Tests/Behat/Features/create_mailchimp_integration.feature

Secrets

Credentials live in .behat-secrets.yml at the application root, referenced from features as <Secret:login.username>. Never commit this file — a .behat-secrets.yml.dist template ships with the oro/e2e-tests package. See secret-management.md.

Self-Healing

Oro ships Reload Page Healer by default; OpenAI Healer is opt-in. Custom healers implement HealerInterface and get the oro_test.behat.healer service tag. See self-healing.md.

Watch Mode

Interactive development mode pauses on error with a "continue from line N" prompt and a cyclic retry loop. See watch-mode.md.

Key Pitfalls

1. Running e2e without `--skip-isolators` — the default isolators try to manage DB state on the remote app: dropping schema, loading fixtures, clearing caches. At best the run aborts; at worst it mutates the target silently. 2. `.app-env.local` vs `.env-app.test.local` — Oro's own documentation is internally inconsistent on the env file name. The e2e docs use .app-env.local for ORO_DB_DSN; the functional-test docs use .env-app.test.local. Pick the wrong one and the DSN isn't picked up — the fixture loader silently falls back to no DB. Full explanation in remote-db.md. 3. Committing `.behat-secrets.yml` — once in git history, rotation is the only remedy. Add it to .gitignore immediately; only commit the .dist template. 4. Local source mismatch with `--skip-isolators-but-load-fixtures` — the fixture loader runs locally but writes to the remote DB. If your local migrations differ from the deployed version's schema, rows land in columns that don't exist or skip columns that do. Fixtures appear to load; assertions fail later with baffling data. Always check out the exact deployed tag locally. 5. Testing production with real customer accounts — e2e tests mutate state permanently. No rollback exists. Create dedicated test users/customers/orders and never point a run at a real customer's data.

See Also

• references/secret-management.md — .behat-secrets.yml schema, <Secret:> syntax, installing the e2e tests package
• references/self-healing.md — Reload Page Healer, OpenAI Healer config, writing custom healers
• references/remote-db.md — env-file naming conflict, ORO_DB_DSN format, local-code-matches-deployed warning
• references/watch-mode.md — --watch prompt, line numbering, cyclic error recovery
• references/v6.1.md — ChromeDriver setup, oro/e2e-tests package, artifact handlers
• references/v7.0.md — 7.1-dev notes (env-file inconsistency not resolved upstream)
• oro-behat-testing skill — integration/functional Behat with full isolation
• oro-behat-debugging skill — screenshots, verbose output, Xdebug for failing steps

Remote DB Access for Fixture-Loading E2E

When you run php bin/behat --skip-isolators-but-load-fixtures, the fixture-loader isolator stays active and tries to connect to the remote application's database from your local machine to seed Alice fixtures. This needs two things in sync: a DSN in a local env file, and local source code whose migrations match the deployed version.

The Env File Naming Conflict

Oro's own documentation is internally inconsistent about which env file holds `ORO_DB_DSN`. Both the current 6.1 stable docs and the 7.1-dev master branch disagree with themselves:

• E2e docs page — uses .app-env.local
• Functional tests docs page — uses .env-app.test.local

The conflict is not resolved in 7.1-dev master as of this writing. Expect to see both forms in blog posts, Stack Overflow answers, and internal team notes. Old Oro docs that predate the split use variations like .env.test.local — ignore those entirely.

For e2e with `--skip-isolators-but-load-fixtures`, follow the e2e docs: use `.app-env.local`. The functional test path (.env-app.test.local) is a different code path used by --env=test PHPUnit functional suites, not the e2e Behat flow.

If ORO_DB_DSN doesn't appear to take effect:

1. Check file name spelling exactly — the two forms differ by one hyphen position: .app-env.local vs .env-app.test.local. 2. Make sure the file sits at application root, next to composer.json. 3. Try the other name as a diagnostic. If the other name works, you've hit the docs inconsistency on your Oro version — document which one works for your team and stick with it.

There is no runtime error when the DSN is missing. The fixture loader silently falls back to "no DB configured" and your fixtures land nowhere, leaving scenarios to fail at the first data assertion with confusing "user not found" messages.

DSN Format

ORO_DB_DSN=postgres://oro_db_user:oro_db_pass@db.staging.example.com:5432/oro_db

• Scheme: postgres:// (or postgresql://) for PostgreSQL, mysql:// for MySQL legacy mode. OroCommerce 6.1 primarily targets PostgreSQL.
• Host must be reachable from the local machine running Behat — open firewall rules, VPN tunnels, or SSH port forwards as needed.
• Default PostgreSQL port is 5432 (not 3306 — that's MySQL; some sample snippets in Oro docs carry a copy-paste bug with the wrong port).
• URL-encode special characters in the password (@ becomes %40, : becomes %3A).

Local Source Must Match Deployed Version

The fixture loader is a local process writing to a remote DB. It generates SQL from local entity metadata — local Doctrine mappings, local enum definitions, local ExtendEntity state. If the deployed application has ENUM columns, migrations, or ExtendEntity fields that don't exist in your local checkout (or vice versa), the loader writes to a schema it imagines, not the one actually on the remote.

Symptoms of a mismatch:

• "Column does not exist" errors mid-load with columns that clearly exist in the remote DB
• Fixtures load "successfully" but assertions fail because enum IDs, extend columns, or relations point to rows that were never persisted
• Random NULL values where fixture data should be — the loader dropped columns it didn't know about

Rule: before running --skip-isolators-but-load-fixtures, check out the exact git tag / commit SHA that is deployed on the target. Run composer install to match vendor versions. Clear var/cache/ so Doctrine metadata is rebuilt from the checked-out code.

Pitfalls

1. Wrong env file name — silent DSN fallback, not an error. Try both names when debugging. 2. Stale local cache after checking out the deployed tag — Doctrine metadata cache from the previous branch leaks into fixture generation. Always rm -rf var/cache/* after switching versions. 3. Port 3306 in a PostgreSQL DSN — copy-pasted from a docs example. PostgreSQL is 5432. 4. URL-encoded `@` inside passwords — forget to encode and the DSN parser reads the password as a second hostname.

Secret Management for E2E Tests

E2e tests authenticate against real applications, which means real credentials have to reach Behat somehow. Oro's e2e framework reads them from a YAML file at the application root, never from behat.yml itself.

File Location and Template

<app-root>/.behat-secrets.yml       # real secrets — NEVER commit
<app-root>/.behat-secrets.yml.dist  # template — safe to commit

The oro/e2e-tests package installs the .dist template:

composer require oro/e2e-tests --dev -n
cp .behat-secrets.yml.dist .behat-secrets.yml
# edit .behat-secrets.yml with real values

Add the real file to .gitignore immediately — before the first git add:

.behat-secrets.yml

Schema

secrets:
  login:
    username: admin
    password: s3crEtPas$
  api:
    token: abc123
  mailchimp:
    api_key: xyz-us12

The structure under secrets: is free-form — arrange keys to suit your scenarios. Dotted paths in feature files dereference nested keys.

Referencing Secrets in Features

Use <Secret:dotted.path> placeholders anywhere a step argument appears:

Feature: Admin login on staging
  Scenario: Log in with the deployment account
    Given I go to "admin"
    And I fill form with:
      | Username | <Secret:login.username> |
      | Password | <Secret:login.password> |
    And I click "Log in"
    Then I should see "Welcome"

Behat substitutes <Secret:login.username> with secrets.login.username from the YAML before the step executes. The substitution runs before table expansion, so placeholders work inside table cells, quoted arguments, and scenario outlines alike.

Scope

• One .behat-secrets.yml per application root. Multi-suite runs share the same file.
• Values are strings only. Structured data (lists, maps as values) is not supported by the <Secret:> resolver — flatten nested maps by adding more key levels.
• The file is read once at Behat startup. Editing it mid-run has no effect.

Pitfalls

1. Committing the real file — git log -p -- .behat-secrets.yml is the first thing an attacker checks. If committed even once, rotate every credential and force-push the history. Use the .dist template for anything that ships in git. 2. Placing the file anywhere but the application root — the resolver does not search subdirectories. tests/Behat/.behat-secrets.yml is silently ignored. 3. YAML syntax errors with special chars in passwords — wrap values containing $, :, #, or @ in single quotes: password: 's3crEtPas$'.

Self-Healing Behat Tests

Oro's Behat framework ships a healer pipeline that intercepts step failures and attempts recovery before the scenario is marked failed. Two built-in healers exist; a tagged-service mechanism lets you add more.

Reload Page Healer (built-in, always on)

When a step fails because an element cannot be found on the page (stale DOM, partial render, AJAX not yet settled), the Reload Page Healer refreshes the browser once and retries the step. No configuration required — it is wired up automatically and active for every e2e run.

The healer only fires on element-not-found failures. Assertion failures, step definition exceptions, and timeouts are not retried.

OpenAI Healer (experimental, opt-in)

The OpenAI Healer sends the failing step, the page source, and surrounding context to the OpenAI API and asks for a corrected step definition. It is experimental, costs money per failure, and leaks page content to a third party — enable only for interactive test authoring, never in CI.

Enable in behat.yml:

default: &default
  extensions: &default_extensions
    Oro\Bundle\TestFrameworkBundle\BehatOpenAIExtension\ServiceContainer\BehatOpenAIExtension:
      api_key: <OpenAI API Key>

The extension must be listed alongside OroTestFrameworkExtension. Store the key via env var substitution or a local override file — not the committed behat.yml.

Custom Healers

Implement Oro\Bundle\TestFrameworkBundle\Behat\Healer\HealerInterface and register the class as a service with the oro_test.behat.healer tag:

namespace Acme\Bundle\DemoBundle\Tests\Behat\Healer;

use Oro\Bundle\TestFrameworkBundle\Behat\Healer\HealerInterface;

class DismissOverlayHealer implements HealerInterface
{
    public function heal(/* ... framework-supplied context ... */): bool
    {
        // Attempt recovery. Return true if the step should be retried,
        // false to let the failure propagate.
    }
}

services:
  acme_demo.behat.healer.dismiss_overlay:
    class: Acme\Bundle\DemoBundle\Tests\Behat\Healer\DismissOverlayHealer
    tags:
      - { name: 'oro_test.behat.healer', priority: 100 }

The optional priority attribute on the tag orders healers — higher runs first. Healers execute in order until one returns true (retry) or all decline (failure propagates).

Pitfalls

1. Assuming all failures self-heal — only element-not-found errors reach the Reload Page Healer. Write steps that wait for the elements they need rather than relying on the healer as a general retry. 2. OpenAI Healer in CI — every failure triggers a paid API call and dumps page markup to OpenAI. Gate it behind a local-only behat.yml import. 3. Missing the tag — a healer class without the oro_test.behat.healer tag is never called. The container doesn't complain; the healer just doesn't run.

E2E Testing — v6.1 Notes

Key Environment

• PHP 8.1+ (matches main 6.1 runtime requirement)
• ChromeDriver matching the installed Chrome major version
• Behat is wired up via oro/test-framework-bundle; e2e-specific scenarios and the .behat-secrets.yml.dist template come from oro/e2e-tests

ChromeDriver Startup

Oro drives the browser through ChromeDriver's native WebDriver endpoint, not through selenium-standalone. Start it with:

chromedriver --url-base=wd/hub --port=4444

The --url-base=wd/hub flag is mandatory — Mink looks for the endpoint at that path. Omit it and every step fails at session creation with "unknown command". Port 4444 is the Mink default; if you change it, also change Behat\MinkExtension.sessions.*.wd_host in behat.yml.

Installing the E2E Package

composer require oro/e2e-tests --dev -n

This pulls in:

• vendor/oro/e2e-tests/Tests/Behat/Features/ — pre-built scenarios for common integrations (MailChimp, DotDigital, Google Tag Manager, etc.)
• .behat-secrets.yml.dist at the application root — template for credentials
• Additional step definitions for third-party service interactions

Run a bundled scenario:

php bin/behat --skip-isolators -- vendor/oro/e2e-tests/Tests/Behat/Features/create_mailchimp_integration.feature

Artifact Handlers

Screenshots, page dumps, and downloaded files from e2e runs go through the artifacts handler chain. The local handler (shown in the hero behat.yml in SKILL.md) writes to public/media/behat. Set auto_clear: false for e2e — you want evidence from failed runs to persist until you manually inspect them.

Other handlers in the 6.1 codebase: s3 for remote storage, null for debugging without writing anywhere.

Common v6.1 E2E Failures

1. ChromeDriver started without `--url-base=wd/hub` — every session creation times out 2. `.behat-secrets.yml` missing when features use `<Secret:>` placeholders — Behat aborts at extension load, not at the failing scenario 3. Running `--skip-isolators` against the local dev environment with an open browser — the run grabs the logged-in admin session from the dev instance and corrupts state you were mid-edit on 4. Chrome version drift — ChromeDriver major version must match installed Chrome; a mismatched pair fails with an unhelpful session not created error

E2E Testing — v7.0 Notes

OroCommerce 7.1 is currently -dev on master. This page tracks e2e-relevant changes observed in the master branch; it will be finalized when 7.1 stabilizes.

Observed State (7.1-dev master)

• --skip-isolators and --skip-isolators-but-load-fixtures flags unchanged from 6.1
• .behat-secrets.yml schema and <Secret:> placeholder syntax unchanged
• Reload Page Healer and OpenAI Healer extensions present with the same configuration surface
• HealerInterface and the oro_test.behat.healer tag mechanism unchanged
• Watch mode (--watch) behavior unchanged

Not Resolved in 7.1-dev

The .app-env.local vs .env-app.test.local documentation inconsistency described in remote-db.md persists in the 7.1-dev master branch. The e2e page still references .app-env.local; the functional tests page still references .env-app.test.local. Treat both forms as live until Oro ships an upstream clarification.

Expected Changes

• TBD — this file will be updated when 7.1 releases and any material e2e changes land.

Watch Mode

Watch mode turns a Behat run into an interactive development loop: each step is numbered, errors pause the run, and you can restart from any line without re-executing preceding setup. Useful for authoring new feature files against a real remote app, where replaying all setup steps for each iteration is slow.

Running

php bin/behat --watch -- path/to/feature.feature

The --watch flag is exclusive to the Oro Behat extension and takes no arguments. Combine with --skip-isolators when targeting a remote app:

php bin/behat --skip-isolators --watch -- path/to/feature.feature

Behavior

1. Before each step executes, Behat prints the step's line number in the feature file:

   #24  Given I go to "admin"
   #25  And I fill form with:
   #26    | Username | admin |

2. When a step fails, Behat does not abort. Instead it prints the failure and drops into an interactive prompt:

   Press ENTER to continue from the current line #26, or enter the line number
   to continue (Ctrl+C to exit):

3. Three choices at the prompt:

• ENTER — retry the step that just failed (after you've fixed it in the feature file or the app under test)
• Line number — jump back to an earlier step and replay from there (useful when the failure's root cause was a setup step two paragraphs up)
• Ctrl+C — exit watch mode entirely

4. After a successful completion the scenario restarts from the top — watch mode is cyclic until you Ctrl+C. This is deliberate: lets you iterate on test assertions without restarting Behat.

Typical Workflow

1. Write a feature file draft with rough step text. 2. php bin/behat --skip-isolators --watch -- drafts/new-flow.feature 3. First failure — fix the step wording or locator in the feature file, save. 4. Press ENTER to re-run from the failing line. Old state from earlier steps is preserved in the browser session. 5. Walk through the scenario until the last step passes. 6. Ctrl+C to exit.

Because watch mode bypasses isolation setup on each cycle, iteration is fast — no schema drop, no fixture reload. The trade-off is that scenario state accumulates between runs, which is exactly what you want for interactive authoring but deadly for suite runs.

Pitfalls

1. Using watch mode in CI — it blocks on a prompt forever. Watch mode is a local-only authoring tool. 2. Assuming each cycle starts clean — browser session, logged-in user, created records all persist across cycles. If earlier steps have side effects, later cycles diverge from the first run. 3. Saving the feature file without Behat picking up changes — Behat re-reads the feature file on each cycle, but not mid-cycle. Edit, then press ENTER.