Oro Behat Ci

OroCommerce v6.1 Behat in CI/CD

Oro's public CI support is Jenkins-only. GitLab CI works but is community territory — the upstream issue oroinc/platform#954 has been open since Oro 2.x with no ETA. This skill consolidates the canonical Jenkins patterns (scattered across oroinc/orocommerce-application, oroinc/docker-build, oroinc/environment) and the community-developed GitLab CI patterns that fill the gap.

Decision: Jenkins or GitLab?

Do not attempt GitLab native services: keyword for Oro. It cannot model the compose fanout.

Canonical Formatter Combo

Always run both formatters — pretty for humans tailing logs, junit for the CI report:

bin/behat \
  -f pretty -o std \
  -f junit  -o var/behat/junit \
  --strict

--strict fails the build on undefined and pending steps. Without it, skeleton steps accumulate silently as tech debt.

Hero Compose Snippet

Oro's mesh requires multiple services wired together. A compose-common.yaml (from oroinc/docker-build) defines them; the root compose.yaml fans out via include: driven by ORO_FILE_STORAGE_SERVICE, ORO_SEARCH_SERVICE, ORO_MQ_SERVICE env vars.

services:
  behat:
    image: ${ORO_IMAGE}:${ORO_IMAGE_TAG}-test
    depends_on: [consumer, operator, chrome, waf-behat]
    environment:
      ORO_ENV: test
      ORO_DB_HOST: postgres
    volumes:
      - ./var/behat:/var/www/oro/var/behat
    command: >
      bin/behat -f pretty -o std -f junit -o var/behat/junit --strict

  consumer:
    image: ${ORO_IMAGE}:${ORO_IMAGE_TAG}-test
    command: bin/console oro:message-queue:consume --env=test

  operator:
    image: ${ORO_IMAGE}:${ORO_IMAGE_TAG}-test
    command: bin/console oro:message-queue:consume --env=test

  chrome:
    image: selenium/standalone-chrome:142.0.7444.175
    volumes:
      - ./vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension:/chrome-extension:ro

  waf-behat:
    image: ${ORO_IMAGE}:${ORO_IMAGE_TAG}-test

Full service list and rationale in references/compose-fanout.md.

Jenkins Canonical Pattern

The canonical Jenkinsfile in oroinc/orocommerce-application commits the Behat stage commented out — you enable it per-project. Pattern:

stage('Behat') {
  sh 'docker compose up --exit-code-from behat behat'
  sh 'docker cp $(docker compose ps -q behat):/var/www/oro/var/behat ./var/behat'
}

docker cp is the artifact extraction mechanism — --exit-code-from does not propagate file state. Full details in references/jenkins-canonical.md.

GitLab CI — the Gap Fill

Use DinD (docker:dind service), run docker compose up from inside. Cache composer, vendor, and pull the init-test image for DB state. Full pipeline template, artifact config, consumer math, composer GitLab auth, Chrome extension mount, and ReconnectingConnection notes in references/gitlab-ci.md.

Init / Init-Test Images

oroinc/docker-build produces four image variants:

• runtime — CLI / PHP-FPM / web
• test — adds Behat + PHPUnit + tools
• init — prod DB + MongoDB pre-loaded (--env=prod)
• init-test — prod DB + MongoDB pre-loaded (--env=test)

The init images are the cross-job state transfer mechanism. Pull the image, restore pre-baked DB state, skip full oro:install (5–10 min per job). See references/jenkins-canonical.md.

Key Pitfalls

1. GitLab native `services:` for Oro's mesh — cannot model the fanout; fails silently on missing dependencies. Use docker compose up inside DinD. 2. Running Behat without `--strict` — undefined and pending steps silently pass in CI while accruing skeleton debt. 3. Parallel × consumer contention — N GitLab shards times --consumers=2 each equals 2N MQ workers competing on one broker. With parallel: 8 and 2 consumers per job that is 16 workers; plan broker capacity or drop --consumers. 4. Missing Chrome extension mount — vanilla selenium/standalone-chrome lacks Oro's custom extension at vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension. Symptom: vague JS errors, missing UI hooks. Mount read-only from the PHP container's vendor tree. 5. Composer install without internal GitLab token — private vendor/* and your-vendor/* packages fail to resolve. Configure composer config gitlab-token.your-gitlab-host.example.com <token> in the job before composer install. 6. No init-test image snapshot — running oro:install fresh in every job wastes 5–10 min per build. Use docker commit or pull the pre-baked init-test image for state transfer. 7. ChromeDriver drift — using :latest breaks when Chrome updates and the driver lags. Pin the full version (e.g., 142.0.7444.175) and bump deliberately.

See Also

• jenkins-canonical.md — oroinc sources, Behat stage, init-image pattern, GNU-parallel sharding, nightly @e2esmokeci default
• gitlab-ci.md — DinD pipeline, artifacts, cache layers, parallelization math, composer auth, Chrome mount, ReconnectingConnection
• compose-fanout.md — the service mesh Oro needs, env-var-driven fanout
• chrome-headless.md — mandatory flags with rationale, ChromeDriver pinning, memory_limit=-1
• v6.1.md — 6.1-stable CI specifics
• v7.0.md — 7.x CI notes (placeholder)

Chrome Headless for Oro Behat in CI

Oro's @javascript steps drive a real Chrome instance via ChromeDriver + Mink. In CI you need headless Chrome with a specific flag set — anything else causes vague JS errors, missing UI hooks, or phantom click failures.

Mandatory Flags

--headless
--no-sandbox
--disable-dev-shm-usage
--disable-extensions
--no-pings
--window-size=1920,1080
--load-extension=vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension

Per-flag rationale:

ChromeDriver Version Pinning

Pin the full patch version, e.g., selenium/standalone-chrome:142.0.7444.175. Never use `:latest`.

Reason: Chrome auto-updates on the Selenium image's base layer. ChromeDriver tracks Chrome, but there's a lag window (hours to days) where Chrome updates and ChromeDriver hasn't caught up. During that window all tests fail with SessionNotCreatedException: session not created: This version of ChromeDriver only supports Chrome version N.

Pin, bump deliberately, test the bump in a branch before merging to the shared compose file.

PHP memory_limit = -1

Non-optional in Oro's test image. Behat contexts hold DoctrineFixtures, parsed YAML graph data, and accumulated scenario state — 512MB is not enough for the full suite. The canonical test Dockerfile sets:

RUN echo 'memory_limit = -1' > /usr/local/etc/php/conf.d/behat-memory.ini

If you see PHP Fatal error: Allowed memory size of X bytes exhausted mid-scenario, your test image is missing this line.

Extension Mount Path

The extension lives in the vendor tree: vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension. It's shipped as source (unpacked, not a .crx), which is why --load-extension=<path> (not --load-unpacked=<path> — that's a different API) works.

In Docker: the path must be visible inside the `chrome` container. The common setup mounts it read-only from the PHP container's vendor directory:

chrome:
  image: selenium/standalone-chrome:142.0.7444.175
  volumes:
    - ./vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension:/chrome-extension:ro

And the behat mink config references /chrome-extension (the in-container path), not the host path.

Compose Fanout: the Service Mesh Oro Needs

Oro's Behat suite touches HTTP, message queue, search, file storage, and a real browser. Neither Jenkins agents nor GitLab services: can model this — you need docker-compose with include: fanout.

The Service List

From compose-common.yaml in oroinc/docker-build:

Backing Services (env-driven)

The compose include: mechanism pulls in different backing stacks based on env vars:

This is how one compose.yaml spans a full matrix test. In practice most projects pin to one combination (e.g., postgres + opensearch + rabbitmq + local + redis) and only test the matrix on release branches.

Dependency Graph

behat
  ├── install-test  (service_completed_successfully)
  ├── consumer     (service_started)
  ├── operator     (service_started)
  ├── chrome       (service_healthy)
  └── waf-behat    (service_started)

consumer, operator, waf-behat
  ├── postgres     (service_healthy)
  ├── redis        (service_started)
  ├── rabbitmq     (service_healthy)
  └── opensearch   (service_healthy)

install-test
  ├── postgres     (service_healthy)
  ├── rabbitmq     (service_healthy)
  └── opensearch   (service_healthy)

The condition: service_healthy dependencies require healthchecks on every backing service. Oro's compose-common.yaml ships them; don't strip them.

Fanout Invocation

Root compose.yaml looks like:

include:
  - path: docker-build/compose/compose-common.yaml
  - path: docker-build/compose/compose-${ORO_DB_SERVICE:-postgres}.yaml
  - path: docker-build/compose/compose-${ORO_MQ_SERVICE:-rabbitmq}.yaml
  - path: docker-build/compose/compose-${ORO_SEARCH_SERVICE:-opensearch}.yaml

Env var defaults via ${VAR:-default} mean a bare docker compose up behat works without setting anything. CI pipelines override per matrix cell.

Why Not Kubernetes?

include: has no k8s equivalent. You would need Helm charts with {{ if .Values.db.postgres }} conditionals for every backing service — roughly 4–6 weeks of migration work for no behavior improvement. Stick with compose for behat.

GitLab CI: the Gap Fill

Oro does not publicly support GitLab CI (oroinc/platform#954, open since 2.x). This document is the community-developed pattern set that makes it work anyway.

Runner Selection

Three options, ranked by maintenance cost:

1. DinD (`docker:dind` service) — recommended. The job runs in a docker:cli image with a docker:dind service; inside the job you run docker compose up. Pro: full compose fanout works. Con: nested Docker, slower than bare-metal. 2. Docker socket mount — the job mounts /var/run/docker.sock from the runner host. Faster than DinD but couples CI jobs to the runner host state. Risky. 3. Shell runner — the runner is a bare VM with Docker pre-installed. Fastest, but you hand-manage runner cleanup and cross-job isolation. Only use if you already have a dedicated runner pool.

Do not use kubernetes executor for Oro Behat unless you have time to rewrite the compose fanout as Helm charts. The compose include: mechanism has no k8s equivalent.

Why Not GitLab Native services:?

GitLab's services: keyword can launch per-job sidecar containers but cannot:

• Model depends_on with health conditions
• Share volumes between sidecars
• Respect compose include: fanout
• Mount files from the job workspace into sidecars

Oro's mesh needs all four. Attempting services: ends with silent failures on the first scenario that touches the search index or MQ. Use docker compose up inside DinD instead.

Minimal Pipeline Template

behat:
  stage: test
  image: docker:27-cli
  services:
    - name: docker:27-dind
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    ORO_IMAGE: registry.example.com/oro/app
    ORO_IMAGE_TAG: ${CI_COMMIT_SHA}
  before_script:
    - apk add --no-cache docker-compose-plugin git
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - docker compose pull
  script:
    - docker compose up --exit-code-from behat behat
  after_script:
    - docker cp $(docker compose ps -q behat):/var/www/oro/var/behat ./var/behat || true
    - docker compose down -v
  artifacts:
    when: always
    paths:
      - var/behat/
    reports:
      junit: var/behat/junit/*.xml
    expire_in: 1 week
  parallel: 4

Artifact Collection

Two separate stanzas:

• artifacts:reports:junit — parses JUnit XML into GitLab's test report UI. Required for the PR widget's "2 tests failed" summary.
• artifacts:paths — captures screenshots, HTML dumps, and the raw JUnit files. Oro writes failure artifacts to var/behat/ — specifically var/behat/screenshots/ and var/behat/html/.

Both must be inside when: always because you need the artifacts when the job fails, which is the common case for debugging.

Cache Layers

Three levels, cheapest to most expensive:

cache:
  - key: composer-${CI_COMMIT_REF_SLUG}
    paths: [.composer-cache/]
  - key: vendor-${CI_COMMIT_REF_SLUG}
    paths: [vendor/]

The hard part Oro does not solve with file caches: Behat needs DB state, not files. GitLab cache is file-based. Oro solves this via docker commit snapshot images (init-test) from the docker-build repo. Pull the init-test image instead of running oro:install in the job.

Parallelization × Consumer Contention

GitLab parallel: N runs N copies of the job. Each job starts its own consumer and operator compose service. If each runs --consumers=2, total MQ workers is 2N. Concrete:

• parallel: 4 × --consumers=2 = 8 workers on one broker
• parallel: 8 × --consumers=2 = 16 workers on one broker

RabbitMQ defaults to 100 channels per connection. You hit queue depth contention long before you hit channel limits. Either drop --consumers=1 or plan broker capacity at 2N * sustained_throughput.

Composer Auth for Internal GitLab

Private vendor/* and your-vendor/* packages are hosted on your-gitlab-host.example.com. Composer needs auth:

before_script:
  - composer config gitlab-token.your-gitlab-host.example.com "$GITLAB_COMPOSER_TOKEN"
  - composer install --no-interaction --no-progress

$GITLAB_COMPOSER_TOKEN is a masked CI variable containing a Personal or Project Access Token with read_api scope. Without this, composer install fails with "could not authenticate against your-gitlab-host.example.com" on the first private package.

Chrome Extension Mount

Oro loads a custom Chrome extension from vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension. It intercepts network requests, injects test hooks, and enables the @javascript step definitions. Vanilla selenium/standalone-chrome does not ship this extension.

In DinD, the path must be reachable inside the chrome container. Either:

1. Copy from the PHP container's vendor tree at compose time via an init container 2. Bind-mount from the job workspace: ./vendor/oro/platform/src/Oro/Bundle/TestFrameworkBundle/Resources/chrome-extension:/chrome-extension:ro

Option 2 only works if composer install runs on the host before compose — which in DinD means running composer in a throwaway container and mounting the result. See the hero snippet in SKILL.md.

ReconnectingConnection

Long Behat runs (>30 min) hit MySQL/Postgres's wait_timeout, the DB connection goes stale, and the next step fails with "MySQL server has gone away" or "SSL connection has been closed unexpectedly".

Fix: configure doctrine to use a reconnecting wrapper. In config/config.yml:

doctrine:
  dbal:
    connections:
      default:
        wrapper_class: Oro\Bundle\EntityBundle\Tools\SafeDatabaseChecker

(Or equivalent Oro/project-specific reconnecting wrapper — exact class name varies.) The wrapper pings the connection before each query and reconnects on failure. Required for CI runs longer than the DB's idle timeout.

behat_test vs test Env Divergence

Oro canonical uses ORO_ENV=test. Some projects diverge and use behat_test as a separate Symfony env. Document the canonical form here; project-specific overrides belong in project-scope skills. If the project uses behat_test, set ORO_ENV=behat_test consistently across all compose services — mismatches cause the consumer and the behat runner to see different caches and the tests silently run against stale state.

Jenkins: Oro's Canonical CI Patterns

Oro publishes its CI patterns exclusively for Jenkins. The pieces are scattered across three oroinc repos:

• [oroinc/orocommerce-application](https://github.com/oroinc/orocommerce-application) — the reference Jenkinsfile at repo root
• [oroinc/docker-build](https://github.com/oroinc/docker-build) — compose-common.yaml and the image build scripts
• [oroinc/environment](https://github.com/oroinc/environment) — ci/behat.sh, the legacy sharding runner

None of these repos document a complete working example in one place. The patterns below are the consolidation.

The Behat Stage (Shipped Commented Out)

The canonical Jenkinsfile ships the Behat stage committed but commented out. Projects enable it by removing the comment markers. Minimal shape:

stage('Behat') {
  steps {
    sh '''
      docker compose up --exit-code-from behat behat
    '''
  }
  post {
    always {
      sh 'docker cp $(docker compose ps -q behat):/var/www/oro/var/behat ./var/behat || true'
      junit 'var/behat/junit/*.xml'
      archiveArtifacts artifacts: 'var/behat/**', allowEmptyArchive: true
    }
  }
}

docker cp is the artifact extraction mechanism. --exit-code-from propagates the exit status but not file state — volumes work, but in Jenkins agents without a persistent workspace mount you must docker cp after the run.

Compose Fanout

The compose.yaml in orocommerce-application uses Docker Compose 2.x include: to fan out service definitions from compose-common.yaml (in docker-build). The fanout is driven by env vars read at compose time:

• ORO_FILE_STORAGE_SERVICE — local | s3 | gcs
• ORO_SEARCH_SERVICE — elasticsearch | opensearch
• ORO_MQ_SERVICE — rabbitmq | dbal | sqs | kafka

This means one compose.yaml produces a different service mesh per CI matrix cell. See compose-fanout.md for the full list.

Image Types

docker-build produces four image variants off the same Dockerfile, distinguished by build args and entrypoint:

The init images are the cross-job state transfer mechanism. Full oro:install takes 5–10 min. Instead:

1. A nightly (or on-merge) job runs oro:install once and docker commits the running container. 2. The commit becomes the init-test image, pushed to the registry. 3. Every PR job pulls init-test and starts from a ready-to-test DB state.

This is the single biggest CI speedup Oro ships. It's also why file-based caches do not help — Oro's state lives in Postgres and MongoDB, not on disk.

Parallelization (Legacy GNU-Parallel)

oroinc/environment/ci/behat.sh contains the reference sharding script. Shape:

find tests/Behat/Features -name '*.feature' \
  | parallel -j "${BEHAT_JOBS:-4}" --line-buffer \
    'bin/behat --strict -f pretty -o std -f junit -o var/behat/junit-{%}.xml {}'

Each worker gets a unique --profile and its own DB template via the docker commit install-snapshot trick. Note: docker-build/scripts does not contain a Behat execution script — only test_behat_wiring_cs.sh, which lints feature file code style. You write the runner yourself.

Nightly Default

The nightly Jenkins job runs:

bin/behat --strict --skip-isolators --tags=@e2esmokeci

--skip-isolators disables the per-scenario state isolators (feature isolator, messaging isolator, search index isolator). This is smoke e2e mode — fast, non-isolated, not a full test run. Full isolated runs are reserved for release branches.

Service Dependencies

The behat service in compose-common.yaml declares:

depends_on:
  consumer:    { condition: service_started }
  operator:    { condition: service_started }
  chrome:      { condition: service_healthy }
  waf-behat:   { condition: service_started }

chrome uses service_healthy because selenium-standalone exposes a healthcheck on /status. Missing the healthcheck means behat starts before ChromeDriver is ready and dies with "unable to connect to the remote server" on the first JS step.

Canonical Env Vars

• ORO_ENV=test — canonical Behat env
• ORO_INSTALL_TIMEOUT=3600 — oro:install often exceeds the Symfony default timeout
• ORO_BEHAT_ARGS — project-specific args appended to the behat command

Oro 6.1 CI Specifics

6.1 is the current stable line. CI-relevant notes:

Image Tags

oroinc/docker-build tags the 6.1 series as 6.1, 6.1-latest, and 6.1.x-y (patch builds). Use 6.1 in CI to get the latest patch automatically, 6.1.x-y for reproducible pins. The 6.1 line ships both runtime and test variants; init and init-test exist but are not published on a fixed schedule — projects typically build and push their own init images nightly.

PHP Version

6.1 runs on PHP 8.3. The test image uses PHP 8.3 with the same extensions as runtime plus xdebug (coverage mode, disabled by default). Enable coverage only on the coverage job — XDEBUG_MODE=coverage adds ~3x to run time.

Symfony Kernel

6.1 is on Symfony 6.4 LTS. bin/console accepts the standard --env=test flag. The test kernel reads .env.test in addition to .env — if your CI job sets env vars via shell export, they override .env but are overridden by .env.test.local (which should not exist in CI).

Message Queue

6.1 supports dbal, amqp (RabbitMQ), sqs, and kafka transports. The DBAL transport polls every second and is unsuitable for CI — scenarios that wait on async messages accumulate seconds per step. Use RabbitMQ in CI regardless of what production uses.

Search Backend

6.1 supports Elasticsearch 7/8 and OpenSearch 2.x. OpenSearch is the recommended default for new projects (licensing). The search isolator in 6.1 deletes and recreates the index per scenario — expensive. --skip-isolators on smoke runs avoids this.

Install Timeout

oro:install --env=test on 6.1 with a realistic fixture set takes 6–9 minutes. Set ORO_INSTALL_TIMEOUT=3600 in CI to avoid hitting the Symfony process default of 60s on any sub-command.

Fixture Loading

6.1 uses oro:migration:data:load with the --fixtures-type=test flag for test-only fixtures. CI's install-test image bakes these in. Projects that add custom test fixtures must register them under the test fixtures type or they won't load.

Oro 7.x CI Notes

Placeholder — 7.x is in development on master in the oroinc repos. As of this writing, no public changes to CI patterns in the canonical Jenkinsfile or compose-common.yaml. The 7.x line uses the same four image variants (runtime, test, init, init-test), the same compose fanout, and the same Behat command surface.

Known Forward-Looking Changes

• PHP 8.4 — 7.x bumps the minimum PHP version. Test images will tag 8.4. No CI pipeline change required beyond pulling the new base image.
• Symfony 7.x — kernel env handling is unchanged for CI purposes.
• Message queue — DBAL transport deprecation discussed on the Oro forum but not confirmed for 7.0. Use RabbitMQ in CI either way.

When 7.0 ships stable, update this file with concrete divergences from 6.1. Until then, follow v6.1.md.