Workflow Debugging

AEM Workflow Debugging — Cloud Service

Production-grade debugging for the AEM Granite Workflow engine, launcher, Inbox, Sling Jobs, thread pools, and purge on AEM as a Cloud Service (AEMaaCS).

Audience

AEMaaCS developers and operators (and the IDE LLM acting on their behalf) diagnosing stuck or failed workflows on a local AEMaaCS SDK or a cloud environment — Developer Console, Sling Job Console, and Cloud Manager Logs available; no production Felix Console JMX or filesystem access.

Variant Scope

• AEM as a Cloud Service only.
• Not for AEM 6.5 LTS / AMS. If the target is 6.5 LTS, stop and load the 6.5-lts variant of this skill — JMX-based remediation, Felix Console runtime config, AMS log filesystem access, and jstack thread dumps documented there do not apply on AEMaaCS.
• No JMX access on AEMaaCS production. Diagnosis is read-only via Developer Console, Sling Job Console, and Cloud Manager Logs. Never recommend JMX-based remediation (restartStaleWorkflows, purgeCompleted, terminate, retryFailedWorkItems) — those are 6.5-LTS-only mechanisms. Use Inbox Retry, Purge Scheduler (OSGi config in Git), custom servlets like StaleWorkflowServlet, and Cloud Manager pipeline-driven config changes instead.
• All remediation lands via Git + Cloud Manager pipeline: OSGi configs in ui.config, custom servlets in core, ACLs in ui.apps/.../repoinit. There is no Felix Console write access on cloud environments.
• See reference.md for runbook locations and diagnostic tool pointers.

Dependencies

This skill is largely self-contained but routes back into the dev skills when the root cause is a code or model defect:

• workflow-development — when the diagnosis is "process step throws / not registered / leaks resources"
• workflow-model-design — when the diagnosis is "model has wrong split rule / missing transition / wrong step type"
• workflow-launchers — when the diagnosis is "launcher not firing / re-trigger loop"
• workflow-triaging — load instead of this skill if the user is mining Cloud Manager Logs across multiple environments rather than diagnosing one

---

When to use this skill

• Workflow stuck, not progressing, failed, not starting, task not in Inbox, purge/repository bloat, permissions, queue backlog, thread pool exhaustion, auto-advancement not working.
• User provides thread dumps, Sling Job console output, or error.log excerpts from Cloud Manager.
• Environment: AEM as a Cloud Service (no JMX; use Developer Console, Cloud Manager logs).

---

Step 1: Map symptom to first action

---

Step 2: Decision tree (workflow stuck)

1. No current work item? → Stale. Deploy a custom StaleWorkflowServlet to your core bundle; call GET /bin/support/workflow/stale?dryRun=true to enumerate, then POST ...?dryRun=false to restart. 2. Participant step → Assignee exists? Inbox visible? Payload accessible? Dynamic participant resolver returning correct user? 3. Process step → Search error.log for instance ID. Check: process.label registered, payload path exists, bundle active, no exception in execute(). 4. OR/AND Split → Condition evaluates correctly? Routes exist? No dead-end branches? Model synced?

---

Step 3: Thread dump & thread pool analysis

Thread dumps and status-producer output on AEMaaCS are obtained via Developer Console → Status → Thread Dump / OSGi / Sling Jobs / Sling Scheduler. For anything not exposed in Developer Console, open an Adobe Support ticket — never attempt to SSH into an AEMaaCS pod.

3a. Sling default thread pool (critical path)

The Sling Scheduler ApacheSlingdefault uses ThreadPool: default. This pool runs:

• Oak observation events
• All Quartz-scheduled jobs — including the workflow timeout-detection scheduler that emits com/adobe/granite/workflow/timeout/job events to the Sling Job system (the job itself then runs on the Granite Workflow Queue, see Step 3c)

Check in thread pool output (Developer Console → Status → Sling Thread Pools, `/system/console/status-slingthreadpools`):

If active count = max pool size AND block policy = ABORT:

• New scheduled tasks (including workflow timeout/auto-advance jobs) are silently rejected
• This is the #1 cause of auto-advancement failure

Check in thread dump:

• Search for sling-default- threads
• If all threads show same stack (e.g. stuck on HTTP call, database, or external service), that's the blocking culprit
• Note elapsed time — threads stuck for hours indicate a hung external call without timeout

3b. Sling Job thread pool

Check `Apache Sling Job Thread Pool`:

• active count vs max pool size
• If saturated, Sling Jobs cannot execute (workflow jobs stall)

3c. Granite Workflow Queue

Check the Sling Jobs page (`/system/console/slingevent`):

Check topic statistics for workflow model:

• Topic: com/adobe/granite/workflow/job/var/workflow/models/<modelName>
• High Failed Jobs / low Finished Jobs ratio → process step throwing exceptions

Check Granite Workflow Queue configuration:

• Type: Topic Round Robin
• Max Parallel: `0.5` OOTB on the AEMaaCS SDK (50% of available CPU cores). Adobe's Workflows Best Practices recommends between half and three-quarters of processor cores. Verify the running value at Developer Console → /system/console/slingevent before assuming. Override via org.apache.sling.event.jobs.QueueConfiguration-<alias>.cfg.json in Git if you need to raise it.
• Max Retries: 10

3d. Sling Scheduler

Check the Sling Scheduler page (Developer Console → Status → Sling Scheduler, `/system/console/status-slingscheduler`):

• This page lists Quartz-style schedulers, not Sling Job topics. The workflow-related entry visible here is the periodic WorkflowStatsMBean collector (used by the Statistics MBean). nextFireTime: null means the trigger was deregistered.
• The com/adobe/granite/workflow/timeout/job topic itself is a Sling Job, not a Quartz job — check it on the Sling Jobs page (/system/console/slingevent), not here.
• Confirm ApacheSlingdefault uses ThreadPool: default — that's how the periodic timeout-detection scheduler reaches the workflow engine.

---

Step 4: Error log patterns

Download error.log from Cloud Manager → Environments → Logs, or use log streaming.

---

Step 5: Configuration checklist (Cloud Service — all via Git + pipeline)

Every config below is an OSGi JSON file under ui.config/src/main/content/jcr_root/apps/<project>/osgiconfig/config.author/ (or config.author.prod/ / config.author.stage/ for run-mode scoping).

AEMaaCS caveat — Sling DefaultThreadPool config may be platform-reserved

Some Sling core configs are filtered or overridden by the AEMaaCS platform layer. The DefaultThreadPool config may land via pipeline and be silently ignored. Always verify after deploy:

• Open /system/console/status-Threads (Developer Console → Status → Threads).
• Find the default pool row.
• Confirm maxPoolSize and block policy reflect your config values.

If the numbers don't change, the PID is Adobe-managed on your environment — do not try to work around it. Open an Adobe Support ticket, attach a thread dump and the thread-pool status, and request Engineering lift the pool size or change block policy for that environment.

Permission and identity gotchas specific to AEMaaCS

• Never list IMS user IDs in cq.workflow.superuser. IMS principals change on re-invite. Reference a JCR group that repoinit creates. Correct repoinit syntax:

  create group workflow-administrators
  add admin to group workflow-administrators

The group keyword after to is required — without it the repoinit parser fails and the entire script aborts on startup.

• Custom process steps that need elevated access must use a service user + ServiceUserMapperImpl.amended-*.cfg.json, not resolver.adaptTo(Session.class) + admin.
• enforceWorkflowInitiatorPermissions=true + an initiator who is a rotated IMS user leaves workflows unterminable except by superuser. On Cloud Service, prefer superuser-group membership for anyone expected to recover workflows.

---

Step 6: Remediation quick reference (Cloud Service)

Pod-restart reality on AEMaaCS: Cloud Manager does not expose a customer-facing pod-restart or env-restart action. The only way a customer can trigger a restart is an Adobe Support ticket. A restart bounces the running author/publish node — in-flight authoring sessions are lost, active jobs are requeued, there is no hot-swap. Treat it as last-resort mitigation, not a fix, and always file the long-term code/config fix in the same support conversation.

---

Step 7: Common root cause patterns (from real incidents)

Pattern A: Thread pool starvation → auto-advance failure

Symptom: Workflow auto-advancement stops; timeout jobs not firing; workflows stuck at participant step despite timeout configured.

Root cause chain: 1. Custom scheduler makes blocking HTTP call without timeout 2. concurrent = true allows overlapping executions on each cron trigger 3. Each stuck execution consumes a default pool thread indefinitely 4. All pool threads consumed (OOTB on the AEMaaCS SDK that's 10; production environments may differ) → pool saturated 5. If block policy has been changed to ABORT (OOTB is RUN), new Quartz triggers are rejected silently; on RUN they instead pile up on the caller thread and back-pressure the dispatch 6. The workflow timeout-detection scheduler cannot dispatch new com/adobe/granite/workflow/timeout/job events 7. Auto-advancement never happens

Diagnosis checklist:

• [ ] Thread pool output: Pool default → active count = max pool size?
• [ ] Thread pool output: Pool default → block policy = ABORT?
• [ ] Thread dump: All sling-default-* threads stuck on same stack?
• [ ] Sling Jobs output: Workflow job topic has high Failed Jobs?
• [ ] Sling Scheduler output: ThreadPool = default for ApacheSlingdefault?

Fix: Request pod restart (immediate mitigation, coordinate with customer — see Step 6 caveat); commit scheduler fix (HTTP timeout, scheduler.concurrent=false) to Git; land org.apache.sling.commons.threads.impl.DefaultThreadPool-default.cfg.json with blockPolicy=RUN and maxPoolSize=50; deploy via Cloud Manager pipeline.

Pattern B: High workflow job failure rate

Symptom: numberOfFailedJobs >> numberOfFinishedJobs for a workflow topic.

Root cause: Process step exception, payload deleted, or process not registered.

Diagnosis: Download or stream error.log from Cloud Manager → Environments → Logs; grep for Error executing workflow step + model name. Cross-check the process.label in Developer Console → OSGi → Components against the model's Process step.

Pattern C: Stale workflows accumulating

Symptom: Workflows in RUNNING state but no work items; Inbox empty despite running instances.

Root cause: Cannot archive workitem during transition; JCR session crash during step completion.

Diagnosis: grep Cloud Manager logs for Cannot archive workitem. For live count, deploy a custom StaleWorkflowServlet and invoke GET /bin/support/workflow/stale?dryRun=true — it returns a JSON report without side effects.

---

Routing back to dev skills

Once diagnosis identifies a code or model defect (not an operational issue on a healthy implementation), route back into the development skills:

---

References

• For runbook locations and diagnostic tool pointers: see reference.md

AEM Workflow Debugging – Reference (Cloud Service)

Quick pointers used by the workflow-debugging skill. Use the SKILL.md Step 1 symptom table for the symptom → first-action map; the entries below are for quick access to diagnostic tools, log patterns, and external documentation.

---

Cloud Service diagnostic tools

Note: /system/console/jmx is not available on AEMaaCS production. The same MBeans exist on the local AEMaaCS SDK, but production diagnosis must use the status producers above plus Cloud Manager logs.

---

Log patterns

• Error executing workflow step – process/step exception
• getProcess for '<name>' failed – process not registered (process.label mismatch)
• Cannot archive workitem – stale risk; archive failed during step transition
• refreshing the session since we had to wait for a lock – contention; reduce queue.maxparallel, do not raise it
• Terminate failed / Resume failed / Suspend failed – permissions (not initiator or superuser)
• PathNotFoundException (workflow/payload) – payload deleted, or launcher config path missing
• RejectedExecutionException – thread pool full with blockPolicy=ABORT; timeout jobs dropped
• retrys exceeded - remove isTransient – transient workflow failed after cq.workflow.job.retry exhausted

---

External docs

• Working with Workflows (Cloud Service)
• Workflow API (AEMaaCS Javadoc)
• Configuring OSGi for AEM Cloud Service
• AEMaaCS repoinit (Deploying overview → Repoinit)