Assemblyai Transcribe

AssemblyAI transcription, Speech Understanding, and agent-friendly exports

Use this skill when the user wants AssemblyAI rather than generic transcription, or when the job benefits from AssemblyAI-specific capabilities such as:

• model routing across universal-3-pro and universal-2
• language detection and code switching
• diarisation plus speaker name / role mapping
• translation, custom formatting, or AssemblyAI speaker identification
• subtitles, paragraphs, sentences, topic / entity / sentiment tasks
• transcript output that is easy for other agents to consume as Markdown or normalised JSON

The skill is designed for AI agents like OpenClaw, not just end users. It provides:

1. A no-dependency Node CLI in scripts/assemblyai.mjs (and a compatibility wrapper at assemblyai.mjs) 2. Bundled model/language knowledge via models and languages commands 3. Stable transcript output formats

• agent-friendly Markdown
• normalised agent JSON
• bundle manifests for downstream automation

4. Speaker mapping workflows

• manual speaker/channel maps
• AssemblyAI speaker identification
• merged display names in both Markdown and JSON

5. AssemblyAI LLM Gateway integration for structured extraction from transcripts

Use this skill in this order

1) Decide whether the user needs AssemblyAI-specific behaviour

If they just want “a transcript”, a generic solution may be enough. Reach for this skill when the user mentions AssemblyAI, wants a specific AssemblyAI feature, or needs the richer outputs and post-processing this skill provides.

2) Pick the best entry point

• New transcription → transcribe
• Existing transcript id → get or wait
• Re-render existing saved JSON → format
• Post-process an existing transcript → understand
• Run transcript text through LLM Gateway → llm
• Need a quick capability lookup before deciding → models or languages

3) Prefer the agent-friendly defaults

For most unknown-language or mixed-language jobs, prefer:

node {baseDir}/assemblyai.mjs transcribe INPUT   --bundle-dir ./assemblyai-out   --all-exports

Why:

• the CLI defaults to auto-best routing when models are not specified
• it writes a manifest + multiple files that agents can inspect without reparsing terminal output
• Markdown and agent JSON become available immediately for follow-on steps

Quick-start recipes

Best general default

Use this when the source language is unknown or could be outside the 6-language Universal-3-Pro set:

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --bundle-dir ./out   --all-exports

This defaults to model routing plus language detection unless the request already specifies a model or language.

Best known-language accuracy

If the language is known and supported by Universal-3-Pro, prefer an explicit request:

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --speech-model universal-3-pro   --language-code en_us   --bundle-dir ./out

Meeting / interview with speaker labels

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --speaker-labels   --bundle-dir ./out

Add explicit speaker names or roles

Manual mapping:

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --speaker-labels   --speaker-map @assets/speaker-map.example.json   --bundle-dir ./out

AssemblyAI speaker identification:

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --speaker-labels   --speaker-type role   --known-speakers "host,guest"   --bundle-dir ./out

Or post-process an existing transcript:

node {baseDir}/assemblyai.mjs understand TRANSCRIPT_ID   --speaker-type name   --speaker-profiles @assets/speaker-profiles-name.example.json   --bundle-dir ./out

Translation

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --translate-to de,fr   --match-original-utterance   --bundle-dir ./out

Structured extraction through LLM Gateway

node {baseDir}/assemblyai.mjs llm TRANSCRIPT_ID   --prompt @assets/example-prompt.txt   --schema @assets/llm-json-schema.example.json   --out ./summary.json

Command guidance

transcribe

Use for local files or remote URLs.

• Local files are uploaded first.
• Public URLs are sent directly to AssemblyAI.
• Waits by default, then renders output.

Prefer --bundle-dir for anything longer than a trivial clip.

get / wait

Use when you already have the transcript id. wait blocks until completion; get fetches immediately unless you add --wait.

format

Use when you already saved:

• raw transcript JSON from AssemblyAI, or
• the normalised agent JSON produced by this skill

This is useful when you want to apply a new speaker map, re-render Markdown, or generate a fresh bundle without retranscribing.

understand

Use when you need AssemblyAI Speech Understanding on an existing transcript:

• translation
• speaker identification
• custom formatting

This command fetches the transcript, merges in the returned understanding results, then renders updated Markdown / agent JSON / bundle outputs.

llm

Use when the user wants:

• summaries
• extraction
• structured JSON
• downstream reasoning over the transcript

Prefer --schema when the next step is automated.

Output strategy

Best default for agents: bundle mode

--bundle-dir writes a directory containing:

• Markdown transcript
• agent JSON
• raw JSON
• optional paragraphs / sentences / subtitles
• a machine-readable manifest

This is usually better than dumping everything to stdout.

Primary output kinds

Use --export to choose the main output:

• markdown (default)
• agent-json
• json / raw-json
• text
• paragraphs
• sentences
• srt
• vtt
• manifest

Sidecar outputs

You can request extra files directly with:

• --markdown-out
• --agent-json-out
• --raw-json-out
• --paragraphs-out
• --sentences-out
• --srt-out
• --vtt-out
• --understanding-json-out

Speaker mapping rules

Speaker display names are merged in this order:

1. manual --speaker-map 2. AssemblyAI speaker identification mapping 3. fallback generic names like Speaker A or Channel 1

This means you can let AssemblyAI identify speakers first, then still override individual display names later.

Example manual map file: assets/speaker-map.example.json

Model and language lookup

Before choosing parameters, inspect the bundled reference data:

node {baseDir}/assemblyai.mjs models
node {baseDir}/assemblyai.mjs models --format json
node {baseDir}/assemblyai.mjs languages --model universal-3-pro
node {baseDir}/assemblyai.mjs languages --model universal-2 --codes --format json

The bundled data lives in:

• assets/model-capabilities.json
• assets/language-codes.json

Important operating notes

• Keep API keys out of chat logs; use environment injection.
• Use the EU AssemblyAI base URL when the user explicitly needs EU processing.
• Uploads and transcript creation must use API keys from the same AssemblyAI project.
• Prefer --bundle-dir or --out for long outputs.
• The CLI is non-interactive and sends diagnostics to stderr, which makes it easier for agents to script reliably.
• Use raw --config or --request when you need a newly added AssemblyAI parameter that this skill has not exposed yet.

Reference files

Read these when you need more depth:

• Capabilities
• Workflows and recipes
• Output formats
• Speaker mapping
• LLM Gateway notes
• Troubleshooting

Key bundled files

• assemblyai.mjs — root wrapper for compatibility with the original skill
• scripts/assemblyai.mjs — main CLI
• assets/speaker-map.example.json
• assets/speaker-profiles-name.example.json
• assets/speaker-profiles-role.example.json
• assets/custom-spelling.example.json
• assets/llm-json-schema.example.json
• assets/transcript-agent-json-schema.json

Sanity checks before finishing a task

• Did you pick the right region (api.assemblyai.com vs api.eu.assemblyai.com)?
• Did you choose a model strategy that matches the language situation?
• If speaker naming matters, did you enable diarisation and/or provide a speaker map?
• If the result will feed another agent, did you produce Markdown and/or agent JSON rather than only raw stdout?
• If the transcript will be machine-consumed, did you keep the manifest or explicit output filenames?

AssemblyAI capabilities reference for this skill

This file is the human-readable summary behind the models and languages commands.

Core STT endpoints

• US base URL: https://api.assemblyai.com
• EU base URL: https://api.eu.assemblyai.com

For LLM Gateway:

• US: https://llm-gateway.assemblyai.com
• EU: https://llm-gateway.eu.assemblyai.com

Recommended speech model strategy

Default recommendation

Use:

{
  "speech_models": ["universal-3-pro", "universal-2"],
  "language_detection": true
}

Reason:

• Universal-3-Pro is preferred where supported
• Universal-2 acts as the broad fallback
• this avoids silently defaulting to a single language when the source language is unknown

When to force Universal-3-Pro

Use explicit Universal-3-Pro when:

• the language is known to be one of its supported families
• you want prompting support
• you want the larger keyterms limit

When to lean on Universal-2

Use Universal-2 explicitly when:

• the language is outside Universal-3-Pro coverage
• you want the simplest “wide coverage” choice
• you do not need Universal-3-Pro-only prompting behaviour

Universal-3-Pro

Supported language families and dialect notes:

• English: American, British, Australian
• Spanish: Castilian, Mexican, Argentine, Colombian, Chilean, Caribbean, Spanglish
• French: Metropolitan, Quebecois, Belgian
• German
• Italian: Standard
• Portuguese: Brazilian, European

Key capabilities called out in AssemblyAI docs:

• highest-accuracy positioning
• prompting support
• keyterms prompt up to 1,000 entries
• built-in code switching across supported languages

Universal-2

AssemblyAI documents 99-language coverage, grouped by accuracy band.

High accuracy

English, Spanish, French, German, Indonesian, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Turkish, Ukrainian, Catalan

Good accuracy

Arabic, Azerbaijani, Bulgarian, Bosnian, Mandarin Chinese, Czech, Danish, Greek, Estonian, Finnish, Galician, Hindi, Croatian, Hungarian, Korean, Macedonian, Malay, Norwegian, Romanian, Slovak, Swedish, Swiss German, Tagalog, Thai, Urdu, Vietnamese

Moderate accuracy

Afrikaans, Belarusian, Welsh, Persian / Farsi, Hebrew, Armenian, Icelandic, Kazakh, Lithuanian, Latvian, Maori, Marathi, Slovenian, Swahili, Tamil

Fair accuracy

Amharic, Assamese, Bengali, Gujarati, Hausa, Javanese, Georgian, Khmer, Kannada, Luxembourgish, Lingala, Lao, Malayalam, Mongolian, Maltese, Burmese, Nepali, Occitan, Punjabi, Pashto, Sindhi, Shona, Somali, Serbian, Telugu, Tajik, Uzbek, Yoruba

Feature notes

Speaker diarisation

Enable speaker_labels=true when you need generic speaker segments such as A, B, etc.

Speaker identification

Use AssemblyAI Speech Understanding when you want those generic labels mapped to real names or roles.

Custom spelling

This is a strong fit for product names, company names, uncommon surnames, technical terms, and branded spellings. Example asset: assets/custom-spelling.example.json

Translation

Use Speech Understanding translation when you need transcript-level or utterance-aligned translations.

Custom formatting

Use Speech Understanding custom formatting when dates, phone numbers, and emails need normalised presentation.

Topics, key phrases, entities, sentiment

The CLI exposes common flags:

• --iab-categories
• --auto-highlights
• --entity-detection
• --sentiment-analysis

These sections are also surfaced in the Markdown and agent JSON outputs when present.

Export endpoints

The skill supports:

• transcript JSON
• agent-friendly Markdown
• normalised agent JSON
• text
• paragraphs
• sentences
• subtitles (srt, vtt)

Important deprecations

Summarisation fields on the transcription API

The old summarisation parameters on the transcription API are deprecated. Use the LLM Gateway for summaries and structured extraction.

LeMUR

LeMUR is deprecated in favour of the LLM Gateway. The CLI therefore includes an llm command rather than a LeMUR workflow.

Bundled machine-readable references

• assets/model-capabilities.json
• assets/language-codes.json

LLM Gateway notes

Use the llm command when the user wants more than transcription:

• summaries
• action items
• structured JSON
• downstream reasoning
• extraction from a transcript that already exists

Why this skill includes an llm command

AssemblyAI's older transcription summarisation fields are deprecated, and LeMUR is deprecated in favour of the LLM Gateway.

This skill therefore prepares transcript text and sends it to:

• https://llm-gateway.assemblyai.com/v1/chat/completions
• or the EU equivalent when configured

Speaker-aware transcript formatting

The LLM Gateway only sees the text you send it.

For transcripts with speakers, this skill can convert the transcript into lines like:

[00:00:00.000] Host: Welcome everyone
[00:00:04.200] Guest: Thanks for having me

That is the default --input-format speaker-aware.

Other options:

• --input-format plain
• --input-format markdown

Structured outputs

When the next step is automated, prefer --schema.

Example:

node {baseDir}/assemblyai.mjs llm TRANSCRIPT_ID   --prompt @assets/example-prompt.txt   --schema @assets/llm-json-schema.example.json   --out ./summary.json

This is usually better than asking for free-form text and trying to parse it later.

Raw request override

If you need a newer or more specialised LLM Gateway parameter, use:

node {baseDir}/assemblyai.mjs llm TRANSCRIPT_ID   --request @my-chat-body.json

The CLI then sends your request body as-is.

Good default pattern for agents

1. generate transcript bundle 2. inspect agent JSON or Markdown 3. call llm with a schema if the next step is automated 4. keep the raw LLM response as a sidecar when debugging

Output formats

This skill deliberately exposes multiple output shapes because different follow-on tasks benefit from different formats.

1. Markdown transcript (--export markdown)

Best for:

• another LLM or agent reading the transcript
• a human reviewing the result
• preserving speaker names and timestamps in a compact way

Structure:

• # AssemblyAI Transcript
• ## Metadata
• ## Speaker Map (when relevant)
• ## Transcript
• optional sections for chapters, key phrases, topics, entities, sentiment, translations, custom formatting, and word-level timestamps

Design goals:

• stable headings
• compact metadata
• speaker-aware transcript lines
• easy skimming
• easy chunking by downstream agents

2. Agent JSON (--export agent-json)

Best for:

• scripts and deterministic downstream processing
• tools that need stable key names
• preserving merged speaker display names without re-implementing mapping logic

Top-level shape:

• schema_version
• generated_at
• source
• transcript
• features
• speaker_map
• utterances
• words (only when requested)
• chapters
• highlights
• topics
• entities
• sentiment
• translated_texts
• speech_understanding

Schema asset:

• assets/transcript-agent-json-schema.json

3. Raw JSON (--export json)

Best for:

• debugging exact AssemblyAI responses
• validating new API fields
• preserving the original response for later re-rendering

4. Bundle mode (--bundle-dir)

Best for:

• AI-agent pipelines
• multi-step workflows
• keeping several exports together without inventing filenames later

Bundle contents typically include:

• *.transcript.md
• *.agent.json
• *.raw.json
• *.manifest.json

With --all-exports, the bundle also includes:

• *.paragraphs.txt
• *.sentences.txt
• *.srt
• *.vtt

The manifest is especially useful because another agent can read a single JSON file and discover the rest.

Speaker map precedence

When rendering Markdown or agent JSON, this skill resolves speaker display names in the following order:

1. manual speaker map (--speaker-map) 2. AssemblyAI speaker identification mapping 3. fallback label generated from the raw speaker/channel token

That means:

• AssemblyAI can generate an initial mapping
• you can later override just one or two speakers without redoing everything

Why bundle mode is the default recommendation

Agents often do better when:

• the main transcript is in Markdown
• the canonical machine-readable copy is separate
• filenames are explicit
• a manifest tells them where everything lives

That is why the recommended default workflow is:

node {baseDir}/assemblyai.mjs transcribe INPUT --bundle-dir ./out --all-exports

Speaker mapping reference

This skill treats speaker naming as a first-class workflow rather than a cosmetic afterthought.

Three ways to get useful speaker names

1. Generic diarisation only

Enable speaker diarisation during transcription:

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3 --speaker-labels

This produces generic labels such as A, B, or Speaker A.

Use this when:

• you only need separation, not identity
• you are not sure who the speakers are yet
• you want to name them later

2. Manual speaker mapping

Provide your own display names or channel names:

node {baseDir}/assemblyai.mjs format ./transcript.json   --speaker-map @assets/speaker-map.example.json

Accepted shapes:

JSON object

{
  "A": "Host",
  "B": "Guest",
  "channel:0": "Agent",
  "channel:1": "Customer"
}

Rich JSON object

{
  "A": { "display": "Host", "source": "manual" },
  "B": { "display": "Guest", "source": "manual" }
}

Simple text / CSV-style

A=Host
B=Guest
channel:0=Agent
channel:1=Customer

3. AssemblyAI speaker identification

Use Speech Understanding to map diarised speakers to names or roles.

Known values

node {baseDir}/assemblyai.mjs understand TRANSCRIPT_ID   --speaker-type role   --known-speakers "agent,customer"

Rich speaker profiles

node {baseDir}/assemblyai.mjs understand TRANSCRIPT_ID   --speaker-type name   --speaker-profiles @assets/speaker-profiles-name.example.json

Use known values when you only know the final labels. Use rich profiles when descriptions help disambiguate similar speakers.

Precedence

When rendering outputs, the skill applies speaker names in this order:

1. manual --speaker-map 2. AssemblyAI speaker identification mapping 3. fallback generic label

That means you can:

• let AssemblyAI do the first pass
• override only the mistakes manually
• keep stable speaker names across rerenders

Channel vs diarisation

Prefer --multichannel when:

• the recording is a call with clean left/right separation
• each channel already corresponds to one participant

Prefer --speaker-labels when:

• the recording is a single mixed channel
• the speakers overlap or change within one channel
• you want speaker turns rather than channel IDs

You can still use a manual channel:0 / channel:1 map when multichannel is the right tool.

Why this matters for downstream agents

A plain transcript often loses who said what. This skill preserves speaker-aware structure in both:

• Markdown
• agent JSON

That makes later summarisation, action-item extraction, and role-specific analysis much more reliable.

Troubleshooting

Missing API key

Symptom:

• Missing API key. Set ASSEMBLYAI_API_KEY or pass --api-key.

Fix:

• export ASSEMBLYAI_API_KEY
• or inject it through the agent environment

403 Cannot access uploaded file

Most common cause:

• the upload was created with one AssemblyAI project/API key
• the transcript request used a different project/API key

Fix:

• make sure upload and transcription use the same AssemblyAI project credentials

Transcript status becomes error

Common causes documented by AssemblyAI include:

• corrupted or unsupported media
• a URL that points to an HTML page rather than media
• a URL not reachable from AssemblyAI servers
• very short audio (around sub-160ms)

Useful checks:

• verify the file opens locally
• if using a URL, try downloading it manually
• prefer uploading a local file when the remote URL is uncertain

Language is wrong

Try one of these:

• explicit --language-code
• --expected-languages
• explicit --speech-model universal-3-pro for supported languages
• --speech-model universal-2 for broader fallback coverage

Speaker names are still generic

Checklist:

• did you enable --speaker-labels when transcribing?
• if post-processing, does the transcript actually contain diarised utterances?
• did you pass --known-speakers or --speaker-profiles in the right shape?
• if the names still need tweaking, add --speaker-map

Output is too large for stdout

Prefer:

• --out FILE
• --bundle-dir DIR

This is especially important for long meetings and word-level timestamps.

Need a newly added API parameter

This skill deliberately exposes raw passthrough options:

• --config for transcription requests
• --understanding-request for speech understanding
• --request for LLM Gateway chat completions

Use those rather than editing the script immediately.

EU routing

If the user wants EU processing:

export ASSEMBLYAI_BASE_URL=https://api.eu.assemblyai.com
export ASSEMBLYAI_LLM_BASE_URL=https://llm-gateway.eu.assemblyai.com

Or pass them explicitly with --base-url and --llm-base-url.

Rate limits / retries

The CLI retries common transient failures such as:

• 429
• 500
• 502
• 503
• 504

If the issue persists, slow down the workflow or inspect the exact request body with --dry-run.

Workflow recipes

These recipes are intended for agent use.

1. Unknown language, best general result

node {baseDir}/assemblyai.mjs transcribe INPUT   --bundle-dir ./out   --all-exports

What this gives you:

• model routing
• language detection
• Markdown
• agent JSON
• raw JSON
• manifest
• paragraphs / sentences / subtitles

Use this as the safest “do the right thing” default.

2. Known English meeting, highest quality

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --speech-model universal-3-pro   --language-code en_us   --speaker-labels   --bundle-dir ./out

3. Product demo with custom terminology

node {baseDir}/assemblyai.mjs transcribe ./demo.mp4   --custom-spelling @assets/custom-spelling.example.json   --keyterms "OpenClaw,AssemblyAI,Universal-3-Pro"   --bundle-dir ./out

Notes:

• use --keyterms when you want the model to pay attention to exact terms
• use --custom-spelling when you want specific renderings in the final text

4. Call-centre stereo recording

node {baseDir}/assemblyai.mjs transcribe ./call.wav   --multichannel   --speaker-map @assets/speaker-map.example.json   --bundle-dir ./out

Use this when left/right channels matter more than diarisation.

5. Named speakers from generic diarisation

First transcribe with speaker labels:

node {baseDir}/assemblyai.mjs transcribe ./interview.mp3   --speaker-labels   --bundle-dir ./out

Then identify the speakers:

node {baseDir}/assemblyai.mjs understand TRANSCRIPT_ID   --speaker-type name   --speaker-profiles @assets/speaker-profiles-name.example.json   --bundle-dir ./out

6. Role-labelled speakers for support or sales calls

node {baseDir}/assemblyai.mjs understand TRANSCRIPT_ID   --speaker-type role   --known-speakers "agent,customer"   --bundle-dir ./out

Good role pairs include:

• agent / customer
• interviewer / candidate
• host / guest
• doctor / patient
• salesperson / buyer

7. Translation with aligned utterances

node {baseDir}/assemblyai.mjs transcribe ./meeting.mp3   --translate-to de,fr   --match-original-utterance   --bundle-dir ./out

This is especially useful when a downstream agent needs:

• speaker separation
• timestamps
• per-utterance translations

8. Structured extraction through LLM Gateway

node {baseDir}/assemblyai.mjs llm TRANSCRIPT_ID   --prompt @assets/example-prompt.txt   --schema @assets/llm-json-schema.example.json   --out ./summary.json

Use this when the next consumer is a script, workflow engine, or another agent.

9. Re-render a saved transcript

If you already stored raw or agent JSON:

node {baseDir}/assemblyai.mjs format ./out/myfile.agent.json   --speaker-map @assets/speaker-map.example.json   --bundle-dir ./rerendered

10. Minimal stdout-only flow

For tiny clips or quick inspection:

node {baseDir}/assemblyai.mjs transcribe ./clip.wav --export markdown

For anything non-trivial, prefer --bundle-dir or --out.