React Native Executorch

React Native ExecuTorch

Software Mansion's production patterns for on-device AI in React Native and Expo using React Native ExecuTorch.

Targets the current published API (v0.10.x). Load at most one reference file per question. For hook signatures, model constants, or config options not covered here, webfetch the matching page from docs.swmansion.com/react-native-executorch.

Decision Tree

What does the feature need?
│
├── Generate / chat with text?
│   └── useLLM                                          → see llm.md
│       ├── Plain chat → standard useLLM
│       ├── Image + text input → useLLM with a VLM model (LFM2_VL_*)
│       ├── Tool / function calling → configure with toolsConfig
│       └── Structured JSON output → getStructuredOutputPrompt
│
├── Understand or transform images?
│   ├── What is in this image? → useClassification      → see vision.md
│   ├── Where are the objects? → useObjectDetection     → see vision.md
│   ├── Per-pixel class → useSemanticSegmentation       → see vision.md
│   ├── Per-instance segmentation → useInstanceSegmentation → see vision.md
│   ├── Human pose keypoints → usePoseEstimation        → see vision.md
│   ├── Read text from image → useOCR / useVerticalOCR  → see vision.md
│   ├── Apply artistic style → useStyleTransfer         → see vision.md
│   ├── Generate image from prompt → useTextToImage     → see vision.md
│   └── Embed image as vector → useImageEmbeddings      → see vision.md
│
├── Speech / audio?
│   ├── Transcribe speech → useSpeechToText             → see speech.md
│   ├── Synthesize speech → useTextToSpeech             → see speech.md
│   └── Detect speech segments → useVAD                 → see speech.md
│
├── Text utilities?
│   ├── Embed text as vector → useTextEmbeddings        → see vision.md
│   ├── Count or inspect tokens → useTokenizer          → see setup.md
│   └── Redact PII from text → usePrivacyFilter         → see setup.md
│
├── Full RAG pipeline (retrieval + generation + vector store)?
│   └── react-native-rag (sibling library)              → see setup.md
│
└── Custom `.pte` model not covered by a dedicated hook?
    └── useExecutorchModule                             → see setup.md

Critical Rules

• Call `initExecutorch()` at app entry, before any other API. The library does not bundle a network/file layer — you must register a resource-fetcher adapter (ExpoResourceFetcher for Expo, BareResourceFetcher for bare RN). Any hook called before initialization throws ResourceFetcherAdapterNotInitialized.

• Check `isReady` before calling `forward` / `generate` / `transcribe`. All hooks load asynchronously. Inference before the model is ready throws ModuleNotLoaded.

• Interrupt LLM generation before unmounting. Unmounting while isGenerating is true crashes. Call llm.interrupt() and wait for isGenerating === false before navigating away.

• Use quantized model variants on mobile. Full-precision variants exceed device memory on most phones. Every supported model ships a _QUANTIZED variant — prefer it unless you've measured otherwise.

• Audio for speech-to-text and VAD must be 16 kHz mono. Mismatched sample rates produce silently garbled transcriptions. Decode with new AudioContext({ sampleRate: 16000 }).

• Audio from text-to-speech is 24 kHz. Create the playback context with new AudioContext({ sampleRate: 24000 }).

• The New Architecture (Fabric) is required. Old architecture is unsupported. Expo Go is unsupported — use a custom dev build (npx expo prebuild). iOS release builds need a real device (the simulator lacks the Metal APIs ExecuTorch relies on).

Minimal Setup

// App.tsx (Expo)
import { initExecutorch } from 'react-native-executorch';
import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher';

initExecutorch({ resourceFetcher: ExpoResourceFetcher });

// App.tsx (bare React Native)
import { initExecutorch } from 'react-native-executorch';
import { BareResourceFetcher } from 'react-native-executorch-bare-resource-fetcher';

initExecutorch({ resourceFetcher: BareResourceFetcher });

Full setup, Metro config for bundled .pte files, custom adapters, model-loading strategies, and error handling: see setup.md.

Hook Quick Reference

Every hook also has a non-React Module counterpart (e.g. LLMModule.fromModelName(...), ClassificationModule.fromModelName(...)) for use outside React components.

Common Pitfalls

Full error code list and recovery patterns: setup.md.

References

External Resources

• Official docs: https://docs.swmansion.com/react-native-executorch
• API reference: https://docs.swmansion.com/react-native-executorch/docs/api-reference
• Source: https://github.com/software-mansion/react-native-executorch
• Pre-exported models: https://huggingface.co/software-mansion

LLMs

Run Large Language Models on-device for text generation, chat, tool / function calling, structured JSON output, and vision-language understanding.

Pick models via the typed models registry: models.llm.<model>({ quant?, backend? }). Calling with no args returns the platform default (quantized variant when one is published). Passing a backend a model doesn't ship is a compile-time error. For full API surface and config options, webfetch useLLM API reference.

---

Functional mode (stateless)

import { useLLM, models, Message } from 'react-native-executorch';

const llm = useLLM({ model: models.llm.lfm2_5_1_2b_instruct() });

const onGenerate = async () => {
  const chat: Message[] = [
    { role: 'system', content: 'You are a helpful assistant' },
    { role: 'user', content: 'What is the meaning of life?' },
  ];
  const response = await llm.generate(chat);
  console.log(response);
};

return (
  <View>
    <Button onPress={onGenerate} title="Generate" disabled={!llm.isReady} />
    <Text>{llm.response}</Text>
  </View>
);

Each generate() call is independent — the hook does not keep history.

---

Managed mode (stateful chat)

For multi-turn chats, configure once and let the hook manage messageHistory:

import { useEffect } from 'react';
import { useLLM, models, MessageCountContextStrategy } from 'react-native-executorch';

const llm = useLLM({ model: models.llm.lfm2_5_1_2b_instruct() });

useEffect(() => {
  llm.configure({
    chatConfig: {
      systemPrompt: 'You are a helpful assistant',
      contextStrategy: new MessageCountContextStrategy(6), // keep last 6 messages
    },
    generationConfig: {
      temperature: 0.7,
      topp: 0.9,
      outputTokenBatchSize: 15,
      batchTimeInterval: 100,
    },
  });
}, []);

// Send messages — appended to llm.messageHistory automatically
llm.sendMessage('Hello!');

Render the chat from llm.messageHistory. llm.response streams the in-progress assistant message; llm.isGenerating is true during generation.

---

Interrupting

Always interrupt before unmounting — an in-flight generation will crash the app on tear-down.

{llm.isGenerating && <Button onPress={llm.interrupt} title="Stop" />}

useEffect(() => {
  return () => {
    if (llm.isGenerating) llm.interrupt();
  };
}, []);

If you're navigating away programmatically, await until isGenerating becomes false before unmounting.

---

Tool / function calling

Define tools with a name, description, and JSON-schema-like parameter spec. Implement executeToolCallback to actually run the tool. Models with strong tool-calling support: models.llm.hammer2_1_*, models.llm.qwen3_*.

import { useEffect } from 'react';
import {
  useLLM,
  models,
  DEFAULT_SYSTEM_PROMPT,
  LLMTool,
  ToolCall,
} from 'react-native-executorch';

const TOOLS: LLMTool[] = [
  {
    name: 'get_weather',
    description: 'Get current weather in a given location.',
    parameters: {
      type: 'dict',
      properties: {
        location: { type: 'string', description: 'Location to check weather for' },
      },
      required: ['location'],
    },
  },
];

const executeTool = async (call: ToolCall): Promise<string | null> => {
  switch (call.toolName) {
    case 'get_weather':
      return 'It is sunny and 21°C.';
    default:
      return null;
  }
};

const llm = useLLM({ model: models.llm.hammer2_1_1_5b() });

useEffect(() => {
  llm.configure({
    chatConfig: {
      systemPrompt: `${DEFAULT_SYSTEM_PROMPT} Current time: ${new Date().toString()}`,
    },
    toolsConfig: {
      tools: TOOLS,
      executeToolCallback: executeTool,
      displayToolCalls: true,
    },
  });
}, []);

In functional mode, pass tools as the second argument: llm.generate(chat, TOOLS).

---

Structured JSON output

Use getStructuredOutputPrompt to build a system prompt from a JSON Schema or Zod schema, then validate the response with fixAndValidateStructuredOutput.

import * as z from 'zod/v4';
import {
  useLLM,
  models,
  getStructuredOutputPrompt,
  fixAndValidateStructuredOutput,
} from 'react-native-executorch';

const schema = z.object({
  username: z.string().meta({ description: 'User asking the question' }),
  bid: z.number().meta({ description: 'Offer in the user message' }),
  currency: z.optional(z.string()),
});

const llm = useLLM({ model: models.llm.qwen3_4b() });

useEffect(() => {
  const instructions = getStructuredOutputPrompt(schema);
  llm.configure({
    chatConfig: {
      systemPrompt:
        `Parse the user's message and return JSON. Don't reply to the user. ${instructions} /no_think`,
    },
  });
}, []);

useEffect(() => {
  const last = llm.messageHistory.at(-1);
  if (!llm.isGenerating && last?.role === 'assistant') {
    try {
      const parsed = fixAndValidateStructuredOutput(last.content, schema);
      console.log(parsed); // typed by Zod
    } catch (e) {
      console.warn('Output did not match schema', e);
    }
  }
}, [llm.messageHistory, llm.isGenerating]);

getStructuredOutputPrompt accepts both JSON Schema (jsonschema) and Zod schemas.

---

Vision-Language Models (VLM)

Some LLMs accept image + text input. They live under models.llm too — pick them by capability. Pass an image alongside text via imagePath (managed) or mediaPath on a Message (functional).

import { useLLM, models, Message } from 'react-native-executorch';

const llm = useLLM({ model: models.llm.lfm2_5_vl_1_6b() });

// Managed
llm.sendMessage('What is in this image?', { imagePath: '/path/to/image.jpg' });

// Functional
const chat: Message[] = [
  { role: 'user', content: 'Describe this image.', mediaPath: '/path/to/image.jpg' },
];
await llm.generate(chat);

imagePath / mediaPath must be a local filesystem path. To use a remote image, download it first (e.g. via the resource-fetcher adapter — see setup.md).

---

Choosing options on the accessor

// Platform default (quantized when published).
models.llm.llama3_2_3b();

// Non-quantized variant.
models.llm.llama3_2_3b({ quant: false });

// Explicit backend — only the backends the model actually ships are accepted.
models.llm.qwen3_4b({ backend: 'xnnpack' });

---

Model selection

Full per-device benchmarks: webfetch Inference time benchmarks.

---

Troubleshooting

• Crash on unmount. Active generation was not interrupted — call llm.interrupt() and wait for isGenerating === false.
• Out-of-memory at load. Pick a smaller accessor or stay on the default (quantized) variant.
• Poor output quality. Try a larger model, raise temperature / topp, or improve the system prompt. For Qwen 3 reasoning models, pass /no_think in the prompt to skip the thinking phase.
• Tool calls never fire. Use a tool-tuned model (models.llm.hammer2_1_*) and ensure executeToolCallback is set.

---

See also

• useLLM API reference
• LLMModule (non-hook) API
• LLM HuggingFace collection

Setup, Model Loading & Utilities

Installation, initialization, model loading strategies, error handling, custom .pte models, tokenization, and PII redaction.

For the full getting-started guide, webfetch Getting Started. For version compatibility, webfetch Compatibility.

---

Installation

npm install react-native-executorch

# Expo projects
npm install react-native-executorch-expo-resource-fetcher

# Bare React Native projects
npm install react-native-executorch-bare-resource-fetcher

Prerequisites

• New Architecture (Fabric) required. Old architecture is unsupported.
• Expo Go is not supported. Use a custom dev build (npx expo prebuild).
• iOS release builds require a real device. The simulator lacks Metal APIs ExecuTorch relies on.

---

Initialization

Register a resource-fetcher adapter at app entry before any other API:

// Expo
import { initExecutorch } from 'react-native-executorch';
import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher';

initExecutorch({ resourceFetcher: ExpoResourceFetcher });

// Bare React Native
import { initExecutorch } from 'react-native-executorch';
import { BareResourceFetcher } from 'react-native-executorch-bare-resource-fetcher';

initExecutorch({ resourceFetcher: BareResourceFetcher });

Calling any hook or module before initExecutorch throws ResourceFetcherAdapterNotInitialized.

If neither adapter fits (custom CDN, private auth, custom caching), implement the ResourceFetcherAdapter interface yourself — webfetch Custom Adapter.

For teardown (e.g. in tests), call cleanupExecutorch().

---

Metro config (bundled .pte models)

To require('../assets/model.pte'), register the extensions:

// metro.config.js
const { getDefaultConfig } = require('expo/metro-config');

const config = getDefaultConfig(__dirname);
config.resolver.assetExts.push('pte');
config.resolver.assetExts.push('bin');

module.exports = config;

---

Model loading strategies

Every hook accepts a model prop. Pick the strategy by model size and UX requirements.

How large is the model?
├── A published Software Mansion model?
│   └── Use the `models` registry accessor (recommended)
│       useLLM({ model: models.llm.llama3_2_1b() })
│
├── Small (< 512 MB) and must work offline from launch?
│   └── Bundle as an asset
│       useLLM({ model: { modelSource: require('../assets/model.pte'), … } })
│
├── Large (> 512 MB) or optional feature?
│   └── Remote URL (downloaded + cached on first use)
│       useLLM({ model: { modelSource: 'https://…/model.pte', … } })
│
└── User-managed / fine-tuned models?
    └── Local file path
        useLLM({ model: { modelSource: 'file:///var/mobile/…/model.pte', … } })

models registry (recommended)

models.<category>.<model>({ quant?, backend? }) — typed accessors that resolve to the right URL and backend per platform. Default is the quantized variant when one is published; iOS prefers CoreML, Android prefers XNNPACK, for multi-backend models.

import { useLLM, useObjectDetection, useOCR, models } from 'react-native-executorch';

useLLM({ model: models.llm.llama3_2_3b() });                 // platform default, quantized
useLLM({ model: models.llm.llama3_2_3b({ quant: false }) }); // full precision
useObjectDetection({ model: models.object_detection.rf_detr_nano({ backend: 'xnnpack' }) });
useOCR({ model: models.ocr.craft({ language: 'en' }) });

Available top-level categories: llm, classification, object_detection, pose_estimation, semantic_segmentation, instance_segmentation, style_transfer, speech_to_text, text_to_speech, text_embedding, image_embedding, image_generation, vad, ocr, privacy_filter. Per-category accessors are listed in llm.md, vision.md, speech.md, and the privacy filter section below.

preventLoad

Every hook accepts preventLoad: true to defer model download/load until you're ready:

const llm = useLLM({ model: models.llm.llama3_2_1b(), preventLoad: true });
// Flip to false (or omit) to start loading

Download progress

Hooks expose downloadProgress (0–1):

const llm = useLLM({ model: models.llm.llama3_2_1b() });
<Text>{Math.round(llm.downloadProgress * 100)}%</Text>

---

Resource fetcher

For advanced download management — pause, resume, cancel, list, delete — the adapters expose a Promise-based API. For the full surface, webfetch ResourceFetcher.

import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher';

// Download with progress
const uris = await ExpoResourceFetcher.fetch(
  (p) => console.log(`${Math.round(p * 100)}%`),
  'https://example.com/model.pte',
  'https://example.com/tokenizer.bin'
);
// uris: string[] of local file paths (no `file://` prefix), or null if interrupted

await ExpoResourceFetcher.pauseFetching('https://…/model.pte');
const uris = await ExpoResourceFetcher.resumeFetching('https://…/model.pte');
await ExpoResourceFetcher.cancelFetching('https://…/model.pte');

const files = await ExpoResourceFetcher.listDownloadedFiles();
const models = await ExpoResourceFetcher.listDownloadedModels();
const bytes = await ExpoResourceFetcher.getFilesTotalSize('https://…/model.pte');
await ExpoResourceFetcher.deleteResources('https://…/model.pte');

BareResourceFetcher exposes the same API but does not support pause/resume on Android — use Expo's if you need it cross-platform.

Downloaded files are stored in the app's documents directory.

---

Error handling

All errors inherit from RnExecutorchError with a code from RnExecutorchErrorCode. For the full table, webfetch Error Handling.

Pattern

import { RnExecutorchError, RnExecutorchErrorCode } from 'react-native-executorch';

try {
  await model.forward(imageUri);
} catch (err) {
  if (err instanceof RnExecutorchError) {
    switch (err.code) {
      case RnExecutorchErrorCode.ModuleNotLoaded:
        // Model still loading — show loading state
        break;
      case RnExecutorchErrorCode.ModelGenerating:
        // Already busy — wait or interrupt
        break;
      case RnExecutorchErrorCode.MemoryAllocationFailed:
        // Device can't fit the model — fall back to a smaller one
        break;
      default:
        console.error('ExecuTorch error:', err.code, err.message);
    }
  } else {
    throw err;
  }
}

---

Custom models — useExecutorchModule

For .pte models not covered by a dedicated hook, use useExecutorchModule to run arbitrary tensor I/O.

Exporting

1. Export your PyTorch model to .pte using the ExecuTorch export tutorial. 2. Pick a backend: XNNPACK (CPU, cross-platform) or Core ML (iOS, uses ANE). 3. Load it via asset, URL, or local path.

Running

import { useExecutorchModule, ScalarType } from 'react-native-executorch';

const m = useExecutorchModule({
  modelSource: require('../assets/custom_model.pte'),
});

const run = async () => {
  const input = {
    dataPtr: new Float32Array([1.0, 2.0, 3.0]),
    sizes: [1, 3],
    scalarType: ScalarType.FLOAT,
  };

  const output = await m.forward([input]);
  // output: TensorPtr[] — output[0].dataPtr is an ArrayBuffer; interpret per scalarType
};

TensorPtr: { dataPtr: ArrayBuffer | TypedArray, sizes: number[], scalarType: ScalarType }.

You own preprocessing (resize, normalize, color conversion) and postprocessing. Shapes must match your exported model exactly.

Non-hook usage

For services or non-React contexts, use the module class directly via fromModelName:

import { ClassificationModule, models } from 'react-native-executorch';

const m = await ClassificationModule.fromModelName(models.classification.efficientnet_v2_s());

Every hook has a corresponding module: LLMModule, ObjectDetectionModule, OCRModule, SpeechToTextModule, TextToSpeechModule, etc.

---

Tokenization — useTokenizer

HuggingFace-compatible BPE / WordPiece tokenizer. Mostly useful for counting tokens before sending text to embedding models or LLMs.

import { useTokenizer, models } from 'react-native-executorch';

const tokenizer = useTokenizer({ tokenizer: models.text_embedding.all_minilm_l6_v2() });

const ids = await tokenizer.encode('Hello, world!');
const text = await tokenizer.decode(ids);

const vocab = await tokenizer.getVocabSize();
const id = await tokenizer.tokenToId('hello');
const token = await tokenizer.idToToken(id);

You usually don't need this — useLLM and useTextEmbeddings tokenize internally.

---

Privacy filter — usePrivacyFilter

On-device PII detection. Returns PiiEntity[] with { label, text, startToken, endToken }. Useful for redacting messages before they leave the device.

import { usePrivacyFilter, models } from 'react-native-executorch';

const pf = usePrivacyFilter({ model: models.privacy_filter.openai() });

const entities = await pf.generate(
  'Email me at jane@example.com — my account is 1234-5678-0001.'
);
// entities: [
//   { label: 'private_email', text: 'jane@example.com', startToken: …, endToken: … },
//   { label: 'account_number', text: '1234-5678-0001', startToken: …, endToken: … },
// ]

Available accessors:

• models.privacy_filter.openai — 8 entity types (account_number, private_address, private_date, private_email, private_person, private_phone, private_url, secret). Label list: PRIVACY_FILTER_OPENAI_LABELS.
• models.privacy_filter.nemotron — 55 entity types. Label list: PRIVACY_FILTER_NEMOTRON_LABELS.

Long inputs are processed in 50%-overlapping sliding windows automatically — no manual chunking required. The window size is set by the exported model's input shape.

For a custom fine-tune, pass a PrivacyFilterModelSources object directly (with your own modelSource, tokenizerSource, and labelNames). Optional viterbiBiases shift the precision/recall tradeoff.

---

RAG with react-native-rag

For retrieval-augmented generation — vector stores, persistence, document ingestion, and a useRAG hook — use the sibling library react-native-rag. It plugs straight into the models registry through ExecuTorchEmbeddings and ExecuTorchLLM wrappers, so you keep one model-selection convention across both libraries.

npm install react-native-rag
# Optional SQLite-backed persistence (otherwise an in-memory store is used)
npm install @react-native-rag/op-sqlite

import { useRAG, MemoryVectorStore, ExecuTorchEmbeddings, ExecuTorchLLM } from 'react-native-rag';
import { models } from 'react-native-executorch';

const vectorStore = new MemoryVectorStore({
  embeddings: new ExecuTorchEmbeddings(models.text_embedding.all_minilm_l6_v2()),
});

const llm = new ExecuTorchLLM(models.llm.lfm2_5_1_2b_instruct());

export default function App() {
  const rag = useRAG({ vectorStore, llm });

  // rag.addDocument(text), rag.query(question) → rag.response (streamed)
  return <Text>{rag.response}</Text>;
}

When to reach for `react-native-rag` vs. rolling your own:

• Use the library when you need document ingestion + chunking + retrieval + generation as one pipeline, persistent vector storage across launches, or want a hook-based API symmetric with useLLM / useTextEmbeddings.
• Roll your own (just useTextEmbeddings + cosine similarity + useLLM) when the corpus is small and ephemeral, or you need custom retrieval logic the library doesn't expose. The useTextEmbeddings example in vision.md shows the minimal building blocks.

Both ExecuTorchEmbeddings and ExecuTorchLLM accept any model accessor from the registry — same { quant, backend } options apply. Custom components (vector stores, splitters, embedders) implement the library's Embeddings / LLM / VectorStore / TextSplitter interfaces.

---

Device constraints

For per-model memory and inference benchmarks: webfetch Benchmarks.

Guidelines:

• Stay on the default (quantized) variant unless you've measured a need for full precision.
• Test on the lowest-spec device you plan to support.
• Provide a cloud fallback for devices that can't fit the model.
• Surface a cleanup UI via ExpoResourceFetcher.deleteResources — downloaded models can be huge.
• Always show loading states. Model download and inference are seconds-to-minutes operations.

---

See also

• Getting started
• Loading models
• Resource fetcher
• Error handling
• useExecutorchModule API reference
• useTokenizer API reference
• usePrivacyFilter API reference

Speech & Audio

Speech-to-text (Whisper), text-to-speech (Kokoro), and voice activity detection (FSMN). All audio is exchanged as Float32Array PCM through react-native-audio-api.

Pick models through the typed models registry:

• models.speech_to_text.<model>({ quant?, backend? })
• models.text_to_speech.kokoro.<locale>.<voice>() — bundles model + voice + phonemizer per language
• models.vad.fsmn_vad()

---

Critical audio rules

• STT input: 16 kHz mono. Mismatched sample rates produce silently garbled transcriptions.
• VAD input: 16 kHz mono. Same constraint as STT.
• TTS output: 24 kHz. Create the playback AudioContext with { sampleRate: 24000 }.

---

Speech-to-text — useSpeechToText

One-shot transcription

import { useSpeechToText, models } from 'react-native-executorch';
import { AudioContext } from 'react-native-audio-api';
import * as FileSystem from 'expo-file-system';

const stt = useSpeechToText({ model: models.speech_to_text.whisper_tiny_en() });

const { uri } = await FileSystem.downloadAsync(
  'https://example.com/file.mp3',
  FileSystem.cacheDirectory + 'audio.mp3'
);

const audioContext = new AudioContext({ sampleRate: 16000 });
const decoded = await audioContext.decodeAudioData(uri);
const buffer = decoded.getChannelData(0); // Float32Array @ 16 kHz mono

const { text } = await stt.transcribe(buffer);

Multilingual

Use a multilingual Whisper accessor and pass a language code:

const stt = useSpeechToText({ model: models.speech_to_text.whisper_tiny() });
const { text } = await stt.transcribe(buffer, { language: 'es' });

Word-level timestamps

Pass verbose: true to get segments with per-word timing, log-probs, and compression ratios:

const result = await stt.transcribe(buffer, { verbose: true });
// {
//   task: 'transcription',
//   text: '…',
//   duration: 9.05,
//   language: 'en',
//   segments: [
//     {
//       start: 0,
//       end: 5.4,
//       text: '…',
//       words: [{ word: 'Example', start: 0, end: 1.4 }, …],
//       tokens: [...],
//       temperature: 0.0,
//       avgLogProb: -1.235,
//       compressionRatio: 1.63,
//     },
//   ],
// }

Streaming transcription

For audio longer than 30 s, use streaming. It applies the whisper-streaming algorithm so audio is chunked without cutting mid-sentence.

import React, { useEffect, useRef, useState } from 'react';
import { Button, SafeAreaView, Text, View } from 'react-native';
import { useSpeechToText, models } from 'react-native-executorch';
import { AudioManager, AudioRecorder } from 'react-native-audio-api';

export default function StreamingStt() {
  const stt = useSpeechToText({ model: models.speech_to_text.whisper_tiny_en() });
  const [text, setText] = useState('');
  const isRecording = useRef(false);
  const [recorder] = useState(() => new AudioRecorder());

  useEffect(() => {
    AudioManager.setAudioSessionOptions({
      iosCategory: 'playAndRecord',
      iosMode: 'spokenAudio',
      iosOptions: ['allowBluetooth', 'defaultToSpeaker'],
    });
    AudioManager.requestRecordingPermissions();
  }, []);

  const start = async () => {
    isRecording.current = true;
    setText('');

    const sampleRate = 16000;
    recorder.onAudioReady(
      { sampleRate, bufferLength: 0.1 * sampleRate, channelCount: 1 },
      (chunk) => stt.streamInsert(chunk.buffer.getChannelData(0))
    );
    await recorder.start();

    let committed = '';
    for await (const { committed: c, nonCommitted } of stt.stream({ verbose: false })) {
      if (!isRecording.current) break;
      if (c.text) committed += c.text;
      setText(committed + nonCommitted.text);
    }
  };

  const stop = () => {
    isRecording.current = false;
    recorder.stop();
    stt.streamStop();
  };

  return (
    <SafeAreaView>
      <View style={{ padding: 20 }}>
        <Text>{text || 'Press start to speak…'}</Text>
        <Button title="Start" onPress={start} disabled={stt.isGenerating} />
        <Button title="Stop" color="red" onPress={stop} />
      </View>
    </SafeAreaView>
  );
}

Available STT accessors:

---

Text-to-speech — useTextToSpeech

Pick a Kokoro preset that bundles the model, a voice, and the phonemizer for that language:

import { useTextToSpeech, models } from 'react-native-executorch';
import { AudioContext } from 'react-native-audio-api';

const tts = useTextToSpeech({
  model: models.text_to_speech.kokoro.en_us.heart(),
});

const audioContext = new AudioContext({ sampleRate: 24000 });

const speak = async (text: string) => {
  const waveform = await tts.forward({ text, speed: 1.0 }); // Float32Array @ 24 kHz

  const buffer = audioContext.createBuffer(1, waveform.length, 24000);
  buffer.getChannelData(0).set(waveform);

  const source = audioContext.createBufferSource();
  source.buffer = buffer;
  source.connect(audioContext.destination);
  source.start();
};

Streaming TTS

Stream chunks for lower time-to-first-audio on long text:

await tts.stream({
  text: 'Long text streamed chunk by chunk…',
  speed: 1.0,
  onBegin: async () => console.log('start'),
  onNext: async (chunk) =>
    new Promise<void>((resolve) => {
      const buffer = audioContext.createBuffer(1, chunk.length, 24000);
      buffer.getChannelData(0).set(chunk);
      const source = audioContext.createBufferSource();
      source.buffer = buffer;
      source.connect(audioContext.destination);
      source.onEnded = () => resolve();
      source.start();
    }),
  onEnd: async () => console.log('done'),
  stopAutomatically: true,
});

Phoneme input

If you already have phonemes (e.g. from a custom pronunciation pipeline), skip the phonemizer:

const waveform = await tts.forwardFromPhonemes({ phonemes: 'hɛloʊ', speed: 1.0 });

await tts.streamFromPhonemes({
  phonemes: 'hɛloʊ wɜːld',
  speed: 1.0,
  onNext: async (chunk) => { /* play */ },
});

Available TTS presets

models.text_to_speech.kokoro.<locale>.<voice> — locale + voice combinations:

---

Voice activity detection — useVAD

Detects speech segments in an audio buffer. Useful for trimming silence, segmenting recordings, or gating STT.

import { useVAD, models } from 'react-native-executorch';
import { AudioContext } from 'react-native-audio-api';
import * as FileSystem from 'expo-file-system';

const vad = useVAD({ model: models.vad.fsmn_vad() });

const { uri } = await FileSystem.downloadAsync(
  'https://example.com/file.mp3',
  FileSystem.cacheDirectory + 'vad.mp3'
);

const audioContext = new AudioContext({ sampleRate: 16000 });
const decoded = await audioContext.decodeAudioDataSource(uri);
const buffer = decoded.getChannelData(0);

const segments = await vad.forward(buffer);
// segments: { start: number, end: number }[]
// start/end are sample indices — divide by 16000 to get seconds

To concatenate detected speech into a single buffer:

const total = segments.reduce((s, seg) => s + (seg.end - seg.start), 0);
const out = audioContext.createBuffer(1, total, decoded.sampleRate);
const dst = out.getChannelData(0);
let off = 0;
for (const seg of segments) {
  const slice = buffer.subarray(seg.start, seg.end);
  dst.set(slice, off);
  off += slice.length;
}

---

Troubleshooting

• Whisper output is garbled. Audio is not 16 kHz mono — check the AudioContext sample rate and that you're reading channel 0.
• TTS sounds chipmunked or slow. Playback AudioContext is at the wrong rate. Always use { sampleRate: 24000 } for Kokoro output.
• `StreamingNotStarted` when calling `streamInsert`. You must start stream() before inserting chunks.
• `StreamingInProgress` on a second `stream()` call. Call streamStop() and wait before starting again.
• VAD segments look wrong in seconds. They're sample indices — divide by 16000.
• iOS microphone is silent. AudioManager.requestRecordingPermissions() must be called and the user must accept; verify the app has microphone Info.plist entitlements.

---

See also

• useSpeechToText API reference
• useTextToSpeech API reference
• useVAD API reference
• react-native-audio-api

Vision

Image understanding and transformation: classification, object detection, semantic / instance segmentation, pose estimation, OCR, style transfer, text-to-image generation, and image / text embeddings.

All models are selected through the typed models registry: models.<category>.<model>({ quant?, backend? }). Calling with no args returns the platform default (CoreML on iOS / XNNPACK on Android for multi-backend models, quantized variant when published). Passing a backend a model doesn't ship is a compile-time error.

Every hook accepts image input as one of: a remote URL (https://…), a local file URI (file://…), a base64 string, or — for bundled assets — require('../assets/img.jpg'). Remote images are cached automatically.

---

Image classification — useClassification

import { useClassification, models } from 'react-native-executorch';

const model = useClassification({ model: models.classification.efficientnet_v2_s() });

const labels = await model.forward('https://example.com/puppy.png');
// labels: Record<string, number> — ImageNet1k label → probability

const topThree = Object.entries(labels)
  .sort(([, a], [, b]) => b - a)
  .slice(0, 3);

For the full ImageNet1k label set, import Imagenet1kLabel.

---

Object detection — useObjectDetection

import { useObjectDetection, models } from 'react-native-executorch';

const model = useObjectDetection({ model: models.object_detection.yolo26n() });

const detections = await model.forward('https://example.com/street.jpg', {
  detectionThreshold: 0.5, // minimum confidence (0–1)
  iouThreshold: 0.45,      // NMS aggressiveness (0–1)
  inputSize: 640,          // for multi-size YOLO models (384 / 512 / 640)
  classesOfInterest: ['PERSON', 'CAR'], // filter
});

for (const d of detections) {
  console.log(d.bbox, d.label, d.score);
}

forward returns Detection[] with { bbox: { x1, y1, x2, y2 }, label, score }. Coordinates are pixel-space relative to the input image.

YOLO models support multiple input sizes — call model.getAvailableInputSizes() to enumerate them.

Available accessors: models.object_detection.yolo26n / yolo26s / yolo26m / yolo26l / yolo26x, models.object_detection.rf_detr_nano, models.object_detection.ssdlite_320_mobilenet_v3_large.

---

Semantic segmentation — useSemanticSegmentation

Pixel-level classification.

import { useSemanticSegmentation, models, DeeplabLabel } from 'react-native-executorch';

const model = useSemanticSegmentation({
  model: models.semantic_segmentation.deeplab_v3_resnet50(),
});

// Pass classesOfInterest + resizeToInput to also get per-class probability maps
const out = await model.forward(imageUri, ['CAT', 'DOG', 'PERSON'], true);
const argmax = out[DeeplabLabel.ARGMAX];        // class id per pixel
const catProbs = out['CAT'];                    // probability per pixel

Tradeoff: resizeToInput: true upsamples to the original image size — more memory and slower. With false, indices map to a 224×224 grid.

Available accessors: models.semantic_segmentation.deeplab_v3_resnet50 / deeplab_v3_resnet101 / deeplab_v3_mobilenet_v3_large / lraspp_mobilenet_v3_large / fcn_resnet50 / fcn_resnet101 / selfie_segmentation.

---

Instance segmentation — useInstanceSegmentation

Per-instance masks (one mask per detected object).

import { useInstanceSegmentation, models } from 'react-native-executorch';

const model = useInstanceSegmentation({
  model: models.instance_segmentation.yolo26n(),
});

const instances = await model.forward('https://example.com/street.jpg');
// instances: { bbox, label, score, mask }[]

Available accessors: models.instance_segmentation.yolo26n … yolo26x, rf_detr_nano, fastsam_s, fastsam_x.

---

Pose estimation — usePoseEstimation

Detects humans and their COCO 17-keypoint skeletons (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles).

import { usePoseEstimation, models, CocoKeypoint } from 'react-native-executorch';

const model = usePoseEstimation({ model: models.pose_estimation.yolo26n() });

const poses = await model.forward('https://example.com/person.jpg');
// poses: { bbox, score, keypoints: { x, y, confidence }[] }[]

for (const pose of poses) {
  const nose = pose.keypoints[CocoKeypoint.NOSE];
  console.log('Nose at', nose.x, nose.y, 'conf', nose.confidence);
}

Use the CocoKeypoint enum to index into keypoints by name.

---

OCR — useOCR and useVerticalOCR

The OCR pipeline ships a CRAFT detector plus per-alphabet CRNN recognizers. Pick one with a language code via models.ocr.craft({ language }).

import { useOCR, models } from 'react-native-executorch';

const ocr = useOCR({ model: models.ocr.craft({ language: 'en' }) });

const detections = await ocr.forward('https://example.com/receipt.jpg');
for (const d of detections) {
  console.log(d.text, d.score, d.bbox); // bbox = 4-point polygon
}

OCRDetection:

interface OCRDetection {
  bbox: { x: number; y: number }[]; // 4 corner points (supports rotated/skewed text)
  text: string;
  score: number; // 0–1
}

Vertical / CJK text — use useVerticalOCR with independentCharacters: true:

import { useVerticalOCR, models } from 'react-native-executorch';

const ocr = useVerticalOCR({
  model: models.ocr.craft({ language: 'ch_sim' }),
  independentCharacters: true, // recommended for CJK; set false for vertical Latin
});

const detections = await ocr.forward(imageUri);

Alphabet matching matters. Latin and Cyrillic share a detector but use different recognizers. Pass the language code that matches the script you want to read. Unsupported languages throw LanguageNotSupported. Full alphabet support list: webfetch OCR Supported Alphabets.

---

Style transfer — useStyleTransfer

Apply one of four pre-trained artistic styles to an image.

import { useStyleTransfer, models } from 'react-native-executorch';

const model = useStyleTransfer({ model: models.style_transfer.candy() });

// Default: returns PixelData (raw RGB buffer)
const pixelData = await model.forward(imageUri);

// Pass 'url' as second arg to get a file URI back
const styledUri = await model.forward(imageUri, 'url');

Available accessors: models.style_transfer.candy / mosaic / rain_princess / udnie.

Generated images are written to the app's temporary directory. Expect a few seconds of inference per image.

---

Text-to-image — useTextToImage

On-device Stable Diffusion (BK-SDM tiny).

import { useTextToImage, models } from 'react-native-executorch';

const model = useTextToImage({ model: models.image_generation.bk_sdm_tiny_vpred_256() });

const image = await model.generate('a medieval castle by the sea', 256, 25);
// image: base64 PNG. Render with <Image source={{ uri: `data:image/png;base64,${image}` }} />

Signature: generate(prompt, imageSize?, numSteps?). Image size must be a multiple of 32. Expect 20–60 s per image depending on device, size, and step count. Use bk_sdm_tiny_vpred_256 on lower-end devices; bk_sdm_tiny_vpred_512 on high-end devices.

---

Image embeddings — useImageEmbeddings

CLIP-based image vectors for similarity / search. Pair with useTextEmbeddings (using the CLIP text encoder) for cross-modal retrieval.

import { useImageEmbeddings, models } from 'react-native-executorch';

const model = useImageEmbeddings({
  model: models.image_embedding.clip_vit_base_patch32_image(),
});

const v1 = await model.forward(imageUri1); // Float32Array
const v2 = await model.forward(imageUri2);

// Returned vectors are L2-normalized — cosine similarity = dot product
const sim = v1.reduce((s, x, i) => s + x * v2[i], 0);

Images are auto-resized to 224 × 224.

---

Text embeddings — useTextEmbeddings

Sentence-level embeddings for semantic search, similarity, clustering, or RAG. Listed under vision because the CLIP text encoder is the cross-modal pair to image embeddings.

import { useTextEmbeddings, models } from 'react-native-executorch';

const model = useTextEmbeddings({
  model: models.text_embedding.all_minilm_l6_v2(),
});

const v1 = await model.forward('Hello world');
const v2 = await model.forward('Greetings everyone');
const cosine = v1.reduce((s, x, i) => s + x * v2[i], 0); // pre-normalized

Text exceeding Max tokens is truncated. Use useTokenizer (see setup.md) to count first.

Building a full RAG pipeline? Don't roll your own — use react-native-rag (sibling library). It wraps useTextEmbeddings + useLLM with ExecuTorchEmbeddings / ExecuTorchLLM, ships a MemoryVectorStore + an op-sqlite persistence plugin, and exposes a useRAG hook. See setup.md for an end-to-end example.

---

Troubleshooting

• `MemoryAllocationFailed` at load. Step down to a smaller accessor (e.g. yolo26n instead of yolo26x) or pass { quant: true } if a quantized variant is published.
• `LanguageNotSupported` from OCR. The requested language has no published CRAFT/CRNN recognizer pair. Use 'en' for Latin or the relevant ISO code for the script you need.
• Style transfer or text-to-image is slow. Both are compute-heavy; show a progress indicator. For text-to-image, lower numSteps and use the 256 model on mid-range devices.
• Image embeddings cosine similarity outside [-1, 1]. Vectors are pre-normalized — the dot product is already the cosine. If you see anomalies, verify you're not double-normalizing.

---

See also

• API reference — per-hook signatures
• HuggingFace collections — pre-exported model artefacts
• setup.md — loading strategies, error handling, custom .pte models