Tree Sitter Language Pack

Tree-Sitter Language Pack

tree-sitter-language-pack is a polyglot code parsing and analysis library with a high-performance Rust core and bindings for Python, Node.js/TypeScript, Ruby, Go, Java, C#, PHP, Elixir, and WebAssembly. It compiles 306+ tree-sitter grammars into efficient parsers and provides code intelligence extraction: structure (functions, classes), imports, exports, comments, docstrings, diagnostics, and syntax-aware chunking for LLM ingestion.

Use this skill when writing code that:

• Parses source code in any of 306 supported languages
• Extracts code structure, imports, exports, and metadata
• Detects syntax errors and generates diagnostics
• Chunks code intelligently for LLM context windows
• Performs language detection from file paths or content
• Validates custom tree-sitter query patterns
• Integrates tree-sitter parsing into polyglot applications

Installation

Python

pip install tree-sitter-language-pack
# or with uv:
uv add tree-sitter-language-pack

Node.js/TypeScript

npm install @kreuzberg/tree-sitter-language-pack
# or with pnpm:
pnpm add @kreuzberg/tree-sitter-language-pack

Rust

[dependencies]
tree-sitter-language-pack = "1"
# With download feature (default):
# tree-sitter-language-pack = { version = "1", features = ["download"] }

CLI

# From source
cargo install --path crates/ts-pack-cli

# Or download pre-built from GitHub releases

Other Bindings

• Ruby: gem install tree_sitter_language_pack
• Go: go get github.com/kreuzberg-dev/tree-sitter-language-pack/packages/go
• Java: Maven: add dev.kreuzberg.treesitterlanguagepack:tree-sitter-language-pack
• C#: dotnet add package TreeSitterLanguagePack
• PHP: composer require kreuzberg-dev/tree-sitter-language-pack
• Elixir: Mix: {:tree_sitter_language_pack, "~> 1.0"}
• WebAssembly: npm install @kreuzberg/tree-sitter-language-pack-wasm

Quick Start

Python

from tree_sitter_language_pack import (
    parse_string, process, ProcessConfig, available_languages
)

# List available languages
print(f"{len(available_languages())} languages supported")

# Parse source code
tree = parse_string("python", "def hello(): pass")
print(tree.root_node_type())         # "module"
print(tree.has_error_nodes())         # False
print(tree.contains_node_type("function_definition"))  # True

# Extract code intelligence
config = ProcessConfig("python", structure=True, imports=True, docstrings=True)
result = process("def hello(): pass", config)
print(f"Functions: {len(result['structure'])}")

Node.js/TypeScript

import {
  parseString,
  process,
  availableLanguages,
  treeRootNodeType,
} from "@kreuzberg/tree-sitter-language-pack";

// List languages
console.log(`${availableLanguages().length} languages supported`);

// Parse code
const tree = parseString("python", "def hello(): pass");
console.log(treeRootNodeType(tree)); // "module"

// Extract intelligence
const result = process("def hello(): pass", { language: "python" });
console.log(`Functions: ${result.structure.length}`);

Rust

use tree_sitter_language_pack::{
    parse_string, process, ProcessConfig, available_languages
};

// List languages
println!("{} languages available", available_languages().len());

// Parse code
let tree = parse_string("python", b"def hello(): pass")?;
println!("{}", tree.root_node().kind());  // "module"

// Extract intelligence
let config = ProcessConfig::new("python").all();
let result = process("def hello(): pass", &config)?;
println!("Functions: {}", result.structure.len());

CLI

# Parse a file
ts-pack parse src/main.py

# Extract code intelligence
ts-pack process src/main.py --all

# Detect language
ts-pack detect src/main.rs

# List available languages
ts-pack list

# Download specific languages
ts-pack download python javascript typescript rust

Key APIs

Language Discovery & Detection

Parsing

Code Intelligence & Processing

Download Management

ProcessConfig Options

Control what analysis features are enabled:

Use ProcessConfig.all(language) to enable all features, or ProcessConfig.minimal(language) to disable all extractions (metrics only).

Common Patterns

Detect and Parse

from tree_sitter_language_pack import detect_language, parse_string

lang = detect_language("src/main.rs")
if lang:
    source = open(f"src/main.rs").read()
    tree = parse_string(lang, source)
    print(tree.has_error_nodes())

Batch Processing

from tree_sitter_language_pack import detect_language, process, ProcessConfig
from pathlib import Path

for filepath in Path("src").glob("**/*"):
    if filepath.is_file():
        lang = detect_language(str(filepath))
        if lang:
            source = filepath.read_text()
            config = ProcessConfig(lang, structure=True, imports=True)
            result = process(source, config)
            print(f"{filepath}: {len(result['structure'])} items")

Code Chunking for LLMs

from tree_sitter_language_pack import process, ProcessConfig

config = ProcessConfig(
    "python",
    structure=True,
    chunk_max_size=1000,  # ~1000 tokens per chunk
)
result = process(open("large_file.py").read(), config)

for i, chunk in enumerate(result["chunks"]):
    print(f"Chunk {i}: lines {chunk['start_line']}-{chunk['end_line']}")
    # Feed chunk["content"] to LLM

Custom Extraction Queries

from tree_sitter_language_pack import extract

config = {
    "language": "python",
    "patterns": {
        "decorators": {
            "query": "(decorator (identifier) @name)",
            "capture_output": "Text",
        },
        "type_hints": {
            "query": "(typed_parameter type: (identifier) @type)",
            "capture_output": "Text",
        },
    },
}

result = extract("@dataclass\ndef process(x: int): pass", config)
for pattern_name, matches in result["results"].items():
    print(f"{pattern_name}: {matches['total_count']} matches")

Common Pitfalls

1. Forgetting to close/free resources: Tree handles and registries consume native memory. Always dispose of them when done (use context managers in Python).

2. Auto-download latency: First call to parse an uncached language triggers a network request and download. Pre-download languages in init() for production.

3. Chunking without metrics: Always enable structure=True when using chunking to get node type metadata for each chunk.

4. Query syntax errors: Validate extraction patterns with validate_extraction() before running against large codebases.

5. Language name casing: All language names are lowercase (e.g., "python", not "Python").

6. Extension ambiguity: Some extensions (e.g., ".h") map to multiple languages. Use extension_ambiguity() to check and resolve manually.

References

• Python API Reference
• TypeScript/Node.js API Reference
• Rust API Reference
• CLI Reference
• Other Bindings (Go, Java, C#, Ruby, Elixir, PHP, WASM, C FFI)
• Code Intelligence Extraction
• Configuration & Download Model

CLI Reference

Installation

# From source
cargo install --path crates/ts-pack-cli

# Or from GitHub releases
# Download pre-built binary for your platform

Global Options

--help, -h          Show help message
--version, -V       Show version

Commands

ts-pack download

Download parser libraries to the local cache.

ts-pack download [OPTIONS] [LANGUAGES]...

Arguments:

• [LANGUAGES]... — Language names (space-separated), or use language-pack.toml if omitted

Options:

• --all — Download all 306 available languages
• --groups <GROUPS> — Language groups (comma-separated: web, systems, scripting, data, jvm, functional)
• --fresh — Clean cache before downloading

Examples:

ts-pack download python rust typescript
ts-pack download --all
ts-pack download --groups web,data
ts-pack download --fresh python rust
ts-pack download  # Uses language-pack.toml if present

ts-pack clean

Remove all cached parser libraries.

ts-pack clean [OPTIONS]

Options:

• --force — Skip confirmation prompt

Examples:

ts-pack clean
ts-pack clean --force

ts-pack list

List available languages.

ts-pack list [OPTIONS]

Options:

• --downloaded — Show only cached languages
• --manifest — Show all available languages from remote manifest
• --filter <SUBSTRING> — Filter by substring

Examples:

ts-pack list                    # All available
ts-pack list --downloaded       # Only cached
ts-pack list --filter python    # Search for language

ts-pack info

Show details about a specific language.

ts-pack info <LANGUAGE>

Output fields:

• Language name
• Known (compiled-in or in manifest)
• Downloaded (yes/no)
• Cache path (if downloaded) or cache directory

Example:

ts-pack info python
# Language:    python
# Known:       true
# Downloaded:  true
# Cache path:  /home/user/.cache/ts-pack/libtree_sitter_python.so

ts-pack parse

Parse a file and output the syntax tree.

ts-pack parse [OPTIONS] <FILE>

Arguments:

• <FILE> — Source file to parse (use - for stdin)

Options:

• --language, -l <LANG> — Language name (auto-detected from extension if omitted)
• --format, -f <FORMAT> — Output format: sexp (default) or json

Examples:

ts-pack parse code.py
ts-pack parse code.py --language python
ts-pack parse code.py --format json
echo "x = 1" | ts-pack parse - --language python

JSON output format:

{
  "language": "python",
  "sexp": "(module (expression_statement ...))",
  "has_errors": false
}

ts-pack process

Run code intelligence pipeline on a file.

ts-pack process [OPTIONS] <FILE>

Arguments:

• <FILE> — Source file to process (use - for stdin)

Options:

• --language, -l <LANG> — Language name (auto-detected if omitted; required for stdin)
• --all — Enable all analysis features
• --structure — Extract functions, classes, methods
• --imports — Extract import statements
• --exports — Extract export statements
• --comments — Extract comments
• --symbols — Extract symbols/identifiers
• --docstrings — Extract docstrings
• --diagnostics — Include parse diagnostics
• --chunk-size <BYTES> — Maximum bytes per chunk (enables chunking)

When no feature flags given, defaults apply (structure, imports, exports enabled).

Examples:

ts-pack process code.py --all
ts-pack process code.py --structure
ts-pack process code.py --all --chunk-size 1000
cat code.py | ts-pack process - --language python --structure

Output is JSON to stdout:

{
  "language": "python",
  "metrics": { "total_lines": 10, "code_lines": 8, ... },
  "structure": [...],
  "imports": [...],
  "chunks": [...]
}

ts-pack cache-dir

Print the effective cache directory path.

ts-pack cache-dir

Output:

/home/user/.cache/tree-sitter-language-pack/v1.0.0/libs/

Usage in scripts:

CACHE=$(ts-pack cache-dir)
du -sh "$CACHE"  # Show cache size

ts-pack init

Create a language-pack.toml configuration file.

ts-pack init [OPTIONS]

Options:

• --cache-dir <PATH> — Set cache directory in config
• --languages <LANGS> — Languages to include (comma-separated); also downloads them

Examples:

ts-pack init
ts-pack init --languages python,rust,typescript
ts-pack init --cache-dir /opt/ts-pack --languages python

Generated file (language-pack.toml):

languages = ["python", "rust", "typescript"]

ts-pack completions

Generate shell completions.

ts-pack completions <SHELL>

Arguments:

• <SHELL> — bash, zsh, fish, elvish, or powershell

Examples:

ts-pack completions bash > ~/.local/share/bash-completion/completions/ts-pack
ts-pack completions zsh > ~/.zfunc/_ts-pack
ts-pack completions fish > ~/.config/fish/completions/ts-pack.fish

Exit Codes

• 0 — Success
• 1 — Error (parse failure, missing file, invalid language, network error, etc.)

Errors are printed to stderr.

Language-pack.toml Configuration

Auto-discovered configuration file:

languages = ["python", "javascript", "typescript", "rust"]

# Optional language groups
# groups = ["web", "systems", "data"]

# Optional custom cache directory
# cache_dir = ".cache/ts-pack"

Discovery order:

1. Current directory and parent directories (up to 10 levels) 2. $XDG_CONFIG_HOME/tree-sitter-language-pack/config.toml or ~/.config/tree-sitter-language-pack/config.toml 3. TSLP_CONFIG environment variable

Common Workflows

Setup a Project

ts-pack init --languages python,rust,typescript
ts-pack download

Download and Parse

ts-pack download python
ts-pack parse code.py

Batch Processing

ts-pack download python
for file in src/**/*.py; do
    ts-pack process "$file" --structure --imports
done

Extract Code Structure

ts-pack process src/main.py --structure --format json | jq '.structure'

Check Cache Size

du -sh "$(ts-pack cache-dir)"

Environment Variables

• TSLP_CACHE_DIR — Override cache directory
• TSLP_CONFIG — Path to language-pack.toml
• TSLP_VERBOSE — Enable verbose output
• TSLP_NO_COLOR — Disable colored output

Code Intelligence & Chunking Quick Reference

ProcessConfig Options

Enable only what you need:

Use ProcessConfig.all(language) to enable everything, or ProcessConfig.minimal(language) to extract only metrics.

ProcessResult Structure

result = process(source, config)

result["language"]        # str: language used
result["metrics"]         # FileMetrics dict
result["structure"]       # list[StructureItem]: functions, classes, etc.
result["imports"]         # list[ImportInfo]: import statements
result["exports"]         # list[ExportInfo]: exported symbols
result["comments"]        # list[CommentInfo]: code comments
result["docstrings"]      # list[DocstringInfo]: docstrings
result["symbols"]         # list[SymbolInfo]: all identifiers
result["diagnostics"]     # list[Diagnostic]: syntax errors
result["chunks"]          # list[CodeChunk]: code chunks (if chunking enabled)

FileMetrics

Available when any processing enabled:

metrics = result["metrics"]

metrics["total_lines"]     # int: total lines in file
metrics["code_lines"]      # int: lines with code (no blanks/comments)
metrics["comment_lines"]   # int: lines with comments
metrics["blank_lines"]     # int: blank lines
metrics["total_bytes"]     # int: bytes in file
metrics["error_count"]     # int: number of syntax errors

Structure Items

Functions, classes, methods, modules, etc.

for item in result["structure"]:
    item["kind"]           # str: "function", "class", "method", "module", etc.
    item["name"]           # str: identifier name
    item["start_line"]     # int: 1-indexed
    item["end_line"]       # int
    item["start_byte"]     # int
    item["end_byte"]       # int
    item["parent"]         # str or None: enclosing class/module if nested
    item["docstring"]      # str or None: if docstrings=True
    item["visibility"]     # str or None: "public", "private", etc.

Example (Python):

# def greet(name: str) -> str:
#     """Greet a user."""
#     return f"Hello, {name}!"

# item =
{
    "kind": "function",
    "name": "greet",
    "start_line": 1,
    "end_line": 3,
    "docstring": "Greet a user.",
    "visibility": "public",
}

Import/Export Items

for imp in result["imports"]:
    imp["source"]          # str: module name ("os", "pathlib", "./utils")
    imp["names"]           # list[str]: imported items (empty = wildcard)
    imp["alias"]           # str or None: "as" clause
    imp["start_line"]      # int
    imp["is_wildcard"]     # bool

for exp in result["exports"]:
    exp["name"]            # str: exported symbol
    exp["kind"]            # str: "function", "class", "constant", etc.
    exp["start_line"]      # int

Examples (Python):

# import os
# {"source": "os", "names": [], "is_wildcard": true}

# from pathlib import Path
# {"source": "pathlib", "names": ["Path"], "is_wildcard": false}

# def greet(): ...
# {"name": "greet", "kind": "function"}

Comment/Docstring Items

for comment in result["comments"]:
    comment["text"]        # str: comment text
    comment["kind"]        # str: "line", "block", "doc"
    comment["start_line"]  # int
    comment["is_block"]    # bool: multi-line comment

for docstring in result["docstrings"]:
    docstring["text"]      # str: full docstring text
    docstring["format"]    # str: "markdown", "restructuredtext", "google", "numpy", etc.
    docstring["start_line"] # int
    docstring["associated_item"]  # str or None: function/class name
    docstring["sections"]  # list[dict]: parsed sections (Args, Returns, etc.)

Symbol Items

All identifiers in the file (variable names, functions, classes):

for symbol in result["symbols"]:
    symbol["name"]         # str: identifier
    symbol["kind"]         # str: "variable", "function", "class", "parameter", etc.
    symbol["start_line"]   # int
    symbol["type_annotation"]  # str or None: type hint if present

Diagnostics

Syntax errors detected by tree-sitter:

for diag in result["diagnostics"]:
    diag["message"]        # str: error description
    diag["severity"]       # str: "error", "warning", "info"
    diag["start_line"]     # int
    diag["start_column"]   # int
    diag["end_line"]       # int
    diag["end_column"]     # int

Note: Non-empty diagnostics doesn't mean unparsable — tree-sitter recovers and continues parsing.

Code Chunking for LLMs

Enable with chunk_max_size (in bytes):

config = ProcessConfig(
    "python",
    structure=True,
    chunk_max_size=1000,    # ~1000 tokens per chunk
)
result = process(large_source, config)

for chunk in result["chunks"]:
    chunk["content"]        # str: source code text
    chunk["start_line"]     # int: first line (1-indexed)
    chunk["end_line"]       # int: last line
    chunk["start_byte"]     # int
    chunk["end_byte"]       # int
    chunk["metadata"]       # ChunkContext

ChunkContext fields:

metadata = chunk["metadata"]

metadata["language"]       # str: "python"
metadata["chunk_index"]    # int: position in sequence
metadata["total_chunks"]   # int: total number of chunks
metadata["node_types"]     # list[str]: AST node types in this chunk
metadata["symbols_defined"]  # list[str]: names defined in this chunk
metadata["comments"]       # list[str]: comments in this chunk
metadata["docstrings"]     # list[str]: docstrings in this chunk
metadata["has_error_nodes"]  # bool: syntax errors in this chunk
metadata["context_path"]   # list[str]: scope path (e.g., ["MyClass", "method"])

Chunking Algorithm

1. Collect units: Walk AST, collect top-level declarations (functions, classes) as atomic units 2. Pack greedily: Fit units into chunks without exceeding chunk_max_size 3. Split oversized: If a single unit exceeds budget, split at sub-boundaries (methods in class, statement blocks in function)

Guarantees:

• Functions never split unless individually too large
• Decorators/docstrings stay with function
• Classes keep method lists together where possible
• Imports grouped at top

Custom Extraction Queries

Use arbitrary tree-sitter S-expression patterns:

config = {
    "language": "python",
    "patterns": {
        "pattern_name": {
            "query": "(identifier) @name",
            "capture_output": "Text",  # or "Node", "Full"
            "child_fields": [],
            "max_results": None,       # None = unlimited
            "byte_range": None,        # None = entire file
        }
    }
}

result = extract(source, config)

for match in result["results"]["pattern_name"]["matches"]:
    match["pattern_index"]  # int: index in query
    for capture in match["captures"]:
        capture["name"]     # str: capture name from query
        capture["text"]     # str or None: matched text
        capture["node"]     # dict or None: node info
        capture["child_fields"]  # dict: extracted child field values
        capture["start_byte"]  # int

Example:

# Extract all decorator names
extract(source, {
    "language": "python",
    "patterns": {
        "decorators": {
            "query": "(decorator (identifier) @name)",
            "capture_output": "Text",
        }
    }
})

Pattern Validation

Validate patterns before expensive extraction:

result = validate_extraction({
    "language": "python",
    "patterns": {
        "decorators": {
            "query": "(decorator (identifier) @name)",
        }
    }
})

if result["valid"]:
    print("Ready to extract")
else:
    for name, info in result["patterns"].items():
        if not info["valid"]:
            print(f"{name}: {', '.join(info['errors'])}")

Common Extraction Patterns

Functions with Type Hints

{
    "language": "python",
    "patterns": {
        "typed_functions": {
            "query": "(function_definition name: (identifier) @fn_name return_type: (type) @type)",
            "capture_output": "Text",
        }
    }
}

Class Definitions with Decorators

{
    "language": "python",
    "patterns": {
        "decorated_classes": {
            "query": "((decorator (identifier) @decorator) @dec)+ (class_definition name: (identifier) @class_name)",
            "capture_output": "Full",
            "child_fields": ["decorator", "class_name"],
        }
    }
}

All Function Calls

{
    "language": "python",
    "patterns": {
        "function_calls": {
            "query": "(call function: (identifier) @func_name arguments: (argument_list) @args)",
            "capture_output": "Text",
            "max_results": 100,  # Limit results for large files
        }
    }
}

Token Counting

Chunking uses cl100k_base approximation: 4 characters ≈ 1 token (matches GPT-4, Claude, Llama).

chunk["token_count"]       # int: estimated tokens in this chunk
metadata["token_count"]    # int: same as above

The chunk_max_size parameter is an upper bound, not exact. Chunks may be smaller (natural boundaries) or slightly exceed (only split point is past limit).

Language-Specific Structure Kinds

Configuration & Download Model Quick Reference

Cache Directory

Where downloaded parser binaries are stored.

Default locations:

• Linux: $XDG_CACHE_HOME/tree-sitter-language-pack or ~/.cache/tree-sitter-language-pack
• macOS: ~/Library/Caches/tree-sitter-language-pack
• Windows: %LOCALAPPDATA%\tree-sitter-language-pack

Override via:

Programmatic API

Python:

from tree_sitter_language_pack import configure
configure(cache_dir="/custom/path")

TypeScript:

import { configure } from "@kreuzberg/tree-sitter-language-pack";
configure({ cacheDir: "/custom/path" });

Rust:

use tree_sitter_language_pack::*;
configure(&PackConfig {
    cache_dir: Some("/custom/path".into()),
    ..Default::default()
})?;

Environment Variable

export TSLP_CACHE_DIR=/custom/path

CLI Flag

ts-pack --cache-dir /custom/path download python

language-pack.toml Configuration File

Define languages to pre-download and cache settings.

Format

# language-pack.toml

# Language names to pre-download
languages = ["python", "javascript", "typescript", "rust"]

# Optional: language groups (web, systems, data, jvm, functional, scripting)
# groups = ["web", "systems"]

# Optional: custom cache directory
# cache_dir = ".cache/parsers"

Creation

CLI:

ts-pack init
ts-pack init --languages python,javascript,typescript,rust
ts-pack init --cache-dir ./.cache/parsers --languages python

Manual: Create language-pack.toml in project root.

Discovery Order

1. Current directory and parent directories (up to 10 levels) 2. $XDG_CONFIG_HOME/tree-sitter-language-pack/config.toml or ~/.config/tree-sitter-language-pack/config.toml 3. TSLP_CONFIG environment variable

Priority: CLI flags > Environment variables > Config file > Defaults

Download Management

Pre-downloading Languages

from tree_sitter_language_pack import init, download, download_all

# Pre-download specific languages
init({"languages": ["python", "javascript", "rust"]})

# Or download after init
download(["python", "typescript"])

# Download all 306 languages
download_all()

CLI:

ts-pack download python rust typescript
ts-pack download --all
ts-pack download --groups web,data

Returns: number of newly downloaded languages.

Checking Cache

from tree_sitter_language_pack import downloaded_languages, manifest_languages

local = downloaded_languages()      # No network
remote = manifest_languages()       # Fetches manifest
missing = set(remote) - set(local)

CLI:

ts-pack list --downloaded
ts-pack list --manifest
ts-pack status

Cleaning Cache

from tree_sitter_language_pack import clean_cache
clean_cache()  # Delete all cached parsers

CLI:

ts-pack clean
ts-pack clean --force  # Skip confirmation

Language Groups

Pre-defined collections of related languages:

Usage

Python:

init({"groups": ["web", "systems"]})

CLI:

ts-pack download --groups web,systems

Download on First Use

By default, calling parse_string() or get_parser() triggers auto-download of uncached languages:

# First call triggers download (network I/O)
tree = parse_string("python", "x = 1")

# Subsequent calls use cached parser (fast)
tree = parse_string("python", "y = 2")

This is convenient for development but adds latency in production. Pre-download languages for production deployments.

Docker & CI Integration

Dockerfile Pattern

Pre-bake parsers into image:

FROM python:3.11-slim

RUN pip install tree-sitter-language-pack

# Pre-download parsers (bakes into image)
RUN python -c "from tree_sitter_language_pack import download; download(['python', 'javascript', 'rust'])"

COPY . /app
WORKDIR /app

# Parsing now offline, no network calls
CMD ["python", "app.py"]

GitHub Actions Pattern

Cache parsers between runs:

- name: Cache tree-sitter parsers
  uses: actions/cache@v4
  with:
    path: ~/.cache/tree-sitter-language-pack
    key: tslp-${{ hashFiles('language-pack.toml') }}
    restore-keys: tslp-

- name: Download languages
  run: ts-pack download

- name: Run tests
  run: pytest

Configuration Files

Loading from File

Python:

from tree_sitter_language_pack import PackConfig, init

# Load from language-pack.toml
config = PackConfig.from_toml_file("language-pack.toml")
if config.languages:
    init({"languages": config.languages})

# Auto-discover language-pack.toml
config = PackConfig.discover()
if config:
    init({"languages": config.languages})

Rust:

use tree_sitter_language_pack::PackConfig;

let config = PackConfig::from_toml_file("language-pack.toml")?;
if let Some(languages) = config.languages {
    init(&PackConfig {
        languages: Some(languages),
        ..Default::default()
    })?;
}

// Discover config
if let Some(config) = PackConfig::discover() {
    println!("Found: {:?}", config.languages);
}

Environment Variables

Monorepo Setup

Shared cache across multiple sub-projects:

# language-pack.toml (at repo root)

languages = [
    # Backend
    "python",

    # Frontend
    "javascript",
    "typescript",
    "jsx",
    "tsx",

    # Utilities
    "rust",

    # DevOps
    "bash",
    "dockerfile",
    "yaml",
    "json",
]

# Shared cache
cache_dir = ".cache/tree-sitter"

Download Model Overview

1. Call get_parser("python") or parse_string("python", source) 2. Check local cache for python.so / python.dylib / python.dll 3. If not cached:

• Fetch parsers.json from GitHub releases (manifest)
• Get platform-specific download URL for current OS/arch
• Download binary to cache directory
• Open via dlopen / LoadLibrary

4. Subsequent calls use cached binary (no network)

Manifest cached locally and refreshed on version upgrades.

Offline Mode

1. On a machine with network access:

   ts-pack download --all
   tar czf ts-pack-cache.tar.gz ~/.cache/tree-sitter-language-pack

2. Transfer to offline machine:

   tar xzf ts-pack-cache.tar.gz -C ~

3. Parsing now works offline (cache pre-populated)

Alternatively, use Docker image with pre-baked parsers.

Platform-Specific Notes

Linux

Default: $XDG_CACHE_HOME/tree-sitter-language-pack (or ~/.cache/tree-sitter-language-pack)

Check size:

du -sh ~/.cache/tree-sitter-language-pack

macOS

Default: ~/Library/Caches/tree-sitter-language-pack

For persistent cache on CI, use explicit cache_dir in config.

Windows

Default: %LOCALAPPDATA%\tree-sitter-language-pack

Typical path: C:\Users\<username>\AppData\Local\tree-sitter-language-pack

Troubleshooting

Parser downloads failing

Diagnosis:

ts-pack cache-dir                        # Check location
ts-pack download python --verbose        # Verbose download
curl -I https://releases.kreuzberg.dev/  # Check network access

Solutions:

• Pre-download on machine with network, transfer cache
• Set TSLP_CACHE_DIR to custom path
• Use Docker image with pre-baked parsers
• Configure corporate proxy if behind firewall

Stale cache

# Clear all parsers
ts-pack clean

# Or manually
rm -rf ~/.cache/tree-sitter-language-pack

Disk space issues

# Check size
du -sh ~/.cache/tree-sitter-language-pack

# Move to larger drive
mkdir -p /mnt/large/ts-pack-cache
export TSLP_CACHE_DIR=/mnt/large/ts-pack-cache
ts-pack download --all

Quick Start Commands

Development

# Create config
ts-pack init --languages python,rust,typescript

# Download languages
ts-pack download

# Parse files
ts-pack parse src/main.py

Production

# Pre-download before deploying
ts-pack download --all
tar czf parsers.tar.gz ~/.cache/tree-sitter-language-pack

# Deploy with cache
# Deploy parsers.tar.gz alongside application
export TSLP_CACHE_DIR=/opt/parsers
tar xzf parsers.tar.gz -C /opt

# Run application (no network calls)
python app.py

Docker

# Build image with pre-baked parsers
docker build -t myapp .

# Run (cache mounted if needed)
docker run -it myapp

Other Language Bindings Quick Reference

Go

Installation

go get github.com/kreuzberg-dev/tree-sitter-language-pack/packages/go

Quick Start

package main

import (
    "fmt"
    tspack "github.com/kreuzberg-dev/tree-sitter-language-pack/packages/go"
)

func main() {
    reg, _ := tspack.NewRegistry()
    defer reg.Close()

    tree, _ := reg.ParseString("python", "def hello(): pass")
    defer tree.Close()

    fmt.Println(tree.RootNodeType())          // "module"
    fmt.Println(tree.ContainsNodeType("function_definition"))  // true

    config := tspack.NewProcessConfig("python")
    result, _ := reg.Process("def hello(): pass", config)
    fmt.Printf("Functions: %d\n", len(result.Metadata.Structure))
}

Key Differences

• Registry is thread-safe, must be closed: defer reg.Close()
• Trees must be closed: defer tree.Close()
• Config is ProcessConfig struct, not JSON
• Errors are returned as second value
• All functions are synchronous

---

Java

Installation

Requires JDK 25+ (Panama FFI). Set TSPACK_LIB_PATH to native library path.

Maven:

<dependency>
    <groupId>dev.kreuzberg.treesitterlanguagepack</groupId>
    <artifactId>tree-sitter-language-pack</artifactId>
    <version>1.8.0</version>
</dependency>

Quick Start

import dev.kreuzberg.treesitterlanguagepack.*;

public class Main {
    public static void main(String[] args) {
        try (var registry = new TsPackRegistry()) {
            var languages = registry.availableLanguages();
            System.out.printf("%d languages available%n", languages.size());

            try (var tree = registry.parseString("python", "def hello(): pass")) {
                System.out.println(tree.rootNodeType());     // "module"
                System.out.println(tree.rootChildCount());   // 1
                System.out.println(tree.containsNodeType("function_definition")); // true
            }

            String configJson = "{\"language\":\"python\",\"structure\":true}";
            String resultJson = registry.process("def hello(): pass", configJson);
            System.out.println(resultJson);
        }
    }
}

Key Differences

• Implements AutoCloseable, use try-with-resources: try (var registry = new TsPackRegistry())
• Config and results are JSON strings, decode/encode manually
• Static methods for download/init (don't require registry instance)
• Specific exception: LanguageNotFoundException

---

C# / .NET

Installation

Requires .NET 10+.

dotnet add package TreeSitterLanguagePack

Quick Start

using TreeSitterLanguagePack;

// List languages
string[] langs = TsPackClient.AvailableLanguages();
Console.WriteLine($"{langs.Length} languages available");

// Parse code
using var tree = TsPackClient.Parse("python", "def hello(): pass");
Console.WriteLine(tree.RootNodeType());  // "module"

// Process code
var config = new ProcessConfig { Language = "python", Structure = true };
var result = TsPackClient.Process("def hello(): pass", config);
Console.WriteLine($"Functions: {result.Structure.Count}");

Key Differences

• Static client class TsPackClient (not instance-based)
• Config is typed class ProcessConfig, results are deserialized objects
• Tree implements IDisposable, use using statement
• Exceptions: TsPackException (inherits Exception)
• Thread-safe with lazy initialization

---

Ruby

Installation

gem install tree_sitter_language_pack
# Or in Gemfile:
gem "tree_sitter_language_pack"

Quick Start

require "tree_sitter_language_pack"

# List languages
langs = TreeSitterLanguagePack.available_languages
puts "#{langs.length} languages available"

# Parse code
tree = TreeSitterLanguagePack.parse_string("python", "def hello(): pass")
puts tree.root_node_type         # "module"
puts tree.has_error_nodes        # false
puts tree.contains_node_type("function_definition") # true

# Process code (returns JSON string)
config = { language: "python", structure: true }.to_json
result_json = TreeSitterLanguagePack.process("def hello(): pass", config)
result = JSON.parse(result_json)
puts "Functions: #{result['structure'].length}"

Key Differences

• All module-level functions (no classes)
• Parse returns opaque Tree reference (not dereferenceable)
• Config and results are JSON strings, convert with .to_json and JSON.parse()
• Errors raised as RuntimeError
• Pattern extraction supported: extract(), validate_extraction()

---

Elixir

Installation

Mix:

def deps do
  [{:tree_sitter_language_pack, "~> 1.0"}]
end

Quick Start

# List languages
langs = TreeSitterLanguagePack.available_languages()
IO.puts("#{length(langs)} languages available")

# Parse code
tree = TreeSitterLanguagePack.parse_string("python", "def hello(): pass")
IO.puts(TreeSitterLanguagePack.tree_root_node_type(tree))  # "module"
IO.puts(TreeSitterLanguagePack.tree_has_error_nodes(tree))  # false

# Process code (returns map)
config = Jason.encode!(%{"language" => "python", "structure" => true})
result = TreeSitterLanguagePack.process("def hello(): pass", config)
IO.puts("Functions: #{length(result["structure"])}")

Key Differences

• All module-level functions
• Config is JSON string, results are maps (not JSON strings)
• Parse returns opaque tree reference
• I/O functions run on DirtyIo scheduler (non-blocking)
• Errors raised as Erlang errors
• Pattern extraction supported: extract(), validate_extraction()

---

PHP

Installation

Composer:

composer require kreuzberg-dev/tree-sitter-language-pack

PHP 8.2+, requires native Rust extension (ext-php-rs).

Quick Start

<?php
declare(strict_types=1);

use TreeSitterLanguagePack\TreeSitterLanguagePack;
use TreeSitterLanguagePack\ProcessConfig;

// List languages
$langs = TreeSitterLanguagePack::availableLanguages();
echo count($langs) . " languages available\n";

// Parse code (returns S-expression string)
$sexp = TreeSitterLanguagePack::parseString("python", "def hello(): pass");
echo "Tree: $sexp\n";

// Process code
$config = new ProcessConfig("python", structure: true, imports: true);
$result = TreeSitterLanguagePack::process("def hello(): pass", $config);
echo count($result['structure']) . " structure items\n";

Key Differences

• OOP wrapper class around procedural extension functions
• Parse returns S-expression string (not tree object)
• Config is typed ProcessConfig class (PHP 8.2 constructor promotion)
• Results are associative arrays (decoded from JSON)
• Use procedural ts_pack_* functions directly if preferred
• Exceptions: Exception base class

---

WebAssembly

Installation

npm:

npm install @kreuzberg/tree-sitter-language-pack-wasm

Browser (ES module):

<script type="module">
  import * as tsp from "https://cdn.jsdelivr.net/npm/@kreuzberg/tree-sitter-language-pack-wasm";
</script>

Quick Start

import * as tsp from "@kreuzberg/tree-sitter-language-pack-wasm";

// List languages
const langs = tsp.availableLanguages();
console.log(`${langs.length} languages available`);

// Parse code
const tree = tsp.parseString("python", "def hello(): pass");
console.log(tsp.treeRootNodeType(tree)); // "module"
console.log(tsp.treeHasErrorNodes(tree)); // false
tsp.freeTree(tree);

// Process code (config is JS object)
const result = tsp.process("def hello(): pass", { language: "python" });
console.log(`Functions: ${result.structure.length}`);

Key Differences

• Curated subset of languages (not all 306) optimized for browser/edge
• No download API (stubs that throw)
• Config is JS object, results are JS objects (no JSON conversion needed)
• Single-threaded (use Web Workers for large files)
• Manual memory management: freeTree() releases memory promptly
• Pattern extraction supported: extract(), validateExtraction()

---

C / FFI

Installation

Header file: crates/ts-pack-ffi/include/ts_pack.h

Link against compiled library:

gcc -o program program.c -L. -lts_pack_ffi

Quick Start

#include "ts_pack.h"
#include <stdio.h>
#include <string.h>

int main(void) {
    TsPackRegistry* reg = ts_pack_registry_new();
    if (!reg) {
        fprintf(stderr, "Error: %s\n", ts_pack_last_error());
        return 1;
    }

    // Check for Python
    if (!ts_pack_has_language(reg, "python")) {
        fprintf(stderr, "Python not available\n");
        ts_pack_registry_free(reg);
        return 1;
    }

    // Parse code
    const char* code = "def hello(): pass";
    TsPackTree* tree = ts_pack_parse_string(reg, "python", code, strlen(code));
    if (!tree) {
        fprintf(stderr, "Parse error: %s\n", ts_pack_last_error());
        ts_pack_registry_free(reg);
        return 1;
    }

    // Inspect tree
    char* root_type = ts_pack_tree_root_node_type(tree);
    printf("Root: %s\n", root_type);
    ts_pack_free_string(root_type);

    // Cleanup
    ts_pack_tree_free(tree);
    ts_pack_registry_free(reg);
    return 0;
}

Key Differences

• Opaque handles: TsPackRegistry*, TsPackTree*
• Error handling via thread-local string: ts_pack_last_error()
• Manual memory management: ts_pack_free_string() for allocated strings
• All results are newly-allocated (caller owns memory)
• #[no_mangle] extern "C" functions for C interop
• Use with cgo (Go), Panama FFI (Java), P/Invoke (C#)

---

Summary Table

Python API Quick Reference

Installation

pip install tree-sitter-language-pack

Language Discovery

available_languages() -> list[str]
has_language(name: str) -> bool
language_count() -> int
detect_language(path: str) -> str | None
detect_language_from_content(content: str) -> str | None
detect_language_from_extension(ext: str) -> str | None
detect_language_from_path(path: str) -> str | None
extension_ambiguity(ext: str) -> tuple[str, list[str]] | None

Parsing

parse_string(language: str, source: str) -> TreeHandle
# Returns opaque tree handle

tree.root_node_type() -> str
tree.root_child_count() -> int
tree.contains_node_type(node_type: str) -> bool
tree.has_error_nodes() -> bool
tree.error_count() -> int
tree.to_sexp() -> str
tree.root_node_info() -> dict  # {kind, is_named, start_byte, end_byte, ...}
tree.find_nodes_by_type(node_type: str) -> list[dict]
tree.named_children_info() -> list[dict]
tree.extract_text(start_byte: int, end_byte: int) -> str
tree.run_query(language: str, query_source: str) -> list[dict]

Example:

tree = parse_string("python", "def hello(): pass")
print(tree.root_node_type())          # "module"
print(tree.root_child_count())         # 1
print(tree.contains_node_type("function_definition"))  # True
print(tree.to_sexp())  # S-expression

Code Intelligence Processing

process(source: str, config: ProcessConfig) -> dict
# Returns {language, metrics, structure, imports, exports, comments,
#          docstrings, symbols, diagnostics, chunks}

ProcessConfig

ProcessConfig(
    language: str,
    structure: bool = True,
    imports: bool = True,
    exports: bool = True,
    comments: bool = False,
    docstrings: bool = False,
    symbols: bool = False,
    diagnostics: bool = False,
    chunk_max_size: int | None = None,
    extractions: dict | None = None,
)

# Static constructors:
ProcessConfig.all(language: str)      # All features enabled
ProcessConfig.minimal(language: str)  # All features disabled

Example:

config = ProcessConfig(
    "python",
    structure=True,
    imports=True,
    comments=True,
    chunk_max_size=1000,
)
result = process("import os\ndef foo(): pass", config)
print(result["structure"])    # List of functions/classes
print(result["imports"])       # List of imports
print(result["chunks"])        # Code chunks for LLMs

ProcessResult fields:

• language (str): Language used
• metrics (dict): {total_lines, code_lines, comment_lines, blank_lines, total_bytes, error_count}
• structure (list): Functions, classes, methods
• imports (list): Import statements
• exports (list): Exported symbols
• comments (list): Inline and block comments
• docstrings (list): Docstrings with parsed sections
• symbols (list): All identifiers
• diagnostics (list): Syntax errors, {message, severity, span}
• chunks (list): Code chunks, {content, start_line, end_line, metadata}

Extraction Queries

extract(source: str, config: dict) -> dict
# Returns {language, results: {pattern_name: {matches, total_count}}}

validate_extraction(config: dict) -> dict
# Returns {valid, patterns: {name: {valid, capture_names, pattern_count, errors, warnings}}}

Pattern config fields:

• query (str): Tree-sitter S-expression query
• capture_output (str): "Text", "Node", or "Full" (default)
• child_fields (list): Field names to extract
• max_results (int | None): Max matches to return
• byte_range ([int, int] | None): Restrict to byte range

Example:

result = extract("def hello(): pass\ndef world(): pass", {
    "language": "python",
    "patterns": {
        "functions": {
            "query": "(function_definition name: (identifier) @fn_name)",
            "capture_output": "Text",
        },
    },
})

for match in result["results"]["functions"]["matches"]:
    for capture in match["captures"]:
        print(capture["text"])  # "hello", "world"

Bundled Queries

get_highlights_query(language: str) -> str | None
get_injections_query(language: str) -> str | None
get_locals_query(language: str) -> str | None

Download & Configuration

init(config: dict) -> None
# {cache_dir?: str, languages?: list[str], groups?: list[str]}

configure(cache_dir: str | None = None) -> None
download(names: list[str]) -> int  # Returns count of newly downloaded
download_all() -> int
manifest_languages() -> list[str]
downloaded_languages() -> list[str]
clean_cache() -> None
cache_dir() -> str

Example:

# Pre-download languages
init({"languages": ["python", "javascript", "rust"]})

# Set custom cache directory
configure(cache_dir="/opt/parsers")

# Download on-demand
download(["python", "typescript"])

# Check what's cached
print(downloaded_languages())

tree-sitter Interop

get_binding(name: str) -> PyCapsule  # Raw TSLanguage pointer
get_language(name: str) -> tree_sitter.Language
get_parser(name: str) -> tree_sitter.Parser

Exceptions

• LanguageNotFoundError: Language not available
• ParseError: Parse or tree operation failed
• QueryError: Tree-sitter query syntax error
• DownloadError: Download or cache operation failed

Example:

from tree_sitter_language_pack import parse_string, ParseError

try:
    tree = parse_string("nonexistent_lang", "code")
except ParseError as e:
    print(f"Error: {e}")

Rust API Quick Reference

Installation

[dependencies]
tree-sitter-language-pack = "1"
# Default features include "download" for auto-downloading parsers
# Minimal (no download):
# tree-sitter-language-pack = { version = "1", default-features = false }

Language Discovery

available_languages() -> Vec<String>
has_language(name: &str) -> bool
language_count() -> usize
detect_language_from_extension(ext: &str) -> Option<&'static str>
detect_language_from_path(path: &str) -> Option<&'static str>
detect_language_from_content(content: &str) -> Option<&'static str>
extension_ambiguity(ext: &str) -> Option<(&'static str, &'static [&'static str])>

Parsing

parse_string(language: &str, source: &[u8]) -> Result<Tree, Error>

// Tree methods
tree.root_node().kind() -> &str
tree.root_node().child_count() -> usize
tree_contains_node_type(&tree, node_type: &str) -> bool
tree_has_error_nodes(&tree) -> bool
tree_error_count(&tree) -> usize
tree_to_sexp(&tree) -> String

// Node inspection
root_node_info(&tree) -> NodeInfo
find_nodes_by_type(&tree, node_type: &str) -> Vec<NodeInfo>
named_children_info(&tree) -> Vec<NodeInfo>

struct NodeInfo {
    pub kind: Cow<'static, str>,
    pub is_named: bool,
    pub start_byte: usize,
    pub end_byte: usize,
    pub start_row: usize,
    pub start_col: usize,
    pub end_row: usize,
    pub end_col: usize,
    pub named_child_count: usize,
    pub is_error: bool,
    pub is_missing: bool,
}

Example:

let tree = parse_string("python", b"x = 1")?;
assert_eq!(tree.root_node().kind(), "module");
assert!(!tree_has_error_nodes(&tree));

Code Intelligence Processing

process(source: &str, config: &ProcessConfig) -> Result<ProcessResult, Error>

pub struct ProcessConfig {
    pub language: Cow<'static, str>,
    pub structure: bool,          // default: true
    pub imports: bool,            // default: true
    pub exports: bool,            // default: true
    pub comments: bool,           // default: false
    pub docstrings: bool,         // default: false
    pub symbols: bool,            // default: false
    pub diagnostics: bool,        // default: false
    pub chunk_max_size: Option<usize>,  // default: None
    pub extractions: Option<AHashMap<String, ExtractionPattern>>,
}

// Constructors
ProcessConfig::new(language: impl Into<String>) -> Self
config.with_chunking(max_size: usize) -> Self
config.all() -> Self
config.minimal() -> Self

pub struct ProcessResult {
    pub language: String,
    pub metrics: FileMetrics,
    pub structure: Vec<StructureItem>,
    pub imports: Vec<ImportInfo>,
    pub exports: Vec<ExportInfo>,
    pub comments: Vec<CommentInfo>,
    pub docstrings: Vec<DocstringInfo>,
    pub symbols: Vec<SymbolInfo>,
    pub diagnostics: Vec<Diagnostic>,
    pub chunks: Vec<CodeChunk>,
    pub parse_errors: usize,
}

Example:

let config = ProcessConfig::new("python")
    .all()
    .with_chunking(2000);

let result = process("def hello(): pass", &config)?;
println!("Functions: {}", result.structure.len());
println!("Lines: {}", result.metrics.total_lines);

Extraction Queries

extract_patterns(source: &str, config: &ExtractionConfig) -> Result<ExtractionResult, Error>
validate_extraction(config: &ExtractionConfig) -> Result<ValidationResult, Error>

pub struct ExtractionConfig {
    pub language: String,
    pub patterns: AHashMap<String, ExtractionPattern>,
}

pub struct ExtractionPattern {
    pub query: String,
    pub capture_output: CaptureOutput,
    pub child_fields: Vec<String>,
    pub max_results: Option<usize>,
    pub byte_range: Option<(usize, usize)>,
}

pub enum CaptureOutput {
    Text,    // Text only
    Node,    // NodeInfo only
    Full,    // Both text and NodeInfo (default)
}

pub struct ExtractionResult {
    pub language: String,
    pub results: AHashMap<String, PatternResult>,
}

pub struct PatternResult {
    pub matches: Vec<MatchResult>,
    pub total_count: usize,
}

Example:

use ahash::AHashMap;

let mut patterns = AHashMap::new();
patterns.insert("functions".to_string(), ExtractionPattern {
    query: "(function_definition name: (identifier) @fn_name)".to_string(),
    capture_output: CaptureOutput::Full,
    child_fields: vec!["name".to_string()],
    max_results: None,
    byte_range: None,
});

let config = ExtractionConfig {
    language: "python".to_string(),
    patterns,
};

let result = extract_patterns("def hello(): pass", &config)?;
assert_eq!(result.results["functions"].total_count, 1);

Compiled Extraction (Reusable)

pub struct CompiledExtraction { /* opaque */ }

CompiledExtraction::compile(config: &ExtractionConfig) -> Result<Self, Error>
CompiledExtraction::compile_with_language(
    language: Language,
    language_name: &str,
    patterns: &AHashMap<String, ExtractionPattern>
) -> Result<Self, Error>

compiled.extract(&self, source: &str) -> Result<ExtractionResult, Error>
compiled.extract_from_tree(&self, tree: &Tree, source: &[u8]) -> Result<ExtractionResult, Error>

Bundled Queries

get_highlights_query(language: &str) -> Option<&'static str>
get_injections_query(language: &str) -> Option<&'static str>
get_locals_query(language: &str) -> Option<&'static str>

Language Pointers (Interop)

get_language(name: &str) -> Result<Language, Error>
get_parser(name: &str) -> Result<Parser, Error>

These re-export tree_sitter::{Language, Parser, Tree} for interop with the upstream tree-sitter crate.

Download & Configuration (requires "download" feature)

pub struct PackConfig {
    pub cache_dir: Option<PathBuf>,
    pub languages: Option<Vec<String>>,
    pub groups: Option<Vec<String>>,
}

init(config: &PackConfig) -> Result<(), Error>
configure(config: &PackConfig) -> Result<(), Error>
download(names: &[&str]) -> Result<usize, Error>
download_all() -> Result<usize, Error>
manifest_languages() -> Result<Vec<String>, Error>
downloaded_languages() -> Vec<String>
clean_cache() -> Result<(), Error>
cache_dir() -> Result<PathBuf, Error>

Example:

let config = PackConfig {
    cache_dir: Some(PathBuf::from("/opt/ts-pack")),
    languages: Some(vec!["python".to_string(), "rust".to_string()]),
    groups: None,
};

init(&config)?;
let count = download(&["python", "typescript"])?;
println!("Downloaded {} new languages", count);

Error Handling

pub enum Error {
    LanguageNotFound(String),
    DynamicLoad(String),
    NullLanguagePointer(String),
    ParserSetup(String),
    LockPoisoned(String),
    Config(String),
    ParseFailed,
    QueryError(String),
    InvalidRange(String),
    Io(std::io::Error),
    Json(serde_json::Error),           // with "config" or "download"
    Toml(toml::de::Error),             // with "config"
    Download(String),                   // with "download"
    ChecksumMismatch { file, expected, actual },  // with "download"
}

Match specific error types:

match get_language("python") {
    Ok(lang) => println!("Got Python"),
    Err(Error::LanguageNotFound(name)) => println!("Not found: {}", name),
    Err(e) => println!("Error: {:?}", e),
}

Common Patterns

Parse and Query

use tree_sitter_language_pack::{parse_string, run_query};

let tree = parse_string("python", b"def hello(): pass")?;
let matches = run_query(
    &tree,
    "python",
    "(function_definition name: (identifier) @fn_name)",
    b"def hello(): pass",
)?;

Chunked Processing

let config = ProcessConfig::new("python")
    .with_chunking(2000)
    .structure(true);

let result = process(source, &config)?;
for chunk in &result.chunks {
    println!("Chunk: lines {}-{}", chunk.start_line, chunk.end_line);
}

Feature Flags

Crate features:

• download (default): Auto-download and cache parsers
• serde: Serialization for ProcessConfig/ProcessResult
• config: Load PackConfig from TOML files

Minimal build (no download):

tree-sitter-language-pack = { version = "1", default-features = false }

TypeScript/Node.js API Quick Reference

Installation

npm install @kreuzberg/tree-sitter-language-pack
pnpm add @kreuzberg/tree-sitter-language-pack
yarn add @kreuzberg/tree-sitter-language-pack

All functions are synchronous. Pre-built binaries for macOS (arm64), Linux (x64, arm64), Windows (x64). Requires Node.js >= 16.

Language Discovery

availableLanguages(): string[]
hasLanguage(name: string): boolean
languageCount(): number
detectLanguage(path: string): string | null
detectLanguageFromContent(content: string): string | null
detectLanguageFromExtension(ext: string): string | null
detectLanguageFromPath(path: string): string | null
extensionAmbiguity(ext: string): AmbiguityResult | null
// AmbiguityResult: {assigned: string, alternatives: string[]}

Parsing

parseString(language: string, source: string): ExternalObject<Tree>
// Returns opaque tree handle

treeRootNodeType(tree: Tree): string
treeRootChildCount(tree: Tree): number
treeContainsNodeType(tree: Tree, nodeType: string): boolean
treeHasErrorNodes(tree: Tree): boolean

Example:

const tree = parseString("python", "x = 1");
console.log(treeRootNodeType(tree)); // "module"
console.log(treeHasErrorNodes(tree)); // false

Code Intelligence Processing

process(source: string, config: JsProcessConfig): ProcessResult

interface JsProcessConfig {
  language: string;
  structure?: boolean;      // default: true
  imports?: boolean;        // default: true
  exports?: boolean;        // default: true
  comments?: boolean;       // default: false
  docstrings?: boolean;     // default: false
  symbols?: boolean;        // default: false
  diagnostics?: boolean;    // default: false
  chunkMaxSize?: number;    // optional, in bytes
  extractions?: Record<string, PatternConfig>;
}

interface ProcessResult {
  language: string;
  metrics: FileMetrics;
  structure: StructureItem[];
  imports: ImportInfo[];
  exports: ExportInfo[];
  comments: CommentInfo[];
  docstrings: DocstringInfo[];
  symbols: SymbolInfo[];
  diagnostics: Diagnostic[];
  chunks: CodeChunk[];
}

Example:

const result = process("def hello(): pass", { language: "python" });
console.log(result.structure); // Functions, classes
console.log(result.imports); // Import statements
console.log(result.chunks); // Code chunks for LLMs

Supporting types:

interface FileMetrics {
  totalLines: number;
  totalBytes: number;
  blankLines: number;
  commentLines: number;
  codeLines: number;
  errorCount: number;
}

interface Span {
  startByte: number;
  endByte: number;
  startRow: number;
  startCol: number;
  endRow: number;
  endCol: number;
}

interface StructureItem {
  kind: string; // "function" | "class" | "method" | ...
  name: string;
  span: Span;
  parent: string | null;
}

interface ImportInfo {
  module: string;
  names: string[];
  span: Span;
}

interface ExportInfo {
  name: string;
  kind: string;
  span: Span;
}

interface CommentInfo {
  text: string;
  kind: string;
  span: Span;
  associatedNode: string | null;
}

interface DocstringInfo {
  text: string;
  format: string;
  span: Span;
  associatedItem: string | null;
  sections: Array<Record<string, string>>;
}

interface SymbolInfo {
  name: string;
  kind: string;
  span: Span;
  typeAnnotation: string | null;
}

interface Diagnostic {
  message: string;
  severity: string;
  span: Span;
}

interface CodeChunk {
  content: string;
  startByte: number;
  endByte: number;
  metadata: ChunkContext;
}

interface ChunkContext {
  language: string;
  chunkIndex: number;
  totalChunks: number;
  startLine: number;
  endLine: number;
  nodeTypes: string[];
  symbolsDefined: string[];
  comments: string[];
  docstrings: string[];
  hasErrorNodes: boolean;
  contextPath: string[];
}

Extraction Queries

extract(source: string, config: object): ExtractionResult
validateExtraction(config: object): ValidationResult

interface ExtractionConfig {
  language: string;
  patterns: Record<string, PatternConfig>;
}

interface PatternConfig {
  query: string;
  captureOutput?: "Text" | "Node" | "Full";  // default: "Full"
  childFields?: string[];
  maxResults?: number;
  byteRange?: [number, number];
}

interface ExtractionResult {
  language: string;
  results: Record<string, PatternResult>;
}

interface PatternResult {
  matches: MatchResult[];
  totalCount: number;
}

interface MatchResult {
  patternIndex: number;
  captures: CaptureResult[];
}

interface CaptureResult {
  name: string;
  node: NodeInfo | null;
  text: string | null;
  childFields: Record<string, string | null>;
  startByte: number;
}

interface ValidationResult {
  valid: boolean;
  patterns: Record<string, PatternValidation>;
}

interface PatternValidation {
  valid: boolean;
  captureNames: string[];
  patternCount: number;
  warnings: string[];
  errors: string[];
}

Example:

const result = extract("def hello(): pass", {
  language: "python",
  patterns: {
    functions: {
      query: "(function_definition name: (identifier) @fn_name)",
      captureOutput: "Text",
    },
  },
});

for (const match of result.results.functions.matches) {
  for (const capture of match.captures) {
    console.log(capture.text); // "hello"
  }
}

Bundled Queries

getHighlightsQuery(language: string): string | null
getInjectionsQuery(language: string): string | null
getLocalsQuery(language: string): string | null

Download & Configuration

init(config?: JsPackConfig): void
configure(config: JsPackConfig): void
download(names: string[]): number
downloadAll(): number
manifestLanguages(): string[]
downloadedLanguages(): string[]
cleanCache(): void
cacheDir(): string

interface JsPackConfig {
  cacheDir?: string;
  languages?: string[];
  groups?: string[];
}

Example:

import { init, download, cacheDir } from "@kreuzberg/tree-sitter-language-pack";

// Pre-download languages
init({ languages: ["python", "javascript", "rust"] });

// Download on-demand
download(["python", "typescript"]);

// Check cache
console.log(cacheDir());

Low-Level Interop

getLanguagePtr(name: string): number
// Returns raw TSLanguage pointer for use with node-tree-sitter

Error Handling

All errors throw standard JavaScript Error objects with descriptive messages. No custom exception types.

try {
  parseString("nonexistent_language", "code");
} catch (error) {
  console.error(error.message);
}

Common Patterns

Detect and Analyze

import { detectLanguage, process } from "@kreuzberg/tree-sitter-language-pack";
import { readFileSync } from "fs";

const lang = detectLanguage("src/main.py");
if (lang) {
  const source = readFileSync("src/main.py", "utf-8");
  const result = process(source, { language: lang });
  console.log(`Functions: ${result.structure.length}`);
}

Batch Processing

import { detectLanguage, process } from "@kreuzberg/tree-sitter-language-pack";
import { readdirSync, readFileSync } from "fs";

const files = ["app.py", "lib.rs", "index.ts"];
for (const file of files) {
  const lang = detectLanguage(file);
  if (!lang) continue;

  const source = readFileSync(file, "utf-8");
  const result = process(source, { language: lang });
  console.log(`${file}: ${result.structure.length} items`);
}