Token Counter

Name: Token Counter
Author: girijashankarj

girijashankarj/cursor-handbook

1 installs
27 repo stars
Updated April 25, 2026
girijashankarj/cursor-handbook

Estimates LLM token counts for files or directories using a word-based heuristic to help manage context budgets and token cost.

About

Estimates token consumption for files and directories to help manage LLM context budgets. A developer uses it to check context size or estimate token cost.

Runs a bundled script using wc as a word-based heuristic
Works on a single file or a directory

Token Counter by the numbers

1 all-time installs (skills.sh)
Ranked #14,102 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
Data as of Jul 22, 2026 (Skillselion catalog sync)

npx skills add https://github.com/girijashankarj/cursor-handbook --skill token-counter

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/girijashankarj/cursor-handbook/token-counter.svg)](https://skillselion.com/skills/girijashankarj/cursor-handbook/token-counter)

Installs	1
repo stars	★ 27
Last updated	April 25, 2026
Repository	girijashankarj/cursor-handbook ↗

What it does

Estimates LLM token counts for files or directories using a word-based heuristic to help manage context budgets and token cost.

Files

SKILL.mdMarkdownGitHub ↗

Skill: Token Counter

Estimate token consumption for files and directories to help manage LLM context budgets.

Trigger

When the user asks to count tokens, estimate context size, check how much a file/directory costs in tokens, or wants to optimize token usage.

Prerequisites

[ ] Target file or directory exists
[ ] wc command available (standard on macOS/Linux)

Usage

Run the bundled script:

scripts/count-tokens.sh <file-or-directory>

Examples

Count tokens for a single file:

scripts/count-tokens.sh .cursor/rules/main-rules.mdc

Count tokens for an entire directory:

scripts/count-tokens.sh .cursor/rules/

Count tokens for the whole project:

scripts/count-tokens.sh .

Steps

Step 1: Identify Target

[ ] Determine what the user wants to measure (single file, directory, or set of files)
[ ] Verify the target path exists

Step 2: Run Token Count

[ ] Execute scripts/count-tokens.sh <target> using the Shell tool
[ ] Capture output

Step 3: Interpret Results

[ ] Present the token estimates in a clear table or list
[ ] Flag any files that exceed context layer budgets:
Immediate context: ~2000 tokens
Relevant context: ~3000 tokens
Extended context: ~3000 tokens
[ ] Suggest optimizations if totals are high (split files, trim comments, use references)

Step 4: Recommendations

[ ] If a single file exceeds 2000 tokens, suggest splitting or using progressive loading
[ ] If a directory exceeds 10,000 tokens, recommend selective inclusion
[ ] Reference the token efficiency rules for budget guidance

How It Works

The script uses a simple heuristic: word_count × 1.3 ≈ token_count. This approximation works well for English text and code. For precise counts, use tiktoken or the OpenAI tokenizer.

Supported file types: .md, .mdc, .ts, .tsx, .py

Rules

The heuristic is an approximation — actual token counts vary by model and tokenizer
For precise counts, recommend tiktoken (Python) or equivalent
Use this for quick estimates and budget planning, not exact billing

Completion

Token estimates displayed with optimization suggestions if any files exceed recommended budgets.

If a Step Fails

File not found: Verify the path and suggest alternatives
Empty output: Check if the directory contains supported file types
Inaccurate results: Suggest installing tiktoken for precise counts

#!/bin/bash
# Estimate token count for a file or directory (rough approximation for LLM context).
# Uses wc -w * 1.3 as a simple heuristic; for accurate counts use tiktoken or similar.

set -e

TARGET="${1:-.}"

count_tokens() {
  local file="$1"
  if [ ! -f "$file" ]; then return; fi
  local words
  words=$(wc -w < "$file" 2>/dev/null || echo 0)
  echo "$file: ~$(( words * 13 / 10 )) tokens (approx)"
}

if [ -f "$TARGET" ]; then
  count_tokens "$TARGET"
elif [ -d "$TARGET" ]; then
  find "$TARGET" -type f \( -name "*.md" -o -name "*.mdc" -o -name "*.ts" -o -name "*.tsx" -o -name "*.py" \) \
    | head -50 \
    | while read -r f; do count_tokens "$f"; done
else
  echo "Usage: $0 <file-or-directory>"
  exit 1
fi