Research Engineer Scientist Tokens

Research Engineer / Research Scientist, Tokens

When to Use

• Frame research questions on tokens, context length, or inference cost
• Design experiments with baselines, ablations, and statistical rigor
• Build benchmarks for tokens-per-successful-task, effective context, cache leverage
• Measure tokenizer and formatting effects on length and model behavior
• Evaluate compression, summarization, routing, or distillation for token savings
• Analyze long-context phenomena (needle, lost-in-middle, attention budget)
• Write research memos with reproducible methods and honest limitations
• Translate findings into actionable thresholds for engineering and product

When NOT to Use

• Executive token reduction program with phased rollout → ai-token-improvement-plan-engineer
• Implement context packing, compaction code paths → ai-context-engineer
• Rewrite one production prompt → prompt-engineer
• General literature survey unrelated to tokens → ai-researcher
• Production RAG/agent deployment → ai-engineer
• Classical ML without LLM token focus → data-scientist

Related skills

Core Workflows

1. Research framing (tokens)

Hypothesis, metrics, baselines, budget.

See `references/research_framing_tokens.md`.

2. Measurement and instrumentation

Token accounting, logging, fair comparison.

See `references/measurement_instrumentation.md`.

3. Experiment design and ablations

Controls, sweeps, power, stopping rules.

See `references/experiment_design_ablations.md`.

4. Context, tokenization, and long-context

Tokenizer, placement, window effects.

See `references/context_tokenization_longcontext.md`.

5. Compression and efficiency methods

Summarization, routing, distillation research.

See `references/compression_efficiency_methods.md`.

6. Reproducibility and research reporting

Memos, artifacts, handoff to engineering.

See `references/reproducibility_reporting.md`.

Outputs

• Pre-registration / experiment plan — hypothesis, metrics, stop criteria
• Results table — mean ± CI; tokens and quality side by side
• Pareto chart narrative — quality vs tokens at operating points
• Ablation appendix — what mattered, what did not
• Research memo — conclusion, limits, recommended next build
• Artifact bundle — configs, seeds, eval scripts, hashed datasets

Principles

• Report tokens and quality together — never optimize one without the other
• Match tokenizer and model — counts from the deployment tokenizer/API
• Control confounds — temperature, system prompt, tool schemas held fixed across arms
• Pre-register primary metric — avoid p-hacking across slice metrics
• Separate science from rollout — research recommends; ai-token-improvement-plan-engineer owns program

Compression and efficiency methods

Table of contents

1. Method families 2. Evaluation protocol 3. Failure modes to track 4. Routing and cascades

Method families

Tag studies as input-token vs output-token vs latency primary.

Evaluation protocol

For each method:

1. Apply at fixed operating point (e.g., 50% input reduction target) 2. Run full eval suite + stress suite (long tools, multi-turn, non-English) 3. Compare tokens-to-success and primary quality 4. Cost human review on stratified failures only

Report Pareto frontier — not single point cherry-pick.

Failure modes to track

Bucket failures for qualitative appendix.

Routing and cascades

Research design:

• Define router features (length, intent class, entropy)
• Train/ calibrate on held-out set
• Measure % traffic to cheap model vs regression rate
• Account router overhead tokens in total

Compare to always large and always small baselines.

Hand off winning thresholds to ai-token-improvement-plan-engineer for rollout planning.

Context, tokenization, and long-context

Table of contents

1. Tokenization effects 2. Context assembly 3. Long-context phenomena 4. Study designs

Tokenization effects

Research angles:

• Cross-model token inflation — same Unicode, different token counts
• Delimiter and markup — JSON vs XML vs markdown overhead
• Code vs prose — compression ratios by language
• Rare characters and emoji — fragmentation spikes
• Tool schema size — function-calling token tax

Report bytes → tokens curves for representative corpora.

Context assembly

Decompose input budget:

Measure marginal value — ablate one block at a time.

Long-context phenomena

Standard probes (adapt to product domain):

• Needle-in-haystack — fact at depth d
• Multi-hop — requires two distant facts
• Lost in the middle — U-shaped accuracy vs position
• Instruction at top vs bottom — constraint following

Plot accuracy vs insertion depth and vs total tokens.

Study designs

• Fixed total window; sweep needle depth
• Fixed task; sweep window size with constant density
• Compare yarn/extended models only with matched eval — avoid conflating model upgrade with context strategy

Link findings to ai-context-engineer implementation choices, not as production spec by themselves.

Experiment design and ablations

Table of contents

1. Baseline ladder 2. Ablation matrix 3. Sweeps and power 4. Stopping and preregistration

Baseline ladder

Always include:

1. Current production — realistic stack 2. Simple strong — full context, best model (upper bound quality) 3. Naive cheap — smallest model / truncate tail (lower bound) 4. Proposed — method under test

Avoid comparing only to a weak baseline.

Ablation matrix

Example for context compression study:

Change one family per ablation wave; combine only after singles validated.

Sweeps and power

• Grid: context cap {4k, 8k, 16k, 32k}
• Grid: compression ratio {0.25, 0.5, 0.75}
• Use paired comparisons on same queries when possible
• Estimate CI via bootstrap over eval items
• Minimum n: justify from pilot variance (e.g., 200+ tasks for 2% pass rate shifts)

Stopping and preregistration

Pre-register:

• Primary metric and direction
• Minimum effect size of interest (δ)
• Max spend (GPU hours or $)
• Early stop rule (only if planned; adjust for multiple testing)

Do not stop at best-looking interim slice without correction.

Record negative results — methods that saved tokens but broke quality.

Measurement and instrumentation

Table of contents

1. Token counting 2. Logging schema 3. Fair comparison 4. Cost translation

Token counting

Log input, output, cached, and reasoning tokens separately if provider splits them.

Document system + tools + retrieval as included in input — not just user message.

Logging schema

Minimum per request:

run_id, model, tokenizer_id, timestamp
input_tokens, output_tokens, cached_tokens
feature_id, experiment_arm, prompt_hash
latency_ms, finish_reason
quality_label (pass/fail/score)
retrieval_tokens (if RAG)
compaction_applied (bool), pre_post_token_counts

Store prompt template version and retrieval chunk IDs for replay.

Fair comparison

• Same eval queries and order (or stratified random seed)
• Same max output cap unless studying stop behavior
• Same tool definitions across arms (tool schemas dominate tokens)
• Warmup discarded for latency; not for token totals
• Report distribution (p50, p90), not mean only

Watch hidden tokens: image patches, audio frames, repeated tool results.

Cost translation

Build price table by model tier:

cost = in_tokens * rate_in + out_tokens * rate_out + cached * rate_cached

Sensitivity analysis when list prices change mid-study.

Separate research GPU cost from production inference cost in memos.

Reproducibility and research reporting

Table of contents

1. Artifact checklist 2. Memo structure 3. Handoff to engineering 4. Publication hygiene

Artifact checklist

• [ ] Git commit or tag for code
• [ ] Config YAML with all arms
• [ ] Eval set version hash
• [ ] Model names and API versions
• [ ] Random seeds listed
• [ ] Raw per-item results CSV
• [ ] Aggregated tables + plots scripts
• [ ] README: how to rerun one command

Memo structure

1. Question and hypothesis 2. Setup — models, data, metrics (primary pre-registered) 3. Results — tables, CIs, Pareto 4. Ablations — what drove effect 5. Limitations — data bias, eval gaps, short runs 6. Recommendation — ship / iterate / abandon with thresholds 7. Appendix — prompts (redacted if needed), failure examples

One-page executive summary with tokens saved at fixed quality ε.

Handoff to engineering

Include guardrail evals required before prod flag.

Publication hygiene

• Cite prior art and baselines fairly
• Do not overclaim from synthetic evals only
• Separate internal vs external communicable results
• No customer data in public artifacts
• Note assisted-by AI if company policy requires on internal memos

Negative results worth archiving to prevent duplicate work.

Research framing (tokens)

Table of contents

1. Question types 2. Hypothesis templates 3. Primary metrics 4. Scope checklist

Question types

Hypothesis templates

• H1: Under fixed eval set E, strategy S reduces median tokens-to-success by ≥δ vs baseline B.
• H2: Quality metric Q is non-inferior (ε margin) while input tokens decrease ≥δ.
• H3: Failure mode F (e.g., missed needle) increases when compressed context exceeds k tokens.

State falsification criteria up front.

Primary metrics

Pick one primary; others secondary:

Always pair with task quality: pass@1, win rate, rubric score, human preference.

Scope checklist

• [ ] Model(s) and snapshot dates fixed
• [ ] Tokenizer/API version documented
• [ ] Eval set frozen and versioned
• [ ] Compute budget cap stated
• [ ] Excluded: safety regressions unless in scope
• [ ] Human eval budget if claiming subjective quality