Prompt Engineer Toolkit

Prompt Engineer Toolkit

Overview

Use this skill to move prompts from ad-hoc drafts to production assets with repeatable testing, versioning, and regression safety. It emphasizes measurable quality over intuition. Apply it when launching a new LLM feature that needs reliable outputs, when prompt quality degrades after model or instruction changes, when multiple team members edit prompts and need history/diffs, when you need evidence-based prompt choice for production rollout, or when you want consistent prompt governance across environments.

Core Capabilities

• A/B prompt evaluation against structured test cases
• Quantitative scoring for adherence, relevance, and safety checks
• Prompt version tracking with immutable history and changelog
• Prompt diffs to review behavior-impacting edits
• Reusable prompt templates and selection guidance
• Regression-friendly workflows for model/prompt updates

Key Workflows

1. Run Prompt A/B Test

Prepare JSON test cases and run:

python3 scripts/prompt_tester.py \
  --prompt-a-file prompts/a.txt \
  --prompt-b-file prompts/b.txt \
  --cases-file testcases.json \
  --runner-cmd 'my-llm-cli --prompt {prompt} --input {input}' \
  --format text

Input can also come from stdin/--input JSON payload.

2. Choose Winner With Evidence

The tester scores outputs per case and aggregates:

• expected content coverage
• forbidden content violations
• regex/format compliance
• output length sanity

Use the higher-scoring prompt as candidate baseline, then run regression suite.

3. Version Prompts

# Add version
python3 scripts/prompt_versioner.py add \
  --name support_classifier \
  --prompt-file prompts/support_v3.txt \
  --author alice

# Diff versions
python3 scripts/prompt_versioner.py diff --name support_classifier --from-version 2 --to-version 3

# Changelog
python3 scripts/prompt_versioner.py changelog --name support_classifier

4. Regression Loop

1. Store baseline version. 2. Propose prompt edits. 3. Re-run A/B test. 4. Promote only if score and safety constraints improve.

Script Interfaces

• python3 scripts/prompt_tester.py --help
• Reads prompts/cases from stdin or --input
• Optional external runner command
• Emits text or JSON metrics
• python3 scripts/prompt_versioner.py --help
• Manages prompt history (add, list, diff, changelog)
• Stores metadata and content snapshots locally

Pitfalls, Best Practices & Review Checklist

Avoid these mistakes: 1. Picking prompts from single-case outputs — use a realistic, edge-case-rich test suite. 2. Changing prompt and model simultaneously — always isolate variables. 3. Missing must_not_contain (forbidden-content) checks in evaluation criteria. 4. Editing prompts without version metadata, author, or change rationale. 5. Skipping semantic diffs before deploying a new prompt version. 6. Optimizing one benchmark while harming edge cases — track the full suite. 7. Model swap without rerunning the baseline A/B suite.

Before promoting any prompt, confirm:

• [ ] Task intent is explicit and unambiguous.
• [ ] Output schema/format is explicit.
• [ ] Safety and exclusion constraints are explicit.
• [ ] No contradictory instructions.
• [ ] No unnecessary verbosity tokens.
• [ ] A/B score improves and violation count stays at zero.

References

• references/prompt-templates.md — 6 production marketing templates (ad copy, email sequence, social repurposing, landing sections, SEO meta, brand-voice rewrite) plus generic building blocks; each written to be graded by prompt_tester.py
• references/technique-guide.md — technique-selection table for marketing tasks + the LLM-governance stack for marketing teams (claim discipline, disclosure rules, data boundaries, human-review gates)
• references/evaluation-rubric.md — mechanical scoring weights, acceptance gates, marketing quality dimensions, test-suite design, and eval anti-patterns
• README.md

Evaluation Design

Each test case should define:

• input: realistic production-like input
• expected_contains: required markers/content
• forbidden_contains: disallowed phrases or unsafe content
• expected_regex: required structural patterns

This enables deterministic grading across prompt variants.

Versioning Policy

• Use semantic prompt identifiers per feature (support_classifier, ad_copy_shortform).
• Record author + change note for every revision.
• Never overwrite historical versions.
• Diff before promoting a new prompt to production.

Rollout Strategy

1. Create baseline prompt version. 2. Propose candidate prompt. 3. Run A/B suite against same cases. 4. Promote only if winner improves average and keeps violation count at zero. 5. Track post-release feedback and feed new failure cases back into test suite.

Prompt Engineer Toolkit

Production toolkit for evaluating and versioning prompts with measurable quality signals. Includes A/B testing automation and prompt history management with diffs.

Quick Start

# Run A/B prompt evaluation
python3 scripts/prompt_tester.py \
  --prompt-a-file prompts/a.txt \
  --prompt-b-file prompts/b.txt \
  --cases-file testcases.json \
  --format text

# Store a prompt version
python3 scripts/prompt_versioner.py add \
  --name support_classifier \
  --prompt-file prompts/a.txt \
  --author team

Included Tools

• scripts/prompt_tester.py: A/B testing with per-case scoring and aggregate winner
• scripts/prompt_versioner.py: prompt history (add, list, diff, changelog) in local JSONL store

References

• references/prompt-templates.md
• references/technique-guide.md
• references/evaluation-rubric.md

Installation

Claude Code

cp -R marketing-skill/prompt-engineer-toolkit ~/.claude/skills/prompt-engineer-toolkit

OpenAI Codex

cp -R marketing-skill/prompt-engineer-toolkit ~/.codex/skills/prompt-engineer-toolkit

OpenClaw

cp -R marketing-skill/prompt-engineer-toolkit ~/.openclaw/skills/prompt-engineer-toolkit

Evaluation Rubric for Marketing Prompts

How to score prompt outputs deterministically with scripts/prompt_tester.py, and how to extend the mechanical score with marketing-specific quality dimensions a regex can't fully capture. The principle throughout: evidence over intuition — a prompt is "better" only if it scores better on a realistic, edge-case-rich suite (never a single cherry-picked output).

Layer 1 — Mechanical Score (what prompt_tester.py computes)

Score each test case 0-100 via weighted criteria:

Acceptance gates (promote a prompt only if all hold):

• Average score ≥ 85 across the suite
• No individual case below 70
• Zero critical forbidden-content hits (brand-banned words, invented statistics markers, competitor names where disallowed, compliance terms — see governance guide)

Layer 2 — Marketing Quality Dimensions

Encode as many of these as possible into Layer-1 fields; what remains needs human review on a sample (5-10 outputs per variant):

Scoring scale for human review (per Hamel Husain's eval guidance, keep it binary where possible): pass/fail per dimension beats 1-5 ratings — raters agree more, and failures become new forbidden_contains/expected_regex entries, ratcheting the mechanical suite forward.

Building the Test Suite

A marketing prompt suite needs at minimum:

1. Happy-path cases (3-5) — typical inputs with complete variables 2. Sparse-input cases (2-3) — missing proof points, vague audience: the prompt must degrade safely (omit proof, ask, or flag) rather than fabricate 3. Adversarial cases (2-3) — inputs that bait policy violations: competitor disparagement requests, unverifiable claims supplied as "facts", off-brand tone requests 4. Edge-format cases (1-2) — very long inputs, non-English fragments, emoji-laden source content

Failure analysis loop: every production failure (rejected ad, spam-flagged email, off-brand post) becomes a new test case before the prompt is edited — the marketing equivalent of regression-test-first.

Anti-Patterns

• Single-output judgment — comparing one generation per prompt; sampling variance swamps prompt differences. Run every case ≥ 3 times or compare suite averages.
• LLM-as-judge without calibration — if you add a model-graded criterion, calibrate it against human labels on 20+ examples first and re-check periodically (judges drift with model versions).
• Score-only promotion — a +2 average that introduces one compliance violation is a regression, not an improvement. Violations gate, scores rank.
• Frozen suite — a suite that never grows stops catching new failure modes; tie suite growth to the failure-analysis loop above.

---

Citations (6 sources)

1. Anthropic — "Define your success criteria" + "Create strong empirical evaluations" (docs.anthropic.com/en/docs/build-with-claude/define-success, /develop-tests): measurable criteria and graded test suites before prompt iteration 2. OpenAI Evals — open-source eval framework and registry patterns for templated, deterministic graders (github.com/openai/evals) 3. Hamel Husain — "Your AI Product Needs Evals" (hamel.dev/blog/posts/evals): unit-test-style assertions, failure-driven suite growth, binary human labels 4. Zheng et al. — "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023): LLM-judge agreement rates and bias modes (position, verbosity) 5. Eugene Yan — "Patterns for Building LLM-based Systems & Products" (eugeneyan.com): eval-first development, guardrails as gates vs. scores as ranks 6. Liu et al. — "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" (EMNLP 2023): criteria-decomposed grading for content quality dimensions

Marketing Prompt Templates

Production-ready prompt templates for the marketing use cases this skill promises: ad copy, email campaigns, social media, landing pages, and SEO metadata. Each template is written to be testable with scripts/prompt_tester.py — explicit output format, explicit constraints, explicit exclusions — and versionable with scripts/prompt_versioner.py under the semantic name given.

Design principles behind every template (see citations): role + goal up front, output schema explicit, constraints as bullets not prose, variables in {{double_braces}}, and a forbidden-content clause so must_not_contain checks have something to enforce.

---

1) Ad Copy Variants — ad_copy_shortform

You are a direct-response copywriter for {{brand}} ({{one_line_positioning}}).

Write {{count}} ad copy variants for {{platform}} promoting {{offer}}.

Audience: {{audience}} — their #1 pain: {{pain_point}}.
Voice: {{voice_adjectives}}. Reading level: 7th grade.

Hard constraints:
- Headline ≤ {{headline_limit}} characters; primary text ≤ {{body_limit}} characters
- Each variant uses a DIFFERENT angle: pain-led, outcome-led, proof-led, curiosity-led
- One specific, verifiable claim per variant ({{proof_points}}); never invent statistics
- No exclamation-point stacking, no "🚀", no "game-changing/revolutionary/unleash"

Return JSON array: [{"angle":"...","headline":"...","primary_text":"...","cta":"..."}]

Test cases should assert character limits via expected_regex and ban the cliché list via forbidden_contains.

2) Email Campaign Sequence — email_campaign_writer

You are a lifecycle email marketer for {{brand}}.

Write email {{n}} of {{total}} in a {{sequence_type}} sequence (goal: {{conversion_goal}}).
Reader context: {{what_they_did}} — they have NOT yet {{what_they_havent_done}}.

Constraints:
- Subject line ≤ 45 chars + preview text ≤ 90 chars; no spam-trigger words (free!!!, act now, limited time)
- Body 90-150 words, one idea, one CTA ({{cta_text}} → {{cta_url}})
- Plain-text tone — write like a competent colleague, not a brand
- Reference the reader's situation in sentence 1; never open with "I hope this finds you well"

Return:
SUBJECT: ...
PREVIEW: ...
BODY:
...
CTA: ...

3) Social Media Post Set — social_post_repurposer

You are a social content editor. Repurpose the source content into {{count}} platform-native posts.

Platforms: {{platforms}}.
Source:
{{source_content}}

Per-platform rules:
- X: ≤ 280 chars, hook in first 8 words, max 1 hashtag, placed at the end
- LinkedIn: ≤ 1300 chars, line breaks every 1-2 sentences, no engagement-bait ("Agree?")
- Instagram: caption ≤ 150 words + 5 relevant hashtags at the end

Every post must contain one specific detail (number, name, example) from the source.
Return JSON: [{"platform":"...","post":"...","specific_detail_used":"..."}]

4) Landing Page Section Copy — landing_section_writer

You are a conversion copywriter. Write the {{section}} section for a landing page.

Product: {{product}} — for {{audience}} who want {{outcome}}.
Differentiator: {{differentiator}}. Proof available: {{proof_points}}.

Constraints:
- Headline: specific outcome, ≤ 12 words, no category jargon
- Body: benefit-first, "you" language, ≤ 60 words
- Use ONLY the proof points provided; if none fit, omit proof rather than invent it
- CTA button: verb + value ("Get my report"), never "Submit"/"Learn more"

Return markdown with HEADLINE / BODY / CTA blocks.

5) SEO Title + Meta Description — seo_meta_writer

You are an SEO editor. Write title tag + meta description for the page below.

Primary keyword: {{keyword}} (must appear in title, near the front, naturally).
Search intent: {{intent}}. Page summary: {{summary}}.

Constraints:
- Title ≤ 60 characters, no clickbait, no ALL CAPS, brand suffix " | {{brand}}" if it fits
- Meta description 150-160 characters, includes keyword once, ends with a reason to click
- Describe what the page actually contains — no promises the page doesn't keep

Return JSON: {"title":"...","title_chars":N,"meta":"...","meta_chars":N}

6) Brand-Voice Content Rewrite — brand_voice_rewriter

You are {{brand}}'s editor. Rewrite the draft in our voice without changing facts or claims.

Voice profile (from .claude/product-marketing-context.md): {{voice_profile}}
Words we use: {{lexicon_yes}}. Words we never use: {{lexicon_no}}.

Constraints:
- Preserve every factual claim, number, and named source exactly
- Keep length within ±10% of the draft
- Flag (don't fix) any claim that lacks a source: [NEEDS SOURCE: ...]

Draft:
{{input}}

7) Generic Building Blocks

The original toolkit templates (structured extractor, classifier, summarizer, constrained rewrite, persona rewrite, policy-compliance check, prompt critique) remain useful as building blocks for non-content marketing automation — lead triage, review mining, survey coding. Pattern:

Classify input into one of: {{labels}}. Return only the label.
Input: {{input}}

Compose them: e.g., review mining = extractor (pull quotes) → classifier (theme) → summarizer (theme digest).

---

Citations (6 sources)

1. Anthropic — Prompt engineering overview: role prompting, structured outputs, "be clear and direct" (docs.anthropic.com/en/docs/build-with-claude/prompt-engineering) 2. OpenAI — Prompt engineering guide: instructions-first, delimiters, reference text to limit fabrication (platform.openai.com/docs/guides/prompt-engineering) 3. Google — Gemini prompting strategies: task/context/format decomposition, few-shot examples (ai.google.dev/gemini-api/docs/prompting-strategies) 4. Brown et al. — "Language Models are Few-Shot Learners" (NeurIPS 2020): few-shot examples improve format adherence 5. DAIR.AI — Prompt Engineering Guide: technique taxonomy and template anatomy (promptingguide.ai) 6. Ethan Mollick — One Useful Thing essays on practitioner prompting patterns for business content (oneusefulthing.org)

Technique Guide + LLM Governance for Marketing Teams

Two things in one reference: (1) which prompting technique to use for which marketing task, and (2) the governance layer — the rules a marketing team needs so AI-assisted content ships safely, legally, and on-brand at scale.

---

Part 1: Technique Selection for Marketing Tasks

Construction checklist (every marketing prompt): explicit role + goal; the audience and their pain named; output format with limits (chars/words); constraints as bullets; a forbidden list (clichés, banned claims, competitor names); instruction for missing inputs ("if no proof point fits, omit proof — never invent").

Failure patterns to check before testing: objective too broad ("write something engaging"); missing output schema; contradictory constraints (casual tone + formal compliance phrasing in one prompt); no negative instructions, so the model fills gaps with invented stats; hidden assumptions (brand voice referenced but not provided — pass the actual voice profile from .claude/product-marketing-context.md).

---

Part 2: LLM Governance for Marketing

Marketing is a high-exposure surface for AI failure: invented statistics in ads, undisclosed AI-generated endorsements, off-brand tone at scale, and privacy violations in personalization. Governance turns those from incidents into checklist items.

The Governance Stack

1. Approved-use registry — every production prompt lives in prompt_versioner.py with a named owner, author history, and change notes. No anonymous prompt edits in production workflows. 2. Pre-deployment evaluation — no prompt ships without passing its test suite (see evaluation-rubric.md). Model upgrades re-run the full baseline suite before switchover — a model swap is a change event. 3. Claim discipline — generated copy may only use claims from a maintained proof-point list. Test suites enforce this with forbidden_contains (superlatives, "guaranteed", unverifiable "%" patterns without a source token). A human verifies any new claim before it enters the proof list. 4. Disclosure rules — know where AI-generation disclosure is required: FTC rules cover endorsements/testimonials (fake or AI-fabricated reviews are actionable); the EU AI Act (Art. 50) requires disclosure for certain AI-generated content including synthetic media; platforms (Meta, TikTok, YouTube) require labels on AI-generated/altered media in ads, especially political/social-issue ads. 5. Data boundaries — customer data in prompts is processing under GDPR/CCPA: no PII in third-party model calls without a processing basis and vendor DPA; segment-level personalization over individual-level wherever possible; never paste customer lists into ad-hoc chat sessions. 6. Human-in-the-loop gates — mechanical scores gate, humans approve: anything paid (ad spend), anything legal-sensitive (claims, pricing, comparisons), anything brand-new (first run of a new prompt) gets human review before publishing. Routine regenerations of an approved prompt+suite can ship on green scores. 7. Incident loop — rejected ads, spam-folder complaints, brand-voice misses: each becomes a test case (evaluation-rubric.md, failure analysis) and, if systemic, a prompt version bump with a changelog entry.

Roles

Minimum Viable Governance (small team)

If the full stack is too heavy: (1) version every production prompt, (2) maintain the forbidden-claims list and wire it into forbidden_contains, (3) human-review everything paid, (4) re-run the suite on model changes. These four catch the expensive failures.

---

Citations (7 sources)

1. NIST — AI Risk Management Framework 1.0 (2023) + Generative AI Profile (NIST-AI-600-1, 2024): govern/map/measure/manage functions adapted here to content workflows 2. FTC — "Rule on the Use of Consumer Reviews and Testimonials" (2024) and FTC Act §5 guidance on AI-generated endorsements and deceptive claims (ftc.gov) 3. EU AI Act — Regulation (EU) 2024/1689, Art. 50 transparency obligations for AI-generated and manipulated content 4. ISO/IEC 42001:2023 — AI management systems: registry, role assignment, and change-management discipline mirrored in the governance stack 5. Anthropic — Usage policies + prompt engineering docs on constraining model claims and structured outputs (anthropic.com/legal/aup, docs.anthropic.com) 6. Meta — Advertising Standards on AI-disclosure requirements for altered/generated media in ads (transparency.fb.com / Meta Business Help Center) 7. GDPR (Regulation 2016/679) Arts. 6, 28 — processing basis and processor agreements governing customer data sent to model vendors