Sentiment Analysis Engineer

Sentiment Analysis Engineer

When to Use

• Define labeling schemas — document-level polarity, aspect-based (ABSA), emotion taxonomies, or multi-label targets
• Choose and implement model stacks — lexicon/rules, classical ML, fine-tuned transformers, or LLM prompt classifiers
• Design annotation programs — guidelines, adjudication, inter-annotator agreement (IAA), and gold-standard refresh
• Run evaluation and error analysis — macro-F1, calibration, confusion slices, and failure-mode catalogs
• Adapt models to domains — product reviews, social posts, support tickets, news, or finance text
• Handle edge cases — negation, sarcasm, entities, code-switching, and demographic or topical bias
• Plan production inference — batch vs streaming, latency budgets, model serving, and API contracts
• Operate monitoring and governance — label drift, score drift, human audit loops, and dashboard integration

When NOT to Use

• Writing marketing copy, brand voice, or content strategy → content-creator, brand-voice-enforcement
• General ML platform MLOps without sentiment/NLP scope → ml-ops-engineer
• Exploratory analytics or predictive modeling without text-sentiment focus → data-scientist
• Social media campaigns, calendars, or channel strategy → social-content, marketing-analyst
• AI safety classifier or harm-benchmark research only → ml-research-engineer-safeguards
• Legal, regulatory, or compliance conclusions from sentiment scores alone → compliance-engineer, legal-risk-assessment
• Production LLM agents, RAG, or copilot features (unless sentiment is one component) → ai-engineer
• Literature surveys without building or evaluating sentiment systems → ai-researcher

Related skills

Core Workflows

1. Scope and problem framing

Clarify unit of analysis (document, sentence, span, aspect), label set, languages, latency, and success metrics.

See `references/sentiment_analysis_engineer_scope.md`.

2. Labeling and annotation

Draft guidelines, pilot batches, measure IAA, and lock gold standards before model training.

See `references/labeling_schemas_and_annotation.md`.

3. Model selection and training

Compare lexicon, classical, fine-tuned encoder, and LLM-prompt baselines; document tradeoffs and compute.

See `references/models_lexicon_transformers_llm.md`.

4. Evaluation and error analysis

Report slice metrics, calibration, and qualitative failure buckets with reproduction examples.

See `references/evaluation_metrics_and_error_analysis.md`.

5. Domain, multilingual, and edge cases

Plan domain adaptation, translation vs native models, and tests for sarcasm, negation, and bias.

See `references/domain_multilingual_and_edge_cases.md`.

6. Production, monitoring, and governance

Specify serving paths, SLAs, drift monitors, human review queues, and analytics handoffs.

See `references/production_serving_monitoring_governance.md`.

Outputs

• Problem spec — labels, languages, domains, latency, and acceptance thresholds
• Annotation guide — definitions, examples, edge-case rules, adjudication process
• Model card — data, architecture, metrics, limitations, and known failure modes
• Eval report — headline metrics, slice tables, confusion analysis, calibration plots
• Serving spec — API schema, batch/streaming mode, versioning, and rollback plan
• Monitoring plan — drift metrics, audit sampling, and escalation triggers

Principles

• Treat labels as product decisions — ambiguous guidelines inflate disagreement and model variance
• Always report macro-F1 or per-class recall when classes are imbalanced; accuracy alone misleads
• Separate offline eval from online impact — production KPIs may differ from held-out test F1
• Document residual risk for sarcasm, negation, and out-of-domain text; do not overclaim coverage
• Prefer human-in-the-loop for high-stakes routing; scores support decisions, they do not replace policy

Domain, multilingual, and edge cases

Table of contents

1. Domain adaptation 2. Multilingual strategies 3. Sarcasm and negation 4. Bias and fairness 5. Entity-level sentiment

Domain adaptation

Techniques: in-domain fine-tune, continued pretrain on unlabeled domain corpus, adapter layers, self-training with confidence filtering (audit high-impact paths).

Always evaluate on in-domain test; cross-domain F1 is optimistic for deployment.

Multilingual strategies

For code-switching: train on mixed examples; evaluate dedicated code-switch slice; avoid translating away language signal.

Document primary languages, script, and locale (en-US vs en-GB) in model card.

Sarcasm and negation

• Sarcasm: often needs context (thread, user history); document-level models beat bag-of-words
• Negation: dependency-aware features, NLI-style models, or LLM with explicit negation examples in prompt
• Do not rely on lexicon antonym flip alone ("not good" ≠ "good")
• Report sarcasm slice separately; do not merge into generic "hard examples"

Bias and fairness

• Audit slices by demographic proxies only when ethical and legal review allows
• Check dialect and identity terms for systematic score shifts
• Avoid using sentiment for high-stakes decisions without policy and human review
• Log disparate impact on routing rates if scores trigger workflows

Partner with governance when scores affect customers or employees.

Entity-level sentiment

• Target extraction then polarity (pipeline) vs joint ABSA models
• Finance: link ticker/company mentions; disambiguate homographs
• News: coreference and headline-body alignment
• Evaluate target F1 and polarity F1 separately

When entities are ambiguous, prefer abstain or human review over forced labels.

Evaluation metrics and error analysis

Table of contents

1. Splits and leakage 2. Classification metrics 3. Calibration and thresholds 4. Error analysis 5. Regression testing

Splits and leakage

• Temporal split for reviews and social streams (train past, test future)
• Group split by author, product, or ticket ID to detect memorization
• Stratify rare classes in dev/test
• Remove duplicates and near-duplicates (MinHash) before metrics
• Hold out adjudicated gold never used for hyperparameter tuning

Classification metrics

For ABSA: micro-F1 over (aspect, polarity) pairs; separate aspect extraction F1 if two-stage.

Report confidence intervals (bootstrap) when test set < 2k examples.

Calibration and thresholds

• Plot reliability diagrams; apply temperature scaling or Platt scaling if scores drive automation
• Tune thresholds on cost matrix (false positive vs false negative), not default 0.5
• Report expected calibration error (ECE) when scores feed dashboards or alerts

Error analysis

Build a failure taxonomy with 20–50 quoted examples per bucket:

Link each bucket to mitigation (more labels, rules, model swap, human review).

Regression testing

• Freeze golden eval set (500–5k adjudicated items) per schema version
• CI gate: macro-F1 drop > 1–2 pts or per-class recall breach
• Compare lexicon, encoder, and LLM baselines on every release
• Log confusion matrix and top regression examples in model card

For production monitoring metrics, see production_serving_monitoring_governance.md.

Labeling schemas and annotation

Table of contents

1. Labeling schemes 2. Guideline design 3. Annotation workflow 4. Quality metrics

Labeling schemes

Choose the smallest schema that answers the business question; extra labels increase cost and disagreement.

Guideline design

• Define in-scope text (title + body, quoted replies, emojis, hashtags)
• Provide positive, negative, neutral, and borderline examples per class
• Document neutral rules (factual statements, questions, mixed without dominant polarity)
• Specify handling for emoji-only, URLs, @mentions, and deleted/edited content
• For ABSA: list aspect inventory (closed vs open vocabulary) and opinion span rules
• For finance/news: separate market sentiment from author opinion when required

Run a pilot batch (50–200 items) before full annotation; revise guidelines when adjudication rate exceeds ~15%.

Annotation workflow

1. Dual annotation on training/eval gold; single pass only for low-stakes exploratory sets 2. Adjudication by senior annotator or SME on disagreements 3. Active learning queue for uncertain model predictions in production (optional) 4. Version guidelines (v1, v2) and remap or re-label when definitions shift

Store: text_id, annotator_id, label, timestamp, guideline_version, adjudicated flag.

Quality metrics

Targets (rule of thumb): κ ≥ 0.6 acceptable for noisy social text; κ ≥ 0.75 for reviews with clear guidelines. Below 0.5 → fix guidelines before training.

Do not treat majority vote on ambiguous social text as ground truth without SME review on error slices.

Models: lexicon, transformers, and LLMs

Table of contents

1. Approach ladder 2. Lexicon and rules 3. Classical and neural encoders 4. Fine-tuning transformers 5. LLM prompt classification 6. Selection heuristics

Approach ladder

Always establish baselines in order:

1. Lexicon + rules — fast, interpretable, domain lexicons 2. Classical ML — TF-IDF + linear models on labeled data 3. Fine-tuned encoder — DistilBERT/RoBERTa/XLM-R for accuracy and latency balance 4. LLM prompts — few-shot or structured output when labels are fluid or low data

Document cost, latency, and maintainability at each step; stop when acceptance metrics are met.

Lexicon and rules

• Use domain lexicons (product, finance) plus negation handling (not, never, without)
• Combine VADER-style heuristics with custom intensifiers and emoticons
• Rules excel for compliance keywords, banned phrases, and bootstrap labels
• Weakness: sarcasm, domain shift, multilingual code-switching

Classical and neural encoders

• TF-IDF + logistic regression / linear SVM: strong baseline on ≥1k labeled docs
• Character n-grams help short social text
• FastText-style embeddings for multilingual baselines with modest data

Fine-tuning transformers

• Prefer encoder-only models for classification and token-level ABSA heads
• Use class weights or focal loss for imbalanced negative/neutral skew
• Early stopping on macro-F1 on dev; avoid optimizing accuracy alone
• Export ONNX/TorchScript when latency matters; quantize after accuracy check

Hyperparameters (starting points): lr 2e-5–5e-5, batch 16–32, 2–4 epochs with early stop.

LLM prompt classification

• Fixed JSON schema outputs: {"polarity":"positive","confidence":0.82}
• Few-shot examples from adjudicated gold; keep prompts versioned
• Use for label schema changes, long-tail aspects, or <500 labels
• Mitigate: temperature 0, self-consistency only for research, cache by hash
• Cost and latency usually exceed encoders at scale — reserve for tier-2 or audit

Selection heuristics

Pair with ai-engineer when sentiment is one tool inside a larger agent; own the classification eval and labels here.

Production serving, monitoring, and governance

Table of contents

1. Inference modes 2. Serving architecture 3. API and versioning 4. Monitoring and drift 5. Human review and governance 6. Analytics integration

Inference modes

Size models to p99 latency; keep LLM tier for low-volume or audit queues unless budget allows.

Serving architecture

• Model server: TorchServe, Triton, HF TEI, or managed endpoints
• Preprocessor: normalize unicode, strip PII, language ID, max length truncation
• Postprocessor: threshold, abstain class, aspect aggregation
• Feature store (optional): cache embeddings for repeat texts
• Shadow deploy: new model scores logged without affecting routing

Document cold start, batch size, and GPU vs CPU footprint in runbook.

API and versioning

Expose stable contract:

{
  "text_id": "…",
  "polarity": "negative",
  "confidence": 0.91,
  "aspects": [{"name": "battery", "polarity": "negative", "span": [12, 24]}],
  "model_version": "sentiment-encoder-v3.2",
  "schema_version": "polarity-v1"
}

• Semantic model_version per trained artifact; schema_version per label definition
• Rollback to prior version without redeploying consumers
• Abstain when confidence below τ or language unsupported

Monitoring and drift

• Sample human audit (e.g., 0.1–1% daily) on live traffic
• Store input hash + score + model_version; avoid raw PII in logs when policy requires redaction

Human review and governance

• Route low-confidence or high-impact items to review queues
• Never present sentiment as legal or compliance verdict
• Align with retention and deletion policies for stored text
• Incident playbook: schema change → re-eval golden set → staged rollout

Analytics integration

• Emit scores to warehouse (analytics-engineer) with scored_at, model_version, dimensions
• Dashboards: volume-weighted sentiment, aspect trends, slice by product/region
• Separate model metrics from business KPIs (CSAT, churn) in reporting

Partner ml-ops-engineer for CI/CD, canaries, and infra; own NLP-specific eval gates in this skill.

Sentiment analysis engineer scope

Table of contents

1. Role boundary 2. In-scope deliverables 3. Lifecycle coverage 4. Engagement checklist

Role boundary

Own engineering and evaluation of sentiment and opinion-mining systems:

In-scope deliverables

Out of scope: using sentiment scores as sole evidence for employment, credit, or legal outcomes without human review and policy.

Lifecycle coverage

Define labels → Annotate → Train/eval → Deploy → Monitor
      │            │          │          │         │
   schema/IAA   gold set   metrics    batch/stream  drift/audit

Document which stages are in scope per engagement (e.g., eval-only refresh vs greenfield build).

Engagement checklist

• [ ] Business question stated (routing, analytics, research, moderation support)
• [ ] Label definitions signed off with product/legal where scores affect users
• [ ] Language and domain list frozen (reviews, social, tickets, news, finance)
• [ ] Baseline and acceptance metrics pre-registered (macro-F1 floor, calibration target)
• [ ] PII and retention policy for text samples in logs and error analysis
• [ ] Production path chosen (batch nightly vs real-time API) with latency budget
• [ ] Handoff owners for dashboards (analytics-engineer) and infra (ml-ops-engineer) identified