hamelsmu/evals-skills
2 skills1.7k installs3.2k starsGitHub
Install
npx skills add https://github.com/hamelsmu/evals-skillsSkills in this repo
1Eval Auditeval-audit is a hamelsmu evals-skills entry for auditing LLM evaluation pipelines developers inherit or no longer trust. The workflow gathers eval artifacts—traces, evaluator configs, judge prompts, labeled data, and metrics dashboards—then runs diagnostic checks to surface issues like missing error analysis, unvalidated judges, and vanity metrics. Developers reach for eval-audit when taking over an eval system, unsure whether reported scores reflect real quality, or needing a structured starting point before building new evaluators. The skill explicitly defers greenfield evaluator authoring to error-analysis, write-judge-prompt, and validate-evaluator, focusing instead on a prioritized problem list with concrete next steps so engineering teams can harden measurement before shipping agent features.1.1kinstalls2Write Judge PromptA method for designing LLM-as-Judge evaluators that check one subjective failure mode code-based checks cannot handle. A developer uses it after error analysis identifies a failure mode requiring interpretation and code-based evaluators have been exhausted.556installs