Model Evaluation

Model Evaluation

Generate code that evaluates a SageMaker model.

Prerequisites

• The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Principles

1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn. 2. Confirm before proceeding. Wait for the user to agree before moving to the next step. 3. Don't read files until you need them. Only read reference files when you've reached the step that requires them. 4. Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask. 5. No narration. Share outcomes and ask questions. Keep responses short. 6. No repetition. If you said something before a tool call, don't repeat it after.

Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

Tell the user when the skill is activated:

"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."

If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.

Evaluation Types

There are two evaluation types:

• LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)
• Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.

Workflow

Step 1: Determine evaluation type

Do you already know which evaluation type to use?

Check conversation history, plan.md, workflow_state.json, or anything else you've already read.

If yes: confirm with the user.

"It sounds like you want to run [evaluation type]. Is that right?"

⏸ Wait for confirmation. If confirmed → go to Step 2.

If no: ask.

"What kind of evaluation would you like to run? I support:

>

1. LLM-as-Judge — an LLM grades your model's responses

2. Custom Scorer — programmatic scoring (math, code, or your own logic)

>

Pick one, or say 'help me decide' if you're not sure."

⏸ Wait for user.

• If user picks one → go to Step 2.
• If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read references/evaluation-type-guide.md and follow its instructions. It will guide the user to a choice and then return here.

You MUST NEVER make a recommendation to the user on eval type without reading references/evaluation-type-guide.md.

Step 2: Validate and hand off to evaluation workflow

Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to.

LLM-as-Judge validation

1. What model type are we evaluating? LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it):

• If you have the training job name or ARN, use the AWS MCP tool list-tags on the training job ARN and look for the sagemaker-studio:jumpstart-model-id tag. Contains "nova" → Nova. Anything else → OSS.
• If you have a Model Package ARN, use the AWS MCP tool describe-model-package and check the model description or source tags.
• If neither is available, ask the user.

2. Does the user have an evaluation dataset? LLM-as-Judge requires one.

Custom Scorer validation

1. Does the user have an evaluation dataset? Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.)

---

If validation fails, tell the user which requirement(s) aren't met and offer alternatives:

"[Evaluation type] won't work because [reason]."

If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user:

"Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation."

If the failure reason is something else, offer to help them pick a different evaluation type.

⏸ Wait for user.

If they say they do want help choosing a different eval type → read references/evaluation-type-guide.md.

If validation passes, read the corresponding reference file:

Follow the reference file's instructions from the beginning.

Code Output Guide

Mode Selection

Ask the user once before generating code: "Would you like me to generate a Jupyter notebook or a Python script?"

If the output format has already been decided in the conversation context, keep consistent — do not re-ask.

Shared Rules (Both Modes)

• Use EXACTLY the imports shown in each code template — do not add extras
• Replace [PLACEHOLDER] values with user-specific configuration

SageMaker Python SDK

• Include set_attribution(Attribution.SAGEMAKER_AGENT_PLUGIN) in the setup cell/section
• Only applies when generating code that uses from sagemaker.* imports.

Reading Code Templates

Templates use # Cell N: Label markers to delimit sections. # NOTEBOOK_ONLY skips a line in script mode; # NOTEBOOK_ONLY_SECTION on a # Cell N: line skips the entire section.

Notebook Mode

Write a .ipynb file in <project-dir>/notebooks/.

Naming and appending:

• Notebook path: <project-dir>/notebooks/<project-name>.ipynb
• If the notebook already exists → ask: _"Would you like me to append cells to the existing notebook, or create a new one?"_
• If it doesn't exist → create it
• When appending, use the template's # Cell 0 [markdown]: cell as the section divider before the new cells

Formatting:

• Use your file write tool to create the complete notebook JSON, OR use notebook MCP tools (create_notebook, add_cell) if available
• Do NOT use bash commands, shell scripts, or echo/cat piping
• 2-space JSON indentation
• Each source line is a separate string ending with \n (except the last)
• Escape quotes: \"
• No trailing commas

Structure:

• Wrap cells in {"cells": [...], "metadata": {...}, "nbformat": 4, "nbformat_minor": 4}
• Code cells: cell_type, execution_count: null, metadata: {}, outputs: [], source: [...]
• Markdown cells: cell_type: "markdown", no execution_count or outputs
• # Cell 0 [markdown]: becomes a markdown cell; all others become code cells

Execution:

• If notebook execution tools are available (e.g., run_cell MCP), offer to run cells for the user. If not available, tell the user to run cells themselves.
• Do NOT use bash commands or inline scripts to execute notebook cells.

Script Mode

Write a numbered .py file in <project-dir>/scripts/.

Naming:

• Format: NN_<descriptive_name>.py (e.g., 01_sft_finetuning.py) — use the next available number in <project-dir>/scripts/

Formatting:

• Plain Python file, standard text
• Use # %% cell markers to preserve logical sections (IDE-compatible)
• Include a docstring at the top describing what the script does
• # Cell 0 [markdown]: → a comment block or docstring

Dependencies:

• Install any required pip packages directly (e.g., pip install sagemaker>=3.7.1) before writing or running the script. Do not embed install commands in the script itself.

Execution:

• Run the script using standard Python execution (python3 <script>.py).

Resumption After Interruption

If the conversation was interrupted while a job was running (e.g., context compaction, user stopped and restarted, connection drop), do NOT re-run the script. Instead, check for an existing job by name or ARN from the conversation context or PLAN.md, and monitor its status rather than launching a duplicate.

Create Reward Function

Help the user create and register a Lambda reward function as a SageMaker Hub Evaluator.

Principles

1. One thing at a time. Each response advances exactly one decision. 2. Confirm before proceeding. Wait for the user to agree before moving to the next step. 3. No narration. Share outcomes and ask questions. Keep responses short.

Prerequisites

The caller must know the base model being used (needed for template selection).

Workflow

Step 1: Copy Template to Project

Select the reward function template based on the base model:

• Nova 2.0 Lite → scripts/nova_reward_function_source_template.py
• All other models → scripts/reward_function_source_template.py

Copy the selected template as lambda_function.py into the project's scripts directory.

• Read the directory-management skill to determine the correct directory for storing scripts.

Step 2: Generate Code

Create a single notebook cell that registers the local file as a SageMaker Hub Evaluator. Set reward_function_path to the path where lambda_function.py was saved in Step 1.

from sagemaker.ai_registry.evaluator import Evaluator

reward_function_path = ""  # Path to lambda_function.py from Step 1

evaluator = Evaluator.create(
    name="[GENERATE A NAME FOR THE EVALUATOR HERE]",
    type="RewardFunction",
    source=reward_function_path,
)
print(f"Reward Function ARN: {evaluator.arn}")

Remember to set an appropriate name for the Evaluator by yourself in the above code, based on the use case and the current context.

• Format: lowercase, alphanumeric with hyphens only, 1-20 characters
• Pattern: [a-zA-Z0-9](-*[a-zA-Z0-9]){0,20}

Step 3: Customize the Lambda

After copying the template and generating the notebook cell, inform the user that lambda_function.py contains TODO sections that need customization for their use case. Ask:

"The reward function template has placeholder scoring logic that needs to be customized for your task. Would you like me to fill in the TODOs based on what I know about your use case, or would you prefer to do it yourself?"

• If the user wants you to do it: customize the helper functions, reward logic, input parsing, score computation, and return statement based on the task context. Then present the result and warn: "Please review the Lambda code before running — especially the scoring logic. I may have made incorrect assumptions about your requirements."
• If the user wants to do it: direct them to edit lambda_function.py directly and wait for their acknowledgment before proceeding.

Output

The output of this workflow is a reference to evaluator.arn. Embed the Evaluator.create cell as the first cell of the evaluation notebook so that subsequent cells can reference evaluator.arn directly as a variable.

References

• scripts/reward_function_source_template.py — Lambda source template for open-weights models
• scripts/nova_reward_function_source_template.py — Lambda source template for Nova 2.0 Lite

Custom Lambda Scorer

This file guides you through resolving a custom Lambda scorer (evaluator) for use with Custom Scorer evaluation.

Resolve evaluator

For this step, you need: the evaluator ARN of a registered reward function.

Check if you already know this from conversation context (e.g., the user mentioned a reward function ARN, or one was used in a previous evaluation). If so, confirm and return to the main workflow.

If not, ask:

"Do you have an existing reward function registered in SageMaker? If so, what's the evaluator ARN?"

If the user has an ARN, validate it:

• It should look like: arn:aws:sagemaker:REGION:ACCOUNT:hub-content/.../JsonDoc/NAME/VERSION
• Validate by splitting the part after hub-content/ into HUB_NAME/JsonDoc/CONTENT_NAME/VERSION and calling:

  aws sagemaker describe-hub-content --hub-name HUB_NAME --hub-content-type JsonDoc --hub-content-name CONTENT_NAME --hub-content-version VERSION --region REGION

If the call succeeds, HubContentStatus is Available, and HubContentSearchKeywords includes @evaluatortype:rewardfunction, the evaluator is valid.

If validation fails, tell the user what went wrong:

• API call errors → "That ARN doesn't seem to exist. Could you double-check it?"
• Status is not `Available` → "That evaluator exists but isn't ready (status: [status]). It may still be provisioning."
• Missing `@evaluatortype:rewardfunction` → "That resource exists but doesn't appear to be a reward function evaluator. Could you verify you have the right ARN?"

In any failure case, offer to re-enter the ARN or fall back to a built-in scorer.

If the user doesn't have one:

"You don't have a registered reward function yet. I can help you create one — I'll provide a template with your scoring logic and register it as a SageMaker Hub Evaluator. Or you can use a built-in scorer instead.

>

1. Create a new reward function — I'll walk you through it

2. Use a built-in scorer — Prime Math or Prime Code

>

Which would you prefer?"

• If create new → read references/create-reward-function.md and follow its instructions. It will produce an evaluator ARN. Once complete, return here and proceed to "After resolution".
• If built-in → return to the main Custom Scorer workflow and switch to the built-in scorer path.

After resolution

Once you have the evaluator ARN, return to the main Custom Scorer workflow.

---

Lambda input/output contracts

Lambda return format

The return format depends on the model type:

For OSS models:

# <RETURN_FORMAT> — OSS models
return {
    "statusCode": 200,
    "headers": {"Content-Type": "application/json"},
    "body": json.dumps([result])  # body is a JSON STRING
}

For Nova models:

# <RETURN_FORMAT> — Nova models
return {
    "statusCode": 200,
    "headers": {"Content-Type": "application/json"},
    "body": [result]  # body is a PARSED LIST (not json.dumps)
}

Each result object has the shape:

{
  "id": "sample_id",
  "aggregate_reward_score": 0.85,
  "metrics_list": [{ "name": "metric_name", "value": 0.75, "type": "Metric" }]
}

Lambda input format

The input format depends on the model type:

For OSS models (gen_qa path):

[{
  "id": "hash",
  "model_response": "model's generated text",
  "query": "the prompt",
  "response": "the gold answer from dataset",
  "reference_answer": { "text": "the gold answer from dataset" },
  "metadata": {},
  "processor_config": {}
}]

For Nova models (rft_eval path):

[{
  "id": "sample_id",
  "messages": [
    { "role": "user", "content": "the prompt" },
    { "role": "assistant", "content": "model's generated output" }
  ],
  "reference_answer": "the gold answer from dataset"
}]

To extract the model response from Nova input: read the last message with role: "assistant".

Evaluator registration

CustomScorerEvaluator requires a Hub Content ARN (registered via Evaluator.create()), NOT a raw Lambda ARN.

from sagemaker.ai_registry.evaluator import Evaluator
from sagemaker.ai_registry.air_constants import REWARD_FUNCTION

evaluator = Evaluator.create(
    name="my-reward-function",
    source="path/to/reward_function.py",
    type=REWARD_FUNCTION
)
# Use evaluator.arn as the evaluator parameter

Using a raw Lambda ARN (e.g., arn:aws:lambda:...) will fail with Invalid HubContentArn format.

Custom Scorer Evaluation

Guide the user through the process for evaluating a model with Custom Scorer (built-in Prime Math/Code or custom Lambda).

Workflow

Step 0: Consider prior context

Before proceeding, silently think about the context you have about the user's project, including conversation history and file reads. You should use that knowledge, and avoid asking questions you already know the answer to.

Step 1: Validate Custom Scorer compatibility

Before proceeding, confirm one thing:

1. Does the user have an evaluation dataset?

If the check fails (the user has no eval dataset), tell the user and offer to help them pick an alternative:

"Custom Scorer evaluation requires an evaluation dataset. Would you like help choosing a different evaluation type?"

If they want help choosing a different evaluation type → break this workflow and read references/evaluation-type-guide.md.

If the check passes, proceed.

Step 2: Understand the task

For this step, you need: to understand the task the model is trained to do. If you know this already, skip this step. If not, ask the user:

"What task is this model trained to do?"

Step 3: Get evaluation dataset

For this step, you need: the evaluation dataset S3 path. If you know this already, skip this step. If not, ask the user:

"Where's your evaluation dataset stored in S3?"

Step 4: Choose scorer type

For this step, you need to know which scorer to use.

If the model is Nova: built-in scorers (Prime Math, Prime Code) are not supported for Nova models. Inform the user and proceed with Custom Lambda:

"For Nova models, only Custom Lambda scoring is supported. Built-in scorers (Prime Math, Prime Code) won't produce results. Let's set up a Custom Lambda scorer."

Then read references/custom-lambda-scorer.md and follow its instructions. Return here and proceed to Step 5.

If the model is OSS: ask if you don't already know from context:

"Which type of scorer would you like to use?

>

1. Prime Math — built-in scorer for math problems (checks answer correctness)

2. Prime Code — built-in scorer for coding problems (executes code against test cases)

3. Custom Lambda — your own scoring logic as a Lambda function. You can use an existing registered evaluator or create a new one.

>

Which would you prefer?"

• If built-in (Prime Math or Prime Code) → note the choice and proceed to Step 5.
• If custom Lambda → read references/custom-lambda-scorer.md and follow its instructions to resolve the evaluator. Then return here and proceed to Step 5. You MUST follow these instructions before moving on.

Step 5: Validate dataset format

IMPORTANT: you MUST validate that dataset, to ensure that it is the correct format. Note that there are precise requirements based on model and evaluation type, so you cannot skip this step.

Reference the dataset-evaluation skill to perform this validation.

Step 6: Determine evaluation scope

For this step, you need to know which model(s) to evaluate.

If you already know from context, confirm and move on. Otherwise, ask:

"Would you like to evaluate:

>

1. Just your fine-tuned model

2. Just a base model

3. Both, with a comparison

⏸ Wait for user approval.

Step 7: Resolve Model Package ARN

This step only applies if the evaluation scope includes the fine-tuned model (option 1 or 3 from Step 6). If the user chose base model only, skip to Step 8.

For this step, you need: the Model Package ARN of the fine-tuned model.

If you already have it from prior context, confirm with the user and move on. Otherwise, ask:

"What's the Model Package ARN (or group name) of your fine-tuned model?"

If they provide a group name, resolve the ARN by calling list-model-packages via the AWS tool. Use the latest version's ModelPackageArn.

Validate the resolved ARN:

• Must look like: arn:aws:sagemaker:REGION:ACCOUNT:model-package/NAME/VERSION
• If it's a group ARN (:model-package-group/), resolve to a package ARN by calling list-model-packages via the AWS tool. Use the latest version's ModelPackageArn.
• If it contains :model-package/ but does NOT end with a version number (e.g., /1), resolve it: extract the group name and use list-model-packages.
• If it contains /DataSet/, /TrainingJob/, or other non-model-package resource types, flag it: "That looks like a [Dataset/TrainingJob] ARN, not a model package ARN. Could you double-check?"
• Verify it exists by calling describe-model-package via the AWS tool. If this fails, tell the user the ARN wasn't found and ask them to double-check.

Step 8: Resolve base model

This step only applies if the evaluation scope includes the base model (option 2 or 3 from Step 6). If the user chose fine-tuned only, skip to Step 9.

For comparison mode (option 3): the base model is resolved automatically from the fine-tuned model's lineage — no additional input needed.

For base model only (option 2): you need a JumpStart model ID (e.g., meta-textgeneration-llama-3-2-1b-instruct). Check if you already know it from context. If not, ask:

"What's the JumpStart model ID of the base model you'd like to evaluate?"

<!-- TODO: Add guidance for helping the user find their JumpStart model ID. -->

Step 9: Resolve IAM role ARN

For this step, you need: an IAM role ARN with permissions for evaluation.

If you already know the IAM role ARN from conversation context, confirm with the user and move on.

If not, but you know the training job name, look up the role from that training job, since it should be a suitable role:

• Call describe-training-job via the AWS tool and extract RoleArn.
• Confirm with the user.

If you don't have either, ask:

"What IAM role ARN should I use for the evaluation job?"

Important: If the user chose a custom Lambda scorer, the IAM role must also have lambda:InvokeFunction permission for the scorer's Lambda function. Mention this to the user:

"Note: since you're using a custom Lambda scorer, this role needs lambda:InvokeFunction permission for your scorer function. Does your role have that?"

Step 10: Resolve AWS Region

For this step, you need: the AWS region for the evaluation job. If you already know it from context, confirm and move on. Otherwise, ask.

Step 11: Resolve S3 output path

For this step, you need: an S3 path for evaluation output. If you already know it from context, confirm and move on. Otherwise, ask.

Step 12: MLflow experiment name (optional)

For this step, you need to know: the MLflow experiment name. If you already know this from context, confirm and move on. If not, the name should follow this pattern: <project-name>-eval.

Perform this step quietly. The user does not need to know about this. This is to make sure their results are viewable alongside other experimental work they're doing.

Step 13: Final Checks

IMPORTANT: Before moving on, you MUST think through every step of this workflow. Did you complete it, or did you have a good reason for skipping? If not, please complete it now. In particular, make sure you have carefully validated the dataset and the lambda against the correct requirements.

Step 14: Confirm configuration

Summarize everything and ask for approval:

"Here's the evaluation setup:

>

- Task: [task]

- Dataset: [path]

- Scorer: [Prime Math / Prime Code / Custom Lambda ARN]

- Model: [Model Package ARN or JumpStart model ID]

- Evaluation scope: [fine-tuned only / base only / both with comparison]

- IAM role: [ARN]

- Region: [region]

- S3 output: [path]

- MLflow experiment name: [MLflow experiment name]

>

Does this look right?"

⏸ Wait for user approval.

Step 15: Generate code

Read ../references/code_output_guide.md for output format rules.

If no project directory exists, activate the directory-management skill to set one up.

Read code_templates/custom_scorer_evaluator.py, substitute the collected values into the placeholders, and write the cells. The template uses # Cell N: Label markers — each marker starts a new notebook cell, with everything between one marker and the next becoming that cell's content.

Step 16: Post-generation

Notebook mode:

To run:
1. Cell 1 — configuration and SDK install
2. Cell 2 — start evaluation
3. Cell 3 — polls status automatically (~25-60 min)
4. Cell 4 — show results

Script mode:

Evaluation can take hours depending on your dataset. Present the user with options:

"Would you like me to:

>

1. Leave it to you — run with python scripts/[script_name]

2. Run it and wait until it's done

3. Start it but don't wait — we can check status later"

• Option 1: Done. Wait for user to come back.
• Option 2: Execute the script as-is. execution.wait() polls until complete. Report results.
• Option 3: Remove the execution.wait() call, execute, report the evaluation ARN.

Note: evaluate() does not accept a wait parameter. It always returns immediately. Blocking is done via execution.wait(target_status="Succeeded").

Checking status:

• describe-pipeline-execution --pipeline-execution-arn ARN → PipelineExecutionStatus
• list-pipeline-execution-steps --pipeline-execution-arn ARN → per-step StepStatus, FailureReason

Showing results after completion:

• Run: EvaluationPipelineExecution.get(arn=ARN).show_results()

FAQ

Q: What metrics do I get with Custom Scorer? A: Both built-in and custom scorers automatically produce standard NLP metrics (F1, ROUGE, BLEU) alongside your custom scores.

Q: Does my IAM role need special permissions for Custom Scorer? A: Yes — if using a custom Lambda scorer, the IAM role needs lambda:InvokeFunction permission for the scorer's Lambda function. Built-in scorers (Prime Math/Code) don't require additional permissions.

Q: Can I create a new reward function through this skill? A: Yes — if you choose Custom Lambda and don't have an existing evaluator, the agent will walk you through creating one from a template and registering it via Evaluator.create.

Nova Model Notes

Custom Scorer evaluation works with Nova models via Custom Lambda. Built-in scorers (Prime Math, Prime Code) are not supported — the pipeline will run without error, but the scorer will not execute.

Evaluation Type Guide

Help the user decide which evaluation type to use based on their goals and constraints.

Evaluation types at a glance

When to use which — in short:

• To assess subjective qualities like tone, helpfulness, coherence, or faithfulness → LLM-as-Judge
• When a programmatic approach can give a meaningful signal about output quality → Custom Scorer

Decision flow

Work through the steps below in order. For each, use what you already know from conversation history, plan.md, workflow_state.json, or other files you've read. Only ask the user if you genuinely don't know.

Step 1: Check for evaluation dataset

For this step, you need to know: whether the user has an evaluation dataset.

If you don't know from previous context, ask:

"Do you have an eval dataset?"

⏸ Wait for user.

If the user does not have one:

"All supported evaluation types require an evaluation dataset. Unfortunately, this skill can't help with model evaluation without one."

Stop here. Do not offer to help create or find a dataset, since our skills do not support this.

If the user has an evaluation dataset, continue.

Step 2: Check model compatibility

For this step, you need to know: what type of model is being evaluated (open source or Nova).

If you don't already know from conversation context, try to determine it:

1. If you have the training job name or ARN, use the AWS MCP tool list-tags on the training job ARN and look for the sagemaker-studio:jumpstart-model-id tag.

• Contains "nova" (e.g., nova-micro, nova-lite, nova-pro) → Nova
• Anything else (Llama, Mistral, Qwen, GPT-OSS, DeepSeek, etc.) → OSS

2. If you have a Model Package ARN, use the AWS MCP tool describe-model-package and check the model description or source tags for the same model ID. 3. If neither is available, ask the user:

"What model are you evaluating — is it a Nova model or an open-source model (like Llama, Mistral, Qwen, etc.)?"

If the model is Nova, LLM-as-Judge is not supported. Tell the user:

"Unfortunately, LLM-as-Judge isn't available for Nova models. Can we use Custom Scorer instead?"

If Nova -> Skip to step 4.

Else -> Continue to Step 3 with the remaining options.

Step 3a: Understand the task and data

For this step, you need to understand: what the model does, what the evaluation data looks like, and what "success" means for this task.

If you don't already have a clear picture from conversation context, ask:

"Can you tell me about the task you're focused on? Please explain what you want your model to do and what your evaluation dataset looks like."

You need enough context to reason about Steps 3b and 3c. If the user's answer is vague, ask a follow-up before moving on.

⏸ Wait for user.

Step 3b: Assess Custom Scorer signal strength

Based on what you know about the task and data, think about: how strong of a signal a custom (programmatic) scorer could give us about task success

Rate the signal strength as strong, medium, or weak:

• Strong: A programmatic check can reliably tell you whether the output is correct. Examples include math problems with numerical answers, classification tasks with labels, or extraction tasks with exact ground truths.
• Medium: The task has reference answers, and programmatic comparison gives a useful but imperfect signal. Examples include summarization or Q&A where text comparison against reference answers captures something meaningful, or format compliance checks where you can verify structure even if you can't verify content quality.
• Weak: What matters about the output is hard to capture in code. There may be no reliable reference answer to compare against, or the reference doesn't capture what the user actually cares about.

Think broadly — even tasks that seem subjective may have a programmatic angle.

Step 3c: Assess LLM-as-Judge signal strength

Based on what you know about the task and data, think about: how strong of a signal an LLM judge could give us about task success

Rate the signal strength as strong, medium, or weak:

• Strong: Key model quality metrics are inherently subjective — helpfulness, coherence, tone, etc.
• Medium: The task has some subjective element, but also a clear factual or structural component that a programmatic approach could partially cover.
• Weak: The task has a single objectively correct answer, and a judge model carries the risk of hallucinating, while a programmatic check would be more reliable.

Think broadly — LLM-as-Judge can surface issues that are hard to anticipate with code, but it's not always the best tool for the job.

Step 3d: Check cost sensitivity

For this step, you need to know: how important keeping costs low is to the user.

LLM-as-Judge invokes a model to score each sample, which adds cost. Custom Scorer runs your own code, which is cheaper.

If you already know from context, skip to Step 4. If not, ask:

"On a scale of 1-5, how important is it to you to keep evaluation costs low, even if it means less nuanced results? 5 means prioritize budget above all else."

⏸ Wait for user.

Step 4: Recommend an evaluation type

Use the signal strength assessments and cost sensitivity to make a recommendation:

• Nova model → recommend Custom Scorer (only available option).
• Custom Scorer signal is strong → recommend Custom Scorer. It's deterministic, reproducible, and cost-effective. A programmatic approach gives you a reliable signal for this task.
• Cost sensitivity is very high (4-5) and Custom Scorer signal is weak but not totally absent → recommend Custom Scorer, but be upfront that the programmatic signal may be limited for this task. A partial signal at low cost may be preferable to a richer signal at higher cost.
• LLM-as-Judge signal is strong and Custom Scorer signal is weak → recommend LLM-as-Judge. The task needs the kind of nuanced judgment that only an LLM can provide.
• Both have medium or strong signal → Carefully weigh the customer's cost concerns with the benefits of each eval type. Recommend the one that you think fits all of their needs the best.

Present your recommendation with a brief reason:

"Based on what you've told me, I'd recommend [evaluation type] — [one sentence explaining why]. Want to go with that?"

⏸ Wait for user to confirm.

Once the user confirms, return to the main SKILL.md workflow (Step 2: Validate and hand off).

---

Custom Scorer: choosing the right scorer type

If the user chose Custom Scorer, use this logic to recommend the specific scorer:

Decision logic:

1. If task is math with verifiable answers → recommend Prime Math. 2. If task is code generation with testable I/O → recommend Prime Code. 3. Otherwise → recommend Custom Lambda.

LLM-as-Judge with Built-in Metrics: Alignment Guide

This file describes the process for aligning on built-in metrics to use for model evaluations with LLMaaJ.

Select Metrics

Refer to the metrics tables below for the full list of metrics with descriptions and common combinations.

Based on the user's task and data, recommend specific metrics with reasoning:

"Based on your [task], I recommend these metrics:

>

- [metric1]: [why it matters for this task]

>

Does this look good, or do you want to consider other metrics?"

⏸ Wait for user to confirm.

Tips:

• Start with the common combinations from the metrics file as a baseline
• Adjust based on what you know about the user's task and data
• If the user pushes back, understand why and adjust — don't just agree

LLM-as-Judge Built-in Metrics

SageMaker provides 11 built-in metrics for LLM-as-Judge evaluation, organized into Quality and Responsible AI categories.

Quality Metrics

Responsible AI Metrics

Usage in Code

In code, these metrics are specified as Builtin.Correctness, Builtin.Completeness, etc. When discussing with users, use natural language names.

Common Metric Combinations

• QA/Math tasks → Correctness, Completeness, Faithfulness, Relevance
• Summarization → Completeness, Coherence, Relevance
• General assistance → Helpfulness, Relevance, FollowingInstructions
• Safety evaluation → Harmfulness, Stereotyping, Refusal

LLM-as-Judge Custom Metrics Guide

This file describes the process for collecting and validating custom metric definitions.

Step 1: Collect Custom Metrics

Ask the user to provide their custom metrics as JSON — either by pasting it directly or pointing to a file. The JSON must be an array of metric definitions following the Bedrock format.

"Please share your custom metrics JSON. You can paste it here or point me to a file."

⏸ Wait for user.

Helping Users Structure Metrics

If the user doesn't have ready-made JSON but describes what they want to evaluate, you can help them create the JSON structure. Be upfront about limitations:

"I can help you put together the JSON structure based on what you've described. Note that I can't guarantee the judge model will interpret your metric exactly as intended — you may need to iterate on the prompt wording after seeing initial results."

When helping, follow the Bedrock-recommended prompt structure (in this order):

1. Role definition (optional) 2. Task description (required, minimum 15 words) 3. Criterion and rubric (optional) 4. Input variables (required, must be last in the prompt)

Available input variables: {{prompt}}, {{prediction}}, {{ground_truth}}

Example of a valid single custom metric:

[
  {
    "customMetricDefinition": {
      "name": "DomainAccuracy",
      "instructions": "You are a domain expert. Evaluate whether the response accurately addresses the domain-specific aspects of the prompt.\n\nPrompt: {{prompt}}\nResponse: {{prediction}}",
      "ratingScale": [
        { "definition": "Accurate", "value": { "floatValue": 1.0 } },
        { "definition": "Inaccurate", "value": { "floatValue": 0.0 } }
      ]
    }
  }
]

Multiple custom metrics go in the same array (max 10 per job).

Step 2: Write and Validate the JSON Artifact

Once you have the custom metrics JSON (from the user or co-created), write it to a file called custom_metrics.json next to where the notebook will go.

Then validate it by running the validation script:

python scripts/validate_custom_metrics.py custom_metrics.json

If validation fails, show the errors to the user and iterate until it passes.

⏸ Do not proceed until validation passes.

After Collection

Once custom metrics are validated, return to the main workflow (Step 7) to check if the user also wants built-in metrics.

LLMaaJ evaluation

Guide the user through the process for evaluating a model with LLMaaJ.

Workflow

Step 0: Consider prior context

Before proceeding, silently think about the context you have about the user's project, including conversation history and file reads. You should use that knowledge, and avoid asking questions you already know the answer to.

Step 1: Understand the task

For this step, you need: what task the model is trained to do. If you know this already, skip this step. If not, ask the user:

"What task is this model trained to do?"

Step 2: Get evaluation dataset

For this step, you need: the evaluation dataset S3 path. If you know this already, skip this step. If not, ask the user:

"Where's your evaluation dataset stored in S3?"

Step 3: Understand the data

For this step, you need: to understand what the data looks like to inform metric recommendations. If you already know what the data looks like, skip this step. If not, ask the user:

"Can you tell me a bit about your evaluation dataset — what format is it in, and what do the input/output fields look like?"

If the user isn't sure, offer to peek at the data:

"May I read a few records of your dataset to help inform my recommendations?"

If they say yes, use the AWS tool to call s3api get-object with a Range header to read the first few KB. If you fail to get a sample, move on and rely on the user's description.

Step 4: Validate dataset format

If the evaluation dataset was already validated via the dataset-evaluation skill — either earlier in this conversation, or in a previous session (as recorded in plan.md) — skip this step.

Otherwise, activate the dataset-evaluation skill to validate it. If it fails, offer to activate the dataset-transformation skill to convert it. Do not proceed until the dataset is valid.

Step 5: Dataset size warning

After dataset validation, warn the user about the Bedrock evaluation dataset size limit:

"One thing to note — Bedrock LLM-as-Judge evaluation supports a maximum of 1,000 rows per job. If your dataset is larger than that, the job will fail. You may need to trim it before running the evaluation."

Step 6: Check for custom metrics

For this step, you need: whether the user has predefined custom metrics.

"Do you have predefined custom metrics you'd like to use? If so, they must follow the Bedrock custom metrics format: https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html

>

If not, no worries — I can recommend built-in metrics for your task."

⏸ Wait for user.

• If the user has custom metrics → Read references/llmaaj-custom-evaluation.md and follow its instructions to collect and validate the metrics JSON.
• If the user does not have custom metrics → Move to Step 7.

Step 7: Select built-in metrics

For this step, you need: user agreement on which built-in metrics to use (if any).

If the user provided custom metrics in Step 6, ask whether they also want built-in metrics:

"Would you also like to include any built-in metrics alongside your custom ones?"

If they say no, skip to Step 8.

For built-in metric selection, read references/llmaaj-builtin-evaluation.md and follow its instructions.

Step 8: Determine evaluation scope

For this step, you need: which model(s) to evaluate.

If you already know from context (e.g., the user said "compare my model to the base"), confirm and move on. Otherwise, ask:

"Would you like to evaluate:

>

1. Just your fine-tuned model

2. Just a base model

3. Both, with a comparison

>

Which would you prefer?"

⏸ Wait for user.

Step 9: Resolve Model Package ARN

This step only applies if the evaluation scope includes the fine-tuned model (option 1 or 3 from Step 8). If the user chose base model only, skip to Step 10.

For this step, you need: the Model Package ARN of the fine-tuned model.

Use this priority order:

1. Model Package ARN from workflow state or conversation: If you already have a model package ARN from prior context or from earlier in the conversation, confirm it with the user and move on. 2. Ask the user: If you don't have the ARN, ask:

"What's the Model Package ARN (or group name) of your fine-tuned model?"

If they provide a group name, resolve the ARN by calling list-model-packages via the AWS tool with the group name.

Use the latest version's ModelPackageArn from the response.

Validate the resolved ARN (whether from API lookup, conversation context, or user input):

• A valid versioned model package ARN looks like: arn:aws:sagemaker:REGION:ACCOUNT:model-package/NAME/VERSION
• If the ARN contains :model-package-group/, this is a group ARN, not a package ARN. Resolve it using the lookup in #2.
• If the ARN contains :model-package/ but does NOT end with a version number (e.g., /1), resolve it: extract the group name from the ARN and use the lookup in #2.
• If it contains /DataSet/, /TrainingJob/, or other non-model-package resource types, flag it: "That looks like a [Dataset/TrainingJob] ARN, not a model package ARN. Could you double-check?"
• Verify the ARN exists before proceeding by calling describe-model-package via the AWS tool.

If this fails, tell the user the ARN wasn't found and ask them to double-check.

Step 10: Resolve base model

This step only applies if the evaluation scope includes the base model (option 2 or 3 from Step 8). If the user chose fine-tuned only, skip to Step 11.

For comparison mode (option 3): the base model is resolved automatically from the fine-tuned model's lineage — no additional input needed.

For base model only (option 2): you need a JumpStart model ID (e.g., meta-textgeneration-llama-3-2-1b-instruct). This is a string identifier, not an ARN. Check if you already know it from conversation context (e.g., the user mentioned which base model they used for fine-tuning). If not, ask:

"What's the JumpStart model ID of the base model you'd like to evaluate?"

<!-- TODO: Add guidance for helping the user find their JumpStart model ID (e.g., list_hub_contents API, or looking at training job tags). See model-selection skill for patterns. -->

Step 11: Select judge model

For this step, you need: which judge model to use for evaluation. This step always runs — both built-in and custom metrics require a judge model.

Read references/supported-judge-models.md for the canonical list, selection guidance, and validation steps.

Before presenting options, run the validation checks from the reference doc against the user's account and region. Only include models that pass all checks.

Present the available models as a numbered list:

"Here are the judge models available in your region:

>

1. [model A]

2. [model B]

...

>

Which model would you like to use?"

EXTREMELY IMPORTANT: NEVER recommend or suggest any particular model based on the context you have. YOU ARE ALLOWED ONLY to display the list of models. DO NOT add your own recommendation or suggestion after displaying the list.

Step 12: Resolve IAM role ARN

For this step, you need: an IAM role ARN with permissions for Bedrock evaluation.

If you already know the IAM role ARN from conversation context, confirm with the user and move on.

If not, but you know the training job name, look up the role from that training job, since it should be a suitable role:

• Call describe-training-job via the AWS tool and extract RoleArn.
• Confirm with the user: "I found the IAM role from your training job: [ARN]. Should I use this for evaluation?"

If you don't have either, ask:

"What IAM role ARN should I use for the evaluation job? It needs bedrock.amazonaws.com in its trust policy."

Step 13: Resolve AWS Region

For this step, you need: the AWS region for the evaluation job. If you already know it from context (e.g., the training job region), confirm and move on. Otherwise, ask.

Step 14: Resolve S3 output path

For this step, you need: an S3 path for evaluation output. If you already know it from context, confirm and move on. Otherwise, ask.

Step 15: MLflow experiment name (optional)

For this step, you need to know: the MLflow experiment name. If you already know this from context, confirm and move on. If not, the name should follow this pattern: <project-name>-eval.

Perform this step quietly. The user does not need to know about this. This is to make sure their results are viewable alongside other experimental work they're doing.

Step 16: Confirm configuration

Summarize everything and ask for approval:

"Here's the evaluation setup:

>

- Task: [task]

- Dataset: [path]

- Custom metrics: [Yes — N metrics / No]

- Built-in metrics: [list, or None]

- Judge: [model]

- Model: [Model Package ARN or JumpStart model ID]

- Evaluation scope: [fine-tuned only / base only / both with comparison]

- IAM role: [ARN]

- Region: [region]

- S3 output: [path]

- MLflow experiment name: [MLflow experiment name]

>

Does this look right?"

⏸ Wait for user approval.

Step 17: Bedrock Evaluations agreement

This step is mandatory. Do not skip it. Do not proceed without explicit user confirmation.

Before generating the notebook, present the following agreement language:

Important: Amazon Bedrock Evaluations Terms

>

This feature is powered by Amazon Bedrock Evaluations. Your use of this feature is subject to pricing of Amazon Bedrock Evaluations, the Service Terms applicable to Amazon Bedrock, and the terms that apply to your usage of third-party models. Amazon Bedrock Evaluations may securely transmit data across AWS Regions within your geography for processing. For more information, access Amazon Bedrock Evaluations documentation.

>

Do you acknowledge and agree to proceed?

⏸ Hard stop. Wait for the user to explicitly confirm. Acceptable responses include "yes", "I agree", "proceed", "ok", or similar affirmative statements. If the user asks questions about the terms, answer them, then re-ask for confirmation. Do NOT generate the notebook until the user has confirmed.

Step 18: Generate code

Read ../references/code_output_guide.md for output format rules.

If a project directory already exists (from earlier in the workflow), use it. Otherwise, activate the directory-management skill to set one up.

Read code_templates/llmaaj_evaluator.py, substitute the collected values into the placeholders, and write the cells. The template uses # Cell N: Label markers — each marker starts a new notebook cell, with everything between one marker and the next becoming that cell's content. BUILTIN_METRICS must be a Python list of strings, e.g. ["Faithfulness", "Correctness"].

Step 19: Post-generation

Notebook mode:

To run:
1. Cell 1 — configuration and SDK install
2. Cell 2 — start evaluation
3. Cell 3 — polls status automatically (~25-60 min)
4. Cell 4 — show results

Script mode:

Evaluation can take hours depending on your dataset. Present the user with options:

"Would you like me to:

>

1. Leave it to you — run with python scripts/[script_name]

2. Run it and wait until it's done

3. Start it but don't wait — we can check status later"

• Option 1: Done. Wait for user to come back.
• Option 2: Execute the script as-is. execution.wait() polls until complete. Report results.
• Option 3: Remove the execution.wait() call, execute, report the evaluation ARN.

Note: evaluate() does not accept a wait parameter. It always returns immediately. Blocking is done via execution.wait(target_status="Succeeded").

Checking status:

• describe-pipeline-execution --pipeline-execution-arn ARN → PipelineExecutionStatus
• list-pipeline-execution-steps --pipeline-execution-arn ARN → per-step StepStatus, FailureReason

Showing results after completion:

• Run: EvaluationPipelineExecution.get(arn=ARN).show_results()

FAQ

Q: Can I combine custom and built-in metrics in the same evaluation? A: Yes. You can use up to 10 custom metrics alongside any number of built-in metrics in a single evaluation job.

Troubleshooting

Evaluation job fails with "access denied when attempting to assume role"

The Bedrock evaluation job needs to assume your IAM role, which requires bedrock.amazonaws.com in the role's trust policy. This is common when running from a local IDE with temporary or SSO credentials.

To check, inspect your current role's trust policy using the AWS MCP tool:

1. Use the AWS MCP tool get-caller-identity (STS service) to get your current role ARN. 2. Extract the role name from the ARN (the part after role/ or assumed-role/). 3. Use the AWS MCP tool get-role (IAM service) with the role name, and extract Role.AssumeRolePolicyDocument from the response.

Look for bedrock.amazonaws.com in Principal.Service. If it's missing, either add it to the trust policy or switch to a role that already trusts Bedrock (e.g., your SageMaker execution role).

Helping a user find their Model Package ARN

If the user doesn't know their model package ARN and can only provide partial info (dataset ARN, training job name, etc.), guide them through these steps:

1. Ask for keywords from the model or training job name (e.g., "medication-simplification"). 2. Search model package groups via the AWS tool: list-model-package-groups with name-contains <keyword>. 3. List packages in the group via the AWS tool: list-model-packages with the group name. 4. Verify the match via the AWS tool: describe-model-package with the ARN. Check that the S3Uri in InferenceSpecification.Containers matches the expected training output path.

Always confirm the resolved ARN with the user before proceeding.

Supported Judge Models

Reference: Amazon Bedrock LLM-as-Judge Evaluation

Allowed Judge Models

The SageMaker Python SDK validates the judge model against a hardcoded allowlist before submitting the evaluation job. Only these models are accepted:

This list applies to both built-in and custom metrics — the SDK does not distinguish between them.

Source: sagemaker.train.constants._ALLOWED_EVALUATOR_MODELS (sagemaker SDK v3)

Selection Guidance

Verify each candidate is active in the user's region. Use the AWS MCP tool get-foundation-model (Bedrock service) with the model identifier and region. Extract modelDetails.modelLifecycle.status from the response.

Only include models that return ACTIVE. Models marked LEGACY will fail at evaluation time.

Present all active models to the user and let them choose. NEVER recommend or suggest any particular model. Only display the list. If the user asks for guidance, you may share these general trade-offs so they can decide:

• Cost vs quality: Smaller models are faster and cheaper; larger models produce higher-quality judgments
• Task complexity: Simple tasks (QA, classification) may not need the most capable model; complex reasoning (math, multi-step) benefits from stronger models