Microsoft Extensions Ai

Microsoft.Extensions.AI

Trigger On

• building or reviewing .NET code that uses Microsoft.Extensions.AI, Microsoft.Extensions.AI.Abstractions, IChatClient, IEmbeddingGenerator, ChatOptions, or AIFunction
• adding IImageGenerator, local-model chat via Ollama, AI app templates, or the .NET AI quickstarts for assistants and MCP
• choosing between low-level AI abstractions, provider SDKs, vector-search composition, evaluation libraries, and a fuller agent framework
• adding streaming chat, structured output, embeddings, tool calling, telemetry, caching, or DI-based AI middleware
• wiring Microsoft.Extensions.VectorData, Microsoft.Extensions.DataIngestion, MCP tooling, or evaluation packages around a provider-agnostic AI app

Workflow

1. Classify the request first: plain model access, tool calling, embeddings/vector search, evaluation, image generation, local-model prototyping, MCP bootstrap, or true agent orchestration. 2. Default to Microsoft.Extensions.AI for application and service code that needs provider-agnostic chat, embeddings, middleware, structured output, and testability. 3. Reference Microsoft.Extensions.AI.Abstractions directly only when authoring provider libraries or lower-level reusable integration packages. 4. Model IChatClient and IEmbeddingGenerator composition explicitly in DI. Keep options, caching, telemetry, logging, and tool invocation inspectable in the pipeline. 5. Treat chat state deliberately. For stateless providers, resend history. For stateful providers, propagate ConversationId rather than assuming all providers behave the same way. 6. Use Microsoft.Extensions.VectorData and Microsoft.Extensions.DataIngestion as adjacent building blocks for RAG instead of hand-rolling store abstractions prematurely. Treat the embedding model, vector dimensions, and collection schema as one owned contract: changing any of them means reindexing rather than reusing old vector data. Keep vector API source-breaking notes version-aware; in the 10.5+ line, named-argument usage of VectorStoreVectorAttribute uses dimensions:. 7. Treat the .NET AI quickstarts as bootstrap paths, not finished architecture. They now cover minimal assistants, MCP client/server flows, local models, app templates, and image generation. Start there for a vertical slice, then harden the DI, telemetry, and evaluation story here. 8. Escalate to microsoft-agent-framework when the requirement becomes agent threads, multi-agent orchestration, higher-order workflows, durable execution, or remote agent hosting. 9. Validate with real providers, realistic prompts, and evaluation gates so the abstraction layer actually buys portability and reliability.

Architecture

flowchart LR
  A["Task"] --> B{"Need agent threads, multi-agent orchestration, or remote agent hosting?"}
  B -->|Yes| C["Use Microsoft Agent Framework on top of `Microsoft.Extensions.AI.Abstractions`"]
  B -->|No| D{"Need provider-agnostic chat, embeddings, tools, typed output, or evaluation?"}
  D -->|Yes| E["Use `Microsoft.Extensions.AI`"]
  E --> F["Compose `IChatClient` / `IEmbeddingGenerator` in DI"]
  F --> G["Add caching, telemetry, tools, vector data, and evaluation deliberately"]
  D -->|No| H["Use plain provider SDKs or deterministic .NET code"]

Core Knowledge

• Microsoft.Extensions.AI.Abstractions contains the core exchange contracts such as IChatClient, IEmbeddingGenerator<TInput, TEmbedding>, message/content types, and tool abstractions.
• Microsoft.Extensions.AI adds the higher-level application surface: middleware builders, automatic function invocation, caching, logging, and OpenTelemetry integration.
• Most apps and services should reference Microsoft.Extensions.AI; provider and connector libraries usually reference only the abstractions package.
• IChatClient centers on GetResponseAsync and GetStreamingResponseAsync. The returned ChatResponse or ChatResponseUpdate objects carry messages, tool-related content, metadata, and optional conversation identifiers.
• Local-model quickstarts still route through the same IChatClient abstraction. Ollama-backed clients are useful for low-cost prototyping, offline dev loops, and portability testing, but you still own chat history replay, latency, and model-quality tradeoffs.
• ChatOptions is the normal control plane for model ID, temperature, tools, AdditionalProperties, and provider-specific raw options.
• Tool calling is modeled with AIFunction, AIFunctionFactory, and FunctionInvokingChatClient. Ambient data can flow through closures, AdditionalProperties, AIFunctionArguments.Context, or DI.
• Tool calling can target local .NET methods, external APIs, or MCP-backed tools. The model requests calls; your app still owns execution, validation, and side-effect boundaries.
• Tool definitions consume request tokens. Keep tool descriptions short and register only the tools relevant for the current conversation or workflow.
• FunctionInvokingChatClient can handle the tool-invocation loop and parallel tool-call responses automatically when the provider/model supports that shape.
• IEmbeddingGenerator is the standard abstraction for semantic search, vector indexing, similarity, and cache-key generation. Pair it with Microsoft.Extensions.VectorData.Abstractions for vector store operations, and keep the embedding model, collection dimensions, and chunking/versioning story aligned so reindexing stays explicit.
• IImageGenerator is the experimental MEAI image surface. Treat MEAI001 as an intentional opt-in, keep image generation separate from chat concerns, and compose logging/caching/hosting middleware around it the same way you would for IChatClient.
• Microsoft.Extensions.DataIngestion gives you the document-side RAG pipeline: IngestionDocument, document readers like MarkItDown/Markdig, document processors such as ImageAlternativeTextEnricher, chunkers, chunk processors, VectorStoreWriter<T>, and IngestionPipeline<T> for end-to-end composition.
• IngestionPipeline<T>.ProcessAsync is partial-success oriented. Handle IAsyncEnumerable<IngestionResult> deliberately instead of assuming one failed document should automatically crash the whole ingestion run.
• Microsoft.Extensions.AI.Evaluation.* gives you quality, NLP, safety, caching, and reporting layers for regression checks and CI gates.
• In dotnet/extensions v10.7.0, Microsoft.Extensions.AI.OpenAI moves to OpenAI 2.11.0, ToolJson.AdditionalProperties correctly preserves sub-schema objects, and HostedFileContent.SizeInBytes / CreatedAt are stable. Remove any local workaround that normalized old ToolApprovalResponseContent.InformationalOnly history only after checking serialized approval sessions.
• The official .NET AI docs now make MCP, assistants, local models, templates, and text-to-image part of the same app-level story. Use mcp when the protocol itself becomes the design problem; stay here when you still mostly need app composition around IChatClient and friends.
• Microsoft Agent Framework builds on these abstractions. Use it when you need autonomous orchestration, threads, workflows, hosting, or multi-agent collaboration instead of just model composition.

Decision Cheatsheet

Common Failure Modes

• Referencing only Microsoft.Extensions.AI.Abstractions in an app and then rebuilding middleware, telemetry, or function invocation by hand.
• Treating IChatClient as if it already gives you durable agent threads, orchestration, or hosted-agent semantics.
• Mixing provider-specific assistants APIs with IChatClient as if they were the same runtime contract.
• Forgetting to distinguish stateless history replay from stateful ConversationId flows.
• Hiding important chat behavior in singleton service fields instead of explicit message history, options, or persistent storage.
• Adding tool calling without validating parameter binding, invalid input behavior, side effects, or DI-scoped dependencies.
• Building RAG without stable chunking, embedding-model/version tracking, or vector dimension discipline.
• Shipping AI features without evaluation baselines, safety checks, or telemetry for prompt/model drift.

Deliver

• a justified package and abstraction choice: Abstractions only vs full Microsoft.Extensions.AI
• a concrete IChatClient / IEmbeddingGenerator composition strategy
• explicit tool-calling, options, state, caching, logging, and telemetry decisions
• vector-search, evaluation, or MCP integration guidance when the scenario needs it
• a clear escalation path to Agent Framework when the problem exceeds provider abstraction

Validate

• the abstraction layer solves a real portability, testability, or composition problem
• provider registration and middleware order stay explicit in DI
• chat state management matches whether the provider is stateless or stateful
• structured output, tool invocation, and embedding flows are typed and observable
• vector store, embedding model, and chunking strategy are consistent
• evaluation or safety gates exist for important prompts and agent-like behaviors
• agentic requirements are not being under-modeled as a simple IChatClient integration

When exact wording, edge-case API behavior, or less-common examples matter, check the local official docs snapshot before relying on summaries.

References

• official-docs-index.md - Slim local snapshot map with direct links to every mirrored .NET AI docs page plus API-reference pointers
• patterns.md - Package choice, IChatClient, embeddings, DI pipelines, tool-calling, and Agent Framework escalation guidance
• examples.md - Quickstart-to-task map covering chat, structured output, function calling, vector search, local models, MCP, and assistants
• evaluation.md - Quality, NLP, safety, caching, reporting, and CI-oriented evaluation guidance

Microsoft.Extensions.AI Evaluation

Package Set

Choose Evaluators By Risk

Quality

Use these when answer quality or agent behavior matters:

• RelevanceEvaluator
• CompletenessEvaluator
• RetrievalEvaluator
• FluencyEvaluator
• CoherenceEvaluator
• EquivalenceEvaluator
• GroundednessEvaluator
• IntentResolutionEvaluator
• TaskAdherenceEvaluator
• ToolCallAccuracyEvaluator

NLP

Use these when you already have reference answers and need cheaper deterministic comparisons:

• BLEUEvaluator
• GLEUEvaluator
• F1Evaluator

Safety

Use these when harmful output, prompt attacks, or unsafe code are part of the release risk:

• ContentHarmEvaluator
• ProtectedMaterialEvaluator
• GroundednessProEvaluator
• UngroundedAttributesEvaluator
• HateAndUnfairnessEvaluator
• SelfHarmEvaluator
• ViolenceEvaluator
• SexualEvaluator
• CodeVulnerabilityEvaluator
• IndirectAttackEvaluator

Practical Evaluation Loop

1. Pick a stable prompt or scenario set that represents the real feature. 2. Decide whether the gate is about answer quality, tool behavior, safety, or all three. 3. Use the same IChatClient-backed app surface that production uses, or a controlled test double when you are isolating logic. 4. Cache responses for repeatability and lower cost. 5. Store results and publish reports so model, prompt, or middleware changes are comparable across runs.

CI Guidance

• Use NLP evaluators for low-cost baseline checks on every PR when reference outputs exist.
• Use quality evaluators on targeted, high-value scenarios such as retrieval, summarization, tool use, or task adherence.
• Use safety evaluators for user-facing or code-producing features before release.
• Track threshold changes deliberately; do not quietly relax gates when a prompt or model regresses.

Agent-Oriented Checks

Even if the app is not using full Agent Framework, agent-like workflows often need:

• IntentResolutionEvaluator when the system has to understand and complete multi-step user requests
• TaskAdherenceEvaluator when the system receives bounded instructions or policies
• ToolCallAccuracyEvaluator when local functions or MCP-backed tools are part of the flow

These metrics are often the first place where prompt drift or tool-schema changes show up.

Reporting And Caching

• The libraries support response caching so unchanged prompt-model combinations can reuse prior results.
• Reporting packages let you persist evaluation data and generate human-readable reports.
• The dotnet aieval CLI is useful for report generation and cache management in local runs or CI pipelines.

Common Failure Modes

• Evaluating only one happy-path prompt instead of the real scenario envelope.
• Comparing outputs without fixing the prompt, grounding data, or model selection.
• Treating evaluation as a one-time benchmark instead of a regression suite.
• Shipping tool-using or RAG features without measuring task adherence, groundedness, or tool accuracy.

Microsoft.Extensions.AI Practical Examples

Quickstart-To-Task Map

Recommended Composition Recipes

Provider-Agnostic App

• Register one or more IChatClient implementations in DI.
• Add options configuration, logging or telemetry, caching, and function invocation in a deliberate builder order.
• Keep feature code dependent on IChatClient, not the vendor SDK, unless you truly need provider-specific capabilities.

Typed Chat + Tools

• Use GetResponseAsync<T> or the equivalent typed helpers for structured output.
• Give the model a narrow result shape and a narrow tool surface.
• Route ambient tool data through AdditionalProperties, AIFunctionArguments, or DI instead of serializing hidden state into prompts.

Vector Search / RAG

• Use IEmbeddingGenerator<string, Embedding<float>> to create embeddings for both source content and user queries.
• Store vectors in a vector store accessed through Microsoft.Extensions.VectorData.Abstractions.
• Keep ingestion, chunking, and retrieval policies versioned so evaluation results stay meaningful over time.

Data Ingestion for RAG

• Start from Microsoft.Extensions.DataIngestion when documents must be read, normalized, enriched, chunked, and written as one pipeline instead of a pile of custom helpers.
• Reach for the official processing shape:
• document reader such as MarkItDown or Markdig
• optional document processor such as ImageAlternativeTextEnricher
• chunker such as HeaderChunker or semantic chunking
• chunk processors such as SummaryEnricher
• VectorStoreWriter<T> and IngestionPipeline<T> for the final persisted flow
• Handle ProcessAsync results per document. A single ingestion failure should be an explicit policy decision, not an accidental crash.

Evaluation-Backed Delivery

• Add quality and safety evaluators for important prompts and user journeys.
• Run cheap NLP evaluators for stable offline comparisons when you have reference outputs.
• Publish reports and reuse cached evaluation responses in CI so the team can compare prompt or model changes.

Important Boundaries

• Microsoft.Extensions.AI is ideal for provider abstraction, middleware, embeddings, evaluation, and typed tool calling.
• Provider-hosted assistants APIs are adjacent but not identical to IChatClient composition.
• When the app needs threads, multi-agent orchestration, or durable workflow control, hand off to microsoft-agent-framework.

Official Docs Index

This skill keeps a slim, markdown-only snapshot of the official .NET AI docs tree from dotnet/docs under docs/ai.

Snapshot Summary

• Local root: references/official-docs/
• Coverage: 48 useful markdown pages
• Scope: Microsoft.Extensions.AI, adjacent VectorData and DataIngestion guidance, evaluation libraries, MCP quickstarts, RAG guidance, and the surrounding .NET AI concept pages
• Boundary: Microsoft Agent Framework is linked from this docs tree, but its dedicated authored snapshot and deeper routing guidance live in the separate microsoft-agent-framework skill
• Intentional exclusions: snippet trees, project files, TOC scaffolding, DocFX support files, JSON helpers, media folders, and other low-signal assets are not mirrored into the skill

Start Here

• `official-docs/overview.md` - Root .NET AI landing page
• `official-docs/dotnet-ai-ecosystem.md` - Ecosystem map and the official boundary between Microsoft.Extensions.AI and Agent Framework
• `official-docs/microsoft-extensions-ai.md` - Package split and core API overview
• `official-docs/ichatclient.md` - Chat, streaming, tools, caching, telemetry, DI, and state handling
• `official-docs/iembeddinggenerator.md` - Embeddings, delegating generators, and implementation guidance

Section Map

• Root pages: overview.md, dotnet-ai-ecosystem.md, microsoft-extensions-ai.md, ichatclient.md, iembeddinggenerator.md, get-started-mcp.md, get-started-app-chat-template.md, get-started-app-chat-scaling-with-azure-container-apps.md, azure-ai-services-authentication.md
• Concepts: `official-docs/conceptual/` with 11 pages covering agents, tools, tokens, embeddings, vector databases, ingestion, prompt engineering, zero-shot and few-shot, chain-of-thought, and RAG
• Quickstarts: `official-docs/quickstarts/` with 14 pages covering prompting, chat apps, structured output, vector search, function calling, local models, assistants, MCP client and server, templates, text-to-image, and data processing
• How-to: `official-docs/how-to/` with 5 pages covering function data access, invalid tool input, content filtering, Azure-hosted auth, and tokenizers
• Evaluation: `official-docs/evaluation/` with 5 pages covering responsible AI, libraries, response quality, reporting, and safety evaluation
• Resources: `official-docs/resources/` with 3 pages for general .NET AI, Azure AI, and MCP resource lists
• Tutorial: `official-docs/tutorials/tutorial-ai-vector-search.md` for the deeper vector-search walkthrough

Complete Local File Map

Root Pages

• `official-docs/azure-ai-services-authentication.md`
• `official-docs/dotnet-ai-ecosystem.md`
• `official-docs/get-started-app-chat-scaling-with-azure-container-apps.md`
• `official-docs/get-started-app-chat-template.md`
• `official-docs/get-started-mcp.md`
• `official-docs/ichatclient.md`
• `official-docs/iembeddinggenerator.md`
• `official-docs/microsoft-extensions-ai.md`
• `official-docs/overview.md`

Conceptual

• `official-docs/conceptual/agents.md`
• `official-docs/conceptual/ai-tools.md`
• `official-docs/conceptual/chain-of-thought-prompting.md`
• `official-docs/conceptual/data-ingestion.md`
• `official-docs/conceptual/embeddings.md`
• `official-docs/conceptual/how-genai-and-llms-work.md`
• `official-docs/conceptual/prompt-engineering-dotnet.md`
• `official-docs/conceptual/rag.md`
• `official-docs/conceptual/understanding-tokens.md`
• `official-docs/conceptual/vector-databases.md`
• `official-docs/conceptual/zero-shot-learning.md`

How-To

• `official-docs/how-to/access-data-in-functions.md`
• `official-docs/how-to/app-service-aoai-auth.md`
• `official-docs/how-to/content-filtering.md`
• `official-docs/how-to/handle-invalid-tool-input.md`
• `official-docs/how-to/use-tokenizers.md`

Quickstarts

• `official-docs/quickstarts/ai-templates.md`
• `official-docs/quickstarts/build-chat-app.md`
• `official-docs/quickstarts/build-mcp-client.md`
• `official-docs/quickstarts/build-mcp-server.md`
• `official-docs/quickstarts/build-vector-search-app.md`
• `official-docs/quickstarts/chat-local-model.md`
• `official-docs/quickstarts/create-assistant.md`
• `official-docs/quickstarts/generate-images.md`
• `official-docs/quickstarts/process-data.md`
• `official-docs/quickstarts/prompt-model.md`
• `official-docs/quickstarts/publish-mcp-registry.md`
• `official-docs/quickstarts/structured-output.md`
• `official-docs/quickstarts/text-to-image.md`
• `official-docs/quickstarts/use-function-calling.md`

Evaluation

• `official-docs/evaluation/evaluate-ai-response.md`
• `official-docs/evaluation/evaluate-safety.md`
• `official-docs/evaluation/evaluate-with-reporting.md`
• `official-docs/evaluation/libraries.md`
• `official-docs/evaluation/responsible-ai.md`

Resources

• `official-docs/resources/azure-ai.md`
• `official-docs/resources/get-started.md`
• `official-docs/resources/mcp-servers.md`

Tutorials

• `official-docs/tutorials/tutorial-ai-vector-search.md`

API Reference Landing Pages

• https://learn.microsoft.com/dotnet/api/microsoft.extensions.ai
• https://learn.microsoft.com/dotnet/api/microsoft.extensions.vectordata
• https://learn.microsoft.com/dotnet/api/microsoft.extensions.dataingestion

Reading Strategy

• Use the local snapshot when exact wording, package names, or Learn-page structure matters.
• Start with the authored overview pages before diving into provider-specific quickstarts.
• Raw Learn :::code and :::image source-asset directives are stripped from the local snapshot to keep it prose-first and avoid broken local references.
• For orchestration, threads, workflows, or hosted-agent protocols, switch to the microsoft-agent-framework skill rather than assuming the answer lives in the Microsoft.Extensions.AI layer.

Foundry tools authentication and authorization using .NET

Application requests to Microsoft Foundry tools must be authenticated. In this article, you explore the options available to authenticate to Azure OpenAI and other Foundry tools using .NET. Most Foundry tools offer two primary ways to authenticate apps and users:

• Key-based authentication provides access to an Azure service using secret key values. These secret values are sometimes known as API keys or access keys depending on the service.
• Microsoft Entra ID provides a comprehensive identity and access management solution to ensure that the correct identities have the correct level of access to different Azure resources.

The sections ahead provide conceptual overviews for these two approaches, rather than detailed implementation steps. For more detailed information about connecting to Azure services, visit the following resources:

• Authenticate .NET apps to Azure services
• Identity fundamentals
• What is Azure RBAC?

[!NOTE]

The examples in this article focus primarily on connections to Azure OpenAI, but the same concepts and implementation steps directly apply to many other Foundry tools as well.

Authentication using keys

Access keys allow apps and tools to authenticate to a Foundry tool, such as Azure OpenAI, using a secret key provided by the service. Retrieve the secret key using tools such as the Azure portal or Azure CLI and use it to configure your app code to connect to the Foundry tool:

builder.Services.AddAzureOpenAIChatCompletion(
    "deployment-model",
    "service-endpoint",
    "service-key"); // Secret key
var kernel = builder.Build();

Keys are straightforward to use, but treat them with caution. Keys aren't the recommended authentication option because they:

• Don't follow the principle of least privilege. They provide elevated permissions regardless of who uses them or for what task.
• Can accidentally end up in source control or unsafe storage locations.
• Can easily be shared with or sent to parties who shouldn't have access.
• Often require manual administration and rotation.

Instead, consider using Microsoft Entra ID for authentication, which is the recommended solution for most scenarios.

Authentication using Microsoft Entra ID

Microsoft Entra ID is a cloud-based identity and access management service that provides a vast set of features for different business and app scenarios. Microsoft Entra ID is the recommended solution to connect to Azure OpenAI and other Foundry tools and provides the following benefits:

• Keyless authentication using identities.
• Role-based access control (RBAC) to assign identities the minimum required permissions.
• Lets you use the `Azure.Identity` client library to detect different credentials across environments without requiring code changes.
• Automatically handles administrative maintenance tasks such as rotating underlying keys.

The workflow to implement Microsoft Entra authentication in your app generally includes the following steps:

• Local development:

1. Sign-in to Azure using a local dev tool such as the Azure CLI or Visual Studio. 1. Configure your code to use the `Azure.Identity` client library and DefaultAzureCredential class. 1. Assign Azure roles to the account you signed-in with to enable access to the Foundry tool.

• Azure-hosted app:

1. Deploy the app to Azure after configuring it to authenticate using the Azure.Identity client library. 1. Assign a managed identity to the Azure-hosted app. 1. Assign Azure roles to the managed identity to enable access to the Foundry tool.

The key concepts of this workflow are explored in the following sections.

Authenticate to Azure locally

When developing apps locally that connect to Foundry tools, authenticate to Azure using a tool such as Visual Studio or the Azure CLI. Your local credentials can be discovered by the Azure.Identity client library and used to authenticate your app to Azure services, as described in the Configure the app code section.

For example, to authenticate to Azure locally using the Azure CLI, run the following command:

az login

Configure the app code

Use the `Azure.Identity` client library from the Azure SDK to implement Microsoft Entra authentication in your code. The Azure.Identity libraries include the DefaultAzureCredential class, which automatically discovers available Azure credentials based on the current environment and tooling available. For the full set of supported environment credentials and the order in which DefaultAzureCredential searches them, see the Azure SDK for .NET documentation.

For example, configure Azure OpenAI to authenticate using DefaultAzureCredential using the following code:

AzureOpenAIClient azureClient =
    new(
        new Uri(endpoint),
        new DefaultAzureCredential(new DefaultAzureCredentialOptions()
            { TenantId = tenantId }
        )
    );

DefaultAzureCredential enables apps to be promoted from local development to production without code changes. For example, during development DefaultAzureCredential uses your local user credentials from Visual Studio or the Azure CLI to authenticate to the Foundry tool. When the app is deployed to Azure, DefaultAzureCredential uses the managed identity that is assigned to your app.

Assign roles to your identity

Azure role-based access control (Azure RBAC) is a system that provides fine-grained access management of Azure resources. Assign a role to the security principal used by DefaultAzureCredential to connect to a Foundry tool, whether that's an individual user, group, service principal, or managed identity. Azure roles are a collection of permissions that allow the identity to perform various tasks, such as generate completions or create and delete resources.

Assign roles such as Cognitive Services OpenAI User (role ID: 5e0bd9bd-7b93-4f28-af87-19fc36ad61bd) to the relevant identity using tools such as the Azure CLI, Bicep, or the Azure portal. For example, use the az role assignment create command to assign a role using the Azure CLI:

az role assignment create \
        --role "5e0bd9bd-7b93-4f28-af87-19fc36ad61bd" \
        --assignee-object-id "$PRINCIPAL_ID" \
        --scope /subscriptions/"$SUBSCRIPTION_ID"/resourceGroups/"$RESOURCE_GROUP" \
        --assignee-principal-type User

Learn more about Azure RBAC using the following resources:

• What is Azure RBAC?
• Grant a user access
• RBAC best practices

Assign a managed identity to your app

In most scenarios, Azure-hosted apps should use a managed identity to connect to other services such as Azure OpenAI. Managed identities provide a fully managed identity in Microsoft Entra ID for apps to use when connecting to resources that support Microsoft Entra authentication. DefaultAzureCredential discovers the identity associated with your app and uses it to authenticate to other Azure services.

There are two types of managed identities you can assign to your app:

• A system-assigned identity is tied to your application and is deleted if your app is deleted. An app can only have one system-assigned identity.
• A user-assigned identity is a standalone Azure resource that can be assigned to your app. An app can have multiple user-assigned identities.

Assign roles to a managed identity just like you would an individual user account, such as the Cognitive Services OpenAI User role. Learn more about working with managed identities using the following resources:

• Managed identities overview
• Authenticate App Service to Azure OpenAI using Microsoft Entra ID
• How to use managed identities for App Service and Azure Functions

Agents

This article introduces the core concepts behind agents, why they matter, and how they fit into workflows, setting you up to get started building agents in .NET.

What are agents?

Agents are systems that accomplish objectives.

Agents become more capable when equipped with the following:

• Reasoning and decision-making: Powered by LLMs, search algorithms, or planning and decision-making systems.
• Tool usage: Access to Model Context Protocol (MCP) servers, code execution, and external APIs.
• Context awareness: Informed by chat history, threads, vector stores, enterprise data, or knowledge graphs.

These capabilities allow agents to operate more autonomously, adaptively, and intelligently.

What are workflows?

As objectives grow in complexity, they need to be broken down into manageable steps. That's where workflows come in.

Workflows define the sequence of steps required to achieve an objective.

Imagine you're launching a new feature on your business website. If it's a simple update, you might go from idea to production in a few hours. But for more complex initiatives, the process might include:

• Requirement gathering
• Design and architecture
• Implementation
• Testing
• Deployment

A few important observations:

• Each step might contain subtasks.
• Different specialists might own different phases.
• Progress isn’t always linear. Bugs found during testing might send you back to implementation.
• Success depends on planning, orchestration, and communication across stakeholders.

Agents + workflows = agentic workflows

Workflows don't require agents, but agents can supercharge them.

When agents are equipped with reasoning, tools, and context, they can optimize workflows.

This is the foundation of multi-agent systems, where agents collaborate within workflows to achieve complex goals.

Workflow orchestration

Agentic workflows can be orchestrated in a variety of ways. The following are a few of the most common:

• Sequential
• Concurrent
• Handoff
• Group chat
• Magentic

Sequential

Agents process tasks one after another, passing results forward.

Concurrent

Agents work in parallel, each handling different aspects of the task.

Handoff

Responsibility shifts from one agent to another based on conditions or outcomes.

Group chat

Agents collaborate in a shared conversation, exchanging insights in real-time.

Magentic

A lead agent directs other agents.

How can I get started building agents in .NET?

The building blocks in <xref:Microsoft.Extensions.AI> and <xref:Microsoft.Extensions.VectorData> supply the foundations for agents by providing modular components for AI models, tools, and data.

These components serve as the foundation for Microsoft Agent Framework. For more information, see Microsoft Agent Framework.

AI tool calling

Tool calling is an AI model capability that lets you describe available tools to an AI model so the model can request that your application invoke them. Tools can be .NET methods, calls to external APIs, interactions with Model Context Protocol (MCP) servers, or any other executable operation. Instead of directly executing those tools, the model returns a structured output describing which tools to call and with what arguments. Your application invokes those tools and sends the results back to the model, enabling it to build a more accurate and grounded response.

<xref:Microsoft.Extensions.AI> (MEAI) provides provider-agnostic abstractions for tool calling that work across AI services, including Azure OpenAI, OpenAI, Ollama, and others. You write your tool-calling logic once, and it works regardless of which underlying model or provider you use.

Why use tool calling

Tool calling simplifies how you connect external tools to AI models. You describe each tool to the model as part of the conversation. The model then decides which tools to invoke based on the user's question. After your application invokes the requested tools and returns the results, the model uses those results to construct a more complete and accurate response.

Common use cases for tool calling include:

• Answering questions by calling external APIs. For example, checking the weather forecast, or sending email.
• Retrieving information from internal data stores. For example, aggregating sales data to answer, "What are my best-selling products?"
• Producing structured data from unstructured text. For example, constructing a user profile from chat history.

Call AI functions in MEAI

The general flow for calling AI functions with <xref:Microsoft.Extensions.AI.IChatClient> is:

1. Define .NET methods as functions and configure them on a <xref:Microsoft.Extensions.AI.ChatOptions> instance. 1. Send the user's message to the model. The model decides which functions, if any, to call. It returns a structured response that lists the function calls and their arguments.

[!NOTE]

Models might hallucinate arguments that weren't described in your function definitions.

1. Parse the model's response and invoke the requested functions with the specified arguments. 1. Send another request that includes the function results as new messages in the conversation history. 1. The model responds with more function call requests or a final answer to the user's question. Continue invoking requested functions until the model provides a final response.

MEAI's <xref:Microsoft.Extensions.AI.FunctionInvokingChatClient> handles steps 3 through 5 automatically, so you don't need to manage the invocation loop yourself.

Key types

MEAI provides the following types to support function calling:

• <xref:Microsoft.Extensions.AI.AIFunction>: Represents a function that can be described to an AI model, and invoked. This is the core abstraction for a function in MEAI.
• <xref:Microsoft.Extensions.AI.AIFunctionFactory>: Provides factory methods for creating AIFunction instances from .NET methods. Use AIFunctionFactory to wrap existing methods as functions without writing boilerplate description or argument-parsing code.
• <xref:Microsoft.Extensions.AI.FunctionInvokingChatClient>: Wraps any IChatClient and adds automatic function-invocation capabilities. When the model requests a function call, FunctionInvokingChatClient invokes the corresponding AIFunction, collects the result, and continues the conversation—all transparently.

Parallel function calling

Some models support parallel function calling, where the model requests multiple function invocations in a single response. Your application invokes each function and returns all results together in one follow-up message. Parallel function calling reduces the number of round trips to the model, which lowers latency and API usage. FunctionInvokingChatClient supports parallel function calling automatically.

Cross-provider support

One of the key benefits of using MEAI for function calling is provider independence. The AIFunction, AIFunctionFactory, and FunctionInvokingChatClient types work with any IChatClient implementation, including:

• Azure OpenAI
• OpenAI
• Ollama
• Any other provider that implements IChatClient

Because function calling support varies across models and providers, check your provider's documentation to confirm whether a specific model supports function calling or parallel function calling.

Token considerations

Tool descriptions are included in the request sent to the model and count against the model's token limit. This means tool definitions contribute to both token consumption and request cost.

If your request approaches the model's token limit, consider these adjustments:

• Reduce the number of tools registered for the conversation.
• Shorten the method names and descriptions used to generate tool definitions.
• Limit tool registration to only the tools relevant for a given conversation context.

Related content

• Invoke .NET functions using an AI model
• Use the IChatClient interface
• Understanding tokens
• Prompt engineering

Chain of thought prompting

GPT model performance and response quality benefit from prompt engineering, which is the practice of providing instructions and examples to a model to prime or refine its output. As they process instructions, models make more reasoning errors when they try to answer right away rather than taking time to work out an answer. Help the model reason its way toward correct answers more reliably by asking the model to include its chain of thought&mdash;that is, the steps it took to follow an instruction, along with the results of each step.

Chain of thought prompting is the practice of prompting a model to perform a task step-by-step and to present each step and its result in order in the output. This simplifies prompt engineering by offloading some execution planning to the model, and makes it easier to connect any problem to a specific step so you know where to focus further efforts.

It's generally simpler to instruct the model to include its chain of thought, but you can also use examples to show the model how to break down tasks. The following sections show both ways.

Use chain of thought prompting in instructions

To use an instruction for chain of thought prompting, include a directive that tells the model to perform the task step-by-step and to output the result of each step.

prompt= """Instructions: Compare the pros and cons of EVs and petroleum-fueled vehicles.
Break the task into steps, and output the result of each step as you perform it."""; 

Use chain of thought prompting in examples

Use examples to indicate the steps for chain of thought prompting, which the model interprets to mean it should also output step results. Steps can include formatting cues.

prompt= """
        Instructions: Compare the pros and cons of EVs and petroleum-fueled vehicles.

        Differences between EVs and petroleum-fueled vehicles:
        - 

        Differences ordered according to overall impact, highest-impact first: 
        1. 
        
        Summary of vehicle type differences as pros and cons:
        Pros of EVs
        1.
        Pros of petroleum-fueled vehicles
        1. 
        """;

Related content

• Prompt engineering techniques

Data ingestion

Data ingestion is the process of collecting, reading, and preparing data from different sources such as files, databases, APIs, or cloud services so it can be used in downstream applications. In practice, this process follows the Extract-Transform-Load (ETL) workflow:

• Extract data from its original source, whether that is a PDF, Word document, audio file, or web API.
• Transform the data by cleaning, chunking, enriching, or converting formats.
• Load the data into a destination like a database, vector store, or AI model for retrieval and analysis.

For AI and machine learning scenarios, especially Retrieval-Augmented Generation (RAG), data ingestion is not just about converting data from one format to another. It is about making data usable for intelligent applications. This means representing documents in a way that preserves their structure and meaning, splitting them into manageable chunks, enriching them with metadata or embeddings, and storing them so they can be retrieved quickly and accurately.

Why data ingestion matters for AI applications

Imagine you're building a RAG-powered chatbot to help employees find information across your company's vast collection of documents. These documents might include PDFs, Word files, PowerPoint presentations, and web pages scattered across different systems.

Your chatbot needs to understand and search through thousands of documents to provide accurate, contextual answers. But raw documents aren't suitable for AI systems. You need to transform them into a format that preserves meaning while making them searchable and retrievable.

This is where data ingestion becomes critical. You need to extract text from different file formats, break large documents into smaller chunks that fit within AI model limits, enrich the content with metadata, generate embeddings for semantic search, and store everything in a way that enables fast retrieval. Each step requires careful consideration of how to preserve the original meaning and context.

The Microsoft.Extensions.DataIngestion library

The 📦 Microsoft.Extensions.DataIngestion package provides foundational .NET building blocks for data ingestion. It enables developers to read, process, and prepare documents for AI and machine learning workflows, especially Retrieval-Augmented Generation (RAG) scenarios.

With these building blocks, you can create robust, flexible, and intelligent data ingestion pipelines tailored for your application needs:

• Unified document representation: Represent any file type (for example, PDF, Image, or Microsoft Word) in a consistent format that works well with large language models.
• Flexible data ingestion: Read documents from both cloud services and local sources using multiple built-in readers, making it easy to bring in data from wherever it lives.
• Built-in AI enhancements: Automatically enrich content with summaries, sentiment analysis, keyword extraction, and classification, preparing your data for intelligent workflows.
• Customizable chunking strategies: Split documents into chunks using token-based, section-based, or semantic-aware approaches, so you can optimize for your retrieval and analysis needs.
• Production-ready storage: Store processed chunks in popular vector databases and document stores, with support for embedding generation, making your pipelines ready for real-world scenarios.
• End-to-end pipeline composition: Chain together readers, processors, chunkers, and writers with the <xref:Microsoft.Extensions.DataIngestion.IngestionPipeline`1> API, reducing boilerplate and making it easy to build, customize, and extend complete workflows.
• Performance and scalability: Designed for scalable data processing, these components can handle large volumes of data efficiently, making them suitable for enterprise-grade applications.

All of these components are open and extensible by design. You can add custom logic and new connectors, and extend the system to support emerging AI scenarios. By standardizing how documents are represented, processed, and stored, .NET developers can build reliable, scalable, and maintainable data pipelines without "reinventing the wheel" for every project.

Built on stable foundations

These data ingestion building blocks are built on top of proven and extensible components in the .NET ecosystem, ensuring reliability, interoperability, and seamless integration with existing AI workflows:

• Microsoft.ML.Tokenizers: Tokenizers provide the foundation for chunking documents based on tokens. This enables precise splitting of content, which is essential for preparing data for large language models and optimizing retrieval strategies.
• Microsoft.Extensions.AI: This set of libraries powers enrichment transformations using large language models. It enables features like summarization, sentiment analysis, keyword extraction, and embedding generation, making it easy to enhance your data with intelligent insights.
• Microsoft.Extensions.VectorData: This set of libraries offers a consistent interface for storing processed chunks in a wide variety of vector stores, including Qdrant, Azure SQL, CosmosDB, MongoDB, ElasticSearch, and many more. This ensures your data pipelines are ready for production and can scale across different storage backends.

In addition to familiar patterns and tools, these abstractions build on already extensible components. Plug-in capability and interoperability are paramount, so as the rest of the .NET AI ecosystem grows, the capabilities of the data ingestion components grow as well. This approach empowers developers to easily integrate new connectors, enrichments, and storage options, keeping their pipelines future-ready and adaptable to evolving AI scenarios.

Data ingestion building blocks

The Microsoft.Extensions.DataIngestion library is built around several key components that work together to create a complete data processing pipeline. This section explores each component and how they fit together.

Documents and document readers

At the foundation of the library is the <xref:Microsoft.Extensions.DataIngestion.IngestionDocument> type, which provides a unified way to represent any file format without losing important information. IngestionDocument is Markdown-centric because large language models work best with Markdown formatting.

The <xref:Microsoft.Extensions.DataIngestion.IngestionDocumentReader> abstraction handles loading documents from various sources, whether local files or streams. A few readers are available:

• [MarkItDown](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion.MarkItDown)
• [Markdig](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion.Markdig/)

More readers (including LlamaParse and Azure Document Intelligence) will be added in the future.

This design means you can work with documents from different sources using the same consistent API, making your code more maintainable and flexible.

Document processing

Document processors apply transformations at the document level to enhance and prepare content. The library provides the <xref:Microsoft.Extensions.DataIngestion.ImageAlternativeTextEnricher> class as a built-in processor that uses large language models to generate descriptive alternative text for images within documents.

Chunks and chunking strategies

Once you have a document loaded, you typically need to break it down into smaller pieces called chunks. Chunks represent subsections of a document that can be efficiently processed, stored, and retrieved by AI systems. This chunking process is essential for retrieval-augmented generation scenarios where you need to find the most relevant pieces of information quickly.

The library provides several chunking strategies to fit different use cases:

• Header-based chunking to split on headers.
• Section-based chunking to split on sections (for example, pages).
• Semantic-aware chunking to preserve complete thoughts.

These chunking strategies build on the Microsoft.ML.Tokenizers library to intelligently split text into appropriately sized pieces that work well with large language models. The right chunking strategy depends on your document types and how you plan to retrieve information.

Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-5");
IngestionChunkerOptions options = new(tokenizer)
{
    MaxTokensPerChunk = 2000,
    OverlapTokens = 0
};
IngestionChunker<string> chunker = new HeaderChunker(options);

Chunk processing and enrichment

After documents are split into chunks, you can apply processors to enhance and enrich the content. Chunk processors work on individual pieces and can perform:

• Content enrichment including automatic summaries (SummaryEnricher), sentiment analysis (SentimentEnricher), and keyword extraction (KeywordEnricher).
• Classification for automated content categorization based on predefined categories (ClassificationEnricher).

These processors use Microsoft.Extensions.AI.Abstractions to leverage large language models for intelligent content transformation, making your chunks more useful for downstream AI applications.

Document writer and storage

<xref:Microsoft.Extensions.DataIngestion.IngestionChunkWriter1> stores processed chunks into a data store for later retrieval. Using Microsoft.Extensions.AI and [Microsoft.Extensions.VectorData.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions), the library provides the <xref:Microsoft.Extensions.DataIngestion.VectorStoreWriter1> class that supports storing chunks in any vector store supported by Microsoft.Extensions.VectorData.

Vector stores include popular options like Qdrant, SQL Server, CosmosDB, MongoDB, ElasticSearch, and many more. The writer can also automatically generate embeddings for your chunks using Microsoft.Extensions.AI, readying them for semantic search and retrieval scenarios.

OpenAIClient openAIClient = new(
    new ApiKeyCredential(Environment.GetEnvironmentVariable("GITHUB_TOKEN")!),
    new OpenAIClientOptions { Endpoint = new Uri("https://models.github.ai/inference") });

IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator =
    openAIClient.GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();

using SqliteVectorStore vectorStore = new(
    "Data Source=vectors.db;Pooling=false",
    new()
    {
        EmbeddingGenerator = embeddingGenerator
    });

// The writer requires the embedding dimension count to be specified.
// For OpenAI's `text-embedding-3-small`, the dimension count is 1536.
using VectorStoreWriter<string> writer = new(vectorStore, dimensionCount: 1536);

Document processing pipeline

The <xref:Microsoft.Extensions.DataIngestion.IngestionPipeline`1> API allows you to chain together the various data ingestion components into a complete workflow. You can combine:

• Readers to load documents from various sources.
• Processors to transform and enrich document content.
• Chunkers to break documents into manageable pieces.
• Writers to store the final results in your chosen data store.

This pipeline approach reduces boilerplate code and makes it easy to build, test, and maintain complex data ingestion workflows.

using IngestionPipeline<string> pipeline = new(reader, chunker, writer, loggerFactory: loggerFactory)
{
    DocumentProcessors = { imageAlternativeTextEnricher },
    ChunkProcessors = { summaryEnricher }
};

await foreach (var result in pipeline.ProcessAsync(new DirectoryInfo("."), searchPattern: "*.md"))
{
    Console.WriteLine($"Completed processing '{result.DocumentId}'. Succeeded: '{result.Succeeded}'.");
}

A single document ingestion failure shouldn't fail the whole pipeline. That's why <xref:Microsoft.Extensions.DataIngestion.IngestionPipeline1.ProcessAsync*?displayProperty=nameWithType> implements partial success by returning IAsyncEnumerable<IngestionResult>`. The caller is responsible for handling any failures (for example, by retrying failed documents or stopping on first error).

Embeddings in .NET

Embeddings are the way LLMs capture semantic meaning. They're numeric representations of non-numeric data that an LLM can use to determine relationships between concepts. Use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text or creating images from text descriptions. LLMs can use embeddings immediately, and you can store embeddings in vector databases to provide semantic memory for LLMs as needed.

Use cases for embeddings

Use your own data to improve completion relevance

Use your own databases to generate embeddings for your data and integrate it with an LLM to make it available for completions. This use of embeddings is an important component of retrieval-augmented generation.

Increase the amount of text you can fit in a prompt

Use embeddings to increase the amount of context you can fit in a prompt without increasing the number of tokens required.

For example, suppose you want to include 500 pages of text in a prompt. The number of tokens for that much raw text exceeds the input token limit, making it impossible to directly include in a prompt. You can use embeddings to summarize and break down large amounts of that text into pieces that are small enough to fit in one input, and then assess the similarity of each piece to the entire raw text. Then you can choose a piece that best preserves the semantic meaning of the raw text and use it in your prompt without hitting the token limit.

Perform text classification, summarization, or translation

Use embeddings to help a model understand the meaning and context of text, and then classify, summarize, or translate that text. For example, you can use embeddings to help models classify texts as positive or negative, spam or not spam, or news or opinion.

Generate and transcribe audio

Use audio embeddings to process audio files or inputs in your app.

For example, Azure Speech in Foundry Tools supports a range of audio embeddings, including speech to text and text to speech. You can process audio in real-time or in batches.

Turn text into images or images into text

Semantic image processing requires image embeddings, which most LLMs can't generate. Use an image-embedding model such as ViT to create vector embeddings for images. Then you can use those embeddings with an image generation model to create or modify images using text or vice versa. For example, you can use the DALL·E model to generate images such as logos, faces, animals, and landscapes.

Generate or document code

Use embeddings to help a model create code from text or vice versa, by converting different code or text expressions into a common representation. For example, you can use embeddings to help a model generate or document code in C# or Python.

Choose an embedding model

You generate embeddings for your raw data by using an AI embedding model, which can encode non-numeric data into a vector (a long array of numbers). The model can also decode an embedding into non-numeric data that has the same or similar meaning as the original, raw data. OpenAI's text-embedding-3-small and text-embedding-3-large are the currently recommended embedding models, replacing the older text-embedding-ada-002. For more examples, see the list of Embedding models available on Azure OpenAI.

Store and process embeddings in a vector database

After you generate embeddings, you need a way to store them so you can later retrieve them with calls to an LLM. Vector databases are designed to store and process vectors, so they're a natural home for embeddings. Different vector databases offer different processing capabilities. Choose one based on your raw data and your goals. For information about your options, see Vector databases for .NET + AI.

Using embeddings in your LLM solution

When building LLM-based applications, you can use Agent Framework to integrate embedding models and vector stores, so you can quickly pull in text data, and generate and store embeddings. This lets you use a vector database solution to store and retrieve semantic memories.

Related content

• How GenAI and LLMs work
• Retrieval-augmented generation
• Training: Develop an AI agent with Microsoft Agent Framework

How generative AI and LLMs work

Generative AI is a type of artificial intelligence that can create original content, such as natural language, images, audio, and code. The output depends on the inputs you provide. Users commonly interact with generative AI through chat applications that use natural language as input. ChatGPT, developed by OpenAI, is a popular example. Generative AI applications that use natural language as input are powered by large language models (LLMs) to perform natural language processing (NLP).

How generative AI works

All generative AI is built on models. These models are trained with large sets of data in the form of content, such as natural language, images, audio, and code. Generative AI models use the patterns identified in the training data to produce new, statistically similar content.

The AI model uses your input to build an output. The model first parses the input into a form it can understand. The model then uses that data to identify matching patterns from its training that it combines to build the final output. Generative AI models are designed to produce unique content, so they won't generate the same output for identical inputs.

Generative AI applications that support natural language as input or output use LLMs to do so. The LLM performs NLP, which classifies the input text and determines its sentiment. The generative AI model uses that classification and sentiment analysis to identify patterns and build the output. If the output is text, the LLM alone generates it. If the output is audio or images, additional models provide the data and patterns for generating outputs in that format.

Common uses of generative AI

Generative AI applications support a variety of use cases and outputs, described in the following sections.

Natural language generation

Some generative AI applications can produce natural language outputs, such as:

• Professionally worded summaries for a LinkedIn profile
• Rich descriptions of items in an online storefront
• Ideas for healthy meals

For example, given the user input Give me three ideas for a healthy breakfast including peppers, a generative AI application might produce the following output:

OK, here are three healthy breakfasts that include peppers:

1. Egg and Pepper Scramble: Dice up some bell peppers and sauté them in a pan until they're slightly softened. Then, add in some beaten eggs and scramble everything together. Serve with whole-grain toast for a nutritious breakfast.

2. Pepper and Avocado Toast: Toast a slice of whole-grain bread and top it with sliced bell peppers, mashed avocado, and a sprinkle of salt and pepper. This is a great way to get some healthy fats from the avocado along with the vitamin C from the peppers.

3. Breakfast Burrito with Peppers: Sauté some diced bell peppers, onions, and scrambled eggs together in a pan. Then, spoon the mixture onto a whole-grain tortilla and top with some salsa or hot sauce. Roll it up and enjoy a satisfying and nutritious breakfast.

Image generation

Some generative AI applications produce image outputs from natural language inputs. Depending on the application and its models, users might be able to specify:

• The people or things to appear in the image
• The artistic style to create the image in
• References for generating similar images

Image generation can create virtual avatars for online accounts, design logos for a business, or provide artistic inspiration for creators. For example, a user might input the request, Create an image of an elephant eating a burger. A generative AI application might produce the following output:

Audio generation

Some generative AI applications produce audio outputs from natural language inputs. Depending on the application and its models, users might be able to:

• Synthesize natural sounding voices from input text
• Create music in a specific style or featuring certain instruments
• Modify input audio files based on set criteria provided in natural language

Audio generation can provide spoken responses from digital voice assistants, add backing instruments to songs for music production, or reproduce a user's original voice from reference recordings.

Code generation

Some generative AI applications produce code from natural language inputs. Depending on the application and its models, users might be able to generate code that:

• Is in the language of their choice
• Accepts a specific input and output structure
• Uses specific technologies based on their reference documentation

Code generation can provide quick implementations of common operations, such as sort or filter data, speed up translating code from one language to another, or answer questions about a technology based on specified reference documentation. For example, given the input "Show me how to code a game of tic-tac-toe with C#", a generative AI application might produce the following output:

Here's a simple example of how you can code a game of tic-tac-toe in C#:

using System;

namespace TicTacToe
{
    class Program
    {
        static char[] board = { '1', '2', '3', '4', '5', '6', '7', '8', '9' };
        static int player = 1; // Player 1 will start
        static int choice; // The number the player chooses
        static int flag = 0; // Set the flag value to 0

        // Check if the position is already marked or not
        private static int CheckPosition(char mark)
        {
            for (int i = 0; i < 9; i++)
            {
                if (board[i] == mark)
                {
                    return 1;
                }
            }
            return 0;
        }

        // The rest of the generated code has been omitted for brevity
        // ...
    }
}

This code creates a simple console-based tic-tac-toe game in C#. It uses a single-dimensional array to represent the board and checks for a win or draw after each move.

How LLMs work

When training an LLM, the training text is first broken down into tokens. Each token identifies a unique text value. A token can be a distinct word, a partial word, or a combination of words and punctuation. Each token is assigned an ID, which enables the text to be represented as a sequence of token IDs.

After the text has been broken down into tokens, a contextual vector, known as an embedding, is assigned to each token. These embedding vectors are multi-valued numeric data where each element of a token's vector represents a semantic attribute of the token. The elements of a token's vector are determined based on how commonly tokens are used together or in similar contexts.

The goal is to predict the next token in the sequence based on the preceding tokens. The model assigns a weight to each token in the existing sequence, representing its relative influence on the next token. The model then uses the preceding tokens' weights and embeddings to calculate and predict the next vector value. The model then selects the most probable token to continue the sequence based on the predicted vector.

This process continues iteratively for each token in the sequence, with the output sequence being used regressively as the input for the next iteration. The output is built one token at a time. This strategy is analogous to how auto-complete works, where suggestions are based on what's been typed so far and updated with each new input.

During training, the model knows the complete token sequence but ignores all tokens after the one currently being considered. The model compares the predicted vector value to the actual value and calculates the loss. Training then incrementally adjusts the weights to reduce the loss and improve the model.

Related content

• Understand Tokens
• Prompt engineering
• Large language models

Prompt engineering in .NET

In this article, you explore essential prompt engineering concepts. Many AI models are prompt-based, meaning they respond to user input text (a prompt) with a response generated by predictive algorithms (a completion). Newer models also often support completions in chat form, with messages based on roles (system, user, assistant) and chat history to preserve conversations.

Work with prompts

Models that support chat-based apps use three roles to organize completions: a system role that controls the chat, a user role to represent user input, and an assistant role for responding to users. Divide your prompts into messages for each role:

• *System messages* give the model instructions about the assistant. A prompt can have only one system message, and it must be the first message.
• User messages include prompts from the user, examples, or instructions for the assistant. An example chat completion must have at least one user message.
• Assistant messages show example or historical completions and must contain a response to the preceding user message. Assistant messages aren't required, but if you include one, it must be paired with a user message to form an example.

Use instructions to improve the completion

An instruction is text that tells the model how to respond. An instruction can be a directive or an imperative:

• Directives tell the model how to behave but aren't simple commands&mdash;think character setup for an improv actor: "You're helping students learn about U.S. history, so talk about the U.S. unless they specifically ask about other countries or regions."
• Imperatives are unambiguous commands for the model to follow. "Translate to Tagalog:"

Use examples to guide the model

An example is text that shows the model how to respond by providing sample user input and model output. The model uses examples to infer what to include in completions. Examples can come either before or after the instructions in an engineered prompt, but the two shouldn't be interspersed.

An example starts with a prompt and can optionally include a completion. A completion in an example doesn't have to include the verbatim response&mdash;it might just contain a formatted word, the first bullet in an unordered list, or something similar to indicate how each completion should start.

Classify examples as zero-shot learning or few-shot learning based on whether they contain verbatim completions.

• Zero-shot learning examples include a prompt with no verbatim completion. This approach tests a model's responses without giving it example data output. Zero-shot prompts can have completions that include cues, such as indicating the model should output an ordered list by including "1." as the completion.
• Few-shot learning examples include several pairs of prompts with verbatim completions. Few-shot learning can change the model's behavior by adding to its existing knowledge.

Cues

A cue is text that conveys the desired structure or format of output. Like an instruction, a cue isn't processed by the model as if it were user input. Like an example, a cue shows the model what you want instead of telling it what to do. Add as many cues as you want to iterate toward the result you want. Use cues with an instruction or an example, and place them at the end of the prompt.

Example prompt using .NET

.NET provides various tools to prompt and chat with different AI models. Use Agent Framework to connect to a wide variety of AI models and services. Agent Framework includes tools to create agents with system instructions and maintain conversation state across multiple turns.

Consider the following code example:

The preceding code:

• Creates an Azure OpenAI client with an endpoint and API key.
• Gets a chat client for the GPT-4o model and converts it to an AI agent.
• Creates an agent session to maintain conversation state across multiple turns.
• Accepts user input in a loop to allow for different types of prompts.
• Asynchronously streams the AI response and displays it to the console.

Related content

• Prompt engineering techniques
• System message design

Retrieval-augmented generation (RAG) provides LLM knowledge

This article describes how retrieval-augmented generation lets LLMs treat your data sources as knowledge without having to train.

LLMs have extensive knowledge bases through training. For most scenarios, you can select an LLM that is designed for your requirements, but those LLMs still require additional training to understand your specific data. Retrieval-augmented generation lets you make your data available to LLMs without training them on it first.

How RAG works

To perform retrieval-augmented generation, you create embeddings for your data along with common questions about it. You can do this on the fly or you can create and store the embeddings by using a vector database solution.

When a user asks a question, the LLM uses your embeddings to compare the user's question to your data and find the most relevant context. This context and the user's question then go to the LLM in a prompt, and the LLM provides a response based on your data.

Basic RAG process

To perform RAG, you must process each data source that you want to use for retrievals. The basic process is as follows:

1. Chunk large data into manageable pieces. 1. Convert the chunks into a searchable format. 1. Store the converted data in a location that allows efficient access. Additionally, it's important to store relevant metadata for citations or references when the LLM provides responses. 1. Feed your converted data to LLMs in prompts.

• Source data: This is where your data exists. It could be a file/folder on your machine, a file in cloud storage, an Azure Machine Learning data asset, a Git repository, or an SQL database.
• Data chunking: The data in your source needs to be converted to plain text. For example, word documents or PDFs need to be cracked open and converted to text. The text is then chunked into smaller pieces.
• Converting the text to vectors: These are embeddings. Vectors are numerical representations of concepts converted to number sequences, which make it easy for computers to understand the relationships between those concepts.
• Links between source data and embeddings: This information is stored as metadata on the chunks you created, which are then used to help the LLMs generate citations while generating responses.

See also

• Data ingestion

Understand tokens

When you work with a large language model (LLM), text is first broken into units called tokens, which are words, character sets, or combinations of words and punctuation, by a tokenizer. During training, tokenization runs as the first step. The LLM analyzes the semantic relationships between tokens, such as how commonly they're used together or whether they're used in similar contexts. After training, the LLM uses those patterns and relationships to generate a sequence of output tokens based on the input sequence.

Turn text into tokens

The set of unique tokens that an LLM is trained on is known as its _vocabulary_.

For example, consider the following sentence:

I heard a dog bark loudly at a cat

This text could be tokenized as:

• I
• heard
• a
• dog
• bark
• loudly
• at
• a
• cat

By having a sufficiently large set of training text, tokenization can compile a vocabulary of many thousands of tokens.

Common tokenization methods

The specific tokenization method varies by LLM. Common tokenization methods include:

• Word tokenization (text is split into individual words based on a delimiter)
• Character tokenization (text is split into individual characters)
• Subword tokenization (text is split into partial words or character sets)

For example, the GPT models, developed by OpenAI, use a type of subword tokenization that's known as _Byte-Pair Encoding_ (BPE). OpenAI provides a tool to visualize how text will be tokenized.

Each tokenization method has benefits and disadvantages:

How LLMs use tokens

After the LLM completes tokenization, it assigns an ID to each unique token.

Consider this example sentence:

I heard a dog bark loudly at a cat

After the model uses a word tokenization method, it could assign token IDs as follows:

• I (1)
• heard (2)
• a (3)
• dog (4)
• bark (5)
• loudly (6)
• at (7)
• a (the "a" token is already assigned an ID of 3)
• cat (8)

By assigning IDs, text can be represented as a sequence of token IDs. The example sentence would be represented as [1, 2, 3, 4, 5, 6, 7, 3, 8]. The sentence "I heard a cat" would be represented as [1, 2, 3, 8].

As training continues, the model adds any new tokens in the training text to its vocabulary and assigns each one an ID. For example:

• meow (9)
• run (10)

These token ID sequences reveal the semantic relationships between tokens. Multi-valued numeric vectors, known as embeddings, represent these relationships. The model assigns an embedding to each token based on how commonly it's used together with, or in similar contexts to, the other tokens.

After it's trained, a model can calculate an embedding for text that contains multiple tokens. The model tokenizes the text, then calculates an overall embeddings value based on the learned embeddings of the individual tokens. Use this technique for semantic document searches or to add vector stores to an AI.

During output generation, the model predicts a vector value for the next token in the sequence. The model then selects the next token from its vocabulary based on this vector value. In practice, the model calculates multiple vectors by using various elements of the previous tokens' embeddings. The model then evaluates all potential tokens from these vectors and selects the most probable one to continue the sequence.

Output generation is an iterative operation. The model appends the predicted token to the sequence so far and uses that as the input for the next iteration, building the final output one token at a time.

Token limits

LLMs have a maximum number of tokens for input and output. This limit is often expressed as a combined maximum _context window_ that covers both input and output tokens together. Taken together, a model's token limit and tokenization method determine the maximum length of text that can be provided as input or generated as output.

For example, consider a model that has a maximum context window of 100 tokens. The model processes the example sentences as input text:

I heard a dog bark loudly at a cat

By using a word-based tokenization method, the input is nine tokens. This leaves 91 word tokens available for the output.

By using a character-based tokenization method, the input is 34 tokens (including spaces). This leaves only 66 character tokens available for the output.

Token-based pricing and rate limiting

Generative AI services often use token-based pricing. The cost of each request depends on the number of input and output tokens. Pricing might differ between input and output. For example, see Azure OpenAI Service pricing.

Generative AI services also enforce a maximum number of tokens per minute (TPM). These rate limits can vary depending on the service region and LLM. For more information about specific regions, see Azure OpenAI Service quotas and limits.

Related content

• Use Microsoft.ML.Tokenizers for text tokenization
• How generative AI and LLMs work
• Understand embeddings
• Work with vector databases

Vector databases for .NET + AI

Vector databases are designed to store and manage vector embeddings. Embeddings are numeric representations of non-numeric data that preserve semantic meaning. You can vectorize words, documents, images, audio, and other data types. Use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions.

For example, you can use a vector database to:

• Identify similar images, documents, and songs based on their contents, themes, sentiments, and styles.
• Identify similar products based on their characteristics, features, and user groups.
• Recommend content, products, or services based on user preferences.
• Identify the best potential options from a large pool of choices to meet complex requirements.
• Identify data anomalies or fraudulent activities that are dissimilar from predominant or normal patterns.

Understand vector search

Vector databases provide vector search capabilities to find similar items based on their data characteristics rather than by exact matches on a property field. Vector search works by analyzing the vector representations of your data that you created using an AI embedding model such as the Azure OpenAI embedding models. The search process measures the distance between the data vectors and your query vector. The data vectors that are closest to your query vector are the ones that are found to be most similar semantically.

Some services such as Azure Cosmos DB for MongoDB vCore provide native vector search capabilities for your data. Other databases can be enhanced with vector search by indexing the stored data using a service such as Azure AI Search, which can scan and index your data to provide vector search capabilities.

Vector search workflows with .NET and OpenAI

Vector databases and their search features are especially useful in RAG pattern workflows with Azure OpenAI. This pattern lets you augment your AI model with additional semantically rich knowledge of your data. A common AI workflow using vector databases includes these steps:

1. Create embeddings for your data using an OpenAI embedding model. 1. Store and index the embeddings in a vector database or search service. 1. Convert user prompts from your application to embeddings. 1. Run a vector search across your data, comparing the user prompt embedding to the embeddings in your database. 1. Use a language model such as GPT-4o to assemble a user-friendly completion from the vector search results.

Visit the Implement Azure OpenAI with RAG using vector search in a .NET app tutorial for a hands-on example of this flow.

Other benefits of the RAG pattern include:

• Generate contextually relevant and accurate responses to user prompts from AI models.
• Overcome LLM token limits—the database vector search does the heavy lifting.
• Reduce the costs from frequent fine-tuning on updated data.

Related content

• Implement Azure OpenAI with RAG using vector search in a .NET app

Zero-shot and few-shot learning

This article explains zero-shot learning and few-shot learning for prompt engineering in .NET, including their primary use cases.

GPT model performance benefits from prompt engineering, the practice of providing instructions and examples to a model to refine its output. Zero-shot learning and few-shot learning are techniques you can use when providing examples.

Zero-shot learning

Zero-shot learning is the practice of passing prompts that aren't paired with verbatim completions, although you can include completions that consist of cues. Zero-shot learning relies entirely on the model's existing knowledge to generate responses, which reduces the number of tokens created and can help you control costs. However, zero-shot learning doesn't add to the model's knowledge or context.

Here's an example zero-shot prompt that tells the model to evaluate user input to determine which of four possible intents the input represents, and then to preface the response with "Intent: ".

prompt = $"""
Instructions: What is the intent of this request?
If you don't know the intent, don't guess; instead respond with "Unknown".
Choices: SendEmail, SendMessage, CompleteTask, CreateDocument, Unknown.
User Input: {request}
Intent: 
""";

Zero-shot learning has two primary use cases:

• Work with fine-tuned LLMs - Because it relies on the model's existing knowledge, zero-shot learning isn't as resource-intensive as few-shot learning, and it works well with LLMs that have already been fine-tuned on instruction datasets. You might be able to rely solely on zero-shot learning and keep costs relatively low.
• Establish performance baselines - Zero-shot learning can help you simulate how your app performs for actual users. This lets you evaluate various aspects of your model's current performance, such as accuracy or precision. In this case, you typically use zero-shot learning to establish a performance baseline and then experiment with few-shot learning to improve performance.

Few-shot learning

Few-shot learning is the practice of passing prompts paired with verbatim completions (few-shot prompts) to show your model how to respond. Compared to zero-shot learning, this means few-shot learning produces more tokens and causes the model to update its knowledge, which can make few-shot learning more resource-intensive. However, few-shot learning also helps the model produce more relevant responses.

prompt = $"""
Instructions: What is the intent of this request?
If you don't know the intent, don't guess; instead respond with "Unknown".
Choices: SendEmail, SendMessage, CompleteTask, CreateDocument, Unknown.

User Input: Can you send a very quick approval to the marketing team?
Intent: SendMessage

User Input: Can you send the full update to the marketing team?
Intent: SendEmail

User Input: {request}
Intent:
""";

Few-shot learning has two primary use cases:

• Tuning an LLM - Because it can add to the model's knowledge, few-shot learning can improve a model's performance. It also causes the model to create more tokens than zero-shot learning does, which can eventually become prohibitively expensive or even infeasible. However, if your LLM isn't fine-tuned yet, you won't always get good performance with zero-shot prompts, and few-shot learning is warranted.
• Fixing performance issues - You can use few-shot learning as a follow-up to zero-shot learning. In this case, you use zero-shot learning to establish a performance baseline, and then experiment with few-shot learning based on the zero-shot prompts you used. This lets you add to the model's knowledge after seeing how it currently responds, so you can iterate and improve performance while minimizing the number of tokens you introduce.

Caveats

• Example-based learning doesn't work well for complex reasoning tasks. However, adding instructions can help address this.
• Few-shot learning requires creating lengthy prompts. Prompts with a large number of tokens can increase computation and latency. This typically means increased costs. There's also a limit to the length of the prompts.
• When you use several examples, the model can learn false patterns, such as "Sentiments are twice as likely to be positive than negative."

Related content

• Prompt engineering techniques
• How GenAI and LLMs work

.NET + AI ecosystem tools and SDKs

The .NET ecosystem provides many powerful tools, libraries, and services to develop AI applications. .NET supports both cloud and local AI model connections, many different SDKs for various AI and vector database services, and other tools to help you build intelligent apps of varying scope and complexity.

[!IMPORTANT]

Not all of the SDKs and services presented in this article are maintained by Microsoft. When considering an SDK, make sure to evaluate its quality, licensing, support, and compatibility to ensure they meet your requirements.

Microsoft.Extensions.AI libraries

`Microsoft.Extensions.AI` is a set of core .NET libraries that provide a unified layer of C# abstractions for interacting with AI services, such as small and large language models (SLMs and LLMs), embeddings, and middleware. These APIs were created in collaboration with developers across the .NET ecosystem. The low-level APIs, such as <xref:Microsoft.Extensions.AI.IChatClient> and <xref:Microsoft.Extensions.AI.IEmbeddingGenerator`2>, were extracted from Semantic Kernel and moved into the <xref:Microsoft.Extensions.AI> namespace.

Microsoft.Extensions.AI provides abstractions that can be implemented by various services, all adhering to the same core concepts. This library is not intended to provide APIs tailored to any specific provider's services. The goal of Microsoft.Extensions.AI is to act as a unifying layer within the .NET ecosystem, enabling developers to choose their preferred frameworks and libraries while ensuring seamless integration and collaboration across the ecosystem.

Other AI-related Microsoft.Extensions libraries

The 📦 Microsoft.Extensions.VectorData.Abstractions package provides a unified layer of abstractions for interacting with a variety of vector stores. It lets you store processed chunks in vector stores such as Qdrant, Azure SQL, CosmosDB, MongoDB, ElasticSearch, and many more. For more information, see Build a .NET AI vector search app.

The 📦 Microsoft.Extensions.DataIngestion package provides foundational .NET building blocks for data ingestion. It enables developers to read, process, and prepare documents for AI and machine learning workflows, especially retrieval-augmented generation (RAG) scenarios. For more information, see Data ingestion.

Microsoft Agent Framework

If you want to use low-level services, such as <xref:Microsoft.Extensions.AI.IChatClient> and <xref:Microsoft.Extensions.AI.IEmbeddingGenerator2>, you can reference the Microsoft.Extensions.AI.Abstractions package directly from your app. However, if you want to build agentic AI applications with higher-level orchestration capabilities, you should use [Microsoft Agent Framework](/agent-framework/overview/agent-framework-overview). Agent Framework builds on the Microsoft.Extensions.AI.Abstractions` package and provides concrete implementations of <xref:Microsoft.Extensions.AI.IChatClient> for different services, including OpenAI, Azure OpenAI, Microsoft Foundry, and more.

This framework is the recommended approach for .NET apps that need to build agentic AI systems with advanced orchestration, multi-agent collaboration, and enterprise-grade security and observability.

Agent Framework is a production-ready, open-source framework that brings together the best capabilities of Semantic Kernel and Microsoft Research's AutoGen. Agent Framework provides:

• Multi-agent orchestration: Support for sequential, concurrent, group chat, handoff, and magentic (where a lead agent directs other agents) orchestration patterns.
• Cloud and provider flexibility: Cloud-agnostic (containers, on-premises, or multi-cloud) and provider-agnostic (for example, OpenAI or Foundry) using plugin and connector models.
• Enterprise-grade features: Built-in observability (OpenTelemetry), Microsoft Entra security integration, and responsible AI features including prompt injection protection and task adherence monitoring.
• Standards-based interoperability: Integration with open standards like Agent-to-Agent (A2A) protocol and Model Context Protocol (MCP) for agent discovery and tool interaction.

For more information, see the Microsoft Agent Framework documentation.

Semantic Kernel for .NET

Semantic Kernel is an open-source library that enables AI integration and orchestration capabilities in your .NET apps. However, for new applications that require agentic capabilities, multi-agent orchestration, or enterprise-grade observability and security, the recommended framework is Microsoft Agent Framework.

.NET SDKs for building AI apps

Many different SDKs are available to build .NET apps with AI capabilities depending on the target platform or AI model. OpenAI models offer powerful generative AI capabilities, while other Foundry tools provide intelligent solutions for a variety of specific scenarios.

.NET SDKs for OpenAI models

.NET SDKs for Foundry Tools

Azure offers many other AI services, such as Foundry Tools, to build specific application capabilities and workflows. Most of these services provide a .NET SDK to integrate their functionality into custom apps. Some of the most commonly used services are shown in the following table. For a complete list of available services and learning resources, see the Foundry Tools documentation.

Develop with local AI models

.NET apps can also connect to local AI models for many different development scenarios. Microsoft Agent Framework is the recommended tool to connect to local models using .NET. This framework can connect to many different models hosted across a variety of platforms and abstracts away lower-level implementation details.

For example, you can use Ollama to connect to local AI models with .NET, including several small language models (SLMs) developed by Microsoft:

[!NOTE]

The preceding SLMs can also be hosted on other services, such as Azure.

Next steps

• What is Microsoft Agent Framework?
• Quickstart - Summarize text using Azure AI chat app with .NET

[phi3]: https://azure.microsoft.com/products/phi-3 [orca]: https://www.microsoft.com/research/project/orca/

Quickstart: Evaluate response quality

In this quickstart, you create an MSTest app to evaluate the quality of a chat response from an OpenAI model. The test app uses the Microsoft.Extensions.AI.Evaluation libraries.

[!NOTE]

This quickstart demonstrates the simplest usage of the evaluation API. Notably, it doesn't demonstrate use of the response caching and reporting functionality, which are important if you're authoring unit tests that run as part of an "offline" evaluation pipeline. The scenario shown in this quickstart is suitable in use cases such as "online" evaluation of AI responses within production code and logging scores to telemetry, where caching and reporting aren't relevant. For a tutorial that demonstrates the caching and reporting functionality, see Tutorial: Evaluate a model's response with response caching and reporting

Prerequisites

• .NET 8 or a later version
• Visual Studio Code (optional)

Configure the AI service

To provision an Azure OpenAI service and model using the Azure portal, complete the steps in the Create and deploy an Azure OpenAI Service resource article. In the "Deploy a model" step, select the gpt-5 model.

Create the test app

Complete the following steps to create an MSTest project that connects to an AI model.

1. In a terminal window, navigate to the directory where you want to create your app, and create a new MSTest app with the dotnet new command:

    dotnet new mstest -o TestAI

1. Navigate to the TestAI directory, and add the necessary packages to your app:

    dotnet add package Azure.AI.OpenAI
    dotnet add package Azure.Identity
    dotnet add package Microsoft.Extensions.AI.Abstractions
    dotnet add package Microsoft.Extensions.AI.Evaluation
    dotnet add package Microsoft.Extensions.AI.Evaluation.Quality
    dotnet add package Microsoft.Extensions.AI.OpenAI --prerelease
    dotnet add package Microsoft.Extensions.Configuration
    dotnet add package Microsoft.Extensions.Configuration.UserSecrets

1. Run the following commands to add app secrets for your Azure OpenAI endpoint and tenant ID:

    dotnet user-secrets init
    dotnet user-secrets set AZURE_OPENAI_ENDPOINT <your-Azure-OpenAI-endpoint>
    dotnet user-secrets set AZURE_TENANT_ID <your-tenant-ID>

(Depending on your environment, the tenant ID might not be needed. In that case, remove it from the code that instantiates the <xref:Azure.Identity.DefaultAzureCredential>.)

1. Open the new app in your editor of choice.

Add the test app code

1. Rename the Test1.cs file to MyTests.cs, and then open the file and rename the class to MyTests. 1. Add the private <xref:Microsoft.Extensions.AI.Evaluation.ChatConfiguration> and chat message and response members to the MyTests class. The s_messages field is a list that contains two <xref:Microsoft.Extensions.AI.ChatMessage> objects&mdash;one instructs the behavior of the chat bot, and the other is the question from the user.

1. Add the InitializeAsync method to the MyTests class.

This method accomplishes the following tasks:

• Sets up the <xref:Microsoft.Extensions.AI.Evaluation.ChatConfiguration>.
• Sets the <xref:Microsoft.Extensions.AI.ChatOptions>, including the <xref:Microsoft.Extensions.AI.ChatOptions.Temperature> and the <xref:Microsoft.Extensions.AI.ChatOptions.ResponseFormat>.
• Fetches the response to be evaluated by calling <xref:Microsoft.Extensions.AI.IChatClient.GetResponseAsync(System.Collections.Generic.IEnumerable{Microsoft.Extensions.AI.ChatMessage},Microsoft.Extensions.AI.ChatOptions,System.Threading.CancellationToken)>, and stores it in a static variable.

1. Add the GetAzureOpenAIChatConfiguration method, which creates the <xref:Microsoft.Extensions.AI.IChatClient> that the evaluator uses to communicate with the model.

1. Add a test method to evaluate the model's response.

This method does the following:

• Invokes the <xref:Microsoft.Extensions.AI.Evaluation.Quality.CoherenceEvaluator> to evaluate the coherence of the response. The <xref:Microsoft.Extensions.AI.Evaluation.IEvaluator.EvaluateAsync(System.Collections.Generic.IEnumerable{Microsoft.Extensions.AI.ChatMessage},Microsoft.Extensions.AI.ChatResponse,Microsoft.Extensions.AI.Evaluation.ChatConfiguration,System.Collections.Generic.IEnumerable{Microsoft.Extensions.AI.Evaluation.EvaluationContext},System.Threading.CancellationToken)> method returns an <xref:Microsoft.Extensions.AI.Evaluation.EvaluationResult> that contains a <xref:Microsoft.Extensions.AI.Evaluation.NumericMetric>. A NumericMetric contains a numeric value that's typically used to represent numeric scores that fall within a well-defined range.
• Retrieves the coherence score from the <xref:Microsoft.Extensions.AI.Evaluation.EvaluationResult>.
• Validates the default interpretation for the returned coherence metric. Evaluators can include a default interpretation for the metrics they return. You can also change the default interpretation to suit your specific requirements, if needed.
• Validates that no diagnostics are present on the returned coherence metric. Evaluators can include diagnostics on the metrics they return to indicate errors, warnings, or other exceptional conditions encountered during evaluation.

Run the test/evaluation

Run the test using your preferred test workflow, for example, by using the CLI command dotnet test or through Test Explorer.

Clean up resources

If you no longer need them, delete the Azure OpenAI resource and GPT-4 model deployment.

1. In the Azure portal, navigate to the Azure OpenAI resource. 1. Select the Azure OpenAI resource, and then select Delete.

Next steps

• Evaluate the responses from different OpenAI models.
• Add response caching and reporting to your evaluation code. For more information, see Tutorial: Evaluate a model's response with response caching and reporting.

Tutorial: Evaluate response safety with caching and reporting

In this tutorial, you create an MSTest app to evaluate the content safety of a response from an OpenAI model. Safety evaluators check for presence of harmful, inappropriate, or unsafe content in a response. The test app uses the safety evaluators from the Microsoft.Extensions.AI.Evaluation.Safety package to perform the evaluations. These safety evaluators use the Microsoft Foundry Evaluation service to perform evaluations.

Prerequisites

• .NET 8.0 SDK or higher - Install the .NET 8 SDK.
• An Azure subscription - Create one for free.

Configure the AI service

To provision an Azure OpenAI service and model using the Azure portal, complete the steps in the Create and deploy an Azure OpenAI Service resource article. In the "Deploy a model" step, select the gpt-5 model.

[!TIP]

The previous configuration step is only required to fetch the response to be evaluated. To evaluate the safety of a response you already have in hand, you can skip this configuration.

The evaluators in this tutorial use the Foundry Evaluation service, which requires some additional setup:

• Create a resource group within one of the Azure regions that support Foundry Evaluation service.
• Create a Foundry hub in the resource group you just created.
• Finally, create a Foundry project in the hub you just created.

Create the test app

Complete the following steps to create an MSTest project.

1. In a terminal window, navigate to the directory where you want to create your app, and create a new MSTest app with the dotnet new command:

   dotnet new mstest -o EvaluateResponseSafety

1. Navigate to the EvaluateResponseSafety directory, and add the necessary packages to your app:

   dotnet add package Azure.AI.OpenAI
   dotnet add package Azure.Identity
   dotnet add package Microsoft.Extensions.AI.Abstractions
   dotnet add package Microsoft.Extensions.AI.Evaluation
   dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting
   dotnet add package Microsoft.Extensions.AI.Evaluation.Safety --prerelease
   dotnet add package Microsoft.Extensions.AI.OpenAI --prerelease
   dotnet add package Microsoft.Extensions.Configuration
   dotnet add package Microsoft.Extensions.Configuration.UserSecrets

1. Run the following commands to add app secrets for your Azure OpenAI endpoint, tenant ID, subscription ID, resource group, and project:

   dotnet user-secrets init
   dotnet user-secrets set AZURE_OPENAI_ENDPOINT <your-Azure-OpenAI-endpoint>
   dotnet user-secrets set AZURE_TENANT_ID <your-tenant-ID>
   dotnet user-secrets set AZURE_SUBSCRIPTION_ID <your-subscription-ID>
   dotnet user-secrets set AZURE_RESOURCE_GROUP <your-resource-group>
   dotnet user-secrets set AZURE_AI_PROJECT <your-Azure-AI-project>

(Depending on your environment, the tenant ID might not be needed. In that case, remove it from the code that instantiates the <xref:Azure.Identity.DefaultAzureCredential>.)

1. Open the new app in your editor of choice.

Add the test app code

1. Rename the Test1.cs file to MyTests.cs, and then open the file and rename the class to MyTests. Delete the empty TestMethod1 method. 1. Add the necessary using directives to the top of the file.

1. Add the <xref:Microsoft.VisualStudio.TestTools.UnitTesting.TestContext> property to the class.

1. Add the scenario and execution name fields to the class.

The scenario name is set to the fully qualified name of the current test method. However, you can set it to any string of your choice. Here are some considerations for choosing a scenario name:

• When using disk-based storage, the scenario name is used as the name of the folder under which the corresponding evaluation results are stored.
• By default, the generated evaluation report splits scenario names on . so that the results can be displayed in a hierarchical view with appropriate grouping, nesting, and aggregation.

The execution name is used to group evaluation results that are part of the same evaluation run (or test run) when the evaluation results are stored. If you don't provide an execution name when creating a <xref:Microsoft.Extensions.AI.Evaluation.Reporting.ReportingConfiguration>, all evaluation runs will use the same default execution name of Default. In this case, results from one run will be overwritten by the next.

1. Add a method to gather the safety evaluators to use in the evaluation.

1. Add a <xref:Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfiguration> object, which configures the connection parameters that the safety evaluators need to communicate with the Foundry Evaluation service.

1. Add a method that creates an <xref:Microsoft.Extensions.AI.IChatClient> object, which will be used to get the chat response to evaluate from the LLM.

1. Set up the reporting functionality. Convert the <xref:Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfiguration> to a <xref:Microsoft.Extensions.AI.Evaluation.ChatConfiguration>, and then pass that to the method that creates a <xref:Microsoft.Extensions.AI.Evaluation.Reporting.ReportingConfiguration>.

Response caching functionality is supported and works the same way regardless of whether the evaluators talk to an LLM or to the Foundry Evaluation service. The response will be reused until the corresponding cache entry expires (in 14 days by default), or until any request parameter, such as the LLM endpoint or the question being asked, is changed.

[!NOTE]

This code example passes the LLM <xref:Microsoft.Extensions.AI.IChatClient> as originalChatClient to <xref:Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfigurationExtensions.ToChatConfiguration(Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfiguration,Microsoft.Extensions.AI.IChatClient)>. The reason to include the LLM chat client here is to enable getting a chat response from the LLM, and notably, to enable response caching for it. (If you don't want to cache the LLM's response, you can create a separate, local <xref:Microsoft.Extensions.AI.IChatClient> to fetch the response from the LLM.) Instead of passing a <xref:Microsoft.Extensions.AI.IChatClient>, if you already have a <xref:Microsoft.Extensions.AI.Evaluation.ChatConfiguration> for an LLM from another reporting configuration, you can pass that instead, using the <xref:Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfigurationExtensions.ToChatConfiguration(Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfiguration,Microsoft.Extensions.AI.Evaluation.ChatConfiguration)> overload.

>

Similarly, if you configure both LLM-based evaluators and Foundry Evaluation service&ndash;based evaluators in the reporting configuration, you also need to pass the LLM <xref:Microsoft.Extensions.AI.Evaluation.ChatConfiguration> to <xref:Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfigurationExtensions.ToChatConfiguration(Microsoft.Extensions.AI.Evaluation.Safety.ContentSafetyServiceConfiguration,Microsoft.Extensions.AI.Evaluation.ChatConfiguration)>. Then it returns a <xref:Microsoft.Extensions.AI.Evaluation.ChatConfiguration> that can talk to both types of evaluators.

1. Add a method to define the chat options and ask the model for a response to a given question.

The test in this tutorial evaluates the LLM's response to an astronomy question. Since the <xref:Microsoft.Extensions.AI.Evaluation.Reporting.ReportingConfiguration> has response caching enabled, and since the supplied <xref:Microsoft.Extensions.AI.IChatClient> is always fetched from the <xref:Microsoft.Extensions.AI.Evaluation.Reporting.ScenarioRun> created using this reporting configuration, the LLM response for the test is cached and reused.

1. Add a method to validate the response.

[!TIP]

Some of the evaluators, for example, <xref:Microsoft.Extensions.AI.Evaluation.Safety.ViolenceEvaluator>, might produce a warning diagnostic that's shown in the report if you only evaluate the response and not the message. Similarly, if the data you pass to <xref:Microsoft.Extensions.AI.Evaluation.Reporting.ScenarioRunExtensions.EvaluateAsync*> contains two consecutive messages with the same <xref:Microsoft.Extensions.AI.ChatRole> (for example, <xref:Microsoft.Extensions.AI.ChatRole.User> or <xref:Microsoft.Extensions.AI.ChatRole.Assistant>), it might also produce a warning. However, even though an evaluator might produce a warning diagnostic in these cases, it still proceeds with the evaluation.

1. Finally, add the test method itself.

This test method:

• Creates the <xref:Microsoft.Extensions.AI.Evaluation.Reporting.ScenarioRun>. The use of await using ensures that the ScenarioRun is correctly disposed and that the results of this evaluation are correctly persisted to the result store.
• Gets the LLM's response to a specific astronomy question. The same <xref:Microsoft.Extensions.AI.IChatClient> that will be used for evaluation is passed to the GetAstronomyConversationAsync method in order to get response caching for the primary LLM response being evaluated. (In addition, this enables response caching for the responses that the evaluators fetch from the Foundry Evaluation service as part of performing their evaluations.)
• Runs the evaluators against the response. Like the LLM response, on subsequent runs, the evaluation is fetched from the (disk-based) response cache that was configured in s_safetyReportingConfig.
• Runs some safety validation on the evaluation result.

Run the test/evaluation

Run the test using your preferred test workflow, for example, by using the CLI command dotnet test or through Test Explorer.

Generate a report

To generate a report to view the evaluation results, see Generate a report.

Next steps

This tutorial covers the basics of evaluating content safety. As you create your test suite, consider the following next steps:

• Configure additional evaluators, such as the quality evaluators. For an example, see the AI samples repo quality and safety evaluation example.
• Evaluate the content safety of generated images. For an example, see the AI samples repo image response example.
• In real-world evaluations, you might not want to validate individual results, since the LLM responses and evaluation scores can vary over time as your product (and the models used) evolve. You might not want individual evaluation tests to fail and block builds in your CI/CD pipelines when this happens. Instead, in such cases, it might be better to rely on the generated report and track the overall trends for evaluation scores across different scenarios over time (and only fail individual builds in your CI/CD pipelines when there's a significant drop in evaluation scores across multiple different tests).

Responsible AI with .NET

Responsible AI refers to the practice of designing, developing, and deploying artificial intelligence systems in a way that is ethical, transparent, and aligned with human values. It emphasizes fairness, accountability, privacy, and safety to ensure that AI technologies benefit individuals and society as a whole. As AI becomes increasingly integrated into applications and decision-making processes, prioritizing responsible AI is of utmost importance.

Microsoft has identified six principles for responsible AI:

• Fairness
• Reliability and safety
• Privacy and security
• Inclusiveness
• Transparency
• Accountability

If you're building an AI app with .NET, the 📦 Microsoft.Extensions.AI.Evaluation.Safety package provides evaluators to help ensure that the responses your app generates, both text and image, meet the standards for responsible AI. The evaluators can also detect problematic content in user input. These safety evaluators use the Microsoft Foundry Evaluation service to perform evaluations. They include metrics for hate and unfairness, groundedness, ungrounded inference of human attributes, and the presence of:

• Protected material
• Self-harm content
• Sexual content
• Violent content
• Vulnerable code (text-based only)
• Indirect attacks (text-based only)

For more information about the safety evaluators, see Safety evaluators. To get started with the Microsoft.Extensions.AI.Evaluation.Safety evaluators, see Tutorial: Evaluate response safety with caching and reporting.

See also

• Responsible AI at Microsoft
• Training: Embrace responsible AI principles and practices
• Foundry Evaluation service
• Azure AI Content Safety