How to Evaluate an AI Agent in Microsoft Foundry

Microsoft Foundry ships built-in evaluators that work like unit tests for agents. You give them agent messages, they give you pass or fail with a reason.

There are three families. System evaluators ask whether the agent got the right outcome. Process evaluators ask whether it took the right steps to get there. RAG evaluators ask whether the answer was actually grounded in your data rather than invented.

Start with three: Task Completion, Tool Call Accuracy and Groundedness. That covers outcome, process and truthfulness, and you can add the rest once you know what your agent gets wrong.

One thing to know before you wire any of this up: several evaluators, groundedness included, have limited support when your agent calls Azure AI Search. That is the most common retriever in an Azure RAG stack. Details below.

What evaluators does Microsoft Foundry actually give you?

Seventeen built-in evaluators across agents and RAG. Here they are with what each one measures.

Agent evaluators, system level. These look at the end result.

Evaluator	What it measures	Output
Task Completion (preview)	Whether the agent completed the task with a usable deliverable meeting all requirements	Pass or fail
Task Adherence (preview)	Whether the agent followed the rules, procedures and policy constraints in its system message	Pass or fail
Intent Resolution (preview)	Whether the agent correctly identified what the user actually wanted	Pass or fail on a 1 to 5 scale
Task Navigation Efficiency	Whether the agent's tool calls matched an expected sequence. Needs ground truth	Pass or fail, plus precision, recall and F1
Customer Satisfaction (preview)	Holistic satisfaction across helpfulness, completeness, clarity, tone, resolution and adaptability	1 to 5 Likert scale

Agent evaluators, process level. These look at how it got there, and they are the ones people skip.

Evaluator	What it measures	Output
Tool Call Accuracy	Whether the agent made the right calls with the right parameters	Pass or fail on a 1 to 5 scale
Tool Selection	Whether it picked the correct tools without selecting unnecessary ones	Pass or fail
Tool Input Accuracy	Parameter correctness against six strict criteria, including groundedness, type compliance and required parameters	Pass or fail
Tool Output Utilization	Whether the agent actually used the tool result in its reasoning and answer	Pass or fail
Tool Call Success	Whether the calls succeeded or threw technical errors	Pass or fail

RAG evaluators. These are the ones you need if your agent answers from your documents.

Evaluator	What it measures	Output
Groundedness	How well the response aligns with the given context without fabricating. The precision side	Pass or fail on a 1 to 5 scale
Groundedness Pro (preview)	Strict consistency with the context, using the Azure AI Content Safety service	True or false
Relevance	Accuracy, completeness and direct relevance of the response to the query	Pass or fail on a 1 to 5 scale
Response Completeness (preview)	Whether the response misses critical information against ground truth. The recall side	Pass or fail on a 1 to 5 scale
Retrieval	How relevant the retrieved chunks were, judged by an LLM, no ground truth needed	Pass or fail on a 1 to 5 scale
Document Retrieval	Search quality against labelled ground truth: Fidelity, NDCG, XDCG, Max Relevance, Holes	Composite metrics

There is also a Quality Grader (preview), which rolls relevance, abstention and answer completeness into one call, and adds groundedness and context coverage when you pass context. It is the same evaluator Microsoft Copilot Studio uses for agent evaluation.

Sources: Microsoft's agent evaluators and RAG evaluators documentation, read 10 August 2026.

What does the groundedness evaluator actually measure?

Groundedness measures whether the response stays inside the context you gave it. Nothing more.

It scores 1 to 5, and the default pass threshold is 3. You give it a response and, ideally, a context. Adding the query improves scoring accuracy. The output includes a reason string, which is the part worth reading, because a bare score tells you nothing about which sentence went wrong.

Here is what a result looks like:

{
    "type": "azure_ai_evaluator",
    "name": "Groundedness",
    "metric": "groundedness",
    "score": 4,
    "label": "pass",
    "reason": "The response is well-grounded in the provided context without fabricating content.",
    "threshold": 3,
    "passed": true
}

Now the important part. Groundedness is a precision measure. It asks whether the answer contains anything that is not in the context. It does not ask whether the answer covered everything it should have.

An agent that answers "I found one relevant policy" when your context contains four is perfectly grounded. It just isn't much use.

That gap has its own evaluator. Response Completeness measures the recall side: whether the response misses critical information compared to ground truth. Microsoft's own documentation draws the precision and recall distinction explicitly, and most teams run groundedness alone and never notice the other half.

Question you are asking	Evaluator	Aspect
Did it make anything up?	Groundedness	Precision
Did it leave anything out?	Response Completeness	Recall
Did it answer the question asked?	Relevance	Neither, this is about the query
Did retrieval even find the right chunks?	Retrieval or Document Retrieval	Upstream of all three

If groundedness passes and your users still complain, check completeness and retrieval before you touch the prompt.

This is not theoretical. The clearest case we have written up is a groundedness regression caught during a first week in production: a system prompt was changed to fix one behaviour, and the evaluation run flagged a drop in groundedness on ambiguously phrased queries where retrieval had started pulling the wrong context and the model answered from it confidently. Nobody would have caught that by reading responses manually. That is the argument for evaluation in one paragraph.

Tip

Reading an evaluator schema and running one against an agent that is failing are different experiences. The free taster lab gets a Foundry deployment running end to end in about twenty minutes, no card needed.

Why do the tool evaluators break on Azure AI Search?

This is the trap, and it is documented in one line that is easy to scroll past.

Microsoft states that if your agent conversation includes calls to certain tools, you should avoid tool_call_accuracy, tool_input_accuracy, tool_output_utilization, tool_call_success, or groundedness.

The tools on that limited-support list:

Tool	Common use
Azure AI Search	The default retriever in most Azure RAG stacks
Bing Grounding	Grounding answers in web results
Bing Custom Search	Scoped web grounding
SharePoint Grounding	Grounding in internal documents
Code Interpreter	Running generated code
Fabric Data Agent	Querying Fabric data
Web Search	General web retrieval

Read that list against the evaluator list and the problem is obvious. The most common way to ground an Azure agent is Azure AI Search. The evaluator you most want for a grounded agent is groundedness. Those two have limited support together.

Fully supported tools are narrower: File Search, Function Tool, MCP and Knowledge-based MCP.

So if you are building the standard pattern, an agent grounded on Azure AI Search, you cannot simply switch on the obvious evaluators and trust the numbers. Practical options: evaluate the retrieval step separately with the Retrieval or Document Retrieval evaluators, which take query and context directly rather than reading tool calls, and evaluate the generation step by passing context explicitly as a field in your test dataset instead of letting the evaluator extract it from tool results.

That is more work than the quickstart implies. It is also the difference between an evaluation suite that means something and one that produces green ticks.

Which evaluators should you start with?

Three, and add more only when you have a failure you cannot explain.

Task Completion, because if the agent isn't finishing the job, nothing else matters. It is a system evaluator and it gives you a single pass or fail on the outcome.

Tool Call Accuracy, because in agent systems most failures are tool failures, not language failures. The agent picked the wrong tool, or the right tool with a malformed parameter. This is the process evaluator that catches both.

Groundedness, if your agent answers from documents. With the caveat above about how you feed it context.

Then add by symptom rather than by completeness:

Symptom	Add this evaluator
Answers are true but thin	Response Completeness
Agent ignores its system instructions	Task Adherence
Agent misunderstands what was asked	Intent Resolution
Agent takes six steps to do a two-step job	Task Navigation Efficiency
Agent calls a tool and then ignores the result	Tool Output Utilization
Intermittent failures with no clear cause	Tool Call Success
Retrieval seems to return irrelevant chunks	Retrieval, then Document Retrieval if you have labels

Running all seventeen from day one is a way to generate noise you will stop reading by the second week.

One configuration note. These are LLM-judge evaluators, so you pass a deployment_name for the judge model. Microsoft recommends gpt-5-mini for complex evaluation that needs refined reasoning, on the balance of performance, cost and efficiency. The judge model costs money on every run, which is a real consideration once evaluation is in CI.

How do you build a test dataset for agent evaluation?

Simpler than people expect. A JSONL file where each line has the fields your evaluators need.

For RAG evaluators, that is query, context and response:

{"query": "What are the store hours?", "context": "Our store is open Monday-Friday 9am-6pm and Saturday 10am-4pm.", "response": "The store is open weekdays from 9am to 6pm and Saturdays from 10am to 4pm."}

For multi-chunk retrieval, concatenate the chunks into one string with a separator between them.

For agent evaluators that need the full interaction, query and response become conversation arrays following the OpenAI message schema, with roles of system, user, assistant and tool. Tool calls and tool results are content items inside those messages. You also pass tool_definitions using the OpenAI function-calling schema so the evaluator knows what the agent was allowed to do.

Include the system message. Several evaluators judge the agent against its own instructions, and without the system message they are guessing what the rules were.

Task Navigation Efficiency has a wrinkle worth flagging because it will waste an afternoon. Its two inputs use different formats. actions takes OpenAI message-schema dictionaries, the agent's real behaviour. expected_actions takes a plain list of tool names. It also has three matching modes: exact_match, in_order_match which allows extra steps in the right order, and any_order_match which allows extra steps in any order.

What does agent evaluation have to do with AI-103?

More than it did a year ago, and this is the part candidates underestimate.

Evaluation is named directly in three separate AI-103 objectives: evaluating models and apps including detecting fabrications, integrating monitoring into deployed agents and performing error analysis, and applying responsible AI instrumentation including evaluators and safety evaluations.

None of that existed as an examinable topic on AI-102 at this depth. If you want the full picture of what moved, we mapped the AI-102 objectives against AI-103 line by line, and evaluation is one of the clearest additions.

It also pairs with orchestration. Once you have read the five multi-agent orchestration patterns, evaluation is the question that immediately follows: you have wired five agents together, so how do you know the thing works? Process evaluators are the answer, because in a multi-agent system the final output can be correct while three of the steps were wrong.

Worth saying plainly: evaluation is not a topic you can revise. There is no fact to memorise. You either have pointed an evaluator at a failing agent, read the reason field, and worked out which step broke, or you have not.

Go and pick one agent you have already built. Write ten test cases in a JSONL file, run Groundedness and Tool Call Accuracy against it, and read the reason strings on every failure rather than the scores. The first ten will tell you more about your agent than any amount of prompt tuning.

Tip

The AI-103 lab set covers evaluation, tracing and agent observability against the current exam objectives, on your own Azure subscription.

Related reading

• What Azure AI Foundry is used for
• Foundry vs Azure AI Studio vs Azure OpenAI: what changed
• Foundry Agent Service: when it is the wrong tool
• AI-103 study guide: all four learning paths