Foundry Agent Service: What It Does and When It Is the Wrong Tool

Microsoft's Foundry Agent Service does something genuinely useful: it takes the plumbing you would otherwise write yourself (conversation state, tool invocation loops, file retrieval, run lifecycle) and runs it as a managed service. That is a real saving. It is also the reason teams reach for it in situations where a plain chat completion call would have been simpler, cheaper and easier to debug.

This post is about both halves. What the Agent Service actually gives you, and the four situations where we tell people in our AI-103 labs to put it down and use something else.

What the Agent Service actually manages for you

Strip away the marketing and the service owns five things.

Threads. A thread is a server-side conversation. You append messages to it and the service keeps the history. You never send the full transcript back on every call, and you never build your own message store for chat history.

Runs. A run is one execution of an agent against a thread. It has a lifecycle: queued, in progress, requires action, completed, failed, expired. The "requires action" state is the interesting one, because that is where the service hands control back to you for a tool call.

The tool loop. When the model decides to call a tool, the service pauses the run, exposes the tool call, waits for your output, then resumes with that output in context. Without the service you write this loop yourself: parse the tool call, dispatch, append the result, call the model again, repeat until no more tool calls.

Built-in tools. File search over uploaded documents, code interpreter for sandboxed Python, Bing grounding, and connections to Azure AI Search or your own OpenAPI-described endpoints. File search is the one that saves the most work, because it covers chunking, embedding and retrieval without you standing up a vector store.

Agent definitions. Name, instructions, model, tool list and metadata, stored server side and versioned by you rather than pasted into every request.

In our AI-103 lab work the practical effect is that a working document-grounded assistant goes from roughly 200 lines of orchestration code to about 30 lines of calls against the service. That is the case for using it.

The shape of a minimal agent

The flow is always the same four steps, whichever SDK you use.

# 1. Define the agent once
agent = client.agents.create_agent(
    model="gpt-4o",
    name="policy-assistant",
    instructions="Answer only from the uploaded policy documents. If the answer is not there, say so.",
    tools=[{"type": "file_search"}],
)

# 2. Create a thread per conversation
thread = client.agents.create_thread()

# 3. Add the user message
client.agents.create_message(
    thread_id=thread.id, role="user", content="What is our data retention period?"
)

# 4. Run and poll
run = client.agents.create_and_process_run(thread_id=thread.id, agent_id=agent.id)
messages = client.agents.list_messages(thread_id=thread.id)

Two things bite people here. create_and_process_run hides the polling loop, which is convenient until a tool call needs your code to respond, at which point you have to drop down to create_run and handle the requires_action state yourself. And an agent is a persistent resource. Creating one per request leaves hundreds of orphaned agents in your project, which is a common mess when people copy a quickstart into a loop.

When the Agent Service is the wrong tool

1. Single-turn work with no conversation

If your call is "classify this ticket", "extract these fields from this invoice", or "summarise this transcript", there is no conversation to keep. You are paying for thread creation, run polling and extra round trips to model something stateless.

A plain chat completions call with a structured output schema is faster, has one network hop instead of three or more, and returns a typed object you can validate.

Use the Agent Service when state across turns is the point. Not when a single prompt would do.

2. When you need deterministic control over the tool sequence

The Agent Service lets the model decide which tool to call and when. That is the feature. It is also a problem when the correct order is fixed by business rules: validate the account, then check entitlement, then issue the refund, never in another order and never skipping a step.

You can push the model toward an order with instructions, but you cannot guarantee it. If a wrong order causes a real-world consequence (money moved, a record deleted, a message sent to a customer), the model should not be the one sequencing. Write the sequence as code and call the model only for the steps that need judgement.

This is the distinction the AI-103 exam keeps circling: agentic when the path is unknown, workflow when the path is known.

3. When you already have an orchestration framework

If you are running the Microsoft Agent Framework, Semantic Kernel or LangGraph, you already have agents, tool calling, memory and multi-agent coordination. Layering the Agent Service under it usually means two agent abstractions, two places where instructions live and two things to debug when the answer is wrong.

Pick one layer. The Agent Service is a good default when the framework's only role would have been the tool loop. A framework is the better answer when you need branching, human approval steps, or several agents handing work between each other. Our multi-agent orchestration guide covers where that line falls.

4. When retrieval quality is the actual problem

Built-in file search is a convenience feature. It uses default chunking and default retrieval settings, and it does not expose the knobs that matter when quality is poor: chunk size and overlap, hybrid search weighting, semantic ranking, metadata filters, per-tenant scoping.

If your answers are wrong because the wrong passages are being retrieved, moving to the Agent Service will not fix it. Build the index deliberately in Azure AI Search, tune retrieval against a labelled question set, then connect that index as a tool. And check first whether retrieval is the right architecture at all, which is the subject of when RAG is not the right answer.

A decision table

Situation	Use
Stateless classification, extraction, summarisation	Chat completions with structured output
Multi-turn assistant over your documents	Agent Service with file search
Fixed sequence with real-world side effects	Your own code, model for judgement steps only
Several specialists handing work between them	Agent framework, optionally over the Agent Service
Retrieval quality is the bottleneck	Azure AI Search tuned, connected as a tool
Strict latency budget under two seconds	Direct model call, no run lifecycle

Things that surprise people in the labs

Runs expire. A run that sits in requires_action waiting for your tool output will expire, typically after ten minutes. If your tool calls a slow downstream system, the run can die while you are still waiting for it. Handle expiry explicitly rather than assuming the run will be there when your tool returns.

Threads grow forever. The service keeps history, which is the point, but a long thread means more tokens on every run. Long-lived assistants need a summarisation or truncation strategy. Nobody adds one until the bill arrives.

Agents, files and vector stores are persistent resources. They stay until you delete them. In shared subscriptions they accumulate quietly and count against quota. Clean up in your teardown path, not manually.

Tool output must be a string. Returning a dict from your tool handler and letting the SDK serialise it silently changes the shape the model sees. Serialise deliberately, and keep the shape stable between calls.

Content filters apply to tool output too. A tool returning text that trips a filter fails the run, not just the tool. This one is genuinely confusing the first time, because the failure looks like a model problem.

What this means for AI-103

The exam does not ask you to recite the Agent Service API. It asks you to choose. Given a scenario, is the right answer an agent, a workflow, a direct model call or retrieval over a tuned index? Objectives around designing agent solutions are written to reward candidates who can justify not using an agent.

If you can explain why a fixed-sequence refund process should not have a model deciding step order, you are answering the question the exam is actually asking. Our AI-103 study guide maps this to the four learning paths, and the Foundry naming changes post is worth reading first if the terminology in the docs looks inconsistent to you.

A 30-minute exercise

Take a use case you are considering an agent for and do this:

1. Write the ideal interaction as a numbered list of steps.
2. Mark each step as fixed (always happens in this order) or open (depends on what the user said or what a tool returned).
3. If every step is fixed, write it as code. No agent.
4. If the open steps are a small set of tool choices, use the Agent Service.
5. If open steps involve several specialists, approvals or branching, use an agent framework.

Most internal use cases come out at step 3 or 4. The framework case is real but rarer than the posts about it suggest.

FAQ

Is the Foundry Agent Service the same as Azure OpenAI Assistants? It is the successor to that surface inside Foundry. The thread and run concepts carry over, and the tool set is broader, including Azure AI Search and OpenAPI tool connections.

Do I need the Agent Service to use tools with a model? No. Function calling works on a plain chat completions call. The service manages the loop for you; it does not enable the capability.

Can I use my own vector index instead of file search? Yes. Connect an Azure AI Search index as a tool. This is the right move as soon as retrieval quality matters.

How do I stop threads from growing without limit? Summarise older turns into a single message, or start a new thread per session and carry forward a short summary. There is no automatic trimming.

Does the Agent Service work with the Microsoft Agent Framework? Yes, and that combination makes sense when you need the framework's orchestration and the service's hosted tools. Avoid it when the framework would only be wrapping the tool loop you already get.