Deploying Azure OpenAI to Production: A Step-by-Step Playbook

Getting Azure OpenAI from a proof-of-concept into a hardened, cost-controlled, production-grade environment is where most teams hit a wall. The demo works great. The stakeholders are excited. Then someone asks "how do we actually run this thing in production?" - and suddenly the room gets quiet.

This playbook walks through the real decisions, real configurations, and real gotchas that separate a Friday-afternoon demo from a system your SRE team can support at 2 AM.

---

Why Azure OpenAI in Production Is Different

Azure OpenAI is not just an API call wrapped in a try-catch block. It sits at the intersection of:

• Regulated infrastructure (your Azure tenant, VNets, Private Endpoints)
• Non-deterministic outputs (LLMs don't behave like deterministic microservices)
• Token-based economics (costs can spiral without proper guardrails)
• Rate limits and throttling (TPM/RPM quotas that behave differently under burst load)

A production deployment needs to account for all four dimensions simultaneously.

---

Step 1: Provision the Right Azure OpenAI Resource

Choose Your Region Carefully

Not all Azure regions have GPT-4o, GPT-4 Turbo, or the latest embeddings models. Before you commit to a region, cross-check model availability:

az cognitiveservices account list-models \
  --kind OpenAI \
  --location eastus \
  --output table

For most enterprise workloads, East US and West Europe have the broadest model availability. If you're building for GDPR-sensitive data, pin to an EU region from day one - retrofitting this later is painful.

Use Infrastructure as Code from Day One

Do not click through the Azure portal to create your resource. Use Terraform or Bicep so your environment is reproducible:

resource "azurerm_cognitive_account" "openai" {
  name                = "openai-prod-eastus"
  location            = "East US"
  resource_group_name = azurerm_resource_group.main.name
  kind                = "OpenAI"
  sku_name            = "S0"

  custom_subdomain_name = "mycompany-openai-prod"

  network_acls {
    default_action = "Deny"
    virtual_network_rules {
      subnet_id = azurerm_subnet.app_subnet.id
    }
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    environment = "production"
    team        = "platform-engineering"
  }
}

Notice default_action = "Deny" - public internet access is blocked from the start. This is non-negotiable in production.

Deploy Your Model

resource "azurerm_cognitive_deployment" "gpt4o" {
  name                 = "gpt-4o-prod"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = "gpt-4o"
    version = "2024-05-13"
  }

  scale {
    type     = "Standard"
    capacity = 40  # 40K tokens per minute
  }
}

Capacity planning tip: Start with your expected peak TPM, add 30% headroom, then validate against actual load test results - not vendor marketing numbers.

---

Step 2: Lock Down Network Security

Private Endpoint Configuration

In production, your application should never reach Azure OpenAI over the public internet. Configure a Private Endpoint so traffic stays on the Microsoft backbone:

resource "azurerm_private_endpoint" "openai_pe" {
  name                = "pe-openai-prod"
  location            = "East US"
  resource_group_name = azurerm_resource_group.main.name
  subnet_id           = azurerm_subnet.private_endpoint_subnet.id

  private_service_connection {
    name                           = "openai-privateserviceconnection"
    private_connection_resource_id = azurerm_cognitive_account.openai.id
    subresource_names              = ["account"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "openai-dns-zone-group"
    private_dns_zone_ids = [azurerm_private_dns_zone.openai.id]
  }
}

resource "azurerm_private_dns_zone" "openai" {
  name                = "privatelink.openai.azure.com"
  resource_group_name = azurerm_resource_group.main.name
}

What This Architecture Looks Like

┌─────────────────────────────────────────────────┐
│                 Your Azure VNet                  │
│                                                   │
│  ┌──────────────┐    ┌────────────────────────┐  │
│  │  App Service │───▶│  Private Endpoint       │  │
│  │  or AKS Pod  │    │  (10.0.1.x)            │  │
│  └──────────────┘    └────────────┬───────────┘  │
│                                    │              │
│                    ┌───────────────▼───────────┐  │
│                    │  Private DNS Zone         │  │
│                    │  privatelink.openai.azure │  │
│                    │  .com → 10.0.1.x          │  │
│                    └───────────────────────────┘  │
└─────────────────────────────────────────────────┘
                             │
                    Microsoft Backbone
                             │
                    Azure OpenAI Service

No traffic leaves Microsoft's network. Your security team will appreciate this.

---

Step 3: Authentication - Keys Are Not Enough

Using API keys in production is a code smell. Keys get leaked in logs, committed to repos, and shared in Slack threads. Use Managed Identity instead.

Assign the Right RBAC Role

# Get your app's managed identity principal ID
PRINCIPAL_ID=$(az webapp identity show \
  --name myapp-prod \
  --resource-group myapp-rg \
  --query principalId -o tsv)

# Assign Cognitive Services OpenAI User role
az role assignment create \
  --assignee $PRINCIPAL_ID \
  --role "Cognitive Services OpenAI User" \
  --scope /subscriptions/$SUB_ID/resourceGroups/myapp-rg/resources/openai-prod-eastus

Application Code Using Managed Identity

from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from openai import AzureOpenAI

# In production: uses Managed Identity automatically
# In dev: falls back to your Azure CLI credentials
credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
    credential,
    "https://cognitiveservices.azure.com/.default"
)

client = AzureOpenAI(
    azure_endpoint="https://mycompany-openai-prod.openai.azure.com/",
    azure_ad_token_provider=token_provider,
    api_version="2024-02-01"
)

response = client.chat.completions.create(
    model="gpt-4o-prod",
    messages=[{"role": "user", "content": user_prompt}],
    max_tokens=1000,
    temperature=0.7
)

Zero secrets. Zero rotation headaches. Zero 3 AM pages because a key expired.

---

Step 4: Build a Resilient API Layer

The Throttling Reality

Azure OpenAI throttles at two levels:

• TPM (Tokens Per Minute) - total token throughput
• RPM (Requests Per Minute) - total request count

When you hit these limits, you get 429 Too Many Requests. In production, you need to handle this gracefully.

Retry Logic with Exponential Backoff

import asyncio
import random
from openai import AzureOpenAI, RateLimitError, APITimeoutError
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential_jitter,
    retry_if_exception_type
)

@retry(
    retry=retry_if_exception_type((RateLimitError, APITimeoutError)),
    wait=wait_exponential_jitter(initial=1, max=60, jitter=5),
    stop=stop_after_attempt(5)
)
async def call_openai_with_retry(client, messages: list, model: str) -> str:
    response = await client.chat.completions.acreate(
        model=model,
        messages=messages,
        max_tokens=800,
        timeout=30
    )
    return response.choices[0].message.content

The jitter is critical - without it, all your retrying clients synchronize and create a thundering herd that makes the throttling worse.

Circuit Breaker Pattern

For high-traffic applications, wrap your OpenAI calls in a circuit breaker:

from pybreaker import CircuitBreaker, CircuitBreakerError

openai_breaker = CircuitBreaker(
    fail_max=5,           # Open circuit after 5 failures
    reset_timeout=60,     # Try again after 60 seconds
    name="azure-openai"
)

async def safe_openai_call(messages: list) -> str | None:
    try:
        return await openai_breaker.call_async(
            call_openai_with_retry,
            client,
            messages,
            "gpt-4o-prod"
        )
    except CircuitBreakerError:
        # Circuit is open - return cached response or graceful degradation
        return get_fallback_response(messages)
    except Exception as e:
        logger.error(f"OpenAI call failed after retries: {e}")
        return None

---

Step 5: Implement Azure API Management as a Gateway

Running Azure OpenAI calls directly from your application works for simple cases, but at scale you want API Management (APIM) sitting in front:

Why APIM for Azure OpenAI?

• Centralized rate limiting per client/team
• Token usage tracking per subscription key
• Semantic caching (cache similar prompts, slash costs)
• Load balancing across multiple Azure OpenAI deployments

APIM Policy for Token Tracking and Rate Limiting

<policies>
  <inbound>
    <base />
    
    <!-- Rate limit by subscription key: 10K tokens/minute -->
    <rate-limit-by-key 
      calls="100" 
      renewal-period="60" 
      counter-key="@(context.Subscription.Id)" />
    
    <!-- Route to the right backend based on model requested -->
    <choose>
      <when condition="@(context.Request.Body.As<JObject>()["model"]
                         .ToString().Contains("gpt-4o"))">
        <set-backend-service 
          base-url="https://mycompany-openai-prod.openai.azure.com/" />
      </when>
      <otherwise>
        <set-backend-service 
          base-url="https://mycompany-openai-fallback.openai.azure.com/" />
      </otherwise>
    </choose>
    
    <!-- Inject managed identity token -->
    <authentication-managed-identity 
      resource="https://cognitiveservices.azure.com/" />
  </inbound>
  
  <outbound>
    <base />
    <!-- Log token usage to Application Insights -->
    <log-to-eventhub logger-id="openai-usage-logger">
      @{
        var body = context.Response.Body.As<JObject>();
        return new JObject(
          new JProperty("subscriptionId", context.Subscription.Id),
          new JProperty("promptTokens", body["usage"]["prompt_tokens"]),
          new JProperty("completionTokens", body["usage"]["completion_tokens"]),
          new JProperty("totalTokens", body["usage"]["total_tokens"]),
          new JProperty("timestamp", DateTime.UtcNow)
        ).ToString();
      }
    </log-to-eventhub>
  </outbound>
</policies>

This single APIM policy gives you chargeback visibility - you can now tell Finance exactly which team consumed how many tokens.

---

Step 6: Observability - You Cannot Manage What You Cannot Measure

The Three Signals That Matter for Azure OpenAI

Signal	What to Measure	Why It Matters
Latency	P50, P95, P99 TTFT (time-to-first-token)	User experience
Token consumption	Prompt vs completion ratio	Cost optimization
Error rates	429s, 500s, timeouts	Reliability signals

Application Insights Integration

from applicationinsights import TelemetryClient
import time

tc = TelemetryClient(instrumentation_key=os.environ["APPINSIGHTS_KEY"])

async def instrumented_openai_call(messages: list, user_id: str) -> str:
    start_time = time.time()
    
    try:
        response = await call_openai_with_retry(client, messages, "gpt-4o-prod")
        duration_ms = (time.time() - start_time) * 1000
        
        # Track custom event with token usage
        tc.track_event(
            "OpenAICallSuccess",
            properties={
                "userId": user_id,
                "model": "gpt-4o-prod"
            },
            measurements={
                "durationMs": duration_ms,
                "promptTokens": response.usage.prompt_tokens,
                "completionTokens": response.usage.completion_tokens,
                "totalCost": calculate_cost(response.usage)
            }
        )
        tc.flush()
        return response.choices[0].message.content
        
    except Exception as e:
        tc.track_exception()
        tc.flush()
        raise

Alert Rules Worth Creating

# Alert when 429 rate spikes above 5% of requests
az monitor metrics alert create \
  --name "OpenAI-HighThrottlingRate" \
  --resource-group myapp-rg \
  --scopes /subscriptions/$SUB_ID/resourceGroups/myapp-rg/providers/\
Microsoft.CognitiveServices/accounts/openai-prod-eastus \
  --condition "avg SuccessfulCalls < 95" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action-group /subscriptions/$SUB_ID/resourceGroups/myapp-rg/providers/\
microsoft.insights/actionGroups/on-call-team

---

Step 7: Content Safety and Responsible AI

Production means real users. Real users will probe, test, and - intentionally or not - send content that your system should not process or return.

Azure AI Content Safety Integration

from azure.ai.contentsafety import ContentSafetyClient
from azure.ai.contentsafety.models import AnalyzeTextOptions
from azure.core.credentials import AzureKeyCredential

safety_client = ContentSafetyClient(
    endpoint=os.environ["CONTENT_SAFETY_ENDPOINT"],
    credential=DefaultAzureCredential()
)

async def analyze_content_safety(text: str) -> bool:
    """Returns True if content is safe to process"""
    request = AnalyzeTextOptions(text=text)
    response = safety_client.analyze_text(request)
    
    # Block if any category scores above threshold
    thresholds = {
        "hate": 2,
        "self_harm": 0,    # Zero tolerance
        "sexual": 2,
        "violence": 2
    }
    
    for category in response.categories_analysis:
        threshold = thresholds.get(category.category.lower(), 2)
        if category.severity > threshold:
            logger.warning(
                f"Content blocked: {category.category} "
                f"severity={category.severity}"