Serving AI to 10,000 customers a day on Azure AI Foundry

The problem

An AI feature that real customers use has none of the glamour of a demo. On the product I worked on, the assistant had to serve 10,000+ customers every day: answer questions in under a few seconds, never leak one customer's data to another, refuse the requests it should refuse, and cost a predictable amount per month. On top of that, the ops team behind it was small — the platform had to absorb load without someone paging around replica counts.

This is a different engineering problem than "deploy a model." The resolution was a multi-service containerized backend running on Azure, orchestrated around an Azure AI Foundry hub that owns models, data, and evaluation in one governed place.

The architecture

The system splits into four logical services, each independently deployable and scaleable:

• Gateway — the single entry point: authN/authZ, rate limiting, and tenant isolation.
• Orchestrator — the RAG pipeline: query rewriting, retrieval, prompt assembly, and model calls.
• Retriever — a vector search service over the product knowledge base.
• Evaluator — offline eval jobs that score answer quality before a model version ships.

The hub is the governance boundary. Models, deployments, data connections, and content filters all live there, defined as code:

resource "azurerm_ai_foundry" "hub" {
  name                = "ai-hub-${var.env}"
  location            = var.location
  resource_group_name = var.resource_group
  storage_account_id  = azurerm_storage_account.ai.id
  key_vault_id        = azurerm_key_vault.ai.id
}

resource "azurerm_cognitive_deployment" "gpt" {
  name                 = "gpt-4o"
  cognitive_account_id = azurerm_cognitive_account.openai.id
  model {
    format  = "OpenAI"
    name    = "gpt-4o"
    version = "2024-08-06"
  }
  sku { name = "GlobalStandard", capacity = 20 }
}

Containerized backend services

Each service runs as a container behind the gateway, deployed on Azure Container Apps with the infrastructure as Bicep — revision-based rollouts and a per-service autoscaler built in:

resource orchestrator 'Microsoft.App/containerApps@2023-05-01' = {
  name: 'orchestrator-${env}'
  location: location
  properties: {
    managedEnvironmentId: envId
    configuration: {
      activeRevisionsMode: 'Single'
      secrets: [{ name: 'openai-key', identity: systemAssigned }]
    }
    template: {
      scale: { minReplicas: 2, maxReplicas: 20 }
      containers: [{
        name: 'orchestrator'
        image: 'acr.azurecr.io/orchestrator:${tag}'
        env: [
          { name: 'AZURE_OPENAI_ENDPOINT', value: endpoint }
          { name: 'AZURE_OPENAI_MODEL', value: 'gpt-4o' }
        ]
      }]
    }
  }
}

Guardrails as code

Safety is not a prompt. Content filters and data protection are configured at the model deployment level in AI Foundry, so every request — from every service — passes the same filters. Tenant isolation is enforced in the gateway: the orchestrator only ever sees the calling tenant's context, and retrieval is scoped by tenant ID in every query.

{
  "filters": {
    "hate": "safe",
    "violence": "safe",
    "sexual": "safe",
    "selfHarm": "safe"
  },
  "blocklists": ["customer_pii_blocklist"],
  "promptTransformation": { "blocklistEnabled": true }
}

Evaluation before release

No model version ships on vibes. An offline evaluation loop scores candidate versions against a held-out set of real customer questions with expected answers. We track groundedness, relevance, and refusal accuracy, and gate the rollout on the score delta — a worse version cannot reach the gateway.

Observability and cost

Every request emits telemetry to Application Insights with tenant, model, latency, and token counts. That gave the business the two numbers they cared about: cost per conversation and p95 latency. When traffic spiked during business hours, the container autoscaler absorbed it; when it drained, the service scaled back down. The small ops team never touched a replica count.

Lessons learned

• Tenant isolation is an architecture concern, not a model concern. A shared model is fine if every request is scoped at the gateway and the retriever.
• Guardrails belong at the platform layer. Content filters in AI Foundry are enforced for every service, so one misconfigured service cannot bypass safety.
• Evaluation is the gate. The ability to score model versions against a held-out dataset is what turns "ship a new model" into a deploy decision instead of a gamble.
• Serverless containers beat managing a fleet. For a workload that scales with customers rather than a fixed baseline, Container Apps autoscaling was the right call.

Serving 10,000 customers a day is not about a smarter model. It is a platform problem — and like every platform problem, the answer is boring, scalable infrastructure with the interesting decisions made once, at the platform layer.