How to Deploy an AI Chatbot on Azure
Using Azure OpenAI and App Service
From zero to a production-grade AI chatbot: provision Azure OpenAI, write a streaming Flask API backend, deploy it on Azure App Service with Managed Identity, wire in conversation history and content safety, and instrument it with Application Insights — all with complete code and Terraform IaC. No API keys in environment variables. No hardcoded secrets. No half-finished PoC patterns.
Architecture Overview: All Components and How They Connect
The architecture for a production Azure OpenAI chatbot has five distinct layers: the user's browser, the App Service hosting the chatbot backend, the Azure OpenAI service processing the completions, the supporting services (Key Vault, Azure Cache for Redis for session storage, Application Insights for observability), and the identity layer tying them together without credentials. Understanding how these layers connect before writing a single line of code prevents the most common production failures — API keys leaked into logs, conversation history stored in server memory (destroyed on every App Service restart), and no visibility into model response times or error rates.
Choosing the Right Azure OpenAI Model for Your Chatbot
Model selection determines the balance between response quality, latency, and cost. For most chatbot deployments, the correct choice is not the most powerful model available — it is the model that delivers acceptable quality at the latency and cost target your use case requires.
| Model | Context Window | Best For | Approximate Cost | Latency |
|---|---|---|---|---|
| gpt-4.1-mini | 1M tokens | High-volume chatbots, internal assistants, customer support automation. Best cost-to-quality ratio for conversational workloads in 2026. | $0.40/$1.60 per 1M tokens (in/out) | Fast — recommended default |
| gpt-4.1 | 1M tokens | Complex reasoning, multi-step analysis, document-heavy chatbots. Use when gpt-4.1-mini quality is insufficient. | $2.00/$8.00 per 1M tokens | Moderate |
| gpt-5-nano | 128K tokens | Ultra-high-volume, cost-sensitive applications where speed matters more than depth. Simple FAQ bots, intent classification. | $0.05/$0.40 per 1M tokens | Very fast |
| gpt-5 | 256K tokens | Mission-critical chatbots requiring the highest reasoning quality — legal assistants, medical triage, complex financial guidance. | $1.25/$10.00 per 1M tokens | Moderate to slow |
This guide uses gpt-4.1-mini as the deployment model. It delivers production-quality conversational responses, supports the full 1M-token context window for long conversation histories, streams tokens efficiently, and costs a fraction of gpt-4.1. Swap the deployment name to upgrade the model at any time — no code changes required.
Create a resource group
All chatbot resources will live in one resource group for clean lifecycle management. az group create --name rg-chatbot-prod --location eastus. Choose a region where your target model is available — check the Azure OpenAI model availability page for your subscription.
Create the Azure OpenAI resource
az cognitiveservices account create --name oai-chatbot-prod --resource-group rg-chatbot-prod --kind OpenAI --sku S0 --location eastus --custom-domain oai-chatbot-prod. The --custom-domain flag creates a subdomain at oai-chatbot-prod.openai.azure.com — required for Managed Identity authentication (API key auth uses a different endpoint format).
Deploy the gpt-4.1-mini model
az cognitiveservices account deployment create --name oai-chatbot-prod --resource-group rg-chatbot-prod --deployment-name gpt-4.1-mini --model-name gpt-4.1-mini --model-version "2025-04-14" --model-format OpenAI --sku-capacity 100 --sku-name "Standard". The --sku-capacity 100 allocates 100,000 TPM. Note the deployment name — it is the value you pass to the API as the model parameter.
Record the endpoint URL
az cognitiveservices account show --name oai-chatbot-prod --resource-group rg-chatbot-prod --query properties.endpoint -o tsv. The output will be https://oai-chatbot-prod.openai.azure.com/. This is the only value your application needs — no API key required when using Managed Identity.
The backend has three endpoints: POST /api/chat (the streaming chat endpoint), DELETE /api/chat/<session_id> (clear conversation history), and GET /api/health (App Service health probe). The streaming endpoint uses Server-Sent Events (SSE) — the Flask response is a generator that yields token chunks as they arrive from Azure OpenAI.
openai==1.84.0
azure-identity==1.23.0
azure-keyvault-secrets==4.10.0
redis==5.2.0
opencensus-ext-azure==1.1.14
gunicorn==23.0.0
import os, json, time, uuid, logging
from flask import Flask, request, Response, jsonify, stream_with_context
from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from azure.keyvault.secrets import SecretClient
import redis
app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# ── CONFIGURATION ────────────────────────────────────────
AZURE_OPENAI_ENDPOINT = os.environ["AZURE_OPENAI_ENDPOINT"]
OPENAI_DEPLOYMENT = os.environ.get("OPENAI_DEPLOYMENT", "gpt-4.1-mini")
KV_URL = os.environ.get("KEY_VAULT_URL") # e.g. https://kv-chatbot.vault.azure.net
SYSTEM_PROMPT = """You are a helpful, professional AI assistant.
You answer questions clearly and concisely.
If you don't know something, say so honestly.
Never make up facts or cite sources you're not certain of."""
# ── MANAGED IDENTITY AUTH TO AZURE OPENAI ───────────────
# No API key. DefaultAzureCredential uses the App Service's
# System-Assigned Managed Identity automatically in production.
credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
credential,
"https://cognitiveservices.azure.com/.default"
)
openai_client = AzureOpenAI(
azure_endpoint=AZURE_OPENAI_ENDPOINT,
azure_ad_token_provider=token_provider, # MI token, not API key
api_version="2024-12-01-preview"
)
# ── REDIS FOR CONVERSATION HISTORY ──────────────────────
def get_redis_client():
if KV_URL:
kv_client = SecretClient(vault_url=KV_URL, credential=credential)
redis_conn = kv_client.get_secret("redis-connection-string").value
else:
redis_conn = os.environ.get("REDIS_CONNECTION", "redis://localhost:6379")
return redis.from_url(redis_conn, decode_responses=True)
redis_client = get_redis_client()
HISTORY_TTL = 3600 # Expire sessions after 1 hour of inactivity
MAX_HISTORY_TURNS = 20 # Keep last 20 turns (40 messages)
# ── CONVERSATION HISTORY HELPERS ─────────────────────────
def get_history(session_id: str) -> list:
try:
raw = redis_client.get(f"chat:history:{session_id}")
return json.loads(raw) if raw else []
except Exception as e:
logger.warning(f"Redis read failed: {e}")
return [] # Degrade gracefully — stateless conversation
def save_history(session_id: str, history: list):
try:
# Trim to last MAX_HISTORY_TURNS exchanges
trimmed = history[-(MAX_HISTORY_TURNS * 2):]
redis_client.setex(f"chat:history:{session_id}",
HISTORY_TTL, json.dumps(trimmed))
except Exception as e:
logger.warning(f"Redis write failed: {e}")
# ── STREAMING CHAT ENDPOINT ──────────────────────────────
@app.route("/api/chat", methods=["POST"])
def chat():
data = request.get_json()
user_message = data.get("message", "").strip()
session_id = data.get("session_id") or str(uuid.uuid4())
if not user_message:
return jsonify({"error": "message is required"}), 400
# Build messages array: system + history + new user message
history = get_history(session_id)
messages = [
{"role": "system", "content": SYSTEM_PROMPT}
] + history + [{"role": "user", "content": user_message}]
def generate_stream():
full_response = ""
try:
stream = openai_client.chat.completions.create(
model=OPENAI_DEPLOYMENT,
messages=messages,
stream=True, # Token-by-token streaming
max_tokens=1024,
temperature=0.7,
)
# Yield session_id first so client can track the session
yield f"data: {json.dumps({'session_id': session_id})}\n\n"
for chunk in stream:
delta = chunk.choices[0].delta if chunk.choices else None
if delta and delta.content:
full_response += delta.content
yield f"data: {json.dumps({'token': delta.content})}\n\n"
# Stream complete — save exchange to Redis
history.append({"role": "user", "content": user_message})
history.append({"role": "assistant", "content": full_response})
save_history(session_id, history)
yield "data: [DONE]\n\n"
except Exception as e:
logger.error(f"OpenAI error: {e}")
yield f"data: {json.dumps({'error': str(e)})}\n\n"
return Response(
stream_with_context(generate_stream()),
mimetype="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no", # Disable nginx buffering
}
)
@app.route("/api/chat/<session_id>", methods=["DELETE"])
def clear_history(session_id):
redis_client.delete(f"chat:history:{session_id}")
return jsonify({"cleared": session_id})
@app.route("/api/health")
def health():
return jsonify({"status": "ok", "model": OPENAI_DEPLOYMENT})
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8000, debug=False)
Storing conversation history in the App Service process memory is the single most common production mistake in chatbot deployments. App Service instances restart, scale out to multiple instances, and receive different requests on different instances — process memory is not shared and not durable. Any history stored in memory is lost on the next restart or load-balanced to a different instance.
Azure Cache for Redis is the correct solution: it is a distributed in-memory store accessible from any App Service instance, with configurable TTL for automatic session expiry. The history stored in Redis is a JSON-serialised array of {role, content} objects — exactly the format the Azure OpenAI Chat Completions API requires.
Create Azure Cache for Redis
az redis create --name redis-chatbot-prod --resource-group rg-chatbot-prod --location eastus --sku Basic --vm-size c1. Basic C1 (1GB) is sufficient for most chatbot deployments. Use Standard or Premium for production SLA guarantees and geo-replication.
Store the Redis connection string in Key Vault
Get the connection string: az redis list-keys --name redis-chatbot-prod --resource-group rg-chatbot-prod --query primaryKey -o tsv. The full connection string format: redis-chatbot-prod.redis.cache.windows.net:6380,password=KEY,ssl=True. Store it: az keyvault secret set --vault-name kv-chatbot-prod --name redis-connection-string --value "YOUR-CONN-STRING". The App Service retrieves it at startup via Managed Identity — no connection string in environment variables.
Set appropriate history TTL and turn limits
Configure two limits: HISTORY_TTL (seconds before an idle session expires — 3600 for 1 hour, 86400 for 24 hours) and MAX_HISTORY_TURNS (number of conversation turns to keep — 20 turns = 40 messages). The turn limit prevents history from growing unbounded and consuming your context window budget. With gpt-4.1-mini's 1M context window, you can afford longer histories — but longer histories increase per-request token costs.
Create App Service Plan and Web App
az appservice plan create --name asp-chatbot-prod --resource-group rg-chatbot-prod --is-linux --sku P1v3. Then: az webapp create --name app-chatbot-prod --resource-group rg-chatbot-prod --plan asp-chatbot-prod --runtime "PYTHON:3.12". Use P1v3 or higher for production — the B1 and B2 Basic tiers do not include autoscaling and have limited memory for Python apps with multiple dependencies.
Configure the startup command
az webapp config set --name app-chatbot-prod --resource-group rg-chatbot-prod --startup-file "gunicorn --bind=0.0.0.0 --timeout 600 --worker-class gevent --workers 4 app:app". The --timeout 600 is critical — streaming responses take longer than App Service's default 230-second timeout. Use gevent workers to handle concurrent SSE connections without blocking. Install gevent: add gevent==24.2.1 to requirements.txt.
Set application settings (environment variables)
az webapp config appsettings set --name app-chatbot-prod --resource-group rg-chatbot-prod --settings AZURE_OPENAI_ENDPOINT="https://oai-chatbot-prod.openai.azure.com/" OPENAI_DEPLOYMENT="gpt-4.1-mini" KEY_VAULT_URL="https://kv-chatbot-prod.vault.azure.net" SCM_DO_BUILD_DURING_DEPLOYMENT=true. Do not set an API key — authentication uses Managed Identity.
Deploy the application code
From the project root: az webapp deploy --name app-chatbot-prod --resource-group rg-chatbot-prod --src-path . --type zip. The zip deploy uploads your code, App Service installs dependencies from requirements.txt during build, and starts the gunicorn server. Monitor deployment: az webapp log tail --name app-chatbot-prod --resource-group rg-chatbot-prod.
Verify the deployment
curl https://app-chatbot-prod.azurewebsites.net/api/health. Expected response: {"status": "ok", "model": "gpt-4.1-mini"}. If the health endpoint returns a 500, check the application logs for startup errors — the most common cause is a missing environment variable or a Managed Identity RBAC assignment not yet propagated.
async function sendMessage(message, sessionId = null) {
const response = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ message, session_id: sessionId })
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
let currentSessionId = sessionId;
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
// Parse SSE events from buffer
const lines = buffer.split("\n");
buffer = lines.pop(); // Keep incomplete line in buffer
for (const line of lines) {
if (!line.startsWith("data: ")) continue;
const payload = line.slice(6).trim();
if (payload === "[DONE]") return currentSessionId;
try {
const data = JSON.parse(payload);
if (data.session_id) currentSessionId = data.session_id;
if (data.token) appendTokenToUI(data.token); // Update DOM
if (data.error) showError(data.error);
} catch (e) { /* ignore malformed chunk */ }
}
}
return currentSessionId;
}
This is the most important security step in the entire guide. Every tutorial on the internet puts the Azure OpenAI API key in an environment variable. Every one of those tutorials is teaching you to create a credential that can leak into logs, error messages, diagnostic exports, and git commits. The correct approach is Managed Identity — no key exists, therefore no key can leak.
Enable System-Assigned Managed Identity on the App Service
az webapp identity assign --name app-chatbot-prod --resource-group rg-chatbot-prod. Note the principalId in the output — this is the object ID you will use for RBAC assignments.
Assign "Cognitive Services OpenAI User" role on the OpenAI resource
This role grants the App Service permission to call the Azure OpenAI Chat Completions API using Entra token authentication: az role assignment create --assignee PRINCIPAL-ID-FROM-STEP-1 --role "Cognitive Services OpenAI User" --scope /subscriptions/SUB-ID/resourceGroups/rg-chatbot-prod/providers/Microsoft.CognitiveServices/accounts/oai-chatbot-prod. RBAC propagation takes 2–5 minutes — your first deployment attempt may return 401 if you deploy immediately after this step.
Assign "Key Vault Secrets User" on the Key Vault
az role assignment create --assignee PRINCIPAL-ID --role "Key Vault Secrets User" --scope /subscriptions/SUB-ID/resourceGroups/rg-chatbot-prod/providers/Microsoft.KeyVault/vaults/kv-chatbot-prod. This grants the App Service read-only access to secrets — it can read the Redis connection string but cannot create or modify secrets.
Disable API key authentication on the Azure OpenAI resource
Once Managed Identity is working: az cognitiveservices account update --name oai-chatbot-prod --resource-group rg-chatbot-prod --api-properties "disableLocalAuth=true". This permanently disables the API key endpoint — even if someone obtains the key value, it cannot be used. This is the enterprise security baseline: no credential-based access, Entra-only authentication.
Azure OpenAI's built-in content filtering automatically blocks harmful content categories (hate, violence, sexual content, self-harm) on both inputs and outputs. This is enabled by default on all deployments. Your application code should handle the ContentFilterFinishReason that signals a filtered response and present an appropriate user-facing message without leaking the filter reason or the model's raw error.
Quota management prevents the 429 errors that crash production chatbots under load. Your deployment has a TPM (tokens per minute) and RPM (requests per minute) ceiling. The most reliable mitigation is exponential backoff with jitter on 429 responses combined with per-session request rate limiting to prevent any single user from consuming the entire quota.
from openai import RateLimitError, BadRequestError
def call_openai_with_retry(messages, deployment, max_retries=3):
for attempt in range(max_retries):
try:
return openai_client.chat.completions.create(
model=deployment,
messages=messages,
stream=True,
max_tokens=1024,
)
except RateLimitError:
if attempt == max_retries - 1:
raise
# Exponential backoff with full jitter
wait = (2 ** attempt) + random.uniform(0, 1)
logger.warning(f"Rate limited. Retry {attempt+1} in {wait:.1f}s")
time.sleep(wait)
except BadRequestError as e:
# Content filter triggered — present user-friendly message
if "content_filter" in str(e):
logger.warning("Content filter triggered")
raise ValueError("I can't respond to that request.")
raise
# Per-session rate limiting using Redis counters
def check_session_rate_limit(session_id: str, max_rpm: int = 10) -> bool:
key = f"rate:{session_id}:{int(time.time() // 60)}" # Per-minute bucket
count = redis_client.incr(key)
if count == 1:
redis_client.expire(key, 70) # 70s TTL for safety margin
return count <= max_rpm
Calculate your expected TPM usage before launch: Average tokens per request × Peak requests per minute = Required TPM. A typical chatbot exchange uses 500–2,000 tokens per turn (system prompt + history + user message + response). At 50 concurrent users each sending one message per minute, you need 50,000–100,000 TPM minimum. The default Standard deployment quota is 100,000 TPM. Request a quota increase via the Azure OpenAI Studio portal before you need it — approval takes 1–3 business days.
A chatbot without monitoring is a chatbot you cannot improve. Application Insights gives you request latency by endpoint, error rates, token usage trends, and user session analytics. Configure it from day one — retrospective instrumentation after a production incident is always more expensive than proactive setup.
Create Application Insights and link to App Service
az monitor app-insights component create --app appi-chatbot-prod --location eastus --resource-group rg-chatbot-prod --kind web --application-type web. Get the connection string: az monitor app-insights component show --app appi-chatbot-prod --resource-group rg-chatbot-prod --query connectionString -o tsv. Add it as an app setting: APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=...".
Instrument token usage as custom metrics
Log prompt tokens, completion tokens, and model latency as custom telemetry on every request. This data drives cost attribution, capacity planning, and performance dashboards. Use the opencensus-ext-azure package already in requirements.txt.
Create Azure Monitor alerts
Set alerts on: P95 request latency > 10 seconds (Azure OpenAI under load), error rate > 5% over 5 minutes (content filter or quota issues), and 429 error count > 0 over 1 minute (quota exhausted). Route alerts to your team's email or Teams webhook.
from opencensus.ext.azure.trace_exporter import AzureExporter
from opencensus.trace.samplers import ProbabilitySampler
from opencensus.trace.tracer import Tracer
# Configure App Insights logger
APPI_CONN = os.environ.get("APPLICATIONINSIGHTS_CONNECTION_STRING", "")
if APPI_CONN:
ai_logger = logging.getLogger("ai_telemetry")
ai_logger.addHandler(AzureLogHandler(connection_string=APPI_CONN))
ai_logger.setLevel(logging.INFO)
def log_chat_telemetry(session_id: str, model: str,
prompt_tokens: int, completion_tokens: int,
latency_ms: float, success: bool):
if not APPI_CONN: return
# Log as custom event — appears in App Insights "customEvents" table
ai_logger.info("ChatRequest", extra={
"custom_dimensions": {
"session_id": session_id,
"model": model,
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": prompt_tokens + completion_tokens,
"latency_ms": latency_ms,
"success": success
# IMPORTANT: never log message content — contains user PII
}
})
# KQL query to monitor token usage in App Insights:
# customEvents
# | where name == "ChatRequest"
# | extend tokens = toint(customDimensions.total_tokens)
# | summarize total_tokens = sum(tokens), requests = count() by bin(timestamp, 1h)
# | render timechart
Terraform IaC: Full Stack in Code
resource "azurerm_cognitive_account" "openai" {
name = "oai-chatbot-${var.env}"
resource_group_name = var.resource_group_name
location = var.location
kind = "OpenAI"
sku_name = "S0"
custom_subdomain_name = "oai-chatbot-${var.env}"
local_auth_enabled = false # Disable API key auth — Entra only
}
resource "azurerm_cognitive_deployment" "gpt" {
name = "gpt-4.1-mini"
cognitive_account_id = azurerm_cognitive_account.openai.id
model { format = "OpenAI"; name = "gpt-4.1-mini"; version = "2025-04-14" }
scale { type = "Standard"; capacity = 100 } # 100K TPM
}
# ── APP SERVICE ───────────────────────────────────────────
resource "azurerm_service_plan" "main" {
name = "asp-chatbot-${var.env}"
resource_group_name = var.resource_group_name
location = var.location
os_type = "Linux"
sku_name = "P1v3"
}
resource "azurerm_linux_web_app" "chatbot" {
name = "app-chatbot-${var.env}"
resource_group_name = var.resource_group_name
location = var.location
service_plan_id = azurerm_service_plan.main.id
identity { type = "SystemAssigned" }
site_config {
application_stack { python_version = "3.12" }
app_command_line = "gunicorn --bind=0.0.0.0 --timeout 600 --worker-class gevent --workers 4 app:app"
http2_enabled = true
}
app_settings = {
"AZURE_OPENAI_ENDPOINT" = azurerm_cognitive_account.openai.endpoint
"OPENAI_DEPLOYMENT" = "gpt-4.1-mini"
"KEY_VAULT_URL" = azurerm_key_vault.main.vault_uri
"APPLICATIONINSIGHTS_CONNECTION_STRING" = azurerm_application_insights.main.connection_string
"SCM_DO_BUILD_DURING_DEPLOYMENT" = "true"
# No API key — MI handles auth
}
}
# ── RBAC ASSIGNMENTS ─────────────────────────────────────
resource "azurerm_role_assignment" "openai_user" {
scope = azurerm_cognitive_account.openai.id
role_definition_name = "Cognitive Services OpenAI User"
principal_id = azurerm_linux_web_app.chatbot.identity[0].principal_id
}
resource "azurerm_role_assignment" "kv_secrets" {
scope = azurerm_key_vault.main.id
role_definition_name = "Key Vault Secrets User"
principal_id = azurerm_linux_web_app.chatbot.identity[0].principal_id
}
Production Readiness Checklist
Key Takeaways
Frequently Asked Questions
Related FAVRITE Articles
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Stop Using Connection Strings: A Step-by-Step Guide to Azure Managed Identities in 2026
- Architectural Runbook: Optimizing Azure AI Search for Enterprise RAG Workflows
- Azure App Service Log Stream with Least Privilege: The Complete Guide