Model Deployment Choices
Level Up Team
This post summarizes Model Deployment Choices, module 3.3 of Level Up’s Azure AI Fundamentals series. The lesson is on YouTube as Azure AI Fundamentals AI-901 Exam: 3.3: Model Deployment Choices. Module 3.2 is Model Families and Capabilities.
Start with the workload, not a familiar model name
Choose a model by the application’s operating requirements: input and output modalities, task complexity, expected quality, context-window size, latency and throughput, cost per request, and safety or deployment constraints. A benchmark leader can still perform poorly against those constraints. Model choice and deployment choice are related, and they are separate decisions.
A deployment makes a model callable
The flow is application request, model endpoint, inference, response. The request carries authentication, a deployment or model identifier, the input, and generation settings. The endpoint returns the model’s generated output.
Deployment prepares a model for inference. Applications commonly call deployed models through REST APIs or SDKs. The deployment configuration selects the model, model version, capacity type, and processing location. Microsoft Foundry primarily exposes models through serverless API deployments, with managed compute available for some open and custom models. See deployments in Microsoft Foundry.
Availability depends on model, region, and deployment type
Confirm current availability before you treat a catalog listing as a plan. Check that the model is available in the required Azure location, that the region supports the deployment type, that quota or capacity is available, that data-processing locations meet organizational requirements, and that the model version has an acceptable lifecycle.
Availability can differ by model, version, region, and subscription. A model in the catalog is not a guarantee that every region or deployment type supports it. Regional choice can affect compliance, latency, capacity, and access to recent models. Global or data-zone deployment types can broaden availability. A regional deployment can tighten control over processing location. Verify the current model and region availability before deployment.
Shared capacity and dedicated capacity fit different traffic
Shared capacity, often described as a standard deployment, lets customers use a common pool and is often billed by consumption. Dedicated capacity reserves processing for one customer and improves how predictable throughput is. A reservation may incur cost even when usage is low. Provisioned throughput is the dedicated pattern: more predictable throughput and less variation in latency. AI-901 stays at that conceptual level. You do not need to calculate provisioned throughput units. Microsoft documents deployment types and provisioned throughput.
| Capacity approach | Behavior | Best fit |
|---|---|---|
| Shared capacity | Customers use a common pool of model-processing capacity | Variable traffic, evaluation, and general workloads |
| Dedicated capacity | The deployment reserves processing capacity for one customer | Sustained demand and predictable performance |
Throughput, latency, cost, and quality compete
| Measure | Question |
|---|---|
| Throughput | How much work can the deployment process over time? |
| Latency | How long does one request take to return a response? |
| Cost | What does inference capacity or consumption cost? |
| Quality | How well does the model perform the required task? |
A live voice assistant needs low latency. Batch document processing prioritizes throughput and cost. A complex legal analysis may justify a slower, stronger model. High-volume classification may favor a smaller model. A deployment can have strong throughput while one request still takes several seconds. Longer prompts, longer outputs, and extra reasoning increase latency and cost.
The smallest model that meets the quality bar is often the right fit
Parameter count alone does not determine application value. Larger models may do better on difficult or unfamiliar tasks, follow instructions more broadly, and handle nuance. Smaller models may offer lower latency, lower cost, higher throughput, and enough quality for a focused task. A focused extraction or classification task may gain little from the largest model. “Most capable” and “best for this application” can be different answers.
Route expensive capability to the requests that need it
A support assistant receives thousands of routine questions a day. Most answers come from company documentation. Customers expect quick responses, and the company must control cost.
- Use an embedding model to retrieve relevant documentation.
- Use a capable smaller language model for routine answers.
- Route difficult cases to a stronger reasoning model.
- Begin with shared capacity while traffic is variable.
- Consider dedicated capacity when demand is sustained and predictable.
| Requirement | Design response |
|---|---|
| Current company information | Ground responses with retrieved documentation |
| Fast routine answers | Use a lower-latency model |
| Complex exceptions | Route selectively to a stronger model |
| Variable early traffic | Use shared consumption capacity |
| Stable high traffic | Evaluate dedicated capacity |
A single large model for every request can raise latency and cost without improving routine answers.
Bring this to your team
Level Up can run a personalized Azure AI Fundamentals workshop for your team. Tell us a little about your group and we’ll reply with pricing.