Configuration Parameters
Level Up Team
This post summarizes Configuration Parameters, module 3.4 of Level Up’s Azure AI Fundamentals series. The lesson is on YouTube as Azure AI Fundamentals AI-901 Exam: 3.4: Configuration Parameters. Module 3.3 is Model Deployment Choices.
Configuration changes behavior without retraining
Generation settings influence variation between responses, response length, where generation stops, and how consistently the model follows an assigned role. They apply at inference time. They do not retrain the model.
Temperature changes variation
Temperature changes how strongly the model favors high-probability tokens. A lower temperature produces more conservative choices and more consistent wording, which suits focused tasks. A higher temperature produces more varied and unexpected wording, which suits idea generation.
Extraction usually wants a low temperature. Factual summarization wants low to moderate. Creative generation can use moderate to high. Temperature changes variation. It does not add factual knowledge.
| Lower temperature | Higher temperature |
|---|---|
| More conservative choices | More varied choices |
| More consistent wording | More unexpected wording |
| Useful for focused tasks | Useful for idea generation |
Top-p limits the candidate pool
Top-p uses nucleus sampling. The model ranks possible next tokens by probability, builds the smallest candidate set whose cumulative probability reaches top-p, and selects from that set. A lower top-p restricts generation to a narrower set. A higher top-p allows a broader range of tokens.
Temperature changes randomness across token probabilities. Top-p limits selection to a cumulative probability range. To see the difference, change top-p while holding temperature constant. A run at temperature 1.0 and top-p 0.2 draws from a smaller pool than temperature 1.0 and top-p 1.0.
Output limits and stop sequences end generation
Generation can end at a natural completion, a stop sequence, or the token limit. Maximum output tokens set a ceiling on generated output, which helps control latency and cost, and can cut off an unfinished response. A token limit is a ceiling, not a required length, and it does not map exactly to a word limit.
A stop sequence such as END_OF_SUMMARY ends generation when the model reaches that sequence. Stop conditions are useful for structured boundaries and multi-part workflows. They need careful selection so generation does not stop too early.
Consistency and creativity sit on a spectrum
Configuration can improve consistency. Generated output remains probabilistic.
| Requirement | Preferred configuration |
|---|---|
| Repeatable structured extraction | Conservative sampling and strict output instructions |
| Faithful summarization | Conservative sampling and a clear length constraint |
| Brainstorming alternatives | More permissive sampling |
| Marketing concepts | Moderate variation with explicit brand constraints |
System instructions define persistent behavior for the request
The application supplies system instructions separately from the user’s request. An example: “You are a customer-support assistant. Answer only from the supplied product documentation. If the documentation does not contain the answer, say that you cannot verify it. Return no more than three short paragraphs.”
System instructions can define the role, behavioral boundaries, response style, rules for using supplied information, and when to escalate or decline. A system instruction guides model behavior. It does not train or permanently modify the foundation model.
Set parameters for the task
Define the required output, start with conservative settings, evaluate representative examples, adjust one parameter at a time, and validate against criteria for that task. There is no universally correct temperature or top-p value.
| Task | Temperature or top-p | Output limit | Instructions |
|---|---|---|---|
| Extraction | Conservative | Enough for the schema | Specify fields, format, and missing-value behavior |
| Summarization | Conservative to moderate | Match the desired summary length | Preserve important facts and avoid unsupported details |
| Creative generation | Moderate to high | Allow room for alternatives | Define audience, tone, constraints, and format |
The lesson compares one prompt at two temperatures
The demonstration keeps the model, system instruction, and user prompt fixed, and changes only the sampling configuration. The prompt asks for a launch announcement with a headline, a description of no more than 80 words, and a one-sentence call to action. The system instruction asks for clear writing for IT professionals, no exaggerated claims, and the requested format.
A conservative run might use temperature 0.2, top-p 1.0, and a maximum of about 180 output tokens. A creative run uses the same prompt in a new chat at temperature 1.0. The conservative run tends toward a direct headline, familiar wording, and close format adherence. The creative run tends toward more varied headlines, less predictable phrasing, and weaker constraint adherence. Higher variation is useful when it improves the result for that task. A regulated support message wants the conservative end. An early brainstorm wants the permissive end.
Interfaces and supported parameters vary by model. Use a model that exposes temperature or a comparable sampling control.
Bring this to your team
Level Up can run a personalized Azure AI Fundamentals workshop for your team. Tell us a little about your group and we’ll reply with pricing.