Speech AI
Level Up Team
This post summarizes Speech AI, module 2.3 of Level Up’s Azure AI Fundamentals series. The lesson is on YouTube as Azure AI Fundamentals AI-901 Exam: 2.3: Speech AI. Module 2.2 is Text Analysis.
Follow the direction of the transformation
Speech questions are usually about direction: spoken language in and text out, text in and spoken language out, a change of language, or a conversational response from a multimodal model.
Speech recognition listens
Speech recognition converts spoken language into text. On AI-901, speech recognition, speech to text, transcription, and automatic speech recognition are the same capability.
A customer who describes a damaged product becomes the text “The screen cracked during delivery.” Common uses are call and meeting transcripts, captions, voice commands, and search across recorded conversations. The exam signal is converting spoken words into written text.
Speech synthesis speaks
Speech synthesis converts text into audible speech. It is also called text to speech or voice synthesis.
Teams use it to read an assistant’s answer aloud, generate accessibility narration, voice automated announcements, and give a virtual assistant a voice. Recognition listens. Synthesis speaks. If the system must produce audible speech from text, the capability is speech synthesis.
Translation changes the language
Converting audio to text is not automatically translation. Translation requires a change between languages. Ask whether the output must preserve the input language.
| Capability | Language changes? | Example |
|---|---|---|
| Transcription | No | Spoken Spanish → written Spanish |
| Speech translation | Yes | Spoken Spanish → written English |
| Speech-to-speech translation | Yes | Spoken Spanish → spoken English |
A transcript is not the same as a response
A multimodal model can respond directly to spoken input. A customer might speak a question while showing a photograph of a damaged product. The model can use the words that were spoken, tone or pacing, conversation history, and additional text or image inputs.
If the scenario needs only the written words, choose speech recognition. If it needs an intelligent conversational response to speech — and perhaps an image — choose a multimodal model. If it needs an action completed, the workload may be agentic AI.
Real time when someone is waiting
Real-time speech handles audio while the speaker is talking. It fits voice assistants, live captions, contact-center conversations, live interpretation, and interactive voice applications. The priorities are low latency, partial or streaming results, and rapid turn-taking.
Choose real time when a person is waiting for an immediate response.
Batch when the audio is already recorded
Batch speech handles complete recordings asynchronously. It fits recorded meetings, call-center archives, podcast transcription, large media collections, and compliance analysis. The priorities are scale, throughput, and rich results after completion.
Choose batch when existing audio must be processed in volume and nobody is waiting on the line.
| Requirement | Approach |
|---|---|
| Display captions during a meeting | Real time |
| Transcribe 50,000 recorded calls | Batch |
| Let a customer speak to an assistant | Real time |
| Analyze last month’s interviews | Batch |
| Interpret a live conversation | Real time |
Match the direction, then ask who is waiting
| The system must… | Capability |
|---|---|
| Produce captions from a lecture | Speech recognition |
| Read an answer aloud | Speech synthesis |
| Convert spoken French into English | Speech translation |
| Answer a spoken question about an image | Multimodal interaction |
| Transcribe an archive of recorded calls | Batch speech recognition |
| Respond during a live phone call | Real-time speech |
A company with 10,000 recorded customer-service calls that wants searchable text by the next morning needs batch speech recognition. The input is completed recordings, the output is text, and nobody is waiting for an immediate response.
Bring this to your team
Level Up can run a personalized Azure AI Fundamentals workshop for your team. Tell us a little about your group and we’ll reply with pricing.