Information Extraction Across Modalities
Level Up Team
This post summarizes Information Extraction Across Modalities, module 2.5 of Level Up’s Azure AI Fundamentals series. The lesson is on YouTube as Azure AI Fundamentals AI-901 Exam: 2.5: Information Extraction Across Modalities. Module 2.4 is Computer Vision and Image Generation.
The modality can change. The goal stays the same.
Information extraction turns unstructured content into structured fields. A support case might become customer: Maria Chen, product: TrailMaster, and issue: damaged screen. Inputs can be documents, forms, images, audio, or video. The input modality can change. The goal is still usable data.
Documents become records
Document extraction populates predefined fields from documents and forms: vendor, invoice number, date, total, line items, or contract parties. Typical inputs are invoices, receipts, contracts, tax forms, applications, and support documents.
A PDF invoice that becomes a structured invoice record is the pattern. The exam signal is predefined fields, not a free-form summary of the page.
Images can yield business fields
Images may contain printed or handwritten text, labels, identifiers, objects, tables, or visual evidence. A photograph of a shipping label can return tracking_number: "1Z8042" and destination: "Seattle".
OCR reads all visible characters. Information extraction returns specific business fields from the image.
Audio can yield more than a transcript
A recording can produce defined fields, not only a transcript. A recorded support call might return intent: replacement_request, product: TrailMaster, and customer_sentiment: negative. Other fields include speaker names, issue category, dates, commitments, and action items.
Speech recognition produces a transcript. Information extraction derives defined fields from the recording.
Video combines frames, speech, text, and time
Video can combine visual scenes, spoken language, on-screen text, and events over time. A training video might yield the topic, the presenter, the equipment shown, and chapter timestamps. Other outputs include participants, products, key events, and action items.
Video extraction may require frames, audio, speech, and temporal relationships. It is not the same as analyzing one still image. The exam signal is fields drawn across frames, audio, text, and time.
Define the schema before you process the content
The schema is the contract for what extraction must return.
| Field | Type |
|---|---|
| Customer name | String |
| Product | String |
| Issue category | Category |
| Damage visible | Boolean |
| Requested action | String |
A schema makes the output predictable, easier to validate, consistent across items, and ready for an application. Extraction is “return these fields,” not “tell me about this.”
Azure Content Understanding is the Foundry capability for this job
Azure Content Understanding analyzes multimodal content and produces structured results. The conceptual flow is content, then an analyzer, then a defined schema, then a structured result. It is designed for documents, forms, images, audio, video, and repeatable extraction workflows.
The current AI-901 objectives associate information-extraction solutions with Azure Content Understanding in Foundry Tools.
Content Understanding and a multimodal prompt solve different problems
Content Understanding is schema-first: “Extract customer, product, damage type, and requested action.” A general-purpose multimodal prompt is question-first: “Look at this photograph and explain what may have happened.”
| Content Understanding | General-purpose multimodal prompt |
|---|---|
| Schema-first | Question-first |
| Repeatable field extraction | Flexible conversation and reasoning |
| Consistent structured results | Usually open-ended responses |
| Many similar content items | Individual inputs to explore or interpret |
| Application and workflow integration | Interactive assistance |
Known fields at scale point to Content Understanding. An open-ended question points to a multimodal prompt.
Different media can fill one business object
One support case can gather fields from several inputs:
| Input | Extracted information |
|---|---|
| Claim form | Customer and order number |
| Damage photograph | Product and damage type |
| Recorded call | Intent and requested resolution |
| Unboxing video | Event and timestamp of damage |
The modalities differ. The business object stays the same.
An insurer that processes thousands of claim photographs, forms, and recorded statements, and must produce the same predefined fields for a downstream workflow, should use Azure Content Understanding. The solution has to process multiple modalities repeatedly and return a consistent schema.
Bring this to your team
Level Up can run a personalized Azure AI Fundamentals workshop for your team. Tell us a little about your group and we’ll reply with pricing.