AI Multimodal Input & Parsing

Many office scenarios involve screenshots, scans, PDFs, and images — not just plain text. Multimodal models can read images, tables, and diagrams directly, saving you from manual transcription.

1) Common Input Methods

• Screenshots / whiteboard photos: Feed directly to GPT-4o / Claude 3 / Gemini. Ask for text output + structured fields.
• PDFs / long documents: Use "file upload + location referencing" mode. Have AI tag page numbers and paragraphs for verification.
• Table images: Have AI extract to CSV/JSON while listing possible OCR errors.

2) Structured Output Prompt

This is a screenshot / image / PDF. The content is [brief description of the scenario].
Extract into JSON:
{
  "title": "",
  "date": "",
  "participants": [],
  "key_points": ["", ""],
  "action_items": [{"item": "", "owner": "", "deadline": ""}]
}
If a field is missing, use null — don't fabricate. Flag anything you're uncertain about.

3) Visual Understanding Use Cases

• Chart interpretation: Have AI identify chart type, key trends, anomalies, then give "1-sentence conclusion + 3 action items."
• Table/screenshot -> document: Have AI generate meeting notes, requirement lists, procurement checklists.
• Image -> text: Poster/marketing screenshot -> AI outputs "copy text + design elements" for rewriting.

4) Risk Controls

• When OCR confidence is low, have AI flag "low-confidence fields." Amounts and dates must be verified by humans.
• For sensitive documents, use enterprise models or private deployments — don't upload to public endpoints.
• Before sharing externally, have AI scan for "privacy info / watermarks / confidential markings" and warn if redaction is needed.

5) Tool Tips

• Desktop screenshot + quick upload: Raycast / ShareX / screenshot tools that upload directly to AI conversations.
• PDF chunking: Split long PDFs by page or chapter before uploading. Summarize segments separately, then merge — reduces missed content.
• If the model supports "citations," require output to include reference links / page numbers for easy navigation.

6) Practice

Take a phone photo of a whiteboard or handwritten meeting notes. Have AI output "summary + action items + items needing human confirmation," then have it generate a sync email for external stakeholders.

---

7) Worked Example: Quote Screenshots to a Procurement Comparison

Suppose three suppliers send quotes as screenshots. The first goal is not “pick the cheapest.” Build a traceable data layer first.

Step 1: Extract One Document at a Time

Process only this quote image.
For each field, return original value / normalized value / page or region / confidence note:
- item
- quantity and unit
- unit price
- tax
- total
- validity date
Use [?] for unreadable characters. Do not guess.

Step 2: Human-Check Critical Fields

Step 3: Compare Only Confirmed Data

The following fields have been human-verified.
Return:
1. directly comparable items
2. items that differ in scope
3. questions for each supplier
4. price, scope, risk, and information gaps — no recommendation yet

8) Multimodal Input Preparation Checklist

• Crop browser tabs, notifications, customer names, and unrelated context
• Confirm the file is allowed to enter the selected AI service
• Preserve page numbers and require page or region references
• Human-check amounts, dates, identities, signatures, and contract terms
• Keep the source file with the structured result

9) Failure Modes and Fixes

10) Chapter Deliverable

Create a traceable multimodal extraction pack: source file, field table, uncertain items, human-check record, and final conclusion. For sensitive material, continue to AI Security and Ethics.