Skip to content

Latest commit

 

History

History
52 lines (37 loc) · 3.44 KB

File metadata and controls

52 lines (37 loc) · 3.44 KB

Problem Statement: Cross-Modal Document Reasoning

Microsoft 365 Copilot already helps users search, summarize, and draft against document content. The problem Lumina targets starts one level deeper: the user asks a question whose answer lives in the relationship between narrative text and visual evidence inside the same document.

Examples:

  • "Does the revenue chart support the CFO's growth narrative?"
  • "Which region misses target in the table, and how is that described in the text?"
  • "The paragraph cites Figure 4. What does Figure 4 actually show?"

When Copilot answers those questions without correctly connecting the text to the visual source, the failure is not cosmetic. It creates false confidence. Users either re-run the query with manually rewritten chart content, or they stop trusting the answer and verify it by hand.

The Problem Statement

Document-heavy knowledge workers need Microsoft 365 Copilot to answer questions that span text and visual evidence inside the same file. Today, that workflow is fragile because good document assistance is not the same thing as reliable cross-modal reasoning. Without stronger structure mapping, source grounding, and uncertainty handling, Copilot risks producing fluent answers that still require manual verification.

Where The Experience Breaks

Failure pattern Why it matters
The answer summarizes nearby text but does not use the chart or table that should change the conclusion Users get a plausible answer that is still wrong for the real job
The answer references visual content without showing which figure or page informed it Verification stays slow and trust stays low
The model reads the figure but loses the narrative context around it The user still has to reconcile the evidence manually
Performance varies materially by file format or document layout Users cannot predict when the workflow is safe to rely on

Users At Risk

User group Why the failure is expensive
Financial analysts and finance leaders Narrative and chart misalignment can change the conclusion presented upward
Compliance, legal, and audit workflows Missing a table or heat map signal turns AI assistance into liability
Research, strategy, and operations teams The cost shows up as re-query loops and slower synthesis across dense reports and decks

Constraints

Constraint Product consequence
Answers must remain fast enough for live analysis workflows Extra structure work only matters if latency stays controlled
Citations need to be correct, not merely present Fake grounding damages trust more than no grounding
Some visuals are ambiguous, low-resolution, or poorly labeled Abstention and uncertainty handling are required
PDF, PowerPoint, and Word behave differently Format normalization is a product requirement, not background cleanup

Evidence Basis

Signal type What it indicates
Public Microsoft product behavior Copilot is already expected to reason over business documents, not just produce generic summaries
Public competitor positioning Document understanding remains a visible point of comparison in AI productivity tools
Public multimodal benchmark results Chart and document reasoning remain meaningfully harder than text-only language tasks
Repo workflow analysis Verification-heavy document tasks reward traceability more than verbosity

Next: Jobs To Be Done ->