Oct 14, 2025
8 min read
Designing for AI (9/12)
Multi-Modal AI Experiences: Designing Beyond Text
Francois Brill
Founding Designer
Humans don't communicate in just one way. We talk, gesture, show images, point at things, and combine all of these naturally. But most AI experiences are still trapped in single-modal interactions: text-only chat interfaces, voice-only assistants, or image-only analysis tools.
The future of AI is multi-modal. It's AI that understands when you say "make this image brighter," processes the visual context, and adjusts the photo. It's AI that lets you start a conversation by voice in your car, then seamlessly continue it in text when you get to your desk. It's AI that reads your gestures, understands your screen, and responds in the modality that makes the most sense.
The challenge isn't building AI that can handle multiple input types. Modern models like GPT-4V, Gemini, and Claude can process text, images, and even video. The challenge is designing experiences that work seamlessly across modalities while maintaining consistency, preserving context, and giving users control.
The Multi-Modal Design Stack
Effective multi-modal AI design rests on three foundational principles.
Modal Consistency
Your AI's personality, capabilities, and limitations should remain consistent regardless of how users interact with it. A voice assistant should have the same "knowledge" as the text interface. The AI shouldn't suddenly become more capable (or less helpful) just because someone switched from typing to talking.
This consistency extends beyond functionality to tone and behavior. If your AI is conversational and helpful in text, it should be conversational and helpful by voice. If it asks clarifying questions in one modality, it should do the same in others.
Example in action
A design AI that helps with color selection provides the same quality suggestions whether you upload an image, describe it in text, or point your camera at a physical object. The recommendations might be delivered differently (spoken vs. displayed), but the underlying intelligence remains identical.
Modal Affordances
Each modality has unique strengths and limitations. Voice is fast and hands-free but lacks visual feedback. Images provide rich context but require processing time. Text offers precision and editability but is slower to input.
Design for what each modality does best. Don't force users to type long commands when voice would be faster, or expect them to describe complex visual concepts in text when they could just show you a picture.
Example in action
A recipe AI uses voice for quick ingredient substitutions while you're cooking ("Can I use honey instead of sugar?"), but switches to visual display for step-by-step instructions with photos. Each modality serves the task it handles best.
Context Bridging
The most powerful multi-modal experiences maintain context when users switch between modalities. The AI should remember what you showed it in an image when you ask a follow-up question by voice. It should understand references like "that one" or "the blue version" based on previous visual context.
This requires more than just logging interactions. It requires building a shared understanding that persists across different input and output methods.
Example in action
A user shows an AI a photo of their living room, discusses paint colors by voice, then switches to text to ask detailed questions about finish types. The AI maintains the full visual and conversational context throughout, understanding that "the wall behind the couch" refers to the specific wall in the original image.
Voice AI Design
Voice interfaces are uniquely powerful. They're fast, natural, and work hands-free. But they also present unique design challenges.
The Visibility Problem
Voice interfaces lack persistent visual feedback. Users can't scan previous responses or scroll back through conversation history. This creates challenges for complex information and error recovery.
Design pattern: Conversation Markers and Confirmation
Always confirm what the AI heard before taking action, especially for consequential tasks. Layer information, starting with brief summaries and offering details on request.
User: "Find Italian restaurants nearby."
AI: "I found 8 Italian restaurants within 2 miles.
Would you like me to read the top 3,
or should I send the full list to your phone?"
This approach gives users control over information depth and provides a natural escape hatch to visual modalities when voice becomes limiting.
The Ambient Context Challenge
Voice interfaces operate in noisy, unpredictable environments. Background sounds, interruptions, and privacy concerns affect how people use voice AI.
Design pattern: Smart Context Awareness
Design for interruption and resumption. Let users pause and resume conversations naturally. Provide visual fallbacks for sensitive information (like passwords or personal details) that users might not want to speak aloud.
Example in action
A banking AI processes general queries by voice but automatically switches to visual display for account numbers and balances, understanding that users may not want to speak these aloud in public spaces.
Visual AI (Computer Vision)
Visual AI unlocks rich contextual understanding. Users can show rather than describe, which is often faster and more accurate. But visual processing introduces its own design challenges.
Progressive Understanding
Computer vision isn't instant. Image analysis takes time, and results often come with varying levels of confidence. Design for this progressive revelation of understanding.
Design pattern: Layered Visual Processing
Show users what the AI sees as it processes. Start with quick, lower-confidence insights and refine them over time. Use visual annotations to communicate what the AI is analyzing.
Example in action
A plant identification app first quickly detects "This is a plant" (instant), then narrows to "Likely a succulent" (2 seconds), then provides specific species identification with confidence levels (5 seconds). Each layer builds on the previous one, giving users immediate feedback while deeper analysis continues.
Confidence and Uncertainty Visualization
Visual AI is probabilistic. The same image might yield different interpretations depending on lighting, angle, or quality. Make this uncertainty visible.
Design pattern: Confidence Regions
When AI identifies objects or features in images, show confidence levels visually. Use highlighting, color coding, or transparency to communicate certainty.
Example in action
A medical imaging AI highlights potential areas of concern with color intensity based on confidence: bright red for high-confidence findings, lighter orange for uncertain areas. This gives clinicians visual cues about where to focus their attention and which findings to investigate further.
Gesture and Touch AI
Gesture interfaces feel magical when they work. They're intuitive, spatial, and accessible. But they require careful design to avoid fatigue and confusion.
Gesture Vocabulary Discovery
Unlike text or voice, gestures aren't standardized. Users need to discover what gestures your AI understands, and these gestures need to feel natural.
Design pattern: Progressive Gesture Teaching
Introduce gestures gradually, starting with the most universal (tap, swipe, pinch). Show gesture hints contextually, when users are likely to need them. Let users customize gestures for personal workflows.
Example in action
A drawing app starts with basic pinch-to-zoom and two-finger-rotate, then introduces more advanced gestures (three-finger-swipe to undo) after users demonstrate comfort with the basics. The AI learns individual gesture preferences over time.
Haptic Feedback for AI Responses
Touch interfaces can provide tactile confirmation of AI understanding through haptic feedback. This creates a satisfying feedback loop that builds user confidence.
Example in action
An AI photo editor provides different vibration patterns for different actions: a gentle pulse when it detects an object to select, a stronger "bump" when it completes an edit, a double-tap vibration when it needs clarification. These haptic cues create a richer feedback channel beyond visual confirmation.
Advanced Multi-Modal Patterns
The real power of multi-modal AI emerges when you combine modalities intelligently.
The Modal Relay
Users start conversations in one modality and seamlessly continue in another, with full context preservation.
Example in action
A user says "Show me red dresses" (voice) → product grid appears (visual) → user taps a dress (touch) → asks "Do you have this in blue?" (voice) → AI shows blue variants (visual), maintaining understanding that "this" refers to the touched item.
This requires maintaining a shared context object that tracks references across modalities and time.
The Parallel Processing Pattern
AI processes multiple modalities simultaneously for richer understanding.
Example in action
A video analysis AI processes both visual content (what's happening on screen) and audio (what's being said) to generate comprehensive summaries. It can answer questions like "What was that building they showed while talking about architecture?" by correlating visual and audio timelines.
The Smart Handoff
AI determines the best modality for responses based on context, content complexity, and user environment.
Design pattern: Context-Aware Modality Selection
When users ask questions, the AI chooses the response modality based on:
- Content complexity (complex = visual, simple = voice)
- User context (driving = voice only, office = visual preferred)
- Content type (numbers/charts = visual, quick facts = voice)
Example in action
A user asks "What's my schedule today?" by voice. If there's one meeting, the AI responds by voice: "You have one meeting at 2pm with the design team." If there are six meetings with complex details, the AI says "You have 6 meetings today. I've sent the full schedule to your screen" and displays a visual calendar.
Users can always override these choices ("read them to me" or "show me that"), but smart defaults reduce friction.
Design Challenges Across Modalities
Multi-modal AI introduces unique challenges that don't exist in single-modal experiences.
Context Preservation Across Modes
The biggest challenge: maintaining conversation thread and context when users switch between modalities.
Solution: Shared Context Layer with Visual History
Build a persistent context object that tracks:
- All user inputs across modalities (with timestamps)
- AI responses and their modalities
- Referenced objects (images, documents, screen regions)
- Inferred user intent and preferences
Make this history visible when users switch modes. If someone has a voice conversation and switches to text, show a summary of the voice discussion at the top of the text interface.
Privacy and Security Per Modality
Different modalities have different privacy implications. Voice conversations might be overheard. Screen sharing reveals more than intended. Camera access feels invasive.
Solution: Modal-Specific Privacy Controls
Give users granular control over each modality:
- Voice: temporary voice-only modes that don't log transcripts
- Visual: blur sensitive screen regions, selective image sharing
- Text: exportable conversation logs with redaction options
Make data handling transparent per modality. Users should understand that voice might be processed differently than text, with clear explanations of retention policies.
Accessibility Across Modalities
Users may not have access to all modalities. Vision-impaired users can't rely on visual output. Motor-impaired users may struggle with gesture interfaces. Design for these realities.
Solution: Modal Alternatives and Assistive Integration
Every critical capability should work in at least two modalities. Visual information should have voice descriptions. Voice-only interactions should have text alternatives. Integrate with assistive technologies like screen readers and switch controls.
Example in action
An AI design tool provides voice descriptions of visual AI outputs ("The AI identified three dominant colors: deep blue, warm coral, and cream white. The composition is balanced with strong vertical elements on the left third"). This makes visual AI accessible to blind users while also helping sighted users understand the AI's reasoning.
Implementation Strategies
Building multi-modal AI doesn't mean building everything at once.
Progressive Modal Enhancement
Start with your core modality (usually text), then add others based on user demand and technical feasibility.
Recommended progression:
- Foundation: Text-based AI with strong core capabilities
- Enhancement 1: Add visual input (image upload, screen sharing)
- Enhancement 2: Add voice input for quick queries
- Enhancement 3: Add gesture/touch for spatial interactions
- Advanced: Add voice output, multi-modal combinations
Each stage validates user demand before adding complexity.
Modal Feature Parity
Ensure core AI capabilities work across all modalities, but add modal-specific enhancements on top.
Example in action
A code review AI provides the same quality feedback whether you paste code as text, upload a file, or share your screen. But the screen-sharing version adds real-time annotations and cursor following, enhancements that only make sense in that modality.
Cross-Modal Testing
Test AI consistency and context preservation across different input combinations:
- Voice → text handoff accuracy
- Image reference maintenance in voice conversations
- Gesture + voice combined interactions
- Modal preference learning over time
Use multi-session testing where users switch devices and modalities to validate context preservation across sessions.
Questions for Product Teams
Before building multi-modal AI, consider:
Which modalities best serve your users' needs? Don't add modalities just because you can. Each one introduces complexity. Start with the modes that solve real user problems.
How do you maintain AI consistency across modalities? Users expect the same AI "intelligence" regardless of input method. What's your strategy for ensuring consistent capabilities, personality, and limitations?
What's your approach to context preservation? How do you track and maintain context when users switch between voice, text, and visual inputs? How long does context persist?
How do you handle privacy across different input types? Voice, images, and screen sharing have different privacy implications. What controls do users have per modality?
Which modal combinations provide the most value? Some modality combinations are more powerful than others. Identify high-value combinations (like voice + visual) and optimize for those first.
Multi-modal AI isn't about adding every possible input method. It's about designing experiences that adapt to how users naturally communicate, making AI feel less like a constrained interface and more like a flexible, understanding partner.
Start with one modality done well. Add others deliberately. Maintain consistency and context throughout. Give users control over how they interact. The result is AI that feels seamless, natural, and genuinely helpful.
Earlier in this series: Testing & Iterating AI Features: Beyond Traditional UX Methods