Multimodal AI systems in 2026: overview
Multimodal AI systems process or generate information across data types such as text, images, audio and video. Their value lies in linking perception with language and, increasingly, decisions or actions. This can support tasks that rely on evidence in different formats, from interpreting medical images with patient text to guiding robots through visual scenes. Yet combining modalities does not guarantee sound reasoning: models can miss spatial relationships, misalign visual and textual representations, or produce answers that are not grounded in the input. Reliable grounding and evaluation remain central technical challenges.
Recent work is pushing beyond image-and-text alignment toward unified models, spatial reasoning and systems that connect perception to action. In October 2026, Canva launched Brush, a vision-language model for editing designs with users; an October 2026 paper, EventVL, explores semantic understanding of event streams. Other current research targets hallucination-resistant clinical reporting, cross-layer vision-language fusion, and model merging to transfer reasoning capabilities. These directions point to a shift from broad visual question answering toward more grounded, task-specific systems that can handle complex inputs and operate in practical settings.
- 1Unified models advance: Unified multimodal models grew 47.4% in the latest comparison, reflecting a move toward shared systems rather than separate models for each input type.
- 2Grounding is the bottleneck: Work on clinical reporting, industrial defect detection and spatial reasoning highlights the need to connect model explanations to specific evidence in images and scenes.
- 3Deployment moves outward: Research and product activity in robotics, industrial inspection and mobile inference is pushing multimodal AI beyond cloud-based image and text analysis.
Top organizations working on multimodal AI systems
Ranked by Linknovate activity score · last 24 monthsThe activity score adds up the relevance of every patent, publication, grant and news item linking an organization to multimodal AI systems. Showing 7 of 3,000 organizations.
| # | Organization | Approach | Signals (24 mo) | Momentum | Why it matters |
|---|---|---|---|---|---|
| 1 | 
Nvidia
Corporate · United States |
Vision-language models | PAT 385 · PUB 73 GRANTS 1 · NEWS 849 |
-40% | NVIDIA has patented training methods for multimodal language models that combine multiple vision encoders, and its software ecosystem supports deploying vision-language models on edge devices. |
| 2 | 
Tsinghua University
Research · China |
Unified multimodal models | PAT 4 · PUB 502 NEWS 50 |
-6% | Recently, affiliated researchers evaluated multimodal models’ multi-step spatial reasoning; other work develops monolithic models that integrate visual encoding and language decoding. |
| 3 | 
Zhejiang University
Research · China |
Vision-language models | PUB 452 · NEWS 8 | +60% | Recently, affiliated researchers introduced DefectCoT to evaluate interpretable reasoning by multimodal models for power-equipment defect detection. Their work also explores VLMs for remote sensing and radio-signal recognition. |
| 4 | 
Google
Corporate · United States |
Unified multimodal models | PAT 48 · PUB 69 NEWS 1,189 |
-47% | Google develops multimodal AI, including models that work across text, images, video, and audio. Its patent activity also covers knowledge-augmented adapters for vision-language networks. |
| 5 | 
University of Chinese Academy of Sciences
Research · China |
Vision-language models | PUB 430 · NEWS 4 | +12% | Researchers affiliated with the university develop vision-language models for Earth-observation imagery, including systems for image captioning and visual question answering, and survey multimodal approaches in the field. |
| 6 | 
Shanghai JiaoTong University
Research · China |
Unified multimodal models | PUB 404 · NEWS 4 | +42% | Recently, affiliated researchers presented work on multimodal models for risk-grounded reporting in heart-valve procedures. Other research develops integrated vision-language models and benchmarks for biomedical image understanding. |
| 7 | 
Peking University
Research · China |
Vision-language models | PUB 360 · NEWS 11 | +29% | Recently, affiliated researchers studied how VLMs reveal spatial and qualitative disparities in urban renewal. The university’s work also includes remote-sensing VLMs and methods for synthesizing image-caption training data. |
| 8 | ··Hidden organizationCompany | Approach | PAT 00 · PUB 00 GRANTS 0 · NEWS 00 | +00% | Available to registered users. |
| 9 | ··Hidden organizationCompany | Approach | PAT 00 · PUB 00 GRANTS 0 · NEWS 00 | +00% | Available to registered users. |
| 10 | ··Hidden organizationCompany | Approach | PAT 00 · PUB 00 GRANTS 0 · NEWS 00 | +00% | Available to registered users. |
Startups to watch and leading research centers
Startups to watch 43 total
AnthropicUnited StatesAnthropic develops Claude models that accept image inputs alongside text, making visual question answering and image analysis part of its multimodal AI work. Its research has also examined image-based attacks on VLMs.
AILabAILab’s research includes mobile vision-language models for understanding app interfaces and multimodal models designed to run on mobile devices. Recently, affiliated researchers introduced a spatially grounded VLM training approach for radiology.
Blackbird.AIUnited StatesRecently, affiliated researchers applied reinforcement learning with chain-of-thought supervision to explainable detection of hateful and propagandistic memes. Blackbird.AI also develops multimodal methods for extracting narratives across platforms.- ··Hidden startupCountryAvailable to registered users, with funding and founding year.
- ··Hidden startupCountryAvailable to registered users, with funding and founding year.
Leading research institutions 2,583 total
- Tsinghua UniversityChinaRecently, affiliated researchers evaluated multimodal models’ multi-step spatial reasoning; other work develops monolithic models that integrate visual encoding and language decoding.
- Zhejiang UniversityChinaRecently, affiliated researchers introduced DefectCoT to evaluate interpretable reasoning by multimodal models for power-equipment defect detection. Their work also explores VLMs for remote sensing and radio-signal recognition.
- University of Chinese Academy of SciencesChinaResearchers affiliated with the university develop vision-language models for Earth-observation imagery, including systems for image captioning and visual question answering, and survey multimodal approaches in the field.
- ··Hidden research institutionCountryAvailable to registered users, with their key researchers.
- ··Hidden research institutionCountryAvailable to registered users, with their key researchers.
Where multimodal AI systems innovation happens
Share of documents with at least one organization from each country.
Open the interactive map with all 3,000 organizationsMain technical approaches
Last 24 months| Approach | What it is | Documents | Leading organizations |
|---|---|---|---|
| Vision-language models | Connect images and text for tasks such as visual question answering; performance can be limited by weak grounding and gaps between visual and language representations. | 2,125 |
Google, Vlm, Zhejiang University |
| Audio-language models | Link speech or other audio with language for understanding and generation; they must handle timing and acoustic variation as well as linguistic meaning. | 613 |
Google, Nvidia, Meta Inc |
| Video-language models | Combine video with language to interpret events over time; processing long sequences and preserving temporal context can be costly. | 787 |
Tsinghua University, Nanjing University, Vlm |
| Unified multimodal models | Use a shared model across multiple input and output types; broad capability can come at the cost of harder training, evaluation and control. | 3,454 |
University of Chinese Academy of Sciences, Tsinghua University, Chinese Academy of Sciences CAS |
| Vision-language-action models | Translate visual observations and language instructions into actions; they enable interaction but raise the stakes of errors in perception and planning. | 695 |
Nvidia, Tsinghua University, Vlm |
Emerging trends
Last 12 months vs previous 36 monthsTerms whose frequency in the last 12 months grew the most compared with the previous 36 months.
Funding and investment
Recent funding rounds
Funding records list 116 grants totaling $170,018,418 since 2020. They also report 55 funding rounds totaling $905,300,499,191, but the listed rounds include infrastructure and debt financing, so that aggregate is not a clean measure of investment in multimodal AI companies.
Patent landscape
Top patent holders · last 24 monthsRecent developments
Latest news, duplicates removed- Oct 2026The in-house model reads layouts, layers and styles to edit designs alongside users, and it debuts for Canva Pro subscribers Canva has a new collaborator, and it doesn’t need a coffee break. The design platform launched…Crypto Briefing
- Oct 2026Where this starts: teaching machines to see Computer vision is one of the foundational subfields of AI. The goal is simple to state and hard to achieve: give a machine the ability to look at the physical world, an image…Hacker Noon
- Oct 2026TeleOCR is an open-source, approximately 1.2-billion-parameter vision-language model for parsing both born-digital documents and camera-captured pages. It is maintained by XingChen-AGI, uses the Transformers library…Hacker Noon
- Sep 2026Dublin, Sept. 25, 2026 (GLOBE NEWSWIRE) -- "Vision-Language Models Market by Deployment Mode; Industry Vertical; Model Type; Region-Market Size, Industry Dynamics, Opportunity Analysis and Forecast for 2026-2035" has…GlobeNewswire: Research Analysis And Reports
- Month 20XXHidden developmentAvailable to registered users with alerts enabled.
Frequently asked questions about multimodal AI systems
What companies are working on multimodal AI systems?
Nvidia, Google and Microsoft are among the corporate organizations active in multimodal AI. Nvidia develops multimodal training methods and software for deploying vision-language models on edge devices; Google works across text, images, video and audio; Microsoft is also active in the field. The company landscape also includes startups such as Anthropic, whose Claude models accept image inputs alongside text.
Which multimodal AI systems startups are promising in 2026?
Anthropic, AILab and Blackbird.AI are examples of startups working on multimodal AI. Anthropic develops models that accept images alongside text, while AILab works on mobile vision-language systems and Blackbird.AI applies multimodal methods to narratives and harmful content. These examples show different product directions; the available information does not establish a comparative ranking of startup prospects.
Which universities and research centers lead multimodal AI research?
Tsinghua University, Zhejiang University and the University of Chinese Academy of Sciences are among the leading research organizations in the available activity data, with 558, 460 and 434 documents respectively. Their work spans unified models, interpretable defect detection and vision-language systems for Earth-observation imagery. Other active institutions include Shanghai JiaoTong University and Peking University.
How much funding has multimodal AI systems received?
Funding records list 116 grants totaling $170,018,418 since 2020, alongside 55 funding rounds totaling $905,300,499,191. The round figure should be treated cautiously: listed entries include infrastructure and debt financing, not only direct investment in multimodal AI companies. The grant total is a clearer measure of recorded grant funding, though it is not a complete measure of all research support.
Who holds the most multimodal AI systems patents?
Nvidia is the largest listed patent holder, with 385 patents. Google holds 48 and Microsoft holds 26, followed by Robert Bosch GmbH with 21. The leading holders therefore include both technology companies and organizations working in industrial applications. These figures describe the patent-holder data for this topic, not the full patent portfolios of those organizations.
Is multimodal AI systems patent activity growing?
Patent activity is down 48.3% in the latest comparison, while publication growth is reported at 173% for the last year. The dataset lists 1,179 patents in its 24-month window. This contrast suggests research output is expanding faster than patent activity in the current data, but it does not by itself show whether individual technologies are becoming more or less commercially mature.
What are the latest developments in multimodal AI systems?
Recent developments include Canva's Brush launch in October 2026 and the EventVL publication on understanding event streams in October 2026. Research is also addressing clinical hallucinations, vision-language fusion, model merging and spatial reasoning. These efforts reflect a shift from matching images with text toward systems that must reason over context, provide grounded outputs and support specific tasks.
What are the main approaches in multimodal AI systems?
The main approaches are vision-language, audio-language, video-language, unified multimodal and vision-language-action models. Vision-language systems connect images and text; audio- and video-language models add sound or temporal visual information. Unified models share capabilities across modalities, while vision-language-action systems connect perception and instructions to control. Their trade-offs range from sequence-processing demands to the greater consequences of errors in action.
Which countries lead multimodal AI systems?
The United States leads with 12,875 documents, followed by China with 10,616. The United Kingdom ranks next with 2,878, while Germany and India each have close to 2,000. These figures show strong activity in both the United States and China, alongside research and development across Europe and Asia.
What are the applications of multimodal AI systems?
Recorded applications include healthcare and medical imaging, creative media and design, robotics and autonomous systems, manufacturing and industrial inspection, and document intelligence. Healthcare and medical imaging has 1,694 documents, while robotics has 1,281. Recent work also applies models to ultrasound assessment, power-equipment defect detection, structured document parsing and robot navigation.