Multimodal

A text-only model reads and writes text. A multimodal model handles several kinds of data in a shared representation, so it can look at a photograph and describe it, read a chart and answer questions about it, watch a video, or listen to speech and respond in speech.

The technical trick is representing different media in a common space, so an image and the sentence describing it land in comparable territory. That shared representation is what allows genuine cross-modal reasoning rather than a pipeline of separate systems handing off to each other.

Most current frontier models are multimodal to some degree, and the frontier is moving toward natively multimodal training rather than bolting vision onto a text model after the fact.

Multimodality removes the transcription step that used to gate a huge class of real work — screenshots, scanned documents, whiteboard photos, diagrams, recorded meetings. A great deal of business information isn't text, and until recently that meant it was out of reach.

Check which modalities a model supports for input versus output — they're often asymmetric, and a model that can read images frequently cannot generate them. Also check whether images count toward your context and cost, because they do, often substantially.