Multimodal models
One model for text, image and sound: how different inputs are brought into the same space and where accuracy suffers.
The idea
A language model understands tokens. Translate an image into tokens as well and the same model can process both at once. That is what multimodal models do: an image encoder produces vectors, a small projection maps them into the space of language tokens, and from then on everything is one sequence.
What it is good for
- Understanding a form without structuring it first.
- Having a chart explained.
- Deriving instructions from a photo of an error message.
- Judging image and text together, for instance a product photo against its description.
The structure
- 01
Image encoder
A pre-trained image encoder produces feature vectors, usually from tiles of the image.
- 02
Projection
A small trained layer maps those vectors into the language model's dimension.
- 03
Insertion
The resulting image tokens are placed in the sequence where the image sits.
- 04
Joint processing
From here there is no difference between image and text tokens.
The cost calculation
| Input | Approximate tokens |
|---|---|
| One page of text | 500 to 800 |
| A low-resolution image | 250 to 500 |
| An image at 1024 × 1024 | 1,000 to 1,600 |
| A high-resolution document scan | 2,000 to 4,000 |
A stack of ten scanned pages can therefore occupy 40,000 tokens and cost more than their text. If you only need the text, OCR followed by text processing is substantially cheaper.
The projection
What is remarkable is how little that takes: in many architectures W_p is a
single linear layer or a two-layer network of a few million parameters. The
overwhelming share of capability sits in the two pre-trained parts, not in their
connection.
High resolution
Since cost grows quadratically with side length, models work with tiling: the image is split into overlapping crops, each encoded separately, plus a downscaled overview.
With six tiles of 576 tokens each plus 576 for the overview, that is 4,032 tokens for a single image.
Which route when
| Task | Recommendation |
|---|---|
| Read amounts and identifiers exactly | OCR with checksums |
| Understand layout, assign fields | Multimodal model |
| Read free text in a photo | OCR, then a language model |
| Interpret a chart | Multimodal model |
| Check an image against a description | Joint image-text embedding |
| Process very many pages | OCR, on cost grounds |
The audit trail
A multimodal model returns no coordinates. Where it must be traceable which part of a document produced a statement, that is a knock-out criterion. The usual way out is a combination: OCR supplies text with coordinates, the model works on that text, and every extracted value can be traced back to a location in the image. See OCR.
Related courses and sources
Learning Transferable Visual Models
Images and text in one shared space. The basis of searching images by description and of image generation.
For understanding image search by description and image generation.