Transcription
The term multimodal AI has been around for years, but how AI models use it has changed quite a bit. So before I get into that, what is modality, and why are there multiples of them?
Well, when we say modal, we are talking about data. So we're talking about a data modality. So if we think of a single modality large language model, for example, we've got an LLM here. A single modality large language model is probably one that tokenizes text strings. So what goes into this thing is text, and then what comes out of the model is also text.
But let's say we want the model to work with another modality. Let's say, we want it to work with images as well. So maybe I want to create a prompt with a little bit of text and then a screenshot of a problem I'm having on my phone. Well, to process that, we need a multimodal AI model, meaning a model that can ingest and or generate multiple data modalities, in this case, text and images. But we could throw in audio, LIDAR, thermal imaging, all sorts of modalities could go here.
So how does a multimodal AI model work with these different modalities? Well, one way to do this is with a modular feature-level fusion approach. This is how a lot of the early systems worked. So in addition to a text-based large language model, so we've got our LLM here that works with text, there's a second model, and that second model is a vision encoder, maybe something clip-based, and that vision encoder extracts features from whatever images we provide. So if I give this an image here, I can pass that in to my vision encoder, and then that passes those features to the large language model. And those features are numerical representations of that image in an array. And it's projected into a form that the LLM can process alongside the text. And of course, the potential problem here is that information might get lost during the transfer between these two models. So the LLM is essentially only seeing a summarized description of the data, instead of the raw signal. It doesn't get to see the actual image; it just sees the numbers. But these models are still used for specialized enterprise tasks because they're cheaper, and they're easier to swap parts. But it's not really the gold standard of multimodal AI today.
But multimodal AI that uses feature-level fusion is where you extract a feature vector, that's this numerical representation from a vision encoder and inject it into the middle of the LLM's processing stream.
So what's the alternative? It is native multimodality. So instead of bolting separate models together, a natively multimodal AI processes different data types or different modalities through something that is called a shared vector space, meaning it is shared between all the modalities. So if we think about how that looks, here's our shared vector space. We've got some text in that shared vector space. We've got images in that shared vector space. We've maybe audio and some other modalities as well. They all get tokenized and embedded into the same high-dimensional space where the model can reason about them together.
Now with text, we already tokenize words and word fragments into numerical vectors. So for example, the word "cat" just becomes a point in this massive vector space. With native multimodality, we can do similar things with images. We chop the image into small patches, and each patch gets its own embedding, its own point in that same vector space. And it's the same with audio and other modalities as well. Break them into chunks and embed them. And the key to a shared vector space is the shared part. All these different modalities live in the same space, so the model doesn't have to translate between different systems. So if we had a picture of a cat in this shared vector space as well—and trust me, that's a cat—well, it would kind of end up near the word "cat" in the same space because they mean similar things.
So this can really beat out the vision encoder approach. With feature-level fusion, the vision encoder processes your image before it knows what question you're asking. It might compress away the exact detail you need, but with a shared vector space, the model attends to text and images simultaneously, so it knows where to look. So if I ask about a tiny icon in the corner of the screenshot I took to address my phone problem, the model can focus attention there because it's reasoning about my question and the image at the same time.
All right, two other things that are useful to know about multimodal AI before we're done. And first, let's talk about video. Now in video, things happen in sequence. There's a bit of a time dimension here. And this is where temporal reasoning comes in. Early multimodal systems, they handled video by sampling a few frames and then running those frames through the vision encoder. But you could lose a lot that way. So show me a single frame of somebody holding a water bottle, and I can tell you that there is a person and a water bottle, but I can't tell you if they're putting it down or if they're picking it up. Yeah, that information lives in the sequence. Thanks, JD.
So newer natively multimodal models embed video with the temporal dimension intact, whereas an image might be processed in these pixel patches that I mentioned before. So perhaps we've got a 16 by 16 set of pixel patches. Native video models, they process data in spatial temporal patches, which are actually little 3D cubes of information rather than just two dimensional. So instead of capturing a flat square of pixels, each token captures an area across a short window of time, like maybe this represents 8 video frames all here, and this means that the motion in the video isn't something the model has to guess by comparing two different images; the motion's actually baked into the token itself.
And finally, I should say that I've mainly covered ingesting multiple modalities here. So we've talked about well, maybe we've got text coming in. Maybe we've got images coming in. Maybe we have got some video coming in. And they're all being received by this multimodal model, but multimodal models can generate across modalities as well. Because all these modalities live in the same shared vector space, we get what's called any-to-any generation. The model can take in any combination of modalities here, and it can output any combination as well. So I could ask the model to explain how to tie a tie. It can generate a few sequences of text telling me what to do. And then, it could generate a short video clip as well showing the process, a natively generated spatiotemporal video chunk of tying a tie. And it's all coherent because it's happening in that same shared vector space.
So that's multimodal AI today, a model that can see and read and hear and respond in kind.