Transcription
Welcome to DataFramed. This is Richie. Retrieve augmented generation, or RAG to its friends, is a popular technique for retrieving supporting information to include in your prompt. This is supposed to reduce the hallucination problem that LLMs suffer. For all its success, I quite regularly see people proclaiming its imminent death in favor of something more sophisticated. Today, I want to know whether the reports of RAG's death are greatly exaggerated or not. Our guest is Da Keela, who was part of the team at Meta that invented RAG. These days, he's CEO of Contextual AI, a platform for creating RAG agents and custom LLMs known as contextual language models. All right, let's find out about the fate of RAG.
A lot of RAG systems like two years ago were what we called like Frankenstein's RAG. All of these parts are kind of cobbled together, but they're not really designed to work well together. In our case, all of these components are designed to be state-of-the-art, and they're also designed to work well together. I think a much more useful definition of an agent is just something that actively reasons. The really exciting technology that that has enabled all of this is just test-time reasoning and the insight that shifting the compute from the training side to the test-time inference side actually has very, very nice properties.
Hi Da, welcome to the show.
Hi, thanks for having me.
Cool. So, I've been hearing a lot in the last year about people predicting the death of retrieval augmented generation. RAG, why do you think people keep thinking this is going to happen?
Yeah. And that that usually happens with with good, simple ideas where where people are trying to sort of rewrite history, maybe from a marketing perspective, a little bit. And RAG is such a simple idea that I think it's a little bit silly to declare it. Um, so, um, I I guess everybody listening in here knows what RAG stands for, right? Retrieval augmented generation. And so the G is just any Gen AI model, and then you want to make that model work on your data, which means you need to augment it with your data. And the way to do that is through some form of retrieval. So it's such a kind of generic paradigm that it doesn't really make sense to to pronounce it dead. I think. But from a marketing perspective, a lot of people keep saying, "Oh, you you know, you don't need RAG, you need fine-tuning," or "You don't need RAG, you need um long context windows." Um, and yeah, I mean, we can we can go into the specifics of each of those. Um, but uh, yeah, I think these are mostly marketing tricks.
Okay, that's good to know. So I suppose it feels a bit like linear regression is very simple and I mean it's been around for like almost a century now and people still use it because it's simple, even though there are more sophisticated models out there now.
Yeah, exactly. Yeah, we even bought a domain name, isragdeadyet.com, where we can point people to a blog post about about whether RAG is dead or not. Yeah, I mean, so these are the types of things that you cannot really declare dead, I think. Uh, but I guess I'm biased.
Okay. Uh, are there any problems though? Are there any limitations to this?
Oh, absolutely. Uh, so so I I I think if you look at agents, right? Agents in general, retrieval is just one of the tools in the toolbox of an agent. Um, and so, um, I think it's it's very true that we should not just have RAG as the only thing that these agents can do. But yeah, that feels a little obvious, right? So, um, in terms of of the current limitations still of RAG, I think part of the solution there is actually long context, right? So, if a retrieval system is imperfect, then ideally you want to cast a wider net. So, have more information that you can put into the context of the language model in the hope that there's something useful in there. Um, and then um uh that's that's where you need a longer context. That that's why I always talk about these dichotomies as sort of being false, right? It's like you need both. It's like you're not going to put the entire like internet in the context of your language model. So that's why you are going to do some search. So RAG, basically. But then you want to ideally put lots of search results in the context of the language model so that it can do its job and and answer the question correctly, right? So uh, yeah, we need all of those things.
Okay, so just to make sure I've understood this correctly, the whole point of RAG is that you're going to reduce hallucinations by just getting or just retrieving relevant information and putting that inside your prompt, and then that's going to help the large language model give you the correct answer. So if you have a longer context window, you can put more information in there that's going to assist it. And so the idea of long context windows and RAG, they sound like they're competing, but actually they're kind of complementary. Is that right?
Very, very complimentary. Yes.
Okay. Wonderful. So I know in the Contextual blog you've introduced the idea of RAG 2.0. So can you tell me what RAG 2.0 is and how is it different from the original RAG?
Yeah. So, so the way we do RAG is is uh a bit different, I guess, from what everybody else is doing. And so we started the company when we saw the world get very excited after ChatGPT and then get very frustrated because the technology wasn't quite ready. And that that was especially true for uh enterprise use cases. So um we knew that RAG would be part of the answer, obviously, given what we knew about RAG. We also knew that RAG was just the first idea there and that and that even in the RAG paper, we talk about sort of what the the vision is and what we really want to achieve, which is that these components are designed to work together, right? So so a lot of um um RAG systems like two years ago were what we called like Frankenstein's RAG, where it's all of these parts are kind of cobbled together, but they're not really designed to work well together. So uh our RAG 2.0 I know approach is about making sure that all of the components of a modern RAG pipeline. So that's not just a vector database and a language model anymore. It's much more complicated as a system. In our case, all of these components are designed to be state-of-the-art, and they're also designed to work well together. And you can do that through uh training on the same uh data distribution essentially, so that all the parts are designed to work well together. They're literally like trained to work well together. And so uh that combination of having very good components and then having a very good compound, that that makes our our system much better at RAG than than anybody else.
Okay. So in theory, that sounds like a very useful idea for the idea that the data in your database and the data in the large language model, they're kind of harmonized in some way. You got all the components working together. Um, do you have a sense of when this might be useful and what sort of benefits you might get from it?
Uh, it's always useful. I mean uh, it's it's the use cases we are focused on as a company. They they tend to be high-stakes use cases with a low tolerance for mistakes and a and sort of high uh high accuracy requirements where you want to have very accurate attributions. So very often this is in regulated industries, and then you want to um um, you know, where you're also sensitive about your data. And if you have those characteristics, then then uh we are much better than than anything else out there.
Okay. Yeah. So I can see if you got a high requirement that the answer you're giving is correct in some way, you need to put more effort into this. Uh, do you have any examples maybe from one of your customers of things that have been built using this approach?
Yeah. So, so one one use case I'm very proud of is the work we've been doing with Qualcomm. Um, so um their customer engineering department is using us where their only GenAI deployment at scale, as far as I know. Um, where um, yeah, we these are thousands of engineers who are using us on a daily basis to answer very complicated questions. So these are not the simple kind of questions where you have like your internal search and maybe it's like, you know, something like Glean, and you would ask it like, "Who is our 401k provider?" or "How many vacation days do I get?" right? Um, that that's not really where the the ROI of AI is going to come from. You want to focus on these much more uh expert uh kind of specialist knowledge worker use cases. And and so that's what we did with Qualcomm, and we can answer very complicated questions where I don't understand what the question means, and definitely don't understand what the answer means, but the system does a really good job at uh explaining the the information and giving the right answer.
Okay. Yeah. So, if you got a simple true or false or give me a number, retrieve one simple fact kind of situation, then RAG 2.0 is maybe overkill. Is that right? But if you got a more complicated question, then that's when it's going to shine.
It's not really overkill. It's more that um you'll So, the RAG 2.0, you know, uh happens during training, right? Not during inference. So during inference, it's still a normal RAG system, but just a really, really good one. Um, so it it doesn't um make things more complicated for you. Uh, if anything, it makes everything easier because we offer it in one platform where you can build these agents really in 10 seconds. Uh, I I can build a a state-of-the-art RAG agent a agent in in 10 seconds, which is I think pretty powerful.
10 seconds. Okay. Well, I'm that's pretty short. I mean, it's about as short a development time as you can get.
Yeah.
So, talk me through uh what do you do then if you're going to build something in 10 seconds.
So, uh we had these two concepts uh of agents and data stores in our platform. And so what you would do is you would create a data store. Um, and then um you would tell the agent that that is the data store that it has to work on top of, and you put some files or a database into that data store, and now you can talk to that data and do RAG on top of that data with a a state-of-the-art RAG system.
Okay, that sounds very straightforward. Just copy some files into a database and say, "Okay, go look at them. So tell me something about what's in there." Um, all right. Nice. Um, and if you're going to look at one of these very strict use cases that you mentioned, that one of these regulated examples, do you need some sort of tuning of that? Um, like some sort of optimization to make sure it all works properly.
Yeah. So, you can do that. So, our hope is that out of the box, our performance is already good enough, which turns out to be the case uh for many use cases. But um, yeah, if you if you work in finance, for example, then you really want to uh uh uh get maximum performance, and so that means that you need to really specialize for the use case, and you can do that through our platform. So we allow you to tune not just the language model or just the retriever, we allow you to tune the entire RAG pipeline for your specific problem. Uh, and so often that that leads to uh to pretty substantial uh performance improvements on top of already state-of-the-art out-of-the-box performance.
Okay. All right. So, it sounds like a lot of the secret to this is around the fine-tuning steps and the other steps around making sure the model quality is high.
Yeah, some of it is. So, it's also just about having very good components, right? So, for for uh folks listening in on the podcast, they can also use these components themselves. We've made them available independently. So if you uh want to have a grounded language model that is designed for RAG and nothing else, so it's really specialized for RAG, then you can use that. If you want to use a reranker um uh that is state-of-the-art and that can follow instructions, um you can also do that, right? And so it's the same for our retrieval pipeline and for our uh document intelligence pipeline. So, all of the components that make up our state-of-the-art uh uh RAG system, they're they're available on their own as well. Uh, because we uh just like seeing what people build with them.
Interesting. Uh, I'm curious as to how you measure what the benefits are. It seems like it's quite tricky to work out um what the quality of the responses are in general. So, how do you go about doing this?
Yeah, evaluation is such an important area and and so underappreciated. And so we we do it in a variety of different ways. I mean, obviously, we look at kind of standard benchmarks, right? And so like RAG QA Arena is a benchmark, and and we are state-of-the-art on that, and so that's what we um care about too. Uh, but that doesn't even necessarily translate to real-world performance. Uh, so we we look much more closely at our our uh actual customer datasets and um sort of the process of UAT, user acceptability testing. So do people actually consider this answer to be correct and useful? Um, and that's ultimately what you care about, right? That's where the the ROI comes from. Um, so that's one thing. I I think the other thing that's maybe worth uh mentioning here is that we have this this framework around natural language unit testing. Um, where um you're familiar with unit testing, right? It's it's the what you do with code is like just step-by-step measuring small, small things about your code, small units, and and then making sure that they are correct. And and you can use the same idea for language model responses, where you can delineate very precisely what the characteristics are of a good answer. So, so you could say, "Okay, like it needs to mention this thing first. That needs to be in this particular style. It can absolutely not talk about this thing," and you know, whatever the characteristics are of the answer, you can write specific unit tests for. And so doing that gives you much, much richer signal than just looking at what everybody else is doing now with LLM as a judge models, which is just like, "Is this generated response equivalent to my ground truth response?" And so maybe that makes sense if you're generating a few sentences, but if you're generating a long answer, then that's just not going to cut it.
Interesting. So, uh, you want to be able to ask specific questions about the response and just check, does it match all of these different criteria? And it can do pass or fail for lots of different things then.
Exactly. But that's really great too, right? Because if you think about like a regulated industry, um, the you have you will have people in like the the model risk management department of the bank, and they need to write out what a good answer looks like and what cannot be uh a part of any answer, right? So if they can just write those unit tests in natural language, and then the system can test against that for every generation, uh, that's something that that you that will make regulators much happier, basically, basically, yeah.
I certainly from my experience of creating software, when you write the test, it really helps clarify what you actually want the software to do. I'm sure it's the same situation with your large language models.
Yeah, test-driven development. Yeah, absolutely.
Cool. So I guess related to this is the idea of maintenance. So if your data is changing very rapidly, how do you go about maintaining your model just to make sure it's continually giving the right results?
Yeah. Uh, so I mean, what RAG is useful for is making sure that things will keep working on your data even as the data changes, right? That's part of part of the the appeal of RAG. Um, so um I I I think you you kind of do that automatically there. But one of the things that we're always thinking about is how can we make sure that we have the best components and continuously keep updating the best components to make sure that they're integrated into the the the overall pipeline. Um, and and so that actually takes a lot of effort, right? Make just staying at the frontier of AI, which is moving so quickly. That is is a lot of work. Um, and and so um, yeah, that's that's exciting to that we get to do uh things like that, right? As a broader kind of research community, research community.
Okay. So in theory, you shouldn't have to do too much maintenance as long as the systems in place and you test it once. So even if your data changes, it's still going to work.
Yeah. I actually I so I was at this uh I was at this this Gartner conference, and it was data and analytics, and and so everybody at the Gartner conference, and this is really not my crowd, to be honest. These are are not not like uh, you know, AI uh people. They are like real enterprise people. And it was really eye-opening for me to just be there and talk to all these amazing people and get their perspectives, which were very different from what I've been hearing everywhere. And the whole conference was just talking about uh making your data ready for AI. And and I just felt like that's a massive cop-out, right? That you should not have to make your data ready for AI. You should make your AI ready for your data. Right. And and so so that in an ideal world, and I think where we're headed, is that you don't have to do anything to your data to make it work. You just have very good AI that works on top of that data, and you and we just have to accept that data is noisy. And so that's why we have like this multi-stage retrieval pipeline. That's why we have this powerful reranker that can follow instructions. That that's exactly because you need to make sense of noisy data and filter out the things that you don't want and make sure you get the things that you do want into the language model. Um, so uh, yeah, in an ideal world, in the long run, I I can't promise it now yet, but in the long run, you want to make sure that AI just works on your data and you don't have to make your data AI-ready.
Okay, I think I just heard everyone listening in who works in data governance just gasp all at once. Uh, just just wait, don't waste your time preparing for uh this work in in the near future.
Okay. So do you have any advice for people who do work in data governance then? So if your company's making applications involves search or retrieval, what do data governance people need to do?
Be very careful about your metadata. Um, so I I think that's really one of the crucial parts of of state-of-the-art modern RAG pipelines is making sure that you have high-quality metadata or annotations for your documents and and database schemas or things like that. Um, so that that is is definitely still going to matter, having rich metadata. Um, and so yeah, the other thing is is really about I think thinking carefully through like your entitlements model, especially when you have multiple data sources that might be very disparate. So you you know, we talked to a lot of companies that have like SharePoint and Confluence and like Jira and Slack and like Google Drive and all kinds of different things, and then they want to work on top of all of those. Um, but so getting that to actually work with a with a proper sort of role-based access control entitlements setup is is not trivial at all. So that's one of the things that that we are uh are good at. But uh if you if you have a more centralized way of dealing with that from a data governance perspective, uh then you can save yourself a big headache.
Okay. Yeah. So certainly I can see how lineage is incredibly important. Do you want to go into a bit more about the entitlements? Uh, it sounds as if different people have got different permissions on data and it makes retrieval a bit complicated.
Yeah. So if you think about like real-world companies with like 100,000 employees, and not everybody has access to the same amount of data um or so, right? Some, like companies actually from a regulatory perspective, have sort of hard fencing between departments. Um, yeah, so you need to make sure that you capture that and that you don't make any mistakes. Um, and so uh, yeah, setting things up the right way for handling that at the scale of 100,000 people in a large company, that's a that's a very interesting problem.
Sure. Yeah. So in that case, I guess you want the model to give different responses depending on who is asking the question because they're going to see different data.
Yeah. So like I always enjoy saying, everything is contextual. And and so you want to get that context into the the model. And so for every user and every user interaction, even um, the the context should be different.
Okay. Nice. So I know handling permissions has been around forever in relational databases. I don't know what the status is with vector databases. So, is it easy to manage who gets access to what data and like, is it as sophisticated as with relational databases?
No, it's not it's not nearly as sophisticated. Uh, it's getting better, but it's not not as sophisticated. And even with relational databases, I I I think it's actually not that that easy. Um, so you can do it in like one one sort of single uh database, but doing it across databases um uh is is also not easy.
Okay. So is there a solution to this then?
Yeah. So the the way we do it is we we synchronize the the data with our um sort of data store concept, which we talked about, right? And then um when we do the retrieval step, we validate using an entitlements API that sort of calls all the upstream APIs to to make sure that the that you still have access. Um, and so that obviously cannot always happen in real time, depending on your latency constraints, but there's some sort of um synchronization uh step that happens in between there. And so yeah, so the model is basically like you ingest, you do the retrieval over that index, kind of a complicated index, and then um when you find the results, you validate that the results you found are actually accessible by the user using an entitlements API.
All right. So, if it sounds like one of those problems where it's going to be much better if someone else solves it for you, like it doesn't sound like the sort of fun thing you want to play around with yourself.
That's what I've been trying to tell people. Stop trying to DIY complex RAG systems. It's not worth your time. Just like you wouldn't build your own database or your own language model, like you shouldn't build your own RAG platform. You should be building RAG agents and applications on top of that platform to solve uh important business problems for your business.
Okay. Yeah. So I mean that seems incredibly important. Uh, so it's like, try and figure out where you can add the most value to solve problems to your business. Uh, can you just talk me through how do you go about thinking about this? Where should you be spending your time? Do you have any examples?
Yeah. So, so the differentiated value for a company is what companies are all about, right? Like your company wants to be better than your competition. That that's sort of the goal of every company. And and so the more you can focus on that um and thinking about, okay, how can we um uh use our data and how can we automate some of our processes or improve some of the things that we do, um the more time you can spend on that and not on like the optimal chunking strategy uh or like how to make sure that your VLM doesn't continuously go down or like basic problems that you have to deal with on the sort of red plumbing side of things. Um, the less you have to worry about the plumbing, the more you can worry about actually solving real problems that that add business value.
Yeah, that makes sense. Just really think about how can I actually make my business better and then try and get as close to just creating stuff on that level rather than worrying too much about the low-level infrastructure if possible.
Okay. So there's something you mentioned a few times, it's the idea of a reranker. It sounds quite technical, but I feel like it's one of those underappreciated components of AI systems that isn't talked about much. So, do you want to explain what is a reranker and why would I need one?
Yeah, the the reranker is super important. I think every modern RAG pipeline in production that I've seen has a reranker in it. And so, the idea is actually very simple. It's when you do retrieval um because you you need to do retrieval over large amounts of data, you cannot have a very big model doing that. So, embeddings models are are you cache the embeddings, right? It's relatively uh uh cheap compute. Maybe you do some like keyword search um and then you get a bunch of results. But now because you you had to do this at a very large scale, you will have made mistakes because you had to do it sort of quickly enough, right? So you couldn't have like a a smarter model take a look at it. So that that's what the reranker is. It's a smarter model that does a second pass at your initial retrieval results and says, "Actually, like this, I you know, I can see why it was sort of relevant, but it's really not relevant," or "You know, this one is actually super, super important for getting the right answer." Um, and so, uh, our reranker is, uh, state-of-the-art by by a large margin. So, it's much better than other rerankers out there, and it's the only one that can follow instructions, which is really important because now you can tell it what your um data hierarchy is basically. So, again, at the level of the user or even of the the individual query, you can say, "This is the priority of my retrieval results based on my preferences." So, "Most recent first," "If it is a PDF, then it's more true than if it's a Slack message," "You know, if it comes from our internal wiki or if the boss wrote it, then it's more true than if it's an intern document and it has draft in the title," right? The these types of of rules for how to prioritize data and how to break conflicts in RAG pipelines, that's what you need an instruction-following reranker for. And and so, uh, yeah, that's that's one of the that we have in our platform, and that's one of the things that makes us better.
Okay. Yeah. So I think the standard approach for retrieving information from a vector database is just a simple dot product of like 19th-century math. So I'm curious, you mentioned if you want to rerank information, you got to have a set of rules on precedence. So is that something you need to write manually, these precedence rules, or can they be automatically generated?
Yeah. So, so we usually have pretty good default rules in place uh for for the reranker. And so that would be things like "most recent first" and kind of common sense uh rules. Um, so you don't have to come up with them yourself if you don't want to. You can also ask a language model based on your data what it thinks the rules should be. Um, um, but uh, yeah, so you you you can do it yourself or you can just go with the defaults.
Okay. Nice. So again, this sounds like something where you could easily get very deep into the weeds and figure out what's the optimal strategy here, but it also feels like again this could be tricky stuff.
Yeah. So if you have a very specific rank problem and there's a very specific thing that needs to happen there, um just because of the problem you're trying to solve, then then you could try to solve it through the system prompt, but ideally you solve it by prompting your retrieval uh pipeline, right? And so you obviously can't do that in your first-stage retriever because it doesn't have the ability to follow any instructions, but you can do it in your second stage. So in your reranker.
So the tricky part here comes when like what happens if you've got information that conflicts with each other? So you're trying to, you give a problem that's trying to give a specific answer. So you pull some information from your knowledge base or whatever, and you've got two different results that give you different answers. Talk me through how do you go about resolving this?
So I mean, it depends on the type of conflict. U but you can tell it through instructions how how to deal with that conflict or how to break the tie, basically. So you could say "most recent first." So if you find two documents and one is more recent, then you uh rank that one much higher, right? Or you could say "different data sources." It's like, "I prefer this data source over that one." Um, so you can you can just put that in the instructions too. And then there are, that's why again why it's so important that you can prompt these components, is because there are different uh strategies for dealing with conflicts. So one is you go for most recent. The other is um you report both results, but you say that one is more recent, right? So uh it really depends, and and that's that's why it's so important that you get control over those aspects of a RAG system.
Okay, so it's good that there's some sort of way of resolving these conflicts. I'm curious, is there any way to feedback from when you have these conflicts uh to then put that into your data governance strategy?
Yeah. So, so we have uh some customers where um when the system, so one of our our uh very special capabilities is our ability to say "I don't know." Which is an underrated feature. It's much better to say "I don't know" than to make up a wrong answer, right? So um um that's a really, really good thing. But then when you have an "I don't know" answer, ideally you want to have the ability to annotate that answer so that the next time you get the question, you do have relevant information in your your documentation for for resolving that question, right? Uh, so that's one feedback loop. And then the other is, yeah, we collect feedback uh through our our UI or through our APIs, and then you can export that feedback and actually train on that feedback. Um, and so that's how you can uh specialize it for the use case over time. Just keep making it.
This sounds pretty useful for creating the feedback that you get. Start off saying, "Okay, we've got some wrong answers," and then you give those wrong answers back to the people who actually curate the data or maintain it, and that's going to feed back into a better system for later on.
Yeah. So that's over time, at least, how you can capture most of the distribution. Obviously, the tail of the distribution, you're never going to fully be able to capture, uh, but you can fine-tune the system maybe to generally be better also at the tail.
Now we talked a little bit before about the idea of a RAG agent. So I am curious as to whether RAG agents have a different architecture compared to other RAG applications.
Uh, yes and no. I guess we should talk a bit about what an agent is, right? Because everybody's very confused.
Absolutely. Uh, so I think an agent is a very general concept. I mean, so it it comes from uh reinforcement learning, uh, or even it's maybe even older than reinforcement learning itself, where it's really just about like a policy that takes actions in an environment, and and that policy has some sort of state, and so it can manipulate the environment um, uh, but it doesn't necessarily have to manipulate the environment in order to be considered an agent. Uh, so another way to put that's maybe maybe a bit closer to home for uh folks listening to this podcast is if you think about an agent that can do SQL queries. Um, some people will say, "Oh, but it's only an agent if it changes the environment," which would mean that it is only an agent if it generates insert queries or update queries. And if it just does select queries, then suddenly it's no longer an agent. So I mean, when you explain it like that, it sounds a bit silly, right? As like, "Obviously that's still an agent." Like Deep Research doesn't do any insert queries or update queries, but it does do a lot of useful stuff, right? So um, I I think a much more useful definition of an agent is just something that actively reasons. Um, so something that thinks about what it's doing, formulates a plan, executes on the plan, and then can revise that plan based on the information that came in. So that's active reasoning. And and So the really exciting uh technology that that has enabled all of this is just test-time reasoning, and the insight that um that uh shifting the compute from the training side to the test-time inference side um actually has very, very nice uh properties.
Okay. Yeah, I like that. So I think uh with your explanation there, it was just introducing all the important bits of jargon from reinforcement learning, just the idea of policies and environments and all those other things. If you're interested in reinforcement learning, please just rewind and re-listen to that last minute or so. Uh, okay uh, I do like the idea uh that you mentioned uh with test-time reasoning. So you're doing reasoning at the time the user asks for it. Uh, and then it can go and check things on its own and uh work through things step by step.
Yeah. No, so I I think like the the simple chatbot is a subset of the overall concept of an agent. Uh, so you can just have a you can have a chatbot agent, and it probably for most chatbot use cases, it doesn't have to do a lot of thinking. Um, so it the the same agent can also power the chatbot use case. It is just more powerful. It depends on how how much um you want to budget for test-time compute. And so if you want to minimize that, then uh you probably um uh, yeah, just have something like a standard chatbot. But but the the boundary is is is very blurry, right? Because when I have a RAG application, which maybe is not really an agent, but as a part of my RAG step, I do query decomposition and I formulate sort of a plan and then I do some filtering on top. It's sort of an agent, right? Like that that is what agents would do, but the agent sort of maybe determines that more dynamically on the fly, sort of what it would want to do. Um, but yeah, the the boundary is not as well defined, I think, as a lot of people like to pretend. Um, but so overall, I think so this really is the year of agents. There's a lot of excitement about them. They don't really work quite yet. I think a lot of people um are are sort of blown away by the potential, but like in practice, I haven't seen any real agent deployments that have like material impact on a company's business uh yet. I mean, I'm sure it's going to come, but it's going to take some time. Um, but these agents obviously, they need to work on your data, too, right? Just like with what we were doing before with NAI. It's like, yeah, we need agents, but the agents need to work on your data. So, they need to be RAG agents, because RAG is the way that you get things to work on top of your data. Um, so um, yeah, RAG is one of the tools that these agents need to rely on for them to be useful, especially in enterprise settings.
Ah, this is interesting. So I think the big difference is that with this sort of standard RAG approach, you've got some software saying, uh, "Okay, let's shove all the bits of information from the vector database into the prompt," and those kind of being pushed to the LLM. Whereas if you're doing things with um inference-time reasoning, then the LLM has to ask for the information, just kind of pulling it in. So there's a a swap, is that about right?
Yeah, that's right. So, but again, right, it's not really like one or the other. It's really a spectrum. Like I think most modern RAG systems, they probably have some kind of classifier that says, "Should I retrieve or not?" right? So, and then based on that, you say, "Okay, so like if I say hello, then then you don't have to retrieve in your RAG chatbot, you just say hello back, right?" So, first thing is you you need to. So that's already kind of active retrieval, where in the old RAG setup, where it's really passive retrieval, is you get a query, you always search for that query in your vector database. You always give the results to the language model, like that's that's prehistoric at this point, right? So it's much more complicated. There's there's like active decision-making involved in these RAG pipelines. There's a lot of uh like filtering. There's the reranker uh that has a huge impact. There's this sort of active retrieval component. There's query decomposition, which is almost like formulating a a retrieval plan. Like that that's all very agentic, but now when you have an agent, you can do that much more dynamically, and that's that's why that's so exciting.
Well, okay. Yeah. So you got more flexibility about what the behavior is going to be, rather than having a more rigid pipeline.
Yeah. Exactly. And it can also like recover from its mistakes, which is very important. Right. So if it retrieves something and it thinks, "Oh, actually, this is not what I wanted. Let me try a different query," and then it gets the result, that's very powerful, and that's something you can do with with uh agentic abilities. Um, and then uh the other thing is is more around sort of multi-hop questions or or like multi-step reasoning, is like like first I need to know this, and then I need to know that, and then I need to compare those things, and then maybe based on the result, do something else, right? That that type of multi-hop uh problem, that that's I think very interesting uh uh in terms of business value as well.
Okay. So you mentioned that there haven't really been any deployments of agents so far that have wowed you. So what do you think would wow you?
I mean, they wowed me in terms of like seeing the potential, but they haven't wowed me in like, "Oh, this company has saved like $10 million this year because they had an agent doing something." Like a like a proper agent using test-time reasoning, and not something that was branded an agent, but that was something that we were already uh capable of doing before kind of uh reasoning models.
So, what do you think the sticking points are? Where are we falling short?
I think generalization, sort of um um so so actually having it work in in real-world settings where it's not in the the toy domain, right? So getting things to work in a toy domain or in a nice demo, that that that used to be my sort of story around RAG, is and that's still very true. People think that RAG is easy because you can build a a nice RAG demo on a single document very quickly now, and it will be pretty nice. But getting this to actually work at scale on real-world data, uh, where you have enterprise constraints, it's a very different problem. And so it's the same with agents, where it's like, "Oh, I can make something like do this one particular thing when I prompt it," and basically just make everything look good for exactly the one thing I wanted to do, but then when you actually have to make this work in in a real-world setting, then then everything just breaks down very quickly still. So that's going to get better over time, obviously. Um, and and so I think the hype is justified. Um, but yeah, it's going to take some time for for these systems to be enterprise-grade enough for anybody to really deploy this uh in a in a critical setting.
So since a lot of companies are just thinking, "We got to get on the AI game, we got to build something." What can you build that is likely to work and to add value?
Yeah. So, um, what you can build um, so you can build basic u solutions for the relatively uh boring problems. So, um, one thing you can do is go for like the basic problems where you ask the basic questions, like internal search, right? But that doesn't really get you value. It's much more like trying to find um workflows that exist in your company that are a little bit boring, but that where it's important that you get it right, and where it requires some expertise. If you can solve those problems, then uh you can you can make your your uh company much better, right? And and so these could be very simple things from like, you know, checking for uh compliance against your set of policies, um, or doing basic research. We have a very nice demo where we fill out Excel spreadsheets on unstructured data on the fly, so that you don't have to manually go and copy and paste. You just directly in your Excel, kind of call a macro and then fill out the spreadsheet with unstructured data from different data sources. Um, doing things like customer support, um, there there's a lot of um low-hanging fruit um in kind of the the codegen side of things. So there a lot of it is happening kind of across the board, right? It's just um um, yeah, doing this this the right way takes time. So there's there's a big gap between sort of where the hype cycle is and where like reality is in enterprises, but I mean, it it's coming. Um, uh, it's it's just u yeah, it takes time.
Okay. So it sounds like the best approach then is to go for maybe slightly more narrow use cases where there's less flexibility needed, less generalization needed. I guess maybe all most build disposable agents. Does if you can build something quickly um that just solves your problems and then be done, that works.
I like I like that idea. You can build disposable agents on our platform, and maybe I should rebrand like that, the disposable agent platform.
Nice. Uh, so just to wrap up, what are you most excited about in the world of AI?
Yeah, so I'm obviously very excited about all the GenAI things. Um, I I think for for me personally, where I I see a lot of very interesting problems is at the intersection of structured and unstructured data. Um, so you have a bunch of documents, but you also have your traditional uh kind of structured relational databases, your Snowflake or your BigQuery or whatever you use, and now you want to kind of cross-sect that information using a GenAI RAG. And so if you can do that, which you you can now uh start doing because of these agentic abilities, that unlocks so much interesting um potential. Um, so I think that's really exciting. The other thing is multimodality is obviously still um very underexplored. I think uh um every time there's like an image generation feature uh uh getting shipped that kind of goes viral. But um I think image understanding is actually much more valuable um from a from a kind of enterprise perspective. Um, so so I think that that's also a really uh key unlock that is coming soon. So chart understanding, for example, and and things like that, understanding um, you know, a McKinsey slide deck that has lots of different diagrams and charts and things in there, like right now, that's not really within the capabilities of these systems. But it it's coming very quickly.
Okay, I got to follow up on that. Uh, so talk me through it. You basically want the ability to understand a presentation, then just throw it a PowerPoint and explain what the output is.
Yeah. So that's the simple case. So the hard case is I have uh a 100 million PowerPoints that my company has made in the
history of my company with a 100,000 people in my company. And now I want to answer questions based on all of that information. Uh so it's not just like one single PowerPoint because that you can kind of start to do even though it's not very accurate but you need to do it at scale right? So over over lots of u uh presentations. So if you can do that then you can do very interesting kind of synthesis on top of it right. So how did our perspective on a particular type of thing change over the years? uh and then you can just find the relevant presentation decks that cover this particular thing. Look at the charts, reason about the trends in the charts and then combine that into a new insight. Um that that's all kind of starting to become possible now.
Okay, that would be very interesting. Although I have to say I've definitely had a few colleagues in the past where even with them talking over the PowerPoint, I am not sure what they've been on about. It's similar to what everybody says, garbage in, garbage out. You cannot be much better than your data. You can try to reason about it intelligently but there are limits there. Absolutely. Absolutely. Wonderful.
So all right. Uh since you've actually been involved in rag since the beginning I mean you were there when rag was created as part of the uh team at meta. Uh so do you think rag has lived up to your expectations? Has it panned out as you expected? That's a nice question. Um I mean I I think that the the original vision was always that we would have um kind of a decoupling between the knowledge and the reasoning where the reasoning is really just like taking whatever the relevant knowledge is and then uh giving the right answer on top of it without um um having any of of the knowledge in it itself. So that didn't really pan out and and so um that's part of the reason why these systems hallucinate and and that that's um I mean it's a longer story but but so for for rag I think when the paper came out uh it was very focused on open domain question answering which is the the domain that you evaluate these systems on um and so it was wellreceived but at the time Gen AI like vector databases basically didn't really exist right that became a thing after uh the paper uh language models didn't really exist. We had like Bart and T5, but there there wasn't really a concept of like an auto regressive uh generative model. So I think like the reason rag became such a popular like paradigm and concept um and why it's called rag is because because of the G. So it's really just because genai became a thing that rag became a thing. Um and there are lots of papers from around that same time. And there's this amazing paper from folks at Google called Realm where that didn't become the name of the paradigm because it didn't have a G in it. It was a a mass language model, right? So um yeah, hindsight is sort of 2020. It is amazing how small changes to the name have a big difference on your success or not. I mean it was it's not just the name, right? is is that we were interested in trying to see if you could generate the answer without uh sort of so the alternative is just predicting the answer which is uh what was much more normal to do at the time. So I guess we were ahead of our time in the right way there.
Wonderful. Yeah, certainly I mean uh it's taken over in so many different ways. Uh it's ubiquitous now. Uh so obviously very successful and uh just finally I want ideas for people to follow. So whose work are you most interested in at the moment? Whose work am I most interested in? Um I think there there is a um a lot of interesting work happening in this new testime compute paradigm. So I I mean I still have my kind of part-time Stanford adjunct professor uh gig which is great for me to kind of uh stay stay at least a little bit up to date on the latest latest research trends. Um, and I I think there's just a lot of interesting research happening around this testime reasoning and and what you can do there. I think we've only just scratched the surface. Um, and and so uh what happened with Deep Seek and things like that has been very encouraging, I think, from that perspective where it's actually not that hard for um for like nonfrontier le um folks to to do interesting things in this space and and have impact. Um, so um yeah and I I follow a lot of uh just smart academics. Uh yeah smart academics always worth following I think. Uh very good genre of people to uh to watch out for. Uh so all right uh thank you so much for your time to thanks for having me. [Music]