Transcription
Were there any specific mindset shifts that led you to become, you know, a leader in AI from just a developer? I think the big one is understanding that I have to. Jason is one of the top AI consultants in the world. He worked with major companies like Zapier and Hopspot on their AI strategies, and his library instructor is now used by some of the largest organizations in the world. It's downloaded over 2.66 million times per month, and it completely changed how we think about AI agents today. I personally use it to build my own framework, and I'm sure many people did as well.
So first, Jason, what are we hoping someone will get out of this episode if they watch until the end?
Yeah, generally it's the case that these AI agents that we think about, we should think about them in terms of a portfolio of tools, and all the work that we need to do and all the kinds of investments we need is about building out this portfolio. Can I do data analysis to add new tools? How do I verify the tools are doing what they can and cannot do? And if we combine them together, then we can get very valuable and economically valuable AI agents.
How do you define AI agents and what makes them different from traditional LLM-based applications?
Yeah, I think now for the most part I just see AI agents as an LLM with a portfolio of tools. And not only is that going to be the case, but you can often times add or remove tools or add different context to make the AI sort of behave the right way. But I mostly think of it as just a tool caller in a for loop. I think honestly the for loop might not be necessary in the future. Right now the for loop does make sense because the agents just can't execute extended tasks, but as we can see on this research—we'll probably show it on the screen—the length of tasks that the AI can execute autonomously is growing exponentially. So now, you know, like 03 is already at like 1.7 hours of autonomous just task execution, which means that we might not actually need the for loop when, you know, using 03 for AI agents. What do you think about that?
Yeah, I guess in my mind, like just the calling a tool, getting the results like that is the for loop. So I think we're actually totally in agreement in terms of like what this looks like.
So can you share any specific practices that you've discovered for making agents reliable in production?
Yeah, I think it would go back down to this like topic analysis, right? This this clustering technique, cuz really what's going to happen is two things. One, you're going to come up with some kinds of eval—like maybe this is good, maybe this is bad. I don't really know what 67 means. But if you can cut up your conversations or cut up your agents in these different clusters—like questions about finances or questions about scheduling or answers that require a lot of tables—now you have all these little different groups, and you can compute the same score for each group. And now you'll see, oh, this one is really, really low. This one is really, really high. Clearly, I should look at whatever is low and fix that problem. And let me look at the conversations. Let's look at, you know, the tools it uses and ask myself, is this eval low because it's really impossible? Is it just because I'm missing some piece of context or some some tool that I know how to build? And you just plop it back in. You see those numbers go up, and then you just sort of do that forever. So basically just setting up the evals from the start and just tracking every metric that you can track. Yeah. But most importantly, grouping things together so you actually know how to cut them up, right? If someone just tells you your marketing is bad, it's really hard to figure out what to do. But if someone said, "Hey, your marketing does really well for people under 25 and really bad for people over 25," now you can make a decision. Do I want to double down on my younger audience or do I need to change my content for an older audience? Right. Now that leads to an action.
Interesting. So just basically like adding more dimensions exactly to the metrics that you check.
And what are some of the most common pitfalls that developers or businesses face when it comes to deploying agents in production?
I think a lot of it is just this belief that the agent can do anything and the agent should be able to do everything. The joke that I always tell is the more I think a company thinks that they need a smarter AI, the dumber the team is, right? Because what it's saying is, oh, I don't want to do any thinking about what my customer wants. I don't want to do any thinking about how my team should operate. I want the AI to do it. Whereas a really great founder usually says, I've talked to a lot of customers, and this task is really important. I watched experts do this task. It looks like they do it this way. Can my team model out this pipeline? The answer is usually yes. And they do quite a good job.
Is there any specific like tech stack that you're using to deploy agents in production? Is there any frameworks that you can recommend?
Yeah, not on my end. Mostly because I let the company sort of pick the stack, and usually the companies that reach out to me already use some kind of technology, whether it's like a, you know, Langchain or like Pyantic, for example. But for the most part, the most important thing is to build something quickly, start tracking that data, and then doing that data analysis. Right? If it takes you six months to build the app, it just means that it takes you six months to make a better decision using that data. I would rather someone make an app as quickly as possible and then spend two or three months looking at that data. So just getting the MVP out as quickly as possible and then collecting some real feedback and improving better based on that feedback, right? Yeah. Like the biggest improvements will always be these like weird things that happen. Like we launched a company once, we launched an agent, all of a sudden it stopped doing well, and we looked at why it's like, oh, all the people using it are from like Turkey and China. We have a multilingual problem, right? And then we spent like a month basically figuring out how to like build different UIs and like showing like Chinese and Turkish on the website. Like those have become the real problems, not the agent.
So it seems like you work with a ton of early startups in AI that are trying to distribute AI agents across the world. So how do you actually productize an agent? How do you make it usable by anyone around the world?
I mean, I think a lot of it really is just the fact that if you try to build something for everyone, it's still very hard. And usually when I see companies do that, it tends to stay away. It's often the case that, you know, when you work with these vertically integrated companies, you know what the useful work is. You have experts that you can talk to and interview. You can hire people who do these things themselves and work on replacing the difficult parts of their job. That's usually when things are really successful. I think it's much harder to like build a good chatbot and not be Google or OpenAI. Yeah, I would probably not want to compete with OpenAI.
Another use case that's currently becoming more and more popular is software development. Like OpenAI says that, you know, right now they have a top 50 programmer. I think GPT is top 50, and by the end of this year they're going to have the number one programmer in the world. All of the tech companies are saying that, you know, AI is already responsible for writing 20 to 30% of their code. So what do you think um about the role of a developer in the future? How do you think it's going to change, and how can we as developers adapt to these changes?
Yeah, I mean the biggest one I think is just the fact that distribution matters more than ever. There's only two problems, you know, distribution and product. If product is made much easier with AI, then I think a lot of the time that you save might be worth going to doing distribution and learning to market yourself. And at least if you're thinking about doing a business, and then when it comes to actually being a developer, a lot of it is now about what is the right organization of code? How do I support like high-quality documentation, and how do I make sure those are in sync? I basically just had Cursor and Cloud Code write I think like 6,000 lines of docs for this library that I've been working on. How do I write code in a way that makes it easy for an agent to work with? You know, is that multiple files? Is that single files? I've changed the way I write documents where documents now always mention the code file so they know where to look.
Oh, cool. So we will definitely get into RAG in a bit. I think there are a lot of things you can share on RAG and on how to improve RAG systems. But first of all, I wanted to talk about a video that I just recently posted on this exact topic. And I said that, you know, essentially developers will have to adapt by the end of this year. And I've gotten a lot of hate for this video, to say the least.
Yeah. Yeah. Makes sense. Makes sense because people are saying essentially that, you know, AI can't work on large code bases. Like it sure can build you a Tetris game, but it it doesn't know how to work with large code bases. And I'm arguing that the reason it doesn't know how to do that is because our code bases are not made for AI. It's not because of AI. It's because of our code bases. So the practice that you've just said is actually extremely interesting to me. Like you add special like links inside the files so they can like navigate between different documentation pages. Is there anything else that you can share for developers and development teams on how they can adapt to AI?
I now have a lot of cloud files like scattered around my repository that explain things like how I do testing. I have a lot of Cursor rules—like I have a lot of Cursor rules on how I do code review, right? It's like I make Cloud Code up its PRs. So I say like, okay, make this big change. Don't make one PR, make three PRs, each one merging to the other one. And now what I do is I'll run the job. I can go get my coffee. And as I'm in line for coffee, I'll get a message that's like, oh, PR1 has been created. It has only 60 lines of code. And then what? PR2 has been created. It's also 60 lines of code. And another 60 lines of code. And each pull request now is really easy to review, right? Because we all hate this like thousand-line, you know, mega pull request that some engineer makes. And now Claude can sort of organize that for me and write much more detailed descriptions. That has been one thing that to me has been really valuable, especially because Cloud Code can also pull the PR comments from GitHub back into the context. So I will be on my phone leaving comments in the code on GitHub, and I'll come back to my desk and I just say /pr comment, pull everything back, fix everything I said, and I go to my next coffee in the coffee shop. That's actually one of the things that excites me about AI agents so much is that once you've built an AI agent, it's basically infinitely scalable. Like, you know, you can just tell one developer to create two PRs to the same codebase at the same time, right? But it's not the case with AI agents. You build an AI agent once, and then it can create a thousand PRs for you, right? At the same time, it can run in parallel as much as you want.
So when AI brings such tremendous scalability to organizations, how do you actually price for a solution like this? Cuz you can't even price it as like an employee anymore because it's like a thousand employees, right?
Yeah. But again, I think this is why we are going to move to a world where most things are going to be on value-based pricing. Like would I pay $2,000 for a marketing AI? Would I pay a 10% commission to a marketing AI? And these are the questions I think that are going to be really important in the future. You know, like if I have an AI book meetings for me, should I pay when the meeting is booked or should I pay when a sale is made? Right? There are people who get paid on value-based pricing. And I think for the most part, people who are being paid value-based, their customers are happier cuz everything is alignment. And I don't even think it's a favor that you—some people charge others hourly. I don't think it's a good business model to begin with. And I'm just really excited to see more and more AI systems go into a world where you might be priced on success rather than priced on usage.
So you also have a course called Systematically Improving RAG Applications, which from what I've heard so far is excellent. We're actually considering purchasing it for our team as well. Can you briefly explain what does it mean to systematically improve RAG applications?
Yeah. So what has happened in the last year was a lot of my consulting was it's not working, it doesn't feel good, like what should we do? And after like six or seven engagements, I kind of came up with a playbook of just the steps you have to follow. And now it's kind of like the playbook I use when these consulting engagements sort of involve improving search systems. Step number one, you create precision and recall evaluations for your search system. Step number two, once you have enough evals, you can think about fine-tuning a model. But you can only fine-tune a model if you have feedback data. So let's go now evaluate and audit the systems you have in place to collect user feedback—thumbs up, thumbs down, customer compliance, etc. And by doing that, you can improve your embedding models and improve your search systems. And then the second half of the course is around tool discovery, right? What tools does my agent need? How do I figure out if a tool is missing? You have to do this topic modeling. Once you find these topics, you can solve each individual topic with a couple of tools. And then it's about how do you make sure that it's using the tools correctly, right? Like, for example, do you give your AI tools for GitHub or do you just say, well, it has access to the command line. Shouldn't it be able to do everything? But it turns out if you give it a GitHub tool, it will make more pull requests. It will create more issues. But if you give it a command-line tool, it will use other things to get the job done. And so it's about breaking up the problem, solving each individual problem separately, and then figure out how to combine them. And those are the two systems I implement when I do my consulting work.
Wow, that actually sounds really exciting to me. So you actually deploy agents without like all of the sets of tools that they might need first. Is that what you're proposing—is like you deploy an agent, you test it, and then you evaluate and then you add more tools accordingly? Is that how you do it?
Exactly. Exactly. Cuz you know, you want something out in the world, and there's just things you can't do, and that's usually fine. But if you discover that, you know, 10% or 30% of your customers are asking for something, shouldn't you discover that using the data that you have, right? Obviously, you want to talk to people, but if you don't have that luxury, looking at the data is the second best thing. I think because it's so different from how we approach traditional workflows because, you know, if when building workflows you just define all of the steps in advance, you know, like all of the actions that you can take, but here it seems like you're taking a lot more dynamic approach.
Yeah, because think about like how we start a business, right? We first start a business, and we go, okay, huh, what do I do next? Okay, I have to go record videos and edit stuff. Okay, well then what? And then you do that and then what? And at some point when you scale, you say, you know what, I've realized that every time I make a video, I should come up with a blog post, and I should do this and I do that. But it's because you've done it so many times that you've discovered the subprocess, the standard operating procedure that you need to implement in your business. And sometimes it's the CEO's job to realize that this thing is happening. Sometimes it's the employees' job to go, "Oh, I should automate this in some way." And doing it with AI should feel the same way.
And regarding the first part of the course, you also mentioned fine-tuning. Do you think fine-tuning is necessary or do you start without fine-tuning?
I think fine-tuning a language model is going to be very hard and generally not going to be worth it, mostly because owning inference is going to be very difficult. Fine-tuning rerankers, fine-tuning embedding models for search are very worth it. They're going to cost you three or four dollars to train a model, and it's probably going to give you 10–15% better performance.
I see. Cuz I thought you meant traditional fine-tuning, and what I find—it's actually—it's nowhere near as useful as fine-tuning your RAG application. I also see that the dev tools for AI agent developers are just becoming so good that most of us don't even like need to think about this anymore—at least for small businesses where, you know, like, for example, OpenAI already has the file search out of the box, and now like Google is also entering the space with like their own ADK and also their own even agent marketplace.
Yeah. So do you still think that like in the future as this space becomes more and more productized and more and more companies release those amazing dev tools that make agents so much more accessible, will there still be space for independent developers like us or is big tech just going to dominate agent building completely?
I think again it's going to be the case that for these large companies they're going to have to build fairly horizontal tools, and so it is just very possible to find like useful niches, you know? Like is it going to be easy for a developer to make a $10 million company? Maybe it's going to be hard, but I can't imagine why you couldn't build a company that's like one or two people making like one or $2 million per person. I think it's actually very possible now, especially with the leverage that you have and also because I would only say maybe 30–40% of running a business is the code, right? Like you have to become more entrepreneurial and more like individual when it comes to these kinds of things. So just taking initiative and focusing on a specific vertical that those large companies can't possibly cover by themselves. We're not talking like, you know, software engineering—that's kind of obvious. We're talking like some niche down solution that requires like some form of a specific expertise and like actually, you know, making it happen—like taking initiative and going for it, right? Yeah, exactly. So you're making over six figures per month now doing consulting in AI, right? Yeah, yeah. So for anyone just getting started in this space, how do you recommend they find their first client?
I mean, I think a lot of it is just figure out what you're interested in and understand what kind of problems and pains you run into. Right. Once you have that in mind, you can just build your own tools. You can just write about them. For example, and free content generally is the easiest way to get started. I mean, I think the easiest way really is getting referrals. Like, for example, I've actually been in machine learning for about 10 years. And so when AI popped up in this past couple of years, I was already pretty well established in the community. But most of my work now comes from my writing, right? I am writing about problems that other big companies are facing today.
Were there any specific mindset shifts that led you to become, you know, a leader in AI from just a developer? I think the big one is understanding that I have to write about things from the perspective of the pain of the customer. If I want to get customers, I can't write about, you know, oh, this weekend I learned like this thing or that thing. It's how you can improve.
Your business, right? It is. It is the mistakes you need to avoid versus just, oh, I learned this this week, or I learned this that week. You know, those things really nobody cares about, right? In the sense that unless, like, your followers could care, but maybe not your customers. And so, really understanding the pain and the benefit of what your solution is, rather than, oh, how valuable am I? How smart am I? Those things really put you in a position where you, I think, for the most part, limit your earning potential, right? If I feel like I am a senior engineer and I deserve $250 an hour, I know exactly how much I want to make in the year. But if I have two customers and one of them has a team of seven people, or one of them has no team, the team of seven people probably has a bigger problem. They might not know how to spend their time. And if you waste the time of four engineers, that's already a million dollars a year that you're wasting. So what's 2% of that, right?
I see. So how exactly do you price your services?
Yeah. So, let me take a step back, which is the first thing I try to do. I try to have a minimum level of engagement, right? It's a number I have on my website. And the goal isn't to communicate, oh, I'm very expensive, like you can't afford me. But neither is it to say, you can afford me, cuz I'm very cheap. But I say typically customers tend to spend about $60,000 to $80,000 over a period of two or three months. And you know, that number on the website sort of helps me filter out like what kind of customers I was looking for. If Walmart goes on my website and they say $60,000, like, oh, what could I possibly get for that little money, right? I am investing, you know, $200 million in AI; McKinsey will charge me $34 million, but I'm probably going to get a lot more. But a serious, you know, seed company that just raised a million, they might go, "Oh, I don't actually know what I want, so I don't know if this is a good bet." So that helps you filter out the right customers. And then once you do that, a lot of it is a conversation on how big is the problem that you're working on and how accountable you are for that solution.
Okay. So once you've filtered all of those initial customers and once you scheduled a call with someone who is okay with this initial price, how do you actually justify it to them?
So a lot of it is understanding what the real pain is, right? So, for example, they might come to me and say, "Jason, we need AI to be more reliable." And it's like, why now? You know, what is at stake in your business that you need this AI to be more reliable? A bad customer might say, "I don't know. I've been trying it out. It doesn't seem like it's doing what I want it to do." Other customers might say, "Well, we launched our product. We went viral. We hit a million ARR, and the churn is through the roof. We're losing 20% of our customers every month." And then if I ask the question, oh, how much does that cost to you? Now it says, "Oh, actually I'm losing $200,000 of ARR every month, and I don't know if I lose this customer if they're going to come back in three months. How much is that worth to you?" Right? That sounds like a problem. That's $100,000 every month. If I ask for $30,000 a month to stop the bleeding, maybe that feels like the right kind of deal. And maybe I have some guarantees on, like, if we don't hit these KPIs, you don't pay me the full price.
I see. So a lot of people when they are starting out, they're kind of thinking about how to price their service. So they're thinking, you know, about the price in terms of the service that they can provide. Yeah. While what you're saying, I think, is a lot closer to some form of like a value-based pricing, where exactly for the value that you provide. Yeah. And obviously it's not always going to be the case that I can just say, well, I want 20% or 30% of the value. But I think what this unlocks in terms of a mindset is you'll realize how often you're not overpricing yourself, right? Like here's a story. This company paid me $700 an hour to interview their senior AI team and to help them hire the head of AI. So at the end of this, I made $20,000. It's like, oh my god, I got paid $20,000 to just read people's resumes and ask them like what the hardest problem they're working on. Like wow, lucky me. At the end of that, we hired four engineers. Each one was being paid $200,000 a year. And we hired a head of AI who had recognized me on a podcast, right? And then I find out the recruiter made $200,000 because the recruiter makes 20% of the payroll of each person. Okay. So, the recruiter made 10 times more than I did, right?
Wow.
Then with the head of AI joining this company, six months later, I see the news. They raise a Series B, and the head of AI is like joining this company, and they raised a Series B to like make more investments in AI, and that was a $20 million like Series B. And so the head of AI's equity probably went up quite a bit. You know, the CEO's equity went up probably like a couple million dollars. And I'm sitting there like six months later going like, "Oh, I I thought I was like winning because I was making $700 an hour, right?"
Right?
But you realize in that situation, I was still the most exploited person in that engagement, right? The employees made more money. The employees got jobs. The head of AI joined the company. They were able to raise the money. The recruiter made like $200,000. The the founders made like a million dollars. I say, "Okay, I need to really think about how I change pricing and what they're getting by hiring working with me, not how much I cost."
Okay, that sounds really exciting. So what's on the road map for Instructor, and what are you personally planning to focus on in the future?
For the road map of Instructor, there's not much to be done right now. I think most of it is actually going to be around like what are the new like LLM tools I plan on investing in. And I think what we talked about earlier is going to be really important, right? We want to invest in some more tools to do this data analysis. So we came out with a library called Cura, K-U-R-A, and that does this topic analysis. And then our goal is to also come out with something that is going to be sort of a light version of Ragus to create an eval library. And then what we'll be able to do is say, okay, given these conversations, I have metrics, and I have ways of grouping the data. So let me group the data, find the average metrics, and just now allow me to just explore more and more what is going on in these conversations with AI and what can be done to make them better, right? And I just think that the conversations we have with AI now is probably the most interesting data set I can imagine having access to.
Definitely. I mean, Chad already knows more about me than my parents.
Exactly. Exactly. Exactly.
So this is basically like a library for exactly what you just described. It's like clustering data, extracting insights, and seeing how you can further improve your AI agents based on that. Yeah. Is that correct?
Exactly. Is it already out, or is it just coming up?
Uh, Cura is out. We're mostly working on the documentation now. Um, I can like send you a link afterwards. We're still uh doing some tests, but we'll have a presentation on uh AI and Engineering Summit.
Okay, awesome. I'd be happy to cover it on one of my videos on this channel later.
Yeah, I would love to have Ivan come on and share what he's working on.
Yeah, that would be great because honestly, EVAs has always been such a challenge for us. Like OpenAI has their own solution, but it's like a bit challenging to set up. Like there's no out-of-the-box solution like you described.
Exactly. Exactly. So, I'm pretty excited for these things to exist, and again, it's going to be just using those tools sort of drive consulting revenue, and then the idea is that these tools can like just stay free and stay lightweight.
All right. So, where can we find out more about your work and learn from you?
You can just find me on Twitter at jxnlco. But generally, if you want to learn more about Rag, we have a newsletter called improving.com. And if you want to learn about consulting and you know the work that we do, check out learnnindconsulting.com. Two little websites that could help you out.
Awesome. Thanks, Jason. We'll leave all the links down below and looking forward to covering that library that you mentioned.
Yeah. Awesome. We'll send an intro afterwards. Thank their.