Transcription
One of the most common concerns I get from people is that they do not know how to optimize their prompts and they want to, but they don't feel they have the expertise. I've written a lot about how to develop that expertise, but I also recognize it's not for everyone.
This method that I'm about to show you is actually a way to make AI optimize your prompts for you. And it's based on a very, very famous Python computer language framework that engineers are currently using for production prompting. And so if you've ever wondered how do people get their prompts to look so nice, well, this is part of how.
What I'm going to do is I'm going to walk through and explain the concepts in this video. And then I'm going to have a whole post that lays out how you actually get started with specific prompts with examples. And I'm going to divide that post into three parts.
Part one is for beginners. This is nobody's ever done this. You should be able to apply these lessons as someone who doesn't want to touch Python code, doesn't want to touch the terminal, doesn't want to see code at all, and you should still be able to get benefits. And that is not something that people have done. People generally say, "If you want to optimize your code like this, well, best of luck to you, right? Like off you go and use the terminal." I don't think that's acceptable.
Instead, I want to give you a 5-minute quick start that lets you take the same principles that engineers are using for production code and apply them yourself in the chat so that you can get some of those benefits, too.
But we're not done yet because if you're an engineer or a builder, if you're not scared of the terminal, I want to give you a reasonably technical explanation of how DSPI works, the principles behind it, and then also in the article, the get started handbook, so you can get there.
And we're still not done because part three, I want to talk about how you scale this across teams. It's a different kind of challenge. If you're a solo builder, you don't need that part. But if you're managing a team and you have production prompting pipelines, understanding how the system scales is actually really important and I want to get into that and get into some of the principles of that. So stay with me.
We're actually going to do a little bit of visuals on this one. I've actually seen some requests for folks to do more visuals in these videos. We're going to get to that here. Walk through for beginners, for builders, and for teams. And then there's going to be lots more good stuff in the post for those who want to go farther. Let's get to it.
All right, here we are. You know, I love my graphics. Uh, fair credit. This is Gamma helping me organize my thinking. Uh, nice little AI tool using AI to optimize AI for prompting. And the framework really does scale from beginner to enterprise. So, with that in mind, what are we talking about?
What is this scary programming language? This is called DSPI. And it's a a fork of the Python language that enables you to work with large language models by treating prompts as programmable code rather than static text. It's not really a fork. It's it's a library.
The framework enables systematic prompt engineering so that you can actually scale LLM applications in ways that go beyond just writing use chain of thought or like some other adjective to make things better. It enables you to be structured and systematic with your prompting so you're much less dependent on individual expertise which has tons of benefits as we'll see.
But don't worry, we're going to start with beginners first. So the first thing to do if you're not sure what I'm talking about is just to get these concepts under your belt and then the next slide we're going to have an actual full beginner prompt to walk you through that you can paste right into chat GPT.
So DSPI essentially provides a bridge. What you're doing is you're saying here's where I want to go, right? You part one, you're defining your task. Then you're saying part two, here's some examples of how the finished product looks like. One example of this is I want you to write a customer service email. Here's some good examples of customer service emails. Part three, you want the prompt optimizer, the DSPI library to automatically refine its prompt structure to optimize to reach those outputs.
And so basically, you want to say my goal, here's what good looks like, and here is an input for that good. You notice it says input output pairs in number two. That's definitely key. You're basically telling the DSPI program, hey, could we have an input and an output that looks like this, but I'm only going to give you the input next time, right? So, it's pattern matching, right? It's not that fancy.
If A equals B and C equals D, then E equals F is what you kind of want it to be doing. In this case, if I give you notes on the customer call and I give you what a good email looks like three or four times, you should be able to get notes on a customer call and produce a good email. That's the core idea.
And yes, you don't have to run DSP to get that kind of results. And I'm going to show you how if you don't want to touch the terminal. Well, what DSP does, it basically optimizes and it iterates. And then once it is able to reliably produce a good email, you can actually integrate it into your production pipeline for AI so that you know that you have an optimal prompt and it wasn't just based on best effort.
And that in turn increases the overall quality of all your prompting because you're actually allowing AI to optimize for AI. You're allowing AI to bridge the gap between your input and the output you want and construct the prompt that links them. And that's really the key idea I want to get across. Let's get into what beginners can learn.
This is a real prompt. You can grab this prompt. So this is not technically DSPI because obviously it's not the Python programming language, but it is a prompt that works like DSPI and works in an LLM or large language model like chat GPT. And so it's very simple. It says I need I need to create a self-optimizing prompt system. This is my task, right? Write an email, summarize meeting notes, whatever it is.
These are my examples. Here are at least three pairs. An input and an output. Input. Output. Input. Output. And make the outputs really good and make the inputs really consistent. So, if you're going to give it inputs and they're all wildly different, you're not helping it. If you're not going to grade your outputs consistently, you're not helping it.
Now, please create a scoring system with specific criteria. And then you have functionality, format, completeness. You can adjust what those criteria are. This is an example. If you don't value format as much, you can drop it and put something else in, right? But you want to as clearly as you can specify how the system should score success when it is practicing.
You are then going to tell the system and chat GPT will just do this in one shot. Please write multiple prompts that could handle my task. In this case I say three. You could do more. Please test every single prompt on the examples I gave you and score the results. So it's basically going to test each of the three inputs. It's going to see how closely it can mimic the output you gave it, and it's going to give itself a score based on the rubric you gave it.
Step four, please take the best one and improve it by fixing whatever element scored the lowest from your rubric of functionality, format, or completeness or whatever you want. And step five, give me the final improved prompt with a scoring system. That is all one prompt in Chat GPT. And that is as close as you can get as a beginner to what it's like to work with DSPI. And you don't have to do the terminal. You can literally do this anytime. And that is the whole concept that we are working with for more complex production pipelines.
But let's say you are an engineer and you want to understand a little bit more what is going on here. This is where we get to part two. We start to talk about what that means. For engineers and builders, DSPI turns prompt engineering from an area of personal expertise into an area of programmable discipline. It basically reduces the ambiguity in the space and turns prompting into a more deterministic science which in turn makes it much easier to provide clarity and control for systems engineering.
And so you can define LLM behavior with signatures. So signatures are really just inputs and outputs, right? You're treating prompts like structured code and you're delivering signatures that enable the Python library to reliably develop a prompt that maps inputs and outputs in what you're giving it.
It is easy to have modular architectures with DSP because you can swap out different components. For example, you can easily swap out the language model that DSPI is calling upon to build these prompts. Super easy. It's like one line in DSPI. And that in turn makes it easier to sustain, easier to upgrade, etc.
You also have the ability to continue to optimize prompts for specific tasks because you can automatically refine as input and output pair systems grow. And so there's a lot of different elements here. We're going to get into it more, but I want you to get an idea of what we're doing. Fundamentally, if you have programmable prompts, if you have a modular architecture and you have some kind of automated optimization loop, you are going to be able to actually build precise LLM applications and not depend on the skills of your best prompter.
Traditional prompt engineering, it had it had defects. I think we all know there's not a systematic way to improve. It's difficult to measure progress objectively. It's really hard to scale it. It's brittle. It is often model specific or it claims to be model specific. I saw someone joking that prompt engineering is just a it's like throwing darts at a dart board, right? Like you're just throwing it and you're throwing it blindfolded and you're not sure if the darts land or not, but you're making big claims about it.
Traditional prompt engineering does work if you don't have better options if you have a skilled prompter and if the skilled prompter is able to evaluate their work honestly. That is sometimes true and very skilled prompters will sometimes still write prompts that are better than DSPI will write. But DSPI scales consistently in a way no human can. And that is why engineers have been preferring it. It is much much easier to scale as a software system.
So let's get into the core philosophy. If you're treating your prompt as a program, if you're treating it as code, which I've been advocating for a while, you're going to insist on clean inputs and outputs, which I talked about. You're going to insist on modularity throughout the architecture. And you're going to insist that you don't treat prompts as strings. Prompts should be treated as code instead.
And you should enable a metric-driven feedback loop. So remember when I talked about automatic optimization a couple slides ago, the way you do that is by defining quantifiable metrics that DSPI can optimize against. So when I gave beginners a measurement system in the chat GPT prompt just now, that is the beginning of a quantifiable metric. And in production pipelines, you go a whole lot farther. You dive much deeper into what you define as acceptable. And that helps DSPI write reliable prompts.
So what are the key components? I talked about signatures. I want to actually get into what they are so it's not confusing. Signatures are input output contracts that specify what your module should do but do not dictate the how. So for example, if the context is question and answer or email draft and feedback to improved email, like those are pairs. You're specifying this is good and this is good, right? The question is good and the answer is good. The email draft and feedback is good and the improved email is good, but you're not explaining how anything happened in between.
You were asking DSPI to essentially write a prompt as an optimization function in between to bridge the gap so that you can in future provide email draft and feedback only. It will apply the bridge and it will get to improved email. Modules are another key component. These are composable building blocks that combine signatures with specific reasoning strategies like React or Chain of Thought. And you can actually chain modules together to create more complicated workflows in DSPI. And that's important because you don't always need inference, right? Not all modules require inference or chain of thought. It gives you flexibility. It's like Lego bricks.
Optimizers are automatic prompt optimization algorithms. An example would be Bootstrap Fshot. and it improves your modules based on training data and defined metrics without any manual intervention. And so it just is always running. And last but not least, the metrics piece. You want to have eval functions that can measure accuracy, that can measure relevance, that can measure format compliance, that can measure custom business metrics because these help you decide what is good. These guide the optimization process and give you feedback that enables the optimizer to work.
So if we look at this in action, what you're doing is you're going to define your task, start with signatures, and then you're going to make sure that you have enough examples of input output pairs that DSPI can learn from those examples. And so in the chat GPT uh light example that we did for beginners, we had three. In real production, we're going to have much more, 10, 30, 40, 50. And DSP is going to learn from these examples to generate effective prompts.
You're then going to specify how to measure quality and accuracy percentages, what format looks like, and you're going to do so in a much higher degree of detail than I gave in the beginner's prompt. It's going to be not three different examples of what good looks like, but quantified examples across six or seven or eight dimensions of what quality looks like. Maybe it's a number of tokens, maybe it's a reading level, maybe it's format compliance. There's a lot of ways to do it and it's going to be dependent on the output you're looking for, but you need to define the output as specifically as you can.
Then you're going to choose an optimizer like Bootstrap Fshot for quick results or there's some that are sort of going to take longer. MERO for complex reasoning tasks is better. So you're going to pick the one that sort of works for you. And then finally, you're going to deploy it and keep an eye on performance. And you're going to allow the DSPI module to adapt to new data as you feed it new training examples. And so it becomes its own self-improving prompt system.
To scale DSPI across teams is a separate challenge. So if you start with personal workflows, you can get significant improvements, right? You can automate email responses, content generation, data analysis. There's lots of good stuff you can do. Individual engineers are using this already and teams are starting to as well and doing so successfully.
But it requires sharing optimized modules across teams through centralized registries. So you actually have scalable architectures and you're not all working off different optimizers. It requires quality gates and cost control. So you are determining the acceptable cost you will pay for quality at a given scale across a range of tasks. And it requires infrastructure for governance, infrastructure for automated model selection.
If you don't do these things, you end up with a complex library of optimizers that individuals are maintaining on a best effort basis. Costs run out of control and you have great difficulty actually building a consistent pipeline for prompting. And so, as much as this may feel like individual engineers want to roll their eyes, if you're a team leader, you have to be thinking about this as you start to scale your production pipelines.
All right, I hope this has been helpful. I want to call out that it actually doesn't it's not that scary to get started that to get into bootstrap fshot and start to optimize right away as long as you have signatures and input output pairs it is totally doable and you can get to applying it to real work quickly week three to four like I know people who've done it much faster than this right like I know people who have gotten into this in just a few days and gotten to actual workflows in the business it's totally possible to do it and the key thing is it removes one of the biggest human dependencies is in the prompt equation.
You now get consistent scaling of prompt engineering expertise by having AI write the prompts and that's pretty cool. So there you have it. That's an introduction to DSPI. That's why I'm excited about it and I hope it gives you a sense of where the state-of-the-art is going as far as using AI to optimize prompts. It's a wild exciting world and uh yeah, I've written a whole post on how to actually get into it whether you're a beginner or whether you're an engineer or even a whole piece on being a a team leader and having the the glorious and fun job of optimizing entire teams in production pipelines for prompt optimization that actually runs. Good luck. Have fun.