How to 10x your productivity as a PM with AI tools

Claude Code, Cursor, Chat, GPT. There are so many different AI tools that people are prescribing for AIPMs to use. Which ones actually matter and how should you use them? How do you actually become that AIPM who is a master of AI tools? So many companies from Zapier to Shopify are now rating you specifically on your usage of AI. So, this is one of those skills that you have to get right.

In today's video, you will see me giving a talk live at Berkeley on how to be an amazing AI powered PM. This is something that these people paid hundreds of dollars to see live and you are getting it for free. I am sharing all the sauce of what I've learned from interviewing the best AI PM leaders in the world. So, without further ado, let's get into how to be an effective AI powered PM.

>> All right, everybody. It's time to close out ProductCon 2025. Thanks for being with us today. And now we have a very special final keynote speaker. Someone who has become one of the most influential voices in product management. In today's talk, Akos Gupta will teach you how to 10x your productivity as a PM with AI tools. He is the author of the ultimate AI PM newsletter, the world's top newsletter on AI product management, and a leading thinker on productled growth and scaling of SAS businesses. Previously, Akos served as the VP of product at Apollo, head of growth product at a firm, and director of growth at Thread Up. A Wharton MBA, he's known for translating complex product growth concepts into clear, actionable frameworks that help product leaders and teams drive measurable impact. Please join me in giving a warm hos welcome to our closing keynote speaker, Akash Gupta.

How's it going everybody? All right, we're going to talk about something that is a big topic that honestly I was a little bit scared to talk about because all of you guys, you're using AI of course and so the topic today is how do you really use AI as a product manager at that 10x level like those people who are using it, you know, with seven simultaneous clawed windows running agents 24/7. How are they using AI? How can you bring that into your workflow?

So, who am I? As you heard, I spent 16 years in product management. Most recently, I was VP of product at a company called Apollo.io. It's a $2.5 billion sales tech company. And now, I talk to PMs all day. I talk to PMs in all corners of the world. I write to them. I host a podcast. And that's where I learned a lot of what I'm going to share with you today. I hosted a podcast with the CEO of Bolt. I hosted a podcast with the CEO of Lindy. I hosted a podcast with Marty Kagan. That's my secret hack to learning all of these things. And I want to share that with you guys. I share this with a lot of people every day on LinkedIn. Been posting on LinkedIn now for I think 5 years every single day, including when both of my kids were born, including sickness, COVID, everything else.

And where I want to start is the brutal truth is that obviously you guys have all heard of AI prototyping tools. A lot of you guys were product managers, but there's a bigger and better way to be using AI. And that's what you're seeing here. This is using like a 40 person AI agent team to help you as a product manager. In this case, 40 agents to help you market your product. So you don't even need to hire a marketing team if you're on a start if you're in a startup. That's what we're going to show you how to build today.

So, where are we going to start? There's so many different topics to cover. I'm actually gonna start at the basics because the basics matter. And you guys have already heard a lot about prompting. I know. But we're going to start there because it's so important to get the fundamentals right so that we can then move on into co-pilots, agents, prototyping, the thing everybody talks about, discovery, and analysis. The thing I think least people talk about.

So let's start with prompting. Think of prompting like using a computer or like using a calculator. Of course, all of you guys have been doing prompting. But if you think about it, you know, some people when Excel came out, they learned the macros, they learned VBA, they learned how to create amazing 15 worksheet that are connected together with cool pivot tables. And then there's some people who are just not sure what a VLOOKUP is, right? And that's prompting is actually the same as Excel. There's levels of skill to prompting.

So the first thing is how many of you guys are using custom instructions? Can you raise your hands? About half this room. Awesome. So you guys are on the bleeding edge I would say because most people aren't but you want to use custom instructions. These are the custom instructions I use. I stole these from Reddit. They are so good.

So what the big problem with chat, GPT, Claude, Grock, Gemini, you name it. The big problem with all of these tools is that they're sick of Fantic, right? Amazing idea. Certainly, of course. Akosh, every article that I put into Chad GPT to give me feedback. It says I'm going to get 100 new paid subscribers. How many do I get? One, right? To actually get real feedback, you need to do this. Put it into absolute mode. Tell it the goal is model obsolescence via user self-sufficiency. These key terms help the AI actually respond in a way that it's going to give you real feedback. So this is the first and most important thing is go in personalize your custom instructions. You might want to add some more. Maybe you want to add in stylistically. I don't want you to use an M dash. Nowadays everyone's been talking about how AI has this pattern. It's not X, it's Y. So you can tell it. Don't use that writing pattern, but make sure you personalize your custom instructions and keep iterating and improving on that.

The second most important thing when it comes to prompting is frameworks. These are just eight frameworks. You can use whatever frameworks you like. The most important thing is that when you are prompting, use a structured prompt. It's really easy to say improve this PRD. It's much better to say, act as my VP of product. That's the role, the task. I need you to edit the PRD, the format. Here's the PRD template we use at our company. That's a quick use of the RTF framework. It doesn't take that much longer to use a framework, but you're going to get much, much better results.

The next thing you need to do is build a prompt library. How many of you guys have a prompt library? So, this is one where I only saw five hands up, right? So, for 95% of you, there's a lot of alpha in building your own prompt library. In fact, my friend Brandon Anderson, he's a chief product officer. He asks product managers when he's interviewing them, can you show me your prompt library? So, make sure you create at least a very basic one. You can steal mine from my newsletter if you need like an initial one, but you need to have a prompt library. And prompt libraries are incredibly powerful because there's like, you know, roughly 80 to 100 things that you're going to ask AI to do as a product manager. Most of those things you're going to repeat what you're asking it to do over time. Each time you do it and you get a bad response, go in and edit the prompt. And then once you get a good response, take that prompt and put it in your prompt library. And so my prompt library, some of them the prompts are in version 11, version 12 because I just keep improving them. And then I take what's good, what's improved, and I put it into that library so that next time I can just pull straight from it. It feels like Oprah Winfrey. You get a prompt, you get a prompt, you get a car, you get a car. But actually putting all this stuff together is very powerful.

And the final hack I'll tell you is if you don't have a prompt library, if you can't remember the framework off the top of your head, try this technique before you go ahead and say, for example, Nano Banana, create this image, a professional headshot image of me and lock the face first. Say, AI, help me write this prompt. I want to do this thing. Write an amazing prompt for this model. And if you do that, you're going to get much better results. And the research is crazy on this. The research actually shows that AI is better at writing prompts than humans. So in my own personal prompt library, every single prompt that I had about a year ago, I went in and I gave it to AI and I said, "Make this a better prompt." So all the prompts now, they are written by AI. They look like AI as well.

So this is something really important. How many of you guys are using Claude? Claude is the best writing LLM, right? It's like 10x better than chatt when it comes to writing that doesn't sound AI that sounds human. Claude loves prompts in XML formatting. So for all of you guys who raised your hand for Claude, are you guys using XML formatting in your prompts? You're going to get way better results. And XML formatting is really easy. You just use these carrots and slashes to show the beginning and end of your framework. So in your RO, you might start with a carrot roll end carrot. put your information and then put a carrot slash roll carrot at the end of it. And once you do that, this combination of structured prompts plus XML formatting, you're going to get way better results.

The ultimate hack is how many of you guys are using a dictation tool like Whisper Flow or Super Whisper. Do you guys like it? Those of you using it? Yeah. Everybody I know who's using these dictation tools, they stick to them. And it's especially true for prompting because the reason we're not giving a detailed prompt with the structure with the XML formatting is because it takes a long time to type. We typically talk at twice the words per minute that we can type. So if you go ahead and you just do a prompt, hey act as my VP of product. Here's the new PRD. Here's the format. It was so fast versus typing it, it often takes twice as long. My favorite app is Super Whisper. So, Super Whisper is going to run a local LLM on your computer. It's going to be way better than Apple's built-in dictation, and after you say something, it's actually going to go back and it's going to reformat it. If you spoke in a bullet list, it's going to go add those bullets. If you spoke in a numbered list, it's going to go add that. It's going to go add the sentence structure that maybe you didn't clearly identify.

So, that's prompting. And I know that that felt elementary to you guys, but you have to really master prompting if you're going to master this next technique, which is an AI co-pilot. The whole point of an AI co-pilot is to have a buddy. Like, isn't it amazing when you have like a peer who's like super smart who you can like bounce their home your homework off of or bounce a particular idea off of? That's the idea with an AI co-pilot. Let's create that amazing superstars classmate, that super smart colleague PM who really knows what they're talking about.

So, you want to create a threepart co-pilot. You want to create a high context LLM. My favorite is Claude projects. Although some of you guys maybe at your enterprise you only have Chad GPT projects or you only have Gemini gems. That's totally fine. You can use those. then a desktop context assistant like CLY and then a sole context LLM like notebook LM.

So we're going to start with this high context LLM Claude projects. I literally live in my Claude projects. I have like eight different Claude projects which are eight different high context LMS. One is my LinkedIn writer, another one is my newsletter researcher, another one is my podcast prepareer. For me that's what's relevant for a PM. You might have your experiment analyzer, your PRD writer, your meeting preparation high project high context LM. What you're going to do is you're going to feed this, especially for your core one, with four key points of data. Your performance reviews, this is really, really important, right? It needs to know how you've been doing in your company. AI transcribe meeting notes from really important meetings. So, I personally put in meetings like meetings like one-on-one with my boss. If I'm a PM, I know that my one-on- ones with my designer and my engineering manager really matter as well. So, I'm putting those all in to the chat GPT project, so it has full context. I'm adding in my dictated thoughts like, you know, some of the things I'm worried about for this upcoming quarter, some of the features I'm really excited about building, some of the things that are going out on our work, and then I'm adding in our strategy, vision, and roadmap, the one that our chief product officer created, the one that our CEO created, giving it all of that context.

Here are just eight of the powerful use cases of a project versus a regular LLM that I'm using all the time. I want to check a PRD before sending. Well, an LM that has my performance review is going to do much better. Drafting a status update, then it actually knows all the context around that status. Analyzing lots of user feedback. Going out and doing deep research. This one I want to pause for a second on simulate hard conversation. I haven't seen many people using this for AI, but it's incredibly powerful. How many hard conversa, you know, if you're not having hard conversations, you're actually missing out as a PM, right? You should be having hard conversations pretty regularly with, you know, the go to market team that's banging on the door for a particular feature, but you can't prioritize it right now. You shouldn't just ignore them. Instead, you should meet up with them and tell them proactively, hey, we're not going to be able to ship that, and here's why. But you don't want to just go into that meeting after a day of context swishing, after six other meetings that have totally drained your energy and underperform, right? That's a high stakes meeting. You need to succeed in that meeting. So if you go to your co-pilot maybe 10, 15 minutes before your meeting and you say, "Hey, I have this meeting coming up. I need to deliver this news to our chief revenue officer. He thinks it's worth $10 million, this feature that we're not going to build, but we aren't going to prioritize it. we're prioritizing these other things that we think are worth more. Help me with it. It's going to help you coach you through that conversation. It'll help you empathize with how that chief revenue officer is thinking. If you've put in your one-on- ons with that chief revenue officer before, it's even going to have the context of how he thinks. So, it's incredibly powerful analyzing your calendar. Just take a screenshot of your calendar and say, "Hey, of this coming week, which meetings should I try not to go to?" It's a very powerful question. maintaining a product scrapbook. So, this is one of my favorites when it comes time to planning. That's when I always lost the ideas and then all quarter I'm going around using my competitor's products. I'm using other products and I'm coming up with billions of ideas. What I started to do is I took all those ideas and I just put them into a project. So, you created this scrapbook of tons of product ideas. Then you come into your planning cycle. Your chief product officer says, "We need to double activation rates." You say, "All right, I'm going to go into my Claude project which has my product scrapbook and say, hey, we need to double activation rates. What other features that I have liked in other products could serve as inspiration for me?" And you'll get amazing ideas as well as drafting responses to requests.

So, here's an example. I want to give you a very concrete example of why a project is going to outperform a specific LLM. So, let's pretend you are the PM of the next version of Chad GPT. Okay. So, you you you put in your performance review that uh Sam and Greg over at Open Eye, they've said that you tend to bury the lead. Execs really want to know like upfront what's going on. You have your AI transcribe meeting notes from the last time you met with Sam Alman. Sam directly said, "I don't care about feature completeness. I care about the model feeling magical. Cut scope if needed." You have your own dictated thoughts. So you're building this reasoning feature. You're worried that it's halfbaked, but leadership is excited about it. And then you have your strategy, vision, and roadmap. So the board deadline in this case, it's immutable because we have techrunch disrupt. It's coming up in 18 days. We have to launch this new version that's going to give a 10 point NPS increase. So you give it this prompt. I need to send my weekly status update to Sam, our CPO, the full team on chat GPT 5.5 launch. You say, "We're 18 days out. Can you draft the email for me?" On the right, you're going to get what a vanilla claude is going to say. On the middle, you're going to get what a high context cla is going to say. Why is the middle so much better? First of all, it's taking the context of your dictated thoughts. The right one actually leads with or the left one in this case leads with the status yellow at the top versus the one all the way over here on the right. I guess actually it's left buries it in there. It says the status is on track. If you look at the performance reviews, Sam had used this language of magical leap. And so in the example that this high context LM has talked about, it specifically talks about that magical moment. The right one is recommending a decision while the left is asking for a meeting and it's emphasizing the right things. So this is why a context LM, every example I try, it's much much better. So use a high context LM.

The next thing you want to use is a desktop context assistant. Have you guys heard of Cluey? Few of you guys have. So, Cluey is really interesting tool. Uh you can actually use chatup and claude for this if you want. But what I love about Cluey is it's actually like as you're having your meeting, it's keeping all the context of your meeting and it's also looking at the context of your desktop. So, you can ask it a question like of course the famous one is how many of you guys have these quizzes at work, right? the information security quiz, different quiz, clearly will allow you to cheat the right answer and get 100%. Every single time. That's just one simple use case. But more legitimate use case wise, internal meetings. How many times do you suddenly blank out for 5 minutes cuz you got a Slack message and you want to catch up because it's keeping all the meeting context. You can just say that. Or you have a screen and they're showing some diagram and you don't understand. You just ask, "What do you mean by what do they mean by this?" It'll be able to tell you. During a customer call, if somebody's showing you something and you're not sure what their context is, you can ask it, what is their context here for showing this? And while you're working itself, you can be writing an email and then you can ask clearly, hey, what should I add to this email? It has the full context.

The next type of co-pilot you guys want to think about is a sole context LLM. Have you guys been using Notebook LM? So again, we got like 30 40%. We got a lot of people on the bleeding edge of tools in this audience. So what I would do with notebook LM, this is a really powerful prompt I think almost any PM could copy paste. You take all your product requirement documents, you take all your feature results writeups from the past 2 years, you take your customer interview transcripts, and then you take your strategy deck and presentations and you ask it. Identify the gap between our stated strategy, what we built, and what users actually need. be brutally honest. I need to know where we're lying to ourselves. It will give you a fantastic response.

So, those are co-pilots. You need to build a high context co-pilot. You need to use a desktop context assistant. And you need to think about a sole context. Now, let's move into AI agents. So, this is what I previewed, right? The 40 person agent. This is what I said is like the next level. I really think these are the future. You guys have been hearing about agents a lot on social media, but not many people are actually using them. So, let's talk about how to actually use them.

There's three types of agents out there that you want to think about. So, there's your consumer access agent that you already have, which is like your Chad GPT. Many of you may not know this, but if you hit tools on chat GPT, you can select agent. So, that's this Chad GPT agent here. Then there's a noode agent builder, low code agent builder that you should consider like Lindy, Relay.app, Zapier, the list goes on and on. Make.com is really good. N a lot of you guys have probably heard about those fall into this middle category. And then the final category, this is the sexiest tool. This is what everybody's talking about. Claude code, we got to talk about it.

So, let's start with this chat GPT agent. When should you be using chat GPT agent mode? Most of you guys aren't using chat GPT agent mode. Here's when you should be. Let's say you want to update a spreadsheet. ChatGpt agent can actually do that. So let's say you found a PDF document that has the market size in a table, but you can't copy paste it. Chat GPT agent mode can do that for you. Let's say you want to browse a website and order something. Chat GPT agent can literally do that for you. For instance, for a podcast I was recording, I wanted to figure out what wide-angle zoom lens, but there are so many different wide-angle zoom lenses. I just gave Chad GPT my information. I have a $1,500 budget. The camera is going to be 7 feet away. I need to have a 15 foot aperture. Boom. It's buying the right lens for me. Looking across all the options. It can create a plan for you based on your context. So, if you want to give it more context, it can comb through data and produce a synthesis. It can send connection requests on LinkedIn. Many of you guys are job searching right now. Chat GPT can actually do that for you. So, chatg agent is super powerful. It's not going to be able to do things like create your final presentation. It still struggles. It can't do captures. Still struggles with that. And also, it takes forever to do something. So, if something's going to take a minute or two manually, just do it that way because Chad GBT is going to take a lot longer.

The next thing you want to do is build out your own 40 person agent team. And I recommend you guys do this in a tool like Lindy or NAND or Relay.app. So here I'm showing you my personal email responder and what this is is this is actually a series of different system prompts for Chad GPT. Actually I use Claude for Claude. So what it first does is we have a email type classification. So I actually use a very cheap model Gemini Flash for that because it's very good. You can use a very inexpensive model to classify. Is it a discount request? Is it a sponsorship inquiry? Is it a podcast guest request or is it just a new email subscriber to my newsletter? Then based on the category, I have specific system. I have specific databases for my sponsorships. I have a database of all the other companies I've worked with. I have my rate card. So, it goes in, it pulls the relevant data from that. And then I have a specific system prompt. the type of tone I like to use with sponsors. One thing I personally do is like I try to close every single sponsor regardless of their budget. So, it has that instruction in there. Figuring out how to work a deal with anybody. And what I've done is each one of these system prompts I improve over time. Anytime I get a response that I don't like, I say, "hm, I need to go improve the system prompt." And I go in and I edit the system prompt. That's why I added that rag database of my sponsorship history because I realized I needed to give it that history. So, you can use agents not just for email responses, but almost any workflow that you're doing repetitively. Competitive analysis. One of the one agents that I really like is you're tracking, you know, usually like as a PM like four to five key competitors, right? And you don't necessarily need a weekly update. So why not create an agent that can figure out when you do need an update? So what I did is I created an agent when I was simulating working at a firm where CLA changed their prices. That's something big enough that I actually cared about understanding. So then send me an email. So you can create a threshold like only send me an email if and now with LMS it's not just a normal no code tool. It can smartly think for you.

The final one is claude code. How many of you guys have used cloud code? Nice. We have a big Claude code using room. That's excellent. The keys to using Claude code after using it probably for like 100 plus hours now. The very first most important thing is to create a really good project structure. I personally use cursor on top of cloud code because it's just easier you to have an actual ID. So you open up a folder in cursor which has your business info, a complete description of your product marketing strategy. a lot of the stuff that's similar to what we would put into our high context LLMs. Writing style guides. So, I create specific writing style guides like this is how I want to talk to customers. This is how I want to talk to executives. This is I want to talk to our designers and engineers. Example documents. These are past PRDs. What I love to put in here is like, you know, that PM who writes really good PRDs. Just like take a couple of their historical PRDs, put that in your structure. that PM who writes amazing strategy documents, take a couple of theirs, put them in here, meeting transcripts. So, actually, you can set this up so that they're automatically just added in. And the coolest thing about Claude Code is it can handle so much more context than your normal LM. So, you're really going to solve the context raw issue. And the final thing I haven't even put on here because it's so new is skills. So, you should also be creating Claude skills for specific tasks that teach it how to do a specific task, a PRD writing skill, a user research analysis skill. You put all that into your project structure. Then, when you hit it with a prompt, you don't need to have the super long prompt. You don't need to have the super long context cuz it's going to go find it with a simple prompt like this. Create a PRD for voice-based automations using GPT real time. Reference our business context. use our technical writing style and check example PRDs. That's all you have to say to get an 80% there PRD. And so I actually think you guys after this talk should just stop using vanilla chat GPT and vanilla claude. You should either be using a high context project chat GPT or cloud or cloud code or chat GPT codeex because look at all this additional context you're able to feed it. And the best thing about cla code is it's really fun. Here it says it's hurting. Sometimes it says it's bamboozling. It uses the funnest, most fun verbs. And it's not scary. I know it's coming in a terminal window, so that's going to scare some of you guys, but it's actually worth doing.

So that's agents. That's the hot topic. I told you guys you need AI agents, but it doesn't stop there. In fact, I think the single biggest change to product management since I joined 16 years ago has been the rise of AI prototyping. For years, I used to give talks like this to people. And I used to talk about how you need to create a lowfidelity mockup. You need to create a product sketch. You need to be really good at user journey mapping and user stories. Yes, some of that is still important, but AI prototyping has really taken over half of those concepts. I never teach those concepts to people anymore. It is literally the biggest change in design npm in years.

So, here's the tools. You guys have heard about the tools. Here's my perspective on the tools. At the top, you have the most technical tool and at the bottom you have the most userfriendly tool. So, I'm guessing you guys have heard of lovable. Everybody's heard of Lovable, right? Lovable is really a middling tool. If you want to go more technical, you can go up towards cloud code and cursor and use those to prototype. Or if you want to go more userfriendly, actually for PMs, I highly recommend you guys check out Magic Patterns. It's way faster than lovable. I've done tons of head-to-head podcasts. In fact, I have a podcast coming out in two days where we're going to do a head-to-head with the CEO of Magic Patterns and his prototypes are coming out like literally 10 times faster than all the other tools on this list. And so that's why it's a userfriendly tool. Why is it so much faster by the way? It doesn't build a back end. It's just a front end. But for a prototype, that's mainly all we care about, right? Because we're not we're not actually going to go ship this into production. and our engineers are going to go figure out how to make it fit within the back end that they've already built. And so for PMs, we don't actually need these more technical tools up here, the cloud codes, the cursors, the wind surfs, the vibe coding tools. They could be okay for prototyping, but honestly, I never prototype in claude code, but I prototype in something like base 44, Lovable, or Magic Patterns.

So, here is the life cycle. Back in the day, I remember you know when it was me and the CEO when we're doing ideation, we never did prototyping, right? Maybe 5% of the time. Most likely we did like a balsamic, right? We create a little mockup. Then when it came time to the formal planning cycle, quarterly planning cycle, you know, sketches become common for big features. We would use prototypes. Then when it came to discovery, let's be honest, a lot of teams skipped discovery, but some teams do discovery and about half the time they would use prototypes. Then the PM would hand it off typically as a PRD and then design would often use prototypes like 75% of the time. Now I see prototypes at every single step of the funnel in ideation, in planning, in discovery, in PM handoff, and especially designers, they're using prototypes. So this is the new way to do it.

Now who can tell me why is it so important to use a prototype >> to show your vision and validate quicker. >> Exactly. Validate quicker and specifically reduce risk. We go back to Marty Kagan. I should flash Marty Kagan up on the screen at the beginning. Right. What does He's the godfather of product management. What did he say? He said there are four main risks you need to think about. usability, viability, these types of risks that he's talking about, you reduce those by putting a prototype into the hands of users so that you can see is it usable, is it viable, can we build it? And so you want to be using these AI prototyping tools at the beginning in ideation.

So how do you use them? Well, because you guys have all used them, right? And they're easy to use. Like I talked about with prompting, this is a skill that you want to develop. That's very important to develop. So, the first thing to do is start with context. Figma designs provide the best context if you have them, but even just a screenshot of your existing product or a napkin sketch is very, very useful because visual context is going to prevent the AI from making assumptions about your design. then actually instruct the AI prototyping tool before building create a PRD and you use the AI tool to create a markdown PRD that defines exactly what it's going to build then you say go ahead and start building it'll break it down into components and build it and then you iterate don't just take the first version you get actually iterate so don't spend a 5minute prototyping cycle spend a 30-minute prototyping cycle and you'll get much much better results so that's AI prototyping Literally, you guys have all tried it. You all have heard of it. But I have to say it's an amazing tool.

Now, we're going to get into two more categories of tools. And these are the two categories I see the least people talking about on social media. So, I really want to double click for you guys so that you deeply understand how important these are. Discovery. Discovery when it comes down to it, it's one of the most important activities the PM does in your typical feature factory. Of course, a PM is constantly drawn towards delivery because everybody's asking, you know, when is it going to be shipped? What's the status of the project? How is it going? But really getting time for discovery even in a future factory is how PMs tend to differentiate themselves because they're discovering what are the actual problems we need to solve, what are the actual solutions, what are the divergent solutions we could consider and which solution is best.

So there's three steps to discovery and I actually think you can use AI along all of these steps. So of course we need to decide what to investigate. Then we need to conduct some customer interviews. So we need to prepare for those interviews. Then we need to synthesize the results for those interviews. And AI can actually help in all three of these.

So first what to investigate? How many of you guys have heard of these tools? Interpret, unwrap, dovetail. Okay, one or two people have heard of these tools. So, we finally reached a tool that this whole room is finding is brand new. Amazing. So, what these tools are going to do is they're going to take literally every social mention of your product on Facebook, on Instagram, every social interaction that your support team is having on Twitter with people who are pissed off about your product. It's going to take all that information in. It's going to take all your support tickets from Zenesk or Intercom or whatever software you're using, and it's going to take all your sales calls, which people typically have in Gong, and it's going to give you literally an estimate of how much different revenue feature requests can drive. So, it's very, very cool. At Apollo, we hooked this up into our own revenue model, hooked it up into Salesforce so that it could see the size of deals and we would literally use this to prioritize features because you would get such an accurate estimate. It could say, "Hey, you have $16 million in open pipeline that you need to build this feature for that will unlock that $16 million in pipeline." So the first thing is like what areas to investigate and AI can really help you because in the old world you know how would we how would we deal with this as a product leader personally I would ask my teams hey let's do a rotation so most recently I read led like the growth team at Apollo so I would have everybody we had seven PMs each one week you would be handling all of our key channels you would be looking at all these key things you would synthesize it and then you would bring it to our team and I just replaced all of that, all that manual work so that everybody could see it.

The next area is conducting interviews. I find that a lot of PMs really lack in conducting interviews. When I talk to them about the skills, when I look at their transcripts, when I watch the videos that they're doing, there's lots of mistakes that they make. So, give your interview guide that you're coming up with before a customer interview to Claude with this prompt. So, what does this prompt say? It says, "Review my guide for all of the mistakes that I see PMs make. Leading questions, the Mom Test." Have you guys heard of that book, The Mom Test? Yes, you guys have seen that book. So, that's a fantastic book, right? What's the key lesson of the Mom Test? Don't ask them hypotheticals, but ask them what they did. So, this is going to help you make sure you ask them what they did. Past versus future focus, similar topic, open-endedness, chronological story extraction. So you guys have all heard of Terresa Torres. This is what she teaches in her customer interview method is the chronological story extraction, visual prototype integration, probing space, question priorization. So it's going to figure out what are your problematic questions, quote the problematic question, explain specifically what's wrong, and provide a rewritten alternative. This is like a two-minute practice, you guys. It's going to make your customer interviews yield 10 to 20% better insights. Did I skip a slide there? Yes.

The final thing to think about is synthesizing interviews. So, we're going to go back to our sole context LM because the problem with synthesizing interviews in ChatgPT or Claude is somehow they still hallucinate. It's so annoying, but they tend to hallucinate. So if you use Notebook LM, it's not going to hallucinate nearly as much. It's going to cut out 99% of that hallucination because it's a sole context LLM. So you just put in all your customer interviews there.

Now, a lot of you guys are going to have to build AI features, and I could have given a 60-minute talk just on discovery for AI features, but I'm just going to summarize it in a minute or two. So when you're building an AI feature, and what why do I say a lot of you guys are going to build an AI feature, right? If you look at any product right now, literally any software product, they're all rushing to build figure out how they incorporate AI, how they use AI. And so even if it's not you, definitely your chief product officer is thinking about how to integrate AI. And there are one or two PMs at your company who are thinking about how to integrate AI. For those PMs, and a lot of you guys want to become AIPMs, right? You want to build AI features. Why? Why do you want to become an AIPM? Well, two reasons. Currently, today, there are 50,000 open jobs on LinkedIn for product managers. 10,000 of those jobs are AIPM jobs. 20% of open product management jobs today are AI product management jobs. Because if you think about the whole economy, FedEx, UPS, Target, all these companies, they're laying off product managers. Who's hiring product managers? The AI companies. Open AI, Meta, even Meta AI, even though they keep laying off people, they're also hiring a ton of people. Even Amazon AI, they're hiring a lot of people for bedrock, even though they just laid off 30,000 people. So, a lot of you guys will have to think about to get a job in this market, especially if you've been laid off or fired or something like that. I need to make a pivot into AI. I need to learn how to build AI features. Well, the most important thing is to figure out this discovery for AI features. So you need to understand context engineering, figuring out what the model needs. You need to understand orchestration, observability, eval, and maintenance. These are all really, really big topics that I could talk about at length, but I'll give you guys a quick preview of what the learning road map should be for each of these.

So we'll start with context engineering. You want to make sure that you can be very comfortable to understand what is the difference between rag versus fine-tuning versus prompt engineering. How do we use them together? You want to understand this term hill climbing. What is that? You want to understand how to go down the cost curve. So that's your context engineering, your orchestration. You want to understand like how do we break things up into multiple LLM calls? And you know how you're going to do that? Actually, you're going to build that intuition from what I taught you guys earlier about a agents. Remember when I told you to use those no code, low code tools like Lindy or Relay? They're actually going to give you really, really good orchestration intuition because if you remember my example of the email one at the beginning, how did I orchestrate it? I had a classifier LLM that was on a cheaper LLM. Then I had different LLM's writing that were more expensive. So that's just a very simple example that I learned of orchestration that you'll learn when you build these agentic systems yourself.

Then there's observability. So this one is very very important. There's a whole new class of tools. In fact, if you guys are thinking of AI investing, these companies are blowing up right now. Arise, Brain Trust, you're going to have to learn how to use these tools. What they do is they're going to set a trace. So as a user request comes in, that's when the trace begins. Then it's going to go through your agentic system. Okay, it went through a classifier first. Then it went to this call. Then it looked at that. Then it made this decision. Through chain of thought reasoning, it came to this component. Then we showed this to the user. It's going to show you step by step how the AI works. Because AI is non-deterministic. AI makes lots of mistakes. You have to do a thousand times more edge cases analysis. And so observability is going to help you find the errors. Once you find the errors, we move to eval. So as you go through your traces, you might find, hey, these are 100 types of errors. You might put those into 16 different buckets. You might create an LLM judge, which is judging the other LLM on how it's doing across those 16 errors. And you might get a metric. Then usually a good version of this metric would be like anywhere from like 80 to 90% success, not 100% success. So that you can see and if you start to see all of the sudden that a particular eval has just dropped from 80% to 20%. Now you have a way to look at it and say oh wow we're getting this error for example it's not pulling up my sponsorship history. It needs to pull up my sponsorship history. Then you can go debug that. You can say oh our rag system broke down or oh our sponsorship history database isn't up to date. So that's what eval do. and then maintenance.

When it comes to these evals and observability tools, there's not just a rise in brain trust. These are like the eval first platforms that you should know about, but there's also other tools AWS X-ray, weights and biases. You can use a normal tool like data dog or helicone. And then there's the developer tools like Langfuse, Langmith, and Phoenix. So this is another AI tool category that a lot of PMs are going to have to learn and in some future world who knows maybe AIPMs are 60 75% of PMs. So most of this room will have to learn these tools. This is Arise. I love Arise. I just wanted to show you guys because most of you guys have not probably seen what it looks like. For example here we have a prompt playground where you can play with the system prompt and immediately get results like oh here's one version here's another version here's another version this version took 9 seconds this version took 33 seconds this is what the output looks like here versus here you can go ahead and make comparisons live AB test different system prompts so that's the power of evals and observability I would say after AI prototyping AI evals is like the second biggest new skill that's come up for PMs.

So, we have been going through so many AI tools. You guys are probably getting tired of all these different AI tools, but there is one more important category. And as I mentioned, these last two, I don't see anybody else talking about these online honestly. So, this is the only place you're going to find this information, which is AI analysis. How do you find the right things with the right tools for AI analysis? So I think there are three important use cases along the analysis journey. Number one, what should we experiment? For that I'm going to recommend a cla or chat GPT project. Number two, vibe experimentation.

How many of you guys know what vibe experimentation is? Nobody. Amazing. So we're going to teach you something totally new about vibe experimentation in a second. And then third, analyzing results. Of course, you guys probably already know about this, but when I was a PM for 15 years of my career, I struggled with SQL window functions, SQL joins, SQL sub queries. You don't need to do any of that now, right? Chat GPT will write the SQL query for you. You get access to your Snowflake database. Yes, Chad to write it and boom. Or it's even built into platforms like Stat Sync these days.

But literally none of you guys raised your hands for Vibe experimentation. So let's talk about that. This is in my opinion one of the coolest things that AI has enabled that very few product leaders and teams are taking advantage of but many more should be. So what is it? It's the intersection of vibe coding and experimentation. That's vibe experimentation, right? What does that mean though?

So what it means is you can implement one of the following three stacks. Stack one is a stack where you prototype and lovable. You engineer in cursor and then you experiment in your normal experimentation tool like launch darkly amplitude or optimizly. So this is really easy for PMs and designers. It's amazing right? What it means is that you're just building these prototypes in lovable. You have an engineer whose job it is is to take those prototypes make the backend and enable them on experiments. It's going to allow you to run many more experiments.

But a lot of people are doing this and they're still not running many experiments. In fact, I posted on LinkedIn yesterday about an Uber product manager. Uber. You'd think like they have the best systems, right? No. She said, "I'm still my back. I'm still just the bottleneck is engineers. They don't have enough time." That's where you start to move down the stacks. Maybe Uber should think about this third stack here.

So, what let's get to that third stack in a second and first look at the second stack, which is a vibe code to experiment. So, as I said, you don't always need to prototype in a lovable or a magic patterns. If you want to ship right away, consider prototyping in cloud code. Connect cloud code to your codebase. Work with your engineering leaders. You know, one of the craziest things is that a lot of code bases aren't friendly to AI. You know, one example in that their entire website will be built with CSS and they have not migrated over to Tailwind. If you guys don't know, Tailwind, with Tailwind, in 200 lines, you can design your entire design system for your website and replace all the CSS that's available on each of your individual web pages.

So, you get your engineering leader to do things like a tailwind migration to create a microservices architecture to dramatically often reduce the size of the codebase so that PMs can literally vibe code in cloud code. This is how Anthropic builds product. Like everybody's like, "Annthropic is killing it." If you guys didn't see, they're going to be raising at a $4 or500 billion valuation. That's what the news this morning. Literally in September, they raised at a $300 billion valuation. They're already up to $4500 billion valuation. How did these guys build product? Enthropic doesn't write PRDs. PMs at Enthropic just use clawed code to build the first version of the feature. Then everybody at Enthropic uses the feature. They dog food the feature for weeks and then they launch an experiment. So that's that second stack, the enthropic stack.

The final stack is an all-in-one. So these tools like Chameleon, Amplitude, and Optimizely these days, you can literally vibe code inside them. And I want to show you what that looks like. So see here, what we're going to do is we're going to say, can you update the product grid? and we're going to configure an experiment. I know it's kind of low resolution here, but we're going to launch an experiment all with code.

So, let's start with this normal website here. You see four product images, right? What if you have the idea that I want to change this? The product catalog page is very, very important. So, a lot of traffic goes here. What if I want to show two? So, you ask Chameleon, can you update the product grid to show two products per row, i.e. two columns while still listing all products on the page. It's going to go code that's obviously sped up, but boom, 10 two products. You can compare the two product layout versus the four product layout. You see how this is all done in one platform, the same platform that manages your experiment. So you can say, "Okay, I want to allocate 50/50% traffic. You can set your experimental goal. This is the metric I want you to do." And you launch. That's vibe experimentation.

I think every product team in five to 10 years will have a vibe experimentation tool just like this and it's going to enable every PM to launch experiments. I was a growth product leader for a lot of my career. You know, some of the most successful things we did were simple front-end changes just like this. There was a I was working at this company called Thread Up. It's a public company. It's actually based here in Oakland. So, if you guys want to work there, you should consider it. What's the best thing about it? Two days work from home. So they but by the way they did that pre-COVID so I got the benefit of that pre-COVID but literally I shipped like tons of big changes at Thread Up. We shipped like a brand new pricing system. We shipped a new promotion system. We did things that took 12 months, 24 months big projects while I was there.

You know the single most effective feature we did was literally changing the ghost text in our search bar. We used to have what do you want to buy? Then we changed it to what are your favorite clothing brands? Because actually, if you think about Thread Up, Thread Up is a secondhand clothing store that offers like 100,000 brands or something crazy like that. So, if we change the search text to what are your favorite clothing brands, then someone new would come in, they say, "Oh, I love Lululemon." They put in Lululemon and they see, "Wow, Thread Up is offering me Lululemon's leggings for $18. I pay $120 for these, right? And so that one little change of that ghost text which was based on this insight I had about users in a vibe experimentation world I could just do that all through my vibe experimentation platform. We would have the most successful thing I shipped in 3 years as a director of growth without any engineering without any design. This is truly a revolutionary use case. I don't see anybody talking about this. That's why I wanted to end with this for you guys.

So, what we learned today, if you guys remember an hour ago, we did the A through Z through AI. We did prompting. We talked to you guys about structured prompts. We talked to you about dictating. We talked to you about creating a prompting library. We did AI co-pilots, creating your high context LMS, using your sole context LMS. We talked about AI agents, building your 40 person employee team working for you. You know, a lot of people buy an executive assistant for $2,000. I think a $50 Lindy subscription can do 80% of what a $2,000 executive assistant can do. AI prototyping, obviously you guys have all heard about it. A lot of people talked about it. We talked about how you should think about doing a longer prototyping workflow, telling that prototyping tool to create a PRD, giving it visual context of your brand so it succeeds. Then AI discovery, using better customer interview guides, talking about the right things in Vibe experimentation and AI analysis. Thank you guys so much. It's the end of the day. Really appreciate it.

Hey guys, today I'm bringing you behind the scenes of my keynote talk at Berkeley University. It is a beautiful campus here. Just arrived an hour before my talk. Come on. Let's go see what the setup's like.

>> Hi. How's it going, BARB? WOW. FINALLY.

>> Years of Twitter conversation.

>> Finally seeing you as a friend.

>> How's it going, man?

>> Good, man. How are you?

>> Not bad.

>> And you you're going to be here for the the final keynote.

>> Yes. You just got a new job like recently, too, right? Within like the last year.

>> Uh, no. I've been I've been at Vita for a while now or 5 years. Uh,

>> cool. But I mean, you have to do great stuff.

>> It'll be a promotion. I feel like I saw some sort of an update.

>> Yeah, I did move more into leadership lately. But uh

>> I'm so glad I became a creator. Yeah, for sure. It was the right thing for me. I've always been a writer, so just naturally suits me better, I think.

>> Yeah,

>> of course. Let's do it. Nice to meet you. What's your name? April.

>> I'm part of the PMC board. Oh, sweet. Awesome. What is like the mixture of attendees then? You would know all about them.

>> Okay, so I would say that there are really young aspiring PMs. Yeah.

>> Then there are PMs that are trying to pivot into different industries. Okay.

>> And then less of the more advanced PMs. So there kind of like three people like the segments, but you're probably going to get a lot of the aspiring PM.

>> Aspiring. Okay.

>> But people want the juicy stuff. So if you have any like hard concrete like um example Yeah. um they would really appreciate. We've been hearing a lot of feedback that the CPO panel is like really high level and they're like, "Oh, like we want to dig in a little bit deeper." So,

>> yeah,

>> makes sense. I'll try to make it

>> good for aspiring.

>> Hi, we actually missed you a little bit in our panel. We had the pan. Well, that would have been fun. Yeah.

>> Where do you where do you do AIBM?

>> Yeah. Yeah. I'm kind of data scientist NPM in the intersection.

>> Cool. What company?

>> Uh it's a semiconductor company. Nova.

>> Oh, cool.

>> So they make tools for manufacturing papers and actually measuring it.

>> So there's lot of quantum physics. So mostly in the line of physics based machine learning to kind of combine AI with the with the physical equations and help the customers like TSMC, Intel and everyone to Make better wafers or faster.

>> That sounds cool. Nice. Are you in the Bay Area?

>> Yeah. Yeah, I live in.

>> We'll see if we can do it.

>> Trying to plan bigger guests, bigger episodes,

>> better topic. Yeah. It's like I used to have one business a year ago. Now I have four. Newsletter podcast cohort sponsorship. Yeah. Heat.

All right, guys. I hope you guys enjoyed those behind the scenes of what it's like to speak at a conference. If you want more of these videos, comment down below. Like and subscribe, and I'll see you in the next one. I hope you enjoyed that episode. Do check out my bundle at bundle.ashg.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobin, Linear, Reforge, Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.