Transcription
How has AI changed experimentation?
Yeah, AI has been the biggest driver of change in experimentation I've seen in all my career. I can't believe more people aren't using based experimentation today.
Fred de Toddaro is one of the world's most knowledgeable people on AI and experimentation. What company do you think is doing experimentation best?
I'm not going to choose the usual one like Netflix, Airbnb, Microsoft and so on. I'm thinking about, you know, a company called Booking.com.
What are the biggest mistakes PMs make when it comes to experimentation?
I think the the top three I I hear the most about in it will slow down.
How would you define the roles of PM, data scientist and AI analysts?
The PM, the data scientist have to work together to make sure that the business context is there, the constraints are known.
How has AI changed experimentation over time?
If you look at the past, I think there has been two big waves in the hand. The machine learning wave which started around 2016 and then the generative AI wave which really kicked off in 2022.
What is the precise difference and what exactly are these bandit algorithms? Talk to me a little bit about content generation. Do you have any stories yourself that your team has used?
So content generation, so you basically you you take content on your website and then you ask the AI to come up with an alternative version of it, right? So that's the goal of content generation.
This is one of the coolest demos out there. Would you be willing to show us how this all works?
Fred, welcome to the podcast.
Hi, thanks for having me.
So, Fred, has AI really changed experimentation much?
Yeah, AI has been the biggest driver of change in experimentation I've seen in my career.
How has AI really changed experimentation?
Yeah, sure. So, may maybe let's let's take a step back for for a second. So, if you look at experimentation, well, it usually comes down to four simple steps. You have an idea, uh, then you make an assumption, if you release that feature in production, it will increase by X metric because of these reasons and these reasons, right? And then you build the experiment, the variations, then you set up the targeting, you know, who who are you going to target with that experiment? Is it all your users or a segment of users? And then the KPIs, which an important, which is an important step as well, how will you measure the success of that experiment? And then you look at the results, right? So it's a simple loop, right? So you iterate as long as you as you learn something with with your experiments.
Okay. So there's four steps and I imagine you're going to map this against how AI affects each of these four steps.
Yeah, exactly. If you look at, you know, the hardest part of experimentation, it has always been the uh, the build phase, you know, right? That's why over the past decade, you know, most of the experimentation tools have been introducing some visual editor to build variations or to let anyone create some some variations. Even dedicated feature experimentation vendors have been introducing them very, very recently. But the truth is, no one really delivered, you know, on that on that premise today. Most tool, most tools still rely, you know, a lot on developers. They're already busy, you know, building the next features in your road map. And so, as a result, most teams, you know, do not AB test the majority of what they deliver to their users and they run meetings to just prioritize, you know, the best experimentation that might be worth building and then they wait one or two sprints to see anything go live.
Yeah, I think this is an important point, right? If we look at the life cycle of experimentation between ideate, build, configure and analyze, build is the bottleneck. That's kind of the tragedy of the commons that we're all facing where very few teams are actually experimenting a high percentage of their releases. And it seems like that's the area where AI could be the most interesting. Before we get into that build phase, I want to talk a little bit about the idea phase. Right? So we have here, you know, it's a three out of five difficulty perceived difficulty for folks, but how powerful can AI be in ideation?
Well, it can be, it can be powerful. Uh, when you look at the history, I think, you know, it has been possible during the generative AI wave, so back in 2022. So you were able to ask an AI to give you some ideas, right? Limited to content generation at the beginning, but then, uh, since the release of the latest model of GPT for, GPT4, well, things have changed a lot. You can really ask the AI to come up with some ideas. You provide a page and then the AI will try to to to come up with some ideas. And so this is where it started, right? Because the AI to some extent has or can assume some context, you know, what's what's your business about, who are you, your users, what is the framework, what you use, you use, you use behind your website, um, and so with all that context, context, then the ideas generation is possible. And so we'll see some of it later during the demonstration as well.
Okay, so I think that this is a really, really powerful set of techniques. Like the way I personally have been coaching people to deploy this if they don't have it built into their experimentation tool is what you want to do is you want to set up the AI with all that context that you mentioned and you want to take advantage of the latest AI feature. So I'll do something like either have a really strong prompt that has context on features we've shipped in the past, the reason we have certain messaging, the metrics that we have seen, or better yet create something like a Claude or a ChatGPT project depending on whatever AI you have access to at your work that loads it up with all of this context so that you can then ask it the right questions and maybe even include like a screenshot of the image. It feels like the better the context you can engineer for the AI, the better the ideas it'll spit out.
Yeah, absolutely. Absolutely.
So, in the build phase, there's obviously a lot that we can build with AI. People are probably familiar with, you know, building with AI idees like Cursor or Wind Surf prototyping with AI prototyping tools like Lovable, Bolt or Replit. But before we get there, we got to stay on this configure step, right? What is the right way to use AI to configure an experimentation experiment that you're going to do? Where should you, the human, be putting an input and where should you be relying on the AI?
Yeah, it's it's a good question. I I think the the role of a product manager is still, you know, absolutely critical there and same for the data analyst or the data scientist. You know, they are they are all working together and better in a way. As a product manager, while your role is still to bring the business context that the AI, you know, does not have. Sometimes you need to deliver that feature for a customer, the AI does not have that context, right? You, you know, the PM comes with all the internal constraints that the AI does not have. Why are we working on that idea? What is the assumption? In the end, and what what success, you know, will will look like as well when you ship that feature? That's the job of a product manager, right? And then the data scientist is still there as well, and I think it, it is kind of the essential human mind in the loop, you know, that that you need, uh, when you need high confidence in the data that you are collecting in your experiments or things like, you know, what is the right models to use? Is it, you know, RAG versus fine-tuned, fine-tuning, tuning model, and so on? So the data scientist is still very, very important, and he's also going to challenge basically the AI's output, right? Is the data, is the data biased as well, to some extent? Are are the results, you know, plausible? That's the job of the data scientist. So if I had to sum up, the AI makes, I would say, the world experimentation process, you know, faster and more accessible to the world organization. For example, the AI can fully determine or identify patterns, you, you know, and suggest some recommendations when an experiment is non-conclusive. You would normally do that, you know, by digging into the data manually or, you know, and it's not possible here behind human scale, there are so many data points to crunch and so on. So the AI is fully capable to do that. But then, you know, the human in the loop is very, very important, both at the product management level and at, I would say, at the data or data scientist level. I think that really sums up very nicely the profiles that each person needs to take, right? So to, if I were to repeat it back, you know, the product manager, they're going to be the person who's taking in all the context and helping to come up with the initial idea where they prove that context into the an AI system. That AI system gives you some initial configuration options. The PM might sanity check that, and then the PM needs to bring in or work with the data scientist or have a process where a data scientist comes in and they're further challenging and hardening the assumptions. And so there's kind of two human in the loop steps, but the PM is really using the AI to make sure that that first version is strong.
Exactly. Exactly. And then the AI can really build that variation for you as as long as you have, you know, very good hypothesis and and some metrics to, you know, on which you will back up your your your measurement there.
And then analysis of features. How would you define the roles of PM, data scientist, and AI analyst?
Yeah. So I mean, like I said, I think the AI is today fully capable to give you some summary of, you know, what happened, u when that experiment was running for your users. It is also able to, you know, identify patterns like, you know, if your experiment is, you know, not providing what you you are expected, sometimes by, you know, uh, uh, bringing down the data by, you know, breakdown in the data by, I don't know, device type of users and so on, you will spot out some opportunities to do better. And that AI can do that for you, you don't have to do that anymore. There are some, you know, many experimentation vendors who have released such capabilities as part of their offering because there is nothing very complex there. And the human at that stage, uh, you know, does not have to be, uh, basically again, the the the PM, the data scientist have to work together to make sure that the business context is there, the constraints are known, and and that the metrics on which you will, you know, measure the success of that experiment are are clearly defined, and, you know, measurable as well. If you run an experiment that you are not able to measure the success, well, that's the point, right? And the AI will not help you. So again, both the PM and the data scientist and the AI now are working, you know, together and and better in in a way. So if we had to summarize here, in the ID8 phase, sure AI is doing some, but it's really just a co-pilot. In the building phase, AI is getting to the phase, and we're seeing a future where AI can really dramatically do most of the build for you. And in the configure phase, again, AI can really dramatically do most of it with some human in the loop review.
Yes.
And then in the analysis phase, again, it can do most of the basics as long as you have predefined the right metrics, the right northstar, the right guardrails, the right secondary metrics. It can go ahead and look at those. And so it's really that ideate phase where humans need to stay super connected, but all all these other steps, it feels like AI is going to be taking over most of the workload.
Yes. Yes. Exactly. Exactly.
So for PMs, product sense here and becoming a great experimenter and coming up with great ideas is really going to be the most important thing to becoming a great experimenter.
Exactly. Exactly. Yeah, I mean, if you look at what a product manager, you know, does in the end, it identifies, you know, features to improve the user experience, but also, you know, the business, right? And so I don't think that what's preventing today PM to experiment is the lack of ideas. Usually, they have already a long backlog of ideas to test there. For sure, AI can come and validate some of these ideas, you know, you provide some context to that, and then the AI can tell you, yes, this idea has a high chance to, uh, to succeed, especially because when AI is fully connected to your, I call it the experimentation knowledge base, because maybe you have been running, you know, tons of experiments by the past and so on, but hey, I can tell you, you know, this thing has been tested already by the past, here are the results, and maybe you did not know it because another team had run that run that that experiment already by the. And so the AI can also bring some context from history. I like to call it UX memory in a way, because you come, you come up with an idea as a product manager, and then the AI will pop up there saying, this thing has already been tested and here are the results that were, you know, discovered at that time. Do you want to go and proceed with that idea because maybe you can test it again? Uh, your users may have evolved, you have new behavior, your website has changed, and so on. So that's something really, really important that the AI can come as well with UX memory, which is very, very important.
Today's episode is brought to you by Jira Product Discovery. If you're like most product managers, you're probably in Jira tracking tickets and managing the backlog. But what about everything that happens before delivery? Jira Product Discovery helps you move your discovery, prioritization, and even road mapping work out of spreadsheets and into a purpose-built tool designed for product teams. Capture insights, prioritize what matters, and create road maps you can easily tailor for any audience. And because it's built to work with Jira, everything stays connected from idea to delivery. Used by product teams at Canva, Deliveroo, and even The Economist. Check out why and try it for free today at atlassian.com/product-discovery. That's a t l a s s i a n.com/product-discovery. Jira Product Discovery. Build the right thing.
Today's episode is brought to you by the AIPM certification on Maven run by McDad Jaffer, who is a product leader at OpenAI. This is not your typical course. It's 8 weeks of live cohort-based learning with a leader at one of the top companies in tech. OpenAI just doesn't stop shipping, and this is your chance to learn how. Run along with product faculty and Mo Ali. The course has a 4.9 rating with 133 reviews. Former students come from companies like OpenAI, Shopify, Stripe, Google, and Meta. The best part, your company can probably cover the cost. So, if you want to get $500 off, use my code AA25 and head to maven.com/product-faculty. That's mavn.com/pect-fac.
I want to dive into that in a little bit in a second. But before we get there, let's take the historical point of view on this. How would you break this down historically? How has AI changed experimentation over time?
Yeah, if if you look at the past, I I think there there has been two big waves in the end. The machine learning wave which started around 2016. So we, um, started to see some machine learning, you know, algorithm coming coming out, and then the generative AI wave which really kicked off in 2022. And so if you look at the, uh, machine learning phase, I think AI helped, you know, remove some key bottlenecks like, you know, finding the right users to target with AI targeting, for instance, or allocating traffic to the best variant. And we saw multi-bandit, you know, contextual bandit, you know, being released there. And analyzing results as well. You don't have to, uh, to use generative AI to analyze, to some extent, some results. So machine learning can do that as well for for you. And so they have all been designed to some extent to help teams optimize, I would say, faster, uh, and, you know, adapt in real time, right? U, and then there has been the generative AI wave, and that changed, you know, everything there. We saw the rise of content generation, you know, to quickly create some, uh, copy variations because at the beginning, I think models were, you know, only good at this, to be to be honest. So many, many vendors, you know, outside of the experimentation world started to include content generation. If you go to Notion, you had, you know, generate that content for me. Uh, I want something more fun, blah, blah, blah. So you, you, you, you started to s to to see this this kind of, uh, of, you know, features being released in any tool. Then AI assistant as well started to, uh, to pop up, you know, often RAG based, the help, you know, answer any question, interpret results, or support users, you know, inside inside the platform. So that was, you know, one year later. And more recently, uh, especially at the at the end of 2023, things are still accelerating so much. I mean, uh, with GPT4, it was a new paring and new model, you know, like again, GPT4 changed what people thought, you know, even was that was even possible. Uh, and suddenly Vibe coding became mainstream, like nine months ago, Vibe coding became mainstream there. And I mean, if you look at experimentation, uh, it change, it changes everything as well, because it means that you can, you can, you can turn any idea into a running experiment just by prompting an AI. And that's why, you know, at Cameo, we released the prompt-based experimentation, so that anyone in the organization, you know, HR, finance, product manager, could directly come up with an idea. I want to improve that there, you provide that prompt to an AI, and then the AI will come up with, you know, variation and an experiment that you can directly test in minutes, and not anymore in, you know, sprint or two sprints, you know, depending on your organization velocity.
So let's take a step by step on some of these for people now that we got the broad stroke of how things go. Right. In the beginning, we use AI to improve targeting. Can you say a little bit more about this? Is this about AI selectively showing which person gets which experience or how do you use AI correctly in targeting?
Yeah. So AI targeting, so the main purpose of AI targeting is to predict, you know, what you will do, right? So, let's say you, uh, browse a website, the AI is fully capable to, uh, give you a score, which is a kind of, you know, conversion intent, like what is your likelihood to convert? And so with AI targeting, we, uh, kind of try to replicate what the sales do does, you know, when someone enters a shop, good sales will understand if you are in, you know, about to buy or not, depending on how you are dressed, you know, how you move in the shop, what kind of products you look at, and so on. And the sales will, you know, score you. And so we try to reproduce that behavior, uh, on the web. So as soon as a visitor lands on a website and depending on what you do, what you consult, and so on, the AI will score you. And then customers can use that, you know, directly to target the right users instead of, you know, creating a segment manually. Uh, you will just use the score which is provided by the AI. So it changes everything in terms of, you know, at the configure, at the configure stage, where you set up the segment you want to target with that experiment. Well, you don't do that anymore. You just use the score from the AI. It's very similar to multi-bandit or contextual bandit, but the the role of the purpose is a little bit different.
I think this is actually really powerful and I want people to look at the left side of the screen. I want people to look at these older things because these are actually probably the most mature and in my opinion probably really strong to just use in production right? So if I'm understanding this correctly right, this is these old school machine learning models which are essentially really strong gradient descent algorithms, really strong regression models which are looking at the user behavior to say okay, this is the user that maybe is very high intent to turn off, so maybe we should give them a higher coupon to keep them here or give them something to keep them here, or maybe they're looking at their beaver and saying this is a user who is almost ready to purchase, so maybe we should just help them get on to their purchase right away. So it's really about in sort of real time based on the user behavior, you can rank your users and you can show them personalized experiences.
Absolutely. Absolutely. Many customers have been using that for, you know, showing, uh, coupon to the right users. So I'm sure, you know, every time you visit a website, you always have that pop-up that comes there, you know, providing you 10% straight away. Well, instead of doing that, maybe you should, you know, keep that 10% there and provide it to, you know, the users who have, you know, that that that are that that are in need of that percentage of discount to buy, you know, to to go to the next step and buy. So that is the main, the main goal of AI targeting.
Love it. And you said it's adjacent and similar to multi-arm bandit and contextual bandit. I feel like I want to really tease this out for everyone listening. What is what is the precise difference and what exactly are these bandit algorithms?
Yeah, sure. So if if you look at at a at a standard AB test, usually your goal is to know if, you know, var, you know, actually improves a specific metric. And so you, you split your traffic equally and when you wait, you know, a sample size, uh, and then you have enough, you know, data, you can make your decision, right? With multi, the the focus is on performance. Uh, so the AI will gradually allocate, you know, more traffic to the, uh, I would say, the best performing variant based on the the data you have at at that time, right? But there is a trade-off. I mean, I like multi-arm bandit there, but there is a trade-off. You lose some accuracy in the data because the AI starts, you know, making some decisions based on a smaller sample, which is fine, I mean, to some extent, in some in some use cases, it makes a lot of sense, like in the media, where you need usually to know sometimes within, you know, a day, which article headline drives more, I would say, clicks or engagement. So you don't care about, don't care that much about the accuracy. Your goal is to make sure that you push the best headline to all your users. So that's multi-bandit. Contextual bandit, on the other end, is a little bit different. You have a bunch of variations, like let's say 10, 100 of variations, and your goal is to push, you know, variation one to the to the group of users that prefer, uh, version one, and you push version two to the segment of users that that are in love with that variation. So your goal is to learn and adapt. So it's still based on machine learning algorithm. Uh, and so the goal is really to push the the best variant to each individual user user.
So it sounds like if you have something highly time-sensitive and you're okay with potentially slightly less confidence in the results, go for one of these multi-arm contextual bandit type things.
Yes.
And within those, can you just tease out really carefully so that I understand it? What's the difference between multi-armed and contextual?
What's the difference? Right. You said between multi-armed bandit and a contextual bandit.
Contextual bandit. So multi-arm bandit, your goal is to push the the best variant to all users, right? With contextual bandit, you push to each user the variation that you know is the most performing for them. So in the end, it's like, you know, having a website which is fully personalized. You know, every user, every user will not see exactly the same offering, uh, the same content. So that's a highly advanced. I'd like to say it's it's more tailored for personalization, where your goal again is to adapt the content based on each user preference. In Ven.
So, when should you be using which? Like which industries, which types of organizations should have the head space of using one or the other?
I think media, as I said, for for multi-bandit makes a lot of sense because usually, you know, you don't want to run an experiment on the headlines or article titles. Your goal is to, you know, go with the best one as soon as possible. Contextual bandit appeals to any kind of industry. Uh, if you have, you know, let's say coupons, for instance, you have 10%, 20%, and 30%. Well, you let the AI push the right percentage depending on each user. So that's one usage of contextual bandit as well. Or if you have, you know, uh, multiple homepages with different contents and so on, and you, your goal is not to AB test or to know which homepage is the best, your goal is to push the best homepage to each individual user. So you are really, you know, in the in the hyper-personalization way of seeing experimentation here.
It sounds like probably your e-commerce and those types of things are doing a lot more contextual bandit, where they have a lot more users, the ability to personalize, maybe your social media, and probably more your B2B type of things are using more multi-armed, where they just have one experience that they can refer to. Is that a high level fair, or is there some what is the nuance to that?
Yeah, what that's for sure is that you need a lot of traffic for both, usually, because again, as you need to make decisions very fast, you need a lot of traffic, right? If you have less than 100k, I would say, of visitors, you know, per month, maybe, you know, that's not the best approach there. But when you look at media, they have millions of, you know, visitors every month, so that's not a problem at all, right? And for contextual bandit, as you know, the the goal again is to learn and act based on that learning, while you need to learn a lot of behaviors before, you know, starting to push, you know, variation one to that user or variation two to that user. So it appeals to use to customers with with, you know, some significant amount of traffic in the end.
All right, and then let's marinate a little bit on this opportunity detection. This was happening right at the end of the machine learning wave. What does this enable you to do in your experimentation?
Yeah, yeah. So the the reason we, uh, we've built or we've seen this this kind of feature is because if you look at the broad statistic of experimentation, well, less than 20% of them are successful. Meaning that when you ship that feature, well, 20% of the time, it will increase the metric that you want to improve with that experiment. And so most of the time, when that that happens, you see a lot of data analysts, you know, trying to to drill down into the data to understand why it happens and to to see if there is an opportunity to do better. Because in the end, maybe that variation does not work for, you know, all of your users. But maybe if you, you know, drill down, you start to understand that maybe that feature works well for mobile devices users. And so instead of doing that manually, AI opportunityation does that automatically. As soon as an experiment, you know, is non-conclusive or it show it is showing a downlift on a metric, then that AI will start crunching the data and, you know, you know, point out some opportunities to do better, like again, this this variation is performing well on, you know, mobile devices. Maybe if you personalize, you know, the experience for mobile device users and you show that variation, you will increase, you know, the performance of your website and ultimately the revenue of your website. So that's the main goal of AI opportunity detection. It's trying to to spot out some opportunities to do better.
Yeah. If 80% of what you're shipping isn't going to hit your metrics, you want to make sure that you're still learning something from that 80%. And so this is really going to help you in that step. Before we dive deeper, let's talk about something every PM faces. Getting alignment on product decisions. You know that feeling when you're trying to explain a user flow to engineering or justify a design choice to leadership, and you're just describing it with your hands. That's where Mobin comes in. Mobin is the world's largest library of real-world mobile and web app designs from industry-leading apps like Airbnb, Uber, and Pinterest. Instead of spending hours taking screenshots or hunting for inspiration, you can instantly find exactly how successful products handle onboarding, pay walls, checkout flows, whatever you're facing. Over 1.7 million product builders use Mobin to benchmark against best-in-class products and show their teams proven solutions. Whether you need to convince stakeholders there's a better way to handle user activation or research how top apps approach feature discovery, Mobin gives you the visual proof to back up your product decisions. Check out mobin.com/acos. That's m o bbb i n.com/ a kh and get 20% off your first year.
Today's episode is brought to you by Maven. The problem with most courses online like Udemy is there's no live component and the instructors aren't experts in their fields. They're professors. At Maven, you get direct live access to experts and operators from the world's best tech companies. You can't get that access anywhere else in any university. And you usually can't find them on YouTube either. I've featured so many of Maven's experts in the newsletter and podcast for that reason. To help you out, I've put together a collection of courses I recommend at maven.com/x/acos. This includes courses like AI prototyping for PMs, product sense for PMs, and getting an AIPM certification. Visit it now at mavve.com/x ak.
Talk to me a little bit about content generation. I'm curious. Do you have any stories yourself that your team has used where AI content generation has led to a win?
Yeah, we see we see that a lot. So, so content generation, so basically you you take, you know, content on your website and then you ask the AI to come up with an alternative version of it, right? So that's the goal of content generation. So we've seen users, you know, putting that kind of experiment when they try to, uh, they create some pop-ups for for instance, or some messaging for banners and so on. They want to make sure that they have the best, you know, performing content, and so they use AI for that. They will generate a content there to make sure that again, they increase the engagement. Uh, because sometimes, you know, uh, you come up with a version and you like to, you know, engineer that with a prompt, you're asking, you know, give me a better version for that goal. I'm sure everybody is doing it today, even, you know, myself, I'm doing it every day. I go and I said, okay, I'm doing a podcast with Hakash, can you provide me, you know, the best thing I should say there? And so I provide my content and then I generate that. I have a, hey, I would generate something which is appealing for our podcast.
Love it. Talk to me about RAG assist.
Yeah, RAG assist. So this is the, uh, the era of, uh, chatbot, basically. Uh, so we, um, we have that in our our tool today. So the idea is to, you know, have an AI that, you know, users can interact with to get answers to anything like, you know, I have that experiment there, tell me what is the best, I don't know, statenine to use for that experiment. We know that it is increasingly complicated. You have, you know, frequentist, Bayesian, and sequential testing, and so on. So users are like, okay, which one should I choose? And so you can, you can go and and talk with that AI there. The AI will ask some context, like, okay, what is that experiment about, blah, blah, blah, and then the AI will tell you, for that experiment there, you should use Cupid, for instance, because I know that you have some some some history of data on the metric you want to improve. If you use that, you can, you can, you know, uh, you know, decrease the duration of of your experiment by maybe two weeks. So that's the main goal of of AI assist. We've seen also that kind of feature being very useful for generating some, uh, some code, some, some, for instance, some of our customers, um, do put in place some feature flags. Well, it's, it's, it's very annoying to look at the documentation because you have a lot of methods, uh, to look at and so on. And so you can go and talk with at AI and say, okay, I have this feature there, I want to release it behind the feature flags. I want to variableize some some behavior of a feature. Well, provide me with a with a code. And so you can, you can directly go and and and and interact with that AI that will generate, you know, some steps to put in place, uh, you know, by using some SDKs so that you can, you can go with with your feature flags as as as soon as possible. So that's
This is huge. I feel like this is like the co-pilot, right? This is all about, especially if it's built into your experimentation platform, you being able to converse with it.
Now, you dropped two pieces of experimentation knowledge there that people's ears probably perked up on, and I want to at least give them the quick overview on. So, you talked about frequentist and Bayesian statistics. So, can you unpack that for us first?
Yeah, sure. So there there there are usually two, you know, methods to, uh, to kind of analyze the experiment. So, um, we look at, you know, the statistical confidence of the results you are looking at, because, you know, an experiment is is basically, um, statistic experiment. You, you take a decision based on the data that you have at one point of time. And so you can use a frequentist approach, which is, which is usually what most of, you know, the users will use. This is a default, you know, statistical engine there. Then there is a Bayesian, which gives you a probability, you know, that variation has 80% chance of being, you know, better than the original or the control. So, you know, it's like, I think two two schools here, some will love Bayesian, some will love frequentist. What we see is that usually 90% of the market, you know, the go with frequentist there. And then you have, uh, Cupid, which is another statistical method, which is very powerful, especially if you have a lot of returning users in the B2B, for instance, industry, that's very powerful because it takes some data as an input, so that when you start your experiment, you actually have two weeks of your metrics already available. And so ultimately, it means that you can get some results much faster than if you start, you know, from scratch. So they are all designed, you know, to answer a specific case, but in the end, your goal is to make sure that you are taking the decision with the best accuracy, and so all these statistical engines are, you know, very powerful and very known by, uh, by by our customers and users in the experimentation industry. And it's Cupid, C U P I D, right?
Yeah.
All right. So, one of the coolest things that you talked about as you were previewing the Gen AI wave was how the Vibe coding has really changed experimentation. Can you unpack this further? What has Vibe coding enabled with experimentation?
Yeah, it's it's a good question. So what what Vibe coding really, you know, has has unlocked is, uh, this ability to take a product idea from a concept to a live prototype in just a few hours, right? So it helps you build things, you know, faster. And I love it. I mean, we are, you, I think all using, you know, Vibe coding in in the product management ecosystem today, like Lovable or Bolt, you, you know, you have your idea and you get a prototype that you can then test with your users. But as as a product manager, my my goal isn't to just, you know, build and deliver faster, it's to learn faster, right? You want to make sure that when you ship that feature, you are able to learn, you know, some insight so that you sure that, you know, that feature is actually being used and increases, you know, I would say the user experience, but also your the business outcomes, which are also very, very important there. And so that's important because that way I can kill features, I can drop the features that don't bring any value to to my users or to the business. So that's what Vibe coding unlocked. You, you go from an idea to a prototype that you can test with all your users. But the real question to me isn't, can you build it? It's really, should you, should you build it, right? And so that's why experimentation is very important because experimentation answers that question there. Should you build that feature or not? And so that Vibe coding won't help that much with with that. Well, for sure, you can, you know, test this prototype very quickly. And again, it's it's outstanding and so on. So you can test that with a few users, and that's very great. But the truth is, you, you will only really know, and I want to say that loud here, if it works, you will only know if you know it works when that feature will be going live, you know, to all your users, because I've seen it so many times. I mean, users say they want something, they say that, you know, they like it. When you show a prototype, and then once you, uh, launch it to to everybody, you launch it broadly, well, nothing happens, right? And I'm sure you, you've got that as well as a as a PM by the past.
And so prompt-based experimentation sits right at the crossroad of Vibe coding and experimentation. Well, we actually call it iterative Vibe experimenting because you can, you can read the same concept in the end. You, uh, you can test ideas in minutes just by prompting the the AI, and you do it right from your website or your web application. You don't do it, you know, from a blank page. You go to your website, you prompt your idea, then the AI will generate, you know, that feature or that variation, and then you can test it directly with all your users. So you, you, you get, you get basically the best of of both worlds here. The speed and creativity, uh, that is provided by, I would say, Vibe coding with prototyping, you know, directly from your with your web apps, and I would say this, the structure or or maybe the discipline of real testing to make the right product calls, right? You are sure that when you get the data about that experiment, you know, or you do not know if that feature will improve your metric. And so this is very, very important.
For my money, this is one of the coolest demos out there. Would you be willing to show us how this all works?
Yeah, sure. So, let's say I want to I want to create an experiment on this shop here. So, it is, you know, classic shop. You have some products there. I have a catalog. I have some some products. I can see I have some pagination there. I can then click there and see the product and have to cart the product. Right? So what I can do there is directly prompt my ideas by using Cameoon. So I'm going to, uh, do that directly from our product. So the way it works here, you create a new experiment. Then you have a choice here by prompt. You give it a name, live. I will select my website, my website here. Then a page I want to optimize, and then you create it. Then your website will open directly there. And so you have, very similarly to what you, you can, you can see with GPT, you have on the right side, uh, you know, a UI that allows you to prompt any changes, right? So what I can do here, so let's say I want, for instance, to change the sort here, and instead of having heat ordered alphabetically, I want to order it, you know, price high to low or price low to high. So what I can do here is that I can directly prompt that with a simple prompt, which would be this one here. So I can say, okay, change the default sorting option to price low to high. And then you hit that button there, and then the AI will start, you know, uh, you know, extracting all the context, you know, he gets from that page here. So what is the framework behind, you know, where is the sorting, you know, being defined there? As you can see here, I haven't selecting anything. I'm not saying, you know, change the default sorting option to price low to high. Uh, the sort is there, uh, here are the default options which are currently available. I'm not providing anything there because the AI is fully capable to understand what I want to do, to localize, you know, where the the sorting is being done on the website, and then, you know, it is starting to share, you know, what it, the AI is going to do. So where is the target element there? So you can see that, you know, the element, so he has found that element there, then what, you know, the AI has to do, what kind of, you know, behavioral changes it has to put in place, the JavaScript, because that thing will generate in the hands, you know, some CSS, some JavaScript, depending on your prompt, the CSS impact, accessibility will be there as well. So the hair is not only, you know, building that blindly, it is going to do every check, like, okay, is this thing also available on mobile? I know that, you know, it has to be accessible as well. I don't have to break that when I will generate the code. So all that context is, you know, taken from the website. So you can see that the JavaScript is, you know, has been has started to be to be generated there, and maybe, but I'm not even sure here, some CSS will be provided. No, no visual changes required. And so you can see there that, you know, the price sorting has been updated. You know, price.
And this was totally live, folks, like that just happened in 2 minutes. So it went through all those systematic steps. Seems like it very much reminded me of like a chain of thought reasoning model where it showed you, okay, these are the five things we're going to do. It went ahead and did those correctly, and boom, like this is Vibe experimenting.
Exactly. So it's straight from the website, you know, it's not a prototype which is created. I can ship that that experiment straight away. I can go there and I can say, okay, I want to finalize that, or I can even simulate. So it's like generating a preview that I can then share with, you know, other stakeholders. But if I want to to to ship that feature live, I just have to hit that finalize button there. And then.
Seems like amazing for a small team, right? Where a PM might be empowered to go ahead and ship those changes. What is the right workflow at a bigger company where you have designers and engineers, how should you stage gate this properly?
Yeah, I think I mean, what's important here is that, you know, execution is definitely much faster. As you can see here, in two minutes, boom, I have something that may have taken maybe three days to build, because I would have to provide a mockup to, you know, I would have to to ask a designer to create a mockup. Then that mockup is going to be.
You know, uh, delivered to engineers with, you know, specification documents, and then it lands, you know, in a sprint. If you are lucky that the sprint hasn't started, and it's good, otherwise, you will have to wait that sprint to be to be ended. And so you end up waiting, you know, three to four weeks before that experiment goes live. Here, two minutes, I have that experiment live. And so, you know, execution is definitely much faster, as I said, with PBX. But that does that does not mean that you, you know, you can break everything and you can skip collaboration there. What I think works best here is keeping the same checkpoints in hand that you, you would have in a normal product process, but you are just keeping them lighter and much faster. Before launching, I can, you know, generate that simulation link here, so you have that CTA there, which I will be able to, uh, to provide to all stakeholders. The designer can check if the variation, you know, fits the brand and the UIX. If you are, you know, shipping something, we, we'll see, you know, some advanced experiments maybe later. But if you are shipping something, you know, uh, with significant changes and so on, the designer can check everything: the brand, the UI, the style. Uh, engineers can jump in and directly look at the code. So we haven't seen that in the previous screen, but I can, I can click on the JavaScript section, and then I see the code. And so the engineers can look at what has been generated by the AI. And the data team here can directly, you know, look at this part here, when they can see, okay, what are the goals? What are we going to measure? Is it add to cart? Is it the, uh, the the number of orders? So they can directly look at the metrics which, which are there, and they can say, okay, this metric is good, or this one will be missing. Maybe you, you need to configure that as well. So that's, that's really important, uh, there. Uh, and so, as we've seen, the great thing here is, you know, everything is built directly on your website. So you can directly test everything. You can check, you know, do the reviews directly based on the real thing, uh, not on specification or or mockups. You will see, the designer will see, the engineer will see exactly what your users will see when the experiment, you know, goes live into production. So that makes a total change here.
>> And how many times have we seen a huge disconnect between the PRD, the Figma, and which actually gets shipped to production? This is really allowing you access to, okay, this is what we're about to ship to production. What changes do we need to make from a design point of view or from an engineering scalability point of view? And I think that the AI is super powerful. What else can it do besides text prompts? Can it take in sketches?
>> Exactly. So, let me show you, uh, that. So, I'm going to open, uh, an experiment which I've created just before here. So, if you look at this experiment there, so there are some some tools as well. So if I want to start from a mockup, I can directly import that mockup there. So let's say you have a new layout with two products per row instead of four. You are adding a dropdown and so on. You can directly, you know, upload that mockup there and say to the AI, you know, build that variation for me. And again, you don't have to provide a lot of context there as well. Uh, you can sketch as well. Uh, so, you know, you can use the sketch feature to, uh, I don't know, build some, uh, some pop-ups if you want. Uh, so, you know, you can do everything there. So you can say, I want an image. Sorry, because I'm very bad at drawing.
>> Better than me.
>> Very hard with mouse there. So you can put it.
>> Impossible.
>> Exactly. And then you said, okay, build, you know, that, uh, newsletter popup. So, of course, you need to provide a little bit of context there. You know, what is the purpose of that of that, um, experiment there. If you don't do that, the AI will come up with some questions as well. You know, do, do you want to generate an image for me? I can maybe show you one that I've built here so that you can see the output. All right. So you can see here. So same story. Use this sketch to build a newsletter popup with a left column. And then the AI, you know, is coming back to me saying, okay, in order to complete this task, you need to provide, you know, the missing information. What kind of image do you want to do? So I said, generate one for me. I want, you know, a basketball player wearing shoes. So the AI will provide me, you know, two images. I can choose one, you know, with a white background or black background. And then same story, the reasoning will be, you know, shared with you. And then you end up with some JavaScript there and some CSS. And then you have, you know, something there. So, sorry, because there is a sketch there on top of that. So that, that's what the AI built here. So from, from that prompt and from that sketch there. So you can see that I draw some lines there. I understood that there is some some inputs, uh, which, you know, will allow the user to, uh, to, to, um, to, to share the email and so on. I can go from a mockup as well. So same story here, I've probably a mockup, and then that's the output there. So the mockup add more information like why should you join, you know, that newsletter. So same story, it has be the wall be over there. So you can do, you know, you can create some elements but that, but that, that do not exist, like pop-ups, banners. You can completely change the layout. So as soon as it is able to do it, you know, front end, front end side, you can build some onboarding flows. I know that, you know, product managers love as well to put some, uh, some flows to point out some features. You can also do that very graphically with, with that, that AI there. So it can build everything.
>> That's amazing. Right. This is the power of what you've been playing with in your AI prototyping tools, but it's actually happening in context in your experimentation system. It is configuring the experiment for you. It is even coming up with your hypothesized variables. All the steps we've talked about, ideate, build, configure, analyze. It can help you with all four of those. But interestingly, right, that build one that we talked about, that one that really is causing all the troubles where people are too reliant on developers, it is just completely obliterating that step.
>> Exactly. You can see here that last example maybe where I have I have asked the AI to, uh, add some quick view and some add to cart, you know, actions directly from the, from the, uh, product listing. So these features were not there. So the AI has been adding them there. So I can have access to the quick view. I can add that to that that product to the cart. I have I have asked the AI to, you know, add a little notification as well. And then you can iterate, you know, by prompting with the AI. I can say, that notification there, maybe you should have a remainder of the product that has been added to the cart, you know, if it makes sense. So you can really hit iterate with that AI. It's just not one prompt. You can, you can add many prompts until you, uh, you know, you come up with the best variation that you want to ship with your, your features with your users. Sorry. All right. So that about covers this amazing change that we've seen in AI affecting experimentation across all features. Now, can you break down for us the second side of this topic? How do we measure and experiment AI features themselves?
>> Yeah, I think there, there are there are some key indicators that you, you have to look at. Well, the first one is, you know, the usage or the adoption. Are people, you know, using your, your feature? The second, the second one is the, uh, the outcome. Um, you know, is that feature, you know, fixing the problem, you know, users came for, for, or is it helping them, you know, complete more tasks faster? You know, they activate more experiments and so on. And, and the last one is, is the, uh, the experience. Once the feature is live, well, you want to ask users, you know, the feedback about it. And usually, there, we, we see, you know, in AI features, you know, the quick thumbs up, down that you see everywhere because this is one of the only ways to get, you know, feedbacks about, you know, is the answer provided by the AI accurate, uh, or is it missing some context there? So very, very important to to include them as well. Uh, and if we look at, you know, what we are following with prompt-based experimentation, it is not just about usage. Uh, it's about how quickly our users can go from an idea to a running. Right? Because in the end, this is ultimately the the main goal of experimentation. So we look at things like, you know, how many prompts it takes to create an experiment. But if it's too many, something is half, maybe, you know, uh, the AI is not providing, you know, the piece of code or the variation that the user wants there. The second thing is, you know, how long it takes from the first prompt to an experiment live. Very important as well. If it is faster than any of our methods, that's a big win for us, right? Instead of taking days, maybe you took five minutes to, to put in a partition your experiment. And the last one, which is also very important for us, is how often developers still need to step in. If you look at the past, before prompt-based experimentation, I would say that maybe 80% or more of the experiments needed code. That's the true, the reality of our market. If we drop that number to let's say 20%, which is our target at Cameon, well, we know it is, you know, a massive game changer for the industry because we haven't seen that, you know, any anywhere else, uh, in the last decade. So those are the metrics we, we do follow. And so again, it's not only about usage, it's about the value that we want to generate to our users. Uh, you know, the feature, uh, needs to, uh, allow them to, uh, to run, you know, experiments faster, to get to running faster, and so on.
>> I love this. It's almost structured as like the wise, the five W's, right? Are people using it? Is it solving the problem? Was it helpful or reliable? And then translate that to your experiment like you've done with prompt-based experimentation. I'm curious, what was the Northstar metric you guys looked at and how did you determine the Northstar metric for success?
>> Yeah, it's a good question. So at Cameroon, we look at the number of experiments, you know, which run daily on our platform. So we have been following that for, you know, many years. Uh, we came up with that Northstar metric because it is, you know, simple to, to follow, to measure. We don't have to ask a data scientist to get access to that metric. And the sec, the second reason is that it is, it is very easy to anyone in the organization, right? It's not something very complex, you know, very specific to a product team and so on. The number of experiments that run daily, that's the metrics we look at because we know that, you know, when there is a drop on that metric, it means that there is some churn at Cameroon. It means that we have less customers, you know, creating campaigns every day. Uh, and if it goes up, it means that either, you know, we are increasing the business with new customers, or we have, you know, more and more users creating experiments. And so this is critical for prompt-based experiments because again, our goal is to have more users, you know, creating experiments so that it is not only limited to a team of users, it is not only limited to product teams or to engineering teams. We really want to democratize experiments as part of the organization. And so the more experiments are created, well, the better it is. It means that, you know, prompt-based experimentation is delivering on the promise, which is again, to be able to have an idea to an experiment in minutes and not days or weeks as it is in most organizations today.
>> So I think there's a lot of hidden takeaways there for people listening. The first is the Northstar metric you chose is basically your business's overall Northstar metric. You're not just choosing some random Northstar metric specifically for this AI feature experiment. If it's a big AI swing, you want to just move the whole business. The second thing is that you didn't choose like a revenue-oriented feature. You actually chose a product input metric that you guys can change, but that you know is highly correlated with impacts on your output metrics like revenue. Are those fair takeaways?
>> Exactly. Exactly. Because with that Northstar metric, well, everyone can, you know, play the same rules of the game. If you look at product teams, you know, they can improve features that drive it forward. That's why we, you know, introduced prompt-based experimentation. It's a product initiative as well. And I know that my product, you know, managers can deliver on that and increase that Northstar metric. Customer success as well. Customer success can, you know, help users get more value out of the product, and their goal is to increase the Northstar metric because if they are, if they have happy customers, if they know how to use the product, you know, well, they will ultimately create more experiments. Marketing can also work, you know, in, uh, you know, publishing some benchmarks per industry, per market, so that, you know, our customers can compare with each other. So every, I would say department, divisions in the organization work towards, you know, improving that Northstar metric. It's not just revenue because revenue, well, you, you can't, you can't all add on that metric. It's very hard in the end. But if we, so find an after metric which is highly correlated to the revenue and that can be again, uh, action, we say actionable by any team, uh, that's something that can change the face of an organization there because everybody is moving into the same direction in the end.
>> So that's how you measure kind of a game-changer feature like PBX, it's really changing the way people use your platform. Talk to us a little bit about how you measure RAG systems.
>> Yes. So RAG systems. So again, this is this is what is usually powered behind this, you know, assistant there. So same story. I think it's very important to look at some business-oriented metrics, uh, there, you know, adoption, the the usage, you know, the outcome, the experience. And then I think there are also some, uh, some technical metrics to look at. And at Cameon, we have been looking at them before, you know, shipping that, that feature into production. Uh, and I mean, there are three things we look at at, at Cameon, and I think it can be applied to any organization using a RAG, RAG system there. The accuracy, you know, is the answer really based on some, uh, documentation that you know, do exist, uh, you know, for instance, if someone asks how is traffic, you know, split in a bandit, and if the AI answers, well, traffic is always split evenly across all variants. Well, that's, that's a miss, right? So, that's a bad, that's a bad answer in terms of accuracy. Relevance is very important as well. So, relevance is all about, you know, does it actually answer the user's question? For instance, if I ask something like, you know, what are, uh, ways to target feature flags, or what are the different options there, and the AI will answer saying, you need to create a new experiment. Well, it, it, you know, it's a miss as well. It is not answering the, the, the targeting logic or the targeting part of the experiment creation. And the last one is the context quality. Uh, you know, where are the documents or documentation it used really helpful? Uh, you know, if it pulls some outdated documentation or some, you know, or not the most recent, I would say release notes, well, some, something is off as well there. So these are good signals to me to check if the AI is giving, uh, answers you, you can trust. Okay. Now, I just wanted to to end on something. There are also, you know, if you look at what we've been doing at Cameon, if I can, if I can say something as well there, one, one method which works well because I'm sure people will ask, okay, but how, how do I do that? How do I make sure that, you know, accuracy, relevance, and so on, uh, you know, is good? One method we've seen working at Cameon is to use, you know, an LLM, so a learning language model, as a judge, right? So you have a judge which will judge the performance of your RAG system. So, for instance, here for the response relevance, which is again, how well does the chatbot answer, you know, match the user's question, one method is to to use an LLM which will generate two free new questions based on the answer which is provided by the first LLM. So, for instance, if I say, how do I enable HI targeting in Cameon, and the chatbot answers me, you need to go to the experiment settings, enable the AI targeting to, and save. That's a good answer, right? And so the LLM will, the alternative LLM as a judge, you ask him to generate two free questions based on that answer, like, where do I find the AI targeting setting, or do, how do I activate AI targeting? And so these questions are very similar to the, to the question being asked by the user. So it means that, you know, the relevance is there. And so you can do that, you know, for the, uh, the metrics and the retrieves context as well, context relevance, you know, the scoring. The LLM can score the answer as well. You can do that with the accuracy. That's what works well, uh, today, I think. So it's that combination of looking at faithfulness, relevance, context quality, operationalizing those into metrics for you, and then building a good eval suite, potentially using an LLM as judge to look at things like response and relevance, context relevance, faithfulness. That's the overall recipe for measuring an AI feature. Are there any big mistakes you see teams making while measuring these AI features?
>> Yeah, I guess one of the biggest mistakes is, you know, to stop at usage again, again, because, you know, mo, most of what we see is when, you know, some users, uh, you know, will complain about, you know, the AI feature, will just stop using it, right? So, I mean, usage is one piece of the of a cake here, but you need to look at the outcome and the user experience. And so some of the insights I've been sharing there are pretty useful for, for you as well.
>> Amazing. Before you go, I have an experimentation expert here with us. So, I have a couple questions for you. First, what company do you think is doing experimentation best?
>> Yeah, so I'm not going to choose the usual one like Netflix, Airbnb, Microsoft, and so on. Uh, I'm thinking about, you know, a company called Booking.com, very known as well. And, you know, they have built such a strong, I would say, culture of, of experimentation. You know, nothing goes live into production, whether you are, you know, shipping features on your website, on your application, on your mobile application, they do everything through experimentation. So I think it's a, it's a great model for any company that wants to, uh, to learn fast and build with confidence.
>> And I think they test almost everything, right? So they've really built that experimentation culture from the top.
>> Yes. Exactly. Exactly.
>> And I think that's incredibly rare, but it's incredibly common amongst winning companies.
>> Yeah. Yeah. And my advice, I think, just to, not, you know, scare every people there, I think it's, it's not, don't have, you know, don't, don't you don't have to go, you know, all in one and and test everything from day one. Uh, you know, at Cameon, we have been, you know, going step by step on this topic, even at Cameon, so we are also shipping features, experimenting with them and so on. And I think you have to start simple at the beginning. If you are not doing any experimentation today, we first use, you know, feature flags just to push code, you know, safely into production. At least you have a turn on, off, you can see how it goes into production. Then you introduce what we call progressive rollouts or targeted delivery, where you ship the feature to, let's say, 20% of your users or to a to a segment of users. And then you learn from that, right? So you learn from the usage, uh, you know, are they using that feature? Are they complaining about the, uh, the user experience? And then you improve that feature before releasing it to every single user. And once you're ready, then you start, you know, adding some heavy testing there to make data-driven, you know, decisions at scale. And I mean, I, I do believe that that people have been putting barriers because of the high build, uh, cost. So that's why I think with AI, that thing is going to be to be removed, uh, so that you can hope, hopefully, you know, going to, uh, to push more things behind experiments as well because you are able to ship them, you know, without having to bother your deliver teams.
>> Yeah, AI is a game-changer to becoming a company like Booking. You have no excuse at this point. What are the biggest mistakes PMs make when it comes to experimentation?
>> Yeah, I wouldn't say there, there are mistakes, but there are, there are some common misconceptions there about experimentation. And, you know, I put three of them here, but there, there are others, of course. And I think the, you know, the top three I, I hear the most about is, it will slow down our delivery. The second one is, we don't have enough traffic there. Or, you know, because you do product discovery, well, I'm fine, you know, I, I've learned a lot from product discovery, so why, why should I bother with experimentation? And so none of these objections, for me, are, are, are perfectly true. I mean, experimentation actually helps teams build, you know, better products. There are many studies about it. There is one which is, which I like to talk about, which is one being done by Harvard Business Review, which basically says this, there is a direct correlation between the number of experiments you run annually and your revenue growth. That's a fact, right? So the more experiments, the faster you can grow there. So if we look at all of these objections, there are, I think the first one actually, it speeds things up. It's not slowing, you know, it's not slowing things down there with experimentation. You can release, as I said, early versions of your feature. You can learn from, you know, real users, you know, using that feature, and then you, you improve as you go, right? That way, you get to the right feature faster in the end. So you don't have to work on the, the best feature before shipping it. You ship the first version of it. You iterate with it. You learn, and then you ship the final version to it, to all your users. Like I always say to my own team, you know, fast gets, you know, good, good quicker than good get fast. So the key message here is that you don't not do not wait for the perfect feature. You ship early and you iterate until you reach, you know, your goal. You will learn so much more in a short period of time. But if you don't do experimentation there. The second one, very quickly here. Well, it's, it's always a little bit funny to me because teams are often fine validating features with, you know, just a few users. They have a prototype. They go and show that prototype to 10 users, and then, you know, they think that all their users will love the feature, right? So why, why it is it is a problem with experimentation there. So it doesn't make any sense to, to me there. And the traffic is, is one point. The number of conversions is also very important when you look at, you know, the sample size you need. You need a good number of of visitors, but you also need to look at the metric. The more conversion you have, you know, the fastest you will get some, some, some insights about your experiment. And the last one, product discovery is enough. Um, I would rather say that discovery and experimentation go hand in hand. Uh, you know, discovery gives you early signals. That's a fact. That's why it's very, very important to do some product discovery. Uh, you know, but, you know, experimentation will tell you exactly what, you know, users do when they interact with that feature. So product discovery will tell you what users say they want. Experimentation tells you what they actually do. So very important for me to, uh, to have both so that you, you get the full picture, uh, in production.
>> Amazing. This episode, our complete course on AI experimentation is coming out an episode after our complete course on AI discovery with Teresa Torres. I think it's the perfect combo of content, perfect ending for our episode today. Fred, this has truly been a master class. If people want to find more, if they want to connect with you, where should they go?
>> Well, directly on LinkedIn. Directly on LinkedIn, you know, I will answer, you know, most of the most of the questions I got.
>> Amazing. Thank you so much for your time today.
>> Yeah, it has been a pleasure. Thanks for having me.
>> So, if you want to learn more about how to shift to this way of working, check out our full conversation on Apple or Spotify podcasts. And if you want the actual documents that we showed, the tools and frameworks and public links, be sure to check out my newsletter post with all of the details. Finally, thank you so much for watching. It would really mean a lot if you could make sure you are subscribed on YouTube, following on Apple or Spotify podcasts, and leave us a review on those platforms. That really helps grow the podcast and support our work so that we can do bigger and better productions. I'll see you in the next one.