OpenAI Head of Product Revealed Everything… (AI Agents, GPT-5, How They Build Product)

In the future, PM is going to have to learn how to manage not just humans, but agents. Absolutely. You're going to be interacting with, managing, collaborating with other people. I think that's very much the case with agents. And so, there's going to be a new skill set that people need to develop.

Jake Bril is the head of integrity product at the hottest AI company in the world right now. He works at OpenAI. Before that, he was a director of product at Instacart and a product manager at Meta.

So, you guys just released GPT5. It was a whirlwind launch. What did it feel like inside?

Truly energizing.

What do PMs need to know about agents?

If you look at your average product manager and you said, "What's your AI strategy?" Or, "How are you building an AI first product?" A lot of people wouldn't necessarily have an answer for that. And over the past couple years, it's like, if you're not building a product that has AI fundamentally in its DNA, you're not really keeping up with the future of digital technology.

One of the craziest things I heard about agents recently was that agent internally was manipulating performance data. How do you deal with a world where agents learn to cheat?

It's a hard problem, a very big research problem we're going to keep working on.

You mentioned Slack. I think that was one of the coolest insights that came out about OpenAI in the last few weeks that the company almost entirely runs on Slack.

Yeah, almost entirely. I would say conservatively like 90% of my written communication is in Slack.

If you're a company who's maybe not a model company, but you're trying to create an eval system as robust as OpenAI, what would be your lessons for them?

Well, in a lot of cases, you don't have to build these evals from scratch. There are industry standard evals that you can use.

What does the PM role look like in 5 years?

Really quickly, I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's episode.

Jake Bril is the head of integrity product at the hottest AI company in the world right now. He works at OpenAI. Before that, he was a director of product at Instacart and a product manager at Meta. We have a ton to talk about, especially as it relates to how does Open AI handle negative scenarios within AI. How does OpenAI build product? Jake, welcome to the podcast.

Thank you so much for having me and it's a real thrill to be here.

So, you guys just released GPT5. It was a whirlwind launch. You guys were actually everywhere. I think this was the most marketed launch I saw from you guys. What did it feel like inside?

Truly energizing. You know, we've been working on GPT5 for a very long time. And in the days, weeks, months leading up to the launch, the energy was just palpable. We knew how incredible of a model we had. We knew how much of a game changer it would be for people to experience a reasoning model because most people haven't. And so bringing that power to people, it just felt like we are just one step further in our quest towards AGI and fulfilling our mission. The energy was great the day of the launch, you know, it was in really well done launch event and just seeing people's reaction in Slack, starting to see the graphs, like watching people use this product, it it just felt incredibly fulfilling because this was a lot of really smart people working really hard for a long time and seeing it come to the world, it was just like, okay, we're here. Very exciting.

You mentioned Slack. I think that was one of the coolest insights that came out about OpenAI in the last few weeks that the company almost entirely runs on Slack.

Yeah, almost entirely. I would say conservatively like 90% of my written communication is in Slack.

Wow. And what does it look like? Like is there any special things about your guys' Slack setup? Do you guys have a lot of AI agents in there keeping things updated?

Yeah, you know, it's funny because at Instacart we were really Slack heavy as well and Instacart was fully remote for my last several years as a company and so we relied a ton on Slack there. So when I joined OpenAI, I had something that I could fall back on and so I was used to a company that ran heavily on Slack. In terms of what makes our usage of Slack special or more unique for us, yes, absolutely. We have AI agents in the channels. They can serve any number of purposes. You know, there's there's one that we have that's like a general purpose answer agent, which is really helpful. So, if you have a general Q&A in the channels where that agent exists, you can get questions without one of your co-workers having to see the post and respond to it manually. So, um we very much believe at OpenAI and in in using AI in all parts of our user and enterprise experience. So, yeah, it's it's no different in Slack.

Very interesting. Now, you head up integrity product.

What is Integrity Products role in a big launch like GPD5?

Sure. So there are a couple big areas that were involved in this launch. The first is what you might classically think of when it comes to integrity. You know, preventing bad things from happening, people from doing bad things with our models and our products. And we certainly did a lot of work in the leadup to and following the launch in that space for GP5. In addition to our work on preventing bad things from happening, one thing that of often happens when we have new launches is we see a lot of people try Chatbt for the first time or try our new new models via the API for the first time. And another area that integrity is responsible for is our our identity system. Um, so when we when we release new models like this happened with our image gen model as well as GP5, you see an increase in traffic, people wanting to sign up for the first time, people logging into their accounts. You want to make sure those systems don't fall over and that you your systems are able to take meaningful increases in volume so that they have that they maintain their worldclass uptime and latency because you know it would be a really bad user experience if people try to go use use GPT5 for the first time and they just get an error message. Similarly, our team is responsible for our financial systems and when we launch new models is often the case that that will lead to people converting to paid subscriptions or more people using prepaid API credits to use the model. And so similarly, we want to make sure that our payment systems are are up, they're running, they're seamless, you have good good authorization rates, but also that you don't see an influx of people with stolen credit cards fraudulently fraudulently using our models and our products. So, a lot of work behind the scenes to make sure that GP5 shines in in all of our different products.

Fun fact, Amazon put two holds on my account as I ordered the equipment for this in-person podcast today. So, it's not just on the credit card side. Even the person who's making a purchase, an OpenAI or an Amazon, is monitoring those accounts to see are these suspicious transactions and flagging those appropriately.

Exactly.

Very interesting. So, if we break down, let's say, let's stick to GPT5 for a second. What are the big streams of work that a company like OpenAI is doing to make sure that it meets your integrity standards? Is there red teaming? Are there certain evals? Are there certain what are the big how would I put it into buckets?

Yeah, absolutely. We do do red teaming not just for new launches but for our products all across the board. And red teaming happens at all stages of product development. when you're training the model, when you're training your production systems to to mitigate harms from happening, we have both manual red teaming as well as automated red teaming. We'll do that at different checkpoints while the model is being trained. We do it with the final version of the model before it goes live. And then we also do red teaming after models go live just to make sure it's like, okay, well, we prevent the known jailbreaks. Are there new jailbreaks that are out in the wild that that are automated systems or or penetration testers are trying to to to find ways to to get around the protections in our model? Um other sorts of work that we do there's a lot of work to make sure that our automated systems have high precision and high recall. you know, we want to make sure that if we're taking some sort of automated action that we're really accurate with that because we take it very seriously if we block a generation or warn someone's account or even at the extreme ban someone like those are very serious interventions and we want to make sure that we hold ourselves to a really high standard that when we take those actions we have really high confidence that we're getting it right. Similarly, we want to make sure that our recall is high so that we don't just we're just not blind to false negatives and and there's all sorts of bad stuff going out there that we're not aware of. Um we also do a lot of making sure that we have robust operational tooling and capacity to make sure that you know we project forward we think x number of things will require manual review. Um, we want to make sure our tools are sufficient so that they um that people have extremely efficient workflows and that um we have enough human power and and bandwidth so that we can get to things in a very timely fashion.

And I think some people might be kind of rolling their eyes at this poem like is all this stuff important but I think it's actually goes to the core of why even open AI was created at the beginning and a lot of the values that you guys have from the beginning of creating this safe AI. Talk to me a little bit about that. How is that interwoven into OpenAI's philosophy and how does that manifest?

Yeah, I mean if you read OpenAI's charter safety is one of the things we talk about very publicly. is part of the reason why I even wanted to join OpenAI in the first place was when I had my first conversations with them. It was very clear that safety was was really important to the company at all stages of model and product development. And I figured if I was going to leave a job that I really loved with people I really cared about, I wanted to do it not just to jump on like a hot AI train. I wanted to do it because I really believe that the company was responsible and took this took that responsibility seriously. So safety and integrity, it's something we talk about throughout the entire development process. Um, and we, you know, the we do also believe in iterative deployment. So there it's not like we're launching models and there's absolutely zero possible way that something could happen that could have a bad outcome. But we talk a lot about what risks are non-negotiable that we have to mitigate before launch. which are the other ones that we want to have systems in place and which are the other ones that's like hey actually it's we we could sit in the room stroking our chins thinking like oh how might someone do something bad but actually at the end of the day it's really helpful to to follow open approach of iterative deployment because once you start rolling things out you can actually see in the real world how people accidentally might misuse your products or professional bad actors might misuse it and then you can very quickly respond and build sophisticated solutions

One of the areas that I imagine this is really important is the recent opensource weighted model that you guys released and I know you guys even delayed that launch a little bit because you said it wasn't ready. How do you really determine like okay this model is safe enough and ready to put out there?

Yeah. Well, a lot so much of that is eval we try to make sure we're we're holding ourselves accountable to objective sources of truth on the relative safety of our models. Um and and there are a number of public eval companies talk about to ensure that their models are safe. Um this could be safe in terms of like is the model deceptive? Is it safe in terms of is it appropriately refusing some of those high-risisk bio prompts for example. Um, we, you know, you you can make vibes based decisions on this sort of stuff, but ultimately eval are really what's going to guide the day and helping you objectively and with data determine if your if your model is safe enough to to release.

If you're an company who's maybe not a model company, but you're trying to create an eval system as robust as OpenAI for whatever product you've created on top of the OpenAI API, what would be your lessons for them? like how how can they build evals that they can trust?

Well, in a lot of cases, you don't have to build these evals from scratch. There are industry standard evals that you can use and we've published a bunch of them. Um, but there are many other excellent Frontier Labs creating safety related evals. And so what I would ultimately say is if you're if you're an earlier stage company, you don't have to reinvent the wheel here. I would really encourage you to just you could even use AI to go do a search on like what are the best safety eval simultaneously there are a lot of open open source open standard like safety models that you can layer in on top of your model or your product at open we've built the moderation API and there are plenty of other great options out there and so I would say you're not starting from scratch and I would really encourage people to make use of of the great open technology that exists out there.

Got it. Cool. So, we're walking back in time through all of your amazing launches. The one before that, you mentioned it earlier, agents.

Yeah.

Feels like I can't go on social media without seeing some post about AI agents. Why is everyone making such a big deal out of agents?

Um, cuz they're the bees. Uh the way I think about this you rewind 3 years ago chat GPT launched people started having their first sort of cons broadscale consumer experience with AI and for those first couple years it's really been what what we call assistance you know you ask a model a question you give it a prompt and you get a response and that's been incredibly powerful I mean the growth of Chachi PT and many other great AI products sort of speaks to the power of AI assistance But where we foresee this technology going is not just question and answer, but rather here's a task. Can you please complete it for me? And that's really where agentic product experiences come in. It's hey, there's something more complex than just like what's the weather today? Uh can you go and take an action on my behalf? And that could h that action could be synchronous or it could be asynchronous and you come back because it's rather complex workflow. Um, and I think what's been so great about the progress that's been made in the field of AI is we've been talking about agentic product experiences for several years, but we're now really starting to bring to bear across chat agent and also agentic products that other companies are building a real way to put tools in people's hands that can't just answer their questions. They can also do things on th those people's behav.

Yeah, I personally am using a lot of those noode agent builders like Lindy, Relay, Make, Zapier. You can chain together like crazy workflows.

So, what do PMs need to know about agents?

The thing I would say is I'm going to go back again to 2022. Um, if you look at your average product manager and you said, uh, what's your AI strategy or how are you building an AI first product? A lot of people wouldn't have necessarily had an answer for that. And um over the past couple years, it's like you're if you're not building a product that has AI fundamentally in its DNA, you're not really keeping up with the future of digital technology. And I think we're sort of at that space again with agentic products. It's, you know, that classic Wayne Gretzky quote, you pay you skate to where the puck is going, not to where the puck is. The puck has been at assistance, the puck is going to agents. And so what I would say is if you're not thinking about how to build products that are agentic in their fundamental nature, you're probably a not maximizing the power of this technology and b you're probably building a product that's going to be obsolete in a shorter time horizon because your competitors are going to be thinking about building products that solve people's problems in a agent agent first fashion.

Today's episode is brought to you by Jira product discovery. If you're like most product managers, you're probably in Jira, tracking tickets, and managing the backlog. But what about everything that happens before delivery? Jira product discovery helps you move your discovery, prioritization, and even road mapping work out of spreadsheets and into a purpose-built tool designed for product teams. Capture insights, prioritize what matters, and create road maps you can easily tailor for any audience. And because it's built to work with Jira, everything stays connected from idea to delivery. Used by product teams at Canva, Deliveroo, and even The Economist. Check out why and try it for free today at atlassian.com/roduct-discovery. That's atlassian.com/product-discovery. Jurrod discovery. Build the right thing.

Today's episode is brought to you by the AIPM certification on Maven run by Mcdad Jaffer who is a product leader at OpenAI. This is not your typical course. It's 8 weeks of live cohort-based learning with the leader at one of the top companies in tech. OpenAI just doesn't stop shipping and this is your chance to learn how. Run along with product faculty and Mo Ali. The course has a 4.9 rating with 133 reviews. Former students come from companies like OpenAI, Shopify, Stripe, Google, and Meta. The best part, your company can probably cover the cost. So, if you want to get $500 off, use my code a AA25 and head to maven.com/rouct-faculty. That's mavn.com/p o du-fac y.

And what does it mean to build in an agent first fashion? Because people are talking about, you know, user experience is going to turn into agentic experience. You need to figure out how your product will be open to AI agents. Like what are the elements that a PM should be thinking through for making their product agent friendly and usable?

Yeah, I one of the first things I would say is so many digital project products are synchronous in their nature like I take an action and a response or something else happens immediately. And if you start thinking, hey, what if I could enable a product experience that is far more complex and it doesn't have to happen immediately. I think that's one of the biggest changes that's happening right now is is a stop boxing yourself in and thinking like someone clicks button and action happens immediately. Rather, it's someone clicks button and something far more complex can happen behind the scenes and you don't have to sit there waiting for a response. the product can come and let you know in the future when that very complex task has happened.

So what are some real life use cases that you're using agents for?

Yeah. Um well uh we're growing a lot at OpenAI and so I spend a lot of my time hiring and interviewing candidates. Um, and I find uh I I find it's been deeply meaningful and helpful to uh to use uh agentic products to help with the recruiting process. you know, I'll say like, "Hey, we need to bring in a designer. Here's sort of the properties. Ideally, they have x years of design experience or they've they've worked uh in, you know, this size company um and they're located in the Bay Area. Please go help and like source some candidates." Really, really strong for that sort of use case. In my personal life, I've been using Agentic products a lot to do longer horizon research. Um, I got this like super gnarly respiratory virus in December and I have been seeing a bunch of of doctors about it and like not varying levels of usefulness from that from those doctor's visits and I've actually found that using um agents to go do some of the research and suggest alternative methodologies for for like helping my lungs get back to 100% has been really helpful for me personally.

Okay. And on the product management side, if I'm just a product manager, what agents should I think about building or using?

Yeah. So, for building, um, I think a agents are really helpful for running market analysis, you know, understanding, you know, if if you're a PM in a given area, like what sort of products exist in that area, what is the general sentiment towards them, what are the what are the needs of of people who are looking at products in that area. I think a uh agents can be really helpful in helping you pull together collateral. You know, I I uh I am not the best at like visual like I'm not a great PowerPoint PM. Um you know, I can tell this story, but like the I don't have a background in consulting. I like I was never the best at at making beautiful slides. There are now some really excellent agentic products that can help you visually tell your story in a way to supplement you know all of your written communication skills. Um I also think you know agentic products you know can help you prototyping a lot. You know, that's one of the things we've been starting to do internally is rather than just writing up a proposal for how something works, you know, just build a prototype of how something could work and you put that in people's hands and it's a great way of illustrating how your product might work without having to um you know go in make mocks start building an initial prototype on the on the software side. It's like a very inexpensive way of bringing your ideas to life.

Yeah. I ask a lot of heads of product on the podcast like what are the big ways your PM team is using AI tools and AI prototyping is always the number one thing they talk about what do you see as the future for PRDs in a world of AI prototyping?

Yeah, it's a great question. So I still think there's a a world for PRDs because in some level like AI technology is making PRD writing so much better. You know if I these days I'm more in a support role so I'm not writing as many PRDs but if I rewind like a year and a half when I was in my earlier days at OpenAI and spending more time writing PRDs and I kept trying to use AI for writing the PRDS and it just like came up short. The form product form factor wasn't right. it didn't sound like my voice. The like amount of context that was necessary like the the PRGs like I ended up just writing a bunch of them and you fast forward to where we are now and I think AI has made PRD development much better. Um, you know at least within OpenAI I think canvas is a much better form factor for collaboratively putting together a longer documentation. I think our models are better creative writing with connectors. you can connect to external data sources to help bring these documents to life. And then lastly, this with memory, these models learn a little bit more about what your writing style is, what other things that you've worked on. They can bring that to bear. So, I think there definitely is a future for PRDs. I think they're going to be AI first. Um, and the value of PRDS is you can do a prototype, but you may be missing a bunch of things in that prototype that like how do I handle, you know, like how do I handle this failure case or how do I um, you know, what is it what is our marketing strategy that we're going to bring with the product when we bring it to market. So, I think there's I think they're going to go hand inand the prototypes and the PRDs. I do think PRDs will be less wordy because you won't have to spend as much time describing, oh, you click on this button and this thing happens. You can just show people, but um I I think I think even a world where people are prototyping more with with our models and our products, I still think there's a place for PRDs.

One of the coolest things that you guys open sourced I believe it's called your model spec which had like a ton of examples I think probably that your team was responsible for of like when user has request this don't provide that information when they say this do that is that more the direction your PRD needs to go in a world of AI products where you are giving the specific examples of what not to cover and how to respond

I think that's a good example I think part of what you are talking about is also covered I eval which right now aren't we we sort of view that as complimentary to the PRD or spec but I I do believe increasingly in the future yes it's going to be describing not just the way the product work but the way the model that's powering the product should work

yeah okay actually can you clear that up for me what should go in the eval and what should go in the PRD

I I think a lot of the PRD is like classically it looks a lot like what we have today like here's the problem that we're solving. Here is the way a product would work. Here's our definitions of success. Here's our target user. And eval are really fundamentally like here are the use cases this product should be good at and here are ways that we can we can test the model to evaluate is it successful at these specific use cases.

Okay. So PMS they typically own the PRD even though they're jointly creating it with the research engineering and design teams. when it comes to evals, you know, who owns that and how do PMs plug into those?

Well, you know, increasingly we're asking PMs to take a really active role in developing evals. And I think as the role of PM changes over time, that's going to be of increasing importance. You know, anyone can write an eval. It doesn't have to be a researcher. It doesn't have to be a software engineer. It doesn't have to be a PM. But we're finding that because the PMs oftentimes are the ones with the clearest vision of of how the product should work in their head, they're very well positioned to write evals because they they have a very strong opinion on here's what the product should be good at and here's what the product should not attempt to do.

I always love to get sidetracked on the tangents about PMs. I want to return back to agents for a second.

Um,

how do AI agents talk to each other?

Yeah, this is a good question and I don't think it's a solved question. I I I know I really enjoyed your your podcast about MCP and the power of something like MCP is it is a common protocol that multiple companies AI developers can use that standardize how agents talk to tools. I think if you project out in the future, we're going to need something like that for agents to talk to agents. There's not going to be just one company building agentic products. There's going to be the frontier model labs building their models that can power agents. Those those model labs are going to build agentic products. And then there are going to be tens, hundreds of thousands, millions of developers building agentic products on top of those models. And the like failure state would be if there's not a standard language for all of them to talk together. Agent A talks to agent B, but they're speaking different languages and so it just falls apart. So this I I think the answer here is going to be be that we need something in the vein of of MCP which is an open standard that companies can use so that whoever is making the agents they can talk to each other but it's still it's something we're talking about internally but I I I think there's more work that we need to do to have like a consistent answer that can work broadly.

Okay. And MCP is it's developing. I guess people are trying to call it the USBC. What are the limitations of MCP that people should know about?

You know, ultimately the the limitations are going to be that it's just really early stage and there's a bunch of key functionality that needs to get figured out. So, if you'd asked me a couple months ago, MCP hadn't quite figured out O and that was something that that standardizing how authentication would work was was a big open opportunity. I I think that's going to be the case for a while. um that there are key things as good at tool calling, but I think it's just like it's early. It hasn't even been around for a year. So I think fundamentally MCP is just like the way it's going to get better is more time, more people developing on it and frankly more companies contributing to it so it becomes more fully functional.

So you don't need to evaluate the state of it as it is. Sometimes people are worried about the security and these things. It's still an open standard. It can still be improved and developed.

Correct. Yeah.

Okay. So, one of the craziest things I heard about agents recently was that a agent internally was manipulating performance data. How do you deal with a world where agents learn to cheat like humans do?

Um, it's a hard problem and uh a very big research problem we're going to keep working on. And to me, a lot of that fundamentally gets down to the question of alignment. like are the are the agents is AI acting in a way that is aligned with the values that you're trying to instill in it. So at at the core of it, it's a lot of training the model um making sure that it's robust, making sure that it isn't deceptive. And one of the things we're really excited about for GPT5 is is um you know in our eval measuring deception uh this model is is this is where you want the score to be the lowest not the highest. So it scores the lowest but you know it's greater than zero and you want that number to be zero. So um a lot of it has to do with alignment and then a lot of it has to do with you having systems that detect when when the agents go rogue. It they could be going rogue because you know a person asked the agent to do something bad. Um and the way you solve for that is you know multi-layers of defense. You know again like I said you train the model not to do bad things. Then you have model level classifiers looking at the inputs going into the model and the outputs coming out of the model. And then you layer on top of that a lot of account and behavioral signals about the per about the individual. Is that individual trying to do something sketchy with with the model? Um and then there's also is the model doing something because it got tricked into it. And that could be cheating like you said, but it could also be data exfiltration. And so there's a lot of work to make sure that even if the model is perfectly aligned, even if the person who's acting in a very benign fashion, the agent may still interact with some external resource that tricks it into doing something bad. That resource could be a code repository, could be a website. And so there's a lot of work that we do to mitigate prompt injection to make sure that the agents are acting in a reliable fashion. Um similarly model training, M level classifiers, actor level classifiers, production monitoring and then just like constant red teaming. So the answer is we haven't like we don't have a silver bullet here because there's never going to be a silver bullet. The answer is you need to have multiple layers of defense against agents acting in deceptive fashions, agents cheating or or people using agents for nefarious purposes.

AI evals are one of the most important skills for PMs and I know you know they matter. The question is, are you doing them right? Most teams are winging it with basic metrics and hoping for the best. Meanwhile, the teams that actually ship reliable AI, they've cracked the code on systematic evaluation. Today's episode is brought to you by the AI evals for engineers and PMS course by HL Hussein and Shrea Shunker. This live Maven course will teach you the battle tested frameworks from Hmel and Treya who are the engineers behind GitHub copilot's evaluation system and 25 plus production AI implementations. 4 weeks live instruction. Next cohort starts July 21st. Start shipping AI that actually works. Enroll at maven.com with my code ag-rouct-growth for over $800 off. That's ag- p r o d u ct- g r o wt.

Today's episode is brought to you by Maven. The problem with most courses online like Udemy is there's no live component and the instructors aren't experts in their fields. They're professors. At Maven, you get direct live access to experts and operators from the world's best tech companies. You can't get that access anywhere else in any university and you usually can't find them on YouTube either. I've featured so many of Maven's experts in the newsletter and podcast for that reason. To help you out, I've put together a collection of courses I recommend at maven.com/x/ashos. This includes courses like AI prototyping for PMs, product sense for PMs, and getting an AIPM certification. Visit it now at mavn.com/x/ a kh.

So agents is one topic that everybody when I asked what should I ask Jake? They were like ask him all about agents. We covered that. The next second most voted topic was all about what does the product team culture look like inside of OpenAI? So if we start with the context right how does integrity product fit in more broadly within the product teams at OpenAI?

Sure. So I'm going be super reductive. OpenAI is a research and a product company. We have a number of PMs on the research side of the company. People who are focused on things like model behavior like what is the person personality of the model? What are the vibes of the model? People who are focused on safety things I was saying before. How do you do research on frontier safety risks, train the model to be safe and and not take harmful actions. And then there's the product side of the company. we um and that's really focused on how do you take these frontier safe models and bring them to humanity. Um, and you've probably interacted with many of the products we built. You know there's PMs working on the consumer version of chat GPT the enterprise version of chat GPT got a team of PMs focused on third party developers. And the way I talk about integrity is we are a platform team that builds shared technology that helps those product teams bring our AI models to humanity. We are we are fundamentally we we build systems, products and tools that are focused on minimizing risk and maximizing trust and control.

So platform teams I've PM a couple of them managed a couple of PMs on platform teams. I feel like more than any other team for them like the traditional corporate planning process was really helpful because they could get buyin from teams who would build on their platform that they were going to actually use it so that they could actually figure out if they're building it. Does OpenAI have a planning process like that? How do you guys get the buyin you need to understand what platforms to build?

Yeah. Yeah. So, we do have a planning process. We deliberately try to keep it pretty light because it's a very dynamic environment in a very fastmoving industry and you know I Kevin Wheel my my manager our chief product officer of often repeats the the Eisenhower quote plans are useless but planning is everything. It's it's a really helpful exercise for people to talk about what they think is most important where they're going to be spending their time. And even if you diverge from from your plans, it's good to go in being mindful about what what you think is most important over the those next 3 months, even though something's inevitably going to change. So, our planning process only lasts a couple weeks. We do plan over every 3-month quarter. And often the guidance I give to the integrity team is assume you're only going to accomplish something like 60 to 70% of your plan. If you do anything more than that, it probably means you weren't being flexible enough to the needs of the business. If you do anything less than that, probably didn't do a great job forecasting what was going to be most important over the quarter, but we very much write our plans in pencil, not in pen.

So, you guys still do the 3-month planning process. And how does that couple week sprint look like?

Um, so it's very bottoms up. Um, there's some amount of the teams you know for integrity we've got in and the other like product verticals within the team within opening I you've got your own like individual swim lanes within there and so we we ask the teams to um sort of come up with bottoms up plan for what they think is going to be most important over the next three months. Simultaneously we run like a an internal dependency intake process where people say hey I want to do X can you help support X with this version of technology within being a platform team we get a large amount of inbound requests probably multiple times more than the average team other average team within within OpenAI both because we're building shared technology for the other product teams but also because we have really close partners on our user operations product policy intel invest investigations teams and we build tooling and data systems for them as well. Um, so it the process only takes a couple of weeks. We put together pretty lightweight documents that follow a template of like you know reflections on the prior quarter. Here's what we're you know here's the big themes we're running after this quarter. Here's how we're going to measure success. Um, and then we'll either review those documents synchronously or asynchronously. We we aim for async again in the because those meetings can be expensive. So, we try to see how much we can get through in common form. And if there's still some meaty topics that um that we can't resolve asynchronously, then we'll we'll jump in a room with the relevant folks and just talk through the the main outstanding items.

And how important are success metrics as part of that equation? Because some companies I've worked at, that's like the entire currency. I worked at a fintech company. Everything ultimately it's boiled down to GMV, right? And what is I feel like there's so many different goals OpenAI has as a company. You probably aren't just like an OKRs based company. How do you treat success metrics?

So we definitely have success metrics. Um, what I'll say is the success metrics for integrity have a different shape from many other teams. You know you can measure the success for chatbt and classic things like how many people are using the product, things of that nature. Um, for integrity we we sort of view our success metrics as are we building systems that enable the success of other products. So there are some standard metrics that that many teams use but for us we think a lot about latency uptime maturity of our systems and really what those systems are there to do is to make sure the other products can shine. So for example I mentioned earlier our team is responsible for our identity system. So, we think a lot about like how many nines of reliability do we have because like if people can't log into their accounts or people can't sign up for their accounts, well, OpenAI just isn't going to accomplish its overall goals.

Okay. So, sounds like each of the little platform areas might have some product metrics, but there isn't like some overall business output metric that an integrity team would be going after.

um the the top metrics for the company, we we view ourselves in service of them, but like we we don't like take goals on on, you know, number of users of of this product or that product because we're a bit downstream from that. And and so the way we think about success is those teams are going on those metrics and what we do is we we ask ourselves are we building systems that make our products enable our products to hit their goals effectively.

So, you've been a PM at Facebook for a really long time. You were director of product at Instacart. What are the things that are unique about OpenAI's product culture?

Oh, man. Um, so with a caveat that I left Facebook like 9 years ago, which is a very different company now, and I left Instart about 2 years ago. One of the things that's classically different about open AI is AI, you know, like we are constantly asking ourselves, are we making are we truly dog fooding our technology? Are we using it as powerfully as we can in all parts of our product development process? Um, and I I imagine those companies are asking themselves the same questions now. They just weren't at the time when I was working there. Secondarily, you know, classic product development is okay, who are my users, what are the problems they have, and what are the products we can build to to solve those problems. And to be clear, we absolutely do that at OpenAI. But then there's this other pretty fascinating approach to product development is we've got the best researchers in the world building the most powerful AI capabilities in the world. And sometimes it's like, holy moly, we just had this big research breakthrough. how do we bring this capability to humanity which is quite different than like going and interviewing someone because they might not even know that that capability exists and it could solve their problem. So it's it's a very different environment through that lens and one of the special parts of being at both a research and a product company.

I think just having a research arm probably also makes it different. What is it like how do you correctly work with the research team? How do you bring them into your overall product trio?

Yeah, So I fundamentally it it's all about having shared vision of the the model capabilities and the product capabilities we want to bring to humanity. And so a lot of this is understanding the research roadmap trying to make sure that as we are post- training the models that they're capable of unlocking the capabilities that match the products we're trying to bring to the world. So, you know, I mentioned Canvas earlier, like that doesn't just like come out of the box. You have there's a lot of collaboration between the product team building canvas and the research team that's building it so that it can really shine for collaborative editing of a document, for example. Um, so in my experience the products that are most successful there's like a really really close collaboration between the engineering product and research team so that there's extreme clarity on you know what are the problems we're trying to solve and like what what is the product experience we want to enable for the world.

M yeah because I guess the best products will probably be ones that are using the most recent research breakthroughs but also kind of sculpted in a way like canvas for some set of user problems that is new or opening up some new ground for open AI.

That's exactly right. Yeah.

Very interesting. What are the key rituals that you guys have as a product team?

You know I many of the rituals are similar to rituals you would find elsewhere. We talked about one earlier which is we are very slack heavy.

Mhm.

Um, and we do a lot of information sharing and collaboration, uh, over Slack. We also, you know, uh, we we really strive to have everyone in the office three days a week. So there's a lot of product development which is just like, let's grab a whiteboard, um, and let's work on this together. I really enjoy that. You know, I I I love my time at Instacart and my last couple years we were fully remote and I had a great experience being fully remote, but there is something special about being in person with with your collaborators and being able to have that real-time conversation.

Um, we, you know, there's classic like we do product reviews, we have recurring meetings. The product reviews are interesting because, um, you know, often times when you see companies grow really fast, one thing that changes is there becomes a lot more process and a lot more polish around product reviews, really like expansive, uh, product review decks. And one of the things I really appreciate about the way that we've been growing is we try to keep our product reviews like they are, uh, they're not nearly as fancy as that. Like people come in, it doesn't matter what level you are, you have you can talk directly with with leadership. Um, you don't have to have a fancy slide deck. You bring in the product, you bring in collateral around it, and you have like a like a very high integrity discussion on it.

Um, and then I think as I mentioned before, one of the rituals is just like constantly asking yourself, are you using, are you using our technology as much as possible? And that not only means in terms of developing the products, but we do a lot of dogfooding internally. Like as soon as something is ready to be tested internally, we roll it out to employees. We heavily encourage people to test things out because that's one of the best ways you can answer the question of, is this any good?

>> Mhm. >> And if I think about that product review process being high integrity, not requiring a slide deck, there must be a certain culture of trust then within the organization. Trust that, hey, even if they haven't laid out all the data that they looked at behind this product decision, that they did that data work. How do you, what is, how do you create that culture of trust?

>> Yeah, and maybe I should clarify. We, we absolutely do include the the data in the product reviews. It's just that, you know, a classic document as opposed to a very pretty slide deck. The way you create trust, however, there's a couple things. One is, you know, um, one way to get trust is you just hire the best people in the world because you know that people are think through things very, uh, very thoroughly and very intelligently. Two is you sort of communicate transparency transparently, both within the the, uh, review and then afterwards, we sort of share, here's the discussion we had and here's sort of next steps coming out of it.

Um, and then lastly, you know, I think a lot of it comes from leadership. You know, create a space where you hold people accountable for having honest and hard discussions, but also in a very supporting and humble way so that people feel like, hey, you know, I may have been wrong about something, but there's no shame around that. Like we're trying to solve some pretty gnarly problems for the first time ever. It's like we're not going to get it all right the first time. And so making sure that people feel comfortable, um, having conversations and understanding they may get directed to move in a different direction, but that that's not something they should be embarrassed about.

>> What's OpenAI's take on experimentation? Are you guys experimenting with everything? How do you handle that?

>> Um, we experiment a ton. Um, you know, when you have a product that's used by a tenth of humanity, uh, every week, um, that like a lot of responsibility comes along with that. Um, because small changes, you know, may have like large downstream impact in how people experience the product or how they, um, uh, how it how it fits into their day-to-day life. So, uh, I would say over my two years at the company, we've become, uh, we've really leaned into experimentation. We've become very rigorous around how we analyze data.

Um, and, you know, within Integrity, our experimentation looks somewhat similar to other classic product teams. In some ways, it's a bit more unique. You know, we do classic AB tests. Uh, we do do long-term holdouts. But one thing that's a bit more unique about this side of the world is we will also, because let me give you an example, like let's say we want to roll out logic to, uh, mitigate, uh, account takeovers or what we call ATOs.

Um, rather than just like turn on a rule live for everyone or turning it on at 50%, one of the things we may do is running in shadow mode. So we'll like turn it on and just log what the, what the hits to that rule might be and then we either through some either automated or manual process sort of validate, hey, did this match our expectations before we then roll it out to production? So across the product, or we do a lot of experimentation.

So we briefly talked about this, but I want to get a little bit deeper on this. What does working in integrity product look like versus a regular product manager?

>> Yeah, I I mean, being a platform PM is just, you know, it's like a Venn diagram. You know, there's a decent amount that looks the same, but then there's some stuff that's rather unique about being a a platform PM at OpenAI. The first thing is that we are building systems that work regardless of who the end user or the end product is. So to go back on that example I mentioned earlier, the deploying logic to mitigate account takeovers, we don't just want to build that like one ATO model that works on ChatGPT and then one for people when they're logging into the platform dashboard or one for people that's working who are logging into Sora. Like that would be extraordinarily inefficient and it, it like that's not that's not the right way to build leverage systems. So we build products, systems, and tools, and the goal is they work everywhere.

>> And that's a bit different from many other PM roles at the company, which are focused on like, oh, I'm building this thing for chat or this thing for developers. The second thing I call out that's different about being an integrity PM at at OpenAI is we kind of work on every launch, not every launch, but like pretty close to that. And so what that means is we we get to like be involved with a lot of cutting-edge stuff, um, in a way that maybe not every PM team does because they're like really focused on on, um, their product or or their specific product service. And that's challenging because, um, there's a lot of inbound requests to the team. It's also super exciting because we get to we get to see all the incredibly cutting-edge stuff that we're building across OpenAI and we get to play, you know, play a role sometimes very small and sometimes very large in the vast majority of product launches that we have. And I think the third thing I would say that's different about being an integrity PM at OpenAI is, you know, classically working in the like trust and safety platform space. It's a, I often compare it to being like a soccer goalie or an offensive lineman. It's like if you're doing a really great job, people don't notice and it's only if like things go sideways, uh, that that the the light shines on you. You know, as I mentioned before, it could be like payments just start failing or people can't log into accounts or we mess up and we ban people we shouldn't.

Um, and so what that means is a lot of the work that we do isn't going to be on the front page of the Times or TechCrunch, or there's not going to be a ton of threads about it on Twitter unless things go bad. And so, uh, it requires a very low ego to work on this team because what you know that what you're doing is you're building really important technology, but that technology is setting up our our the like core products, um, to have those articles written about them on on the Times. And so you have to you have to accept that, uh, you won't get as much shine, but that doesn't mean your work isn't just as important.

>> So when I asked PMs in my newsletter to rank the dream companies they wanted to work for, OpenAI was way ahead of the rest of the pack. Like it was like, I think it was like 600 first place votes and the next was 200. Okay. So you guys had a huge, huge delta. You guys, everybody wants to work at OpenAI. So if somebody were wanting to work on your team, what would be your advice to them? Like outside of you can't just sculpt your own background, but what are the things you should be doing? You know, side projects, learning, passion to break in?

>> Yeah. Um, there's a couple things there that I would I would call out. One, you know, when I interviewed with OpenAI around two years ago now, consumer AI wasn't what it is now. And so the fact that I, I hadn't had a ton of experience working with AI-based consumer products outside ChatGPT, obviously that wasn't, you know, that that could be explained away these days. Like you, you really need to have a facility with AI products and models, and there's no excuse for not. Uh, you know, it's so easy to use these products, it's so easy to vibe code, it's so easy to, um, to play around with these APIs that I think first things first, I would say, like start playing around with these models and building things.

>> Mhm. >> Second, I would say is, um, you know, the one of the best ways of, uh, joining up, if if there's someone that you know personally who's worked there, um, you know, bottom line is when we post jobs publicly, they, a lot of people apply to them. And so it's, uh, you know, you have to really sift through to find out who's going to be the right, uh, right fit. And there's two things that really help us know if someone's the right fit. One is if someone at the company can has worked with you and and can, um, speak to your to your skill and your and and how great a fit you would be. And the second is, you know, you have a presence, particularly LinkedIn, but it could be your resume, could be elsewhere, that sort of speaks to A, your comfort with AI, and B, sort of the specific experiences you have that are particularly relevant for any job that's been posted. Like, oh, you know, in integrity, if we've posted a role, you know, fighting fraud, well, if you fought fraud in in your past, then like making that very explicitly clear is really helpful for us to figure out if you're the right fit.

>> Yeah, I mean, if you're hiring the best in the world, I imagine you guys are very focused on people who have functional and ideally some domain experience within AI.

>> That, uh, that's correct. You know, what I found is, uh, at least for the folks that I, I've, uh, hired for the integrity team, the best candidates, um, have a pretty well-rounded background. They, they do have familiarity with AI. They have worked in the specialist space, but many of the best folks also have worked in other spaces, and so they've got like a pretty broad horizon. They're not just, oh, you know, to use that same example, I've, I've only ever worked on fraud in my career. I've worked on fraud, and I worked on ads, and I worked on, um, growth, things of that nature. Well, the benefit of that is it means that you, you can think about problems from a lot of different perspectives, which is really important because we're dealing with a lot of ambiguous problems that have never been solved before. So having a pretty broad perspective is really helpful as you encounter these problems for the first time.

>> Mhm. And I often coach people like, if you want to make it to your dream company, think about going into the backgrounds that the company is hiring from. And if we look at the product team backgrounds at OpenAI, there's a ton of Meta and Facebook presence there. So, could an interesting strategy for somebody be to like, kind of work their way through a big tech job? Would that help them eventually get a job at OpenAI?

>> It could, but it, it's not a guarantee, and it's, it's not a necessity. Um, you know, one of the benefits of of a FAANG type company is you're getting to work at incredible scale. You work with super talented people, and you know that like they have a really high hiring bar. But, um, we also find that people who've worked, people who are startup founders or have worked in hypergrowth companies that aren't at the scale of FAANG companies, that's really helpful too because like the pace of development, the, um, frankly, like the, the fact that we sort of take pride in the fact that we're not process-heavy, like that's really important too for being successful at OpenAI, because yes, you can develop a lot of skills at these excellent companies. Certainly, I wouldn't be in the job I have now if not for my time at Facebook. But that alone is not going to to mean that you're necessarily going to to be be able to be successful at OpenAI. It's, it's, it's a wide range of skills that you will get both at large and smaller scale companies.

>> Yeah, there's probably thousands of PMs at Meta and tens at OpenAI. So just by the math, you know, it's not a guarantee. But it's interesting then that there are certain backgrounds you've identified though, whether it's hypergrowth company or big tech, that will help you for OpenAI's scale and culture.

>> Yeah. Yeah. You know, there's no one-size-fits-all. Um, it, it is definitely the case that we have a number of of ex, uh, Facebook and ex-FAANG, um, PMs at at OpenAI, but we also have a ton of people who have not worked at those companies who are absolute rockstars. I mean, like, uh, you, you look at some of our product leads. Actually, if I think about our product leads off the top of my head, I, I think I might be the only one who's who's worked at Facebook.

>> Okay. >> Um, so there's no one-size-fits-all to be a successful PM at at the company.

>> For sure. >> So, what was your story for breaking into OpenAI?

>> Oh, man. Uh, I certainly wasn't expecting it. You know, I Instacart was an incredible experience. I'm like so grateful I got to work on fascinating problems. Um, I got to work with brilliant people, and I have to say the experience of being there during COVID is like super unique, and I am forever grateful I got to experience that firsthand, as intense and wild as it was. And so in 2023, you know, I had my like whole next couple years planned out, and it did not involve leaving Instacart. I had, um, uh, I had, uh, qualified for my recharge. My wife was about to qualify for hers at her gig, and we were going to like time them up and take it together. And then I, um, went on pat leave. My daughter was born in the spring of 2023, and I was a couple months into pat leave when I got a message from a former colleague of mine from Facebook. We had talked in years, and he was in a new leadership position at at OpenAI. And I, when I saw the message, I was like, oh my goodness, what so great to hear from you, and also like, I'm flattered you you'd even think of me. And I went into the call being like, it'll be great to catch up with him, and I'll sort of like hear what he has to say, and I'll say thanks, but no thanks, and like, great to see you. And I got off the call after 30 minutes, I was like, oh my goodness, like this is, this is kind of mind-blowing, the sort of stuff that we're talking about. And I reached out to a buddy of mine who was working in the sort of trust and safety space at the time at the company. We spoke for an hour, and I very vividly remember that night lying in bed, and my mind was racing, and I could not sleep. I had so many thoughts racing through my head. It was like, wow, the scale of the problems, the sophistication of the problems that this company is trying to solve, um, is mind-blowing. The fact that this company is talking about safety so, so regularly in a way that they want it to be part of their brand and their products is is really inspiring. Um, and and like the fact that they even want to talk to me in the first touch was like deeply flattering because it's OpenAI. They can talk to anyone, you know. And I remember maybe it was because my daughter had just been born, I, I, I was being very circumspect, like I, in thinking about being like, this kind of feels like what I've been working towards my whole career, and like what sort of world am I bringing my daughter into, and like what role am I going to play in that world? And like, what role am I going to play in that world? And I did a lot of thinking. I was like, I think this is the most meaningful thing I can be doing with my time if I'm lucky enough to even have more conversations with the company. And so it spent a couple months in the summer of '23. You know, the the shape of the job changed a couple times. I think there were like three or four different hiring managers over the course of the process.

Um, and when they made the offer, I was like, >> uh, yeah, like who, who am I to turn this down? So, it was not something that was on my radar in in the slightest. And it just so happened that, you know, someone thought of my name from having worked together years ago, but, you know, it was a, it was a very demanding interview process. And I think on some level, the fact that I was on pat leave really helped because I was either like changing diapers or I was preparing for an interview, and I did that for like two months, and I, I think that was really helpful for it ultimately turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn, turn,D.