Transcription
Why does Chach PT give different responses to identical questions? Take, for example, the age-old question: Should I text him? In this conversation, Chach PT says no. But in this conversation, using the exact same phrasing, Chach PT says yes. So why is it changing its mind?
AI models like ChatGPT are non-deterministic, which means they have a little bit of randomness built into them. For every single word the model is typing out in its response, it first generates a list of the most likely next word tokens. You might assume the model just chooses the top one. It actually performs better when it has more variety and choosing some of those words that are further down in the list. So computer scientists have something called temperature, and it tells us how likely the model is to choose one of these lower-down options. You can actually see this on the OpenAI playground, where you can set the temperature manually to one. And if you regenerate the last word of the sentence nine times out of 10, you are not going to get that top option. OpenAI doesn't publish what their actual temperature is, but it's a reasonable guess to say it might be somewhere around 0.7, which means it's boosting that top option, still allowing for a lot of variety.