How far are we from "Her"?

Do you remember the movie Her from 2013? It's about Theodore, an introvert who falls in love with his AI assistant, Samantha.

"Hey, where'd you get that name from?"

"I gave it to myself, actually."

Fast forward 13 years to July 2026, OpenAI releases GPT Live, promising a meaningful upgrade to their ChatGPT voice assistant.

"What do you think?"

"Go up. It's getting a kind of baggy."

"Although baggy is very popular now, don't you think?"

And OpenAI are not the only ones trying to bring Her to life. Just 2 months ago, Thinking Machines announced their interaction models. They respond in real time and jump in very proactively.

"Let me know if I start to slouch."

"I've got you. You're starting to slouch forward."

So, why are the frontier lads rushing in this direction all of a sudden? And what is it about these new voice agents that allows them to be so much more fluent and responsive? Well, I found the one person who can answer these questions. Nelly Sagidora spent years at Fang building the audio tech behind products like Google Meet and Notebook LM. He then founded Q Tai, the Paris-based nonprofit lab that built the very first full-duplex assistant. That's the new class of models behind the recent launches.

But before we get to the bleeding edge, let's remind ourselves where we were less than a month ago. When I looked at the ChatGPT app, I noticed there are actually multiple options for a voice, including standard, advanced, and live. About a dozen papers later, I realized that these are not just quality increments. They're conceptually different ways of modeling conversations. Let's take a look at standard.

"Hey, I'm in a library right now. Can you tell I'm whispering?"

"I can do that. Imagine I'm lowering my voice."

"Hey, am I whispering right now?"

"It does sound like you're speaking more quietly than normal. Yes, I don't actually measure your volume precisely, but your voice comes across as a whisper."

"All right, could you beatbox for me?"

"Boots and cats and boots and cats. BST st bird cuts cuts cats. BFFs bird do do me cats."

Okay, so what in the world is happening here? The issue is standard mode is basically an LLM wrapper. If you ask the assistant, it will tell you it was built on top of GPT 5.5, which is a pure language model that cannot process or produce audio on its own. There's an entire pipeline of additional components that gives it a voice interface. A voice activity detection or VAD model continuously analyzes the audio waveform captured by the microphone.

"Like, when did you give it to yourself?"

"Well, right when you asked me if I had a name."

"Every 30 milliseconds or so, it makes a binary decision. Is the user speaking or are they silent? After long enough silence, it concludes the speaker is done talking. Then every identified speech portion is passed on to an automatic speech recognition model or ASR, which transcribes it. Next, a pure language model like GPT 5.5 reads the user input in text form and produces a text response. So, by the time the message gets to the LLM, the acoustics are lost in translation. I mean, transcription. And finally, a text-to-speech or TTS model reads the LLM response out loud. This design is representative of a large class of voice assistants known as cascades. Cascades can be quite slow since latency accumulates across components. OpenAI hasn't published latencies for GPT-5.5 in particular, but its predecessors took multiple seconds to respond with voice. For reference, a human takes a little over 200 milliseconds on average."

Now, let's see how this compares to advanced mode.

"Yo, can you beatbox a beat and I'm going to rap on it?"

"All right, let's do it."

"Yo. Yo, what's up? Keep it going when I'm rapping."

"[panting] Yo, yo, what's up? I'm I'm"

Okay, at least we're making progress. It might not beatbox perfectly, but at least it feels more audio native. The model behind it is GPT-4o, where O stands for omnimodal, meaning it can process text, audio, image, and video without passing them through a text bottleneck. It falls into a category that is the complete opposite of cascades, namely end-to-end models. An end-to-end voice assistant isn't necessarily a monolith, but its components are all trained together on speech data. So, input acoustics are never lost. And removing the LLM bottleneck brings latency down to almost human level.

But GPT-4o is not exactly Her. The problem is it can only do one thing at a time, either speak, i.e. beatbox, or listen, but never both. Even regular conversations can feel a little bit like passing the baton back and forth between you and the assistant. But GPT Live promises to change that.

"Can you beatbox for me?"

"Sorry, I can't do beatboxing, but I can help in lots of other ways."

"All right. Tell me, what's the plot of the movie Her?"

"Theodore is a lonely letter writer in a near future LA, and he buys this new AI."

"What I just did there is called backchanneling. I exaggerated a little, but humans do this often to acknowledge that they're paying attention. It might sound mundane, but it's actually a challenge for voice assistants. They need to decide whether your intervention should interrupt the conversation or not. AI's ability to overlap naturally with humans is what brings GPT live closer to Her. Listen to Samantha's interventions in this scene."

"[laughter] Call me."

"Wow, your feet really?"

"Yes, he's obsessed."

"What?"

"[laughter] Well, obsessed and now you have to show them to me. Come on, show them to me. Let me see. Wow, he's right. They are really hot."

"[laughter] See, I told you Tatiana, you have hot feet. Face it. They're my favorite thing about her."

"Really?"

"That's it, my feet."

And here's what's interesting. GPT-4 and GPT Live are 2 years apart, which is like 200 in AI years. So, you would think that the difference in quality comes from scale. More parameters, more data, faster GPUs, but that's not necessarily the case or is a very small part of it. The true unlock comes from the fact that GPT Live models conversations entirely differently. These segments are called speaker turns. In a real conversation, they often overlap. Samantha repeatedly chimes in while other characters are talking like here. The problem with GPT-4 is that it forces this whole conversation into a single sequence of alternating turns. But, models like GPT Live completely throw away the concept of speaker turns.

"They're not turn-based. They're more like time-based interaction where they're continuously taking in audio, text, video, and continuously providing output. Now, we cut this up in chunks of 20 milliseconds, and this enables you to catch these things like interruptions and simultaneous speech."

This was Mira Murati in a Bloomberg interview. She's the former CTO of OpenAI, now founder of Thinking Machines. They're building something they call interaction models.

"Can you be spontaneous and answer questions as you hear them?"

"Absolutely, I'm ready when you are."

"What is the smallest prime number?"

"Um what is the capital of Jamaica?"

"The capital of"

"What's the largest mammal on Earth?"

"The largest mammal is the blue whale."

"Who painted the Mona Lisa?"

"The Mona Lisa"

The core idea is to view this conversation as two separate concurrent audio streams, one for the user and one for the assistant. Crucially, neither stream is ever discontinued. When the user goes quiet, their stream simply carries silence, which is modeled like any regular piece of audio. So, there are no more turns to hand over. For instance, here, the agent is still answering the second question while the third one is already coming in. It's listening and speaking at the same time. A turn-based system would have to make an impossible choice here, either cut its answer short or miss the incoming question. Formally, this is known as a full duplex. The channel carries communication in both directions at the same time. This is exactly how OpenAI describes GPT Live. The traditional turn-based systems are half duplex. The channel still works in both directions, but the agent is either listening or speaking, never both. The classic half duplex device is the walkie-talkie, invented around 1940 by Donald Hings and widely used during World War II. It had a single antenna that acted both as a transmitter and a receiver, so it physically couldn't do both things at once. That's why soldiers said over at the end of each message.

GPT Live made headlines, but it's not the first full duplex in the world. In fact, it comes 2 years after Moshie. Moshie is an open-source voice assistant from Qutai, which is Nao's nonprofit lab in Paris. With less than 10 people and in less than 6 months, they set a completely new direction for voice assistants. Here's Nao on what motivated them.

"It's almost ideological, I would say, right? In a way, when you impose by your human hands a cascade in several gesture rings or signal network, you know, you go against the philosophy of machine learning. So, you know, there is always this attraction to trying doing things end-to-end. What was not obvious for us is that we did not even know what full duplex meant. And when we had to write the tweet about the announcement, someone made me notice, 'Hey, you know, your thing is called full duplex.' And I looked on Wikipedia and I was like, 'Okay, it's full duplex.' And so, then we started bragging about having a full duplex modem. So, until early July 2026, it was the only a full duplex modem in the world for almost 2 years."

Since Moshie is a full duplex, it takes in parallel two audio streams, one for listening to the user microphone and one for generating its own response. This multitasking ability enables it to handle backchanneling. When the user says something like, "Mhm." the assistant is no longer forced to switch turns and can continue uninterrupted.

So, how does Moshie work under the hood? Well, to generate audio, we need to split it into small building blocks that can be predicted one at a time. Ideally, we could predict raw amplitudes like my animation is showing. But, for real-time speech, that would require at least 24,000 sequential predictions per second, which is a challenge even with the newest GPUs. Instead, the most common approach is to segment the stream into fixed-size frames, something like 80 milliseconds. These are the audio equivalent of text tokens. Though, the analogy is not perfect because audio is a richer signal. It combines semantics, or what is being said, with acoustics, or how it's being said.

Now, An and his team proposed this decomposition back at Google with Audio LM. Following the same principle, Moshi decomposes a single audio frame into one semantic token and seven acoustic ones. Now, these sub-tokens do indeed act very similarly to text ones. They can be mapped to integer token IDs, as well as real-valued embeddings. Audio tokenization is an entire subfield itself, and I covered it in this previous video. Even though my diagrams are showing a single audio stream for the assistant, there are actually eight of them, one for each sub-token. By keeping them separate, we can artificially inject delays between them. Specifically, acoustic tokens are delayed by two frames during pre-training and one frame during fine-tuning and inference. Why is that helpful? Because we're imposing a conceptual hierarchy here. First, generate coarse semantics, the equivalent of phonemes or words, and only then vocalize them with fine-grained acoustics. And the conceptual hierarchy doesn't stop here. You see, over 5,000 years ago, we invented the top of the abstraction pyramid, written text. Text might actually be one of the reasons why we're thriving as a species. It's a very efficient signal.

Now, Moshi by design removed the text bottleneck in Cascade Assistants. But, its objection was forcing everything through text. As long as the audio signal also goes through, text is pure upside. Moshi brings it back in the form of an inner monologue. It's a text stream output in addition to the audio stream. But, I want to make something clear. This is very different from the chain of thought in LLMs, where the model gets extra room for reasoning. In contrast, the inner monologue doesn't delay the response at all. It advances at the same pace with the audio. Its job is to keep the speech tethered to language. Without it, Moshi might sound right, but say nothing of substance. Moshi is basically an augmented LLM with a multimodal vocabulary that includes both text and audio tokens. In fact, a big portion of its weights are initialized from an LLM and fine-tuned with the rest of the weights on speech data.

Now, at its core, Moshi is a speech-to-speech architecture. But, the inner monologue makes it more versatile. If we delay the text stream by a couple of seconds, the model learns to transcribe the audio it just heard, making it a speech-to-text model. The other way around, if we delay the audio stream, we can train a text-to-speech model. Of course, at inference time, the TTS model will read user text rather than its own inner monologue. So, with a single model architecture, we get three different behaviors. Moshi really paved the new way for voice models. It was trained on about 100 H100 GPUs for about a month, which might sound like a lot, but for Frontier Labs, that's just a rounding error. Yet, Moshi responds faster than the average human, and if you quantize it, it even runs on your MacBook.

So, if full duplex models are so enticing, then why aren't they everywhere? Well, here's something that might surprise you. Now, who founded Qute the nonprofit in Paris, recently spun off Gradio, a for-profit arm that builds building blocks for, well, cascade systems.

"What changed your mind? Why did you join the other camp?"

"So, basically, we look at what it was on the market. What it was. So, it was very surprising for us, but after Moshi, we did another release called Unmute. So, we had developed a streaming speech-to-text and a streaming text-to-speech. And so, Unmute was just an open-source framework to take any LLM and make it speak, right? And in a way, it drew much more interest from big companies than Moshi, right? Because I think they they saw much more directly how they could use it. Because, you know, in a way, the adoption of text models has been more mature. And so, a lot of companies already had a text model, and they just wanted to turn it into a real-time conversation system."

The major disadvantage of end-to-end models is that you can't easily plug in the newest and greatest LLM. You have to retrain the entire pipeline end-to-end, and that is if you have access to the LLM weights. So, cascades are the practical approach today. Arguably, they're the right thing if you're, say, a bank implementing customer support. But, that's the short-term view. Companies like OpenAI and Thinking Machines have enough budget and talent to think longer term. That's why they invest in end-to-end models. And there's quite a bit of work they need to do in order to bring over all LLM capabilities.

"several thousand emails regarding LA Weekly, but it looks like you haven't worked there in many years."

What Samantha just did here is tool calling. She had to call the Gmail API to access all those emails. Both OpenAI and Thinking Machines have confirmed that they can already handle this. The core speech-to-speech model delegates to an asynchronous background model, presumably a powerful LLM, while keeping the conversation going at the same time.

"So, agent is going to do a lot of processing in the background. It's going to do tool calling. It's going to reason. It's going to do web searches and so on. But you don't want to just wait while all of this is done, right? It's going to take seconds to dozens of seconds. What you want is to keep a natural flow of conversation that is kind of asynchronous with the computations that happens in the background. So, that's something we we introduced with Moshi RAGs and this also is used by the interaction model from Thinking Machines. So, instead of having one model, you have two models."

Moshi RAG is a follow-up to the original Moshi. It can pull in knowledge from external documents. It does so mid-conversation without pausing. The moment Moshi realizes it needs outside help, it drops a special token into its inner monologue. This token fires off a request to a retrieval system running in the background. Of course, retrieval takes time, maybe a second or more, but Moshi RAG was trained to fill that silence. Either with a rough answer or small talk like, "Let me check that for you." That's exactly what customer support humans do while their slow playbooks load. When the document arrives as a sequence of embedded tokens, it's simply streamed into the model. More precisely, it's like a fourth stream added to the existing three, and the output is now grounded in the retrieved facts. This strategy can be generalized well beyond RAG for any kind of delegation to a background model.

So, what would it take for end-to-end models to actually become the dominant paradigm?

"My intuition is at this point it's going to be mostly a post-training question. Which is creating the right synthetic in situ data and finding the right reward models. And this one is very interesting because when you look at text models computing the reward for a verifiable problem it can be exactly computed and so you can scale it very well the post-training. Asking the question, is this conversation natural? Very hard, right? To find an objective automatic metric of that. What does natural mean? Is it that the pauses were at the right time? Was it that the tone was right? Is it what was said that was meaningful? How do you pick the best out of two 10-minutes conversations?"

"In the limit, the end-to-end systems are going to catch up, but what do you think that timeline is?"

"Being a bit pessimistic, I would say 1 or 2 years. We're going to see very impressive stuff in the next year or so. And then I will say we're going to see a completely new wave of changes. With two things I think. The first one being miniaturization, so being able to make this stuff run on device. Maybe most importantly, integration in robots. Or anything that is not a smartphone or or a computer."

And now the big question, how far are we from Her? Well, sort of close and sort of far. It's the same story with AGI where the goalpost keeps moving. With every release, AI becomes superhuman in one dimension and super dumb in some other dimension. It just has a very different intelligence distribution compared to us. But I wanted to end the video with a different question. Should we pursue Her? Now, I'm personally all in favor of better customer support when I call the bank, and I might be on board with an AI therapist as well. But friendship and love is where I clearly draw the line. AI doesn't love you. And even if you're feeling extremely lonely right now, I promise to you there's someone in flesh and blood among your family or friends who really does. That's it for me. If you want to watch the full interview with Mel, it's available to my YouTube and Patreon subscribers. Thanks for watching and I'll see you next time.