Google DeepMind Distinguished Eng (L9): How To Land a Job at a Frontier Lab | Vlad Feinberg

Every single time you go up for a pre-training run, you're about to put in more flops into this run than you've ever done before. This is Vlad Fineberg. He's Google DeepMind's pre-training area lead. And I asked him all about how to get a job at a frontier lab.

"That was a particular skill that I see voracious demand for across all the different labs. The research skill set is going to become increasingly important. If you do the scaling book exercises and, you know, send me a video of yourself doing them, I would love to, you know, interview you."

Here's the full episode. You wrote this post that was titled, "How to get a job at a frontier lab. What are the skills that are kind of in demand in Frontier Labs?" Maybe we can talk about the shape of the work. There's quite a range of different things that Frontier Labs require at this point. LLMs are artifacts that are connected to uh research and product in ways that machine learning really hasn't been as connected to before. And so it it really touches on so many different things. The goal of my post was to propose just a couple tangible directions in which labs could require a certain set of skills, not not to be fully exhaustive. And really the ones that I I dive into have to do with uh kernel development and a low-level engineering to accelerate the runtime for these LLMs uh in practice. And so that that was a particular skill that I see voracious demand for across all the different labs and uh among different projects within the labs. So that that seemed like a very sharp one to call out as uh an overall need. Uh and so specifically whenever we're doing a research project that involves changing the architecture for the neural net in a particular way or rethinking how we might do serving to uh you know do better KV caching or something like that again across the stack you just need to be able to implement these new techniques in efficient ways and uh the inner loop of all of these different changes is creating software artifacts that can function at large scales with high throughput, low latency. Uh, and this is just fundamental work that's tied to classical backend engineering thinking. Uh, so yeah, it seemed like a very open thing for people to specialize in.

My friends that work at OpenAI and Anthropic, there's this distinction of an applied org and the research org and I was wondering if DeepMind has a similar uh distinction and if you could speak about what that difference is. So we we have different focus areas and like you know for instance within GDM there's a team that focuses on how uh we can use our Gemini LLMs to better inform search results and so like that might be some you know you know in some way like an applied version of the LLMs but I I am hesitant to you know make a very sharp distinction here because there's so much actual like hard research that has to go into this kind of level of product integration like specifically for the one I mentioned uh quite a lot of work goes into making sure that these LLMs are factual and can cite sources uh to have very precise grounded answers assessing the quality of these sources to make sure that you're not referring to anything that's like sarcastic or a joke. This is uh I guess a good example of how even in like product specific quote unquote applied AI verticals you're still doing research. Uh that being said, there's definitely what I would say is like very classical LLM research teams, pre-training, post-training. These are things that are still standalone uh teams inside of GDM that are focused on what I would say is like you know creating soda models, you know, pure research. Again, the caveat is the the pure research that we do like the extent that it matters is the extent to which we can realize it. And so, you know, we're just as responsible with uh delivering these models and making sure they train stably and actually being like the SRRES of sorts for the training run to make sure that the model training is going smoothly. Uh, as we are for coming up with the recipes to make these LLMs and you can't separate those two roles. It's it's really crucial to kind of wear both of those hats. So yeah, I think you can you can draw up a spectrum between research and applied. Uh but uh no matter what in today's world, I think uh everyone needs to be fluid across that spectrum.

I noticed there's also another spectrum of software engineer to pure AI researcher and like how do you think of that spectrum like software engineering versus like AI researcher roles? So I guess in um in in my case specifically I think a lot of what we do and a lot of the new techniques that we develop the groundwork is laid in infrastructure investment. So um I can walk through what my team does a little bit more detail uh later but one of the verticals is uh distillation and in order to do uh distillation it's it's some way of of transferring the knowledge or some form of statistics about the underlying data set through a teacher model into the student model to make the student model better than if it hadn't ever seen these auxiliary statistics from the teacher. And when you're talking about statistics derived from a massive LLM applied to trillions and trillions of tokens, uh you're talking about a level of flops investment that you know is, you know, millions and millions of dollars. And that in turn means that you have to be able to think through how do you uh optimize the system to be as efficient as possible because every operation that we're performing is is multiplied by such a large factor that yeah every second counts every bite of storage counts and quite a bit of that work is you know good old-fashioned software engineering. And so uh in particular the infrastructure for distillation has evolved through maybe three to four generations at this point. And in each one we've taken a step back looked at what kind of research methods have we been applying for distillation holistically thought about how do we broaden what the infrastructure is capable of. And there's definitely a couple discreet points where rethinking the system design of how we perform distillation enables us to do research on distillation methods much more quickly. And so it's this kind of investment that like okay this like four month or whatever rewrite of our distillation infrastructure uh then results in a dramatically new understanding of uh distillation scaling laws that translates to really strong models. So it really requires just work across the stack and I you know I can't yeah I can't imagine that we would have gotten results like flash 3.0 know without having made those distillation infrastructure investments that are at the end of the day things that started with a good old-fashioned design dock and thinking about what the right abstractions are for uh generating these teacher statistics coming up with the right storage system for them thinking through what could support uh the reading and writing across uh multiple different data centers at this scale really classical distributed systems problems.

"Yeah, I mean it sounds like there's there's a lot of software engineering engineering backend infra type problems given just the scale of the compute at this point. It still feels like though there at some point in that spectrum there is some crossover where there's these new skills like somewhere where if you had took a arbitrary backend engineer and you placed them to I don't know adjust the model architecture or something like there that is like a bit of a jump more than the infrowwork. um like how do you see that distinction?"

"Yeah. So I think there is a crossover point in terms of doing research where research is an endeavor where the payoffs become a lot higher risk higher reward and we have this notion of uh kind of research taste which is you know some high level intuition about what path you should be proceeding through the DAG of the multiple uh different milestones that you need accomplish in a particular project. In some sense, we can view software engineering projects through a similar DAG where you know you have all of these intermediate artifacts that you want to hit in a software program to uh get to the final result. But in the software engineering case, the DAG is more or less deterministic where you you know build one service then a different service then a third service and you know you figure out your storage infrastructure layer first uh that kind of thing and you can just make monotone progress. But in the research case, you have to uh kind of explore this DAG which is now stoastic because some of the nodes which might be some research ideas or some you know aspect of getting to a final goal uh may or may not work out and I think that requires a bit of a mindset shift and that that kind of mindset shift takes a while to learn and it takes specialized skills to learn. uh this would be the kind of skills you pick up in a PhD. For instance, one succinct way I could put it, there's a really excellent post by this uh professor Jacob Steinhart and I I love to frame a lot of the research work that I do in this way and it's research as an MDP. So MDP here markov decision process uh it's again we have this highle idea of a stochastic dependency graph between different milestones in a research project where you might need to have a pertinent certain kind of result or prove a certain kind of theorem before you get to a certain kind of conclusion. Uh similarly for a machine learning research project you might need to have this and that featurization working before you can get this and that imageet accuracy or something like that. um and expanding those nodes in this graph. It's this stochastic endeavor where these approaches may or may not work out and whether or not one works out opens up a set of new possibilities for you. And so the approach that you might have in the software engineering case where you could fully write out here are all the paths to the goal across walking this graph. what's the shortest path to your goal? That approach is not optimal in the research case because if all of a sudden the transitions between the edges in this graph become unreliable and uh some of the nodes you might not even be aware of. It might be a hidden MDP. Then the way that you might approach this problem would really differ. And in particular, you have to factor in the success rate and the time investment that you're going to be putting into uh these different research ideas as well as a priori estimating what those different rates are. And that's a very different exercise than writing up what the you know design for your software engineering project might be. And it's it it's this skill set of of building an intuition of how likely an approach is to work out without having yet done that approach that I think people often you know correlate with this uh research taste notion. But that's exactly the one that you need to build up in order to properly uh traverse this MDP."

"For the the research projects and just like generally the nature of the research work. And it sounds like you're you're saying that there's a lot more uncertainty here. I'm still trying to get a sense of the the nature of the work. If you threw this backend engineer into a team that's doing research, like what are those like concrete examples where they fall short? Like I think the the very first thing that comes to mind is having the right context for the research landscape in which you're operating. So quite a bit of research work involves like almost uh this kind of um you have to take on this like very humble viewpoint of there's been quite a lot of investment in related work in the past and until I know the sum total of humanity's bleeding edge in this topic I'm definitely not going to be able to further that bleeding edge. So building up uh a solid understanding of of past work uh in a particular area uh and doing that related literature review is maybe the first thing that I would imagine people might stumble on is uh having read and having the skills to effectively traverse you know historical uh citation tree for a particular topic because you don't have the time to read all of these different papers. is you need to build up a sense of uh what are the high value papers and what are the ways in which I can assess if a paper is worth reading without fully reading it. That's like the first thing that comes to mind as the you know skill that people need to build up even to be able to read these research level papers. You have to have a background in machine learning in um some you know computer science and uh you know depending on the paper and depending on the domain there might be all sorts of prerequisites in terms of like the underlying math and coursework that you would want to have to properly understand. So that's that's quite important to be able to have a deep understanding of what methodology is available because you really won't have a lot of hope of improving upon the methodology if you don't understand what's there already. So, so I think I like mentioned earlier, one of the things that my team works on is is distillation. And in order to advance our understanding in uh distillation for large language models, you have to have a good understanding of like what we're trying to do with LLMs. And uh just to give a cursory overview here, the name of the game for LLM research is especially in pre-training is uh is scaling laws. And so what are scaling laws? People focus a lot about like you know this power law structure and the fact that like you you know have this and that exponent but like what matters is less so the functional form. What matters is for a given recipe of scaling up your LLM. So as you invest more and more flops into the pre-training run of an LLM, you have to be able to predict what the final test loss of this LLM is going to be. And why why do we care about this question? Why do we care about predicting what our uh generalization error is in the classical machine learning world? Like say we're trying to you know win imageet we have our test loss which is our classification uh error for you know a thousand different classes and uh you run your VGG or your ResNet proposal to get that uh classification error that's an estimate of how well that model does at classifying amongst those thousand classes various different images. we can estimate how good our method's going to be by taking a validation set and then whenever we have an architecture idea for a neural net we just train it and then we uh do a bunch of uh validation set runs and we get a cross validation error that is itself an estimator of our final test error and so in this way you can just iterate on different ideas uh through this process but what's different in LM world is every single time you go up for a pre-training run you're about to put in more flops into this run than you've ever done for. So it's in some sense like a oneshot version of this imageet problem. You never get to see the full imagenet training data set. You have to practice on emnest and then cf and then maybe based off of those you try to come up with a method that just works right off the bat on imageet. And if you were to just do that by itself, as I'm sure many people have tried, like certainly when I was learning how to do all of those different things, you get something, it works really great on emnest, it maybe even works on CFR, and then all of a sudden it breaks on imageet. You'll find out that like things don't just generalize easily across scale like this. And so much of what we do for LMS is coming up with recipes where a recipe is this function that goes from number of flops you'd like to train on to a training routine for this LM. And if you can couple this recipe with a prediction rule that can predict accurately what your LM accuracy is going to be, then um you're able to make decisions about how to improve your recipe because you can use that prediction. That is all a ton of context on what uh LLM research looks like in general. But that's like an understanding that we got to that we even thought was feasible thanks to so much uh initial LLM scaling work that we've seen across the Kaplan paper across Chinchilla. Since those two papers, there's been a lot more work in terms of like what other factors are there beyond uh number of params and um number of tokens that you train on that influence your prediction accuracy uh like number of unique tokens for instance. But like I would say like those two foundational papers for LLMs uh those are informed by uh an even even longer line of uh different uh scaling works uh going back to like say the original uh GPTs and then Google has had a ton of scaling work across its palm papers. This is just a set of works that have informed that viewpoint that I described earlier that you you kind of just need to build up by having gone through that literature review yourself. If you were, for instance, if uh you were trying to pick someone that was going on your team and the the way that you would judge their fitness to help you push the frontier is their understanding of the frontier, including the existing literature, which requires all these prerequisite.

"I think you called it mathematical maturity in your post."

"Yeah. So I think I I I think it's easy to read and understand those papers once you have mathematical maturity. So I guess the ones I mentioned in particular nowadays they're table stakes. So I I would expect candidates to be familiar with them. Um I think um the the general skill set is being able to dive into uh a paper of that level and then understanding it. uh you know being able to take a research idea uh from a paper and implementing it yourself like that's that's just a a very important skill set to be able to have like we get you know all sorts of uh different ideas presented you know they might not all directly apply to our domain but if you can deeply understand them then you can iterate on them and you can improve them inside of uh inside of our domain and so when we assess for people who can work with the mathematical concepts in these machine learning papers. That's that's I guess the the key skill there that would be evidence that you can go pick up this arbitrary paper and see to what extent these ideas carry over uh in the Google setting. This probably won't be exhaustive, but I'd be curious to hear other domains that maybe people could dig into to see what kind of matters in frontier AI research. So you'd mentioned distillation, you also mentioned kernels. It sounds like kernels are helpful everywhere. Um, but are there other areas that come to mind if you were just raffle off areas that are not necessarily exhaustive? One thing that I think is is quite powerful is uh actually programming language research. So by looking into how we can create abstractions at the programming language level, we could facilitate kernel development. I think Thunderkittens is a really good example of this. Like coming up with an ab an abstraction that allows you to write kernels through four functions instead of arbitrary globs of C++ code uh allows you to move really quickly uh in uh developing algorithms that fully utilize hardware. So like it at that point it's um it's not about the PL research itself. It's about having a passion for you know these kind of programming language abstractions and and working with low-level hardware um you know uh people who you know are interested in and will try to work with like cute DSL this kind of thing where there's a lot of hardware specific uh domain specific languages one other thing that comes to mind besides PL and uh scaling law literature would be reinforcement learning literature. uh so in particular ever since uh RHF uh I think we've seen that DRL algorithms uh like PO do have a place in production systems and you know there was a time where that was in question but uh now it's you know uh pretty unanimous that we see these kind of algorithms applied to real production systems and the uh theory behind that uh you kind of have to start with the basics for reinforcement learning and work your way up to you know the myriad uh value based methods and and uh policy gradient methods that we have today. That's that's another domain that I think is just like a very rich literature tree to crawl. Um, and then for more of the backend engineer folks, just beyond just the kernels themselves, there's I think a pretty fun overlap between distributed systems and optimization work where uh figuring out how to design neural net training algorithms that allow for training across many GPUs. There's all sorts of fun challenges between asynchronicity, how upto-date your gradients are, how pipelining affects the staleness, uh all of these system choices that you could make in your training algorithm design will impact convergence and the final quality of your neural net and uh those are things that can be analyzed independently of the LLM setting uh and have been for a while. So uh you know especially if you're kind of more infrain inclined then having a good understanding of like uh how those different algorithms works work is a is a really good place to start.

Do you see any difference between the the demands of the different frontier labs? So for instance if someone wants to work at DeepMind is there like a particular area that you see DeepMind cares about more than Anthropic for instance? I think in terms of the skill set, it's probably pretty similar. Yeah, I think I think there's maybe differences in like business strategy and uh you know the set of offerings that's a function of uh the specialties of the labs and uh like the kind of different uh you know customers that the labs could have. Uh but uh I would say that there's there's quite a lot of overlap between the labs in terms of what people look for and like yeah like when I posted uh my post you would you would see like you know people from both OpenAI and thropic saying like yeah like we agree with this advice and so you know I I think um that that's just a little bit of evidence towards that.

I think one reason for the the huge demand for wanting to go closer to AI research is because people are thinking oh software engineering is not going to be as important in the future. Is there a similar thought in when it comes to research where LLMs is also going to handle a lot of that work as well? So there's no reason to favor AI research versus software engineering. Um, so I think the the research skill set is going to become increasingly important. Uh so I would say like being able to handle stoastic components in the planning of your work is is just going to be a larger and larger part of how we approach our jobs. figuring out how to leverage AI in whatever thing you work on, which doesn't even have to be software related, is just an important muscle to start building right away. Um, because these components aren't deterministic. And thinking about how do I construct systems around these LLMs to do my job more effectively, uh, that's that's going to be the thing that sets you apart in the future. And I think that's true no matter what you're going to be doing. Look, I think I think there's there's FUD everywhere, especially with with some of the approach to marketing that some people have in terms of AI. It's FUD that is being intentionally leveraged. And so I I feel like people should really just focus on themselves and and trying to uh be more productive themselves. I I don't think that like AI is going to replace all of our roles. And so the reason for that is that one of the important aspects of what we do as humans in an organization which is really this web of trust from like you know this organization that is you know this pool of resources and this pool of people that manages these resources. One of the important things that we do is we allocate those resources towards c certain goals and um even when we can accelerate our execution there's an element of making decisions around how we allocate these resources that will always be something that needs to be attributable to a human making that decision. And uh that's simply because you can't hand off blame to AI. So we at this point have LLMs that really deeply understand law and they could, you know, review your contract for you or something like that. But they can't represent you in court because they can't be disbarred. And so that's that's I think like a a really you know sharp way that I might describe like okay this is why the legal profession will go on even though LLMs are really good at recalling precedent is you want to have someone who is responsible who can validate the output of AI to perform uh legal work more effectively for you rather than hand off your legal defense to an LLM.

"Yeah, I think the FUD that was actually the original motivation for your post."

"Yeah, I mean I I really think that the mindset that people should have is is a constructive one. And so there was a tweet that I saw I think by Dee that was like some long form you know fear-mongering about you know uh uh AI permanent underclass or something like that. And uh it's easy to get stuck in that loop, but I think the important thing to think about is like we all have agency over our future and we can start investing in uh skills that matter for tomorrow today and um that's that's really the only thing you should be doing, right? Like you know worrying about it is not going to not going to help you. And so part of why I wanted to write this post is is in response to that uh because it it was something that I could see echoed. You know, I gave a lecture at Princeton a while back and you know, a big question that came up is like, you know, how do I work at DeepMind? And and it's something that like uh yeah, just when people find out what I do, that's the top question people ask. So, I figured it would be helpful to add a little bit more constructive, you know, direction to the discourse here. One last thing on the post, cuz you know, if you think about getting a role, there's obviously the skills and we talked a lot about the skills and your fitness for the role, but there's also kind of the uh signaling for that role and like what is kind of valued if you were to be saying marketing yourself to one of these frontier labs. What signals um matter most?"

"Actual evidence that you've created something of uh of use to other people uh along the line of kernels, right? Like you can take any of the many open source LLMs that we have and optimize them. You don't have to make them better in every case. You could show that, oh, I have an improvement for this and that setting. It doesn't even have to be something that speeds up the model on GPU. There's all sorts of open- source stacks like VLM. There's a lot of other um things that you can do besides accelerating the LLM inference on device. The serving stack that surrounds LMS is a very sophisticated distributed system that has to maintain this KV cache memory and deal with uh all sorts of like load balancing and uh request queuing and and very common problems for for back-end servers. Uh and these projects are always looking for help. So, you know, contributions to VLM or SGLANG uh or demonstrations with Tensor RT uh they have, I think, a a a distributed system called Dynamo that uh allows for disagregated serving where you could show that you you made a project using these components, you improve these components like that would be an extremely positive signal uh for any candidate that I'm looking at uh and and a very welcome contribution to uh open source."

"I I think also a lot of what we said is kind of assuming the path of external hire into frontier lab. Um, but a lot of these frontier labs have large organizations that aren't necessarily doing the cutting edge uh frontier work. So let's say yeah for instance I mean you know Google DeepMind versus let's say there's some infrastructure that's working on search and they have the backend skill set maybe not as much domain context and they try to internal transfer to Google DeepMind does any of your advice differ in that kind of case for like an internal transfer versus uh someone who's coming from external"

"There's someone who I worked with closely on the search side who actually did transfer to my team uh Nate Linds and he's amazing and now he owns so much of uh like what we do on my team in terms of inference code design for uh like flash and flashlight and I would say like he's a really great example of this where his approach was you know how do I help my PA my product area adopt this technology as effective ly as possible. So I think there's you know definitely if you're in a organization that isn't directly generating these models but in some way trying to leverage them there's a very big gap in terms of applying these LLMs effectively serving them effectively within uh your organization and becoming someone who does that really effectively. not only creates a ton of value uh in terms of like the uh you know specific business need for your org which will definitely elevate you in your org. Uh but it'll also be the case that you're going to just naturally become the partner that we work with uh on the research side to make sure that our models are effective within your org. And so at that point, you know, you may or may not want to transfer. uh definitely if you transfer we'd you know be happy to work with you but like at that point I think you're you're you're already doing something that is cutting edge which is integrating this new technology into uh you know a real product that people use and so yeah that'd be my advice there is a towards the end of this post as we as we leave this topic you had the concrete invitation because I know you were hiring uh do you want to say what that was"

"Yeah, so I just trying to think of like, you know, you know, how do I put my money where my mouth is? Um, how do I demonstrate, look, this is a good way to show that you have, you know, at least some evidence of of like the the skills that I called out as important, you know, intent, mathematical maturity, grit. Uh and so I listed out a couple of exercises that demonstrate you know some initial knowledge of scaling laws, some willingness to get into the weeds engineering wise in terms of implementing a real transformer and uh sort of willingness to pick up the kind of bread and butter bread and butter math that we use uh every day to size uh these LLMs and uh you know I I won't I won't recall the full list of like the exercises that I expected here. But like uh you know if you do the detailed like uh handwritten uh version of the scaling book exercises and you know send me a video of yourself doing them along with the transformer exercise on my post then that's something if you can work in the uh New York office I would love to you know interview you for and quite a few people reached out to me about that. I actually already have had a couple submissions and we're proceeding with the loop with those people. So yeah, it's it's quite a bit of work, but uh impressively I got a response within like I think a week of posting. So uh it's definitely doable. Uh yeah, I mean I don't have unlimited headcount. So I mean the offer is on the table, but the you know I can only hire so many people. The good thing is though that is such a strong sign of you know self-development that not only is this a something that you should be doing for its own sake regardless of whether or not you will get a job at at DeepMind specifically but I think it'll be something that you know lets you basically prepare for interviews in other places. Certainly if you reach out to me with these uh exercises completed like even if you know I do all my hiring there's tons of people who I know who are hiring as well and I'd be happy to refer people as well."

Open AAI enthropic cursor and Verscell all use this product to make their lives better. And the problem it solves is when you're building SAS or an AI product and you want to sell to other companies there's all these requirements you need to meet. There's SSO, there's SKIM, there's arbback, there's audit logs. These are all things that take time to integrate but aren't the main focus of your app. Work OS is an API layer that lets you meet all of these requirements in just a few lines of code. So, let's say you have a new SAS product and you want to sell to other companies. Work OS will solve all of these critical feature gaps for you. You can check them out at workos.com to learn more and get started. and I appreciate them for supporting my work and sponsoring this podcast.

On the next topic, I mean, I saw you're the the the area lead for pre-training on Gemini, and I just thought it might be interesting to hear you give um uh kind of like a highle overview of what pre-training is in your words and maybe what are the the highle challenges in the area. We can talk about that.

"Yeah. So there's there's quite a lot of work that we do in pre-training um as an area lead for it. The specific things that my team is responsible for delivering uh include uh the flash model, the flashlight model. These are models that get used for AI overviews and AI mode in the search bar. Uh as well as some other uh oneP models that are used by different orgs like ads and YouTube. Besides this, we're also key technical PC's for the uh Google Apple partnership. Uh and so we do technical work there. Those are the actual like product level deliverables uh for my team. Uh beyond that, we do research to make sure that these deliverables are state-of-the-art and also we do general pre-training research that contributes to the Pro Series model as well. And the nature of the research I would say generally breaks down into three different verticals. There's distillation which I mentioned earlier. There's what I like to call inference code design. So uh creating neural architectures that are efficient uh to run inference on. So coming up with the network topology, the shapes of the matrices that the matt moles uh use uh inside of uh uh gating and linear layers for this transformer as well as the attention shapes, num heads, that kind of thing. So that that is effectively utilizing the hardware that you're serving on. And then the final uh pillar here is new quantization methods. And so quantization is just something that's uh been near and dear to my heart that I've been working on the research side for ever since I joined Google. And it really changes what's feasible for uh the first two. So uh that's why you know furthering the state-of-the-art in terms of how you can compress models is is also a very important pillar in the research that my team does. Generally uh uh quantization uh refers to reducing in some sense the size that the neural nets take up uh in order to represent their weights. So typically a neural net when you're training it uh is represented as a a series of numbers that make up the matrices inside of the neural net uh that are stored in FP32 32-bit floatingoint weights. Um, it turns out that when you do these computations, you don't need all of that extra precision to still maintain the quality of your neural net. And you can with pretty simple methods reduce the precision at which you store these weights down to four bits. So uh all of a sudden this huge range of numbers uh that we would take you know this float 32 to represent uh something that gets you down to like you know seven digits of precision uh can you know with somewhat high fidelity uh still be um uh represented well by 4bit ins which you know just cover this uh tiny range of like minus 8 to 7 and um It's it's kind of a miracle that you can do this. But what's even more of a miracle is that you can apply these kind of quantization transforms to the runtime activations that the neural net processes. And as soon as you do that, the actual math that you're performing because you're taking much smaller operands to your map mole, the amount of electricity that it takes to compute the neural net drops significantly. And what's interesting is that like 99% of the total cost of operation for AI hardware comes from the uh power that it takes to run these chips. And so if you can do these operations, you could just make neural nets run more cheaply, run more efficiently. That helps uh uh in terms of like serving more requests and helps in terms of latency. So the name of the game for quant research is how do we push the frontier beyond like this like 4bit range.

There's this take that I see on Twitter all the time um which is just talking about MFU and someone who's not in the space or model flops utilization. Someone who's not in the space they see a number in the low tens and they think wow they're wasting all of those GPU resources. Um, I was curious if you could just clarify that for people why a low MFU or I guess naively low is actually not low at all and maybe also explain what MFU is.

"Yeah. So when we compute MFU, you want to divide the actual number of flops that the neural net is performing here by the total number of flops that the accelerator could have done in the time of your request. And so in some sense this is giving us the uh percent of time that we're usefully utilizing the flops rate of the accelerator. And to get to 100% MFU, you would just need be need to be fully utilizing uh the matmo unit of uh whatever accelerator uh you're doing here. So it would just have to be doing like a bunch of matt moles in a loop uh without reading any memory or doing any other operations. That's not a very useful computation. Uh and in practice, neural nets have to apply activation functions or do attention or write intermediate outputs back to uh HBM. All of those different operations will require utilizing the memory bus or utilizing vector processing units. uh or simply they might be a mathematical operations that the underlying hardware performs more slowly than they than uh it might perform a maple. And so all of those things contribute to not running at the full speed that the processor is rated at. And so that's why you might not see 100% MFU all the time is cuz you know part of the time your neural net was you know reading and writing to memory or part of the time it was doing an operation that uh you know fundamentally runs slower than certain other units on your on your device. And I think quite a bit of this inference code design work that I talked about earlier is across all of the different um capabilities of the chip. So uh communication to other chips um memory bandwidth the speed at which we can read parameters for memory flops of course uh this can be metal flops this could be flops for processing uh vectors. So like things like doing activations uh all of these have different rates in the hardware and a given computation isn't going to match the natural hardware's rate uh of each of those operations. So when you design a neural net, you want to be able to choose shapes for this neural net that fully saturate all of those hardware units to get you as high of an MFU as possible. um when you are doing uh inference here. What makes this more than just an algebra problem is that those choices translate to different quality outcomes when you actually train this neural net. So the process of this kind of inference code design is how do we come up with neural architectures that scale predictably have a good prediction so are high quality and still make the MFU as large as possible during inference. And so this kind of joint optimization is what makes uh inference code design really fun. uh and also this kind of evergreen problem because as the hardware changes all of those relative constants of flops to memory bandwidth to communication bandwidth change and those will have different implications to what's the optimal neural net shape should be."

On another topic, Google has this idea of a spot bonus where someone can kind of give you a a oneoff lump sum of money as a thank you for like good performance. And I I saw on your resume that Jeff Dean, the legend himself, gave you a spot bonus. And you know, if you can tell that story, I'd love to hear why did he give you a spot bonus.

"Yeah. So, that one actually was at the very beginning of the Gemini program. Uh he gave out his spot bonus to people who hopped on and launched the first version of Bard. And like I had a you know very small contribution to a very very large project at the time. I helped with uh SFT for uh one of the first versions uh of uh supervised fine-tuning for one of the first versions of uh Bard that got released like right you know the biggest lesson out of uh that experience was you know at that time I was just doing like pure research in um uh Google brain and I was super focused on just how do I maximize the number of first author papers at Nurib Ciclair and I remember distinctly thinking like I I had this instinct of like oh like you know should I just keep my head down and try to write more papers and luckily at the time uh my my manager Roana Neil like he really encouraged all of us to get involved in uh you know this space and that was just the right motivation that I needed to like roll up sleeves, do a bunch of hyperparameter tuning and engineering work to get uh this uh model running on uh uh some like really old TPUs to get some extra you know cycles in for for uh SFT attempts. that very small initial engagement that was recognized by Jeff Dean I I think blossomed into more and more investment on the LLM side by me and ultimately led me to where I am today. Uh so yeah, I would say you know it it's less so about you know you know how much that like SFT helped the initial release and it's much more about uh uh recognizing that like there there's quite a bit of work some of it not glamorous some of it just like you know hyperparameter tuning and golfing the XLA compiler to make your program fit in a certain memory amount that contributes to a wider business goal that is is really quite important for getting involved in in very high value projects."

You've been working on Gemini for a while now and because it's a top priority, there has to be some, you know, incidents or war stories that you've been involved in. So, I'm curious, you know, what's your favorite uh war story when working on Gemini?

"So, I think my all-time favorite would have to be Flash 2.0. Uh, so this one this one was quite a challenge and a very long journey to get there. But uh, one of the main things that we were optimizing for which which Flash 1.5 established is this category of very fast low latency model that's still quite good. Um, and you know, in particular, it has to be fast because it's it's used by search to serve uh uh responses in in AI mode uh very quickly. Because of that uh for flash 1.5 and before we we focused on dense models which uh allow you to respond very quickly even though at the time we we knew about models and how they increase capacity and so I think um one thing that like came up was okay like we sure would like to use this new architecture but it it's difficult to just simply switch"

To ANE. Because what happens with ANE is it uses a lot more parameters in general. And because it uses more parameters, it takes up more HBM. These chips that we serve on have a finite amount of HBM. So you have to shard them across uh multiple different chips. So if you have, you know, whatever N experts, then you might shard it across N chips or, you know, some factor of N.

And what this causes is a lot of communication in the middle of the model when you have a token that needs to be routed to an expert, and that token might live on the first TPU but it needs to go to the last TPU. That's a lot of communication that you're inducing in the forward pass. So the latency of this operation like increases dramatically with N. Uh, and uh, you know, the challenge with ANE is they increase N. So, uh, that that that like kind of really bottlenecked this approach.

And one interesting thing that happened was, uh, we we definitely knew about, uh, pipeline serving for a while. It's just in the dense case, uh, it never really ended up mattering. Like I distinctly remember a very early conversation I had with Shto about it, and Shelto's like, "Oh, yeah, you're like so flop bound and so pipelining is just not going to change your prefill profile." And and he was right. I tested it out and like then abandoned the idea.

But what's interesting is, I I had a very small team at the time and and one of my reports, uh, Gen Yan, uh, had a very nice idea. He was working with Rahul Aaria and a couple folks from, uh, the Israel team at Google. And that was to apply pipeline prefill to pipelining. Pipelining is a technique where instead of paralyzing those N machines, experts across those N machines, you paralyze layers across those N machines. So instead of on a particular layer, you have to route tokens from machine to machine, now one layer does the computation for one subset of your prefill request and then hands off, uh, the processed tokens to the next machine to process the second layer, and then the third layer, and the fourth layer. And all of the experts can then stay resident to a single machine or a smaller set of machines.

So, uh, what this does effectively is it changes the communication pattern from something that required a lot of token exchange on every single layer to, uh, something that actually can be, uh, hidden behind other computation because you can do this, uh, pipeline prefill across different parts of your request. Uh, so, uh, while layer two is working on the first thousand tokens of your request, uh, layer one on the first chip, uh, is processing, uh, the second thousand tokens of your request. So it was a way of breaking this HBM constraint by moving layers across the machines rather than moving experts across these machines. And because of that, the communication overhead has gone down, and all of a sudden latency looks really attractive.

Now, this, you know, the the Gemini 2.0 report says like it's an ANE series of models. And the thing that made that possible is, you know, or one of the things that made that possible is is this, uh, uh, you know, serving time innovation. Darkesh and Reiner have an amazing post about exactly this optimization that you can write up in the algebra of the scaling book. And it's just a wonderful example of how, uh, this kind of change can, uh, have really dramatic implications on LLM quality. What really made Flash 2.0 rewarding is this, you know, giant decision. And it sounds like a small technical decision at the time, but people were really worried about whether or not the latency of this would, uh, actually be reasonable.

Luckily, I was able to run like a very transparent technical process to get to the bottom of this. And by the end of it, uh, you know, we we made the right call. Uh, but then we had to train it. So this was a bigger model than we've ever trained before at the Flash scale. And like we knew this would be the right call, but it was just going to be 40 days of grueling work for like a really, really small team. Like we probably had like five people on the rotation for training this model. I remember, you know, all of us just kind of like rotated day by day, handing off like, you know, all of this like SRE style work of, uh, keeping the training job alive, which at the time was was a very interactive thing, cuz, uh, you had to make sure that everything was moving stably, that, you know, you have tuned data iterators that aren't slowing down your job, that, you know, if there's like a gap in the data somewhere or or an indexing issue, you have to like really quickly put up a fix because it's, you know, wasting all of this GPU time.

>> What about at nighttime and on the weekends?

>> So, yeah, like I think, you know, for those 40 days, we did not do a lot of sleeping. Like we had to like do like kind of these dual shifts across like the Paris office and Mountain View. And like the thing that makes it so rewarding was when this model came out, like around the same time, uh, DeepSeek V3 came out. And, uh, the Wall Street Journal put out this article that was like this giant Red Scare article about how China's going to take over AI with open source models. And I remember my friend sent me a screenshot of this table of the LLM arena leaderboard. And, you know, all the way at the top right, you've got, uh, ChatGPT. And and, uh, DeepSeek right behind it. And like, oh, DeepSeek was trained for whatever, few million dollars, you know, and and like they're right there. Uh, and then my friend was like, "Oh, like, you know, Gemini is so behind cuz they had, you know, a version of like, I think 1.5 Pro or something in that table at the very bottom."

And then I looked at it. It's like, "Oh, that's really interesting. I was just looking at this leaderboard cuz we just released a model and it definitely doesn't look like that when you go to the website." So, turns out there was kind of some, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,

uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,

uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,