10 Years of UX Research Experience in 4.5 hours - Ultimate Crash Course

I spent over 10 years as a UX researcher working at companies, the biggest ones: Google, Uber, Volkswagen, StubHub. I've worked on big teams, small teams, midsize teams, and even consultant startups. I've learned a lot. I've been helping thousands, I've lost count, thousands of people learn the fundamentals of UX research and transition into the field.

And now, I've distilled some of the most important fundamental lessons that I've learned into this ultimate crash course for UX research. So, this video is, is actually a preview, a short preview of a collection of the trainings in my master class. My full master class is over 100 hours long. So, check out the link in the description if you're interested in learning more about the full class, which I spent 5 years or more, actually, in monk mode. I quit my job to create that training. That's how serious I am about proper UX research education.

Some companies and teams have adopted my templates. They literally show my videos to their UX research teams. So, I'm going to include my templates for free in the description below. Let's show them how it's done.

I've split this video into five main sections. You're going to walk away with a better understanding of the context of UX research. I called it the intro, and you might be tempted to skip it, but too often, I see people get mixed up with certain terminologies in our field. You know, people using trendy words. I'm going to attempt to clarify those once and for all. You'll learn how thinking like a product manager, speaking the language of the business, is a key to being a strong UX research leader. That's what business acumen means. You'll learn how to do what is needed and not do what's wanted. If you've ever never been confused what a UX research roadmap is and how to make one, I'll, I'll share my screen. I want to show you step by step, every little thought, every little consideration. I'm going to dissect my brain for you. I'll guide you through it. You'll learn how to gain influence, no matter your title or level. It doesn't matter what your title or level is.

Over the years, I've learned the term leadership. It's overrated. Like I said, I wish I could say this is everything you need to know about UX research. It's not. It's not everything you need to know. There's a lot more, like how to say no, even when it's a CEO, how to stop managing stakeholders and collaborate with them instead, how to do B2B recruitment, how to make your data analysis smoother. So, so much more. But I picked five main topics for us to get started. After all, it's supposed to be a short crash course. I told you this video is just a fraction of my entire course. It's just the beginning, but we got to start somewhere. You're not going to learn this from any other UX course. You're not going to learn this from a master's program. This is years of real-world tested knowledge, stripped of all the noise and fluff, so you can get real results. Let's have some fun, shall we? Let's get [Music] it.

What is UX research? Now, seriously, what, what is it? Can someone tell me? No, I'm just kidding. I will do my best to define what UX research is and what it means. Is it a job title? Is it a way of thinking? Uh, or is it something else? What does it mean to do UX research? What does it mean to be a UX researcher? So, we'll try to dissect this term. Now, before we get into what UX research is, just being aware that maybe a few years from now, it might be called something else, or there'll be groups of people who want to specialize and call it something else. Who knows? The title keeps evolving. Whether it's UX research, user research, customer experience (CX), service design research, experience research, human factors research, or something else. For all intents and purposes, I will refer to what we'll talk about as UX research.

Now, in order for me to tell you what it is, let me tell you what it's not. It's not about designing wireframes. It's not coding or programming, unless you're a quantitative UX researcher who needs to manipulate data in R. You don't need to program. It's not A/B testing. It's not usability testing. It's not asking people what they want. It's not collecting data and pain points. Hm, wait a minute. Is this guy for real? Wait a minute. Maybe your eyebrows are raised. Sure, what I mentioned are methods that researchers use, but that's not the whole picture. And let's go on a little side quest, a little tangent. There's a little semantic debate on the term user testing. How you frame research as a practice to your stakeholders is going to be pivotal. As long as you explain to your team that we're not testing the user, it's all good, because we test our products, our services, with our users. You don't test the users. Got it? English is hard, right?

What did he say? So, what is it? It is the study of people's behaviors, needs, and motivations through systematic methods, with the purpose of guiding the design and innovation of a system. For those of you who did basic science experiments as a kid, you may be familiar with the scientific method. You have a research question, you conduct some background research (aka literature review), you formulate a hypothesis, you define your objective, design your procedure with certain methodologies, and once you've collected your data, you analyze it and you present the results. If you have ever read a scientific journal article or paper, this is pretty much the format in which they write.

Now, compare this to the UX research process. Wait, the research process is pretty much the same, except we do the research with specific business goals in mind, and we need to make actionable recommendations. The difference between academic research and UX research is, I'm just generalizing, okay? One is for the pursuit of knowledge, and the other is business pursuit. So, at the core, UX research follows the scientific process and is the foundation of product development. Without proper research, your design is just a pretty art project.

UX research is also not just about fixing existing problems and making them better. As much as we are problem solvers, we are problem spotters too, or put another way, opportunity spotters. As Henry Ford may have said, "If I asked people what they wanted, they would have said faster horses." We've all heard it before. Usability testing and evaluative research are great. We need them, but inherently, we are focused on an existing concept, an existing idea. To truly innovate, we need to discover new problems, discover new questions. With foundational, generative research, which we'll talk about later, we will identify opportunity spaces and work with our team, our product team, to develop a vision that will turn these ideas into validated business models. And at every step of the way, guess what's important? Research. Research is the Keystone, interlocking all those pieces [Music] together.

For all the activities, we draw from a multitude of different disciplines: from human factors, psychology, business strategies, leadership, statistics, ethics, and of course, design. I hope you can see by now that UX isn't just a job. It's a skill. It's a competency. It's a way of thinking, a problem-solving process. That's what it is. And that's why I don't really care what it's called. Because at the end of the day, if we let go of the name, let go of the term UX research, CX research, whatever, at the end of the day, it's all about understanding people to unlock opportunities to better improve products, services, and processes. So, the next time someone assumes UX is about making things pretty, you hopefully now have the vocabulary and philosophy on how to shift their perspective. Who knows? Maybe you've done UX research in some capacity already, just without the UX title.

Let's talk about why UX research. Well, let me start by telling you a story. In 2015, a San Francisco-based startup raised $10 million in venture capital. The product was a Wi-Fi-enabled, voice-controlled, sleek, sexy-looking product. But four years later, it failed. No, that company was called Juicero. They built a Wi-Fi-enabled, voice-controlled, sleek, sexy-looking, polished turd for $700. The machine helps you squeeze juice packets, and not just any juice packets, Juicero juice packets. You couldn't use any other ones, and each of those packets cost $5, $7, just to squeeze a few more bucks out of you. No pun intended. But wait, there's more. Are you kidding me? That this is the silliest part. You had to be connected to Wi-Fi for it to work. No Wi-Fi, no juice. It didn't take long for people to realize that you could just squeeze those packets by hand. No expensive juice machine needed.

Now, why do I tell you the story of Juicero? Well, hey, it's a sleek-looking product, and it looks pretty easy to use. There's one button. But that's just it. Doing research to improve a product's ease of use is just one part of the whole. Good UX doesn't mean it's valuable. Was there a real problem that needed to be solved? Now, some of you smart folks might point out, "Well, Kevin, what about people with Parkinson's, where people without the hand strength to open the packets?" Mmm, very keen question. I like where you're going with it, but we should be asking a different question: Is the problem with the machine or with the juice packets? Then, mmm, anyway, come with me to the other side of the bay, and I'd love to tell you about our next story of when user research is used and why it's important.

This is the MRI machine at the University of Pittsburgh Medical Center. The room itself would be dark, with those flickering fluorescent lights, right out of a horror movie. And for children, it was a scary experience. Kids would cry, they'd get anxiety, and wouldn't want to go in, rightfully so. And even when they got in, they wouldn't be calm enough for the machine, often leading to doctors repeating the scan. Now, this took a lot of time and money for the hospital, and it wasn't pleasant for the kids. Now, if the staff only gave Band-Aid solutions, they might have enticed the kids with candy or told them, "Oh, it doesn't hurt, I promise." Okay, yeah, look at the machine, it's going to eat you.

So, you know what the researcher did? They went to a design thinking workshop at Stanford University. He learned to build empathy with the patients and the parents. He identified those anxieties and fears associated with the MRI machine. Then he prototyped something called The Adventure Series. So, what used to look like this, turned into this. It wasn't a fancy tech solution. It didn't cost hundreds of millions of dollars. It was simply art, but art that solved the problem, removing fears in children. It helped the hospital not have to do those repeat scans anymore, not have to use anesthetics, which meant that you were also saving time and precious mula. And the kids loved it so much that they wanted to go back inside.

So, those two stories I gave contrast what happens when you don't do research and what you're able to accomplish when you do do research. Do you, with UX research, we can design and develop with direction, answer questions, but also inspire new questions and opportunities, help people achieve their goals, and build stuff that people actually need. The right way. Yeah, that's why UX research is crucial.

Just who is a UX researcher? You, you, bass UX leader, right there. Well, UX researchers come from all kinds of different backgrounds: the arts, design, social sciences, pretty popular, physical sciences, economics, data, English. Your homeboy has a biology and psychology degree. What background do you have? And it doesn't matter what background you have. I'd say it's a good thing that people have different backgrounds and different perspectives because everybody brings something different to the table, and that's what makes UX research so great. Okay, all right, enough of the pep talk, though.

What do UX researchers do? And how do they do it? We identify user needs, behaviors, and motivations. Inspire ideas through data and storytelling. We uncover what makes an experience intuitive, accessible, and delightful. And more importantly, we bridge a gap between our users' voice and our stakeholders and our teams, so that we are the advocate for our users. That's what we do. But how do we do this? Well, through conducting primary and secondary research. We look for patterns and insights to inform product decisions. What we do, we hold the flashlight and we shine the light to guide our teams on where to go. And speaking of team, we collaborate with our partners, our stakeholders, to develop valuable products and services for users to achieve their goals. As, as the bridge between stakeholders and users, we hold a unique position to be both a leader and expert influencer.

And some of the qualities that UX researchers possess, uh, I mean, some of the things I've noticed in UX researchers are: they're people persons, you got to like people, curious, compassionate, not just empathetic, but you're taking the step to make an action, you're humble, obviously, a good listener, ethical, collaborative, inclusive. You want to work with others. Sometimes you need your alone time, I get it, but we got to work with our teams too. And probably the most important quality to have in any kind of business setting, but especially for UX research, is a tolerance for ambiguity, uncertainty, precarity. There's no need to, there's honestly no need to stress about something you don't know the answer to, because, hey, we can go find out through research. Get feedback, talk to your team, get ideas. It's going to be okay.

Some of the skills that UX researchers have are: they're master storytellers, they're analytical problem solvers, but also problem spotters, communication, communication skills, you got to have those when you collaborate, obviously. The technical skills of knowing your methods, experimental design, how to look for the right participants to do your studies, and not just knowing the methods, but knowing when and why you would use a method, that's more important, which we'll cover in modules four, five, and six. And finally, some product management skills, you know, tying in that business acumen. We'll learn all of this and more. Don't worry if you don't have the skills yet. You can always learn these either through this class, on the job, reading books, and all that. So, but if you have the right qualities, you're in the right place.

Up until now, we've talked about the UX researcher, who we are, what we do, how we do things. But we don't work alone. So, who are these people that UX researchers work with every day? What is a stakeholder? What does stakeholder mean? Uh, well, a stakeholder is someone who has a vested interest in something, a particular area, product, an initiative. People who are part of a project team. Our BFFs, best friends forever, include product managers, designers, developers, data scientists or analysts. These may be the four most common partners that you collaborate with on a product team. But everybody has different working styles. Every company, every team might have a different kind of partnership. So, just remember to meet your teammates, ask them what their needs are, what their roles are, what kind of questions they have, and how you can best collaborate with them to help drive your product or initiative.

Let's start with the product manager. They are the crucial stakeholder to UX researchers in any product team. We might work with them to define product vision and the goals and objectives, uh, and why there's a particular business case to tackle a certain initiative. They also look at user opportunities and they create accountability within the team to execute, uh, go-to-market strategies for the product. What they need is to make informed decisions for product strategy to maximize confidence and minimize risk. Maximize confidence, minimize risk. So, research is going to be extremely crucial to what they need to do. Their goals are essentially to deliver impactful, successful products for both the customer and the company. So, that's the gist of a product manager.

How do you collaborate with them? Well, since they need research, they need validation, they need confidence, research will come into play. So, sync with them to define goals, the scope, the objectives, the questions early, and get alignment on those topics. Don't just execute what someone tells you to do. Ask those questions, which we'll cover in module four, but ask those key questions to make sure they truly have the right problems in mind. Involve your product manager throughout your research process. Invite them to your studies, have them watch participants, uh, get feedback on a study plan, analyze data together. Like, do everything, basically. The, it's, it's our research, not my research. Okay? Um, and by the end of it, nothing, nothing should be a surprise, right? And by the end of it, really, you have to give recommendations on, on clear actions to take next.

UX designer, product designer, whatever you want to call it. They are a researcher's best friend. They are responsible for crafting prototypes, wireframes, based on research needs and goals. They also work with research to develop many deliverables, like personas, uh, user journey maps, and, and whatnot. What designers need to know are data-informed decisions on design. What's the best design for the US user? And solid user research will help guide them here. Their goals are to determine the best design approaches that are useful and meaningful to users and meet business needs. Right? You kind of see how everything falls into place between business and users' needs, and research is the Keystone that holds every piece together.

Now, how do you collaborate with UX designers or product designers? They might have project questions or requests. Sync with them, just as you would with a product manager. Sometimes you might even include all three in the same meeting, uh, or everybody in the same meeting. The more, the merrier, okay? Get alignment, same thing you would do with product managers. Involve them in your research. To summarize, the designer is someone who will take your research insights, your recommendations, and bring them to life. Now, if you're at a startup, you might be doing both, right? If you're at a startup and you don't have a specific person to do each one, you might be doing both.

Let's talk about the developer or engineer. Software, they take product and design requirements and implement it into the code, and they actually build the product. So, they're act, they're really good source of knowing what is feasible or not. Uh, and in many teams, they include front-end, back-end, full-stack, mobile platform, so like either iOS or Android engineers. They have specific focuses. They really need exact design and product feature requirements. So, research will help designers know, know what requirements there will be, and then the designers will kind of hand it off to the developer team. But I think that user researcher could do a good job of helping build empathy and reminding the developers like, why we're doing it. The user stories that they come up with might not be real user stories that users care about. So, that's why research is important to kind of just bridge that gap. Remember the theme of the bridging gap. Their goal is to implement those requirements into code and start building.

Just a side note, developers might need to build things quickly, and we hear stories of like, "Oh, you know, teams always build first and then test later," which is why UX is so bad and you have to save time. The reason why developers need to build quickly sometimes is just like the business or like leadership forces them to come up with something. Like, "We need to, we need to come up with something quick." So, don't blame them that, you know, blame the low UX maturity in the company. Okay? So, that's the gist of the engineer or the developer.

How do you collaborate with them? Same process. Depending on the function, they can tell you if something is feasible to build, uh, or how resource-intensive it will be. Um, so then you can take those points and prioritize what can we feasibly do now, and what can we feasibly do later? So, give recommendations to design, and the designers will also give requirements to engineers.

Finally, let's talk about our fourth BFF, the data scientist, aka analyst. The data scientists or analysts, they're separate from the product team, but they're invaluable. They track hyper-granular user behavior to make inferences, projections, correlations, predictions, and trend analysis, and they might visualize data with a lot of models, okay? So, they, they know what's going on at a large scale. What they need are data. Just like we need data. They focus on quantitative. They focus on the "what" is happening. When we triangulate with quantitative data, we build the whole picture. We give color to the "what."

So, how do we collaborate with data scientists? They might not be as involved in your research, but they know what's going on. Sometimes they want to find out why. So, sync with them and look at the data together, right? They know what, and you know why. This way, you might be able to distinguish expected user actions from actual user actions. Uh, you will cross-reference your samples and your main questions, making sure the way you collect data are going to be a little bit different. Uh, so, just ask those questions first, like, "How, where did you get the data? How did you get the data? Who did you sample?" Because it'll be a little different from how you do the qualitative side. Okay?

So, we went through the four most common stakeholders that you might work with on a product team. Um, but at the end of the day, remember, just meet your teammates, learn their needs, understand how to collaborate with them, and go from there, because every team's going to be different. Involve them in your research.

What's a day in the life of a UX researcher? Before I jump into it, I do have two YouTube videos that are, you know, they're fun, more lifestyle videos about what the day in the life looks like for a UX researcher. One is pre-COVID, and one was during COVID. Um, so, go check it out on my YouTube channel, "Zero UX Day in a Life." With that aside, a day in the life, a week in the life, it varies greatly. Before I transitioned into UX research, I thought the majority of the day would be conducting research. Woo, researchers conducting research, who would have thought? I thought maybe 60 to 70% of my time would be, uh, talking to users. Well, in reality, most of your time is going to be spent meeting with stakeholders, planning, planning about meetings, meeting about planning, scheduling research, scheduling participants, analyzing data, writing up your report, and then having more, you guessed it, meetings. Meeting, meeting, everybody, meeting. Okay, all right, maybe it sounds like I'm complaining about meetings, just a little bit, but I just wanted to set the right expectations. Meetings are fine, I guess, but research is finer, which sounds like funner, because it is.

So, here's a couple of screenshots of my weeks at work, uh, from two different companies. And, you know, every week, it's going to be very different. Some weeks, I'm not doing any user studies. Like, some weeks, I am. Full days of doing research, daily syncs, daily stand-ups, team meetings, weekly meetings, one-on-ones, all-hands meetings, like different kinds of meetings. So, yeah, it's going to be a little different. But just to give you an idea of what a week in the life might look like.

Now, now, for a specific project, let's say you're doing a research study. Um, here's maybe an example timeline for a project. By no means is this representative of a lot of projects, but based on my experiences, a lot of them are going to be maybe a two or three-week turnaround time. If you're doing longitudinal studies, like a longer diary study, or you're planning contextual field research, yeah, your timeline's going to be different. But let's, let's just use this as an example. So, day one of week one, you're planning research. You kick off the study. We'll talk about how to do all these things in later modules. Uh, you plan a research, you start recruiting participants, you write it up, you finalize about a second week, and then you run your studies. A lot of debriefing of your stakeholders, tweaking your study plan if you have to, and then you're going to spend some time analyzing your data. And then by the end, you're going to read out your recommendations, your final report, and there's your project.

Now, this is all fine and dandy, looks so smooth and streamlined, but in reality, it looks more like this. So, while you're running one study, you might be wrapping up a second study. While you're running the study, you might have to plan your next study. So, we UX researchers are juggling projects here and there. Sometimes I'm doing three projects at the same time. So, it really depends on your team's needs, your priorities, and things like that. What I'm trying to say is, your schedule can be hectic. Therefore, it's important to have time management, prioritization, and communication skills, all of which we'll talk about in later modules.

The research team structure. Typically, there are two types of models. The first is embedded, and the second is more of a consulting, centralized style. In the embedded model, this means you work on and support a particular team. So, you're part of one team. The pros are that you become an area expert on that team, on that topic, and you get to build closer relationships with your team. You can also see projects go from being a little seedling into launching into the real world, right? This way, you can track your impact more readily. You can see how things change at each step, and you can make those changes through research. The cons of the embedded model is that because you're working on the, the one team, on the one product, you don't really get the context or the bird's-eye view of what's going on cross-functionally between teams. And really, your users don't care. Like, your users don't look at your product and say, "Oh, that's this team's work, that's this team's work." They look at your product and they see one holistic experience. So, working on the one little thing, you might not get that bird's-eye view. So, you will have to rely on collaborating cross-functionally to get that context.

The second type of team structure is this consulting or centralized or studio model. You're kind of like a, a project consultant, basically. You live here, but you support multiple different teams, right? So, one day you can work for this team, and next day you can work for this team, and so on. The pros of this are, you get to work on many different kinds of problems, right? Many different kinds of products, potentially, and build relationships with many different teams. The cons of the consulting model is, you know, you do the research, get the insights, and you make the recommendations, and then you kind of say, "Bye." So, it's harder to realize whether your research actually made an impact, like whether or not they implemented the changes or not. So, it's a little harder to track the impact, I guess. The only way to do it is reach out, you know, see, "Hey, you know, how is the, how's the project going along?" So, those are the two different kinds of models. One's not better than the other. It's just what's needed for the team. For example, when I was working at Uber, I was embedded on a particular team looking at trucking companies. I got to see in-depth one side of the trucking experience. So, I had to collaborate with other teams to know, like, "Oh, well, the trucking experience spans different parts of the user journey, and we're only focusing on one part of it." At the end of the day, the user's experience spans the holistic product or the service, right? It's agnostic of how we define our teams.

Now, those are the two main ones. There is one more that I've worked on before, and this is a combination of both embedded and the consulting model. So, when I was working at Google, I was embedded on the wearables team. So, I only did research for that team. However, within that team, there were multiple sub-teams, and I would support all of them. So, in a way, I was embedded on the bigger team, but I was consulting many of the sub, the smaller teams. So, that's a hybrid of both. So, we just covered how researchers fit into different team structures: the embedded model and the consulting model, and a hybrid model as well. At the end of the day, neither is better than the other, just whatever team prefers. There are pros and cons to each. So, just make sure to collaborate cross-functionally with your partners to get the bird's-eye view of your entire user experience.

So, you're learning UX research, backpack on your back, books in hand. You're learning different types of methods, reading Steve Krug, reading Nielsen Norman articles, and then you come across different terms for types of research: generative, formative, strategic, and then you come across evaluative, foundational, summative, exploratory, tactical, need-finding. Have in you suck. When I first started learning UX research, I was like, "What the heck?" I read multiple articles on these terms so many times, and I still got confused. How to do generative research? How to do foundational research? How to do discovery research? And I'm like, the meat of the articles, they all talk about the same thing, but all called it something different. So, I was like, "What am I missing something here? What, what's the difference?" And I kept trying to reconcile the differences. I'm like, "They have to be different because they're called different names, right?" And so today, I'm going to define these terms once and for all, so that you don't have to struggle the way I did trying to reconcile the differences between these terms. And I'm going to show you how they fit into the product development process, why they're different, at what stage do you do some of these research. And oh, yeah, we're also going to learn about the product development process, and then I'll define all the different types of research along each phase of the development process.

All right, so taking a look at this slide, we're going to start with a development process that has no business model. There's, we start at zero. We identify a problem or an opportunity. Anytime there's a problem, there's also an opportunity. And then once we do some research, generate some ideas, we're going to start testing concepts. We're going to have concepts in mind. We have, at this point, concepts. So, they're unvalidated business models. And at every step of the way, we're going to do research. Research is crucial. Research is the Keystone for every single step of this process. And once we test these concepts, we're going to have start, we're going to start solutioning. Once we have solutions, we're going to evaluate them and test them out. The ones that make sense, we're going to start actually deciding on those and test at scale, and showing the business value of that, uh, development. Therefore, we're going to have a validated business model. Does it make sense to users? Does it make sense to the business? If so, we're going to decide on that, ship it, get feedback by doing research, iterate at every one of these steps. Eventually, going to market, launching it, right? And once we launch, we're not done yet. Once we launch, we still have to evaluate this. Summative research, which I'll explain. But once you launch, you're going to want to measure what is going on in the real world. Scale, is it meeting user needs? What are the, uh, reviews people are leaving? And then can we get reach out to into new markets?

And I've separated this development process into two main buckets of research: foundational/generative research, and then evaluative research. Coming down here in the map, we identify a need, some user need or business need. Define what it is that makes this project or product successful. Start learning about our users, understanding, deep dive into our users. What that existing market looks like. What about our competitors? What are they doing? What's that problem space look like? We're going to start brainstorming ideas. And this product vision, we're going to learn more in module three of how to craft a UX strategy and product vision as part of that. And then we're going to have defined those business values. And then you're going to see these diamonds. Now, if you learned Double Diamond from IDEO, well, you can do Double Diamond. But in reality, you're doing triple diamonds, quadruple diamonds, quintuple diamonds. You know, the process really depends on your team, your product, your launch, your deadlines, and everything else. And each one of these steps, research is involved. Every one of the steps. Measure, learn, pivot, getting feedback, new ideas, building, getting feedback, iterating, new ideas, meaning iterating, decide, getting feedback, shipping. So, every single step, research is involved.

Now, with that in mind, this is the product development process. Let's talk about the types of research. Now, bucket types of research into two main ones. See the card sorting right there. The first is strategic research, which is the same thing as foundational, generative, exploratory, need-finding, pathfinding, discovery, all the same. They have so many different names. This is what I was so confused about. On the other hand, we have tactical research, aka evaluative research. Same term. Now, evaluative research has two kinds: formative and summative. Uh, strategic research focuses on a big picture, long-term, adaptive goals. It's exploratory in nature. And this use, this type of research usually includes understanding who your users are, building an understanding of something. It could be your user behavior, market opportunities, sentiment, motivations. It is used as the foundation of everything else. On the other hand, tactical research is more of that nitty-gritty stuff. It's the implementation of design, specific design recommendations that all serve to achieve the overall product or business strategy. And these, this is what we know as usability testing, A/B testing, um, where we have prototypes, we're designing, and we're evaluating these concepts.

Foundational research, as I mentioned, so many different names. This is focused on exploring that topic or problem that hasn't been identified. It's foundational because you're building up the foundation of knowledge on your users. Our users don't tell us what the problems are. It's our job to synthesize and spot those problems through research. And we can do them by interviews, contextual inquiries, observing what people do, when, where, why, how. Evaluative research is testing an idea, concept, or prototype with the purpose of improving an existing idea. So, we're trying to see what works, what doesn't, and why. In module seven, we'll talk about something I dub the research death spiral, the tactical research death spiral, which I've given talks at conferences several times on.

Now, let's break that down. Evaluative research, there's two types. There's formative research, and then there's summative research. Formative research is what we know usually as evaluative research. So, like usability testing, heuristic evaluations. This kind of research we do often in early. All right, so when we talk about tactical evaluative research, usually we're referring to formative research. Formative usability testing. Summative evaluation focuses on the overall experience, usually this is quantitative, like once you launch something and see how it performs in the real world. Now we have data. Now we have users using our product. We can actually track things. So, we might track things like, like metrics, different metrics, sentiment, satisfaction, NPS, whatever we define as our success metrics. H.

There is one more worth mentioning. This is called concept testing, or you might hear it being called concept validation. What is concept testing? Is to see if an idea resonates with users, to validate or invalidate solutions, or to see if the idea is valuable or not, to find a product-market fit. Now, concept testing, it, it has been really confusing to me because I thought it was a method at first. Concept testing itself is not a method or it's not a methodology. And teams might call it validation. Now, this slight nuance is very biased. Okay? When teams try and ask you to validate something, they're trying to tell you, "Go find out how to make this work." That's biased. Our job is to do both. It's what works and what doesn't work. Our goal should also be not just to validate, but to invalidate concepts. And that's something no one taught me, but now I know. It's not just validation, it's invalidation.

Concept testing looks to answer these kind of questions: Do users care about this idea? What works? What doesn't work? You might run concept testing after you come up with ideas, after ideation sprints that are based off of research, um, or you can just still test prototypes based on assumptions or best practices. How can you concept test? Well, like I mentioned, it's not any one method. So, you can use any kind of method to test your concepts depending on what your goals are. You can do surveys, interviews, focus groups, participatory design, usability testing. Just be sure to be clear on your goals. If it's just an idea, you shouldn't care about the usability just yet. Just whether the idea works or not. That's the purpose of concept testing. Don't usability test it. It doesn't matter if people can use it or not at this point. It's just, is it valuable or not? And it can also just be an idea at this point, right? You don't need prototypes for concept testing. You could just tell people what you have in mind, or you can ask people what they've done in the past.

Here's the map of all the different types of research, and now I've added concept testing. Now, we're almost done with this. What term should you use? I would say use whatever term your team is already using. Right now, for simplicity's sake, when you speak with leadership, you know, like directors or VPs, I prefer to use the word strategic and tactical research. It just sounds more busy. Now, on the other hand, you're speaking of product teams, you might use the words generative or evaluative research. You don't want to confuse everybody. So, just stick with what your team's already using. Pick one and stick with it. Now, we learned the different types of research, their definitions, and how they fit into the product development process, and now we don't have to be all confused with these terms anymore. We'll be using these terminologies throughout the course and beyond.

In one episode of Hotel Hell, Gordon Ramsay visits a hotel that's having troubles with their business. The hotel manager and CEO were wondering why they were hemorrhaging money. The guests complain that the rooms have mold, that the beds are creaky, and the hotel manager just blames the guests for being too needy. Gordon Ramsay, obviously in disbelief, uh, gathers a bunch of executives, the hotel executives in a room in a circle and asks them, "Who is the most important person in the hotel?" And at that moment, you on the TV, you can literally see the faces of the entire hotel staff. They obviously hadn't thought about the guests. They were so focused on increasing revenue that they forgot to focus on the core problems. I know, I know, how, how could they forget that? It's so obvious. We're learning UX, okay? We're learning UX and being, how, how to be more user-focused, user-first. But this notion isn't obvious for many businesses. I think things are changing for the better, but in the thick of things, you'd be surprised how many businesses forget about their customers.

Now, how do we make sure our investors are happy? How do we make sure our staff are happy? They forget about the users. How do we shove ads in a more user-centered way? Ads make money. If we just shove it in a more user-friendly way, that's good UX. So, that's good business, right? Am I right? Hey, editor Kevin here. There's a camp of people who argue we must focus on the business needs first because, hey, whether we like it or not, we live in a world where we serve our capitalistic overlords. If we work for them, they pay us. That's why we have a job. It's a good point, but I think it's missing the point. First, I ask, why is that the norm? Why should that be the norm? I own a business, and all I think about is you, the learners, the people I serve. And I want to teach. And if I can get paid for my efforts, amazing. And a lot of small businesses focus a lot on those service to the customers. When they become large, somewhere along the way, suboptimal structural incentives lead them to take shortcuts, cut things where they shouldn't cut. And large businesses fail too, just because I'm not a billion-dollar business doesn't mean billion-dollar businesses don't fail. Why? Because they couldn't keep up with what their customer demands or needs are. Anyway, why should UX researchers learn about business?

But first, what is a business? Uh, so, let's look at the interwebs and look at a few definitions. A person's regular occupation, profession, or trade. An activity that someone's engaged in. Commercial operation or company. What the heck are these definitions? Okay, here's another one. The goals of the business. That might be getting a little closer to defining it. The primary purpose of a business is to maximize profits for its owners or stakeholders while maintaining corporate social responsibility. Huh. The primary goal is to maximize profits for its owners. Instead. All right, I'm not satisfied with any of those definitions, especially the last one. Who even, who wrote that? Let's, let me share you a screen. Here's a traditional business thinking model where we have multiple problems, and you try to come up with one solution. Here's what we do: design thinking, and a little bit more of systems thinking these days too. We had to zoom out and think of systems as well, which we cover in this class. Okay? So, design thinking, we first understand the problem through abductive, divergent thinking. You diverge first, and then you identify multiple problems, and then you converge upon unique solutions, not a one-size-fits-all solution. In UX, we do this process and test our understanding. We test our solutions and we iterate and test those iterations. Okay? We, we know this as the Double Diamond approach.

So, going back to our original question, what is a business? A business solves a problem. Solves a problem. Just like design. Design solves a problem. Design for the sake of looking pretty is just art. Design in everyday life solves problems. And businesses don't exist without customers, unless it's a gym membership, 'cause, you know, you pay the gym membership, you don't show up, but they still make money. Knowing that you probably won't stick with your resolution after the New Year's, unless all of you, since you're bad at it, sh... leaders, do. But if you don't have customers, you don't have a business. So, the next time people want to push customer problems to the side, you remind them of this fact. Businesses exist because you have customers who believe in your solution, and you're solving problems for them. And if you forget about the users, you lose your users, and then you lose your business. Remind them of that fact.

Ask any product managers what they do, and a lot of them will tell you they have no idea what they're doing. Not for any reasons of incompetence, but rather their job entails so much that they sometimes just go by a different name. Unofficially, a mini CEO. They're essentially CEOs of their own products and are more strategic in nature. Uh, another role you might hear of is called the product owner. Product owners are more tactical, do the day-to-day execution of the vision. Some product managers are also product owners, but not all product owners are product managers. If you're still following, honestly, though, it has never affected my work what their title was, nor does it really matter what their titles are called. I just involved them all in the process. I mean, just look at our own titles: UX researcher, experience researcher, design researcher, product researcher, user researcher, UX researcher, etc. You, you get the idea. So, for simplicity's sake, I'm going to refer to, uh, product people that we work with as product managers or PMs.

So, what is product management? According to a last's definition, it's a function that guides every step of a product's life cycle from idea to development to execution to pricing. Kind of like conceiving a baby and raising it into adulthood. Of course, you wouldn't put a price on your baby. Except the average cost of raising a baby in 2022 to the age of 18 is, uh, about $300,000. So, you can put a price on your baby. Product management is the intersection of technology, business, and user experience. PMs deal with a lot of opportunity sizing, defining the vision of the product, and getting their team aligned on it, staying on top of, uh, marketing, competitive trends, and prioritizing features. You might be saying, "Hey, Kevin, that sounds a lot like what we do as user researchers." And, actually, yes. But the role of PM really differs from small to large companies. At small companies, they might be hands-on, talking to users, prototyping, and doing everything, including what researchers and designers might do. At larger companies, they're going to be more specialized. You know, where design team does design, research team does research, and helps the product or kind of make that vision come true. As UX researchers, we are trained in research. PMs may or may not be. So, it's our job to make sure that the research we all do is valid and reliable. If there's one book that will encompass what it means to think like a product leader with business sense, it is this one: Inspired by Marty Cagan, "How to Create Tech Products Customers Love."

It's about product management at the core. It's written by a product manager for product managers. I can't say that their definition of user research is entirely accurate, but this is a great book if you want to understand what it means to think like a product leader, what you think, how to think like a product manager. So I definitely recommend this one if you want to get into the heads of how product managers think.

But the book, you know, is way more than just product management. Uh, I can't recommend it highly enough. "Not everything that counts can be counted, and not everything that can be counted counts." Now, I think this quote by Albert Einstein really nails the fact that metrics aren't everything. We try to count things when they don't even matter.

Physicists have been trying to find an all-encompassing Grand Unified Theory of Everything that attempts to explain all physical aspects of the universe with one equation. But I think the closest they've gotten so far is Super String Theory or M-Theory, which basically describes everything as made up as vibrating springs, and every vibration corresponds to a particular frequency, which corresponds to a particular particle. Anyway, like theoretical physics, there isn't a grand unified UX metric. But perhaps there is no need for one. People, you and me, are too complex to be boiled down into one metric. Anyway, we change behavior based on social context, our moods constantly shift.

Having said that, we operate in a business, so we'll need measurements to help keep track of our progress. However, as in the title of this lesson, measurements are not always the answer. Measurements are merely inputs and a way to minimize risk. And companies, by and large, are all about minimizing risk. Don't believe me? McKinsey, Harvard Business Review, and others have all done studies on companies and found that companies are definitely more risk-averse, even if they say they like taking risks. And if we only ever focused on minimizing risk, we might miss out on bigger, more innovative opportunities. We end up too focused on testing, getting marginal, incremental gains to make our investors happy, and we forget to focus on the outcome for our users. Being too metrics-focused might actually take us off track. Literally, I'm not even joking. The kinds of conversations I hear at work are, "Getting conversion rate from 1.2% to 1.3% in the next quarter." It bores the hell out of me. You don't have to measure everything. Okay? Focus on the user need, and everything else will fall in line. So remember to take a step back and ask yourself and your team, "Why are we going to measure this? What are we going to do with this information? Measure what counts. Measure what's important to driving strategy." And I'll see you in the next lesson.

The product roadmap maps out how the product vision will be carried out on a high level. In the product roadmap, it should include research activities to get us there. And that's where the research roadmap comes in. It's almost like a roadmap within a roadmap. So in this, I'll talk about what the research roadmap is, what it's not, how it's planned, why have one, and how to create one with an example that we'll do together.

What is it? Well, a research roadmap, it shows what research you need to do and when. We don't go high-level like product roadmaps. So if you ask me honestly, a research roadmap is just another word for research timeline. When research happens, annual roadmap planning happens during pumpkin spice latte season. That's either autumn, and if you're in a Northern Hemisphere, or spring flower blooming season if you're in a Southern Hemisphere. In Noma-land, joke's aside, not really a joke. Roadmap planning usually happens quarterly, or every three months. But it's good practice to anticipate the future and think ahead too, especially if you have foundational research you need to make time for.

However, before we get too deep into the parochial nature of roadmap planning, I want to tell you the expectations versus reality. There's a saying for driving on the racetrack: your eyes should look towards where you want your car to go for the next turn, not right in front of you. This helps orient your car better. But it's also useless to think about the next five turns if you haven't even finished the current one, because how you enter the next turn depends on how well you exit from the previous turn. Got to carry that momentum.

Don't just take it from me, though. I posed this question to the User Research Collective group on Facebook: "Does anyone ever stick to their timelines that they have planned?" Overwhelmingly, from the responses, the answer is no. No matter how far you try to plan out your roadmap, it's an exercise in futility or adaptability, if you're more of an optimist. In other words, there's really low confidence for sticking to the roadmap once you plan out beyond three or four months. Many times, we are asked to plan out two, three, even four quarters into the year, the entire year. Realistically, there's no way. Based on my experiences at many different companies, at different projects over the past decade, there's not been a time where I've been able to stick to those timelines beyond a quarter. There's no way we're going to stick to those timelines. But that's okay. If we only plan for the short term, we'll be working very reactively. It's okay to have a long-term vision, anticipate what needs to be done to future-proof yourselves. Even if you don't stick to the timeline, at least we have a vision to reach for. "If the plan doesn't work, change the plan, not the goal." That's adaptability. Just keep track of the work as you go and be sure to have an open mind for when new insights spark new opportunities.

Here's an example of what a research roadmap or timeline looks like: Month one, project one; month two, etc. You're basically breaking down projects and you're putting them into a chart or Gantt chart by month, and you're going to break the months down by week, and then you're going to go even more granular. We also make or update a research roadmap when new initiatives come about so you can prioritize and reprioritize current objectives to fit those in. So that's what a roadmap looks like. Not rocket science.

What it's not, on the other hand, and that's important to point out. What a research roadmap does not contain: problems we don't include problems that are outside of UX, like marketing, engineering, customer support. Those aren't included because those teams have their own roadmaps. It's strictly user research studies we have to worry about. That's not to say we won't collaborate with other teams, but this helps us with our focus. Of course, it's a discussion to have. If you're going to do research on your target audience, who are they? Market sizing and whatever. A lot of the overlap might happen with your market research team as well. That's a discussion to have. You might have some copy message: "Does this resonate with your target audience?" You know, again, marketing or your UX copywriters. So always good to, uh, discuss up your team and why have a research roadmap.

Why is it important? The benefits of having a research roadmap are that we are creating a shared understanding of the research process itself. We get alignment on priorities, unveil why research is done at certain points. It can itself be an educational tool for your stakeholders, and we can get a bird's-eye view of what kind of research you and your team does most often. We can figure out, "Hey, you know what? We spend a lot of time on fix-it testing. Can how can we get ahead of that type of testing? Make more time, carve more time out for foundational research." And we can use this information for improving our own research practice. The research roadmap isn't just about, "Here's what we need to do, here's what we need to do when." But you can kind of look at it as a, a tracking tool for your research practice. "Oh, it looks like on this roadmap, everything is usability testing. Is that where our efforts are most valuable? Sure, we got to do some usability testing, but what about the big questions? Where do those fit in?" And that can help spark some discussion.

Let's talk about the steps to creating our own research roadmap. We'll try an exercise together. Let's make one together. Let's practice making a research roadmap together. So let's pretend that we're badass UX researchers in our team, and we're doing quarterly planning. Fun. Uh, so we're going to plan the next three months, or about 13-ish weeks. Uh, some months have four weeks, some have five. I don't know what. Let's just plan 12 to 13 weeks. Let's pretend that we've spoken to our stakeholders, learned what projects need to be done and their priorities, and we're prepared to write up our timeline. Okay? So let's say that we have two projects going on for the next three months so far.

Project one, let's say that there is a major priority foundational study. Okay. Uh, we're trying to understand a segment of our customer base better to build something for them, right? Try to understand their unmet needs and build a product for them. Uh, and assume that we haven't built anything yet, but that's the plan.

Project two, let's say the team is currently iterating on an existing feature. Okay? So something that is already live, but wants to test it before the launch. And the launch is going to happen in three months. Okay? So we got a bit of time for that. The team says, and they even said themselves, it's not the largest priority at this moment, but they still need it eventually. Okay? So, uh, in summary, we have one high-priority project and I would say this is a lower-priority project. In real life, you're probably going to have more projects than two projects per quarter. Okay? But I just wanted to keep it simple for the sake of example.

All right, so I suggest planning your roadmap starting with the highest priority projects. So let's start off with project one. And let's, you know, uh, I color-coded it in purple. I like to color-code, but you don't have to. Let's say that we learned about it in the product planning meeting, but now we have to figure out the research details. Okay? We just know that the project is needed, but we haven't talked about the project at all yet.

Over here, in this different sheet, I have the roadmap. Uh, now I have visualized it into month one, month two, and month three over here, and I've broken it down into weeks. I assume each month has four weeks. I know sometimes they have five, but whatever, just keeping it simple here for example. Okay? So let's start with project one. Let's just, the only thing about project one, and by the way, your timeline might vary based on conversations you have with your stakeholders. So it's actually best to plan your roadmap after having a kickoff meeting. Um, but in this case, we're going to plan that into our roadmap too. So let's get that going as soon as possible, preferably week one of month one. So let's say the kickoff for project one kickoff, let's make it purple, happens in week one. What else happens in week one? Maybe some secondary desk research. Uh, maybe we, after the kickoff, we understand who we need to recruit. So we might even start recruiting for the study. Okay? And maybe start writing up the plan, draft out research plan. Okay? All of this, week one. And there's plenty of time to do all this in week one.

Week two, let me, let me do the RP text. Okay, week two. I'll pretend we did enough planning and aligning with our team. So we might want to do a dry run or a pilot study on week two. Pilot study. And maybe, maybe we're able to recruit someone for the study by then, hopefully. And, uh-oh, maybe recruiting takes a bit longer than we anticipate it, which often, often happens in real life. Uh, so we can add some buffer time for it. Okay? So maybe week two is like, recruiting again, right? Recruiting again. Um, so I, by the way, I have laid it out this way. You don't have to do this. In fact, I'm going to scroll down really quick because I've created a different type of visualization. Something that I actually much more, much prefer. It's kind of a calendar view, right? Monday, Tuesday, Wednesday, Thursday, Friday. And let's assume, you know, like it's January 1st on a Sunday, and then second, third, etc., etc. So this is how I like to have it. So if you were to use this visualization, it'd be like this: Project kickoff on Monday. Project kickoff, oh my gosh, can't type right now. Um, Project one kickoff. Let's, you know, make sure to designate that, color code it again. And then we might do some desk research on the same day, cuz you don't, you know, your project kickoff isn't going to take the whole day. All right, kickoff, desk research, maybe start planning, drafting out your plan, and recruit. So yeah, these things can happen first two days. All right, so like I said, this helps me more to plan my own research. Uh, people don't really need to see when you're doing stuff to the T, to like the exact day and stuff. So sometimes you could use kind of a more high-level thing like, "Oh, this week we're going to do this activity generally." Um, but if you want to break it down into days, you could do that too by using this kind of visualization. Okay? Yeah, so that's what I like to do. But let's focus on this one for now. But if you'd like this version, uh, let me know. I actually like this one better. Cool. So where are we? All right, so we're conducting. Let's conduct a study. So, oh yeah, let me back up. Project one, we're trying to build a product. So let's think ahead three months. What do we want at the end of three months? Well, preferably we learned that during the kickoff, but I assume that we're going to start brainstorming, developing an idea, concept testing it, and then iterating on that all within three months. So this may move pretty quickly, as you'll see.

All right, cool. Week two, let's, uh, plan recruit. Um, oh yes, what else? Pilot to study. Let's say that, um, before we pilot this study, we obviously would like to, uh, align. Or this also happens: feedback on plan. Okay? We also want to get the feedback from our stakeholders, make sure they're aligned, finalize the study plan. You know, hey, you know what? I know I'm going back and forth, but this is real-time, baby. This is real-time planning. Okay? Finalize the study plan, pilot the study, recruit, and keep on recruiting. And if we recruit fast enough, uh, I actually prefer that we could maybe do a couple of sessions on week two, if we could do that. Interviews or sessions? Let's not assume they're interviews. Uh, I assume this interview cuz we're understanding customer segments and their unmet needs. So yeah, whatever interview sessions it's going to happen. I know you don't usually choose methods beforehand, but you know, if you're trying to understand customers, you better interview them, or that's definitely a method to use. Cool. Um, all right, conduct it. Conduct it. Maybe we do more sessions in, you know, this week too. More sessions. Maybe we have a debrief meeting with our stakeholders. Okay? This is all project one. Everything in purple is project one. Okay? Uh, I'm just saying sessions might take two or three weeks. I'm not sure, but let's allow time for it. As we're conducting, we want to provide constant debriefs on what we're noticing. After we're done conducting, we'll have data. Okay? So we'll need to spend some time synthesizing and analyzing data. Now, maybe an analysis can happen on week three after you're done for your session. Maybe it bleeds into week four. You know, I like to give myself some time for analysis. Uh, couple days at least. And maybe you even have a co-synthesis session that you plan with your team. What's a co-synthesis session? I'll talk about more of that in a later module, but essentially, it's just doing analysis with your teammates together, like a workshop. And then set a date for the final readout. I think if we had sessions on week two and three, we could do analysis and have a readout by the end of week four. So, you know, readout. And hey, you know what? I'm going pretty aggressive here. All right, some of you are like, "Oh, I'm done faster than that." Whatever. You depends. This is the best-case scenario. This is the best-case scenario. Maybe your readout is on week five. That's okay too. Just be flexible. You, there's no way you can plan three months out in advance without encountering some kind of hiccups. You, it always happens. Your timelines get extended. That's all right. That's all right. It happens. You just have to keep your stakeholders in the loop at all times. Okay?

Now, this is a foundational study. Remember, since we're doing foundational research, I anticipate needing to do some brainstorming sessions, right? Ideation workshops, uh, with the team to take action on those foundational insights, brainstorming solutions that we can concept test later. So let's, let's add that in after the readout. But hey, you've done a hard, you've done hard work for these past four weeks. Let's do the, the workshop here, right? Brainstorming, brainstorming, ideation workshop, week five. Okay? Okay. Wow, look at that. We've just planned our timeline for our foundational study. Cheers. So that puts us here, week five, week six. Um, and you know what? If it goes to week six, I think that's still okay. We're still good on time here. Now, we will do the brainstorming session. We're going to prioritize ideas and then, you know, concept test a few. If we want to validate those findings with a larger scale study, we might plan a survey also, and that could take another few weeks to do, depending on if we have to, if we do the survey ourselves or if we're using a vendor. So at this point, the timeline can get a bit more cloudy. For the sake of this timeline, let's just pretend we're going mostly qualitative research. Okay? But if you wanted to do quantitative research, I personally like to do quantitative research after doing qualitative research because, like, after analysis, uh, if you really want to see to what extent customers out there feel your insights, right? I might do a survey like here, right? Month two. So I could run these in parallel, right? Let's just put this in metallic. Survey. And surveys, they take a long time, and they are tough, right? They're tough. Surveys are the hardest method to get right, easiest to get wrong. But let's again, let's say for the purpose of this study, we're going to stick with qualitative, just to make this easier for planning. Okay? Usually after brainstorming in a sprint, the design team may start to conceptualize and prototype. We're going to need to work closely with our design partners to see how long that they think they could do this and plan accordingly. Okay? So we're going to have to give more buffer time here. Let's do a, like, gray boxes. Um, let's pretend it'll take them about a week to prototype. Um, so, you know, this is not something we're doing. Uh, but designers prototyping, perhaps. Maybe they're fast, but let's get them some time too. Okay? So during week seven, we may do another kickoff, right? Kickoff part two for the same project. This is for project one. Project one kickoff part two. Why do we do this? This because now we're going to do concept testing. Okay? We can do this even if the prototypes aren't ready. Uh, so kickoff again. So it's not kickoff for the project, it's just, hey, here's the next phase of the research we need to do. So we've done the foundational part, now it's conceptualizing, concept test. So let's reign, let's call it realignment, you know, just for our sake. Realignment part two. Usually, we'll know by now what kind of users we need to recruit. So I might put in the recruitment request here again. So recruitment for that for the next study. And we could pilot the following week, right? And you can even be, it could be sooner if you like, but, let's just give ourselves some time cuz we have 12 weeks. And hey, you know what? If we reach week 12 or 13 and we're getting tight, we can shift some of these around. You could do that. This is not finalized. We're just planning, uh, for now. So let's do pilot plus sessions. Oh, no, no, I don't want that. I want this one. Okay? So now we're in week eight. Pilot and conduct more sessions. And, uh, hey, if the design isn't finished by then, we have to be flexible and shift the timeline a bit. We'll cross the bridge when we get to it. We have to be adaptive. And here's, I guess, here's my pro tip for you: whether it's on paper or just mentally preparing for it, always add buffer time whenever you plan projects this far out in advance. Anticipate roadblocks. Okay? For roadmaps, you probably have roadmaps, uh, and roadblocks. So anticipate it so you're not stressed out when priorities change. It's, it's a very down-to-earth mindset. Like Bruce Lee said, "Be like water." I imagine since this might be a concept test, we could probably knock out the sessions within the week and even start analysis the same week. Okay? So maybe we're getting pretty aggressive. Uh, now let's give ourselves some time to do, you know, focus on sessions here. We'll do some debriefs here and there. Uh, and then week nine, let's do analysis. And this, this is a whole week, right? Five days. Uh, I, let's give ourselves a day or two, or even three if you really need it, to do the analysis, which means you can do the readout at the end of the week. Okay? Readout for, you know, the concept test. That's where to. I hope this is becoming a little clear of how we plan this roadmap out.

All right, so given, give, give this design team some time to iterate based on these findings. Okay? So the design team iterating. Okay? Let's say that they're going to be doing this. And we're going to repeat the process for evaluation. Let's say that we find the concepts that work and we scrap the concept ideas that don't work, and we're going to iterate and start building out the ones that do work. So in the last three weeks, let's do that, right? Let's repeat this, uh, uh, which can fill up the remaining three or four weeks here. Okay? So, uh, once we have the concepts in mind, we're going to like kick off our usability, probably. I don't know yet, but I anticipate this is what's going to come up. Kickoff, recruit, you know, conduct sessions, uh, analysis, and then readout. There we go. You know, this is, you know, in one quarter, one quarter, which is three months, we went from not knowing users to getting foundational knowledge, conceptualizing ideas, brainstorming and ideation, I love prioritizing ideas, which I'll teach you all later how to do, to prototyping something, to usability testing it, and then we can launch it, most likely, maybe. Okay? So three months, we planned it out. Congrats. Awesome. And again, you may use this visualization if you prefer. Again, I prefer this, uh, because I like to just see like what days we're doing, you know, sessions, conduct, conduct, like this, and conduct, and then here debriefs, and then here maybe, maybe here we do, uh, analysis. Whoops. Hey, that, I, I might edit this out or not. You're just going to see real-time. And then, you know, let's de, readout here. Yeah, so stuff like this. So like this, I want to tell you something. Remember, we only did this for project one. We still have project two to plan out. Let's plan with project two. Let's not forget about that one. All right, let's show this baby some love. Having just one project for a quarter ain't going to happen. Got to have two. Let me just remove this survey thing here for now. Okay?

So let's see. Remember this project, the second project is evaluating an existing feature, unrelated to project one. Let's assume. And they don't need it immediately, but that doesn't mean they don't not need it immediately. Okay? It just gives us some flexibility for when to do this project number two. We, we definitely don't want to leave it for the last, last, you know, minute. Okay? Because the team is still waiting for insights. And maybe because we might find some insights that affect the timeline overall. Okay? It's more of a nice, it's, it's more of a, uh, it's nice to have data soon, but prioritize your large project first type of situation. So on my roadmap, uh, I'll create, um, I'll just use these bottom rows for second project. I'll take a look at project one, right? Where do you think we're the busiest? Well, I definitely don't want to conduct two different studies during the same week. Okay? So when we're conducting studies for project one, I don't want to be conducting studies for project two. Let's stagger that timeline. Okay? The kickoff for project number one is on week one. Kickoff for project two could probably happen week three or four, depending on how much work you think you'll have then. Boom. Add in recruitment, conducting, analysis, and readout components. All right, so let's, let's see. This is what it might look like. What color was Project 2 again? Was blue. Okay? So this color. Kickoff for project two. All right, maybe I, instead of the, oh, instead of I could just make all these cells blue text color. Okay? So kickoff, recruitment for project two, maybe pile. Oh, oh yeah, draft plan. You see all these components are the same for any kind of research study. Draft out research plan. Let's wrap text. Sorry, you're just going to see this live, raw, unedited. Draft. Research plan. Get feedback on research plan. Finalize. Oh my goodness. Okay, here we go. Finalize research. Then thank you for your patience as you watch me fumble, fumble around this, uh, Google sheet. Yeah, we're piloting a study with. Oh, this goes first, right? Finalize research plan. Pilot the study. And what else would we have? You know, maybe more recruiting. Who knows? And since it's week two, we could probably do more sessions. Yeah. And, and since we have a bit of time to play with, maybe we don't have to do all our sessions on the one week. If it's an iteration project, I assume it's usability. It's some kind of evaluative testing. Evaluative testing, right? So it's not going to be as long. It depends, right? Everything depends. You could be recruiting very niche participants. So if you need it, you have more sessions on week five. Here. Here you can also do analysis and then maybe the readout here for the project two out for project two. Okay? Boom. Done. After you finish this study, keep track of the findings and recommendations and make sure to inspire the team to keep on iterating. They might need you for a second round, so leave your timeline open for that possibility, right? So now we have a lot of buffer time. Maybe design team for project two is iterating here, and maybe we can use all this time here, uh, to do an additional round if they require it. Okay? So, phew, that's a lot. Thank you for sitting through this one. Uh, hopefully that gives you an idea of how one might plan out a research roadmap for a quarter. Again, this was purely hypothetical with very little information about the projects themselves. In reality, you would have more information that could better help you plan out the timelines. Okay? Just remember, the timeline serves as a guidepost to get certain milestones done, but they aren't set in stone. Roadmaps are not set in stone. Remember Bruce Lee's quote and be prepared to adapt. Uh, because in my experiences, and this is a true story, I have planned entire studies out and in the middle of conducting a project could completely get pulled for whatever reason, and you'll have to overhaul your timeline on the spot. Okay? I've had projects scrapped out of thin air. Okay? So remember, plan it out, keep it open enough for any changes, and remember to add some buffer time, either literally on paper, not literally on paper, with the spreadsheet, or just mentally in your head. Like, just mentally prepare some buffer time. For example, for this video, for this lesson, I was going to write out this timeline exercise on a whiteboard. Look at this. I have everything here. And I thought, okay, this is kind of a pain. So I decided to adapt and do this on a Google spreadsheet, which is probably what you're going to be using. Like Google spreadsheet, you might use Coda, other tools. I won't talk about the tools you use for roadmaps. You could use anything, honestly. Just whatever your team uses. Okay? So thank you, and I'll see you in the next lesson.

The UX research process. Okay, this is the one. This is the ultimate mega module. We're going to go far beyond the basics of just writing a discussion guide or study plan because now we have a strong foundation in UX psychology and product strategy. In this module, we're going to get into the process in depth and learn by doing. In fact, I'm going to give you a real-life job interview exercise as our starting point so we can actually think about how to plan for research as we learn the process. Learn by example, right? That's the best way. We'll cover how to dissect the problems, define research objectives, and use those puzzle pieces to formulate our questions. When we've defined some questions, we're going to talk about how and where to recruit participants for your research with some actual scripts that I'm going to give to you that you can actually use today to email participants. So print those out and frame it on the wall. And whenever we talk about participants, we have to talk about sample size, right? I know everybody's favorite topic. Uh, I'll show some unconventional wisdom in terms of how to ensure your sample size works. And of course, once we get those pieces down, we got to talk about methods, or rather, choosing methods. I'm willing to bet at one point, you've probably Googled this exact term before: "When to use which UX research methods?" Am I right? Mhm. You probably saw that chart from the Norman Group. It's a great start, but I'll break it down even further for you. In this module, we won't talk about methods just yet. That's coming up later. But knowing what problems you need to solve first is going to be way more important than choosing what method you need to use. That comes later. In this module, we're all going to learn what happens after you choose the method. Well, you got to conduct a study. Sounds simple, right? But there's so many things we got to prepare for to be aware of. I'll show you the art of moderation and listening. I even will show you how to take notes with my notes template so your team can be involved. Once you've collected the data, we have analysis. Ah, of course. Extracting insights to make winning strategic and tactical recommendations. Yeah, we'll go over ways to analyze qualitative data. Then, for any researcher or storyteller, for that matter, if your recommendations go nowhere, then does your research even matter? Does it even impact? No. So I'm going to teach you how to tell the story of our users the right way for the right audience. And all the while, throughout this whole process, I'm going to show you how the process itself is inherently a way for you to amplify yourself as a badass UX leader. Seriously, it's a team effort, and you're involving your stakeholders is paramount. And by the end of this module, you'll hopefully gain the confidence to conduct rigorous UX research and maybe even a study plan you can refer to whenever you have a hypothetical take-home exercise for a job interview. You know, just a nice little bonus. Are you ready? 'Cause I'm ready. Let's get it. Do it.

How does NASA organize a party? They plan it. Hey, welcome to the first lesson in module four. Let's learn to process by doing. I think this is the best way to understand how the step-by-step process works when planning rigorous user research. So to ground ourselves, a lot of the work will revolve around something called the study plan. In UX research, a study plan is a document that summarizes all the research details. However, it's much more than just a document with your research methodologies and procedures. It also serves as a central place to align with stakeholders on the key objectives, to align our research questions, and as a record of how the research was conducted for future reference, in case anybody else wanted to know how the research was done. This also serves as a feedback mechanism because you will not just have one version of the draft, you're going to have multiple iterations of the draft. Change it over time, you're going to ask feedback and update.

But going back to the study plan, I'll give you a quick breakdown of the different sections of the study plan. We have the background, the problem or opportunity space, objectives, your research questions, your timeline for the project, the methodology section, which should contain your methodologies, participants, sample size, and links to the materials you're using. Include the DAISY framework in there. And finally, last but not least, your study plan should also include the procedure, AKA discussion guide, AKA script, AKA protocol. Okay, they're all the same: procedure, discussion guide, script, protocol. You've heard different terms, they're all the same. Don't get confused. I have one in the course materials, so feel free to download the study plan template from that section if you want to follow along. So I basically included all the information on the template itself. If you ever want to use it, just delete the information that's there.

Let's walk through each section really briefly. The background or the problem: this is the section we want to talk about where did this initiative come from? Why is it happening now? Here you might want to link to relevant background documents like the PRD or maybe the PRF and Q or product roadmap vision documents with, you know, stuff like that. The next section, objectives: what do we want to learn? What are we going to do with the results? And I also like to have a subsection called non-objectives. Sometimes you might have huge objectives over very broad ones, and you want to scope it down. You want to make sure you align with your team on this first to know, hey, what are we studying and what aren't we going to study, right? 'Cause a lot of times people are going to have so many questions, they're going to want to like, bam, ask this, ask this, ask this. I understand nothing. But you, you want to scope it enough so you can have a focus. So you might have a section called non-objectives and say, hey, maybe this could be a different study for later. That's all right. Totally okay.

Next, we have research questions. So these are the specific questions pertaining to the product or service. I also have success metrics. There will be metrics you'll be measuring to know whether or not you're on the right track. So typically, if you're running foundational research, there's no real like metrics. Okay? There's no like, "Oh, we're going to increase revenue, blah, blah, blah, blah," 'cause you're just building an understanding at that point. So for foundational research, there's, you know, not metrics as numbers. Just keep that in mind.

Timeline: uh, timeline is, is, you know, well, it exactly what it sounds like. An overview of how long the research activities will take and the deadline for each activity. Usually, I keep the DAISY framework along with the timeline. So DAISY is just who's working on what and when. It's really important to conclude because I've made the mistake once of not actually confirming who's doing what and when, and that actually resulted in the team pushing back on each other and then blaming one another. So yeah, it sounds stupid and trivial. Oh, process. So much process. I know process isn't the sexiest thing to talk about. Systems work. And trust me, there's a reason I included it in the study plan template. It's not for you. It's not for us. It's for your team. Something to keep your team up to date on what's going on.

Then we also have methods. The methodology section, right? What methods are you using? You want to describe them here. In this section, you also want to include your participants, your criteria, who you're talking to, materials. You know, typically I link to any assets or prototypes that is going to be used, and I'll tag my designer, whoever's in charge of those designs. And finally, we're going to have the procedures, discussion guide, script, AKA protocol, whatever. All that stuff. This is a formal or informal script containing the questions and the exact wording that you're going to use during a study. The procedure is like a cooking recipe. Anyone reading it should know exactly how your study is done, and they're able to replicate it if they wanted to. So that's the study plan for you. Keep in mind, this is a living, breathing document where you and your team should be centralizing your research activities upon. Now, like I said already, this document is more than just a recipe for research, but it's like, it's like a home base. Everyone can feel warm and fuzzy because you're all aligned on the same problems.

So now that we know the structure of the study plan, let's put this baby to practice. You don't want to miss the next one because we're going to do a real-life take-home exercise that I've gotten from a tech company before, a real-life interview take-home, and we're going to use our study plan to do it, showing you what goes into a study plan. But how long should it take to create one? As a newbie, it's very easy to want to make the perfect study plan and take hours and hours to do it. But in reality, I never spend more than two hours creating one. Of course, when I was newer to the field, it took me a long time. As I got better, as I got more experience, I realized you don't need to spend that much time on it. Not because I'm super smart or anything, it's quite the opposite, but because you don't need the perfect plan to show your stakeholders. After the badass kickoff meeting we'll have, I have a pretty good idea of the background, the objectives, the non-objectives, the research questions, success metrics, timeline, and probably whom participants we need to speak with. That's plug and play. All you need to do is just take all your notes from your meeting and just plug into the study plan. That shouldn't take you more than like, I don't know, a few minutes, 10 minutes, half an hour, probably go in and out, 20 minutes. Adventure. The sections that take the most time are the methods and the discussion guide, AKA script, AKA protocol, AKA procedure. We got to think about which methods make sense, who we're going to speak with, how many segments of people do we have in a sample size for each segment. Then we have to actually write out a discussion guide to help guide our conversations. This goes for qualitative research. If you're doing a survey, you might not be writing a discussion guide per se, you'll be writing your survey questions. The reason I don't take a lot of time on a study plan initially is because I always treat my first draft as a first draft. Yeah, yeah. I treat my first version as a draft, and I'll share that with my team to get feedback and thoughts. And I will make sure when, and I do share with them, that I emphasize and I stress that, hey, it's a draft, it's not finished yet. I want to get your initial look at it to make sure I'm not missing anything or that we're getting second pair of eyes on it, just to see, hey, now it's on paper, do we need to change anything? And that's how you can avoid stressing out about having a perfect study plan. Really, honestly, no lie. I had major, major imposter syndrome like writing out study plans because I wanted it to be perfect. But as soon as you think about, hey, this is a draft, it will be better. I'm going to involve my stakeholders throughout this process. So it's almost as if we're writing it together. In fact, I usually write my discussion guide after I get some feedback from my team when I send the first draft. Sometimes I don't even have the discussion guide ready. That way I don't use up all my energy writing something if it's not even aligned the right way. Also, pro tip for getting feedback: don't just email it to people and say, "Hey, here's the study plan, leave feedback." People don't usually just open and look through the whole thing unless they're like, really, really cool, badass partners. What I like to do is send the email, but tag people in specific parts of the plan to review. Therefore, it gives the illusion that you're only asking for something small. When people open it and review the section, they're naturally going to look over the rest of it, right? Yeah, yeah. The more you know.

But let's say for some reason nobody gives you feedback. Sad. Then just do your best to get a version ready and then pilot your study. I would do it at all costs. I will get feedback from the stakeholders before I even run the study. But if for whatever reason, maybe you're working in a very immature UX practice, people don't have time, they don't really want to. That's a different problem. I would still try to in front of them and say, "Hey, maybe you set up another meeting where you can all synchronously at the same time look at it together." Just do your best to get their feedback before you actually run the study. But I'm just saying, last resort, if you cannot, like, let's say everybody on your team dies or something. Knock on wood, right? No one dying. Let's say everybody is not able to give you feedback. Pilot your study. In other words, do a dry run. This is the last resort. You're going to do this anyway, regardless of feedback. You want to do a pilot study. We'll talk about that more. But essentially, it's another way to get feedback on whether or not you're getting the right data you expect. So there you have it. A couple of badass UX tips to help you not stress out by your study plan.

There's one tip I want to give to you that can help you organize your thoughts in your study plan. This applies to your methodology section, your sampling, and your screeners. So here's what I mean. Let's say you're talking about your methods that you want in your study plan. I want you to create a table in your study plan that says method. And let's say you're doing, you know, interviews, IDIs as one method, and usability as another. In your table, put these methods here. In the next column, I want you to put what you're trying to learn from this method, questions you want answered. Okay? So for IDIs, it may be understanding backgrounds or attitudes, etc., etc. For usability, you're trying to, you know, identify pain points, whether they can complete tasks, etc., etc. And here's another tip: you can also add another column for participants, right? Let's add it to the left. I don't know if you see this, but add it to the left, uh, sample. Okay? 'Cause maybe you want to do IDIs for only certain participants, and for usability, you want to do for others. Let's say, for example, usability is for non-users. Non-users. You also want to do IDIs for current users. You might want to run them for both. So you might even, you know, add an extra column say for current users and non-users, and you want to specify. I'm sorry for the chicken scratch, by the way, but specify what users are doing what method and what you want to learn from them. And you can also, you know, create a sub-row for this. It's like IDIs also, uh, for, you know, current users, you want specific questions. For non-users, you want specific questions. You see what I mean? Okay? So that's one way to organize your methods and your sample.

Now, on the other hand, I want to share with you how to organize your screener questions. We're also going to go with a table, uh, kind of format. So on the left column, I'm going to have track. I'll, I'll tell you what this means later. Your screener question goes here, and then the last column is why. So track means, let's say I want, uh, everyone to answer a question about their age. Not typically a screener question, that's a demographic profiling question, but let's just pretend for ease. Age. Track. I want everyone to answer this. All. Okay. And here, why is the purpose? Why are we asking this screener question? And I might call it a screener question, but maybe, uh, I'm profiling people. Yeah, I know profiles have a negative connotation, but in a way, sometimes we don't want to screen people out for certain questions.

We just want a quot of different people. For example, let's say the options for age is 18 to 24, 25 to 35, 36 to 45, 46 to 55, and 56 plus, whatever, right? Everyone answers this question.

The purpose is to, um, let's just say, uh, screen out people under 18. I want to screen people out under 18. This is the only purpose of this question. So anyone who says under 18, they are out. So if they're not 18, they're older than 18, they can move on to question two.

So, question two, let's say it's an attitudinal. Um, how willing are you to try fried chicken and waffles? Chicken and waffles for me, that's an extremely likely. So there's a 1, 2, 3, 4, 5, okay? So not at all likely and extremely likely. I hope you can see this or at least get the idea of what I'm trying to write, 'cause my chicken scratch fried chicken. Um, anyway, yeah. So track anyone who says over 18, 18 plus. So anyone who's 18 plus can see this next question.

And now, let's say I'm interested in talking to people for my study. I'm interested in talking to people who are, uh, at least somewhat open to trying chicken and waffles. Okay? I don't want the people who are not interested at all. So 3, 4, 5, I will accept them. So what I'm trying to do here is screen, screen out people who answer one or two. One and two, okay? So I'm going to keep 3, 4, 5. People who, who answer 3, 4, 5, I'm going to keep, and they get to see the next question.

So let's call this question, uh, one. This is question two. Now, this is question three, my third screener question. The track is if they answer 2C, 2D, and 2E. A, B, C, D, yeah. Uh, this is messy, but hopefully you're getting my drift here. Uh, if they answer 2C, 2D, 2E, they can proceed to the next question, whatever it is. Okay? I, we, but hopefully this makes sense.

So to organize your thoughts in a screener, you could do a similar thing as you would with methods and sampling. So so far, I've, I'm using questions to screen people out. But I have many times where I'm not trying to screen people, I'm trying to get a quota. So what I'm doing instead is profiling, profiling or quota. What that means is I'm interested in getting a mix of people who answer different, different ways in the next question. So what's the next question? Um, I don't know. How likely are you? You know, what are your thoughts about hot sauce? Okay, maybe my study is a lot all about hot sauce and fried chicken and waffles. So if this question could be about hot sauce, do you, you know, do you like hot sauce or not, whatever? And I'm interested to talk to people who aren't into hot sauce and also people who are into hot sauce, everything in between. So really, the next question is just profiling. I want a mix of people who like hot sauce or don't like hot sauce, and I want to talk to them. So that's the purpose of, of profiling a quota, a question. Uh, and hopefully, yeah, this format helps you organize your thoughts a little bit better. You can add it to your actual study plan. Your audience would actually appreciate that too. It's an easier way to find out why you're asking certain questions. So hopefully that helps to illustrate just how important a study plan is.

Let me tell you a story about something that happened to me at work. So one time, my team and I were planning research to study user onboarding to our platform. We knew that onboarding was, uh, not great. Let's just say it's a, a very small percent of people make it through the initial process. And onboarding is one of those things you have to get right because it sets the tone for the rest of the experience for users. So naturally, I wanted to know how the product prepares or doesn't prepare new users to the platform and why people drop off before trying out the platform. However, as I was planning to research, the design lead, who's been at the company for about seven years, kept commenting on our research plan. How is it different from previous XYZ research? We have data on blah, we have data on this, etc., etc. I've gone through the previous research, right? And there had to be a reason why we're doing this new research. But I started second-guessing whether or not we needed this research because the design lead kept asking these questions. And then my PM also started second-guessing the research. Do we even need this now? So even though I had already planned the research, this doubt was killing me. So the PM, designer, and I, we sat together again. We're like, "Hey, let's talk about it. Let's talk it out. Let's play Devil's Advocate. Let's pick this research apart. Do we really need it?" And I really had to think, "What is it that we don't know about our users yet? What were the limitations of the previous research? What gaps do we need to fill in here? What does the designer need to know in order to start designing a better experience?" Eventually, we did the research by looking at non-users and their expectations around the platform. And it turned out to be useful because the previous research all have been done within the platform with our current user and stuff like that.

The point of the story is not the research study that I did, right? That's not the point of the story. Is this: If you can't convince yourself, don't expect to be convincing anybody else. And if you are sure about your research questions and why you're doing it, then be sure. Get stakeholders in a room, just jam on it. It's better to not know before the research than to not know while you're conducting the research. And the other thing I want to point out, if you work at a place where people have been on the team for a long time, they're going to reference previous research. They have a lot of institutional knowledge. They're going to be like, "Oh, we already have this. We know this." And you need to start considering how the previous research was done. You need to look at how the methodology is run, when it was done, because the product may have changed a lot. What questions were they asking? Were they asking the same questions? Maybe some of the findings are applicable and great, great if they are, you save some time, that's perfect. I would rather have that. But chances are, your product has changed over time. And you know what? Yeah, we have researched in the past, but we're still seeing the same problem. It still hasn't been fixed. So anyway, that was a quick little detour to show you the realities of study planning. Not everything's going to be smooth like butter. So if you're not sure, it's okay. Take the extra meeting. Don't just jump into the research if you're not sure. If you're not sure as the researcher, that's not good. So be sure. Ask your stakeholders. Ask them again. And just go in at, like, "Hey, you know what? I've been thinking. I've been thinking. Uh, yeah, yeah, yeah. I can align with y'all. I want to clarify, is this what we're studying? We have this data here. What do we think about that, right? What do we think about this previous research? Why can't we use this research right now?" And go in it with an open mind. That's how you can do research with intention.

In this lesson, I want to give you a very simple framework in which to think more purposefully about your research questions. The reason being, we don't want to just ask questions for the hell of it without thinking why we're actually asking these questions. Otherwise, our research will be all over the place. Again, harking back to the importance of having clear objectives, which can be attained by aligning with your stakeholder team. Anyway, here it is. I don't have a sexy name for it or anything. We can start with building out a table with three columns. First column: Who are we learning from? The second column is just your research question. And then the third column, probably the most important one, is why you're asking this question. What do you expect to do with this information? I found this table helps structure our questions a bit better.

The "who" column is easy, it's your target audience. But you might have multiple target criteria, like current users or ex-users or users who dropped off in the middle of your process. And you wouldn't ask the same questions to everybody. Next column, two, which is questions. This is pretty obvious, just put your research questions in here. And finally, the last column is why we're asking and what we expect. And this helps us stay focused. A lot of my students ask a question like, "How many people click X or click Y or how often do users check the app, blah blah blah?" And I remind them, qualitative research won't tell you this information. Think about what kind of questions you're asking and whether or not you need qualitative research to get it, or can you refer to analytics to get those answers? So, "How many people click on blah?" If you're recruiting five, six, seven, or eight participants, that's not going to give you the information you need. That's not what qualitative research is for. "How many people click on blah?" That's analytics. Now, if you have a prototype and it's not live, "How many people click on blah?" is not a valid question because you don't have the data. Does that make sense? We can ask it to our participants, but what they answer isn't important in qualitative research. It's *why* that's more important for us to understand. And to take it a step further, we then anticipate what information we expect to get. Think to the future, what possible answers could you get? And think about how you're going to use that information, kind of like the premortem I talked about in the previous lesson. This framework, then, is a great way to organize and structure our study plan. So start including this in your study plans.

One very interesting question I got from a student of mine was, "How much domain knowledge do I need to approach a research session?" This is a really good question because some people say you need to be an expert. Uh, and in fact, one time I interviewed for a TV entertainment company, and they said, "We're looking for someone with TV experience." And I said, "What? I've been watching TV my whole life." Uh, so anyway, they didn't take me. They said they wanted someone who knew the industry. Fair enough. But catch-22, am I right? How are you supposed to get experience without the experience? And I think watching TV as a consumer is experience. But anyway, besides myself, I don't think you need to be an expert in a certain domain, but I think you should have enough domain knowledge. So the industry jargon, for example, you got to know what previous research exists about it and the processes. Why? Because we don't want to waste time asking the obvious questions, right? We want to get to the main objectives. We don't want to spend so much time doing the buildup and building up our own knowledge so that we can converse on the same level as our participants. We don't need to be an expert, but we have to know the basics, the baseline, so that we don't duplicate research, so that we know who our competitors are and how we stack up, so that we can understand what the participant is actually talking about, so that we can actually probe and know what to probe on and what they're talking about, and also can be taken seriously.

Here's an example of one industry that I worked in where we were required to understand the jargon. And this was when I was working at Uber Freight. So at the, as a lead researcher there, I needed to know what terms were used in the freight industry, what they mean, and how it worked. If I didn't, there's no way I could actually research participants because I would ask, "Oh, what do you mean by LTL? What do you mean by BL? POD?" I'm like, "You should know this, right?" And that's not the research that we're interested in. We don't care what BL means. So I don't want to look stupid. I don't want to waste time asking the stupid questions. If I didn't understand the jargon, I wouldn't know what's important or not. I wouldn't know what to ask or probe on, and I wouldn't be taken seriously. So in certain industries where you're new to it, learn the jargon. So what happened that when I was at Uber Freight, we had to learn all these terms. And we had an orientation, we had a training meeting where we were tested on all these terms. It's really cool. I learned a lot about the freight industry. And one of the reasons why we were trained on these terminologies was because a lot of us were not working in the freight industry, and we're talking to people, truck drivers, you know, dispatchers, fleet managers. And they knew the freight industry very well, you know? So when they're talking to someone from a tech company, they're not going to take you seriously unless you understood the language.

So I would argue that not having extensive knowledge, but at least a baseline knowledge or language, right? You got to share the same language as a baseline, but you don't have to have the expertise in the domain. This gives you an advantage because this gives you a fresh outsider's perspective on approaching a problem. While others may already have preconceived notions, right? When I was working in the freight industry, everyone in antiquated methods, right? Paper, they still use a lot of paper tracking. The sentiment is, "This is how it's always been. It's hard to change." If you don't speak the same language, so if you don't know the domain, that's okay. If you're interested, do the research. Yeah, learn about it. It's very interesting to learn more about anything in general, right? That's the advantage of not knowing. Approach everything with curiosity. And that's my two cents.

Believe it or not, a form of being a badass UX researcher is including your stakeholders during your research mindset. Beyond that, including them throughout your research. It's the team's research, not just yours. My philosophy is, by the time I come out with the research, nothing should be a surprise. But the point is, I want you to put this philosophy into practice. We spoke about what to do before your sessions. Now let's talk about how to involve stakeholders during your session. The best practice is to set expectations and ground rules before the call and get a designated person to field the questions for you. But things happen, such as stakeholders didn't read your discussion guide, stakeholders invited others to join, which is great, but they might not know the ground rules. If that happens, it may manifest itself into a not-so-smooth session. Okay, let's establish some ground rules, pretty boy.

Here are three real-life situations I come across a lot, and I want you to think about how you might approach them. And after I say them, pause the video and think about what you might do. Brainstorm with a buddy if you want to. And in the next lesson, we'll go over some possible answers. Here's the first scenario: Imagine you're conducting a remote interview over Zoom with a participant. You have five other teammates join remotely as well. One stakeholder forgets to mute their microphone, and another forgot to turn the camera off. What do you do?

Second scenario: Imagine you're conducting a remote interview with a participant. A stakeholder is watching and then pings and messages multiple questions to ask the participant, some of which you're going to get to eventually. Okay, so they're pinging you a lot. "Ask this, ask that." What do you do?

Third scenario: Imagine you're conducting a remote interview over Zoom with a participant. Again, remote interview. The stakeholder has a question and suddenly unmutes themselves and asks the participant directly. "M, did you ever to go to school and have those thermos? You know what I'm talking about? Thermos? Thermos?" "No, a thermos." "Thermos?" "No." "What do you do?"

All right, and to think, these are just a few examples. In fact, if you're lucky, you might even encounter all three of them in the same session. No, God, please, no. No, no, no. I'm just kidding. That's not lucky. So think about how you might approach them, then we'll chat about it in the next lesson.

I posed three real-life situations that come up during a call where it will require us as moderators to juggle many things at once. For those of you badass leaders who took the time to think about it, good job. Now let's go over scenario one, where teammates join a call with their microphones and cameras on. Don't call anyone out over the video call. "You're fired." Just politely remind people to turn their microphones and cameras off. Oh, right, you're fired. I need you to come here. And use this opportunity to introduce the participant to your observers, so it's not awkward. If you didn't do that already, you know, say something like, "So we have you, we have Jen, May, Alex, Encore, Peter on the line, and let them say hi. The part of the team, and we'll be just taking notes. No right or wrong answers. It's the prototype, not you." Blah blah blah. And I might relay some of their questions to you, so I'll have them chat me in.

In the second scenario, we have a very eager and curious stakeholder with lots of questions. "Are we there yet?" "No." "Are we there yet?" "No." "Are we there yet?" "No." So I've asked my master class students before, and some of them gave me really good answers. But some people said we should ignore the stakeholder and focus on the participant. Don't ignore the stakeholder. Message them back, acknowledging their question, and you'll get to it when, when appropriate. Otherwise, they might get frustrated and think you're ignoring them. "Louis, Louis, Louis, Mom, Mom." In some cases, stakeholders might have joined the call late and missed earlier questions, okay? And we don't want them to think you're ignoring them. They might even unmute themselves and ask the participant directly. So here's a few phrases you can type back to them: "Got it. Sounds good. Going to ask soon." "Sounds good to me. I'll cover it later." "Already covered. We'll debrief with you later." Or, "Thanks, I'll circle back later." Or simply send them a thumbs up emoji. Basically, some indicator to let them know that you see them. And make sure to reward their questions in a non-leading way.

Finally, in the case of stakeholders gone wild, scenario three, we have someone who just unmutes and asks the participant something directly. But the incidence was dramatically in India. "2 + 2 = 5?" "There, right?" Well, at this point, what are you going to do? Cut them off? Just let them finish and see what the participant says. "Excuse me, Miss, uh, do you have cancer?" If it was a leading question, maybe reward and try to rephrase the question again. I'll admit, even if it's a bad question, sometimes participants may respond in a different angle where we didn't expect. So here's where we can take advantage of that and probe more deeply. So sometimes you might have a leading question, they still answer. Maybe it's a good jumping-off point. Now, to prevent the stakeholder from doing it again, ping them privately and politely ask them to relate the question to you first. "Hey stakeholder, thanks for the question. I'm happy to relay your questions to the participant. That way, we can keep the session running smoothly. At the end of the call, I can open up the room for you if you like." So be prepared for such situations and handle it diplomatically like a badass UX leader. Remember, you are the research expert. You are in control.

As UX researchers, we don't only deliver findings. We must provide actionable insights and sound recommendations. And findings are the "what." Insights are the "so what?" And recommendations are the "now what?" But we should be careful how we word our recommendations, else they might end up sounding bad or even ugly. So what's the difference between a good recommendation and a bad one? Well, a recommendation should be based on data and gives direction on what changes need to be made. But it shouldn't give the actual change to be made unless it's super obvious.

So let's look at what factors make a good recommendation. Well, it has to be data-driven or backed by evidence. The recommendation ties directly back to the user needs and the pain points. It also describes opportunity. And sometimes you might have opportunities to address multiple unmet needs and look at the bigger picture. Sure, it's great, you know, and it's very normal to just recommend one thing for one problem. But if you can come up with a bigger picture and say, "Hey, here's my recommendation, and it can," sorry, "birds," but it can kill multiple birds with one stone. It also has to be feasible. You know, if you recommend creating a spaceship to outer space with VR tech to address, you know, a cooking process, that's not very feasible. Just making something up. But it has to be realistic to technical constraints. And I'm not saying don't make kind of blue-sky recommendations. And by blue-sky, I mean things to think outside the box. A recommendation has to be doable within reason. We have to also think about the next factor, which is priorities. Yes, you can have blue-sky recommendations, and your team might be all aligned with it, but it might take months, even years, to get to that recommendation. That's a different priority compared to, "Hey, we are losing users left and right every single day. We got to do something now." That's a different kind of priority. So prioritize based on the impact to your users, to your business, your severity, prevalence, frequency. You can't just give a list of recommendations to your stakeholders because they're going to ask you, "What should we focus on first?" From your UX research badass UX leader expertise, where should we focus our efforts first? And sometimes that might have to come from a discussion with your team. What you can do, what we can do, is give a recommended priority based on what we've learned from user research. Here's what we recommend. You, you know, marketing team has some other data, analytics team might have other data that say otherwise. And then we'll work together as a product team, as a unit, to determine what the priorities should be. We'll talk about prioritization, uh, in the later module where we have some frameworks to internally and also externally prioritize stuff.

All right, next, a recommendation doesn't blame the user. Don't blame me, blame your mother, okay? So don't blame anybody. It can't be too specific nor too broad. Okay, I know that one doesn't help a lot. But for obvious problems, you can be specific. It's like, "All right, this isn't, this is a clear issue. This is an easy fix. Don't freak out. Don't freak out. Easy fix." Puzzle code in my head for everything else, we might want to provide direction, but without giving too much specificity to it. So we'll practice with some practice problems soon to give you an idea of what that means. Uh, but it, it varies case by case. That's why a factor that it can't be too specific and it can't be too broad for certain contexts. And last but not least, a recommendation has to be concise. If you're recommending something, just give the recommendation. When you add too much context, that should belong in the context, right? When you present your research findings, you present all your findings, and then you have a dedicated section, and usually I color in a different font, I even put a box around it. Here's my recommendation. It lives alone, but it ties back to the insights, ties back to the needs, and why I'm recommending this recommendation. So don't add too much context. Don't add the explanation to your recommendation. That should just be separate, like one sentence for your recommendation.

All right, on the other hand, what makes a poor recommendation? Obviously, kind of the opposite of everything I mentioned just now. But here's some factors that make it a poor recommendation. It's too solution or feature-focused. Some findings can have multiple solutions, but if you give only one possible solution, you're limiting your team. You're kind of pigeonholing them to only think about that, and that makes a bad recommendation. Instead, what we want to do is brainstorm together. Instead of just saying one thing, we'll go through some examples of that. A poor recommendation is also not data-driven. It's based on assumptions. It's based on only attitudinal data and not any observed behavior. Sometimes your participants might give you great ideas. It's like, "Oh, that's actually a great recommendation. I think that would work." And your stakeholders, you know, they were listening in on the call, and they thought, "Wow, that participant's suggestion is really great. So let's just do that. Yeah, let's do that." That's not a bad recommendation. But what you want to say instead of, "Hey, we recommend what the participant told us," instead, what you want to say is, "A participant suggested doing this. You know, let's open up this discussion for that. How might we, you know, bring that to life? How might we actually dig into the root problem of why this was asked?" Another poor recommendation, this might be a very common one in academia, which is "more research is needed." This is great for exploring new topics. If you identified some new topics, some new themes that your team doesn't really understand yet, great, let's explore more. But if you set out to answer a question and you couldn't really answer it with your research, and your recommendation is, "Oh, we need to do more research," then we might have to go back and look at our scope. Our project scope might have been too big, um, or maybe we didn't probe enough. Maybe we didn't understand the deeper insights too well. So it's useful as a means to explore new identified themes, but if you're going after a specific research question or objective, we better have some good actionable recommendations for our team. Otherwise, it's going to end up being another kind of, "Hey, UX research wasn't very effective" conversation. And last but not least, and finally, last but not least, a poor recommendation blames a user.

You finish your presentation. Yay! Throw those high fives, round of applause, and get celebrating. Unless you were super nervous and fainted during a presentation two minutes in and bombed it, in which case, uh, let's go hide in the corner together. Psychic. We're badass UX leaders. We ain't doing that. No judgment if you do. Anyway, after the presentation, we're close to wrapping up our study, but we're not done yet. So close.

In this lesson, you'll learn a few quick steps to take that'll ensure your insights get actioned on. First things first, you're going to write up a summary of your discussion points and action items. And this is discussion points from your actual meeting, from your presentation with everybody who aligned on it. This is because not everybody is probably going to be able to attend your meeting. So the point is to summarize the main takeaways and next steps so your team is in the loop. And if your team ended up with more questions than answers, that's good. Remind your team that that's a good thing, and we can start planning the next steps for digging deeper. And just because we finished our presentation doesn't mean we're done with our research communication. The best UX leaders maintain proactive relationships with stakeholders and track impact. Say you identified usability problems with your product. Don't just read your report and expect your stakeholders to take action. If they do, great, okay, great, keep them for a long time. But if they don't, which has happened to me, do it for them. Go on Jira, Trello, AirTable, Confluence, wherever your project management tool is that your team uses, or, uh, any kind of simple spreadsheet. Doesn't have to be a fancy tool. Like Google Sheets works well. And I talked about this in a previous video before. Create those tickets, create those lines of items to track them, and assign stakeholders to their tickets. If one way doesn't work, try presenting your recommendations in a different format, like in a very findings-oriented spreadsheet. Remember the previous lesson, I refer back to the earlier lesson on when to use which report format, and the time I had to switch report formats. When you're done with that, keep up those one-on-one meetings with your partners to see how the project is progressing, how the action items are being played out. This is key to amplifying yourself as a proactive badass UX lead. Later, in module 10, I think module 10, I'll talk about more things we can track, how to track them, and how to achieve high impact in our work.

Miro has a set of AI tools, including generating mind maps, user stories, images, and other things. And I'll post a link to the help center page in the video description. The AI tool is available to everyone, which is great, even a free plan, but you have to be signed in, so you have to have an account. In this lesson, I'm going to share my experiences with the AI tool for analysis. Then we're going to practice using the AI for analysis with the same kitchen renovation dataset as a previous lesson. The reason I'm showing Miro AI is because, well, Miro is currently one of the most common collaborative tools user researchers use for analysis, besides maybe Dovetail. But who knows, in a few years, another tool will pop up integrating AI in it. AI evolves so quickly that the tips I show you today might be different tomorrow.

So I've set up this Miro board with our three participants here. P1, I've color-coded as green. P2 is yellow, and P3 is blue. Set up an identical one using our same fake dataset. And the way you have to do this, if you want sticky notes, right? If you want these little post-it sticky notes to move around, you have to paste it from like a spreadsheet tool. If you paste it from the Google Doc that I gave you, it will just paste as, you know, text. So you have to transfer it over here first, or at least this is the way I figured it out, right? You have to transfer it over into some spreadsheet or table tool, and then paste it. And it'll ask, Miro will ask you, "Do you want to paste as table or paste a sticky note?" We want sticky notes because we can move them around. If you paste as table, it's just going to, you know, give you the spreadsheet. So let's get rid of that and paste as sticky notes instead. Now, you'll notice in relation to the actual setup that I have, these are super tiny. Miro does that sometimes. I think this is their default size. And I've just made everything bigger. Um, but yeah, just know that you might have to go hunting for it sometimes. Now that we've set this up, you will see that this is just our participants' quotes, not in any order or arrangement. In other words, unclustered, unorganized data, ripe for our affinity diagramming.

Now, I have had my live master class students run an experiment. And the question is, how does human analysis compare to AI clustering? More specifically, Miro AI clustering. Now, before I go any further, I want to say, as of this recording, the Miro AI is still in beta. AI is always evolving and getting better. So what you see today may be different in a few months. Maybe they've gotten super good in a few months. So keep that in mind. I've had my live master class run an experiment. First, I have them make two copies of the original data. And this is, this is just academic researcher habit in me, which is I never touch the raw data. I just make copies of it, and I'll do the analysis in it, and then, you know, kind of keep track. So first set is your data. Then make two copies. The first copy, I want you to do analysis by hand. Do your affinity diagram by hand, and I want you to find your own themes. Then in the second one, run the analysis with the Miro AI. Only at that point, compare how your themes compared to the AI themes. And then you could be done. But I want you to take one step further. After you've done the hand analysis, make a copy of that, and then run the Miro AI on it. So rather than just running the Miro AI right away, you're going to do the hand analysis and then make a copy of it, and then run the AI on it. In my opinion, I believe the human plus AI path is the most useful result, as it summarized and allowed for different perspectives on the story. And that's what I recommend doing as well. Since this lesson isn't about having you doing analysis by hand, I'm not going to force you to do it. But bonus badass points if you do, because perfect practice makes perfect. Not that much data anyway. And if you want, you can do a co-synthesis with a buddy, make it a little faster. But yeah, you're going to see why the human plus AI approach is superb. So don't be lazy, practice.

All right, so create your own Miro board with the exact same setup. And here's how you do the Miro AI, kind of simple. So let's go over here. We're going to apply the Miro AI. Only highlight all your sticky notes. And this is the icon you're looking for: Miro AI. So let's just, blue star icon. And you can summarize the data, you can cluster by keywords, cluster by sentiment. I believe there's another option where you could, uh, cluster by tags. Oh, cluster objects. Oh, yeah, you can cluster by color, tag, author, keywords, or sentiment. In fact, you don't even have to use the AI button. You can just go straight here to cluster objects, which they've always had. But yeah, as you can see, the beta is the AI sentiment is AI, color, tag, if you've tagged it, if you have any other color coding schemes, then you can cluster by that too. But let's try the first option, which is summarize. It's going to give you one sticky note. Oh, uh, let me make this bigger. It's going to try to summarize all your sticky notes, which in this case, it doesn't really make sense because we have three different participants with three different needs and three different motivations. Um, and it's going to summarize into one. "Couple renovates farmhouse style home, gathers inspiration from online resources, and tracks budget on Google Sheets." Now, if you read all the quotes from the participants, they're fake participants, you'll realize that this sticky note only applies to participant one. It doesn't summarize participant two or three. But you notice I highlighted all of them and only summarized participant one. So it's not correct. I don't know what it's doing in the backend. But let's try it again. "Retired couple." You know, so retired couple refers to participant three, not participant one or two. So yeah, it's not exactly correct. Let's see. Let me just grab like one and a half of it too and see what does something. We're just doing live here, right? I, I'm not scripting this, just seeing what the AI does. All right, first summary was about P1. Second summary with retired couple. This is about third participant. And then, uh, okay, again, retired couple. Participant one and two are not retired. It was only participant three. So these summaries are not correct. In my experience, I have never found the summary AI to work very well. So I don't even use it. Maybe it'll get better. So ignore the summary for now. You can summarize better, I'm sure.

Let's try clustering by keywords. What happens is Miro would generate a cluster, clusters with black boxes around it, and you can treat these as possible emergent themes. Okay, let's try that. Okay, so the first thing is, um, it went kind of outside our frame. You can make the frame fit it, or you can, uh, re, you know, put these over here. And, uh, you notice that I dragged the black title. It took me forever to realize that you had to click this to drag the whole set, or I kept trying to like drag this. Sorry, I kept trying to do this, but it wouldn't do it. So you have to actually drag it by, uh, the black title. Okay, so let's do that first. After every AI result, my recommendation for you is to always look at it, check it, don't take it at face value. It's pretty neat that the AI can, you know, all you have to do is click a button and it summarizes, uh, uh, clusters things maybe 60% of the way. You'll notice compared to the human analysis, one, two, three, four, five, five. This one's not clustered yet, but five clusters. And here we have two, three. We have a lot more than five, right? There's 11 clusters. The question is, what is going to be the most useful way for you to analyze your data? What's the most useful story you can tell from your data? For that, you go back to your research questions, your main objectives. What are you trying to extract from your analysis?

What you can do also with the AI tool is first cluster by keywords, and then you can apply cluster by sentiment. But before you do that, and again, I mentioned my researcher habit, which is always make a copy before you do something. And then apply sentiment. I don't think so. What I imagine this will do, so they're clustered by keyword, and this cluster only has two sticky notes, right? I don't think there's going to be much insight drawn from that. If you, if you try to apply the cluster by sentiment to everything, it's just going to treat it as one bunch, almost like clustering this set here. So it's not very useful. What you could do is take a cluster and then further cluster by sentiment. So let's apply only there, right? It's going to take your theme or your emergent theme or possible theme and then cluster by sentiment. So that's pretty useful, I think. Find some main clusters and then further cluster by sentiment, which I think is pretty cool. Um, so let's try it with this bigger group too. Um, highlight that cluster by sentiment. Boom. Uh, okay, well, it got in the way over here, but there you go. Create some sentiments, create some new, you know, black frames around it. Sometimes I don't fully agree with the clustering Miro AI does. So you can undo it or just drag the stickies where you think makes most sense. The takeaway I want you to get from here is try comparing the data that you get versus Miro AI only, and then try applying AI to your human analysis to see if you can extract a different perspective. Maybe the AI can give you a different perspective on the data, which I think is probably the most useful use case of AI in analysis. And when I mention AI, I'm specifically talking about Miro AI. It's a very common tool to use. So yeah, and there you go. That's data analysis using Miro AI. Saves us a bit of time, or at worst, provides us some different perspectives that we can approach our data with and our story. Again, at the end of the day, AI results, any tools that you use, treat it as a starting point. Don't take it at face value. Use it to augment yourself as a badass UX leader. Don't use it to replace our work. But if you have additional tips or questions about analysis using Miro, let me know.

Empathy map. Redundant? What? But the Nielsen Norman Group says I'm going to tell you in this lesson why empathy maps are useless and redundant, and why I've never made one. I don't plan to. And I don't just say this because I want to be controversial. I say this because I've given it a fair chance. I've tried to find a use for it, but honestly, they're redundant. And I'll explain why in this next lesson. Also, the deliverable that makes this redundant is the user journey map. But in this lesson, we're just talking about the empathy map. So let's start off with what is it? What is an empathy map? Well, here you can see that it takes four elements of what a user says, thinks, does, and feels, right? What are their quotes? What are they thinking? What are they planning? What do they do? What are their behaviors and their emotions? How are they feeling? And you map that out into this quadrant, and you show your stakeholders. Here's the empathy map to have them build empathy with your user. My honest opinion, you know, my opinion, it's redundant. And the reason for that is that if you do the generative or foundational research, you're going to come up with the same information, the same data that you could use to produce a user journey map, which I argue is much more useful. It's way better visualized, more actionable than an empathy map. So why have an empathy map at all? To me, it's duplicate data, and you're just creating a deliverable that is just for the sake of creating another deliverable, which is not going to be very useful. I've never made a formal empathy map. My stakeholders have never asked me for one, and they never needed an empathy map. And I don't plan to.

Here's a message exchange with one of my alumni who works at a large retail company as a UX researcher. They told me that their company really values journey maps and empathy maps. And I asked them, "Oh, how do you use empathy maps?" And this was the response: "Honestly, I find empathy maps useless. My director requested persona, journey map, and empathy map, and it felt redundant to be honest." So don't, don't just take it from me. This is a real researcher working at a large company also finding it useless. So in my advice, you don't need it. Don't waste your time with empathy maps. What you do want to spend your time on instead is use a journey map.

All right, lovely researchers, let's do a pop quiz in this lesson. Let's start off with this question. Let's say you conducted a study with six participants, and something occurs with two of them. Doesn't matter what the study is, doesn't matter what you found. What I want to know is, what do you report? A, 33%? B, two of six? C, a few? Or D, one-third? Let you think about this for a second. Two of six participants, you saw something. What do you report? Now, if you said B, you'd be correct. I would also give credit if you said C, if you said "a few." And we'll talk about why in a second. But let's try two more of these questions.

All right, let's say you did a study with 20 participants. What do you report here? Do you report 40%? 8 of 20? Or some? You said B or C, I give you that. Let's try one more. How about this one? 10 participants. All 10 people will observe something. A, 100%? B, 10 of 10? Or C, all? All right, it's B or C. B or C are okay with me, just not the percentages.

Now, in this lesson, I wanted to talk about how do you actually report small sample size numbers. Okay, reporting percentages with small sample sizes is misleading. Reporting percentages with small sample sizes, especially less than 30, is misleading. We can reference the Central Limit Theorem for the reason here. I'm not going to cover that, but you can definitely look that up. Central Limit Theorem, basically the rule of thumb for when you can start doing percentages or a lot of statistics power is kind of a minimum size of 30. That doesn't apply to every scenario, of course, but that's kind of just your packet of napkin rule of thumb. Sample size less than 30, don't use percentages.

Now, what do you report? Now, you notice how there are alternatives to using numbers, raw numbers. You can also use words. For example, if we had a sample size of eight, and depending on how many people observed or experienced something, this is how you can translate or label these numbers into words. Now, first, before I go into this, this is a hotly debated topic. There's many discussions. A lot of researchers will live and die by one way, and others don't really care. I've experienced both of them and across many stakeholders. Uh, I think I found a good way, a good compromise to not piss anyone off and also to remain clear and not confusing. So let's walk through this table. First, if no participants, you could just say none or zero. One person, then you can say one, two, or three. I would say few. If it's four out of eight, then it's exactly half. Five to six people out of eight, that would say some participants. And if it was seven out of eight, I would say most participants. Uh, you can also use many when it gets to kind of that six to seven range. But let's keep it simple. And

Obviously, eight out of eight, you could say. All, let's try a different sample size. 20. Okay, 20 people. This is how I kind of break it down. Now, this isn't a science. Uh, this is more up to your discretion, but just not, don't be misleading. So, zero, same thing, it's none. Uh, 1 to 4 participants out of 20, I would say a few. 5 to 8, I'd say several. 9 to 11, about about half. 12 to 16, I would use the word some. 17 to 19, I would use many or most participants did this, and then 20, obviously, would be all.

Now, my takeaways from the many years and working with many, many stakeholders here's what I recommend. Do use the raw number or the absolute number and focus on the insights. If you are going to use the number, you know, the left column version of it, do include the number even if you use the quantity term. So, the the English word of it. So, if you use the word, I would also include the number because a few could mean two to three to me, but a few could be perceived as something else to someone else. So, it's very important to also include the number. Now, whether that's in a fit note, a small italics or gray text, or just write at the end of your sentence, it doesn't matter as long as you have it somewhere for some people, for people to reference it.

The reason I do this is because when I tell the story of small sample sizes, I don't want my stakeholders to focus on the numbers. I don't want them to focus on the numbers. I want them to focus on the insights, the qualitative portion of my research. Uh, and this applies to small sample sizes. Okay? So, if I'm doing 10 participants, anything less than 30 or 20-ish, then I will use the word "most participants did this," and we saw that, and here's why. I want to guide them away from the numbers and focus on the qualitative insights here. Obviously, I don't want you to use percentages if your sample size is less than 30. Okay?

So, remember these key, uh, ways to report small sample sizes. Uh, and here's just a note on reporting numbers in general. Let's say we do have of a sample size larger than 30. If you do report the percentage, I would also recommend including the confidence interval and your sample size as well. So, mention your sample size, your N equals blah, your confidence interval, your CI, and that's the safest thing to do. We're going to cover confidence intervals later in the in the stats and the Quant module. Uh, but just keep that in mind. Keep this term ready for later.

And since we're talking about qualitative studies, uh, I had a few words I want to talk about about the prevalence of usability issues. When you're conducting a usability test with, let's say, five to eight participants, the real-life observed prevalence or the rate of your observed issue may be significantly higher, but your stakeholders might not believe it. And then we have a question about frequency and impact or severity. So, we have three questions we talked about in the previous lesson. This is prevalence, severity. What, what I say? Frequency, prevalence, severity. Severity is the impact on your user. Is whether it's going to be positive or negative impact. Prevalence is how many users. And frequency is how often participants are going to see this issue. So, the question to ask ourselves is, how critical is this problem? If your answer to all of that is yes, yes, yes, it is severe, it's prevalent, and is frequent, it's a high priority issue. And so, we get into the realm of prioritization of our issues, and that's a really important thing to do too. We don't want to just report our findings, we want to report the priority of our findings as well. Um, cuz, you know, think about it. If it's, if your product is a hand warmer for using a cell phone, I don't, correct me if I'm wrong, I don't think anyone's going to die if those gloves aren't going to work for your cell phone. But on the other hand, if an issue accidentally leads to people deleting their personal data off of their account, that's an issue that you want to be certain fewer people will screw up. In fact, you probably want zero people to screw that up, right? This is a more severe issue.

Essentially, when talking about usability issues or any issues in general, we have to tie back to the business implications, user implications, set the context for our audience, for our stakeholders. The exact numbers again for small sample, any study is less important than the severity of the issues. So, this is the classic debate between quantitative data and qualitative data. They don't work against each other, they work together, they should work in tandem, in harmony, to provide more context for one or the other. So, folks, when you're doing sample, small sample size studies, focus on that qualitative aspect when you're telling the story. Just like I mentioned, when I report my research, I, I want to guide my stakeholders away from the numbers when it's a small sample study. I don't want to focus on a number, I want to focus on the issues and the stories.

Qualitative UX research methods. We'll cover some of the most common methods used, how to conduct them, when to conduct them, how to analyze them, pros and cons, and then show you examples of some of these methods in action. We go deep. Zooming out, I want you to know that you'll likely come across hundreds of different research methods with different names, like discourse analysis, content analysis, cultural probes, picto, whatever. For an overview, you can check out designresearchtechniques.com. And wow, if you look at it, look at all those methods. Scary, right? How the hell are we supposed to learn all of them? Don't worry. I assure you that once you learn the core fundamentals, you start to recognize that all of the other ones are just some variation of the core method. Not all of them, but most of them. Ask both a good thing and a bad thing. Good because the fact that there are variations means that you can get creative and adapt methods to fit your specific research needs. And that's the part I love about challenging research because maybe you don't have the budget to run it the way you're supposed to, or maybe you have the Gucci money and you want to try out crazy ideas like strapping body cameras to participants. The bad, well, I've been in your shoes trying to learn UX research, and people write articles calling the same things different names. It was frustrating. So, that's why in this module, we'll go over the common core methods in user research so you don't have to struggle during the learning process like I did. All right, enough jubber jabber. Let's get started with some research methods.

I remember seeing an old photo from the 1970s. Two men standing next to a fax machine, one of those big printers, you know? And they tried for a while but couldn't figure out how to make it work. And if you were that fax machine company, you might have assumed that these two guys are just dumb. Now, we know better than to blame users. Remember our story for module 2 about how Boeing B-17 pilots were blamed for crashing the planes? Yeah, don't blame users. Well, in fact, those two men fumbling around with the printer and fax machine were actually two really smart engineers from IBM. And I don't know about you, but even today, I still can't figure out how to use one of those things. I find it hilarious and sad because in the office, I lost count how many times I see the same person walking to and from the machine. I used to sit next to it, it was pretty noisy, and I would see people like, "Ah, how do you work this thing?" If anyone has figured out how to use one of those, please help. Thanks.

We learned in module 4 how to plan for research, which includes aligning with our team on objectives and planning, and getting feedback on your study plan. Always keep those steps in mind. Those are always step zero. In this section, in this section of module 7, we'll learn specifically about usability testing. What it entails, different types of usability tests, its pros and cons, how to write tasks, and different ways you can conduct it. It was my bread and butter when I first started out. I've done at least a thousand of them. Yes, more than a thousand usability tests. I probably 2,000. I've lost count, but I know it's in the thousands, or at least a thousand. It's like the pull-up, the gateway to other bodyweight exercises you've got to master to pull-up before you could do other things. And that's like usability testing. It's a necessary method for being a modern user researcher. I'm not saying people have to know usability testing to be a user researcher because not all roles require you to do it, but if you're taking this master class, you better bet that UX leaders do know how to do it. So, for reference, I uploaded a YouTube video on usability testing back in 2019. So, if you want to watch a younger, better-looking, more energetic Kevin without back and knee pain, I invite you to watch that to really let the information sink in. Repetition helps. Repetition helps.

Usability testing has its roots in ergonomics in the early 20th century. It's an evaluative research method where we ask participants to complete certain tasks using our product or service. It can help us see how users perform on an interface, uncover problems with design, find out new opportunities to to improve, and learn about users' behaviors and preferences. And usability testing is what we do to prevent people from struggle-busing around fax machines, questioning their sanity. It's what we do to save people's lives from dumb design oversights. It's what we do to help our users feel confident and happy. And when we say usability testing, most of the time people are referring to something called task-based scenarios, which are user goals turned into tasks, and we ask participants to show us how they would complete that task. But believe it or not, there are many kinds of usability tests, some of which you may have heard of before, like the 5-second test, write testing, first-click testing, gorilla usability, and quantitative usability testing. So, in the following videos, I'll dive deeper into each of these variations and we'll talk about how we can perform usability tests, whether you choose to do them remotely or in person, moderated or unmoderated, whether you could do them as purely qualitative or comparatively quantitatively. Yes, there is such a thing as quantitative usability testing as well.

Looking at the different ways we can conduct usability testing, we can see that we don't just have to pick one way to run it. We can use a combination, mix and match depending on your needs. You can do in-person unmoderated, you can do remote unmoderated, you can do remote moderated. The possibilities are endless. And we should do usability early and often. Usually, we conduct them after a product or concept has been validated so we can focus on optimizing the experience because who cares if people can use something if you don't even know if people will use it in the first place? So, don't be fooled, just because it's evaluative doesn't mean usability testing cannot inspire strategic changes. So, here are some common topics that usability testing can help us better understand. We can understand user awareness, user onboarding, if users know something exists or not, whether they perceive some benefit that this feature brings, whether they think the feature might help them. We can also understand desirability. Users choose not to use the feature because they don't value the benefit. We can also understand discoverability. Do they know some feature exists but they can't find it? And then finally, I mean, obviously usability, where people know that a feature exists, they know where to locate it, but it's somehow unusable. As you can see, while some of these areas are usability fixes, some of them require larger strategic efforts too.

Now, we learned what usability testing is for. Let's chat about sampling and sample size. We'll learn how to sample for usability tests and what sample size to use. The big question: how many people? So, really quickly, let's talk about who to recruit. I would say, do not recruit people who are you. What do I mean by that? I mean, don't recruit your teammates, don't recruit yourself, not your grandma, unless your grandma is a target user. Essentially, anyone who knows too much about the product, do not recruit them because you're not going to learn anything. Recruit your target audience, someone who would be using our product or service. So, consider demographics, psychographics, and any specific criteria relevant to your project. So, if you're making an app for nurses, you're not going to go and test it with lawyers or construction workers. Also, consider people of accessibility needs too. Get a range of people in your target demographic. For example, for one project, I'll be studying business owners, entrepreneurs, hiring managers, basically people who hire others for their business, and I made sure to get a range of ages, gender, business types, and I ended up with participants from around the world, from the UK to US, Italy, Kenya, and many other places.

But Kevin, what about sample size? Are we supposed to get saturated for each criteria? That's a great question. So, let's talk about sample size. You probably heard it before, the magic number five. Five participants for usability tests, which can uncover around 80-85% of usability problems. Early 1990s, Robert Virzi and Jacob Nielsen did a lot of studies on how many problems can be caught with a certain number of participants. Now, you've probably heard of the big daddy, Jacob Nielsen, from Nielsen Norman Group, but Robert Virzi and Jim Lewis contributed a lot to this magic number five as well. And fun fact, when I interviewed with an OG UX research manager, I mentioned Robert Virzi and his paper, and this guy was surprised to hear that I actually read that paper and knew his name. Robert Virzi. Since Nielsen wasn't the only one who came up with that sample size of five, bonus points, read that paper. By the way, I'll link an article from MeasuringU for you to read. So, anyway, five users can get us 80% of the problems. And that if we tested more people on the same design, we would get diminishing returns. We're not going to learn much more from the 20th person. And the point of usability testing is that it's quick, it's iterative. The point of it isn't to catch all problems. A better use of resources would be to test with five users, catch these major problems, fix them, and then test with a different five. And in my experience, I'd say it's spot-on. Five is usually enough for one design. But I also like to recruit a few extra between, you know, six to eight people total, just in case somebody doesn't show up and to have a little bit of extra information.

Now, I don't know if you caught that just now. What I just said, I said five is usually enough for one design. Why? Because it's also important to consider how many prototypes you're testing. Because if you're testing more than one, we need to consider whether to do this within-subjects or with between-subjects, and that can affect our sample size as well. But what if it instead of multiple prototypes, we had multiple criteria? Now, for my study, I got five participants total. I think it was six or something, but each person was from a different country: the US, Italy, Kenya, Singapore, etc. Is that important? Do we need five from each country or just five total? That's a good question, my friend. I had a question like this myself too, and we have to ask ourselves, do we care about saturation for certain criteria, or do we just care about finding usability issues? So, it depends on our goals. The classic "it depends" answer. When it comes to usability testing, a total of five target users should do the trick. Now, if you have a strong hypothesis that user needs in the UK are going to be drastically different from user needs in Kenya, then you might need a separate study to localize and design and co-create for your users. So, sample size for usability testing, five should do it. But practically speaking, I think between 6 to 8 is okay. So, in short, recruit your target users and a minimum size of five should do the trick.

Have you ever done any online shopping only to be confused by their categories on the website? Oh my gosh, the organization or categorization of items is called Information Architecture. IA for short, not AI, IA. And in this lesson, we're going to be talking about card sorting and information architecture. What it is, how we conduct it, and how to analyze it with or without software. Yes, without software. Let's get started. Let's go find out how things should be organized. We conduct something called an open card sort study. And card sorting is when you give users cards of the items and have them arrange them in a certain way that makes sense to them, which helps make navigating your product more intuitive. Just like how you might sort your clothes for laundry, or how you like your spice rack to be. A change, whatever the case. The point is, we all group things into groups that make sense to us. And if people aren't able to easily find things on your website because maybe we've haphazardly put menu items all over the place, then it's not a good time. So, we use card sorting to identify how we should group things together more intuitively. At times, we might do a heuristic evaluation or just look at the dang thing and realize it's harder to navigate around the site than you thought, or users may tell you that they're going to the wrong places, or it's just hard to find something, or we can check analytics and see that users are spending a long time finding what they need on your website. All of these are possible signals that our website ain't too organized. Often times, we might even identify that it's not the navigation itself that's confusing, but the jargon and copy that we use could be confusing to users as well. So, a card sort task can help us organize a website and gather some comprehension on certain words.

There are different ways we can run a card sort. We can run it open or closed. We can do it in person with physical cards or digitally online. We can choose to moderate it or go unmoderated. And I'll discuss a variation called the Dyadic card sort. So, let's talk about the open versus closed card sorts first. Open card sorts are used to develop a new information architecture. What we do is give participants the cards without any order and have them arrange them. And the task, what we give is basically, "Organize these into groups that make sense to you." Pretty easy. In a closed card sort, we already have predefined categories or spaces. So, we use this method to validate an existing information architecture and see where people might organize things underneath our predefined categories. And the task we would give them is simply, "Organize the cards into the groups you see here." And that's the main difference between an open and a closed card sort. So, why ever do a closed card sort? Well, for example, one project I did at Google had both open and closed elements to it. We wanted to find out how to organize our menu options for our smartwatch product. In this case, which card sort would you use? Open and closed? I would use an open card sort. But another research question that we had for the study, the designer had, was which menu options should be on the phone, and which one should be on the smartwatch, and which one should be on both devices? In this case, we do a closed card sort because we had three predefined options already: phone, smartwatch, and both. Well, with that quick intro to card sorting, let's head over to the next lesson where we'll talk about moderated versus unmoderated card sorts, sample size, how many cards you should have, and how to actually conduct one.

To moderate or not to moderate, that is the question. What kind of stupid ass question is that? Whether it is nobler in the mind to probe and ask why, or to take our sort online to provide higher power for analysis? Shakespeare wrote that. I'm just kidding. Let's talk about the pros and cons of moderated card sorts versus unmoderated cards. So, moderated studies, we get to probe and ask why as your participants sort their cards. We know that we have the ability to troubleshoot and help them out if they get stuck, and we have more control over the situation to gather the data that we need. Of course, with any moderated studies, we'd have to spend time moderating and scheduling people. Common pros and cons of moderated studies. Unmoderated studies, we can use certain tools online and have participants do the card sort on their own time, thus saving us time. So, in a way, a pro is that we don't have to moderate. Although that's also the con, where we're unable to probe or ask why for certain items. Sure, we can provide open-ended comment boxes for people, but if you have 60 or more cards, I can assure you they ain't going to do that. They're going to comment on the most prominent ones, which is still helpful, but they're not going to comment on every single card. If you're able to get folks to think aloud during a card sort, that's great.

Sample size. Let's talk about that. I think you can get away with fewer participants for a moderated study because you're able to probe deeper into the rationale. A good number could be between 10 to 15. For unmoderated studies, according to Jeff Sauro and Tom Tullis, the sample size can range from 20 or more than 30 participants to give us enough power for analysis. For safe reasons, I would go for 30. Central Limit Theorem, minimum sample size of 30. If you're going to go unmoderated, and if you're going to moderate it, 10 to 15 people, I think. So, another question is, how many cards do we give participants? Should we give them everything we have? Every single item? What if we have like a hundred items, 100 cards? Well, if you're conducting this in person, I'd say anywhere from 30 to 60 cards is a good number to have. People might fatigue if you give them more. Have fewer cards here because it's more cognitively taxing. If you're going to have more cards, make sure to give people breaks. Now, if you're conducting this online, anywhere between 30 to 80 cards is okay. Now, some articles say you can go up to 100 cards, but I'll say this: if you have 100 cards, why do you have 100 cards to begin with? That's a lot. That's a lot of menu options. Yeah, sure, you maybe have a complex website. We need to trim it down, speak to our stakeholders, see which one we know for sure we keep, and which ones are not that important to know about. There may be cards already that you know where to place them. So, maybe you don't give all of them to users, or you can split them into two separate studies, or you can kind of mix and match groups of users with certain groups of cards so they don't have to be fatigued.

Speaking, speaking of digital, I'll mention a few tools that you can use for online card sorting. The most popular one being OptimalSort from Optimal Workshop. UXtweak also has it. WebSort also has it. There's a lot of tools with this ability. Some tools have free capacity, or after a certain amount of cards, you'll need to pay. If you're totally on a budget and want to hack your way through it for free, you can use something like Trello, something where you can just drag and drop cards. Miro can even work for this. Google Jamboard, if anyone uses that. Do. Anyway, here's five easy steps on how to conduct a card sort. Step one: Prepare the cards on your tool or table. If you're doing it in person, label them with numbers so you can refer back to them easily, and also have some blank cards as well. So, that's a setup. Number two. Step two is instruct participants to think aloud and then rearrange until they're satisfied. If they are unsure, we can probe and ask why, and then we can note down what was confusing. We can also let them know that they can put cards aside in a separate pile if nothing makes sense to them for now. If they think something is missing, that's what the blank cards are for. They can create their own using those blank cards. So, while they're conducting, we're going to observe them, we're going to note down, and ideally, we would record their progression. Step three: After their participant finishes, take a look at the groups they formed. How many groups did they form? And ask them the rationale. If it looks like there's too many groups, ask if any can be combined. If there's too few groups, ask if any could be separated. Step four: Have participants label the groups. They've probably already did this in their minds, but this is just formality. Have them actually name it. And finally, step five: Take a photo or screenshot of their groupings. You can try this on your own too. Just go to any website, any website of your choice, your friend's website, Trainer Joe's Best Choice Market, HB, and look at their navigation menu. Write down all their sub-menus on the cards and ignore their initial categories. Then have a few friends or participants to try this with. Who knows, you might even find some awesome improvements to the navigation. And you know what? Here's a secret pro tip for you. You don't have to sort just words. You can also sort images or icons onto a page. For example, you can provide an empty wireframe to participants and give them icons, images, even entire sections to sort out. See how creative we can get with our methods. In a way, this is kind of like a co-design exercise, but it is related to information architecture and hierarchy and how things are organized. So, you can also do it with images.

[Music]

So, that's how not to do field research. Field research itself is an umbrella term because it can refer to an amalgam of methods such as observation, contextual inquiry, and diary studies. I like to point out something that might sound controversial though. UX researchers often say they do ethnography as a form of field research. Ethno, it's Greek root for ethnos, which means people, nation. Graphy, graphy is study of. To all the anthropologists or social scientists out there, I apologize on behalf of user researchers who say we do ethnography. We do not do true ethnography. We do ethnographic research in the form of contextual inquiry or observation, but not true ethnography. Ethnography is the study of people's cultures, customs, and behaviors over an extended period of time. An anthropologist might even live with the people for years to get all the nuances and dynamics of everyday life, or dress up as a zebra and experience what it's like to be in the plains of Africa, lion meat. What we, user researchers, do in fact is observe participants in the wild, in their natural environments, but we're usually doing it no longer than a few hours or a few days at most. And the scope is much more focused on certain user goals or experiences within a particular context. And that's the difference. Now, for the purpose of this lesson, we're going to talk about specifically contextual inquiry. Now, throughout the next few video lessons, I'll break down the steps to plan, conduct contextual inquiry, along with who we recruit, where to go, a bunch of logistics, and how to involve our stakeholders.

Contextual inquiry is the most common form of ethnographic field research where we combine observation with interviews in order to observe participants' behaviors in context. Examples can include going to people's homes, their office space, going to a hospital to research healthcare professionals, manufacturing plants to observe inefficiencies or efficiencies, and many, many possibilities. The word "inquiry" literally means the act of asking questions, which to me is the interview portion of contextual inquiry. Now, combine observation with interviews, and we have contextual inquiry. Voila. We know what interviews are. What's observation? Nice. Observation is a classic research methodology. It could also be called fly-on-the-wall observation, where we observe people in their natural environment. Most of the time, we might not interact with the subjects at all, kind of like creeping on people. Example: you want to see how people move through a subway station, or count how many people use their vehicle turn signals, or how people greet each other in different cultures. Another field research technique is going undercover as a user and seeing how people interact with us. This is supposed to be a mustache. I forgot to bring one. Anyway, these going undercover studies are called mystery shopper studies or service safaris. What happens is we pretend to be a customer and ask questions about the product to staff or customer support. Alternatively, you can ask questions to other customers and learn about their decision-making processes and purchasing or not. Now, we discussed the ethics of this research method back in an earlier module, and since it's not instilling any harm on participants, this is fine to do. Now, with this method, mystery shopping, we can look out for opportunities for improvement in the service process, knowledge of customer support agents, or and you can compare it to other stores, also other things that a shopper would experience.

So, we talked about what contextual inquiry is. How about when do we do contextual inquiry? Well, we can do it if the team is struggling to find the root causes for a problem that your product is supposed to solve. We also do contextual inquiry when we want to learn more about the user contexts for tasks, unmet needs, uh, or to test your product in the wild. I'm in the wild suburbs. These are great times for contextual inquiry to understand the foundation of what our users are doing and provide us with more reliable sources of truth when the team begins brainstorming solutions. Yay!

We finally get to talk about my favorite research method: diary studies. Mainly because I love snooping around my brother's diary as a kid. Just kidding, I don't have a brother. Ha, gotcha. But it is my favorite method because it's just so versatile in how we can really get creative with how we conduct it. Diary studies aren't just run one way. In this section, we'll talk about what diary studies are, how to conduct one, the pros and cons, and I'll share two really in-depth case studies with you from my days at Google and Uber where I used this method.

Diary studies, also known as cultural probes, are a great substitute for field research. It's a longitudinal method where we can ask participants to keep a diary of their day-to-day lives to capture attitudes and behaviors over a period of time. And that's what longitudinal means: over a period of time. And we can run a diary study for as long or short as we want, from a few days to even years. Now, when might we do diary studies? When might we want to use it? Well, to find out user motivations behind their behaviors in context, understand how small actions over time affect larger decisions, and why. We can also see how people approach their goals in different ways and get really, really rich data about it. Uh, we can also learn how experiences change over time, whether it's learning a new tool or something else. We can learn about things that people don't normally do and why, why not.

Diary studies are very useful to help us find out about, for example, your current product or prototypes. We can send people your product or prototype and have them interact with them over time. We can also learn about general behavior and activities, understand what and how people do things in different contexts, specific activities or tasks. We can understand how people do certain things like buying a flight ticket, what goes into buying a flight ticket, not just the act of buying a flight ticket, but what comes before and after over time, what's their emotions, also ordering delivery, or how they achieve their personal goals. Very foundational use cases for diary studies.

Now, how do we conduct one? Like I mentioned, there's many different ways you can conduct it. I'll go really in-depth with two case studies so you can see that there's a variety of ways that you can conduct a diary study. But let's get the basics down first. What's the first step in every research study? That's right. And this is automatic for you now. Step zero is defining the objectives and research questions. Picking good participants is also crucial. So, get your target criteria and recruit a sample. Then figure out our stimulus for the question that we have at hand, your product or service or understanding, and send your participants, if you have something that you need to send them. And finally, figure out how often you want your participants to do your diary. Is it daily, at the beginning or end of the day, every 2 hours, or do you want to fill a diary study only after a certain trigger or specific event? This is something you have to decide based on your research goals. After that, I highly suggest bookending our diary study with in-depth interviews. Meaning, start the study off with a user interview, then off they go, diary study in between, and we can close the diary study off with kind of a, what we call an exit interview, a final exit interview. That I find is the best way to let participants know that there's a human being on the other side willing to help you, as well as recapping the study once you're done. And I don't do my diary studies without those two interviews at the end.

And lastly, let's talk about the pros and cons of this diary method. Pros: Everything. Cons: Nothing, cuz I love diary studies. I'm not biased at all. I'm just kidding, I'm kidding. Who am I? The pros: When you go do field studies and you're observing someone, you may unintentionally come across the Hawthorne effect. But here, with the diary study, we minimize that Hawthorne effect because we're not actually observing them. We don't look over their shoulders and watch them. We can be when and where your participants are without actually having to be there, which is a really strong advantage of diary studies. It's a very efficient way to collect data on natural behavior and see trends over time in different contexts. And as we'll see soon, it's so versatile and it's, you can use it with really adaptive methods. Not to mention that it is a, a good alternative to field studies. While the pros are amazing, I do have to admit there may be some cons to this favorite method of mine. It is pretty time and money intensive. You can't do a diary study one day. I mean, you could, but you won't get much data that way. And the point of diary studies, you have to remember that the advantage of doing diary studies is so that you can see behaviors and attitudes, trends over time. And one day isn't a lot of time for you to see any trends, even two days. So, you really got to think about how long you want to do your diary study for. But anyway, it is time intensive in that aspect. Diary studies are also based on self-report, which may elicit inaccurate data because people might only report things that they think are interesting and omit things that they think are not interesting. In which case, there are workarounds. This is the crux of user research in general. We see and report what we see and what we hear, but it's also important to read between the lines. Read between the lines. What aren't people saying? What pe-, what aren't people doing? And that's when we have to ask follow-up questions. That's the workaround. Now, one common con or disadvantage of diary studies is that people, researchers might say, "Oh, but you can't observe behavior directly." In my next couple videos, I'll show you how exactly I got around that con by using body cameras or video vlogs, video vlogs, video vlogs. One really major con of the diary study method is attrition rates or drop-off rates because the longer you study, the more people that are going to drop out of your study. We can shorten the study, focus it a little bit more, or provide some milestones for incentives to keep them engaged.

This one we're going to talk about between versus within-subjects design. Let's go. If you're planning a usability test with two prototypes, we can do several things. One, test each prototype independently with different participants. Or two, test both prototypes with the same participant. Now, if you're testing them independently, this is something called between-subjects design. Participants are seeing both prototypes. This is called within-subjects. Just for fun, these, I'm just going to, I'm going to throw more jargon and confusion at you. You know, because I want you to suffer. No, I'm just kidding, I'm just kidding, I'm just kidding. It's because I want to prepare you for the real world, which consists of people using way too many terms for the same thing. And that's why you're taking this class so I can cut the BS. So, here it goes. Between-subjects design is also known as between groups or independent measures. Within-subjects is also known as within groups or repeated measures. Now, for the purpose of this master class, and honestly, let's make the world a simpler place by using the same term. I'm going to stick with using the words between and within subjects. Since they're dense words and people always get them mixed up, like me, I came up with some terrible analogies to help you remember what they, which one's which. For between-subjects, remember, between-subjects is when participants independently experience your conditions separately, right? Let's say you have two prototypes, you know, one person takes a prototype and another person takes the other prototype. Each person does not see both. So, anyway, within-subject design, I like to imagine a river that's in between two lovers standing across from each other. The lovers never interact because they're separated by a river that's in between them. #sadstory #getbetterhashtags. It, I told you these are dumb. Now, for within-subjects design. All right, this is when participants experience multiple treatments or conditions or prototypes. In that case, imagine you have multiple superpowers within yourself, which you all do. You get to activate all of them. That's the power within. So, go forth, badass UX leader, unleash all your powers. With great power comes great responsibility. I hope it helps because I always forget like, all is between-subjects, the one where they do both or you. Anyway, between-subjects, something comes in between them, present prevents people from interacting with another thing. Within-subjects, you have multiple powers within you, you get to use them all. Okay, so those are my analogies. I don't know, right? If you can come up with a better way to remember, I'm all for it.

"Kevin, um, is one design better than the other? Is within-subjects or between-subjects better?" Ah, well, I'm glad you asked, young Padawan. Anytime we ask if something's better than another, expect a conversation about pros and cons. Let's break it down then. Between-subject studies, the pros are there are no order effects, learning, or fatigue effects, uh, that may bias the second condition. And what I mean by that is, if you're doing a between-subject, that means participants are only seeing one variation, and they only see one variation, so they don't get tired. You don't get any order effects. Order effects meaning one variation affects the perception of another. Since they're only seeing one, you don't get that. Uh, shorter sessions, that's another pro because people only see one third. Between-subject studies are kind of easier to set up because you only have to set up one thing. You don't have to worry about order effects. Now, the cons of between-subjects are that you may need a larger sample size because participants only see one condition. If you want enough power or statistical significance to compare these two groups, if multiple variations, then you need a larger sample size. Uh, another con is that there may be individual differences that may affect the result. Of course, there's always individual differences, but there, they may be a little bit more pronounced when you're comparing one variation to another. Since we identified the con of between-subjects, there are ways around those cons. The best way to resolve the cons is randomly assigning participants to your conditions. Okay? So, let's say you have two prototypes, you just randomly assign your participants to either version A or version B. Randomly assign them because now those individual differences are not the result of the condition, but you, since you randomly assigned them.

So, let's talk about the pros and cons of within-subject studies. For within-subject designs, the pros are that individual differences are reduced because the same participant is seeing all the versions. Okay? So, if one person's really, really interested with one variation, you can see their behavior reflected with all of the variations. So, the individual differences aren't as pronounced. Within-subject designs, another pro is that you don't need as large of a sample size because they get to see all conditions, right? You don't have to pay more people to see the variations, you can just pay fewer people to see all the variations. Random noise is reduced. It's kind of related to individual differences. As finally, you can compare data within the designs. Okay? That's one thing you cannot do with between-subjects is that you can compare, get comparison data, right? With within-subjects, participants get to see more versions. Naturally, they're going to be like, "Well, compared to this one, it's blah, blah, blah." And between-subjects, participants only see one version, they have nothing to compare it to. So, you have no comparison data. So, those are pros of within-subjects. The cons. Okay, there's always pros and cons. Since you have to, you, there's no way a participant is physically able to see all variations without being affected by a different one. Okay? And the way you present the order in which you present them may bias participants. So, you see version A and then they see version B, they're going to compare it to version A. So, this is known as order effects. The order of the conditions can and they may affect their behavior. The performance may also be better because either, you know, so order effects can come in many, uh, flavors. Okay? It could be fatigue, right? I'm seeing one version, two version, three version, so on. I'm going to get tired and maybe by the fifth version, I don't want to do this anymore. I'm bored. I don't want to start anymore, right? Another thing that, another order effect flavor is that maybe by the fifth version, they get really good at the task that you presented them. For example, they've gotten practice by them. They could be fatigued, they could get better, they could be tired. So, all of these are related to order effects, and that's one major con to the within-subjects design. However, like I mentioned, with any con, there are workarounds, right? That's nice to know in life. When they're, when going gets tough, there's always a way. Now, the way around that con for within-subjects is something called counterbalancing. We can counterbalance the order that participants see a particular design. Counterbalancing, if you've never heard of this word, that's all right. We're going to talk about it in the next lesson.

All righty then. So, as you can see, to answer your question, little Padawan, neither design is better than the other because it depends on what your goals are. It's great that there are ways to resolve the issues with both experimental designs. But what the heck is randomization? What's counterbalancing? Another great question, lovely Padawan researcher. See, there's so much to think about when planning research, right? Oh, what are we measuring? What are we manipulating? Do I get comparison data, or do I want to have task times? Do I want to focus on one variation? Do I want them to see all variations? And if I want them to see all variations, how do I get rid of that order effect? Please stop. This is the rigor that goes into research planning, and y'all are on the way to being badass UX researchers by knowing this. So, with that, let's dive into counterbalancing and randomization.

If we have eight participants and three prototypes, we can't just show the prototypes in the same order for everyone. We can't. If everybody sees version one first, that will introduce order effects and affect the validity of our results because maybe version one affects how a user uses version two and so on, or maybe people just get tired over time and perform worse on version three. Whatever the reason for order effects, we must consider this order effect in which you show something to participants. Now, to reduce order effects and influence of these extraneous variables, it's another vocab word I want you to look up, extraneous variables. So, to reduce these order effects, we can do something called counterbalancing or randomization of the order in which we show the stimulus. It's really easy when you have two stimuli or designs. Okay, let's start with that example. Two designs. Let's do it for participant one. Actually, let me use these little buddies. It's really easy if you have two stimuli.

Or designs, let's call them A and B. And let's say we have six participants here, okay? So A and B for participant one, we can show design A and then B. Okay, so pretty easy, right? For second participant, we could just switch up the order, B first and then A. Participant three, switch the order, you know, switch it back. Easy peasy lemon squeezy.

All right, so that's pretty easy. Two designs, just switch them back and forth. Or you could even do this: A, B, A, B, A. And for these participants, B, A, B, A, B, A. Uh, in practical terms, I would switch them up every participant. Just because, you know, maybe you do A and B first day of your, of your study, and then see B and A for the second day. I'd like to see a low variation of both, just so I could tell my stakeholders, or at least share with my stakeholders, "Hey, there's some interesting things, and it has nothing to do with order effects." But it doesn't really matter as long as at the end of the study, you got both counterbalances.

Now, two designs, two stimuli, very easy, right? Just switch it up. But what if you had three designs, A, B, and C? So participant one, easy, right? Design A, design B, then design C. Only it's getting harder, you see, right? A, B, C. Now, what order should participant two be? Pur two? What order should that be? Hmm. Well, we could do design B first. Move this to the top. Yeah, this is hard. [Music] And then C, and then A to the last one. Yeah, this works. What would you do for third participant? Okay. Oh, yeah. How about this? We can move the orders up again. Design C, then A, then B. Whoa. Now, what about the fourth participant? Mmm. How about we start over with A, B, C? That makes sense, right?

And if we repeat the treatment orders like this: A, B, C, B, C, A, C, A, B, and then restart. This is what's known as the Latin Square design. But we want to avoid the Latin Square. Why is that? Here's the problem with the Latin Square: A always comes before B, and B always comes before C, and so on. So inherently, the Latin Square still has order effects.

So what do we do instead? Let's do some quick math. We're going to do some factorial combinations here. Two plus two is four, minus one, that's... Please ignore the old markers because I haven't used this in a while, and it's all solidified on there. So for N number of prototypes, we find out how many possible order combinations there could be. This is called N, N-fact, N factorial. So the factorial, this exclamation mark just means you're going to multiply all the numbers with that digit. So, for example, if you have three factorial, it's 1 * 2 * 3. This gives us six. Six possible combinations for having three prototypes. So easy, okay? Right.

So let's, let's try that. A, B, C, B, C, A, C, A, B. We did this before. Now, what are the other three? Because the factorial only tells you how many possible combinations there are, but it doesn't actually tell you what the combinations actually are. So let's try this: A, B, C, B, C, A, C, A, B. A, C, B. Let's switch these around. Uh, B, A, C, and C, B, A. And those are the six possible combinations that you can have if you had three prototypes.

Let's try it with four. I'm not, I don't think I could write every one of them out. But four times four, this gives us 24. 24 combinations. Now, I ain't got time to write out 24 of these combinations. We got the first six. If you want to find out the other 18, be my guess. But the good thing is, you probably won't be testing with 24 participants, nor should you be testing with four prototypes anyway. And honestly, whenever I do prototype testing, I always ask my stakeholders to limit their choice down to three maximum.

Now, if you want to pseudo-randomize assigning people to your conditions, here are the steps to do so. First is to find out N factorial to find out how many order combinations you have. Then you write down the combinations. And at three, randomly assign an order to your participants. I call it pseudo-random because we still have some power over it. It's not true random. That's more like a, um, if you majored in math, you'll know what true random means. Uh, to find a random number, you can use a function in Google Sheets with this formula: equals RANDBETWEEN(your low number and your high number). Your low is one, and the high is N factorial.

Stanford University, 1971, was one of the most notorious experiments in modern psychology. Professor Philip Zimbardo attempted to study the effects of perceived power between prison guards and prisoners. But really, the overarching question was, does the external situation outside of your control dictate your behavior, or do the intrinsic morals of your being allow you to rise above those negative environments? What happens when you place somebody in a place of power? So college student participants were randomly assigned to be either a prison guard or prisoner. And for the next 14 days, the guards and the prisoners would assume their roles, unsure of the impact that the study would have on their psyche.

Later, the prisoners were made to do push-ups. They were beaten, humiliated, and other psychological torture. Guards would do this naturally, or by the instruction from Zimbardo? Yes, the professor, the scientist, actually intervened and instructed the guards to amp up their harshness. So if the "and characteristics" were a thing, it would be this study. A few days into the study, the prisoner participants would demand to withdraw from the study, but they weren't allowed to. What started off as a study began to muddle into the unknown. Even the professor didn't see what was happening. A study that was supposed to go for 14 days stopped just after six. And that's only because Zimbardo's wife told him to stop.

Fun fact: I ran into Zimbardo in the Stanford restroom before. I didn't talk to him in there or anything. Just 10 years earlier, in 1961, a Yale psychologist named Stanley Milgram conducted another notorious study, what would be known as the Milgram experiment. So he wanted to study to the extent to which people would obey authority. The experimenter was in a room with the participant while a fake test subject sat in another room, separated by a wall. Now, the fake test subject was someone who was in on the study. The participant was told that they would have to shock the student as they answered quiz questions. Now, the student, or the fake test subject, would pretend to answer quiz questions wrong, in which case the participant was told to shock them, and they would have to increase the voltage. The more they got wrong, and the test subject on the other side of the wall was instructed to scream. Now, the screams were fake because the test subject was just pretending to feel the pain, but the participant didn't know that. They thought they were actually shocking people, and they would question the researcher. They would want to stop, but the researcher would urge them to keep going, sometimes keep shocking them, sometimes administering fatal levels of electricity to the point where the test subject would stop screaming. So the participant thought they passed out, or worse.

So why do I share these two stories with you? Well, because in the first study, participants wanted to stop, but they weren't allowed to. And the participants were not debriefed until years after the experiment was finished. There's a lack of informed consent because they didn't know what was involved in the study. And finally, there were the "and characteristics" where guards would act the way that the researcher wanted them to act. Which ain't badass research to me, that's just bad research.

In the second experiment, the Milgram experiment, participants were exposed to extreme emotional stress and inflicted insight where a participant realizes a deep flaw about themselves. And despite f informed consent, the experimenter didn't ensure the well-being of the participants.

I'll leave you with one more example that was a bit more recent. In 2014, Facebook ran a massive psychological experiment on 700,000 users without them knowing. They manipulated people's news feeds to either contain more or less negative content and assess whether that affects people's emotions. Any lab could tell you that, "Oh, duh, of course it does." Why do you need to run a study to find this? Just do some secondary research. Like, what was the point of this study anyway?

These stories show the dark side of research and how it should not be done. We must learn to recognize what is bad research, not just protect ourselves, but our team, our company, and above all, our users. Let's learn how to keep our users safe and secure by conducting research responsibly.

Ethics are a set of moral principles to help us function for the greater good of society. And in the case of UX research, the safety and well-being of our customers. We're the Jedi of the UX universe, with the force of good. I couldn't find a lightsaber, so this is it. Why do we need ethics, though? Apart from defeating an ethical Sith Lords? To promote truth by prohibiting the falsifying or misrepresentation of data. Uphold respect, health, well-being, and safety of people. And, uh, to protect your ass, your company's ass, and our users' entire being.

If you come from academic research, you'll be familiar with the Institutional Review Board, or IRB. Biomedical, human behavior research, and academia. IRBs exist to ensure research methods are ethical. Their job is to analyze the risk-benefit of studies and determine what to approve or reject. They make sure appropriate steps are taken to protect the rights and welfare of human participants. Many people ask, "What resources are there to learn about research ethics?" And unless they come from a US, like, academic research background, most UX researchers don't know because nobody teaches this stuff. So I'm going to teach you some today.

The Belmont Report, 1974, was created because of an infamous study called the Tuskegee Syphilis Experiment, which involved recruiting Black Americans to study syphilis and test out different treatments on them. It was a cruel study, and it's the reason why we have strict ethical principles for research today. The Belmont Report would eventually be replaced by the American Psychological Association's Ethical Principles of Psychological and Code of Conduct. Oh my goodness, oh my goodness, long name, I know.

And in summary, there are five golden rules: Respect for Persons, Beneficence, Justice, Fidelity, and Responsibility, Integrity. So let's talk about each one. Respect for Persons means we protect the autonomy of all people and treat them with courtesy and respect, and allowing informed consent. We must be truthful and debrief participants if we ever use deception. Beneficence means do no harm while maximizing benefits for the research project and minimizing risk to the participant. Justice means we ensure reasonable, non-exploitative, and well-considered research procedures that are administered fairly and equally, the fair distribution of costs and benefits to potential participants. Fidelity and Responsibility refers to establishing trust for the study and taking responsibility for any repercussions that may occur. And last but not least, Integrity. I've had a manager who used to omit bad data to make herself look good. Well, all that does is hurt the users in the end. I'm sorry. Oh, that's okay. I think I kind of had that coming. Don't be like her. Ensure honesty and accuracy with all your activities. It's not about you, it's about your stakeholders, it's about the users. Which again, I mentioned again and again, without users, we don't have a business. So we better be good to them. We must follow these guidelines. This isn't a choice, we have to. This maybe new to some of you, so keep this close, print this out, and frame it on your wall. That's how important this is. And I just hope more UX researchers will treat this as part of the core curriculum of being a [Music] researcher.

Impact. You've probably heard this term a bajillion times. Take a shot every time you hear the word "impact." Makes for a great UX research conference topic because it feels like every UX conference has at least one speaker talking about UX impact. So UX research teams produce a lot of insights, recommendations, and opportunities. Tracking how those insights and recommendations made an impact can be a bit challenging. To give you an example, not too long ago, I had some really good research for my team, but got switched to a different team to do a reorg, reorganization of the team. A few months later, my previous teammate messaged me and said, "Hey, that research that you did increased our metrics on all fronts, conversions, signups, etc." It was great news, but I wouldn't have known about it unless my teammate messaged me about it because I switched teams.

So in this module, we're going to talk about what impact is, what to measure, how to measure it, and the reasons why sometimes research doesn't always lead to impact and how to overcome those. So let's begin. What is impact? Impactful UX research, I believe, can be broken down into four main kinds. The first is product or strategic impact. The second is organizational impact. The third is impact on your UX research team. And the fourth, we mustn't forget, our own personal impact. When we understand who our users are, we can unlock clarity and opportunity to impact these four things.

Let's start with product or strategic impact. That means you can help your team or product area progress in measurable ways, which often times is the most straightforward to track. Right? You do research, make the product better, your user experience better, easier, or identify opportunities for the product to take a certain direction. The next is organizational impact, being able to shift your organization from a solution-first mindset to a user-first mindset. It's hugely impactful because then we'll be focusing on what really matters the most. And when your immediate team sees the value of being user-first, it'll naturally trickle up to leadership, and then you'll have impacted leadership too.

Now, the third kind of UX research impact, or the impact that you have on your team, that's more specific to the user research practice at your company. This is where you, as a researcher, a badass UX leader, make your research team a better functional unit. Are you establishing processes or functions that make things more efficient for your team? Are you growing your team's skills and abilities? Are you literally growing the team and hiring more people? All of these things will naturally trickle into the other two categories of impact.

And finally, personal impact. Don't forget, you are a badass UX leader. Come on, you're the badass. You know, keep track of those things where people mention your name, uh, mention how many times they reference your research. What I like to remind my mentees is always to take screenshots of any positive comments that were made about your research so that you can use them for later. In the next lesson, we'll go over six steps to measuring UX success.

Last fall, it has to work. And if it doesn't, that's fine. Impact. What exactly does tracking impact mean? People talk about it like it's some advanced thing, but it's really not that hard. Uh, tracking impact is basically knowing how your product or service, your process, your team, how anything is doing in relation to before. Are metrics getting better or worse? Staying the same? Are the intangibles getting better or worse? After all, as Einstein said, "Everything is relative."

So, step one to measure UX impact is, well, decide what we want to measure and attach those metrics back to the objectives and outcomes. Remember when I said about measuring things? Measurements are inputs and a way to minimize risk. And if we only focus on minimizing risk, we'll never innovate. So the point is, you don't have to track every single thing. Measure what matters to your objectives and desired outcomes. But that's not the same. Metrics won't come in handy later because goals might change. But the point is, don't do it for the sake of doing it. This is exactly why so many product teams fail. They focus on the wrong numbers when they should be focusing on the core user needs. Another point is, I like to make data is just data. Uh, it doesn't mean anything until you interpret it. And it doesn't mean anything unless you interpret it. Just something that matters to you. Don't, it doesn't mean anything to me.

All right, step two. Once you've decided it would make sense to measure, then we got to start planning our study. Step two is basically plan your study. And by now, we know to step by heart. What are the goals? What are research questions? How can we answer those questions? And then plan out what study methods we'll use to find the answers. That's step two. Pretty easy, just plan out your study.

Step three, believe it or not, is just conducting the study. Way we conduct the study for the very first time. This is known as our baseline study. You might also hear a term called benchmark. Nothing fancy. It's not some special type of research. Benchmarking literally is just using our first study as a point of reference to compare future data to. That's literally what the word benchmark means. When Einstein said that "Everything's relative," this is what he meant.

So what if you got a System Usability Score of 65? You know, who cares? You don't know what that means unless you keep doing it. Just keep doing it so you have something to compare to, right? Uh, it's just like any kind of goal in life. If I wanted to, you know, pull, pull up my body weight, I have to first measure how much I weigh. And can I do it? No. Maybe I start with some resistance band to help support me. And then when I get better and better, and I get stronger and stronger, then I could do more and more weight. That's the only way you can track progress. So it's just tracking progress there. The only one way to find out.

And this is where step four comes in: comparing and measuring against our new experiences. The more you do it, the more opportunities to improve your experience. So this is a continuous, iterative process. This is why race car drivers have lap timers so they can see exactly how fast they are compared to their previous laps. In fact, we could even get more granular with the metrics. Next, right? Track out further. Not only would we have times for the entire lap race cars, we could even see specific metrics for specific turns, like, "Oh, on turn four, you were 6.7 seconds slower because you took the turn, you know, 2 feet wider than you should have than the previous lap." You know, something like that.

Uh, step five to measuring UX success is, finally, as you've guessed it, we need to visualize our data and just tell the story. Show the before and after metrics visually because everything is relative. Without our benchmark, aka the baseline data, it's hard to understand how far we've come and where the experience stacks up. Sometimes it's hard to understand. However, one thing I learned from race car driving is that numbers don't tell a whole story. Now, sometimes I get input from more experienced friends who would give me suggestions. Sometimes it's the feel of the car, the confidence I have in the, the tires' grip around the corner, the sound of the tires starting to screech is an indicator to me that the car is losing grip. But these things aren't numbers. Which is why qualitative inputs are just as important as the quantitative ones.

And that's it. Five steps to tracking UX metrics. And as a reminder, continuously track your data because research impact never ends. In fact, this whole lesson, for those of you hawkey and badass leaders, much of it was just following the general research process we learned from module four. The only difference is that you're using the first study as a comparison point.

So now that we've learned how to measure UX success, what exactly can we track? And what does it look like to achieve impact? So in the next lessons, we'll talk about the types of impact that we can track as user researchers. When it comes to talking about impact, we usually think of numbers. We increase conversion by 300%, decrease bounce rates by 87%, reduce project completion time by 15%, decrease the number of shits I give by 99%. Four kinds of research impact you can have internally at a company. Uh, and this chart is a compilation of my experiences having worked as a user researcher to more than eight companies for 10 plus years during more than a thousand projects, being 20% more office snacks than I should have.

Impact on the actual product or strategy. We can have impact on your organization or team culture. We can have impact on our research team, our immediate research team. And finally, as Maya Angelou said, "You are worthy of your love." So don't forget about your own personal impact too.

In this video, let's talk about the impact we can have on product and strategy. The first thing is hard metrics, of course. Things like revenue, churn, engagement, sign-up conversion, user satisfaction, you know, the typical business KPIs. Of course, you can't really track those unless your research actually made changes to certain elements of your product or service. So track those UI changes that were based on your research recommendations or strategic changes that you made. Did your research change the design of any buttons or navigation? Better flows? Better copy? You know, these are kind of like the, the interim impact. The numbers will come later. It takes time after implementing design changes. And that's another thing we have to be aware of because if you make a change to a product, you won't see the numbers change immediately. You can't see them change overnight. You got to give it some time. How much time? Well, that depends on your product, how many users you have, and how often you're getting that feedback cycle. What are your real users saying in the real world? Getting feedback from real users will be a sign of impact, especially the good ones. And if not, hey, negative comments are also good. You get a chance to improve it. You know what people are complaining about. So negative is good.

Um, something else that's really important to track, and this is more strategic in nature, is whether your research prioritized or deprioritized something. What you could do is inspire new initiatives, spotting problems, not just solving them, right? It goes to huge impact later when research from one study informs multiple initiatives or efforts, right? You're not just doing siloed research. Research can have an impact on the rest of your company too. And finally, I kind of coined this word at my company's before with "evergreen insights" that can be evergreen, like something that lasts forever. Of course, all insights change because users evolve or people evolve over time. So your insights might not change, but there are some things that can be referenced for months, even years down the line. So that's huge impact.

A lot of newer user researchers ask me, "How to show impact if we don't have any hard numbers?" Well, you're in luck because, as you saw, not all UX impact is about numbers. We can have impact on the team, we can have impact on the road map, the strategy, our organization. In fact, so let's talk about that next.

Have you ever wondered what kind of impact you can put on your case studies besides hard numbers? Well, wonder no more because I'll give you yet another list of research impact that you can track. And this time, it's about impacting our team, our organization, or the culture on the whole. These are some of the impact that I have seen and that have promoted in my own work, and also I got some feedback from my students and mentees and people I've met over the years. Um, let's get started in order from small to huge impact. And this is for your team, your organization, your culture. Is when more research is requested. And when stakeholders ask for studies similar to previous research, or request even more research, more, more, more, more. Especially, it's really impactful when you're working with people who didn't really believe in research before, or they never really worked with UX research before. They're basically new to user research. Once you worked with them, they start seeing the value. I mean, that's huge impact on its own, but that's not even the hugest impact we're going to get. Bigger than that, right? You can work with them, and the key word is "work with." Once you build those partnerships with stakeholders and they care to listen to you, that you're becoming that go-to expert that they can seek out to, you know, actually listen to your user insight. Once you do more research, hopefully, it's just not, "Hey, we built this product, and can you do this research for us to validate?" Hopefully, we turn a reactive culture into a proactive research culture. And that's getting to a larger impact because now you're not just doing research, you're inserting yourself, or research has been inserted into the fabric of how things are done at the company. And being proactive means they're not going to come to you at the very end. They're going to come to you early. Then we get to dissemination of information, right? Where research is actually shared, praised, and utilized across communication channels. And I think the ultimate impact is when your stakeholders start talking about your user insights, and they rely on insights rather than guessing or solutioning, and they understand what the process requires, and they, they start sharing things for you, and they become an evangelist for search for you. And I think that's the ultimate huge impact. So I hope that gives you a few ideas of things to track that aren't necessarily just hard numbers and are equally, if not more important, in our line of work. And that's team and organizational impact. Some leadership impact because you might influence some people who are in leadership positions.

Let's wrap this up. Let's do the next one, which is influence on our UX research team. In another edition of impact, it doesn't have to be just numbers. We're going to talk about making a better user research team. Badass skills. I think the first thing you can influence, uh, this is starting small to large again. You have a team with complementing skills. And this is going to be mixed methods. You have Quant people, you have Qual people, all doing the research together and kind of working together to triangulate these informations. Then you have collaboration with other research functions. So not just user research, but other research functions. Anyone who does research. So that could be data scientists, analysts, market researchers, product managers sometimes do it too. So it really depends on the maturity of your UX culture. But when you're starting to work with different research functions, that is, you're going to combine, you're going to be even more of a research powerhouse. Next is establishing any kind of research operations processes to kind of make your streamline your operations more efficiently. Ample research operations to support, you know, recruiting, participation, incentives, participant tracking. All those take time. And I worked in a company as a, you know, solo researcher before, and I had to recruit, I had to source participants, I had to come up with incentives and like work out vendors to do that. So it's a lot of times if you can build up a research operations, that's huge because then the researchers can spend more time doing research. Other impact that you can track and have influence on is if you come up with better processes for yourself, for your team. Share it. I maybe you've done things, uh, quicker this way. Maybe this way is more efficient. This way, share with your team. If everyone can have a higher efficiency, that's impact. And and measure that too, if you can, right? That could be processes, how you come up with different documentations for things, how you ask the right questions. Do you have a template for stuff? You know, come up with templates. And it's exactly what I shared in module four. A lot of the things that you can kind of pull out your toolkit here are the pros and cons of research methods, just make this faster. Here's how to recruit participants. Here's our templates for outreach. Here are incentives, vendors, and whatnot. Every little thing that makes your life easier, you know, systemize it. Systemize it. You don't do it step by step every time from the ground up. Now you have a system. It's like, "All right, if we need to do this, these are the steps that we need to take." And here's a checklist too. And here's all the resources that is tied to this process. Makes it way easier. And I think this is more of a philosophical, but it research is a guiding light, not just for your organization, your immediate team, but they serve as a guide and light function. Every team has user research capabilities. That'll be even better because now you're not just, you know, who do I have to support? It's every team starts with research, and every team has a researcher. Now, being great, that's a great way to be scaled. It's not normally the case, but that's kind of an idealized state. And if you come up with any more that I've missed here, please let me know. I'm constantly updating this chart. Plus, that empty box at the bottom bothers me. I've been able to present my research directly to the CEOs of StubHub and Upwork before, which is pretty cool. And the CEO gave me a shout-out once, and I screen, I screenshotted that. So quick. And this is more personal impact to track. Hey, if anyone gives you a shout-out, recognition, kudos, praise, feedback on how your research project impacted a project, hey, screenshot that and make sure you keep it for performance reviews later on. I'll definitely talk more about this type of impact on the career module. Make sure you give yourself some love too. I love myself. You're giving research process, you're giving the culture, you're giving the product some love. Give yourself some love too. That means if anyone else is giving you love, you got to screenshot that. Count them with the number of references to your research insights, your presentations, and decks. And finally, you know, who and how many people attend your session, your sprints, your presentations. Like, how scalable are you making your influence on research and the team and the product and insights? Okay. And how often? This is more of a minor thing. I've seen charts similar to this, but they only mentioned the first three. They don't really mention the personal impact. So I hope I gave you some ideas on what to track because these aren't always as obvious to people that they are impact. Not hard numbers all the time, because that's not the only impact you can have. As I, hoping that you now can see. So thanks for watching, badass. You later. And get out there and make some impact. But come on back later because I'll talk about why research doesn't always lead to impact. You want to stay tuned for that one. Ah, wow, pretty cool.

Well, when you think of the word leader, what do you think of? Who do you think of? What comes to mind? What do you think the characteristics are of a leader? A mentee once said to me, "I feel like the field of UX research is for leaders, but I don't think I'm meant to be a leader." And that was painful to hear because I knew exactly how she felt for a very long time. Creating my YouTube channel was difficult because I didn't consider myself a YouTuber. When I rewatched my recordings to edit, I thought I was too boring, spoke too slowly, no one would want to hear what I had to say, and I was scared of being judged. In fact, I recorded my first video a year before I ever uploaded it to YouTube because I was scared. I hated how I sounded on video, and I thought only leaders in the field should make content. Besides, who was I to make videos about UX research? But when I thought about all the people asking me about how to get into the UX research field, I knew I could help, and at scale. At the same time, I was taking a UX leadership class at my Master's program, and it inspired me to think what the word leadership really meant. I had this notion that leaders were masters of public speaking, loud, articulate people wearing suits who are able to direct masses of people. And that's what I thought of when I heard the word leader, and I could have never put myself in those ranks. And I knew that's exactly how my mentee felt. Not everyone wants to be a leader, and that is okay. But when people think they might not be good enough to be leaders, that humility just may be the exact reason why they might end up being the best ones. You don't become a leader from taking a workshop, finishing a class, or reading an article on Forbes, or wearing a suit. To be a leader, we must know what it's like to follow. And since everyone has followed, anyone can be a leader.

Lao Tzu said that "A leader is best when people barely know that they exist. When their work is done and their aim is fulfilled, they will say, 'We did it ourselves.'" We often look to leaders, but without followers, there are no leaders. According to Forbes, leaders exhibit different actions like making hard choices, self-sacrifice, pushing people to be their best, helping them grow, and serve a great cause. According to Inc., leadership is a process that maximizes the efforts of others towards a greater good. There are no direct reports, no power or authority, no mention of personality traits, attributes, or titles, only a greater good. A few more I might add: being a leader does not mean you are better than others. A leader does not have all the answers, and they don't pretend to. They actively ask questions to mobilize others to achieve a superordinate goal. And in doing so, exhibit vulnerability where others can feel the psychological safety to do the same. Vulnerability in that maybe they've never done this before, they may have fear or of the unknown, but in spite of it, they stand for their values and don't let that fear stop them. There are people who have become leaders not for their titles, but for the movements they have inspired. They are leaders who have certain values and don't tell people what they want to hear. They're ordinary people like me and you, ordinary people doing extraordinary things, which I know you will.

The purpose of this module isn't to give a single definition to leadership. No, it is to show you us that there are different kinds of leaders, and therefore no all-encompassing qualities. The untold opportunities for the making of a badass UX leader beckon. Across history, we've learned to perceive leadership in certain ways, and I want to introduce you to three theories of leadership. Starting with what occurs in the natural world: the dominance hierarchy. This leadership theory is typically seen when members of animal social groups create a ranking system. Members either compete for resources and mating opportunities, uh, they're born into power like aristocrats, or fight they become the leader. This type of leadership system is based solely on power and authority. Machiavellian approach to ruling with fear over ruling with love. And different groups can be on the spectrum. Would I rather be feared or loved? Um, easy. Both. I want people to be afraid of how much they love me.

The next theory, it comes from Thomas Carlyle from the 1840s, who suggested that great leaders are born, not made. That leadership qualities are innate. You either have it or you don't. This was called the Great Man Theory, and he suggested that these leaders have intrinsic characteristics like charisma, intelligence, sociability, and confidence. Nowadays, we know his theory to be scant, inadequate, and quite frankly, damaging. This is the theory, believe or not, that I believed in, that made me and my mentee think that we are not leaders because I didn't have that charisma, or so I thought. When I saw people in powerful positions, that's what I thought leadership was, and could not ever imagine myself as a leader. The Great Man Theory ever is a very limited construct, built upon the dominance hierarchy of power based on fear. Of course, if Carlyle was wrong about the Great Man Theory, then where does that leave us?

The evolution of leadership theories brings us to transformational leadership, which basically says anybody could be a leader. That there's no higher or lower rank. And this style is idealized influence, where leaders walk the walk and not just talk the talk. The transformational leader considers the needs of followers and helps others to succeed beyond their own interests. When it comes to the range of efficiency and engagement, transformational leadership takes the most engagement and may be very efficient at moving teams along. Now, on the other hand of the spectrum, we have laissez-faire leadership, which is basically letting people do what they want. Now, you might be wondering, why is that a leadership style? Imagine a group of people who are way more experienced than you. You probably want to hear from them and take their suggestions rather than tell them what to do. In the middle of that spectrum, we have transactional leadership, which, you know, when it comes down to it, isn't really leadership more than it is management. Here, in transactional leadership, we're keeping things the same. Don't do your work, get punished. If you do your work, get rewarded. Simple as that. And understanding these theories in depth gives us a healthier look at what leadership means and gives us a 360-degree approach to becoming a badass UX leader, which I emphasize so greatly.

1995, the year Post Malone was born, and O.J. Simpson found not guilty. And the same year that Daniel Goleman published his New York Times bestseller, Emotional Intelligence, which stayed on the bestseller list for over a year. In the book, he talks about six different leadership styles. And in this module, I want to go through them because it's important to recognize, again, that this job isn't just about understanding our users through research. It's as much about understanding our teammates and stakeholders. And when we recognize the type of working styles there are, we can more easily communicate across divides that can help create trust. Who knows, perhaps this will even help us understand ourselves better too. So the six styles are: Affiliative, Visionary, Coaching, Democratic, Coercive/Commanding, and Pace-Setting. I'll quickly say that the first four styles I mentioned are good defaults. The last two styles are bad, I mean, um, good for certain situations. As I describe each leadership style, you know, think about the times you've seen or used them in the past. I want to be clear, though. Everybody has a dominant style, but the best leaders know when and how to switch up their styles in different situations. You know, they know how and when, but also they do it. I know how and when, but it could still be very difficult for me to take on a different style. That's usually not me. For example, according to the leadership style quiz, my dominant style is Visionary, then Democratic, then Affiliative, then Coaching, then Pace-Setting, and then the Commanding style. These styles do not define you, but rather think of them as, you know, another tool in your UX leadership toolbox. Because as I got into teaching and mentoring students, I've learned to utilize different styles at different times. I had to. So let's learn about each, the pros and cons of each, and when you might want to channel a certain style. And if you want to find out which is your dominant style, head to this website to take the quick quiz: idealist.org.

Dale Carnegie, author of one of my favorite books, "How to Win Friends and Influence People," once said that "If you want people to be enthusiastic about your ideas, make it about them, not about you." A great quote that embodies what our work as badass UXers is all about. As we go through the next section on influence, stop and think about stories you can relate to what you've encountered before. How and when you can apply these learnings. Fostering strong human relationships only works if you actively apply what you learn in real life. You won't get anywhere just reading things.

So why learn about influence in a class about UX research? Although if you haven't already survived much of our job is to influence design through means of storytelling and building empathy and compassion for users. Whether you're in a mature UX company or not, we still need to influence our stakeholders on what is best for our users, and maybe even what's best for your user research team. There are a couple of great books on how to build relationships practically: "Emotional Intelligence" by Daniel Goleman, which is where we learn about the six leadership styles. This one right here, "How to Win Friends and Influence People" by Dale Carnegie. "Real Influence" by Mark Gerson and John Olman. "Exercising Influence" by Kim Barnes. Yeah, there's other great books related to UX specifically, like Tom Greever's book, "Articulating Design Decisions," which is awesome. But if you understand the fundamentals of influence, you know, you might not need all the other books.

Now, before we talk about influence, we keep bringing up this term "empathy." What the heck is it? Let's learn about empathy more and see what exactly has to do with building influence. So please watch a brief video about empathy by Brené Brown, a professor focused on leadership, shame, vulnerability, and empathy. Author of "Dare to Lead," "Atlas of the Heart," lots of bestsellers. Okay, that I recommend, especially "Dare to Lead." Another book I recommend on leadership besides this class, teaching a cohort, working consulting on UX research projects, dabbling of cars, helping ladies cross the road, and in general hooliganism. I also do mock interviews for UX researchers. And one time, I did a mock interview with a, a mentee, and I put all my love and effort into helping them because I really wanted them to succeed. Gave them templates, questions, co-shir through questions and how to answer them. When she didn't get the job, she called me, and she was crying. I really didn't know what to do, and I felt terrible. It reminded me of when I was going through the transition into UX research. So I knew exactly how she felt, and I told her my story. I'm not sure if it helped, but maybe that's all I can do sometimes. I checked in with her a few days later to see how else I can help because I, I want, I still want people to crush her interviews. You know, one doesn't work out, hey, it's okay. You got interviews, that's great. You can get another one. So I offered a quick chat with her again. Without realizing it at the time, the story I just shared, I guess exhibited three types of empathy described by Daniel Goleman in the book "Emotional Intelligence." There is three types: cognitive empathy, emotional empathy, and compassion. So cognitive empathy is knowing how others feel and think. Emotional empathy is feeling along with the other person. And compassion is one step beyond empathy. In fact, when we talk about empathy in UX, we should really be talking about compassion. When you take the action to help, that's compassion. So not to turn the UX world upside down, anything, but you know, yeah, we're really talking about compassion, or we're talking about empathy. Empathy is vulnerability transcended by connecting with others while being able to connect with something inside that you can relate to. In UX and design, it translates to being able to know what it's like to be your users, not just usability-wise, but their context, their fears, and motivations too. In leadership, empathy is knowing what motivates or scares your colleagues so that we can garner growth. Empathy is the underpinning to influential leadership, which we'll talk about right now.

Influence is to connect with others to build strong relationships. It's not taking selfies or Instagram photos of food, right? The social media influencer influence. It's not, "How can I get them to do this for me?" And it's certainly not, "Do this or else, right?" Influence focuses on the other person, or as Gerson and Olman puts it, "their there, they're there, the other person's there, the other person's mind, presence, what is their perspective?" Depending on how you go about it, it's either persuasion, manipulation, or coercion. But what the heck is the difference? Intent. But why should we care? Well, remember how I say everything in life comes down to semantics and first principles, basically? Okay, not everything in life, but a lot of what we do. Once we define these, we can be more aware of our actions in the workplace and in your own home. So let's start with the bad ones. Manipulation. Oh, it's already bad. It's getting control for short-term gains, typically for yourself, right? Selfish gains. Manipulation, one side wins, one side loses, and there's no cooperation. Okay, that's manipulation. Coercion. I, I mean, that's this is the worst kind. It's "do it or else." It's forcing or threatening somebody against their will to do something that you want them to do. It's also a crime. So, uh, I know none of you will do such a thing. So that leaves us with persuasion and influence. What's the difference? We know manipulation and coercion are bad. Persuasion, influence. It's interesting because I had a hard time distinguishing the two before. But I attended a talk once, I forgot who the speaker was, but they were a leadership expert, and I asked them what they thought the difference was between influence and persuasion. And they gave what I thought was a pretty good response. He said, "Persuasion is something you do."

Influence is something you earn. I like that a lot.

Persuasion is convincing people to take action, even if they might not agree with you. But still, they have to eventually agree. Persuasion is disconnected influence, and that's what most Business Schools and workshops teach. Persuasion makes the other person into an influencing target, right? It's a target for influence. Persuasion is about getting people to do something your way. Persuasion works for the short term, and sometimes is what we do to earn influence later.

So, if you want a high level of commitment and long-term motivation to your idea, then involve people in the creation of your idea, as we've been learning this whole class, right? Influence, on the other hand, influence takes time. By cooperating with others, understanding their point of views, long term, it's a win-win. You're not forcing anybody to align with your objectives and vision. Influence seeks to make authentic connections, allows for dialogue and collaboration to build ideas together, and both parties commit willingly.

So, how do we gain influence? Let's go over some brief ways in the next lesson. Influence is really just about building relationships. So, what are some steps to gain influence? Well, before I jump in, a number of things to remember on your influencing journey. There's no step-by-step, one-size-fits-all process. It's nearly impossible to influence everybody, otherwise we wouldn't have wars and a lot of disagreements. You may be right, or only partially right. Be honest with your intentions and expect some resistance. Not everything's going to go smoothly as you try to build relationships, especially with tougher teammates or stakeholders.

And the books I recommend, the ones that I did recommend, "Real Influence" are Mark G. Stone, John Olman, "Exercising Influence" by Kim Barnes. They define different ways to influence. I'm not going to go into their technical frameworks or snazzy definitions because once you break down all of their points to first principles, it really comes down to two things. First is showing compassion with active listening and providing possibilities and ideas and working together with the other person. In fact, you know what? My mom had always taught me. She said, "Just be respectful, humble, listen first, don't offend people, don't brag, don't be the first to talk. People will naturally like you, or at the very least, won't give you any trouble." And when you get that respect, then people are more likely to listen to you. So, thanks, Mom.

Having an open mind is easier said than done, though. But it is important to know that you may not be 100% right all the time. Influential people inspire us to see beyond where we are now to where we could be. They understand where we're at now, where we want to go, and inspire us towards possibility, maybe even to places we couldn't even imagine ourselves. It's really a blend of all of the leadership styles we talked about.

But before you try and go out to convince everyone to join your, you know, Hell Kitty Fan Club, remember to expect resistance and keep in mind some common tendencies of human beings. People don't like others telling them what to do without asking. A lot of times, people may simply resist and do the things it was before because it's easier to just continue doing the things we've always done. People may not commit to any changes based on principle, just to resist it. If the change is imposed directly, it could even create resentment, even if the change was for the better. If you don't work with people to explain your rationale, people won't understand why you're changing things.

So, here's some practical advice. If you want to suggest changes to anything, start with an affiliative leadership style by asking non-judgmental questions to understand people's perspectives, which is our typical approach to UX research. So, why not apply that to anybody you know? We can ask questions like, "When do you think of it now? How do you decide to do this behavior? Have you always had it like this? Why or why not? What would you like your ideal situation to be? Why? Where do you get your inspiration for, etc.?" You know, affiliative leadership, understand the other person. Then, you might want to take a democratic leadership approach to offer relevant suggestions and also get their opinions. "I think XYZ approach, but what do you think about doing this? What are your thoughts, etc.?" Giving people a "why" usually helps them get closer to where you're coming from. Can you see the resemblance of good leaders to UX research? Like I said, our job inherently requires leadership qualities. So, apply the same research approach that you do with participants with your team. Cool.

Welcome to this lesson. And this lesson, we're going to go through a thinking exercise. I saw a post in a forum about a person researcher who had done a roadmap, a research roadmap workshop with their stakeholder team. This is the prompt. Okay, we give you the prompt. The person runs a roadmap workshop with their team. You know, in the workshop, it goes really well. You define the timelines, prioritize projects. Only come to find a few weeks later, your stakeholders had given it to a different team. All these projects, no longer user research, but now it's marketing or data analytics. Someone else that basically they said, "These research questions are for other teams, not for user research." I mean, first of all, how would you feel if someone did that? You did all the work, planned the research, here the research questions, they all came together with, set the timelines, prioritized everything. You had the impression that it was going to be user research. Who goes and deals this product? But the stakeholders gives it to other teams. Take it. How would you feel? And what would you do? That's a question. So, pause the video, think about it. And when we come back, I'll share what other people have said, what I would recommend. And then this is a thinking exercise in case you ever encounter it. Hopefully, you don't, but it'd be helpful to kind of think about these things on influence, persuasion, conflict resolution. All right, welcome back. Hope you got some time to think about how to approach this situation. It's a little difficult, I would say. But I'll share some responses from what I see on the line. Some people, there's two camps, it seems. There's some people who say, "Just do the research anyway. If you find value in it, they'll they'll come to you. Oh, they'll realize what how valuable it is. Damn you, I will not be ignored. Get back in here, get back in here, and love me." On the other camp, there's people who say, "Well, I wouldn't do research at all if we had no alignment." And there's another camp of people who like, "Well, why did they give it away to a different team? Why didn't they raise this concern during the workshop or in any prompts to that? And was the initial impression that user research was going to do this, but then something changed where they gave it a a different team, and you just weren't looped in?" So, many different approaches. So, I'm curious to hear what you thought of. Here's what I personally might do. And there's some really good responses here too. When some people say, "Well, and and like I mentioned in module three, you you don't ask them research questions. You ask them, hey, what are your plans for the next year? What are your product plans? What's your vision? What are your risks and benefits and goals? And what things do you need to do in the next six months or three months or nine months or 12 months? What decisions are you going to have to make? Do you anticipate anywhere you might pivot or you anticipate something changing that a lot of kind of just projection into the future and premortem, so to speak?" Definitely that's the understanding bit. When it comes to re-influencing your stakeholders, understand what was the state of mind when research wasn't involved in their decision here. And you got to do it. It's something you can't avoid the action of doing research regardless of their input. I personally wouldn't just because, like I mentioned before, if you do research, no one's involved, no one knows what you're doing, you come back, and no one really had input to begin with. They're going to be like, "Well, where are these questions? Where are these questions?" So, these things change. What happened to that? A tree falls into forest, do you hear it? Right? Exactly that kind of case. If you did research and no one does takes action on your recommendations, it didn't happen. Didn't happen.

M is really crucial for any project. It's not like you can go rogue or vigilante researcher and hope to do a great research project and say, "They'll see the value." They're probably super busy. Of course, sometimes it might help, but get the buy-in. Involving the stakeholders is an inherent part of what we do. And finally, another approach you could take is, okay, well, since they moved it over to a different team, let me just see if that team needs help or my expertise. Need some help. Maybe that other team didn't even know they were supposed to do this. But you do, 'cause you planned all this and prioritized with the team all alone. So you can go in there and say, approach it in a friendly manner. It's like, "Hey, you know, I know you have this research project. I can help. How can I help you?" And that could be another approach. So, hope that was a good exercise in influencing the research and make sure it goes in the right direction. It can be difficult if you encounter people's decisions that leave you out of the picture. And sometimes maybe it opens up avenues for yourself or research opportunities, as long as the research has been put in appropriate hands, so to speak. You want the research because you planned the workshop, or is the research definitely a research project that you need to take on? I could imagine some sense of ownership, like, "I did the research prioritization roadmap workshop. Like, these are all supposed to be my projects." And but say, go with a collaborative approach and say, "Hey, if you think it makes sense to go there, let me see how I can help. Or here's why I don't think it makes sense." But let the ego go. My ego, your poor, spited ego. There's no right answer here. There's no one approach that you could take. And that's the point. I want this is a difficult situation, and I want you to you think about how you might approach it. How if any of these ideas inspired a different approach for yourself. Thanks for watching again. The research isn't the hard part. People are. And which is exactly what the next video is going to be about. So, I'll see you there.

The hardest thing about user research isn't the research, it's people. Like any job, we can probably do the thing we're hired to do. But people are so f***ing difficult. Today, we'll delve into the art of working with stakeholders who might not be as cooperative with research. Remember, we want to create a harmonious and effective working relationship that benefits not just the research team, not just stakeholders, but our end-users, who upon their needs being met, ultimately will have this symbiotic relationship and thus help the company as well. But life isn't always so fine and dandy. What happens if you run into pushback from your own teammates? So, let's explore some effective strategies together.

First, let's talk about mindset. When faced with challenging situations, remember, research is easy, people are hard. Don't take any resistance personally. Instead, focus on the common superordinate goal we share with our stakeholders. We need to understand their working styles and preferences. Employ persuasive techniques that align with their needs. Building influence and trust takes time. So, you know, keep a record of your attempts and maintain a positive, collaborative approach.

The second thing I need to talk about is due to the long game. Maybe not today, maybe not tomorrow. Like I mentioned, building influence and trust takes time. That's exactly what we might need. Not every alignment is going to happen immediately. What matters isn't aligning people with our point of view, but aligning on something together. Start by understanding the "whys" behind any resistance and pushback. And then implement those tactics that can help you achieve your long-term goals. Play the long game, right? It might not happen today. You need to build up relationships, and that's, take that takes time. Ask some questions, understand where they're coming from, have them understanding where you're coming from, and then find a solution to move forward together.

The third tip for working with people who might not be as cooperative is them. Maybe it isn't you either. Maybe this is just a misunderstanding and clarity and understanding. Misalignment happens because of differences an understanding of what's needed. Sometimes it's the words we're using. Maybe we're assuming the way they're using a certain word is a way that is something else compared to what word referring to. People have a different definition or perception of something, and that's why I always start by asking how people are defining things to to get to the root. So, start by clarifying how different stakeholders perceive key concepts to ensure everyone's on the same page. Everyone's on the same page, then effective discourse can happen. You know, I had a a VP who said, "I don't want to get into semantics right now." And I'm like, "Well, if none of us understand what your definition of this term is, how can we even begin to understand your process and implementation of it?" So, you got to break that down.

First, step four. You know, here's some like communication strategy. We'll have a whole module on this next, but the importance of thoughtful communication cannot be emphasized enough. We have to choose our words carefully and steer clear of, you know, phrases that could be perceived as dismissive or belittling, especially during times of disagreement. So, the fourth tip I has for you is to be aware when and what language we're using, depending on the level of trust. I say, depending on the level of trust, because there's two ways things can go. Take for example, these phrases: "Well, from a research perspective." "Well, from a design perspective." "Well, from a insert here perspective." They seem innocuous, right? And I've seen these exact phrases used in two different kinds of environments. One where the team had great trust and open-mindedness, and the other where the team didn't have a high level of trust at all. Okay? So, do you see? It doesn't matter what you say, it depends on how people might be perceiving it or past history. These may be great to use when the team is vibing, everybody's having a good time, they're having an open-minded discussion, and they want to hear different perspectives, right? "From a research perspective, I want to hear from the research side what you have to say." "Oh, I want to hear from the market research side what you have to say about." On the other hand, when relationships aren't at an all-time high, I see people completely go off because maybe they're perceived as dismissive. Instead, right? Instead of collaboration, it's now defensiveness. It's like, "Oh, well, from a research perspective, dismissive." So, how fickle human relationships are, right? Remember, it's not about our point of view, it's not about their point of view. It's about, you know, at the end of the day, trying to do something for the users, ultimately leading to a better business. It could be hard when there's no trust. Everything you say, even if you have the best intentions, could be perceived not good. That's where we might employ the affiliative leadership style.

In conclusion, here are the few key principles to foster effective collaboration. Don't be confrontational. Carefront. Focus on the overall goals, not somebody's personal goals, including yourself. And if your goals don't align, you got to go back and understand where you're coming from, where are they coming from, why aren't we aligning? What is it that's forming the rift between us? And then start from there. And also, the language of true leaders is honest, open, and compassionate communication.

HRDQ is a research-based training program that provides training and they call it soft skills. I just like to call it human skills. And I want to introduce you the four different communication styles because it's useful to one, improve our interactions with others. Two, recognize what kind of styles you might encounter your arch nemesis. Three, and perhaps more importantly, it's a way of understanding ourselves too. What makes us tick? What styles do we vibe with? And what styles do we butt heads with? All of this can help influence our success and relationships in a positive way. So, let's get into it.

Here I'm showing the four styles in two dimensions: low to high assertiveness and low to high expressiveness. We have direct, spirited, considerate, and systematic. Let's walk through each one. The direct style tells it like it is. There's no mincing words. They're highly conscious of time, mission-oriented, results-driven, and willing to confront issues head-on. Spirited style tends to be enthusiastic, intuitive, very friendly, and bubbly. So, therefore, great building strong relationships and is an expressive, persuasive, motivating, and inspiring style. The concerted style uses a soft, warm tone of voice. They have good listening skills. They're good team players and managers who use a caring, empathetic, and appreciative style. Last but not least, we have systematic. This is thorough, precise, facts-driven, organized, and diplomatic. So, they typically won't get on everyone's bad side and mind their own business. Sounds great. I bet you could probably pinpoint which is your dominant style. But just as we have positives with each child, we have negatives as well. Or rather, instead of negative, let's call it blind spots. Direct communicators tend to be poor listeners, likes control and competition, may be impatient, discounts feelings, doesn't mind arguing. The spirited communication style focuses on a big picture, tends to forget the details. They tend to exaggerate and generalizations, maybe procrastinate, responds poorly to criticism. Considerate communication style avoids conflict, might give in easily, resists change. They emphasize feelings and tends to keep opinions to oneself. And finally, last but not least, we have the the blind spots for the systematic communicator, which is they focus too much on the fine details, and they care about logic over emotions, trust and risk-averse. Think of a friend. I bet you could probably pinpoint which style they have, your manager, your colleague, maybe yourself. But there's a lot more to these styles than just verbal communication. Paravocal too, body language, and also personal space. And I'm also wondering if you could draw any parallels to the leadership styles we spoke about in the previous module. I'm not sure if there's any significance to this, but it seems to me that direct communicators might take on a commanding leadership style. Systematic communicators might be the pace setters. Spirited might be visionary leadership style, where considerate communicators might have a mix of affiliative, coaching, and democratic. What are your thoughts? Anyway, I'm always trying to draw patterns between things. Personally, my dominant communication style is considerate. But these style theories aren't meant to put a box around us. They're simply a starting point to help us understand how people might communicate. Think of them like proto-personas. In fact, I mentioned my dominant style is considerate, which means that I might take a different approach in different situations, just like those leadership styles we talked about. The reality is, most people are a mix of different styles and may switch up styles depending on context and situations.

Now, here's a question for you. And I'm sure you already sensed it. Which style do you feel is the opposite of yours? Who do you tend not to get along with? I'm willing to guess it's the style that's diagonal to yours according to that chart. People in diagonal quadrants tend to have polar opposite communication styles from each other. Direct and considerate butt heads, while spirited and systematic tend to butt heads. We can recognize the different styles that exist. Isn't that cool? This is the first step to being better communicators. Simply understanding others first.

First, holy moly, you made it! Four and a half hours of leveling up your UX research game. What a ride. Give yourselves a pat on the back. And thank you, thank you for watching. Virtual high five. Whether you jumped around the video, skipped sections, or rewatched sections, I hope you found something to take away, something that was helpful and gave you a sense of the the right mindset and some pro strategies to be that UX leader shaping the future of our field. But remember, this is just the beginning. Now that you've got a foundation, you can keep the momentum going. Like I said, this video was just a tiny fraction of my entire master class, which I spent five years on. So, if you thought there was a lot in here, you'll love my master class. It's my baby. Like I said, I quit my job and went monk mode to make it. That's how serious I am about UX research education. So, I invite you to check out my self-paced master class to dive even deeper, sharpen your skills. It's step-by-step, everything, everything. I have a preview on my website there for you to watch. At least go check it out, check out the curriculum, see what's in it. In it, you'll find extended lessons, real-world case studies, actionable exercises, role-playing exercises designed to take you from good to great. And you know me, right? I will ask you real-life scenarios. I want you to do it. You do it, or you're not moving to the next lesson. So, keep asking questions, stay curious. Most importantly, keep pushing yourself to be the kind of badass UX leader that sets the standard, not follows it. Thank you so much for watching. Mad love. Peace.

[Music]