Transcription
How do these pieces fit together so that you're pulling the right seven references instead of 12 redundant references? So if you're training a model for that, the actual training set you use is very different than what you would do if all you're trying to do is get the right answer. But it's also a lot more robust because then what you're trying to set up is what is the right paper at any given point in time given the day as things change as opposed to what is the right answer given a question that might change tomorrow, right? That answer might actually change. So those were the kind of retrieval questions I was very interested in but the start of it and then of course you also have to get the answer right. Right. So those are both equally important but having both of those as pillars I think is really critical to thinking about what goes on under the hood in open evidence.
>> Hi and welcome to another episode of any jam AI grand rounds. I'm Rajan Ry. I'm here with my co-host Andy Beam and we have a really exciting and and fun conversation to bring you today. This is with Travis Zach, Dr. Travis Zach, the CMO of Open Evidence. And so, if you're a physician or clinician practicing, there's a pretty good chance that you've already been using Open Evidence for some time. And this was a amazing opportunity for us to just really dig into the background of not only open evidence but how Travis thinks about clinical safety uh the challenges of sort of growing at this pace and this scale and to learn about his journey from his sort of early days to joining open evidence and now in a leading role as their CMO. Andy I thought Travis was incredibly thoughtful throughout the conversation. I really came back with a sort of new appreciation for not only him but the work that he's doing at Open Evidence after this.
>> Yeah. Really thoughtful. He was described to me before the episode by someone who I consider to be very smart to be one of the smartest people that this person knows. And I think he lived up to that billing. Amazing background, very non-traditional. I think a smart person who has done very very hard things and been successful in many different arenas. Perhaps none harder than making a very fast growing health tech company. Three or four years ago, having a company valued at 12 billion dollars that sells directly to clinicians, I would have told you that's impossible. But Travis has been a big part of that. And hearing how he thinks about both the durability of their remote, the product that they think they're building, which is different than I had thought about it before this conversation, and then what they're building in the future, I thought was super insightful. And again, very measured takes on everything. being very thoughtful as you said Raj and to your point probably of high interest to our listeners because as Travis told us half of clinicians in the United States currently use open evidence and so again anticipate this being a widely listened to episode. The NEGM AI Grand Rounds podcast is brought to you by Microsoft Vizai Lyric and Elevent Health. We thank them for their support. And with that, we bring you our conversation with Travis Zach on AI Grand Rounds. Well, Travis, thanks for coming on AI Grand Rounds. We're excited to have you today.
>> Yeah, I'm stoked. I can't wait.
>> Cool.
>> Travis, it's really great to have you on the podcast. So, this is a question that we always get started with. Could you tell us about the training procedure for your own neural network? How did you get interested in artificial intelligence and what data and experiences led you to where you are today?
>> Sure. I try not take up the full hour and a half, I suppose. I did my undergrad in condensed matter physics. I was planning on being a pretty basic science researcher. I went to bioysics for graduate school interested in single molecule imaging, the crystalallography, crym, scanning, tunneling, microscopy things. And then actually during my rotations in graduate school, my mom got breast cancer and I got interested in doing something a little more practical that might have some real world impact at some point in the next 50 years. To that extent I rotated in some oncology labs and got interested in algorithm development for genomics. So worked at the broad and got the bug for medicine and went back to medical school after that. I was just telling Andy that part of that was exploration. I had been doing statistical modeling work. Got interest in machine learning and how that could potentially interact with real world data. That is we have the genotype with the genome. I really wanted to understand the phenotype better and how to use machine learning for that. Ran into Andy and Zach and got excited about that. So decided to do research during medical school residency and fellowship in real world data and how that pairs with genomics. So finished in the HST program there at Harvard Medical School and then did my residency and fellowship here at UCSF. So that's my long journey to where I'm at in terms of training.
>> Travis, that's a great recitation of your CV, but let's go back further. Like that's one of the things we like to do in the opening is >> how did you get interested in biohysics as an undergrad? What were you like as a high schooler? Could you take us back a little bit further? What's your origin story?
>> Sure. Sure. I was kind of a non-traditional background. I grew up in Lahina, Hawaii and uh was actually homeschooled for a fair amount of my growing up. I, you know, moved to California and then to UC Berkeley and was very open. I had no idea what I wanted to do. I had a vague idea of what a unit was when I entered undergrad. I was coming in as a philosophy major and uh was excited about I still love philosophy. I think a lot of, you know, like big questions. I like big question stuff. And what actually happened during undergrad is I couldn't get into the philosophy classes I wanted. I had these breath requirements. I was in a dorm with a bunch of engineers and I had to take a physical science. I was like, I'll just take physics with you guys. And then they laughed at me. They're like, "Come on, you're a philosophy major. You're not going to take physics for scientists and engineers." And I was like, "Well, screw you. I'm going to take this class and I'm going to do great." And it was by far the hardest class I'd ever taken in my entire life. But I don't know, I like hard problems. So I was like, you know what, this is amazing. These are also big questions, kind of fundamentals of how things work. So end up going into physics. My parents were not science or engineers or physicians. In undergrad, I actually had no concept that physicians did research. It's just like did not, you know, I thought they fix your knee, they give you antibiotics for your cold. So it was definitely not on my radar to do anything. I wanted to do you know answer big questions. So I want to do research kind of the academic track. So got into bioysics. Uh, this is okay I'll tell you one more fun story. I applied to all physics programs and I had a undergraduate mentor uh in physics department who had seen that I had taken a bunch of biology classes and I had taken all those biology classes because they had great field trips. Uh, I was like you know I liked invertebrate zoolology. I liked the forestry class. We went to go hikes and stuff, went on boats and hikes and um, he's like, "Wow, you've taken a lot of biology. You should apply to one bioysics program." My I think he said his daughter I think he his daughter did biohysics and and she really liked it and that was John Clark. He won the Nobel Prize this year. Um, and so I applied to one uh bioysics program and then I I toured there. I was like, "Wow, this is amazing. This is exactly what I want to do." And then I had no other opt. So that's why I went and it's the program that when I was in high school, I was touring. I was in Boston in February. And I remember telling my mom, I'm like, I will never live here.
>> I was coming from Hawaii, right? I was just like, I will never live here.
>> The contrast between Boston and February and Hawaii, I'm sure, is stark. And so, yeah, makes total sense.
>> Yeah, it was like 17° blowing sideways with snow. And then I lived there for time, right? Um, but yeah, so I
>> Okay. Well, it was Harvard biohysics first because I did my PhD first and then went back to medical school.
>> Cool. Um, super cool. And I think maybe we're going to dig in, of course, your work with open evidence and what's been going on there and what what you see as the next couple of years. Uh, but maybe you can tell us about how you got involved with open evidence before we sort of dive into the current work. Like how did you meet the team? I always find these stories fascinating. Like it's easy to it's easy to sort of connect the dots backwards, right? But a lot of times people like, you know, you can't really predict exactly which way it's going to go. And so I I love these stories.
>> No, no, it's like Yeah. Yeah. Um, most lucky set of four things in a row to happen in my life. The long story there, I hope you you I don't know if we'll have time to talk any actual things. If I
>> This is This is great. This is great. when I was when I was transferring you know going from end of intern year to residency um in medicine you you like have this there are a lot of existential crises in medicine but there's this you know there's this increase in responsibility is multi-dimensional from intern to resident and of course you're responsible for more patients you have more ownership you're running a team but there's a whole other aspect which is you are now responsible for the education for your team right? You're supposed to come up with how to teach, how to teach your interns, how to teach your students while taking twice as many patients and being responsible for all of them and making sure they don't die. And I've always really loved education and and it's been part of my identity. So, I had been thinking about, okay, how am I going to teach? I hated the resident. I thought it was less than ideal for residents to just read the up-to-date page for the patient that they're seeing. It's like, I can read the up-to-date page, so I want to do something different. But I didn't want to have to do a bunch more studying. So instead, what I did is I was like, you know what, be great is if I created a retriever model for the CPC cases so that I could basically submit information for the patients on our floor and figure out which CPCs I can teach on because the pathophysiology in those CPCs are amazing, right? So if I can create that retriever model, that would be awesome. And so you know I and the person, you know, Gupri who is who is my PCP instructor, right? He was the one who was my I did clinic at the VA.
>> Amazing.
>> Correct. Yeah. Yeah. He he had this >> we we he do something similar for education but his was this file system he'd have.
>> So you built the digital version of your ret. Yeah. So he had like printed out all these and he's you know he's he's got his retrieval his internal retrieval model is just like a vice right. So like we'd be talking about a case. He's like, "Oh, let me go back to 2004 and pull this case." And then he'd talk about it from his file cabinet, right? So I was like, "Okay, I'm going to create that." And so long story short, this was early days like Burke kind of model stuff. So I was like, "Okay, I'll build a retriever for that. Then I will have cool cases to show my interns when it's time to teach." So through that, I got connected to the New England Journal to do that in a very basic way. Group and I published a paper on this kind of idea, you know, preGPT and so New England I think um uh yeah, Pete Solvitz,
>> who you guys know, just saw him yesterday, bumped into some graduate students. Yeah.
>> Yeah,
>> he's amazing. Um, he had a couple graduate students interested in a project that used the CPC cases and Jeff Drazzen connected them to me cuz they're like, "Well, we don't have the system to set you up that way, but there's this MD out there who's like very interested in that stuff doing it." And the first graduate student he sent me was Matt McDermott who I know you guys both know.
>> Um, and then Matt's understudy was Eric. So then I started working with Eric after Matt kind of moved on from Sulvit's lab uh on these kind of ideas of diagnostics, how to train, you know, how to train good ML models for these things using the CPC cases. Again, preGPT explosion stuff. And then Eric was part of the first few guys part of Open Evidence. And so I became like he would just as he was building the retriever for open evidence, he would ping me and be like, "You're an MD. What do you think about this answer?" And I'd be like, "This is terrible. I'll never use this. This is total crap." And then he'd be like, "Okay." And come back a week later. And he's like, "How about now?" I'll be like, "Yeah, it's still pretty much total crap." Uh, and and so then they kind of sucked me in more and more and I I kind of, you know, got more consulting. You just kind of uh getting involved in quality and product and stuff. I was basically like a Slack hanger on for a long time. That that's how that connection happened. Very long story. I'm sure you can edit
>> that. No, that I don't think we're going to edit that down because that's honestly uh it's amazing just to sort of hear. It's like there's no way you would have predicted that you would have ended up in open evidence from sort of starting with any GMC CPCs, right? Or starting with starting with what you were doing. I it was it was a total you know my my research is was kind of like real world data in oncology predictive models understanding toxicity all these things and then I had this total side project on medical education for retrieving from the literature that was total passion project did not think it would have anything to do with my academic career or anything it's like wow I get to work with group pre talking about the CPCs and AI like I have to do this and that kind of created this connection with with what became open
>> I have to just uh remark and I think Andy knows a lot about this like not a lot of this already but I just have to remark at the sort of like wild similarities that you and I have in some of our interests and some of our backgrounds. So I also studied physics as an undergrad. I then went to HST and then in HST I became obsessed with the CPC's and so in the CPC's it was like you know just the way the differential diagnosis would be formed how you would then get the correct answer and then how we just just have this amazing hundredyear history that is not just today but looking back and how diagnoses were approached and what diseases were sort of the final answers back in the 20s and 30s how they sort of blend into the background now right and into the presentation of case and so I I you know I got obsessed to the CPCs and then we worked on building the system Kabot named in homage to Richard Kat which I think Travis we've talked about a little bit before and we published a CPC using the that Kabot generated the AI diagnosis of and this was last fall and of course who did they select as the human diagnostician one of the best diagnosticians ever Gerri Daliwal who I didn't know that you actually knew also that you had worked with and then Gerri and I we started talking and then there was a nice back back and forth and I haven't actually even had time to tweet this out or anything but there was a nice back and forth and we got to work with Gapri and write up a response to our NGM CBC where someone was asking back on the topic of CPC retrieval why didn't Kabat pull this case from like 2019 was similar when it was approaching the case that we were solving that Garri was solving and we had this just sort of amazing back and forth and Gapri has talked about this concept of wayfinding or navigating to the correct diagnosis as opposed to just spitting it out and so we invoked that and we we essentially talked about like you know it got the it localized the lesion right it recommended the test and we just got to reflect on sort of what is useful AI for a clinician who's actually using this for a second opinion or even in other ways versus what are the ways in which we're currently benchmarking them or what do we tend to fixate on which is like final diagnostic accuracy and the CPC is actually this great data set or trove of reasoning right the sort of reasoning chains in the differential diagnosis and then by the care teams and so I I just have to remark like we have like this wild sort of similarities and in interest that have uh that's pretty cool too.
>> Yeah.
>> But I I wanted awesome.
>> Yeah. Go ahead. Go ahead. Yeah.
>> I I was just going to say yeah um
>> again kind of the project is a little half it was halfbaked at the end because I ended up having
>> Turns out that being CMO of a of a rapidly growing startup takes a bit of time that I just a little
>> anticipate. But we had a project similar uh you know with different intent with the idea this you know Socratic guide that takes an agent and walks in an adaptive way through the CPC's but to your point, you know, that I think the reason that attracts us so much is not the answer it's the wayinding that is properly diagrammed by the expert that comes in
>> either the idea that the answer is not the important part here right those experts a lot of I think we actually When I was working with Matt, we actually proved that sometime in the early 2000s, they started giving enough more information such that people start getting it right more often.
>> Yeah.
>> Because it it's not about whether they get it right or not that matters. It's about that reasoning process that gets
>> Yeah. And I think like one thing led to another and we were initially focused on like writing up the differential diagnosis with the reasoning then using them as like a benchmark for different reasoning tasks that are evidenced in the cases and then eventually just can we create a model to present a slidebased presentation and narrate it through that physicians can watch that you would see on on Zoom if Gerri were dialing in and uh presenting his uh his deck as sort of the inspiration. Um, okay I think that's a great transition. So you were working on the CPCs. I think you actually started as an assistant professor, right? You're at UCSF and you're working more and more with the open evidence team and then now at some point you join as the CMO, right? Or you join the team, you become the CMO of Open Evidence. And maybe we can just start with a very practical kind of question just to start digging into the work that open evidence is doing with how like could you walk us through how a physician actually uses open evidence at the point of care and maybe compare it to tools like it almost sounds a little bit silly to ask this now but there probably some physicians who haven't used the tool as yet but I think through your lens like how physicians are using this like what's a typical use case and maybe also how that is different than some of the other tools that they would reach to that are in their quiver like up to date or a Google search or or something else or just chat GPT.
>> Sure. Yeah. Yeah. Absolutely. And there are a few distinctions there. I think it comes down to what we started this conversation at which is what was the problem we were trying to solve and how do you train a model for that? You could imagine that starting with this okay from the outside one would be okay there's a medical question. what I want to do is train a model to give the right medical answer. And that was not the problem that we started trying to solve. Right? The problem that we were trying to solve is given a medical question, what are the right references to answer that question given today's date and given the person asking it. Um, with that last one being a little nuanced, we we talk a little about about that, but that's that is a slightly different question, but it also dictates how you train the model, right? In a very strict way, right? And so our goal is two-part but the first one is given a medical question how do we identify the right references to answer that question given the extreme complexity and multiple dimensions that in the medical literature that exists on right so one is how relevant is that paper to the question being asked uh two is how recent is that and is it still up to date three is kind of the strength of evidence within the paper and the last is kind of authority and I want to be clear like there's nuances and there's actually qualitative information in all of those, right? And then there's a last piece which has been more and more exciting for us which is think about rather than think about this as a single retrieval problem, think of this as an optimization problem like an APSAC problem kind of thing, right? Where well actually how do these pieces fit together so that you're pulling the right seven references instead of 12 redundant references or you know 12 references where four of them are redundant. So if you're training a model for that, the actual training set you use is very different than what you would do if all you're trying to do is get the right answer. But it's also a lot more robust because then what you're trying to set up is what is the right paper at any given point in time given the day as things change as opposed to what is the right answer given a question that might change tomorrow, right? That answer might actually change. So those were the kind of retrieval questions I was very interested in but the start of it and then of course you also have to get the answer right. Right? So those are both equally important, but having both of those as pillars, I think, is really critical to thinking about what goes on under the hood in open evidence. So how do doctors use it? In a lot of ways, for what doctors see, it's a lot like a chat interface, right? You ask a medical question, you get an answer, but that answer is not just in isolation. It's with all the references that help support why we believe that that is the right answer that you can go and double check. We're moving to more and more multimedia, right? So, it's not just text answers. You also have the Kaplan Meyer curves and exactly why they say what they say. You have clinical images for dermatology, your X-rays to show, okay, this is how it looks in three different real world examples. We have the flowcharts for how you would practice if you're an oncologist in different areas. So, that's how it looks for a physician. How physicians are using it is obviously something I'm really excited and look into. What I like to say is based on our data, doctors are often like most of the time they're using it, it's to doublech checkck things that they think they already know. It's this idea that as you're practicing in outpatient, inpatient, for every patient you see, there are probably four decisions you have to make. And I bet three of those four decisions you're only 95% sure of. You're probably 100% sure, but you're not quite there, right? And obviously if you're 95% sure, you're not going to spend 20 minutes for every single patient getting to 100% sure. And then there's a question that maybe you're 60% sure or or you know very much lesser or maybe you have no idea. And what you know what we're seeing is a lot of times it's just being used as a double check to make sure I didn't miss anything. Does the chronic kidney disease actually affect the answer? You know, I haven't seen this kind of patient in 4 months. Have the guidelines changed? So those kind of double check questions on things they're pretty sure of. And then you know there are these kind of wow I haven't thought of IGG4 disease since I was in medical school. Can you remind me what the heck is going on or differential diagnosis stuff but you know in our data things like differential diagnosis which is stuff the New York Times likes to talk about with relate to AI is really only 7% of questions that get asked by our platform.
>> Yeah, it's treatment and management. It's most I mean I know we're not slideshows. I'm a big SL you know I can show you the data. You don't have to. So it's mostly treatment and management around a number of different axes. Um, and it's things like next steps. If it's diagnostics, it's not like I have no idea, give me the differential. It can be, but that often is trainee dependent, right? It's like they do that a lot more as interns than they do as end of attendings. And uh even the diagnostic stuff, it's I'm thinking of these two things. What are some features that might help me go one direction or another or what's the next diagnostic test that will help me distinguish it?
>> Yeah. Exactly. So that that's kind of how doctors are using it. With regards to how it differs from other tools out there, there's a big distinction between what we do and up-to-date in kind of orthogonal ways. And so and I think it's important to to kind of think about that. We're precision medical information retrieval. So you ask a very specific question, we are going to answer that very specific question. So that's great from a ergonomics time optimization perspective, right? There are downsides to that and I I like I like to talk about it and then recognizing that and then figuring out ways that we can be better at it. The downside of precision medical information retrieval is you lose a lot of the residuals you get from a browsing experience. So if I want to ask a question about coxidium micosis but I don't really think about coxidium micosis I might ask okay what's treatment for coxidio micosis and I'll get very specifics on how to treat but if I don't know anything about coxidomy micosis I go to up to date I'll probably just type in the term and then with that I'll have to go through okay what are the symptoms I should be thinking about what are the differentials what are the diagnostic steps I have to actually accomplish you know there's a lot more residual information that you have to get through before you get to your specific answer. And that information may trigger different neural firings on what to do next because you're actually being exposed to it in ways that precision medical information search um doesn't expose you to. So I think that there are benefits to both, right? There's a So, what we're trying to do in the next steps is give you the precise answer, but then learn from where you're at in your practice setting to suggest better follow-up questions that could get you towards information. You may not have thought about the kind of unknown unknowns in your process where you didn't know that you should know this information and kind of bring that browsing experience back in a way that's still very targeted using machine learning. And then with relation to other tools, I think it's pretty obvious, but you know with regard to other geni tools, a lot of it is about the information we use, right? So we are very conscientious about our content. And I mean that in two different ways. One, there's the walled garden, which means we don't take things from sources that are not trustworthy. We don't take things from Facebook, Reddit, the wider internet. Um, you know, Chad and other foundation models are doing better and better at that. So I think that's true. There's another thing that gets discussed a little less, but you know, there's so much of our system of basing medicine on the evidence that requires good faith. And what I mean by that is publishers like the New England Journal have a business model and they have to keep that business model running. And that business model relies on people paying for their content in some way, shape or form. And I know that from a legal perspective that is being contested by a lot of foundation models, right? They're like, you know, this is public information. We're just going to use it whatever way we see fit. And I think the whole system collapses if publishers that are producing this, especially groups like New England Journal and Societies, none of that is actually recognized. So, you know, the other piece that I'll say about the way we work with content is we pay for that. you know, we pay for it in a very transparent way where we want to make sure that this works for both us and the group that's creating that content because that's very important to continuing this in the next 3 5 10 years.
>> So Travis, just to dig into the comparison cuz I think a lot of clinicians are using open evidence of course, right? But a lot of them are probably just typing things into chat GPT2 or one of the other models and asking it its thoughts. And so I think you touched on this, but one of the first experiences you'll have when you use these models, and I think this is true even today, although I think it's less true than it was a year ago, and even less true than it was 2 years ago, is that the baseline or the frontier models tend to hallucinate. And I think this is still a problem or confabulate, right? make up references and get the sort of the veneer of the argument correct or even the paper correct, but then you know invent who the authors are or change maybe some more important and salient things about the paper itself or the core argument that was being made by that paper. And I think I I imagine that you spend a lot of time and there's a lot of effort on your team at guarding against hallucinations. And if I'm just to think out loud about how I would design this, I think it's not at one step of the process. I think it's probably at every step of the process. There's risk that comes with the sort of inputs that go into the system. There's risks that come with the outputs. And then there's ways of gating or guarding what is actually surfaced to the user in a way that it's maybe drawn from a set of trusted references or verified or checked. And all of these are I think potential ways that I know others who are dealing with chat bots that are facing patients or facing other users have to have have tried to tackle or trying to tackle right now. I guess my question is how has your lens on the sort of importance of the problem of hallucinations or confabulations? How has this changed over the last couple of years as the sort of core base models have gotten better and uh and also as you've sort of been in this period of of very rapid growth as well where presumably you're learning a lot about the usage and you know a lot about the usage of these systems in detail.
>> Totally. Yeah. Yeah. I think you annunciated pretty well that you know inaccurate information that doesn't have a verifiable source can leak in at many different steps and the first one comes with inadequate retrieval which is where you get a lot of these hallucinations for references that don't even exist and or not supported by anything you know just pulled from weights directly which has its place right that absolutely has its place in a lot of generative AI so inaccurate retrieval can lead to hallucinations Absolutely. And then you know the second step and I think so that being said I think we're foundation models are getting a lot better at the idea of hallucinating whole cloth references right I feel I actually have even in the last couple weeks you know people have said that they're seeing this in in different foundation models blah blah blah but it's much less a problem than it was 2 years ago right but that doesn't resolve the problem of inadequate or inac accurate retrieval itself. And I'm still speaking about hallucinations, right? I'm not talking about inaccurate answers. I'm talking about actual hallucinations where if it's done a search, but it doesn't pull what it needs to pull to answer the question, it will still due to syncopancy hallucinate. Um, maybe not hallucinate a reference because again, it's getting they're getting very good at tell, but they'll hallucinate what is almost more dangerous, right? Because they don't have information,
>> right? And and so and actually right so I am
>> it's a failure mode that is shared with people too to be clear.
>> Yeah. Exactly.
>> Totally. Yeah. Yeah. So so we're actually moving to more and more nefarious problems with relation to hallucination. The one of just the reference doesn't exist. You know yeah lawyers get caught up in that. Obviously MDs could as well but it is something that's more easily verifiable because Google's actually very good at identifying okay can I go pull this paper? Right. And then there's the issue of inadequate inadequate or inaccurate retrieval which can force the model to hallucinate if it's trying to answer the question with information it doesn't have. So that's the next the the next two are actually very hard and those are hallucinations relation to information the model thinks it has but doesn't and we talk a little bit about that. And then the last one which is borderline hallucination and more just reasoning failures but can look like hallucinations which is when synthesizing from multiple sources how to make sure that the model actually reasons and answers a question
>> right
>> accurately given the information presented and there's a gray line there between is that truly a hallucination versus a reasoning error but but to kind of walk through those. So given a set of sources and this is another reason that we feel like it's so important to partner with publishers with full text is um making sure that the answers that we get are not just based on the abstract um because what we found more and more is anything with problems with reasoning often is because we haven't been able to show the full full text for the article and I'll give you a very simple example but you know if there's a table four that involves a side effect that involves the rate of rash in RCT T in the new New England Journal and then the question is oh what's the rate of rash if given this drug if it doesn't have the full access to the FDA label or if it doesn't have the full access to the New England Journal that that showed that it might say oh I have this content it must be here somewhere I'll just make it up right so that's one where it's like okay it actually pulls the right reference so it's not incorrect reference inaccurate reference but you can still end up with confabulation because even with the right reference the right information is not present because is the model doesn't have access to the right piece of that reference. So, so that's one. And then the last one I mentioned which I've been very and actually you know Sam on our team has been very excited to work on which is this idea of how do you take let's let's assume perfect recall where you take all the information for a given you know rare disease or or a specific question. How do you accurately measure and then actually allow the model to reason across that to create a synthesis that is very accurate given that it has to pull information from multiple different sources. And that is the next step. I think that's where we're we're at right now with regard to the frontier of language models. That is still challenges. And I still think humans do it better where if you take the expert who's the endocrinologist who's been doing the guidelines for 20 years, I mean I I know it sounds obvious, but they're going to do a much better job of synthesizing what the pros and cons of these 12 papers are and how to create a specific set of recommendations than if you ask a model to do it. And so we're kind of trying to learn and work on that to think about maybe that's moving further a field to reasoning failures, but they can often look like hallucinations in the right context. So I add that to that list of the continuum. And as I was mentioning in terms of nefariousness, those are the ones that often are are really hard, right? Where only the expert who really understands our field and the 12 papers will be able to say, "Yeah, but this is a midterm endpoint for the overall survival. So even though it's statistically significant in isolation, I really want to take it with caution because it's not kind of what was the primary objective was or or you know things like that.
>> Awesome. Shout out Sam Finlay, shared friend of the show and uh currently working at Open Evidence on some cool stuff. I wanted to talk a little bit about the rocket ship ride that you've been on. I know that you started this as like a hanger on in Slack and as a consultant before we started the show. You mentioned that we had a chance to chat way back. I would have made the mistake that you guys did not. Uh me and Zach worked on a lot of like predict the diagnosis versus retrieve the information to support the diagnosis. And one of the things that was never clear to me was like what's the business model and it strikes me that you and other companies like a bridge have found meaningful business models for healthcare AI products. I think of open evidence as DTC where the C is direct to clinician and it's clear that clinicians are willing to pay to access open evidence and be able to have that sort of trusted source um of of information as part of their practice. Maybe you could walk
>> Can I can I please can I skip in just real quick there because um maybe I'll talk maybe I'll I'll kind of um
>> elucidate a little bit because I I do want to create one correction, right? So
>> maybe willingness to pay is what I should have said because they there's a lot willingness to pay. Yeah. Willingness to pay.
>> Willingness to pay. Yeah.
>> Okay. I'll let you continue.
>> Thank you. Thank you. Yeah. Yeah. Um my wife's a clinician and and she she uses it all the time. Maybe you could just talk like walk us through what the trajectory of open evidence has been like and kind of like what it has felt like to go from this kind of like small group of folks in the Cambridge area. I think to date you guys have raised something like over $700 billion at a $12 billion valuation. So just like a complete rocket ship. So just quality like from a personal perspective, what has that felt like and like what lessons have you learned along the way as the company has scaled so quickly?
>> Oh, those are big topics. Um, I guess I'll I'll just spend a little time talking about our trajectory as I see it. again not as the co-founder but as a hanger on early on the most important thing that I saw early on was there was a focus on how do we make sure that this ends up being a business but more importantly first of all is we'll figure that out what is the problem we're trying to solve and that problem was information retrieval in medicine and I think you know Daniel and his previous companies had a lot of experience in information retrieval and he was very passionate about this idea of there's needle and hay stack problem in medicine just like I've seen in other places. How do we solve that problem in a really robust way? And I know it's such a problem that once we solve it, we'll figure out something to do with it, you know? And um we had some partnerships early on that were not this direct to consumer model. They were fine. It wasn't that there was any problems with those. What happened was in the end, Daniel was like, I want to release this for free to clinicians. That's what I want to do. I want to see how it grows. I want to use that as a flywheel to understand what we can do better. Um because they're the ones, yes, we, you know, Travis, you're great, but the ones who really matter are the everyday clinicians and we need hundreds and thousands of them to tell us what's wrong. Um so I think that was the big turning point where we're like, we're just going to release this um and we're going to get feedback and we're going to monitor. I think the other thing that was so much fun is we had an, you know, an email and we were like, "Please email us with anything you see." And Eric and I and Zach, you know, we'd look at that email every single morning and be like, "Okay, this person said there was a problem right here. We're going to catalog all these and figure out what was going wrong and how do we make it better?" And so that kind of release early, release free and uh just allow be very open to criticism to make things better I think really allowed for the kind of rapid growth. Um
>> I mean that that's incredible and I think like the playbook for software product tech startups but had had been non-existent I would say in like healthcare IT startups here like up until this point. Normally you're trying to land a meeting with the right CIO or something like that. So I think credit to
>> credit to you guys for like Yeah. Yeah.
>> Yeah. Yeah. No, no, it's not. Yeah. So I wasn't part of any of those, but the way Daniel the way I've heard it maybe I was either directly from Daniel or or you know
>> they had a couple of those meetings
>> and Daniel was like this is the biggest waste of time. These are not the people I want to benefit from this product. I want doctors to benefit from this product. So, we should be giving this to doctors. Again, that's secondhand. I at that point, I was fortunate enough I didn't have to be any part of those meetings, but I I hear it like and it's like you said, I think it's very much from the software, you know, that the Silicon Valley playbook, which is you have to get it in front of people who you actually want to use the tool.
>> Cool. Um, so maybe forwardlooking like what does growth and success look like for Open Evidence over the next year or two?
>> Yeah. Um, so we've been exploring a few different things. Just to call out the elephant in the room, right? We are quickly approaching market saturation in terms of users, right? Um, we have well over half the doctors in the country. Uh, it's like 650 or 700 thou. I think we just hit 700,000 clinicians in the United States uh, who've used us in the last month.
>> But what fraction of the global market is that? There's always a bigger dam.
>> Yeah. Yeah. Yeah. I get that. So So that's that's part of what we're talking about, right? So that's that's us you know this is a product for clinical use and so credentiing is very important as part of that right so if you think about just very strictly dimensions one is you can increase users or two you can increase use right? So those are the the two ways to grow so with it with regard to increasing users international market and so to do that we have to be very thoughtful and careful about it right because we were very careful here you know is a you had to have a NPI to log in. We wanted to make sure that that it was designed for clinicians. We wanted to keep that. So, you know, we
started Canada. We're working to, you know, make sure the credentialing works there. And similarly in EU and and others, one step at a time, making sure we go through all the security, the privacy, the regulations so that we can fully expand in those areas.
>> Cool.
So, that's one side, which is and and you know, we're doing a lot of fun. Actually a project of mine which I don't mind shouting out a little bit if you don't mind is how does this look in different resource settings right so uh we've been working with Result to save lives in Rwanda to think about what is the right way to adapt open evidence when there are resource constraints like in low-middle-income countries and that's a really challenging topic with a lot of ethical quandaries because on the one side the right answer is potentially the right answer anywhere right you know evidence-based literature the goal is we're all humans they you know, we do a randomized control trial and for all its faults, the idea is this is expandable. But at the same time, if we're constantly answering oncology questions with $200,000 a year drugs that are not available in that resource constraint, that's not very useful to the clinician, right? So, how do you adapt for low-middle-income countries with that balance in mind? So, that's another thing we're doing in that regard. So, that's one.
And then the other one that's been really fun for me to think about is what are some other ways that we can build for use. So that is how do we make open evidence more useful. So more users, more useful. Um and so we're building a lot of things both within the health system. So we're now full circle um now working with health systems be like okay it's great for your doctors, your doctors love using us. What are some ways that we can help the health system? And if that involves you know AI scribe capabilities with evidence um built in, if that involves kind of more inpatient management and understanding evidence gaps and those, if it involves you know a simple thing that we've done that has actually been really effective is taking information from the health system about how they want to practice right like um antibiotic sensitivities, classic example right like yes the IDSA says this is what you do for CAP but the infectious disease department at a hospital knows what the drug sensitivity activities are for their local strains. We should make sure that that's incorporated. So when a doctor asks that question, we can provide that. So those are kind of some of the early things that we're doing in the next 1 to 2 years is always way too long a timeline, but let's say the next 4 to 6 months.
>> Yeah, the next um 1 to 2 hours um is probably the right timeline. One last question before we move to lightning round. I'm obligated to ask this to all startups. I ask it of myself all the time. What is your moat? I think that the frontier, the space, the surface area covered by the frontier models grows all the time. A lot of them are big in healthcare and life sciences. How do you think about using the wedge that you've created to create like a durable moat and a sustainable product in light of the sort of >> exogenous forces that I'm sure you think about all the time.
>> Yep. Totally. And so I think the two moats that we've created, one is based on effort and the other is based on we started early basically early effects. One is, you know, the moat that I talked about with working directly with the people creating the content so that we're using it responsibly. And what I mean by that is we haven't quite crossed the Rubicon yet where yes, I know Foundation models are stealing anything. They probably are watching my screen right now, right? They're stealing everything available. But in terms of reproducing it, they haven't quite crossed that Rubicon of reproducing figures, reproducing images, reproducing surgical videos that someone's created and edited. And so by licensing that content and be able to provide that, we are visual beings, right? And it's one thing to see the full statistics of the whole randomized controlled clinical trial, but it's much easier for me to understand where the gap starts to develop if I get to see the Kaplan-Meier curve or if I'm an ED doc and I'm looking at a hairline fracture, it's really nice for me to be able to see here's exactly what that image looks like across a number of distributions. And so one is because we started working responsibly with these groups. We've heard from a lot of them that they are very reticent in the idea of working with these foundation models because of you know they're starting from a place of begging forgiveness. So that's that's one. And then two of course is the data flywheel right we have 200 million questions. Obviously foundation models have a lot of questions too. The difference is those questions are often asked by consumers and the questions asked to our model are very very complex questions and they're tied directly to the specialist where we know this is how long they've been training, this is where they practice, this is their specialty, their subspecialty in GI oncology etc. And you know we never share that information outside our system but it does help us in a lot of ways learn both for retrieval but also for writing on how to make our product better. So those would be the two that I'd point to immediately.
>> Travis, can I just ask that's kind of related to what you just answered, but I think it it ties to maybe some of our our conversation from earlier. I think what's amazing is that I walk around the Harvard hospitals and I go to the residents' room or go see my colleagues there and a lot of them have OpenEvidence open on their phones or on the computer screens there or chat or another LLM and it's not like they had like a class that taught them how to use it, right? I think the medical schools and the fellowship and residency programs, they're like very slowly contemplating with committees how to integrate this stuff and then committees on committees how to integrate this stuff into care. And the doctors aren't waiting, right? The clinicians who are actually seeing patients are not waiting. And then the patients are not waiting either. They're I don't think they're using OpenEvidence unless they're a doctor and a patient, right? But they're using chat or one of the other models. So, sort of a two-part question and they're they're kind of related to each other, but yeah, maybe not. The the first is how do you imagine medical school changing? How should it change in this AI era? Um, should we have classes on how to use LLMs and OpenEvidence or they're going to figure that out on their own? What should physicians learn about AI tools while they're in medical school to prepare them for their future clinical life, if anything?
I mean, I think a reasonable answer is like Yeah.
>> Yeah. Well, so I I think there's a more fundamental question about what what should we what should we be learning in medicine?
>> And then, you know, as part of that, you know, how how does >> the fact that I'll be practicing with an AI tool um fit into that? And we have the same background. So, I I think you won't be surprised that I'm going to lean on that bias to say that.
>> I think everyone should be learning physics. We should be learning more more basic pathophysiology. I'll say I I'll go a couple more levels of abstraction than that, but I think the basic pathophysiology is the stuff that will never change. Um, and I think it's it's really fundamental to understanding how everything else you're going to be doing in medicine works. And so I, you know, so maybe not the Krebs cycle, you know, but you know, things like why does an ACE inhibitor work, right? Uh, for example, or like you know, the RA system more generally, right? Because in the end I don't think it's probably going to be very useful to memorize second-line hypertensives in a CKD patient because one we can look that up and two the shelf life for that information is just so short, right? It doesn't it doesn't last very long. But um, I do think it's important to understand, you know, the potential reasons why these side effects happen or the potential reasons why this treatment might work because those are foundational concepts that help you interact and ask questions and and get to the right answer and know the answers, right? Understand that right answer, know when maybe you didn't provide enough context to an LLM to get that kind of specific treatment management one. So that's from a global perspective, you know, what I think we should be teaching more in medical school. And so that leads to my next thought on what we should be teaching about AI. And uh again, biases disclosure. Sam and I created a open-source education tool for this. But the idea of that lecture series was look, if you're an orthopedic surgeon, you may never need to know about, you know, the kidney that much, but in medical school, you still have to take a couple lectures on the glomerulus. No, no offense to orthopedic surgeons. Maybe maybe they have to use it a lot. But you know, there are these foundational pathophys or physiology and pathophysiology concepts that you need to know because we have a very interconnected system and it turns out that your patient may be on one of these drugs and you're going to stress them during surgery and this is something that may come up, right? You never know. And I think very similarly, understanding the physiology and understanding the pathophysiology of AI systems is very important. So no, you don't need to understand the different pathways of a transformer model, but it's good to understand what pre-training is. It's good to understand where that data comes from in pre-training, right? It's really good to understand what post-training is, you know, in a in a very basic sense, you know, like what is reinforcement learning? Why what is the human feedback portion of that? And understanding that physiology helps you understand the pathophysiology of ways this can break and what to be on the lookout for and what are the the potential risks of adapting out-of-distribution, for example, or or things that you might be doing um and not even be aware of it if you don't have that physiology and pathophysiology background. So, and and maybe it's prompt engineering. You know, someone else might say, you know what, all we need is prompt engineering. Just make sure they know how to interact with it and they'll be fine. But for me, I think one of the great things about medical school is it isn't a technical college in in some ways, right? Like it makes you you first have to understand that we still have this belief you have to understand the basic science. You have to go to OChem at some point. You have to learn something about the basic physiology. And I think now that we're going to be interacting with these AI systems either in an active mode um where you're aware of it or more and more probably in passive modes, right, where it's an operation support and you don't even know what's happening underneath the screen. I think that kind of basic physiology and pathophysiology of AI is really important.
>> It's a great answer. I think it's a little bit different than what I I would have guessed, but I I uh I think it's a great answer and I think it to me >> if I if I didn't answer the question, I'm happy to to you ask.
>> No, you to you totally answer the question. I I essentially was asking how medical school should evolve in the AI era. And I think you you know if I'm paraphrasing, I think you said there's a lot that medical school actually is doing correctly already, which is giving you the sort of understanding of pathophysiology and sort of mental schema, a way of organizing knowledge and knowing sort of what threads to pull on for for accessing information uh relevant to your patient. And of course the Krebs cycle got mentioned, which is great, but I think people are imagining very different like very different sort of futures from a very practical training on prompt engineering like you're saying or using the tools to uh I think trying to turn physicians into computer scientists in the middle of medical school, which is maybe the the other extreme and so I I think yeah >> the the other the other thing I'll mention sorry the other thing that why I feel strongly about this is like another phenomenal thing about medical school is it actually allows a lot of maintenance of differentiation, right? Or pluripotency. Yeah. Right. So like >> if we are like okay, we're going to make medical school now just how to interact with prompts so that you can use OpenEvidence to treat your patient. We're never going to have the physician scientists making new drugs >> or we're never going to have the physician scientists who are running clinical trials to test those drugs. Right? or we're never going to have the people who understand the business of a health system and how that works, you know, and with boots on the ground. So anyway, th those are my thoughts. You know, I would love to maintain that pluripotency of medical school because those foundations allow for doctors who do many things, not just one thing.
>> Amazing, Travis. I I think um I think that brings us to the lightning round, Andy. Right.
>> Yeah. Awesome. So, we're going to hop to the lightning round now. These are going to be a bunch of rapid-fire questions. Uh, keep your answers brief and we think if they if we think they require further elaboration, we'll ask you for them. So, one that we like to start with just because it tends to be like more revealing than you think it might be is um, what was your first job?
>> Well, I worked in Lahaina. I grew up in Lahaina. My first job, I got paid to be a magician's assistant. Um, where I juggled and rode a unicycle for a July 4th parade.
>> That's pretty cool.
>> See, that's why that's why this is a great question. Like, that's why this is a great question.
>> Can Can you still juggle? Is it like juggling? Like Yeah. Once you once you juggle, you could juggle, right?
>> Juggling.
>> I probably couldn't um ride on a unicycle. Probably could ride a unicycle, but yeah, it's been a long time. The thing is that the taller you get, the worse balance is, you know? It's just like there's a biomechanical property there where surfing and unicycle riding just aren't as easy anymore.
>> Amazing. All right, here's our next one. If you could instantly digitize one thing in medicine that's still done on paper, what would it be?
>> Probably pharmacy um prescriptions. Um, I mean that I I'm just trying to think of what I use paper for anymore.
>> Not that much. Yeah.
>> Yeah. Taking taking your prescription to Walgreens still seems really stupid. I know you don't always have to do that, but seems like it's pretty frequent.
>> Cool.
>> Cool. Yep. Um, so this is a callback and uh you can choose um to answer one part of the question or other part of the question depending on how much you remember of your first year of undergrad. Who is your favorite philosopher or what is your favorite topic in philosophy?
>> Oh yeah. I mean, uh, Dostoevsky or depending on how broad you want to get, uh, Herman Melville.
>> Okay. Oh, wow. Great choice. Dark horse there for favorite philosopher. Yeah. Awesome.
>> Travis, what is your favorite non-medical use of AI?
>> Um, I've had a lot of fun um giving people different facial hair for um lab.
>> Nice.
>> Awesome. Um, this is a question addressed to your role as CMO of a very successful startup. What is your least favorite part of fundraising?
Um, sorry. I I you know, I'm going to have to pass on that, but I'll tell you a very good reason because I don't have to do any of it. That's my favorite part. Can I say my favorite part?
>> Yeah, that's your favorite. Fair enough.
>> Pretty cool. That's Yeah, fair enough.
>> Yeah. I mean, you know, we have Daniel and Zach. Incredible.
>> I just saw Andy's facial expression at the same time.
>> Lucky you. Lucky you.
>> Amazing.
>> I mean, I I They let me focus I They probably don't want me anywhere near that. I focus on clinical like how to make the product better, how to make the partnerships better, how to make the quality better.
>> Andy, what's your least favorite part of fundraising?
>> I also would like to pass that question. Thank you.
>> All right, we'll go to the next one. Travis, uh, this is the next one. Actually, this is the last one in the lightning round. Who is your scientific hero or whose lab would you join today if you were starting grad school?
>> Oh, wow. Oh, man. That's really tough. Um, well, I'll answer two separate ones. You know, um, my side, I I had great mentors. So, you know, my mentor in graduate school was Romain Baroo Kim. He's phenomenal, just smartest, nicest person I ever met. And then Ali Van Allen, which I can say the same. Um, and then, you know, I got to work with Atul for 8 years here um, and the amount of lives he impacted um on a personal level, uh, was It's just incredible to me, you know. Um, he Oh, sorry. It's a lightning round. Anyway, so Atul is uh, you know, the the the mentor, kind of the person I look up to the most in science. Um, he passed about a year ago actually. Um, very close to a year ago. So obviously wouldn't join his lab. Um, but I think I'd probably I'd love to go back to kind of one of the the kind of computer science labs and still learn. Uh, I never was formally computer science trained. Um, so, you know, maybe even Pete Soldovit's lab or someone like that. I don't know if he still takes graduate students, but um,
>> Amazing, amazing. Thank you, Travis. I think that's the uh end of the lightning round. So that was absolutely fantastic.
>> I'm very, very bad at lightning rounds. I probably my my shortest answer is probably like four minutes.
>> Yeah. No, you did great. Um, cool. So I think uh the last uh little bit here we like to zoom out and ask some bigger picture questions. You mentioned correctly that making predictions is hard, especially about the future and doubly so in AI. Um, I think we're like four years post GPT-4 or GPT-3, I forget, but we're still like very early as they say. What do you think the next four years are going to look like in AI for medicine? Do you think actually and let me give you a foothold on that that big question.
>> Do you think change is going to come fast in healthcare or do you think it's going to be slower?
Yeah, I mean, it's already been faster than most other fields, right? Like it's been us and it's been the two fields I work in, right? Which is software engineering and uh medicine, I feel like have had the biggest changes in how people practice.
>> That's interesting.
>> How people do their jobs.
>> Software engineering, totally agree. But if I look at non-clinical things like drug discovery, like like AlphaFold, like things like that feel radically different, whereas like healthcare feels like it's starting to change, but maybe not changing as fast as law where there's like lower barriers to entry and people can use an OpenEvidence like Harvey for law or I it just feels like there's some viscosity in the healthcare system. Maybe there always has been, but like with OpenEvidence, you know, we now have, you know, like I said, over 50% of doctors using us.
>> Completely DTC as you mentioned, right? Like that kind of pace in the last 18 months or 2 years is pretty incredible to reach saturation with an AI tool that is being used at the point of care to make clinical decisions. That that's pretty fast.
>> Agreed on adoption, but has to have things fundamentally changed as a result?
>> Great question. So then any, but you know, there are other OpenEvidence is one, and I would say that there's been a lot of change in practice thinking about primary literature with that. AI scribes are another one, right? Where um, there's been rapid adoption across, you know, what is now like part of the onboarding or part of the offer letter is like, do I get to use an AI scribe for for what I do clinically? Right? Because it's so important for for what how people practice. So I think in the next year, we're going to see a lot more around operations, so things that are going on behind the scenes for good or like I I have a lot of concerns about AI tools for operation management, but hospitals are under the gun in a lot of different ways and they'll probably adopt those things. Maybe I'll ask flip it back to you. Do you actually think that in science generally, there's been faster adoption than in healthcare?
>> Um, I think in certain areas of drug discovery, generative models have fundamentally changed the way that drugs are designed. Like you still have to get to the clinic, but the way that you go after like a new target, >> that timeline has been compressed from like 5 to 7 years down to like less than a year um in in many cases. And the way that the work is done, it feels qualitatively different than the way that it it had been done. That's a small slice of the drug development.
>> No, no. I mean, I'm I'm I'm very excited that you're so excited about it. Um
>> I likewise like your answer. I'm like, "Oh, that's awesome. I thought you were going to say >> Yeah, exactly. Cuz you have much you have much uh you're much much closer to the target than I am for sure in that realm. My other thought with drug discovery is I I just feel like there have been rumblings on this for a longer stretch, right? People, you know, like you mentioned AlphaFold, right? This is pre uh generative language models. Um, people have been trying to do this and I think it's just, you know, now we have the tools to actually do it very well, whereas pre-GPT nobody was talking about this kind of stuff outside of Dragon. Um, so, so from 0 to 60 feels like a little faster. Gregory, you know, I mean, it's hitting a lot of different industries or outside of general contractors, most people are worried about what my job looks like in the era of AI.
>> Yeah, it's a great point and I think uh I I totally see where you're you're coming from. It's like the maybe the absolute velocity isn't as high, but the derivative, like the acceleration probably in healthcare is surprisingly high.
>> Exactly. Totally. Travis, I think the as as you've pointed out and you know, as we've just seen by just talking to my colleagues and talking to the residents around here, uh, there really is very fast uh and very sort of durable adoption by clinicians of these tools in in clinical practice. But what I think I haven't seen yet is sort of an RCT or and I I know that this would be actually very hard to conduct and difficult to sort of get off the ground, but an RCT or a real sort of clinical appraisal against a hard outcome of how these tools impact clinical practice. You know, from your perspective, do you think, and I think this kind of echoes one of Andy's questions just a moment ago, right? There's really there's a lot of adoption, but like how practice is changing? Are errors going down? Are better decisions being made? Is management that saves lives happening earlier? Maybe as a this a little bit of an unfair final question, but can you maybe just give us sort of your thoughts about what we need for this kind of next phase beyond adoption by physicians of their products to really have the sort of clinical-grade evidence of the impact of these tools on care. And the scribe analogy actually is pretty apt because there's also been fast adoption there. And now we're starting to see some of the RCTs and I think we're learning that about the, you know, time savings, maybe the lack of time savings from some of these tools, uh, and also the impact on physician well-being and and other measures that, uh, that that mean a lot. So, um, do we need similar types of studies for, uh, for OpenEvidence and for for tools?
>> Yeah, let me let me hit a couple of those points because, you know, I'm not a I'm not an AI C person. and we have an AI scribe, but you know, it's not um my focus, but you know, I was actually just on a panel with Karen Singal um at OI and he said something I really love. We were talking about graduate school and what graduate student, you know, projects focus on or at least that's how he took it and I loved what he said. He was like, you know, I these kind of AI and healthcare projects for graduate students take two flavors. One is betting against the AI and one is betting on the AI, right? Like and it's a bit simplistic, but the idea is like, is your big finding that AI sucks or is your big finding AI is transformative, right? And he's like, I would bet on the graduate student projects that bet on AI, which I think is very interesting because AI is only going to get better. So for example, in the AI scribe solutions, yeah, I've also been underwhelmed by how much time they save and the errors and, you know, the corrections that have to be made and things like that, but physicians adopt it and they like to use it and it's only going to get better, right? These these things are only going to improve. And my guess is if we did this in a year, those RCTs that show there's not much improvement are going to be outdated by scribe options that show okay, there are going to be. So that's what I'll you know, I would say that I'm betting on AI. I think that it's only going to get better. And if physicians are enjoy, you know, using it, here's where I'll transition. These adoptions with some exceptions have not be driven from the top down, right? This is not like you have to use this AI C scribe tool because it's good for us. Physicians are choosing to use these tools. And so if you're choosing to use this tool for your job, I kind of see it's useful in two ways. One, it makes your job easier. Or two, it makes you better at your job. And in my mind, AI scribes make your job easier. It makes your life easier. At least physicians have felt that they make their life easier or they would stop using them, right? More or less. I I'm speaking broad generalities, right? And I actually think that physicians use OpenEvidence and we have less data to show that they make their lives easier if you do it in isolation. I think they make them better at their jobs because they are looking up and finding evidence where in absence of OpenEvidence, they would just choose to go with their gut. So they basically pull from weights versus actually retrieve their weights being their internal weights. Um, and so, you know, yes, we're still working on okay, how do we make that pairing, right? Cuz uh, a lot of we're we're EHR external, right? So when the doctor asks us a question, we don't know who the patient is that they're looking at, more or less, you know, we do. We're starting to create partnerships with health systems, but in terms of looking at outcomes, looking at changes of behavior, we're a little bit on the outside looking in. And we're very excited about exploring those, but my thing is, you know, nobody's forcing doctors to use OpenEvidence. So in large part, they think they're, you know, they think that this is either making their job life easier or making them better at their jobs. And so I agree with you. Um, we'll see in the next year, you know, what we can do in terms of health system partnerships to actually prove that out with primary outcomes.
>> Awesome. Well, I think that's a great note to end on. Travis, thanks so much for coming on AI Grand Rounds. This was really fantastic.
>> Yeah, thanks for having me.
>> Yeah, really awesome. Thanks for like lots of really thoughtful answers and um loved getting that opportunity to chat. Yeah, cool. This copyrighted podcast from the Massachusetts Medical Society may not be reproduced, distributed, or used for commercial purposes without prior written permission of the Massachusetts Medical Society. For information on reusing any JM Group podcasts, please visit the permission and licensing page at the NJM website.