Transcription
On today's show, Charity Anastasio, attorney, ethics nerd, and leading voice on practice management, legal ethics, AI adoption, and law firm risk management. We'll be talking about why every law firm needs an AI policy. Now, the hidden risks of ungoverned AI use, and how to create practical guard rails that protect client confidentiality, minimize hallucinated citations, improve oversight, and support responsible firmwide AI adoption.
You're saying what products are approved. These are the products we've vetted and the ones that we want you to use. And then it says how to use them. Use this one only for legal research because it's a legal specific product [music] that has legal guard rails. And then it says, "Who can use it?"
Welcome to the Alpha Lawyer Podcast, a show dedicated to exploring what makes law firms [music] thrive. I'm your host, Jason Marsh. Join me as I interview some of the smartest [music] thought leaders and accomplished law firm owners in our space. Together, we'll dive deep into law firm operations, marketing, technology, finance, and everything else you need to build an incredible firm that allows you to live your best life.
Hello, welcome back to the Alpha [music] Lawyer podcast. I'm Jason and joining me today is Charity Anastasio. Hey Charity, it's great to see you. I read your article in uh ABA's Law Practice today on why law firms need an AI policy like yesterday. And and honestly, my guess is that this is one of those topics that a lot of law firms, they're either avoiding it completely or assuming they'll deal with it later. We were talking before the show about some of the percentages, which we definitely want to get into because it is pretty pretty shocking. But the reality is whether firm owners or leaders know it, AI is being used in their firms. And so without clear guidelines, as you've pointed out, this is creates some real risk here. Whether it's confidentiality or relying on completely fake citations or incorrect case summaries. So today I want to break down why an AI policy uh not having one isn't an option anymore and then what the risks are that firms are actually facing and so how these firms can put something practical in place that's going to protect them from the risk but you know not completely jam up their whole team. And so you're going to help us do that. Uh but before we dive in, let me give you a proper intro here. Charity Charity Anastasio is a lawyer. She's the practice and ethics council for the American Immigration Lawyers Association where she advises lawyers on ethics, practice management, and essentially how to think through real world risks on evolving issues like technology and AI in their firms. So in fact, Charity just did a presentation on this very topic at ABA Tech Show. So if you were not there, you're going to get a good taste of it here today.
So okay, Charity, so you've talked about hey AI, it's not some future issue. It's here. It's being used in firm and again whether leadership knows it or not, people are using it and they may not have any formal guidelines around its use. So that's the reality. Staff's using it, drafting research, maybe just getting regular work done faster, but without any real guidance or oversight. And so as we said, there's a real legitimate risk there. And not because, right, lawyers are being careless or their teams are being careless. Maybe that's arguable, but ultimately this tech is just evolving so quickly and people are using these tools and they may not fully understand particularly their limitations. And again, no guidelines on what's allowed, what's not. And so when that happens, you know that firms, they're dealing with client confidentiality. They want to be accurate, I would assume, in their court filings, professional responsibility. So there's real consequences here. Is it an issue like whether or not these firms are using AI? That's happening. So the question now is how to manage it. So Charity, I want to start there. Why is having an AI policy no longer an option for law firms? And what are the real risks that that they're taking by not putting one in place?
Well, thank you, Jason. It's really great to be here and and thanks for that great setup. Uh it's no longer optional to have an AI policy because the bottom line is if you haven't discussed it with your staff, the chances are they're using it and and they're using it for good purpose and good reasons. It's because it cuts down on the time that it takes to do their job. And they might be having like this relationship with their AI where they're having this wonderful opportunity to reduce the time that it takes to do the work and they don't really even want to tell anybody that. So instead of having them secretly be getting this benefit without having your very smart guidance, make sure that you are involved in that process.
Yeah, that it's such a great point you're making there, right? Because I think we are currently in a stage where people are using AI but they're not really letting people know that they're using they want people to think this is just my work product here. This is all me. Maybe a little bit of AI but we don't really understand the extent. I know I went from losing a little bit to basically every single thing I do all day touches AI probably in some way. So let's start with the big picture here because I think you know there's still some lawyers and firms that are not sure how seriously to take this AI. So, first let's talk about the shift of AI. Like what are we really talking about here? Is this like some incremental tech improvement or are we talking this is a massive shift in the way we work?
Oh boy. I feel like you're opening a can of worms with that and we could talk the whole time just about that. There's books being written on this, right? I think it is a fundamental shift in the way that we do work and it's it's not all going to happen at once or all across the board or all evenly. It is it is it sounds like from experts going to change employment. So some people will be laid off, some people will have new types of jobs or their job will shift and change. Maybe some people will get better jobs and and more fun in the actual practice because they won't be doing menial tasks anymore, but a lot of people might be doing they might get laid off for those people that do do menial tasks.
Yeah. And and but but let's take a step back, right? We've seen technology advances, you know, and significant ones really over the last couple of decades. And of course, every time that affects the way all of us work in some way. But my question for you though is how is this AI and and how is that different from past technological changes in the legal industry specifically? What what what are we seeing that's different here?
Yeah. Well, I'm an English major, so I'm not going to talk in in geek speak. I'm going to tell it to you the way that I understand it, which is large language models work with language, right? Who works with language more than lawyers? Not many. Not many people except for maybe the English majors that actually get a job doing it, right? So, it's actually a perfect fit for us in many ways. And it and it is about a certain amount of automation. We've been talking about automation for decades, right? It is to a degree automation. It makes automation incredibly easy compared to what it used to be. If you just look at Bob Ambrosia's blog, it has a a story today about, you know, building it over the weekend with vibe coding. I'm telling you, it can do miraculous things that lawyers could never do before. So, it actually increases the ability of lawyers to do things that they didn't used to be able to do because it just uses language to do it and helps them. But, it also is going to change the way that we do our actual work. It's going to help us draft things, find things. I actually love the aspect of AI where it's just sucking data in and putting it in the right buckets. It's such a simple concept and it's one of those things that I feel like I always wanted my technology to do and it never did. Right. So there's these are like broad concepts, but also I guess I just say it's so incredibly flexible and it and it and if a lawyer picks it up, they'll kind of fight with it like you always do with new tech for like a couple months until they have a couple successes and maybe they get a little creative and then the next thing you know it's it's in everything that they do.
And and it's a different everything from lawyer to lawyer, right? and it still fits with them. So, it it's going to be a very different world in the next 5 to 10 years. I don't know when everything is going to catch. I don't know what which one's going to be the big fire that it's going to spread across. But this is going to change things fundamentally and I think lawyers have woken up. I you know the data shows that between like 45 to 75% of them are using it in work now and even like 30% of them are using it on a daily basis. And for immigration lawyers at least in this study from let's see 8 a.m. It's Nikki Black study the 2026 legal industry report uh she said immigration lawyers are using it at 40% daily. Shout out to those immigration lawyers who are a little more tech savvy. I'm super proud of them. I wish I could see a great
I don't expect higher numbers. I mean who I don't can't even imagine who's not using AI for some aspect of their work today. But let me ask you this like what are you seeing in terms of actual adoption of AI and use cases inside firms today? What what are some of the main areas you see?
What are some of the areas that they're using it?
Uh boy, marketing copy. um drafting things for their website, drafting almost anything. Legal writing, uh writing the legal drafts, research, where to go for lunch.
Yeah. I mean, that's the thing, right? You're you're doing some marketing copy. It's like, okay, that's one thing. I mean, what are the risks there? But the bigger risk is when you mentioned you're drafting legal copy, things you're going to file with a court, things that you're representing a client with. And so I think those are the things to be a little a little more scared about. I want to you you point out something I liked. It's called shadow AI and I want to talk about reality and the risks is what you're talking about there because it's the idea here that even if a particular firm owner leadership hasn't approved the use of AI either generally or in particular use cases the reality is as we said it's it's being used somewhere. And so can you talk a little bit bit about like what is shadow AI and where are you seeing it show up potentially in not great ways uh inside firms.
So shadow AI, the term comes from shadow IT, which is this IT term for all this technology that people want to use and they're really not supposed to be using because it's not the official product of the firm or the designated and controlled product of the firm. And it's a serious problem for IT teams because they can't they can't wipe it of data if your device gets lost or stolen unless they have sufficient control of that device. They don't know what you've put in there. It just means that there's less control over it. And I think that one of the reasons that it's become especially problematic and and you're hearing this term shadow AI is because Open AI thought they were just doing an experiment and they opened it up to the public in a free version early on. So chat GPT is one of them that is just very accessible, easy for people to figure out and and all you need really is a browser and you can sign up. And so the IT departments I think are struggling right to keep up with this. But I I think one of the places that you see it a clear evidence of it because mind you people are a little bit sneaky is that you see the hallucinated cases and and the hallucinated cases or just uh not accurate facts, not accurate quotations from cases sometimes are super problematic in our courts right now. Uh I I quoted it in the in the article that I I went to Charlatane's site.
Yeah. Dam Damian charlatan which which by the way uh he
so I had Jennifer Ellis on a couple of weeks ago and we were talking about AI hallucinations and she mentioned uh Damian charlatan which I I which by the way I love that his name is charlatan even though it's not spelled that way but he talks about you know fake you know AI coming out in legal
it's kind of koi right I wonder if it's
I want to try to get him on the show and see if he could uh yeah walk us through some of more uh ridiculous instances of this uh maybe scare some people straight. But but in terms of like what Damian's talking about hallucination in court filings, what are what are the there's real consequences for that stuff. What are we talking about here?
Well, there are real consequences financial and to your license and just embarrassment, public embarrassment. But what is interesting to me is if you look at those cases, it's mostly US cases about 75% before it's around 68% now US. And and of course the majority of those are going to be prosay clients. That doesn't mean that it's not a problem for us, right? Because prosay means it's a problem for both the judges and the lawyers on the other side.
But then it's it's about 35 to 38% lawyers doing this and it hasn't gone down. Every time I talk about it, I go and I check it and it's between about 35 to 39% lawyers doing this in the US. And I'll just compare that to it's about 1% judges and 1% experts doing it. So, it's our problem. But when you actually talk to the lawyers about how this is happening, well, some of them have really lame excuses about like, I didn't know. But the majority of them say, "I had my parallegal do it. I had my associate do it. I thought they were trained. I thought somebody else was checking this. I thought they would tell me. I thought I told them not to do it." And it's just it's a dog ate my homework type of repeated answer over and over. And it tells me that that's that shadow AI problem, right? that that somebody maybe you should fire them, maybe you should have fired them before or maybe you should train them better or maybe you need this AI use policy because somebody was using it and they used it in a poor way because they they don't know. They're not seeing the news the same way that you are.
Right? Especially if the attorney isn't double-checking their work, right? I mean, I I I am not an attorney, but I believe anything an attorney files in a court, it doesn't matter who produced the work, it's going to go under their name. And so,
take credit for it. It's all yours. Yeah. You have a responsibility. The chapter five of the rules of professional conduct say you're going to make sure that others that you supervise adhere to the rules of professional conduct, too. And so, it is it's totally on you even though those other people are doing it. And I think that shoots fear into lawyers hearts and maybe stops them from adopting it, right? Or maybe just adopting it for the marketing copy and we're not going to do anything else. But I I I urge everybody that's watching to fight against that that that feeling of like I just I'm not I can't I'm going to wait until it settles down because first of all, I don't think it's going to settle down. This is this tech is moving too fast. But second of all, I think what you have to do here is get smart. Not get scared, but get smart. Look at charlatans or charlatans website. See why it is that people do this and see what happens to them if they do. And you know, not all of them are equal. Some of them are you think, gosh, that lawyer was really good. Everything they did was great. He's a past judge. I can't believe he did this. It wasn't really a, you know, there's a lot of nuance there. And and I think the major message is we all need training
and we all need to know how this thing works in the legal space.
And and isn't that like any aspect of inside of a business or a law firm? There's policies and procedures for lots of different things. I mean, it could be how to how to handle the refrigerator and when things get thrown out in the you know, but I mean, as simple as that. So, well, but I mean, I'm saying like
it's the next policy you do.
Yeah. Yeah. That's I mean listen that's that's a different podcast handling the law firm refrigerator but
but that's the thing here and I rather than avoid it just look at it directly and I don't think it has to be over complicated but to start thinking about it and I and I want to get into why getting in an AI policy is important but I think I read something in your article that essentially said when AI is being used on a let's say this is a parallegal or an assistant or not the attorney but the attorney's getting work from somebody that they need to specy speify in a formalized way maybe just a couple of lines this was the AI tool that was used this was how it was used and this is where it was used and so for an attorney it's very much like they've always practiced legal work is there's citations and they need to verify that the citation relates to the you know the matter at hand and so forth and essentially that there's basically a trail of where this stuff came from and so people know it right is does that make sense
yeah I think that this is we I In the article, I talk about the use policy, but really it's a change in both your workflow or your process there and a policy. And it and it does have to have an element of accountability and transparency to it. And I think there's a lot of creative ways that lawyers are starting to do this. I heard from one that said they they put what what product they used, where they used it, and then any information that is from that product, they highlight it in neon green, right? So that the person that is reviewing it actually really sees it clearly and they don't forget where it is because that is a possibility with complex legal writing and and they don't change the color until it's been verified. And really, I think too, the verification needs to be one of the elements there. this is how I verified the results because that's also proof that they that they internalize the training about what they're supposed to do and that they're following it. And then if they do make a mistake, well, it just is a better story to the judge, too, right? You're saying, "Your honor, I'm sorry that happened. Let me show you what our policy is and the documentation of what we did on this case and how that leaked into there, and I apologize for it." is so much better than I don't. They didn't use it. We told them not to. I'm
Well, you know, council, we think they did,
right? It's a different story.
I I think that's some really great advice and and you can almost see how simple it is. Just highlight it. Call it out so that, you know, the attorney can see what was AI and they can verify it. because it seems to me you get caught in court citing a fake case that how humiliating that must be and and just in front of the judge potentially in front I mean imagine having to explain that to your paying client. I mean it just it seems unimaginable to me like it really does and so all right so let's talk about
AI policy just very simply like what does a good AI policy actually do for the firm?
Well, it it lays out some of the things that we've talked about already. Uh this is the way that I usually uh think about it is you're saying what products are approved. These are the products we've vetted and the ones that we want you to use. And then it says how to use them. Use this one only for legal research because it's a legal specific product. It's legal guard rails. Use these other ones for whatever you want. and and then it says who can use it and it might spec specify the trainings that you're going to give people to use those things and maybe some requirements for them to get through certain hurdles to get to certain ones. I've talked to a lot of lawyers that at least at this point they say I don't want my associates to start in AI. I think they need need some more base knowledge and some deeper understanding of the topic before I have them do it or I don't let the interns use it. I want them to struggle through it. So, making those nuances clear in the in the policy. And then I think it's also a good idea to set out like how to how to get more. If I want a product, how am I going to vet these products? I'm going to take it to soand so. I'm going to say this would be good for this. This is what I need to give them to have it considered carefully. And then we'll make a determination. And that way you avoid that shadow AI problem where they're like, "But this is the best tool. I want to use this tool." If they want to use a tool, bring it to us, right? And and then two, I think it's a good idea to set out that process that you're going to adopt and emphasize strict adherence. If you're kind of a stricter firm, this would maybe be my style, but maybe you even say what the punishment is if it's not followed.
Like like this could be this process if it's not followed and and then you follow up with the training and and you give everybody the training. In Nikki Black's uh survey, interestingly, I I was looking at the numbers today. Um 54% say that they have no plan or current training on AI, generative AI, and another 17% say um no, but we plan to in the next 12 months. 11% say yes, it's mandatory. Well, bless them. They get a gold star. And yes, optional. And 7% are unsure. So, of course, unsure people go in the 54%. That's over 60% of firms that have no plan to even train somebody. So, we said lawyers are adopting this and they are.
There are some that are doing it loosey goosey and they're not setting people up for success.
So, so there's there's two pieces here, right? There's adoption, right? Just that we're going to adopt it. It's going to help us work better, more efficiently. And then the other part is a policy around how it can be used. And so you like what you're describing doesn't sound I mean, it's work but it doesn't sound infinitely complex and you could probably have AI help you actually do it but
but one thing I was thinking there's maybe like two buckets of firms right there's a little bit larger firm they have maybe uh you know a CTO or an IT firm or an in-house IT director or something and and in those cases it sounds like okay those can be the people that can start to craft this policy with the leadership of the firm and then you have smaller firms that they don't have they they might not even have like a a very regular IT firm and and so they've got to kind of manage this you know maybe they have a couple staff an associate or something and they're going to have to manage this uh directly but all of which is to say like listen these are the tools that are approved if there's a tool that you want to use that isn't on that list let's come and talk about it
and one thing um and I'll and I'll let you go is or you know let let you jump in here I spoke to an attorney and he said one of the things that they do and they have like a weekly meeting about whatever and you know he has a smaller team but probably you know five or six people but he encourages people to talk about and I'm not sure they even have an AI policy yet or maybe they do at this point but they encourage people to just talk about like how are you using AI and in a positive way to help share that knowledge across the firm and of course that can help identify oh maybe that's not the best way to use it um but what do you think of something like that just having open conversations about it that lead to creating a policy
I love love love it. I think that one of the uh one of the interesting things about AI is in not everybody has the same entry point or the same uh not everybody has joy with it when they first start. Look, honestly, I can be seriously bumbly. And so I I got in there and asked a bunch of questions that didn't answer them very well. I was like, "Oh." And lots of lawyers do that. And I'm sure lots of people do that, but uh you need a little work on it. And so it's really good for people to talk about their use. And it also sparks these ideas because sometimes I'm not as creative as I think I am. And I hear somebody else use and I go, I'm so jealous. I got to try that. Right? So, it spreads the word and it and it also helps people get comfortable with that idea of like, like I said, like I don't really want to tell people about this great like thing that's saving me lots of time and making me look better than I ever looked before. And by the way, you guys probably do know because you you have some assistant that used to give you terrible copy and now it looks great and you're like, "When did that happen?" I mean, I have I have people on my team, they I get an email and I start seeing the M dashes and I'm like, I don't even think they wrote I mean, hopefully they at least read what they sent to me because it's fine. It's what I wanted. But I'm like, I'm pretty sure like AI drafted this email and wrote the whole thing and whatever.
You know what? I have so much I have I'm so mad at AI for being into M dashes and N dashes because I use them all the time and now my writing looks like it's AI and it's not.
Oh man, that's funny. [laughter] I I can't I literally go in them all the settings and I'm like do not ever ever use any dashes of any kind. No M, no M N, no nothing. I can't stand it. So yeah, I see what you're saying. Hilarious.
But you made a really interesting point a minute ago and I want to go back to it because you talked about like when you have these these conversations again just making it transparent and comfortable that everybody's using it, but you make discoveries and I think this is the cool thing about technology. we've seen this is when you when you marry technological advances with human ingenuity and creativity and how they use it. And we've seen that forever. And I think AI is just going to be another version of that, which is really cool because like you said, people are finding really creative, interesting ways to use it that aren't standard way. And that's amazing, right? Shows you that yeah, humans still have a role to play here and and that it's really a useful thing like any other aspect of a firm is just sharing knowledge across the organization. The AI tech giants would like to tell us all we need is their AI and it's going to fix everything. But the truth is their AI is crap without our content and our deep knowledge. And that is the role of lawyers here is to be the shepherds of this and and really we we should be at the forefront of this. They should be asking us about the ethics and and so it's paramount upon us to claim our seat at the table of this this revolution or whatever they want to call it. It shouldn't be all theirs. It needs to have us because they don't understand the stuff that we understand. That's the truth. And and two creative purposes uh you know my boss had the best purpose. We have we have all these surveys that we've done of lawyers that we do this deep study and we have to pay somebody to collect the data and give us the data. But then we also we cannot hold more than a decade of studies in our head. So we took all those and we put it in an agent and I'd quiz it about the trends over years. And that's something that I never could have done in in past. My last study was the best study that I ever did because I also I argued with it plenty of times because sometimes it read the data wrong, but I know the data enough that I could argue with it. If you don't know the content enough, and that's what those lawyers that are saying like, I don't want my intern to do this are saying if you don't have a certain route, you can get fooled by the confidence quotient in AI.
That that's something Jennifer Ellis, she mentioned. It's it's the confidence that the AI presents this information that you know can still be incorrect. It's improving and it'll get better, but that confidence can fool you into just going just taking it by for its word. Um, so here's what I want to do. I want to talk to you about the core components of an AI policy, right? Like break it down into what needs to be included in that policy and then after that
checklist.
Cool. All right. So let's let's talk I've got some questions around it but so let's talk about the policy and then we can talk about the training of the of the policy and how it works and how to how to provide some oversight but let's start with just the tools like how should firm think about which AI tools are permitted right there's the ones that are available to everybody there's the legal specific ones I feel like AI is just creeping into literally any tool that you you might have and so you know even like I don't know like this app that we use to recorded the podcast on now has AI features built right in. So people could be using AI. So you really have to figure out what tools not just AI are permissible for use within the firm. So how do you do that?
Uh, so I think probably it's a two-tier for me what I would recommend is a two-tier setup. one generic product because I do think the generic products definitely have a place here and they're they're a little easier to learn on and they're more generalized. They're generalized tool. So, I use Copilot which is built into Microsoft 365. It's a $30 per user per month fee. Not very high. It sees everything that I see. It's brilliant in that regard. It's like a closed universe of my things. But I know firms that use Gemini or Chat GPT or Anthropic has the Claude. All of those are generalized products too. I think one of those makes sense. And you're looking at the use policy, the security measures, the client base that you have, frankly, and how concerned you are about uh somebody wanting to infiltrate that. And look, we know that general hackers like all law firms because data is data, but you might actually have some particular clients or reasons. Hey, if you're suing Microsoft, maybe Copilot isn't the one for you, right? So picking that product and then uh picking a legal specific one and that might uh relate to what practice area you have or it might be a generalized uh one if you have a lot of practice areas. So you have co-consel uh you you have boy I think all of them have something in it right Lexus have their little product I like describe AI by the way they did a great research project on that confidence quotient issue the worst one in the generic products still got 88% on the bar exam mind you
still pretty good right
incredible
So let me ask you this like so as a firm you're you you're going okay we're going to go through we're going to create a list of and like they've already done right I mean there's specific programs you use to write your documents and spreadsheets and so you add it to like your tech stack you have your list of approved AI tools do you then define like who's allowed to use what tools in other words certain roles within the business are allowed to use this particular AI tool others maybe not or they're use something else
his role within affirm an important thing to define usage as well.
I think it is for some lawyers. I would probably do it more on a training basis than a role basis, but I do see a lot of lawyers defining it by role and and probably more my perspective would be if I was setting up a firm, I do that general product for everybody, but I'd say there's some uses that you can't use it for, right? like what
if you're doing legal research, I want you to do the legal research in this platform, but you can still use this to rewrite your content once you get it done.
So that's a great example. So yeah, because that can be a great use case for uh chatbt or you know, similar tool. Yeah, pull out all the research, confirm that's real, and then you can use it to help organize your your outline or actually rewrite something or edit something that you've written. And then what about client confidentiality when it comes to using some of these AI tools, right? I know some have it built in. Other ones, not so much. And you probably should at a minimum be anonymizing all of the, you know, information you're putting in there. But what do you recommend when it comes to that?
You think you should anonymize all the information you put in the product that you own? Boy, oh boy. I think I'd rather try and ask them for zero knowledge on my paid version. So I think what you're saying is like don't put it in a free version, right? Please, please.
Yeah. Do not put client names in your base version of chat GPT or all
Oh, goodness gracious. Yeah. And do understand the different levels and then if you're not certain, get an IT professional to help you set the right settings so that you are not having it trained the model. There's there's even for me it some firms are more riskaverse than others. So if your firm's very very riskaverse, you maybe are only dealing with client information in the products that are legal specific and that don't only not train on the data but also um don't save that data for a certain amount of time because this can be a tricky issue where they can see and they scan. This is something that people don't talk about very much. They scan the different uses to verify that they are not bad uses like really bad uses that we don't want like criminal uses. But the problem is that
like what do you when you say how to build a bomb or something like that?
Yeah. Because lawyers could be dealing with something that's talking about pedophiles or child pornography or could be dealing with things that are talking about terrorists. This person was accused of being a terrorist but they are not and they have right to be immigrant here. So So a legal specific product that has paid to not have them scan that is is an important
So so I think what you're talking about here and I think this is super important, right? because we're like, "All right, these are the tools that we're going to allow." But then, and this is not something new when it comes to technology. There's a million settings, you know, behind the scenes that you can set up to operate this tool the way you want, right? And and that seems super critical and all the things you're just talking about. Is the model training on it? Is it storing the data? How all of those things are being handled? And so what is your recommendation as far as putting a policy in place when it comes to just the settings of an individual tool? It seems like somebody has to have some pretty solid knowledge of how the thing works in order to set it up properly. So So that's I would assume generally the IT professional the firm's working with. And it sounds like if you don't have one, you should have one because at this point it's not worth the risk.
Yeah. the age of having your your your son set up your IT system on when he's home from college. I think that's over and I think it's over for good reason. Look, I think it was over before uh AI came in, but now we have an additional risk. And I I I'm I'm going to hedge a tiny bit because hey, I'm not that IT professional, but also I think your policy needs to be kind of generalized. And so, you may not have every specific in there. And and and my third reason I'm going to hedge and say you're going to have to do the research and you're going to have to keep on looking at it is because that's a moving target. And we've had this problem with confidentiality for a long time. All the technologies we have, they have too much access to our client information. And they always say, "No, no, no, no. We don't we don't look at that though. Don't worry about it. It's no big deal." They won't we won't look at that. We promise. And what time and time again, they actually have looked at it. And there's been some bad actors and that gets exposed. And so, I feel like there's a reckoning to happen at some point in time with this issue. If you want to be the most rigid, you are going to have somewhat inflexible systems and processes because it's going to be hard to maintain it. Uh because it'll have encryption and it'll have keys and it'll have zero knowledge and so you're not going to get help from like it's hard to do this all yourself.
Yeah.
But most of us are to a degree exposed, right? This is the this is the fact of our modern-day technology. We're all a little bit at risk. Nothing is 100% safe.
So, I don't think that changed with AI. The big thing it changed is the ease and access with which we can expose client data without realizing it,
right?
And and maybe there there's more places to put it, right? There's like a lot of places you could put it and it could just spit back out because that's how AI works. It's a little
It seems like over time it's it's easy to become complacent as well. Like initially you start you're like okay these are all the the guardrails I have to be in but then you start using it and you're just putting all kinds of per I mean god knows the amount of personal information that I put in the you know my chat and so forth but
oh my goodness medical stuff right
of course I mean everything like you and and and you know without really thinking long term like where wait a minute this is might be stored somewhere because and you mentioned this a minute ago when when you they often say oh no no no we don't look at this but when you actually look at the term terms of service, you know, it's a mile long. Well, now you can maybe help have a uh AI tool help read and summarize it for you, but usually it's not quite that cut and dry. And it turns out you read that and you find there are plenty of loopholes where they may have access to your data. So, okay. So,
and and lovely contract of adhesion, which I hate, and they change the terms of service on a regular basis.
Yeah. Right. Yeah. So yeah, you'll get an email. Oh, we've updated our terms of service. Make sure you read it, right? I mean, I don't
Yeah,
nobody reads that stuff. So, and of course, they rely [laughter] on that. All right. So, policy, right? Let's just say that policy alone, that's that's where you need to start, but that's not going to be just enough because people need to know how to use the tools in the way that the policy defines and then know, as you mentioned, that they're going to be held accountable. Maybe there's some punishments that they can met out. Uh, but what does Let's talk about that. you know, what does an effective AI training inside a law firm look like for you?
Oh gosh, that's a great question. It will
people say that.
Yeah, it it will be practice specific. So, it has to have examples that are from their work. This is this is how I do this is my generic answer to that. I like ones that start with really simple small wins like where do I find my stuff and and help me draft this tiny inconsequential blog and really small like tests like point at a document and help me clean this up and and getting them those basics down first. Then going a little bit more technical like let's train on how you use it in the marketing content. How you develop agents that people could then use to answer questions internally. Then maybe you go externally, right?
You start low stakes and start to work your way up that ladder.
And the low stakes is because it's easier for people to conceptualize and just feel safe and learn. Uh then and then in that probably middle middle part, you're also talking about prompts, more complicated prompts and and maybe financial management stuff. Like there's a whole bunch of key performance indicator stuff that firms can learn to to do that they've never done before because they have all this data, but they don't know how to understand it. Well, AI does a pretty good job of that. Just put it in there and it will give you answers back. And then we're really getting to the legal research, the the more technical things like immigration lawyers can use some of these products that cut down on the time that it takes to evaluate information. Kind of flips it on its head where you probably get 500 pages from someone that you're supposed to evaluate and then write. But what it does is it would scan everything, put them in the buckets, and then essentially say, "These are the ones that are low on information." And so it changes the way that you do work and and those things are harder for people to digest from the beginning. I think I think they're impressive. They get excited about it, but it's harder for them to understand how to use this as a collaborative tool. And so I wouldn't want them to move into this until they understand the critical thinking you have to come at that product with and how to kind of work with it. Yeah, that's okay, but do this. The iteration and the improvement or the nope, that's wrong. I think people that argue with AI are some of the best users. That tells me that they have a really strong confidence on what they're saying. And that says that they're I'm more confident in them using it because they're not like falling for it.
Yeah. And and so that goes to verification whether you just know it as you were talking about earlier. You know the data sets and so if it's you can push back on things that you think are being said that aren't accurate. And so for you, what does a a proper verification process look like? Particularly if we're talking about something that where the stakes are higher, where you need to be citing accurate like real cases, but also based on real summaries and outcomes of those cases because I know that that can be it could be a very real case, but you're summarizing it incorrectly or applying it incorrectly to to the legal uh
right. By the way, humans do that all the
time, too. We always talk about shepherdizing, and these, this one does it better than the other one. I like, you know, Westnotes' notes are better. This is, it's not infallible. Um, I think that, well, let me say this. I went to law school at Seattle University School of Law. Nobody on the East Coast even knows what this is, but it has one of the best legal writing programs in the country. One out of two that like runs like neck and neck for the best. My teacher said at least a hundred times, read every case, analyze it carefully, brief it. These are things that are the practice of law.
And yet, uh, and we all say we do it. We all say we do it. And yet this somehow, this technology has identified that some of us are not. And I think a lot of these might just be somebody just putting, like they want a little more confidence in that string of cases. So they, they just look for a few more cases that say the same thing and they, and they just, you know, cut corners. And that's where they muck it up. And that's where they muck it up.
So, I do think it's just getting back to the old-fashioned practice of law, which is you, you do this research and you see what the text says, but then you don't take it on its face because creative lawyers are the ones that actually read those cases and go, they did not, they did not get that that note right. That's not necessarily the truth. So, that's the goal is, is to point out mistakes that the opposing counsel has made and say, "Oh, they're, they're using this citation completely wrong." So again, if you're on the other side of the table there, you can obviously be wrong. That happens. But if you're, or face, you look like a clown. Even Lexus Nexus or Vlex, they got that wrong. That, that they, they read that case. But what's important about this case is this fact. This one that they almost ignored was important.
So let me ask you this. Can you, can you, is that, can you take some AI output and then use another AI as your verification process? Is, is would that be a reasonable use case for you?
It's so funny. Everybody asks me this, and I guess this is my take. Um, I mean, my feeling that has to be absolutely not, right? Like, at least not yet. I mean, it's nice to maybe have a different tool, but you still have to, I think, manually verify, especially the things with the greatest import, right? I think there might be some uses for multiple products because sometimes they have different strong suits, right? And or you can instruct it with a different angle. It is funny to me when I, when I do, I do articles with AI very collaboratively with it, where it does some stuff and I'm like, "Yeah, but I'm going to add a whole bunch." And then it does some more, and I do some more, and it even corrects its own stuff. When, when you go to the next step, it'll be like, "Yeah, but I, this word is not as good." And I'm like, "You put that word in there, not me." So, it's not a bad thing to do.
But I also think this is the thing. My son, he's 17, and I said, "You could just use AI to answer that right now." He goes, "Mom, I only use AI when it's necessary. Do you know how much it costs us globally to use that? How much water it uses? If I, there's another way for me to reach it, I'm going to reach it that way." And I was so impressed and put in my place with what he said because he's right. I actually, uh, think it's the interaction between the lawyer and the product that is going to be the best result. So, I probably wouldn't just take another product and say, "Look at this," and tell me.
I do think, by the way, some really creative uses that are excellent is once you're done with it, say, "Where are the holes? What did I miss? What are the arguments that could be made here that I didn't make?" And and it gives you so many, and some of them are totally stupid, but maybe two of them are brilliant, right? Yeah. And and so, and the same thing, like you said, of looking at what the other people have done and just getting a different view on it. Look, that's one of the biggest things that for my uses, the biggest thing AI did for me was just have a colleague that I can bounce things off of because frankly, I'm an extrovert and I spend way too much time by myself. And this, and it was just a, it's a freeing thing to be like, "What do you think, robot?"
So, if it ever asked me to call it something weird, I'm out.
Well, but that's probably the case for so many people. Like, I'm very introverted, but it is nice to just, you don't have to actually go and find another human being. You could, you have this tool, you can bounce things off of, and like you say, it can come up with ideas that you hadn't previously thought of. Some are crap. Other ones are like, "Wow, that is really good."
Let me, let me ask you this last question on the training or like, like doc about documentation. We touched on this earlier. Do you think it's a good idea as a policy, particularly when, where things maybe for everything, but particularly where the stakes are higher? Should firms require, as you mentioned, documentation on how AI was used in a specific work product, whether it's highlighted, it's what tool was used, how it was used, so forth. Is that something you recommend as part of a policy?
If it's a use that is used on everything, maybe you don't need it completely documented, and it's just like one of the standard checklist things. Or maybe you put it in, in an agreement or an addendum, like client. I use this at ABA opinion about how to use, I think it's 512, how to use AI. It was kind of vague on what to do about clients and what to tell them. And and my theory is, hey, it doesn't hurt to tell clients because you want them to be on board with it and to be impressed with you. And so, uh, I think yes, document everything, partially because we're lawyers and we should document everything and we know how good it is to have that information, but partially just to get that system built in place for everybody and so they have confidence about what they're putting in there and they learn it along with the, this is how I use it, the loosey-goosey fun. They learn the, the structure, the verification. So, we reduce the potential errors and risks. And then I didn't say this about the, you could always have that part too in this policy. Let's, let's make sure that we talk about how we're going to talk about it with clients and how we're going to talk about it with the courts, because you may be required to tell the court how you used it. The, the 512 really says you don't have to tell clients or court, or they don't talk about how to tell the courts. So, you don't have to tell the clients if it's a standard use that everybody does. And that's good because I think, as you said, it's in everything now. Not everything, but a lot of things. It's just like you include citations in your work. You can include essentially documentation of how the work was produced if it was used with AI. It seems reasonable. I don't know that, you know, we're there yet, but but that does seem like a reasonable request, almost with many of the things that again have real import that we're working on. Okay, just let's just have a, a paper trail of what was you or whatever.
AI trail of what was used here and how did, how did we arrive at here?
Have a trail of what's used, and especially if it's any unusual use or unusual product. This is our, this is our own product that we built, is one that you definitely need to explain to clients, but also like, "Wow, that looks great." So I wouldn't even differentiate. I just tell them both. But I wouldn't, it is a balance. You don't want to get into minutia. We, we use this for writing our documents. We've used this for writing our. Yeah.
Within reason. And and so when it comes to that stuff, is there a balance that a firm really wants to try to strike between having real control over these IT tools and how they're used, but also where you allow the experimentation to find creative new ways to do cool things within the firm and then of course be transparent about them and share them with the firm so other people can do them and and maybe vet them on their own, almost like, uh, you know, kind of like research. It's like, hey, we're going to come up with a thing and then other, uh, people are going to test it.
I think it's, I think it's appropriate in some firms to have a sandbox, maybe, uh, you know, not using it on cases. We're just using it for this to experiment with it, or we are using it on a case, but we're only using it on this one case. We've discussed it with the client. The client is on board. Right. There may be clients that, look, my immigration attorneys have lots of tech folks who are these AI experts and they're doing extraordinary ability visas or H1Bs, all sorts of things, or, you know, investment visas for them. So, are they going to be on board? Heck yeah. Right. They're even creating things as we go. They go through their, they go through their immigration process and they go, "This system is so broken. Well, maybe I could help a little bit by helping the lawyers do it." So, there's been this influx of really excellent products into our, you know, practice area. It's kind of thrilling. That client may be less risk-averse than the grandmas over here.
So, the grandmas are not as like scared as people think. I will say.
Are you sure?
I'm not a grandma. I'm just saying that.
You're, you're [laughter] just setting it up for the day when that 17-year-old son of yours has a child and you do become a grandma. Hey, I've been supporting grandmas for years here. I'm not, I'm not one of those. Well, let me ask you this. How do firms handle situations where maybe the toothpaste is out of the tube here, right? The staff, they've been using all say of unapproved tools. How do, how do you rein all that in? And and and of course you mentioned earlier you could have in your policy consequences for using tools that are not approved.
Yeah. I don't, I'm not sure. So Nikki in her excellent report also talked about the data for how many have a policy that they like. It's only 9%. But I think it's about, let's see.
9%.
9% have a written and enforced policy. 10% have one, but it's informal and not consistently enforced. And then the other numbers are really low for everything.
There's just a disaster.
And that's, you know, the, the 10% they're not bad. It's just it would be better to, look, it's a living document. You can feel kind of unsure about it, but have some flexibility within the document so that you can have these experiments. This is our sandbox. These are who are going to be in it, or maybe anybody can be in it, but these are the requirements, and then do enforce it across the board. The other thing I thought interesting about the study is lawyers have a lot more access to the tools than the other staff. And and I know part of that is, well, we don't want people that don't have as much education as us to have access to this thing. But when you're,
that don't have, they should have the tools to help them.
It says it staff only have 14% have access to it. What a shame. Wait, say that one again.
Here it says, at least in law firms, it says which roles are being prioritized for AI training and resources. 14% IT technology staff. What a shame, because who could do some really amazing stuff with it, right? My husband's in IT. He's a project, you know, director of project management and stuff. They saved our state millions of dollars by being bold enough to use AI when they were getting a, a, you know, bid that it was going to be over a million dollars, and they figured it out in a week. In a week.
Yeah. I mean, that is amazing, right? I mean, a lot of the service providers that are out there, SAS products, all these tools are being either replaced or or or, you know, partially replaced by by this AI, uh, technology. But all right, let's, let's try to bring all of this together. So in terms of firm, doesn't, I mean, seems like nobody's got an AI policy. Even the 10% that got one are barely using it. So, um,
9% are happy with their 9%.
Everybody else, this is who we're talking about.
Everybody else, that's who we're talking to you guys.
All right. So for a firm that does not have an AI policy or one that they're actively using and enforcing, what is the just the first step they can take to get started?
I would tell them to open up one of those generic products and put in, in the most garbled but consistent language, "Look, I think I want it to say this, this, this, and this, and this." And do a draft. And then you're not going to like it, but it's going to give you ideas. Because the one thing about AI that's really fun, well, I've said lots of things about AI that's fun, is that it understands the shape of stuff that are super common, and it doesn't mind drafting boring things like a policy. So, use AI to do it. You'll probably need to go three or four times back to it. You'll have some more ideas, and then you have a draft, and then you float it to whomever you want to work with to help you do it. If it's just you, then even if it's not perfect, this is something lawyers are so uncomfortable with because we, we do get a little perfectionistic. It doesn't actually have to be perfect. I'd rather you just put it in place and start having the conversations and the trainings.
And you said it's a living document. This stuff is evolving fast, fast, fast. So, it has.
Join that 10%.
Join the 10% that's like, "We don't like it, but at least we have something." And then iterate on it. So, so that sounds pretty easy, right? I mean, basically, we, we need an AI policy. Let's use AI. And I'm guessing you could dump in here's all the tools we're using. Here's the concerns I have. Just, just a brain dump of everything going on around AI, and then it'll spit out something, right? And that's the great thing about AI. It really tackles the blank page incredibly, right? So, now you have something to work with and refine, get it to a point, right? I guess that you like it, and then you can just kind of go from there. But at least, all right, now you're in the 10 percentile of having something that you're relatively happy with, right?
Yeah. And it gives you ideas because you're doing that. We think out loud nicely, right? Or or even just on paper, we think well. And and you could actually put it back in there and go, "Well, what did I miss? What did I miss?" Copilot. And it'll say, have a long list. And it'll be like, "Oh my gosh, I missed everything." No, it's just you gave it a different task. That's like, "What did I miss?" And so it'll give you everything, and some of those again are dumb, and a few of them will be great, and you'll be like, "You're right. Let me put that in." Do that a few times. Get a decent policy. Float it to people. If there's a bunch of resistance, that resistance is normal. That resistance is every, if you've ever rolled out any technology, it's what happens. And you work through it. You keep talking about it. You keep sharing like that guy that you talked about who has people just hang out and chat about things. And and you keep getting that buy-in and and pushing towards. And by the way, if any of you were like, "I'm still not going to use it. I'm never going to use it." Still make the policy because that's when you need it more than ever, because if your plan is to never use AI in your firm, your people need to know that, right? In a big way. And and then, you know, they may just be using it anyway. And and then because you're not using it, you don't even have any clue how it actually works. And so you could be the leader of a firm that makes this edict, but really you're putting yourself at a serious disadvantage.
But in any case, let me ask you this. Maybe the last question we can wrap up on on this, but in terms of like, what's the biggest mistake that you see that you think firms are making when it comes to using AI right now, aside from sort of not having the right policy?
Not getting in there and trying it.
Yeah. People just being, they, they get it, but we do what we always do with new tech that we know we need, but we don't schedule time to learn it and to practice in it. And they just think they're going to get it through osmosis. And you don't get any benefit from osmosis. So, uh, my biggest mistake I see people make is just being timid about getting in and trying stuff. That's, yeah, I think that's the one. And of course, the hallucinated cases, but those are not that many. [laughter]
Yeah. I was looking at this, uh, that thing and charlatan, charlatan, but, you know, it had it taken down by country, and I'm looking at K's got this one has a few. This one has a few. And then I go like, where's the US? US had like 900. And so, um, and that, those are just the ones that that he's aware of. So, but [clears throat] yeah, you know, I was thinking about that. I was like, why does the US have so many more than other countries?
It's a big country. There's a lot more going on here.
It's a big country. It's very legal.
Yeah.
Like we, we like our law. We, we do a lot in court. But we also, we're very public. We're very published. We're just organized. Zimbabwe as organized. Do they think that it? I think it's partly a little deceptive, right?
Yeah. I mean, and we're one of the major, we got these tech companies that are huge that are creating it. Several of them here. So, I mean, maybe I'm just trying to make myself feel better, but I don't think it's just because we're all terrible lawyers. And actually, I looked at the numbers. The percentages for for lawyers versus pro se in in non-US is actually higher. The the lawyer percentage was somewhere in the 40s last time I looked at that. So, maybe we are better, a little bit, a teeny bit.
Yeah. On like a per capita basis, however you normalize that data, but yeah, I'm sure obviously. Yeah. Um, all right. Anyway, this, this has been great. Thank you. So, I'm, I'm here with Charity Anastasio. Uh, I think I pronounced it the wrong way now.
Great. No, you didn't. You did fine.
I know you said it can be Anastasio. Anastasio. So, okay. I'm here with Charity Anastasio. And we've been talking about why your firm needs an AI AI policy ASAP and how to put it all together. In terms of connecting with Charity, I'm going to, uh, share her LinkedIn in the show notes and all that, but you can also read, uh, her article on the ABA's Law Practice Today, "Why You Need an AI Policy Yesterday and How to Write It." So, I'll include a link to that. Anything else you want to add, Charity?
Feel like I, feel like I, feel like we've, we've really, we've covered it all. I do feel like we've covered it all. We, we, I, I can talk your ear off, but actually, I just will say it was really fun talking with you about this and and I, I hope folks look at this as a positive and find the silver lining in it and really, like I said, take our seat at the table of this because they shouldn't be doing it without us. They shouldn't.
Great words. Well, thank you, Charity. I, I really do appreciate you joining me today. Charity Anastasio, everybody. That's it for today. We've got lots of great shows coming up on all sorts of different practice management topics. So stay tuned, check it out, and we will see you next time. Thanks for listening to [music] the Alpha Lawyer Podcast. I'm your host, Jason Marsh, and I appreciate you joining us today. If you enjoyed [music] this episode, please subscribe, leave a review, and share it with other lawyers who might benefit. You can find more resources and insights [music] on building a thriving law firm at alphalawyer.io or the Alpha YouTube channel. Till next time, keep building the firm of your dreams so you can live your best life.