Transcription
Hello everyone, Jake Van Clee here with the amazing Katie Bakewell. I actually said Blackwell the first time I got to know her, and apparently that's a common thing that we hear a lot. Uh, so apologies for that, but Bakewell. Katie Bakewell. Now I am with NLP Logix. We're in the NLP Logix building. They've been around since 2011. Talk about being an AI before AI. Uh, Katie, I know you came up through math, right? DNA computing, time series on commodities. How does a math background change how you look at problems, and could you give a bit more detail on all that?
>> Yeah, I think the way that I look at problems really is shaped by that math background. It is not looking for a specific algorithm that is step one, step two, step three to solve a problem. It's going through and saying, what are the things that we could be doing? What might we be missing? I always love—somebody once told me that the secret to math when you get past, you know, algebra and calculus, when you're looking at proofs, the secret is knowing when to add zero and when to multiply by one, because you're making these small changes, and you know, if you add the same thing to both sides of the equation, it still equals, but you're getting a much better result. So the way that we look at problems is essentially, what are the things that we can add or change in order to solve something?
>> And almost a subtraction side of it too, right?
>> And wow, that's such an interesting concept. Then I guess the next follow-up question kind of would be, what brought you to NLP Logix? What made you want to dive into this space versus what you were doing before?
>> I hated statistics when I took it in high school. I thought it was the worst class ever, because I don't care what the probability is that somebody gets a king when there's an ostrich laying an egg. Like, that's really not important. And then, you know, we started digging into statistics for school and really loved it. It's a super interesting subject, and it's really great to sort of dig into problems and, and see, can you figure out, you know, what are the correlations that we're seeing? And got really excited about the space, and you know, had an opportunity at NLP to start here, and we had fun stats problems. I really, just at the end of the day, like solving a problem, and that is what I do every day, so it makes me happy.
>> Absolutely. And I think that's what's interesting about that, right? Is we look at statistics, we look at the history of it all the way back to the 1800s of people doing different things with ledgers and whatnot, and all the way to modern day where people are saying, you know, they're vibe coding and saying that they're doing statistics and whatnot. So take me back before this was really called AI. It was more natural language processing. It was statistics, you know, right? You were scoring debt, correct? Right. And you were looking at, you know, building health alerts, matching people to jobs. What did the work feel like then, and what were you actually building, and then how does it feel very different today, or is it different at all?
>> It is super different today. Uh, I think the one thing that hasn't changed is the technology behind it is machine learning, which is just statistics on steroids, but we use AI to sell it. It's a whole lot easier to explain AI to somebody than to go and start talking about machine learning. I think the favorite question I've ever gotten from a customer is, "Am I going to have to buy one of those machines?" And so, you know, we, we, we sell through AI, but you know, the underlying technology is still really similar. The thing that's changed is the difficulty in implementation.
>> So it used to be something where if you wanted to build a machine learning model, you were writing the code from scratch, and you were going through the whole thing building out just every single line of code. And now there's really great models. So like in computer vision, you used to have to write the entire neural network. Not as transformers, and you get to stand on the shoulder of giants, and that's fantastic. So, it makes it a little bit easier, but definitely, um, same underlying technology, just a little easier on our, our hands on the keyboard.
>> Absolutely. And you hit on two important points there. One, we'll, we'll hit on a little bit later, which is the kind of marketing of AI. The other one is right, you're standing on the backs of giants. I think it was 2017 the paper came out from Google on transformers, and you know, people started doubling down, and it wasn't really until like 2020, 2021 that we start seeing more consumer outreach process, but—
>> You had mentioned this concept of implementing, right? That's the important part.
>> So a company comes to you and says, "We want AI," right? This is obviously a big part of what you all are doing today. Walk me through what you think happened next, what you care about, and what do you really listen for in those conversations or problems?
>> So if you come to me and say, "I want AI," the first thing I want to say is, "Let's do some discovery and let's figure out what your problems are." I think it's really easy to get tech for the sake of tech. And that's why you're seeing the MIT paper came out that was 5% of projects are making it past a pilot. And you know, I think when I looked at our numbers, we were at 70% make it to production and stay there a year or longer, which is fantastic. And it's not because we have better math, you know, we don't know statistics that no one else knows. It's the planning in the beginning, going through and saying, you know, what are the, what are the actual problems that you have, and what's the impact of solving this? So rather than just saying, "Let's get tech. We want AI because it's cool." Well, let's find the AI that's going to make sense in your organization. I don't think there's anybody that doesn't need it. But it's just finding the right way to apply it. And if that AI is machine learning methods, statistical methods, analysis, or if it's actually these generative models. And like you said, there's a balance there, which there's something interesting, right? How often do you see either companies coming to you all or just in general asking for AI when they don't need it?
>> I would say 50% of the time almost. It's crazy how much. And it's not that they don't need AI, it's that they're trying to solve the wrong problem with AI. You know, I think you've got people that are coming in and you've got a problem, and man, you want to solve it, and you see AI coming in and solving all these problems for all these other people. Well, it can solve my problem, too. It's really exciting, but that's not necessarily the way that it always works. So, you know, I think a lot of the time we're figuring out sometimes it's just process change, and then sometimes it's AI that, you know, you do want to get with an org to build, and sometimes it's like, you know, really, can I just teach you a little bit about chat GPT and it's going to make your life a whole lot easier? [laughter] Well, and that's the funny part. I think, I, because I work with a couple companies here or there, and one of the common things I see is this weird obsession with finding a solution then seeing the problems you can apply it to, rather than seeing the problems and then seeing what solutions exist for it.
>> For sure. I would say I actually did a presentation that was Indiana Jones themed, because I will really commit to a bit, and I had a hat and a whip and everything. And the reason that that came about was because I felt like 2023, early 2024 was the boulder scene from Indiana Jones. Okay.
>> And you know, I'm, I'm Indie running away, and there's this boulder of "can you build me a chatbot?" It was like, "Hey, I need to forecast what my profits are going to be next year. Is the chatbot going to do that?" And you know, as people just you hear it, and I mean if you're looking at generative AI in any way and not saying, "Wow, I want to use that." You're crazy, right? Like there's amazing things that can happen, and like yeah, there's scary sides to it, but you know, if you can use it safely, like why would you not want to?
>> But that's putting the tech first, you know. Figure out your problems. And that's really the whole point of us doing discovery, why we care about doing discovery.
>> Yes. And, and that's like, I find one—I absolutely love that analogy. That is so cool. I wish I was there for that presentation. I would have loved to watch. Was it recorded? Is there anywhere we can—
>> Not recorded, but Ben did capture a picture where the whip is tracking?
>> Oh, we'll have to share that. I'm sure people would love to see that. That's actually, that's an interesting analogy, right? Is you have all these—I feel like there's a lot of data scientists and mathematicians and programmers who probably feel like you right now, where you feel like you have either leadership or you have companies or you have market pressure because you run your own company, and you just have this boulder of people saying, "I want this AI, this LLM," and you're like, "Okay, but is that really what you want?" Like maybe you want a chatbot because dialogue is a good interface, and that's great, but what you need first is, like you said, a stat, some sort of statistical model that's capturing the data and then handing it to the chatbot. That loop—
>> For sure. It's getting, getting the data in the right shape really is one of those big things. It is—if you're, if your data, you know, everybody tells me that they have the worst data and it's awful, and nobody's ever got terrible data. You know, you've got data that might be a mess, and you know, guess what? There's great new tools to get that data out. You know, we did something where it was sign-in sheets that are handwritten pieces of paper, but we're at a point where, you know, data capture works. Like, it's—I would say a quarter of my time is spent looking at data capture, just because it's an amazing tool for AI to do, you know, but it's figuring out, what does that data look like? Am I capturing the right things? And what else could I be capturing? If I caught A, B, and C, then I'd be able to have an even better outcome. And honestly, I would rather spend the time figuring that out with somebody and setting them up to capture that data than I would, "Hey, can you build this model with maybe the wrong data in it?"
>> Yeah. No. And you see what's interesting there is the question of simplicity, right? Uh, a lot of what I work and teach, and I, I have a research paper, is like, hey, you're all building these complex things and trying to use AI for everything when you could just use stuff that's existed forever. Some folders and markdown files 90% of the time can work better. So the question then, when you're making a decision, and obviously it's very dependent on context, but do you have any rules or things or patterns you look for on when you're looking at a problem you go, "Oh, this definitely needs some sort of model," or even a custom model, or a spreadsheet and a good rule will handle that? Where do you kind of draw the line?
>> So I'm going to give you my other rule first.
>> Okay.
>> So if I'm deciding whether or not this is a problem you should tackle—if you can't tell me what the ROI of doing this would be, I'm going to tell you don't do it. And honestly, like for companies, you know, there's obviously the warm and, and squishy, and we want to do nice things, and we, we have those, but there's also the side of this that is I really want to see increased revenue, decrease cost, decrease risk. If you can't tell me one of those and it's for a business, you might not be doing the right thing. So, you know, going through that, but like on the other side, those rules that you're talking about—I look at, is it a deterministic process? Is this something where I'm trying to predict something? If you're trying to predict an outcome, guess what? We're talking about machine learning.
>> Yep.
>> And the thing that kills me right now is I am going to get an agent that is going to follow these deterministic steps. Agents make non-deterministic decisions. All you're doing is adding risk to the process and not solving your problem as well.
>> Yes. 100%. Right. And it's—yes, you can create the structures. You can create some sort of error handling to handle non-deterministic processes, but at a certain point, you can't allow that risk. Right. And when you can allow that risk, that might be when you want these models.
>> For sure. But if honestly the steps are one, two, three, and it's always one, two, three, and there's no decision, all an agent's doing is costing you more money than software would.
>> Yep. Absolutely. And one thing that I do actually a lot is this kind of weird reverse. I don't know if anyone else—I've seen a couple of people doing this—where I will use some sort of agentic flow or I'll use a model to do some stuff, and if I see the model doing the same thing over again, I actually stop using the model and just build some sort of custom code that'll do that thing and hand that as a tool to the model.
>> 100%. I love that. It is so many places where the simplicity is important and the traceability is important. And if you're talking about nets, like they're great, they produce really good results on things, but they are not super explainable, and it's—if I can look at a net and go, you know, this is where what we're seeing, these are the output that we're seeing, if I can capture that using something more simple, like do the simple thing.
>> Absolutely. You've been giving absolute gold, by the way. Thank you so much. This is so great. I have also—I have a lot of a technical audience as well as business-oriented. Um, so just to give you an idea of my audience on YouTube, um, if you look at the data, um, I think the top 10% of income earners are 60% of my, um, viewership, and of those, they're usually either CTO, they're in kind of some sort of data architect. I have a lot of people who are like 60, 70 who have been in the game for a while that are kind of watching our stuff. So if you ever do want to get a little technical, you're more than welcome. However, I also understand the need to stay simple, all but just giving you that there.
>> And yeah, moving on from that, one thing that I kind of want to think about—imagine what you've done, right? You scored trillions of dollars in debt, right, in terms of your algorithms and what they need to do. When you look back at being able to accomplish that type of data processing, that level of success in algorithms, how much of it was this AI in these models, or how much of it was just getting data in the right shape?
>> Um, I would say it's three things. So the models piece, you know, going through and building the model and doing the feature engineering, like that is definitely an important step. Getting the right data—and you know, for the debt scoring example, I love it because you're messing with people's finances at the end of the day. It is really important, you know, from the debt collector standpoint, they don't want to call you if you're broke, and if you're broke, you don't want the debt collector calling you. And so, you know, it's really important to do the right thing there. But it's also really important to take the time to make sure that's non-biased, to go through with the team and sort of figure out like what data can we use, what can't we use, what would be helpful. So getting that in shape is probably a third. And then that last third really is working with the customer, you know. At the end of the day, um, I really love an example, um, in debt collection actually. Somebody worked with a Silicon Valley company to build out a model, and they shipped their data over to them, and they said, "You want us to explain the data?" And they were like, "No, data is data, just send it over, throw it over the fence." And then this company spends, I think it was 12 weeks, and then they come back and they've got seven neural nets that are being ensembled by an eighth neural net. It takes 18 hours to process a file. And one of the things they found was people who had both letters in the state capitalized paid more frequently. So they're like, "Oh, you know, this is, this is a win." Like, and what it was is the data that they received, it was both capital from one customer who did pre-charge off debt. So, this is accounts that haven't even gone into collections yet. So, of course, they pay more frequently, but the signal is not what you think it is. It's taking that time and getting your hands dirty and looking at the data. And if you don't, you're going to miss it. You've got to get that tribal knowledge. I'm never going to be a debt collection expert. I'm never going to be a healthcare expert, but I like the AI and I like having a customer that wants to be a partner. They let me get nerdy with them. Like, there's nothing better than listening to somebody talk about the thing they love. I don't care what it is. I watched somebody determine a ping on a submarine for 20 minutes 'cause he was so excited about it. Like, just talking to excited people is fun.
>> Yes. No, absolutely. And well, and that's where we find in a weird way some of the greatest ROI in business comes from excitement, comes from this passion, and having people who are passionate about it. And I, I can immediately see your passion on that. What's, what's so funny is you mentioned this issue of neural nets, uh, and the signal within them, the bias in the data. That's before we've diluted that with even higher levels of transformer models and the bias that gets embedded in those. What's some of your views on bias, and how do you, how do you personally try to mitigate it within data?
>> So I am super, super proud of the group that we do the HR scoring with. It's a group called Opley, and we did a presentation with them, uh, about this, and it is important in hiring over anything else. Again, you're messing with somebody's livelihood. If you're making hiring decisions, they cannot be biased.
>> And you know, I was lucky to go to a conference and the CTO from John Snow Labs is there talking about a new package that they put together called Lang Test. And Lang Test is a package that takes natural language data and replaces things. So let's say I had Jack Johnson from Harvard University. If I replace that with Jamal Johnson from Howard University, I better not see a change in the likelihood that this person gets a job.
>> So we can go through and we can use that framework to create synthetic data and understand what are the changes that are happening. You know, it's easier to spot in tabular data processes, but when we're looking at something that's language based, like this tool will actually let us go and start doing bias testing. And this group that's finding, you know, they're finding people jobs. We need to make sure that we're not doing anything crazy there. Like, it's super important to make sure the bias is, is excluded in any way you can.
>> Absolutely. And what was the name of that again?
>> It's Lang Test.
>> Lang Test. I'd be very interested in that. So, um, background on me—I, I, we haven't talked much, uh, before this, but my master's thesis was on using psychometric scales on language models to basically figure out, and I would do it at scale, not like, hey, copy and paste it into chat GPT, I mean sending thousands of these to it and see if there ended up being a pattern. Um, and we did it across different psychometric scales, and then what we started doing is sending that—not, "Hey, I am this person asking you to do this," you are this person, so you are a Republican and you are a liberal, you are a, uh, you know, Labor party in the UK. And then we would have them take psychological, uh, personality tests for politics and see how Brock, Claude, Gemini would respond. And you end up finding very, very deterministic bias within probabilistic systems, which is absolutely wild to me. And the reason I'm interested in Lang Test is it sounds like there's a really good layer I could work with on that, just in that general. But I, I guess then this is a little off script, but what do you see dangers-wise for people who don't take the approach that NLP Logix does? Like we see a lot of companies just throwing it at the wall and going for it. And yes, I know speed and business is important, but not having that deep technical knowledge despite everyone saying that it's not useful anymore. Um, what do you think that means for them? What do you, within reason of course—
>> There's two pieces. One is if you are following regulations—it's not just the liberal states, it's not just the Republican states. Texas and New York, Colorado leading the charge in how are we doing this? We are going to see regulation come in with AI, and taking the time to be careful right now and do the right thing right now is going to stop this from, from failing in the future. Honestly, I don't want a lawsuit. I don't want any of my clients to have a lawsuit. Like, it is super important to me that we are doing everything we can now to protect against that, rather than, you know, "Oh, just, uh, ask forgiveness," right? So I, I don't want to be in that boat. So it's really important to, to take that time. And again, you know, it's sort of in the same way—like you go and you do the bias testing, versus you go and you do the discovery. If you don't do the discovery, you're going to build something that doesn't have ROI and it will fail.
>> Yes. I actually I operate by one simple rule: in a world full of cheap answers, questions become valuable.
>> Love that. Love [clears throat] that so much.
>> Well, because it becomes this thing where, um, I have this video that, uh, it basically was me describing what an AI expert is. And I go through this long thing of like, "Oh, are you a machine learning expert? Are you statistical analysts? Do you do vision models?" Right? There's experts in those fields. "Are you in business?" Right? "Do you understand how AI should be applied to business? Do you understand the levels of it? Do you, are you of the history of AI? You go into the 1950s or with Ada Lovelace in the 1830s and her arguments, are you a historical expert?" There's so many versions of experts, but all of them have one common theme. They ask the right questions. And I think this discovery phase is exactly that. It's not just to discover what they're doing. It's to ask, "What do we need?"
>> Right.
>> Right. And I don't know if people do that enough anymore, especially in this AI space.
>> I 100% agree. It is something—it's everything can move so fast, and you know, it's, it is amazing and terrifying to watch what you can vibe code. Like you have an entire application, and it's the first time developers move faster than product. Right? You know, you see developers that are able to, to move faster than anyone else in the org. But you know, in the same way that you have this with developers, I will tell you, like, big pet peeve is if you took two minutes to write down 10 words and submit them to chat GPT. It generated me 30 pages, and now you want me to go and read those 30 pages. [snorts] We're not best friends right now.
>> And it's the same with code. If you vibe coded this entire thing, I mean, great. It is an amazing foundation, and like, holy cow, I don't want to sit right in the hands keyboard. Like, I don't want to do that either. I totally get it.
>> But it's not an excuse to stop reading, to stop learning. Like, you have to know. And when that thing breaks, I don't want to trust Cloud to go and fix it or chat GPT to go and fix it. You better know how that code works, right? It's—
>> It's—I think, you know, sometimes I'm seeing this as an excuse to not understand, and you know, I don't—lifelong learner, like you know, I, I want people to learn, like use these tools to help you learn.
>> Well, and I, I got—it was so funny. I got questioned by my friend the other day, who's very brilliant and pushes a massive amount of very good code. He's been programming for a decade before AI. So, I know he's a good programmer, but he's like, he's like, "Yeah, I usually just get summaries of these papers." But I was reading a PDF on my favorite little notebook I have here, and it was a 1962 "Augmenting Human Intellect" by, um, Douglas Englebart, and he's the one who did a lot of work with the mouse and the keyboard and getting that all together. And I'm reading through it, and he's like, "Why are you spending this time?" He's like, "The human is a massive compute computer. We are still powerful. Yes, the AI can output so much." Some of these, these models are extremely impressive, but the connections that I'm able to make are so efficient. Sometimes I'll read a sentence and I'll connect seven other little papers that I had read in the past, like I should do something there. Then I can use the AI to augment that idea, right? But I would have never gotten it. And I, I mean, I've tried many times, and it's not to say that if someone did it once and it did it the right way that Claude or Gemini or any of these things wouldn't be able to eventually come to that conclusion, but the idea is I had a cup of coffee and a pair of chips and I was able to do it, versus spending that money on the compute layer bouncing through it. The human in the compute layer, I think, is something people ignore, and I really wanted your opinion on this. And what I mean by that is people are spending a lot of money on API fees, all of these kind of computing costs, or they're rejecting them entirely. To me, there's this middle ground, which is the same way that the human was in control of the computer, and we automated processes—we didn't get rid of accountants, we gave them Excel; we didn't get rid of salespeople, we gave them the internet. What does it look like to, not get rid of what we're doing today, be that programming or data, but give them this tool in your head?
>> I mean, I think it's incredible. Like, you know, [clears throat] I love that you're going back to, to old papers. Um, you know, we're solving a problem right now that is looking at behavioral patterns. And so, it's steps that somebody is taking in a process. And, you know, everybody takes them a little bit of a different way. And essentially, this goes back to the DNA stuff, where we're looking at, you know, essentially the steps are letters and you're making a word. I, I want to know how much did you misspell to see how far are you deviant from normal pathways, and that's all automata based, right? Like this isn't about the stuff that is happening today. It's all like dig into this really, really old stuff, which is super, super cool. But you know, I think it's, it's this space where everybody is, everybody's excited, everybody wants to do stuff, and you know, we're losing out on—you need the human in the loop just generally. Like, yet for accuracy sake, like please keep a human in the loop. Please don't get rid of humans. But you know, when you give somebody an AI tool, you're not—it's never going to be perfect, and you're still going to need that human to do things, but they're doing things that aren't miserable. Like I talk to people that are doing data entry.
>> And you know, you can capture anything off a page. Like I am so excited about VLMs in how they capture data off a page, because there are things that we couldn't capture before that now—that's table stakes, and that world is so exciting to me, because you know, you've got people that are spending billions on on BPO, but like, that job is miserable. When I talk to, you know, companies that have data entry folks, the throughput on that team—like they get real fast, and then they leave, and then you've got somebody that comes in and is slow, and it's, you know, how do I get people out of this miserable piece?
>> It's not that we're all gonna lose our jobs and never have anything to do. We're gonna have better jobs. Like, I don't want to till a field either. You know, [laughter]
>> This is such a good point, too, is I, to our detriment and to our benefit, I believe there's no limit to human desire and abstraction. And what I mean by that is you could automate every job on Earth. We're going to find a new place to add value.
>> Yeah, which, you know, leads to a consumption argument and I'm here to have that. But there's the other side of this. We end up solving problems at massive rates that we never were able to solve before, in ways—I mean, some of the jobs that I'm doing today that I have done didn't exist 100 years ago. And I love them. I absolutely adore some of the work that I get to do. And what I read, and, and this is why I like going back into the old papers, is you get to see the patterns that never go away. The same pattern that existed in the 1960s or the 70s, be that with how human problems are moving, how technology is moving, you can end up finding that today. And I think that's really why I like going back. I mean, a lot of my research papers are based on stuff from like 1972, like Unix methodology, and it's like yes, I'm applying it in a new way, but it's still the same stuff. And it dove into this concept of best, and I'd love your opinion on this.
>> Yeah.
>> People are worried about AI taking over everything, all this aspect, but my argument is—you can, and maybe I'm wrong here—models are trained on a reward function, right? Some sort of thing. This is the idea of best.
>> Y.
>> Best becomes relative at certain levels of expertise. What is the best food? What is the best politician? Even in programming, if I said, hey, uh, I want you to build me a calculator—the people at NASA would write it in Fortran, the people at, you know, KPMG would hire it off to someone to write it in Python. You might choose it a little different language. The output is the same, but the underlying opinions are different. And so, can AI—or do you see a world where AI is capturing that, or is that really the bastion of human thought? Is that taste, judgment, best?
>> I—that is such a good question. I—
>> Take your time to answer.
>> No, I love the idea that you're talking about, like where it really is—you're gonna get different opinions, and best doesn't necessarily mean the same thing. And we use this argument all the time of, you know, if you ask me, "I would like to buy a car." Um, I'm going to tell you to buy a Pagani Zonda because it's super cool, man. Pagani, go get you a $5 million car. And then you're going to say, "I can't spend $5 million on a car." And I'm going to say, "All right, have you considered I found this Toyota Echo from 1998 on Craigslist? It's a hundred bucks." Like, "Well, no, that's not reliable enough for me." Okay. And it's figuring out your need. And again, it goes back to the discovery piece. It goes back to figuring out like what is your actual need, because you're gonna end up buying a Pagani Zonda that you—like, honestly, take it for me. I bought myself a little fast car, and I drive about four miles to work, and I will say the happiest time in my morning is when I floor it on the on-ramp, but that's the only time I ever use it. Like I bought this car that I really didn't need, and it's like, was that the best choice? Probably not. And I, you know, it's the same thing with, with the models—let's figure out like what do you actually need, right? Do you need the Pagani or, you know, what is best for you?
>> Such a perfect—
>> Yeah, I love SLMs. I like the small language. Like, I love going to the side where it is—it doesn't all have to go through an LLM. The LLMs are amazing and they do incredible things, and it's never—saying get rid of them because if you take them away, I will quit today. But if you just look for, you know, LLMs and that's the only solution, like, man, you're missing out on some really good Toyota Corollas out there.
>> Yeah. Yes. No, 100%. And I, I, I will—I love that analogy. I've never thought about it, because for me, right, I, I really want to—one thing I'm saving up for is the Audi RS7. Um, which is just, it's this beautiful four-door sedan. You wouldn't ever look at it, but it makes 620 horsepower to the ground, right? I want it because it has the four doors. I can put people in the back. It's got storage, but it can beat a Ferrari, uh, some of the older Ferrari models. That's kind of cool in my head, right? And that's my version of best, right?
>> And oh my gosh, that's—I can't believe I've never thought of that allegory before. The car industry is a perfect example of this. What your version of best is very different. Some people—okay, yeah, Ferrari won some sort of formula race or Porsche did. Well, I, I want a big old truck that can go out on the, on, you know, hit a mud and go to mud holes. Oh, okay. The discovery is always the thing, because the question I guess then is, what is your version of best, right? Can the models always do that? Maybe, but they're going to be biased towards whatever they've been trained on as best, right? And it's this self-feeding loop where if everyone's asking the models what's best, everyone comes to the same idea of best, which leaves companies like you and others who are asking your own questions a huge layer to hit value on. And I think that's where we're going to be for the next decade.
>> I totally, totally agree. I think it's looking at discordance, right? It's what—in a problem where there's very little discordance, that's somewhere where yeah, great. Use the thing off the shelf. You know, if I look at a, a piece of paper and say, "What's that word say?" Probably going to get the same answer from almost everybody. And great, use an out of the box tool. Go use one of the LLMs, go use a VLM, like it's going to do a great job. If there's a differentiator for your company, if there is, you know, something where the way that we look at it isn't quite the same as everyone else, that's where you need custom. You got to figure out like what makes sense for us. And I, I love it.
>> We're learning from each other today.
>> I know. Well, and that's the whole point of these, right? Is, and this is a perfect example. Podcasts or questionnaires or interviews in the past would focus on a very specific set of questions and for a very specific goal. And while we obviously have that, we want to get ideas out there. At the end of the day, it's what happens that you didn't expect. What are the conversations that are going to happen? And I, I think one of the most powerful things that humans can do is take action against the data. Right? All the data says do this. Well, I'm going to do this instead.
>> Right?
>> For some reason, that combination of being able to look at the data informs a non-data-based action. And I think you need both. I don't think you could ever live without one or the other, right? Like in no way am I saying do that all the time. But I see this like this action, and, and maybe that's the thing that makes it—it's not the taste, it's not the judgment, it's not—I think everyone's talking about that about what's the last human. It's the idea that we don't need data to make a decision. These models do. And that's an interesting—I don't know, what, what are your thoughts on that?
>> Well, I think you've got the data in your head, right? Like you know, it's—you're, you're making a model when you decided how long did it take to get here. You had to make a prediction in your head, and you've got your own little model in your head that says, you know, for me it's Jacksonville. Everything's 30 minutes away. I'm going to set 30 minutes for no matter where I have to go, because things are 30 minutes apart in Jacksonville.
>> And that's probably a pretty, you know, shitty model. But [laughter] at the end of the day, like you, you're—we make models in our heads all the time. True. But the thing that I love—there was an argument, I think it was the CEO of Hugging Face or the founder, somebody from Hugging Face made an argument that was whether or not AI can make new. And it's this question of—Einstein isn't Einstein because he knew everything. Einstein is Einstein because he thought of things that hadn't been thought of before. And an LLM isn't going to do that. Not right now. We don't have AGI. It's—it knows what it knows because it has been trained on this set of things, and without that human thought to be able to say look outside the box, it's not going to do it. AGI, right? Artificial general intelligence, general intelligence, which I think is a heated topic, right? I spent a lot of my masters looking at psychometrics. And a big part of that is what does it mean to be intelligent? And you just mentioned Albert Einstein. I think his definition was the measure of intelligence is your ability to change, right? Your ability to change from prior assumptions, which is I think the huge argument that a lot of people make about current AI is once it's there, it's there, right? Can't change its backup assumptions. And I think there's ways to solve that mathematically to an extent, but is there really? What is intelligence to you, I guess, is the good question.
>> Oh.
>> This is a hard one. You don't have to. There's no correct answer.
>> It's such a good question. You know, so like one of my favorite quotes about AI—whenever I do a presentation, I try and give you both sides of the coin—where Sam Altman saying like, "This is going to eclipse everything that we've ever done. It's more important—" no, I think it was Sun that said it was more important than electricity, which given that electricity runs the AI, I, I kind of think it's probably not, but like that, that's one side. And then you've got Yann LeCun on the other side that said, "If we're going to get to human intelligence, we might want to get to dog intelligence first."
>> And you know, okay, I'm going to go, I'm going to go back to my school days. And I am a really bad student. I do not do homework. Like, I'm just not that interested in it. So every test is a brand new test for me, right? Like it is just a matter of, of can you learn? And there were two sort of schools of thought in, in math because you're writing proofs, right? Like I'm trying to show does this thing work, yes or no, and make an argument in your head. And there was a group of students, and they were fantastic students, and they would for every type of proof learn every example they could find. Like for going into a test, they have got 400 proofs memorized, and as long as the questions are one of those 400 proofs, then they're getting 100% on that test. And maybe I'm biased because you know I am very much not that way. For me, that's not intelligence, um. That is the ability to recite information that you receive versus critical thinking, and you're just training a model at that point. Right? And it's like, it's really great—you know, if you look at the models, like they really can answer any question you ask them, not always right, but you know, for the most part it is. But like when you get into critical thinking and like let's start thinking about new—this is a problem I've never seen before, how am I going to solve it? That's where I think we have the human advantage right now.
>> The ability to think of new—what was, I can't remember who said this, but it was, um, all of the AI and all of the humans were lined up in war together, and all of the AI had brought swords and crossbows because statistically that is what most wars had been won with.
>> Sure.
>> Right. And it's this idea of new isn't, is, isn't just simply how do I make something new. It's how do I create something that breaks away from what everyone else was taught? How can I put things together in a way to do the same thing but with completely different technology or, or methods, right? Like what always blows me away is the electric car was before the gas car, right? But the reason we didn't use the electric car is because it wouldn't last long, couldn't carry as much. And then we discovered new ways to create electricity and movement. Now we're switching back again. It's this idea of the ideas and the technology has been around for hundreds if not thousands of years. In some cases, when someone was running a local model on a 1994 computer, right? And they just hooked up a bit extra memory. Obviously, it ran extremely slow and it took a day to put a sentence out. But our technology had the capability of doing what we're doing today with these models. We just didn't understand how to organize it. Yeah, Eliza. I always ask people like when I go and do like a big group presentation, here is, you know, a screenshot from Eliza, because there's people that have Eliza emulators right now. And you can go and say, "Okay, this is the first chatbot. What year is it from?" And the answers I get are 2000, maybe 1990s, and it's 1966. We've got chatbots from 1966. It was terrible. Like, trust me, go try play with an Eliza emulator. Like, it's not good. But it's a chatbot and it worked, and it is from 1966, and it's not an LLM or anything like that. But like these aren't new ideas. They're not new concepts. We are improving the skill on concepts that have been around forever.
>> Right. Well, it's again that's again why I'm reading these papers. It's an untainted view on the way to solve the problems we're solving today.
>> Right.
>> Right. It gives you this idea of someone else who didn't have 70 years—like I have 70 years of patterns that have been taught to me.
>> Yeah.
>> They didn't. And so they're looking at tech and the way to do things. Like in that 1962 paper, he is talking about what it looks like to collaborate, and he uses a fictional scenario, right? I think I have the paper here. Oh, I remember it. He's like, "Okay, this is going to sound wild." Obviously, he does much more academic than this, but he's like, "This sounds crazy. I'm going to use a fictional scenario here, but imagine this. Imagine someone sitting at a desk and in front of them is a large three-foot screen, and they are—"
Using it as a tool to help do architecture design. And he describes the whole thing like what they click, how they he didn't even call it a click, he called it a movement because the mouse hadn't been invented yet. Uh, which I think is just so cool because he's the one who invented it. So he's like writing about it before it exists. And it's like that idea. We're at that stage right now with a lot of this tech.
And so what's interesting to me is to come to the people who have been in the field looking at the pattern. Someone like yourself. What is some of the stuff that you all are working on that you see and are building around? I know you all are working on some interesting things here or there, but I think we had chatted about too beforehand that you think is a good program that people have been diving into.
>> So Cap Navigator is data extraction. It is, yeah, like >> I can talk about that all day. And then Discovery is just we just call it Discovery.
>> Okay. So what I really want to know is you've told me that some of these kind of ways you want to solve problems, the way you combine things together in what you're building. I think you had mentioned Cap Navigator and then, of course, your Discovery process. Someone watches this video who runs a team somewhere and they have a budget and pressure to do AI. What does it look like talking to you going through that? And what's different about them going through that process three Mondays from now?
I don't know that there's a ton different between three Mondays from now. I, because honestly, it's answering human questions. >> And I love that. You know, we could we could put together a survey and do a lot of discovery, but it is getting in a room and getting people comfortable to talk. And if you don't take that time, like there are human connections. And, you know, it's funny, when I started at NLP, uh, I told Ted I would work here if and only if, because as a mathematician, you kind of, if and only if, um, I work here if they never made me speak to anybody. Put me in a corner with my computer and my numbers and leave me alone. And, you know, Ted the next day, he's like, I need you on a sales call. So, you know, small company, you're going to kind of do that. But you, you go through this process and it's, you know, how do you make a human connection? And I, you know, I wish that I would say like, oh, my greatest skill is like, I'm the world's best programmer. I know more machine learning than anybody else. And I don't think that's what separates us. I think it is that taking time to go and listen to somebody's problems >> and figure out how do I get you to talk to me about your problems because people are guarded. I mean, honestly, if you watch the news, either AI is going to be the best thing ever, [snorts] and we're all going to just live on a beach somewhere, or we're all going to be homeless, it's taking our jobs, everything's awful. And I'd like to think there's a middle ground >> and it's not scary. You know, I you have to build trust. And, you know, doing discovery that I, I've tried Miro and it's okay. Um, you know, you can you can do it online, but honestly, getting in a room with somebody in a pile of sticky notes >> and let's just talk about your problems. You know, it, if you've ever done therapy, man, it is relaxing. You come out of it and you're like, "Wow, that felt really good." And it's the like, I I like to think that we're sort of doing business therapy in that.
>> And, you know, the Cap Navigator thing, I'm super excited about because this is, you know, we we have a background in document processing. um 2016, I think we were doing a ton, got three patents for the company, four patents for the company on document processing, on, uh, you know, how do you work with blurry images? How do you do like different types of processing for getting capturing data off the page? And that was months to get things off a hard-coded form because you've got the form, but when it gets scanned in, maybe it's off at a slight angle. So, you've got to sort of resize and everything like that. And then how do you do bounding boxes? And I mean, it was incredible technology. It was running millions of docs a day. Like, this is fantastic to get information off these forms.
>> Absolutely.
>> But if you said, I need a new form. Okay, we'll see you in a couple months. And, you know, we built this tool using BLMs or LLMs and OCR that takes documents and captures the information that you need out of it. And >> adding a new form is going and saying, "Here's the schema of information that I need. Here's some descriptions that I can provide for each of the things that I need." And it's going to do it. It's not months of work. It's five minutes of work to put a schema together. And like that is radical and game-changing. Wow.
>> You know, really like huge huge amounts of of effort like there's just things that you know I wish I could say I'm a forward thinker and I knew all this was coming but, you know, I if you would have told me a year ago that it was possible I'd have said no.
>> Wow.
>> You know, I think the other one, like this isn't us obviously, but, you know, Meta's SAM 3 [clears throat] >> like that we were halfway through a project. We're doing a project um with Hubs Institute for, you know, we can occasionally get to do things where the ROI is warm and fuzzy and not, you know, dollars coming in. And this is identifying dolphins in the Indian River Lagoon.
>> Whoa.
>> And so dolphins, all their fingerprints are unique or their fin unique. So you can identify dolphins by this >> and we were no idea.
>> Yeah. Yeah. And we're halfway through labeling these and, you know, bless our team. Um, and Ted's niece, we actually got her involved. She loves dolphins. So, she was psyched to to be part of this. And we were labeling dolphin fins by putting dots around them. So, to create the segmentation mask, how do you do that? And it was ours because dolphin fins are unique. The thing you're trying to capture is the little wavy bit. So, you have to capture all of that in the mask. And then SAM 3 came out and you could say find me dorsal fins and you had the masks. You had to go edit them. They're not perfect, but like >> but it's close enough.
>> Yeah. Everybody talks about LLMs being game-changing and they are and they're incredible. But like that to me is one of the biggest breakthroughs we've had in the last couple years.
>> See, and that's what's always why I'm so excited to interview people like you because you've been in the game. You're a mathematician. You know the statistics. You know the the hype and the BS. You know what to separate. Finding what excites you in this space is always exciting to me because it shows that's where crazy real innovation is and seeing how you all are adapting to that is is is I mean, it must be mind-blowing to watch in real time. I mean, you're again, I said at the beginning of the video, 2011, you're almost two decades ahead of most of the game in terms of like what it comes to AI this companies in general. And then you coming in and being able to be a part of that and then I had no idea you were working on on that type of thing. I think it had mentioned to me, but I didn't have time to look into it. That's wild.
>> It's super cool. And I mean, it's it's super cool from two sides, you know. It's cool from the AI side, but from, you know, a person who knows nothing about business and, you know, I I'm a little bit of a not so much rule follower. So, like the entrepreneurial spirit is something I love. And to be able to watch this company go from, you know, I always joke with people that, you know, Ben was our first employee, I was our second. Um, you know, Ben and I go way back before this. We did, you know, our math degrees together and we shared a monitor. Like we took turns with who got the extra monitor because it was, you know, it was hard and you're trying to sell machine learning to people who are like, can I buy one of those machines? So it it was, you know, watching this company go from, you know, five of us sitting in an office to 160 people like that. Honestly, it's so fun. And, you know, AI is the reason that happened. Like caring about AI and honestly caring about our customers is is the reason we got here.
>> Super cool.
>> No, I I agree entirely and I I think that's a good point to end on here, which is there is ROI in so many things. There's ROI in being able to build a better business, find out your financials, build better models. But as we're entering the AI stage, as everyone's automating everything, as everyone gets refinement easily, I think that idea of trust, that idea of authenticity on what best is our ability to make decisions outside of the data is really where the ROI is going to be. And I think you all are doing an amazing job sitting in that space. I I think it's very exciting to see. Uh, I'm I'm just happy that you all have invited me in to talk and chat and see what we can build in in fun little ways together. And I uh I can honestly say this has been an amazing conversation.
>> Yeah, same for me. Super exciting. I thought this was going to be, hey, five minutes, answer some questions, then you're done. But like this has been fantastic.
>> I'm so glad, Katie. I uh I wish you the best of luck. And to anyone out there um that would like to get in contact with NLP Logix, I'm sure I'll have a whole bunch of stuff in the descriptions wherever you're seeing this from. Um, and hopefully if uh Katie is willing to, there might be opportunities to to reach out their way and and chat with them. So,
>> Please do. I'm never going to turn down a nerdy conversation.
>> Please. And well, we've got plenty of people watching who could chat your brain off, that's for sure. So, thank you so much.