Transcription
Hello everyone. Another wonderful NLP Logix interview today. Uh, a lovely talk. I'm here with Matt Bers, which we've actually chatted before. Many of you have seen the video. You've asked for more. Uh, our last conversation got unfortunately cut off by some Microsoft issues. I think if they were building on top of your platform, maybe they would have a little less issues. But, you know, we'll see how that goes.
But, uh, kind of want to dive back maybe into some similar questions that we had chatted about last time. But, you know, just kind of the history because I think it's very rare to have a company in AI that has been around for as long as you have and studying specifically what you do, right? NLP Loggix 2011, right? Natural language processing was the name before. Now everyone's just calling everything AI. What did you see back then that made you realize this is what I want to spend the time in? This is where I think we should focus our effort. What was your bet?
Specifically with uh natural language was the, and this isn't like a deep insight or anything, but you looked at how data was stored at that time. I know a lot of relational data and a lot of times in an enterprise you would have kind of these catchall fields like comments or notes and then kind of this free form text and it was, you know, it was kind of the challenge of like, um, gosh, we're storing all this data and anybody who was in software at that time, you probably did the same thing, which is you just ignored those fields. No one ever used the notes and the comments fields. You just, it was just, it was data that was collected and just constantly ignored.
And then you think about, and that's just in like, a kind of forcing that that unstructured data into the structured format. You got to put it someplace. But then outside of that, just, you know, most of the data in the world is, you know, semi-structured or unstructured and it's like, you just kind of from an analytic standpoint, is just this feeling of, um, I mean, we shouldn't just ignore all this data, like what can you do with it? And it was interesting because I, I feel like the, the part, like I remember one of the early uh projects we were looking at was, it was actually analyzing some of the, some customer feedback for, for our customers. So it was, it was one of our customers and they had a process where they collected a tremendous amount of feedback from their customers and it was, uh, you know, basically an ignored field and we were helping them with customer churn and they were really trying to get ahead of customer churn, predict it, how to cure it, how to, how to basically, you know, prevent it from happening and doing something at the time as simple as, you know, putting a sentiment analysis model like on top of that free text field and relating just the tone of the conversation to kind of the, the ultimate outcomes of, you know, a lost customer. And I was like, "Okay, it looks like through that one lens, we just bought the customer like three weeks more time." Like an earlier indicator, another signal that they could use to get in front of sort of the business outcome they were looking to change. So like that's really simple, but you know, you could kind of see, you know, okay, that's interesting. And you know, how do you build on that? And and as models got more complicated and or, um, you know, they, they could do more things, it all just kind of stacks and then, you know, lo and behold 15 years later, yeah, we're at, we're at where we at now.
Yeah. And you say simple, but in my head, like having tried to build my own computer or my own models on my computer and go through all of that, like even sentiment analysis is like a really challenging computer science project if you've never dove into it before. And to be doing it that early on.
Right? Yes, sure. Now we can throw a transformer model at it and it'll give a sentiment very easily. I can just throw a junction of text, but to get to that point, we had to build off the back of giants. And it's in my opinion, it's still magic in so many ways. However, there's a lot of hype around that magic unfortunately as well, right? I, I always say it's underhyped and overhyped at the same time.
Yeah. Where do you think people are overhyping artificial intelligence, just the term AI today? And what is different than previous times you've seen hype around this technology? Because you've been through, you know, kind of more than one iteration, I think, of it than than most people have.
Yeah. So where, where is it overhyped right now? Is that the first question?
And I agree with what you said. I think I've said that statement a number of times. It's like, it's weird for something to be overhyped and underhyped at the same time. So in, I, I, I 100% agree with that. Where are cases where I think it is overhyped? I mean, I think in general, the, the case where the, the mindset is, um, turning that, that there will be that, that that we will be turning over tremendous amounts of work that will be done by agents that are totally autonomous. And, and even as I say that, it's hard to say that because I believe there actually will be a, a tremendous amount of work, but it's like, it doesn't mean all work and to me, it doesn't mean, um, that there isn't more work to be done after that in other cases. So it's, I guess that's kind of a weird answer because,
Um, where, where is it overhyped? I guess maybe, maybe where we are right now today, maybe there is some overhyping of what today's capabilities truly are.
Uh, and maybe that's the case of just, you know, kind of being lazy and, and, you know, having AI do everything for you when it, it maybe shouldn't be. Um, but I, you know, as, as I think about 12 months and two years and three years and four years from now, um, there's going to be more and more capabilities and it's going to be able to do more and more things. So I guess that would put me, that would put me more in the side of, um, I just, I think there's a lot, I think that there's a lot more capability in the technology we have today. Like if we, I, if, if we could not advance any of the foundation models any further anywhere, I think we would probably still have five to 10 years of building to really take advantage of the capabilities that are like right at our fingertips.
No, and I agree so wholeheartedly with that. Actually, since our last conversation, I've been diving into, um, actually older papers from the 60s to the 80s, um, kind of to just really get this untainted view of technology then, right? Looking at the same arguments they were making and I, I ended up diving into, um, Douglas Engel who's, uh, kind of a visionary around, um, keyboards and mouses, but he wrote a paper called Augmenting Human Intellect and the whole concept was we're going to automate processes, not intelligence and it's like, yes, you're automating what we consider intelligence today, but a lot of the time it's just to get to that next job, that next thing and he even, the 1960s said like, look, our problems are going increase as we automate stuff, not decrease, even though we're solving more problems every day because we're able to see more problems that we maybe didn't notice.
One thing that I absolutely loved is another paper I came along and that's when we introduced like computers in the 1980s like Apple and all these things, productivity actually decreased slightly because we were trying to figure out how to use them and it wasn't until the 1990s that we actually figured out our workflows. Do you believe that there is a specific way to kind of redesign workflows around AI? Or even further, I guess this is a better question. People are buying a tool.
Does that equal adoption? Does that equal use case? Do licenses equal capabilities?
Yeah, it, you know, for us, that the easy answer to that second question is, is, is no. Is that, you know, provisioning licenses is probably, um, you know, it's, it's something, it can be obviously done very easily in the enterprise, but the, you know, how you take and leverage that license to be more efficient or drive better outcomes in your role or your job function, like that is, you know, that is not necessarily something that's just gonna happen. Um, you know, a couple of things there, I think that are interesting is, um, you know, the, you know, how do you scale AI adoption, kind of across an organization? It's like, you know, there, there is a certain amount of, hey, everyone can probably, you know, at some level, we're all, we're all doing some of the same types of work throughout our job. And yeah, you know, there's a certain amount of training and adoption that you can do to get everybody better at some of those core blocking and tackling things. But then when it comes to role specific, um, I think it's, I think you really need to work hard to connect. Here is the tool. Here are the places in your job that, that you can use it. Actually, here are places where you really shouldn't use it. Here are some, here are some, you know, guard rails or some best practices. And then while you're doing all that, I think you really got to keep in mind, especially, you know, these last, you know, I don't know, three years, but really the last six, 12 months, it's like,
While you're doing all that, the tools are changing dramatically. Yes. You know, there's new surface area, there's new best practices, there's new, um, you know, I won't say like the, the voodoo magic, but I mean, oh, you must, if you're prompting, like you must be doing this and do this and it's like, all that is advancing the model, the capabilities are improving in the models, the harnesses are getting more nuanced and, you know, and so it's, it's a, it's a constant right now, at least it feels like it's kind of this constant, oh, you can do that, you know, let me change this to like, how do I, how do I take that, you know, piece of functionality and embed it into my team or my department. Um, it's all happening very fast. Yeah. So, it's probably the furthest thing that the, you know, if you think taking a license and, you know, giving it to somebody is going to just immediately change how things work. I think if you looked at the usage and the adoption of the teams, like from the, the data that the tools emit, you will very clearly see that what you would probably expect, which is, you're going to have a, you're going to have a small group of users that are really super users and then there is a, there's a long tail,
Of adoption across teams, even departments where everyone is is basically doing the same kind of work, drastic differences in how tools are being used today. Absolutely. And I, I think, um, I mean, I've seen what you all kind of go through and we talked with Katie and the rest of your team, kind of on what discovery looks like, what it looks like to actually find the problems first, right? A lot of people seem to want to, a, get AI as a feature and a solution that they like and then find the problem for it, rather than start with the problem and find the feature. And I think that goes to the, the problem that you're mentioning. These tools are getting better, right? And they are. I say the quotations because better does not mean your business solution, right?
Right. Uh, once you get into the world of solutions, best becomes very relative, right? Uh, and I'd be curious, kind of what is your view or NLP logics really try to tackle when they're looking at what a best solution is, especially for AI? And do sometimes it's not even a transformer model. I know you all dive into kind of machine learning. Yeah. Uh, what is kind of your, your view, your opinion, how do you try to guide the people under you to kind of look at that?
Uh, you know, ours, I mean, the, the way we try to think about that is a couple of, couple of key things. One is, you know, we don't, we don't like to get into, rarely are our, really good engagements, ones that are solely technology driven. So someone, if someone comes and says, we have to use a chatbot for this, you know, this has to be part of the solution. Um, that's usually the wrong foot. And where we try to go immediately is, and I'm, I'm certain the rest of the team talked about this, but is really good understanding of the problem and what are the outcomes? Like, what, what are the outcomes we need to drive? And a lot of times, uh, components of the outcomes are the cost. Like, what is, what is the cost? Not just the, the cost to create the solution, what's the cost to support and maintain it, what's the infrastructure cost if you're using, you know, models to inference it, like, what is, like, you need to really look at it as a system and all those pieces can play off each other. You also need to think about, especially right now, you need to think about the, some of the rate of change on some of the cost parameters and one of them being that infrastructure and the models.
You know, is, is it, is it, is it a decreasing token cost that you're going to plan for, or are you planning for increasing? You know, are you going to need use more capable models next year, or are you going to be able to downgrade and get cheaper models because you have the performance that you need? Like, there is a lot to it. Um, and so we, we always try to come in, like, you know, really open-minded on understanding the problem, really making sure we understand the drivers for the business. What are the objectives the business is going to have to hit? And then,
You gotta, gotta build the solution, select the technology that kind of, that kind of fits into that mold because,
You know, not every, you know, some, if, if Katie was in here earlier, she maybe have said something about maybe you don't even need a model. I, I know we all want to use AI, but, you know, there's basically two really good business rules that, that if you embedded those in the right places and maybe you changed a piece of your business process, that's, that's a great way to get a really great outcome, uh, quickly. Like that might be the best approach.
No, and you're, you're spot on. That's exactly what Katie said earlier, is [laughter] she essentially was like, I had asked the question like, how often do you find people searching for AI solutions when they don't need one? And she's like, honestly, 50% of the time, like, it's half and half. Sometimes they'll come and say, we want to solve all this, but really, it's not even a model problem. It's just access to the data. It's just a structure problem. It's a spreadsheet and a good rule. Yeah.
Right. And it's not to say that you shouldn't be actively doing it. She, uh, kind of partnered with that saying like, you can use AI everywhere, but that does not mean that it is the solution everywhere. It is just another feature in some places. Um, and I guess what's interesting and, and what was great talking to Anton and talking to everyone else on your team is I could see they're extremely passionate. They are ridiculously passionate about their specific area. And some of them come from completely wild different fields. I think Katie went from, uh, finance and healthcare and modeling for hiring, all the way into the kind of deep level side of, kind of the statistics of it. But there was a pattern across all of it and Anton came from, you know, microelectronics all the way into this data science side. But again, there was a pattern of first principles and a thing that they were passionate about. What has kept you passionate in the last decade, I would say, and more importantly, has it changed?
There is definitely changed. So it definitely has changed. I, I'll take you back maybe even longer than a decade. Uh, really when we started the business, um, one of the things that, and I, I know the same is for Ted, from my business partner, because he's, we've talked about this plenty, but when we started the business in 2011, it, it was almost at, it was at the, I won't say the tail end, but big data, like just recognizing the fact that everybody wants to capture and store data is, uh, you know, the thing we all need to be doing because there was always this piece of like, we'll store the data now because if we don't store the data, we have no chance of extracting value from tomorrow. So just, just store, store, store, store. And we got pretty interested in thinking about and watching how challenging it was to really unlock the power that was in that data. And that, that was like the first kind of like real challenge and passion for both Ted and I was, okay, we have all this data, we can get so much out of it from analytics and reporting and BI, uh, you know, we can educate the business on how to make some decisions, but, you know, here's this thing called machine learning, like, could we predict, like I talked about the churn, right? Can we predict right now which customers are going to leave next month? And if you can do that, how can you change the business? Like, what would you do if you kn, if you have, and we would do this, we have, we would produce a list, here are the top 100 customers are at risk. You know, if you do that for a business, if you did that for our business, what would we do? We would, we would, we're going to do something, right? Let's do some outreach. So that was like, it was, you know, thinking that even probabilistically that you could have a glimpse into the future, like that's pretty interesting, right? Like you can see a month in advance in like a, you know, in, in this way that's not just like, um, forecasting out, you know, a simple forecast, but like it was some nuance. So that, that was like, that was like the first thing to me that got, that we got really excited about and thinking about, you know, how do you connect that to the business and change outcomes? And then, you know, that, that kind of morphed, at least for me personally, into when this is probably like 2014, 15, around there, where computer vision kind of had its moment.
And seeing for us, it was really concrete because we were working in, in healthcare and we, we were, we entered this, um, machine learning challenge called the Chameleon 16 and we were, uh, basically, you know, in digital pathology. We were identifying cells that were, you know, positive or, or a pathologist would have identified as like a cluster of cancer cells. And so we were doing that at, you know, at scale. And by at scale is, if you think about a, a wide sl, like a slide that has, um, you know, pathologists would look at and it's digitized, you know, there might be a million cells on that slide. It's really the data sets were massive. And using computer vision technology to basically, you know, supplement or augment what a pathologist was doing and, and the result of the challenge being, you know, the algorithm was, um, beating the pathologist in some cases. It was like, you could feel, and that got pretty exciting because it's like, you know, was, it's, it's awesome and, uh, when we were modeling like 2D data, like tables of data, like the churn, but then seeing computer vision, which the computer vision models were so fun because it, it is seeing the world in a 2D picture, like you're seeing it, right? So it's like, when it made a mistake, you know, you can see, oh, it made a mist, I see the mistake, I can also spot that and when it's identif, identifying something, it's like, yes, I can see, I can see that. So that to me was, was really interesting and thinking about how we could take that kind of an approach and bring it to, uh, we, you know, we did some work in the transportation industry. We did plenty of work in healthcare, um, and just, and, and we use the same technology to look at, uh, like faxes of data, like based from a back office standpoint. So that was like this whole new domain. It was a whole new modeling technique. It was the transition from like, you know, tabular modeling to like deep, I don't know, deep learning was like the crazy. I bet you if you looked at Google trends in like 2015, 14, 15, 16, like deep learning would have been,
What LLMs are now, like it would have been only research if you were to do research at that time in an academic setting. The only way you're getting funded in computer science is if it was deep learning because that was, that was like the rage and then, and then now, you know, obviously,
You know, where we're at now, it's just the, the step off of it. Yeah.
So that, that was kind of, you know, my lineage and I feel like personally for me, it was, um, you know, answering the question, what are, what can we really do with this data? I don't, I don't, we can report it. We can aggregate it. We can slice and dice it. That's interesting. Really interesting. We don't want to stop doing that. But how can we look into the future,
In ways that maybe we didn't think we could, like that? So that was just intellectually exciting, building models and algorithm, algorithms to do that. And then now it's, you know, thinking of our, you know, 70 customers and thinking across all their industries and thinking, we have this amazing suite of tools right now. We basically have intelligence as a service, which is like amazing. And we have still all the capabilities to build models from scratch in a super bespoke way for those specialized use cases. We got this whole suite of capabilities, you know, how can we help our customers use this, you know, in ways that just weren't possible just years ago?
Well, and there's this common thread of like passion pairs with usability, pairs with outcome, right? Is you're excited because you saw the capability, right? You're excited because you see the capability of something else. You see where you can grab these patterns, right? I guess a question that I want to ask, and I, I can imagine a lot of people would be interested in this too, is you're passionate about these things. You see these opportunities. You've obviously been very successful in, in, in actually getting the outcomes of these as people are learning and coming up today. Who are you looking at hiring and bringing into the company? What skills do you think exist in data science and AI that are really needed for the next decade, if you were to make a bet? Obviously, it's a hard one to say, but,
Yeah. And I, and this may be a bit of a, a copout because it's not, it's not like a, it's more of a, I guess it probably falls in the soft skill thing, but the, the characteristic that I would look for is really a genuine curiosity. Some, someone who has the genuine curiosity to ask good questions, uh, to challenge status quo a little bit, and to try to really understand, uh, systems. If it's systems of software, you know, I look at the, you know, when the, the core machine learning algorithms, like those are little systems, like really understanding, not that you have to go and build a, a deep network from scratch, but like to understand the components of it, to understand where it's good and where it's not good, to get an intuition for that stuff, to get to understand talking about systems, to understand, um, outside of technology, like, like a business, like, what, what are the systems that are in the business? How do they drive the towards the goal and the mission of the business? Like an understanding, if you can understand that stuff and then start to connect it, you know, but, because if you can only, if it's only the technology, I, I feel like that, that has too many opportunities to do interesting things that are adjacent to the outcomes you want to try to drive. Like it's, I think it's too easy to get,
Down a rabbit hole of technology, technology, technology, you know, if you can't, if you can't ground yourself in business and outcomes, then it's just, it's just a research experiment, which there's research institutions for that. We're a commercial enterprise. Absolutely. No, there's a difference there, but this hits on such an amazing point and, uh, we talked about this with both Anton and Katie. Yeah. Which is this idea of the human in the compute layer. A lot of people are focusing on human in the loop from an ethical standpoint or just a taste and judgment, but we forget that it's efficient. A lot of the times, like humans are really powerful at being able to do things without data. Yes. Right. And it's this concept of like, I've worked with companies where they were trying to build this crazy automation to like connect to data and they kept not being able to do it because sometimes you needed certain data and it's just like, well, how often are you using it? Well, only like once a week. Just have someone copy and paste it.
Yeah. And it's like, yes, you could automate it and that could be useful, but when you're looking at the business as a whole, you're already paying this person to look at it. They're going to know what to pull over. And even today with a lot of AI stuff when we're looking through it all,
Having people know when not to do something becomes useful, right? This idea of why. And I'm very curious your thoughts because I'm, I'm writing a paper on this now too, which is, right, we have this cost of API rising. We have all of these things. And yes, I think we should be automating a lot. I think it very much is useful, but sometimes having a person in control of that process isn't just a judgment thing or a value thing. It's just more efficient because sometimes they see something. So what do you have any thoughts on that that maybe, uh, you'd want to dive into?
I, I would say I agree with you wholeheartedly. It's, um, you know, it is, you know, humans are like the original general purpose computing machine, right? Like, like, like we are very good generalists, right? And we can do so many things. We can do it pretty efficiently as well. And so that's, um, yeah, I mean, I gotta say, sometimes we'll, we'll talk to, you know, a prospect or even just have a conversation and, you know, someone's getting excited about automating. Oh man, actually, it was, I was talking to a gentleman last night, a little bit different scale, but the same kind of concept. He was, he was talking, he works in healthcare, but he was talking to a, uh, potential partner and they were getting really excited about automating this process that he has. And his point was, his point was, yes, this is great, uh, but the, the part, the process you want to automate is 2% of this whole picture. While that's interesting, the big rocks are someplace else, you know, and go,
Not that we couldn't do this and maybe you get a decent ROI even on the investment, you know, maybe you spend 50 grand, you get a 100 grand back for it next year or something, but, but it's the opportunity cost and the time and the big rocks are somewhere else and, and let's go get the big rocks and move those first. Mhm.
And, and sometimes that's, you know, that's, I think that's harder at times because I think identifying the areas of where technology can be applied. I mean, some, honestly, the, a lot of times the, the big prizes are just identifying what the big rocks are. Like that's a lot of times that's hard enough to do. Like, what really, if you, if you could only do three things in your business next year, what would you do? Like, well, like, what, what are the three things you would do? And then, you know, okay, what if, what if technology doesn't help two of those three, right? You know, so,
I do feel like you can, especially right now, I think you can get caught up in,
Automate everything anyone does,
Um, but I think again, grounding it back to trying to identify for your business or your objectives, like what are the big things, you know, how do we advance the ball? Maybe with this technology, you have new opportunities. Like the big rocks just changed because of this technology lift that we have now. It used to be this thing. Now it's going to be this thing. Okay. You know, let's go have that conversation and drive that.
Oh, absolutely. I think that's such a good analogy and I'm just picturing the rocks right now moving around. Um, is this aspect of, right, in the, when in the internet.com bubble, right? We had people wanting to be internet companies rather than a company that uses the internet. And now I think we have a lot of companies that are trying to be AI companies rather than a company that's just using AI as another tool, right? Every time Excel updates, every time operating systems updates, I shouldn't be going for all the news articles and be like, oh gosh, what changes about everything? It means you haven't figured out your value stack yet in the AI age. Not to say that you shouldn't be looking at it, but if Claude or Gemini or something updates, my systems don't change too much, right? My workflows, which means my workflows are probably going to last a little while. Yeah.
If every time Opus comes out and your whole company might go down the drain,
It means you've been building too much towards that tech rather than solving a problem.
Right.
What ways do you think, and I already know a lot of my answers in my head, um, we talked with Anton about Logic Forge and things like that. What ways do you think NLP logic is building their workflows and their services rather than being an AI company, even though you, that is your kind of tick?
Yeah. And I, I would say we are, like, our DNA is always going to be an AI company, really, because of, like I said, how we were formed, when we were formed, like just our formative years, AI was such a piece of it. But I think that especially now, it is thinking more about being for our customers, a really good technology partner and really an outcomes company. Like, like that. That is what we want to do. Is we want to create outcomes for our customers. And I was, you know, talking to somebody just the other day on that of like, you know, the, if, when we engage, it's like, we really want to be working on the big strategic initiatives. Like that is what we want to do. It's our highest and best use. It's, it's why we're talking and engaged with the client most of the time. And like, that, that's how we want to move the needle. Um, so those are the kinds of engagements that, that work best for us. But I mean, as far as like, like right now, I mean, I agree with what you said. It's like, you have this interesting dynamic of the, you know, the, the, the foundation models are getting stronger and, and better and they're kind of eating away at probably a lot of, you know, I think they, you know, maybe they were called wrapper AI companies last year, they kind of get eaten away and stuff and it's, it's almost like, you know, this isn't my analogy, but I like it, which is, you know, the, the yellow brick road, which is like, I, if, if you think of the surface of everything, you know, that, that you're building, that you could be building and investing in in your company,
And you have kind of the yellow brick road in the middle right now, maybe it's five feet across,
And it's going to continue to get wider. The foundation models are going to continue to have be more and more just capable out of the box. And so if you're looking at where to be building, I think you should be thinking, you should be planning on that and thinking about where are, where are the, where are the places that are unique and bespoke to our business that are especially if you're going to invest in building, is is away from the, the yellow brick road and kind of off, and that's where I would be building.
Um, and then yeah, I mean, I like how you said it too. If you're, you know, if your workflows are changing because of a minor rev in a model, oh, that, that sounds pretty fragile, right? Like, like that, what, you know, I would definitely be be looking at that and thinking about that. [clears throat]
One thing that I want to take from what you said there, right, is this specific focus on away from the yellow brick road. And in my head, when I hear that, I hear productionize your opinion. And what I mean by that is, I, I think I use this at least once a day now. If I gave you, uh, a company problem and I said, I need a calculator, and I gave that to four other companies without any explanation, a few people might use Python, they might use whatever to build that calculator. But then if I say, it's for NASA, it changes everything. You're like, "Oh, okay. We probably need to use 4R and we have to figure out who's going to be using it. Does it need to be able to handle particulates and radiation in space?" Right? Everything changes. The outcome is the same. I need to calculate numbers.
Yeah. The difference is who and what those numbers are for, which become questions. And I think,
All the models are really good at training towards a reward function of best. Best becomes relative at the highest level. When you become a really good company, or you're at, you know, the best artist, the best food, it becomes highly relative. And when you work with your clients and your companies, their version of best for that outcome is not going to be solved just by the model. It's going to be solved by how that model is used.
So, how do you see yourself and people productionizing opinion as time goes on? What do you think the best ways in which to do that are?
Uh, and I, I don't know if this exactly what you're looking for. And I would say it's, it's funny too, when someone, you know, when you said, um, I need a calculator, and, and you made great points about who's, it's for, and what's, first thing is, why do you need a calculator, right? Because that, that is how,
Like, that is often how we, maybe sometimes it's easy to get off on the wrong into the wrong path is when I tell you I need a calculator, you build it, I use it, and I realize, well, I don't, that wasn't, I didn't need it, you, it, it wasn't what I needed, right? And to, to not dig into the context of that.
Um, but, um, but yeah, I mean, I, I would say the, the way that we look at that is to, so, so you, you're going to have the capabilities in the underlying foundation, and then, you know, you have your, like, you're, you're the way you described it is basically like, hey, uh, the preferences of today's underlying foundation model is not actually aligned with the preferences that we need for our business outcome. And one of the things that we do almost always, and I'm, I'm trying to, to not say always because maybe there's some cases that I'm not thinking of, but one of the things that we feel pretty strongly in,
Is making sure you have a really good baseline and understanding. We call it a truth deck, but it's basically a way of encoding your preferences and inputs outputs. Like, if this is my input, my preference is the output is this, right? M
And the reason why we do that is because when we have a situation where we have a model, and models can change, and you might want to choose different models based on cost and performance, is having it anchored in always being able to answer the question, uh, how do we evaluate quality, cost, and latency for whatever model plus whatever thin layer that we may put on top of it against the outcomes we're trying to achieve.
And if, if you put that into place, you know, was talking to somebody this last week about this, and he's like, "Hey, Matt, uh, you know, I did a lot of software engineering 20 years ago. I know a lot about unit tests. You're just kind of describing unit tests." I'm like, "Yeah, that's, that's pretty much it, is making sure you, you take the time to do that." And then if you do that, you save yourself a heck of a lot of heartache and it does, you know, you don't get quite as locked in. You can make some decisions on, uh, you know, on the modeling front because you have this really robust way of understanding change and comparing.
Absolutely. And it's so funny you bring up that last point, um, because I always find that extremely technical individuals, they go into two rabbit holes, which is either everything's super interesting and I'm down to explore it all, or everything's super simple, don't explain it to me. Um, and I find that being a very challenge because there's that middle ground, which is the business. Uh, and it's the idea essentially, yes, that is basically a unit test, or yes, that's basically Unix's philosophy, right? Like, a lot of my videos, I just describe folder structure for agents. And realistically, it's from 1970s papers. It's already been explained.
Lots of people comment, "Oh, I've been doing this for decades with software." It's like, okay, but are people using the way you're using it? You have to describe it and give it to customers in a way that they can actually use it. And that's the difference between AI expert and expert in the AI industry and expert in business who uses AI, right? And there's a huge difference, I think, which eventually you get to the end, which is where, um, and feel free to, to comment on it. I really like what you are all doing with Agent Forge and kind of Logix with your platform, is you're giving people the opportunity to use these highly technical but quote unquote old patterns of software first principles in a way that's simple, in a way that actually lets them access it. Just because it's been done before doesn't mean it can't be done better or accessed better. So I guess, uh, yeah, my question there would be, where, where do you see that platform or NLP Logic's pushing that platform over the next decade? Where do you see that being most useful?
Uh, you know, it, it isn't, it, it's, it's the common accelerator pattern, which is, if, if we're engaging with a customer, one of the things that we want to do is we want to bring our toolkit of patterns to the table so we can say, you know what, we're going to start on second base because we've got the plumbing for these, this componentry already implemented. And then the, the way we like to look at that is, um, is we want to make sure with our customer that, that when we, when we do that, it's not, it's not held back to an exclusive license. And it, it's a way for us to bring something to the table to help everybody get to the outcome a little bit faster. And I, I fully expect that, um, what we're going to be doing in the strategy is to always, as a company, invest in building and maintaining our accelerators so we can be differentiated. When we come to talk to you, I would love it when we've got, you know, a thousand different accelerators. And when we think about the solution, we know we can pick this one and this one and this one. We're going to take this one, but we need to modify these sections of it. But how can we come to the table with as much as we can bring possible? Like, like how do you know, we've been in business for 15 years, you know, how do we encode as much of, we much of that experience of the 15 years, like how can we encode that and bring that as something that, um, you know, that we can bring to the next solution?
Absolutely. And I think that's just ridiculously valuable. I say it all the time is, starting in the AI world is hard. Starting in the AI world 15 years ago when people didn't care about AI is even harder, right? And to be able to sit there and then get into the wave and then even grow further now, which, you know, behind us is a big expansion of, of your, your office and everything like that. So congratulations on that. I'm loving to see it expand like that.
That type of of work doesn't come from knowing the answers. It comes from asking the right questions. Right? In a world full of answers, questions become valuable, right? And I, I think I say it in every video, but I guess that also comes with changing your mind sometimes. I think it's, I said this with Katie, Albert Einstein's quote, the is, um, intelligence is a measure of your ability to change, right? And one of the things that a lot of these models don't always do well is making decisions based on no data, right? You can't even have a model without data. And humans seem to be very good at doing that.
So I guess what are some beliefs you had around machine learning and AI that maybe have changed or no longer or you've kind of, I might have been wrong there.
Do you have any of those or have?
Oh, I've certainly been wrong. [laughter] Yeah. So, uh, no, no shortage of being wrong. I mean, uh, you talk about totally wrong was being in the field and still being surprised by what ChatGPT was and knowing what it was within, not knowing what it was, isn't seeing the next three, four years, but like using it and knowing that things are going to be different in all forms of what we do from here on out. So like, not, I, I'm actually, it's surprising to me,
In a way, because, um, you know, being, I, I really thought that computer vision at the time in 2014, 15, 16 was going to have a bigger impact than I think I thought language models relative to where we are would have had an impact in 2022. So like, totally did not see that. I also, I also feel like, um, something I'm, you know, a big miss conceptually for me was, um, and I know there's big issues with hallucinations and that's a big issue and, you know, having a mo, you know, using ChatGPT to just confirm what you wanted to do or, but I am absolutely amazed at how I never thought we, I never thought there would be a model that would be as general purpose purpose as the foundation models are today. And I know there's a lot of engineering and a lot of stuff goes into the harness and this and that, but no way would I have thought that you could take this one tool and it's a, you know, under the hood, it's a, it's a large language model, but it's really one interface and I have a developer using it, I have finance using it, I have HR using it, I have the CEO, we have, you know, it's like, to me doing all kinds of different work and even if there are hallucinations and it's not perfect, but the fact that like, if we were to walk down the hall over there, I think you would see one, if it's not Claude or GitHub Copilot, ChatGPT or M365, one of those tools is going to be on a window of every desk you're going to go by. And I, like, to me, it's, it's from where we were, never would I have said that that would I, you know, I would have said
You know, 40, 50 years, basically, is what. And and it's so funny because we were working with large language models or early versions of GPT. We, you know, and then, but to see it was something different to me.
Yeah. Wow. And it takes a lot to, one, I I I think it says a lot about you and your company being able to recognize that that you had had that opinion and now you're very focused on, okay, how do we get the most value out of it now that we see the value? Because I see a lot of companies and people who still haven't changed their mind. And to me, that's very, I understand and I agree. I think we need to be careful of risk. We need to do all this. I mean, my master's thesis was on the bias in these data sets of large language models. I that was my entire master's degree. But even during my master's degree, I'd had PhDs telling me three, four years ago, language models are useless. I don't know why you're studying them. And it's like, how could you look at that today and say, okay, yes, sure, there's problems. I don't think there's any evidence saying they're useless. I don't think I agree.
[laughter] As you said, a tool's use can be judged by the people using it.
Right? You can say it's simple or what it does or what it doesn't, but if every person's using it and they're getting use out of it, it's pretty useful, right? Um, and again, I think that's where the misconception I think people had, and maybe I'm wrong, is in its probable nature that we would be able to make deterministic outputs or at least outputs that are within a risk threshold that are basically the same as deterministic.
And I think every year that's what we're doing. We're we're building stronger models, but we're building the structure around them, the data structure, the context, which provides us the output we want. And yeah, okay. So what? Like for me, I don't necessarily use the models for perfect outputs. In fact, I do the opposite. I try to have it generate a whole bunch for me to get me to where I can make a perfect output. And it does work. It works much faster, much more fluidly. Um, and I just find it wild to me that even myself when I I remember the first time it was a beta version of OpenAI's chat GPT and then I had used a version of Burp uh from Google and I just remember sitting here being like, I'm talking to a computer.
Like, well, like not like, oh, because I I think what the Katie was telling me the first chatbot I think was 1967, um, Eliza.
Yeah.
Eliza. Yes. Eliza and obviously not that great, but it's still a chatbot.
But it took 40 to 60 years to get to the point where we like my mouse is now dialogue.
That's mindboggling to me and I think that means a lot.
Well, I think then that kind of falls us on to one final question that is, I think it'd be very useful. Imagine someone who's watching this and uh, you know, I have a lot of people who are kind of in this position. They're running a company. They're looking to make a company. Maybe they've they're inside leadership at a at a very large company. What questions should they be asking themselves? Not because we shouldn't be giving them answers. We should be giving them questions. We figured that out. What questions should they be asking themselves about AI and their company? What do you think are some of the biggest questions they could ask themselves?
Uh, to me, the best question, the most general question that I think um everyone should try to answer is to is is to just get an intuition for uh how AI works, which sounds super intimidating, but I remember I'll give you an example like uh, this is a long time ago when I was in college. It was it was I think I think this is pretty common, but it was like, you know, you had to answer this essay question which was, in as much detail as you can provide, what is everything that happens when you request a web page?
Okay.
Including like trying to understand uh, you know, getting into the um IP protocol and packets and and what happens on the server and then on the server, you know, how is information fetched and retrieved and even if it's not totally correct, right? But the intuition as far as, you know, things, you know, literally packets of data are traversing to a server that I don't know where it is, that server is, you know, doing things, access. So when it comes to AI, the reason why I say that is because I think the worst viewpoint is being is anything around it's just a magic box. I don't know. I I type something in and I get some answers. But it's like, it's I don't think it is all that difficult to get. If you could probably spend as little as a half hour, an hour, but to get an intuition as far as how is a model created, like what what is what are some of the steps? What do you have to do? What what? Because I think if you understand that, you know, a model is kind of a representative, you know, represents the data it was trained on. Oh, okay. Well, what does that mean? The data hasn't been in, like, what you're getting out of the model is heavily influenced by the data that goes into it and you know, how is that data crafted and you know, we have researchers answering questions and certain and just like, okay, so this isn't this isn't really magic, right? There are these pieces that I think you could line out and then um, and if you get an intuition for that, I kind of feel like you're really set up well for what I would think of as this current generation. Like if you know um, Yan Lun, you okay.
[clears throat]
So Yan Lun, I love. He's got he's got a really, I like Yan Lun because he's a little bit counterculture. He doesn't seem to be on today's hype train. I feel like he's got a nice long-term view of the world and I love his, he's got a really great phrase or sentence which is, machine learning sucks. Like that's his word. And I like that because, you know, there's some appreciation for that phrase because machine learning sucks. So if you understand why he's why he says that, is one, what is machine learning? It is all models today, all the frontier models are basically learned from data. So we are constrained at, we have to provide the data and data is finite. We're at a point now where creating more data requires more humans to do more complicated things. And if we can't do that, we're going to have we're going to be down to like using tricks to like squeeze out that next percent of the model. And I think what and Yan Lun's point is machine learning sucks because we already know we're going to be constrained at what we can create because the input is is scale kind of constrained. And then I think his second point is kind of like what I was saying in in 2015, 16, 17, like if it wasn't deep learning, it wasn't anything. And I kind of feel like that's today. Like today is if it's not a large language model, what are you even talking about?
And so I like his point being to really, to get the next paradigm shift, we need more people looking at things that aren't today's generation, you know. So his, I interpret his thing as machine learning sucks is is hey, there's going to be a next paradigm. If we're going to get to it, we got to have some people looking past what we're doing today and trying to build that new frontier, which obviously he's doing with his world models, but.
Yes.
Um, but but yeah, I mean, that that's, you know, so but getting an intuition, I think goes in a lot of areas. Just get an intuition for how does it work, you know, don't rely on it being magic. And then I think you can actually make some decisions about how do I use it at home with my kids? How do I use it? You know, I use it, you know, recipes, whatever you're doing. I use it for my golf swing or how do I use it at work? Like you can get some intuition on what what's the right way to think about it and use it properly.
Absolutely. You don't need to know how to build an engine to at least have some intuition on how a car works.
Great.
Right? You put some gas in, you press the pedal, some explosions happens and it moves. It's not magic that you move the car, right? Electric engines, a little bit more magic, but still you can see the uh, the motors moving and things like that.
And that's really what it comes down to. And then the other side of it, which that last part, what you said, not just use it, but use it on something that you already know. Because then you can see its limitations, but you can also appreciate its ability. Because a lot of people say, "Oh, use it on something you're an expert at and you'll see its limits." And I'm like, well, I'm an expert on it, so if I give it enough information, it's able to amplify my expertise, which I absolutely love. So, no, I think that's such amazing advice. Well, with that then, Matt, I think we'll call it a night here. Uh, thank you everyone. If you've made it this far in, I appreciate it. Of course, there'll be links in the bio to reach out to talk to me and the team. Um, if there's any other questions, please leave them in the comments. As always, happy learning. Thank you.