Transcription
there's been a democratization shift in
the industry and then because of that
shift most people using MLAI today are
working at a higher level abstraction.
So rather than intense focus on model
architecture and training, most users
now just are ahead of this object, this
extremely powerful new computing
primitive and what do I do with this?
And this is where I've been operating
for the last few years at Langchain
building on top of these LLMs, prompt
engineering, context engineering,
fine-tuning, building agents, all of
these new disciplines built on top of
this new primitive that's being offered
through an API by a small number of
players. So I think those are a few
major shifts that we've seen in the
landscape over the last few years. That
was Lance Martin, machine learning
engineer at Langchain, outlining a key
shift in the generative AI era. We're
moving from the challenge of training
models to the new engineering discipline
of orchestrating them at scale. Lance
used to build and scale production ML
systems at Uber, including self-driving
technology, and now builds tools at
Langchain to help teams across all
verticals build and deploy AI powered
applications.
In this episode, Duncan Gilchrist and I
speak with Lance about what's changed
since the early ML days and how core
principles like simplicity and
observability must be adapted for
non-deterministic systems. Lance brings
so much insight from the bleeding edge
of the space with examples from Claude
Code and Manis discussing the practical
disciplines of context engineering
including context rot or why the
effective context window is often much
smaller than the token limit and the
three-part playbook to manage it reduce
offload and isolate. This also includes
using multi- aent architectures for
context isolation. We also cover the
emerging architecture of the agent
harness which manages tool calls
essentially how LLMs can do things and
how builders can use only a few atomic
tools like a bash tool to expand the
agents action space dramatically. This
episode is a deep dive into engineering
discipline and also gives technical
leaders clear insight into how teams are
building and delivering value at the
bleeding edge where the foundation
models powering your applications are
constantly and exponentially improving.
High Signal is brought to you by Delina
the AI agent for data science and
analytics. If you enjoy these
conversations, please leave us a review,
give us five stars, subscribe in the
newsletter, and share it with your
friends. Links are in the show notes.
I'm Hugo Ban Anderson. Welcome to High
Signal. Let's jump in. Hey there, Lance
and welcome to the show.
>> It's great to be here and I've known
Duncan for many years. It's a pleasure
to be on and great to meet you as well.
>> Totally. And look, I'm so excited to
hear about what's happening at Lang
Chain and all the wonderful things that
you're affording people to do there. But
what I'm also really excited about is
you've worked on production ML systems
at places like Uber prior to working on
generative AI tooling at Lang Chain. So
you have a wonderful perspective on
what's changed. So I'm wondering if we
could open by you letting us know what
feels fundamentally different about
building and maintaining ML systems
versus what people are doing now with
generative AI and LLMs in particular.
>> That's right. Yeah. So you know Duncan
and I over overlapped at Uber. This was
back in 2015 2016 era. And I think
there's been a few interesting shifts in
the ML AI landscape since that time. I
think one is architectural
consolidation. So we saw the emergence
of the transformer architecture
extremely expressive. We saw driven by
scaling laws and compute data model
size. Models get much bigger. We saw
other architectures like CNN's RNN's
that are a little bit more specialized
kind of get swallowed by transformers.
So we had architectural consolidation
and scaling loss driving much much
larger models. That's thread one. And
then thread two was I worked in
self-driving for a number of years at
Uber than after Uber. And in that era,
there was approximately the same amount
of orgs that were training models were
housing or using them. So it was one to
one in the sense that each self-driving
company was trying to training their own
models. It was highly proprietary. a lot
of in-house expertise at really all
these companies and I think obviously
beyond self-driving where ML is being
deployed recommener systems typically
organizations that were using ML were
also training the models now that's
entirely flipped you've had the
emergence of a small number of
foundation model providers models become
extremely large and most people using AI
today are not actually training models
so that's that kind of this there's been
a democratization shift in the industry
so architectural consolidation
democratization shift and then because
of that shift most people using MLAI
today are working at a higher level
abstraction. So rather than intense
focus on model architecture and
training, most users now just are handed
this object, this extremely powerful new
computing primitive and what do I do
with this? And this is where I've been
operating for the last few years at
Langchain building on top of these LLMs,
prompt engineering, context engineering,
fine-tuning, building agents, all of
these new disciplines built on top of
this new primitive that's being offered
through an API by a small number of
players. So I think those are a few
major shift that we've seen in the
landscape over the last few years.
>> I think that kind of contrast from
classic or traditional ML to geni is so
interesting and you and I have a lot of
war wounds I think from and battle scars
from the Uber days. I'd love to explore
with you like which lessons from
traditional ML still actually apply in
GI systems and which start to to fall
apart.
>> Yeah. So that's an interesting one and
what I would say here is actually even
though we're now handed these incredibly
powerful models through for example APIs
offered by these frontier labs
simplicity remains essential I think it
is very important to start with the
simplest possible solution we see many
organizations that we work with the lang
chain and others that I've consulted
with will jump to agents are in the air
agents are a buzz word I want to build
an agent and I think we'll talk about
this more in detail later but really
thinking through the problem you're
trying to solve there's many different
ways to do it prompt engineer ing a
simple workflow, building an agent with
simple contact engineering, all the way
up to maybe building an agent with some
kind of RL in the loop or reinforcement
fine-tuning. There's a spectrum of
solutions you can use with these models
depending on your problem. And I think
starting simple is very important. The
other thing that's really critical is
actually observability evaluation. So
you can build an agent, but actually
having the ability to understand what's
happening and to actually evaluate it in
a rigorous way is obviously extremely
important. So at Lang, we do a lot of
work on observability and tracing and
also a test suite and evaluation suite.
Beyond just simple unit tests, I think
many software ors are familiar with a
simple unit test. Working with ML
systems, particularly LLMs, which are
nondeterministic, needs a new kind of
evaluation, which we can talk about
later. I think the final point is a
really interesting observation I heard
from Jason Wei who was at OpenAI for
many years and now MSL meta super
intelligence lab. He talks about this
idea of verifiers law which says that
the ability to train an AI to solve a
task is proportional to how easily
verifiable a task is. So verification
means coding you can just like compile
the code you can run it make sure it
runs that that's kind of what
verification means and tasks that are
easier to verify are actually easier to
for example apply reinforce and
finetuning to that's what he's referring
to. So actually setting up evaluation
and part of evaluation is establishing
some verification metric is actually
very helpful and important foundation if
you ever want to apply reinforcement
fine-tuning or [clears throat] training
train model fine-tune models for a
particular task. So setting up
evaluations with clear verification or
criteria is very important for quality
and also if you ever want to move into
fine-tuning or more advanced things. So
those are three things that are still
very important today with these new
systems which were always of course
important in the prior eras of ML.
>> There's so much meat and insight in
there that we will get into throughout
this conversation. [clears throat]
One thing that I do think we're we'll
talk about is how we go about building
and evaluating agents and it does feel
kind of like a two cultures thing in
some ways because as you've said
starting and building small is
incredibly important to be able to
inspect evaluate and deliver value and
yet we'll talk about a blog post you
wrote about context engineering in Manis
typical Manis call has 50 tool calls and
anthropic has told us to start slow with
workflows and yet they also have
published about their multi- aent
research system which is a big sprawling
ing behemoth in in a lot of ways. I am
interested in before we we get to that
point to talk about a wonderful post of
yours that we'll link to in the show
notes called learning the bitter lesson.
And you quote Rich Sutton by saying
general methods that leverage
computation are ultimately the most
effective. And I'm wondering if you
could expand on this quote unquote
bitter lesson and tell us how it's
shaped the way you approach designing
systems today. Yeah, actually I think
this is one of the most interesting
challenges associated with now building
on top of this new kind of LM compute
primitive that we all have access to. So
basically Rich Sutton 2019 put out this
very important like kind of seminal
essay called the bitter lesson and the
intro line follow goes the biggest
lesson that can be read from 70 years of
AI research is that general methods that
leverage computation are ultimately the
most effective and by a large margin. So
simple methods throw more compute often
beat more complex methods with more kind
of human biases and like a classic
example of this is in vision we had
convolutional narrow networks. They
encoded various inductive biases about
how we should solve vision tasks like
classification or object detection.
Transformers ultimately became
state-of-the-art for that particular
task. Transformers are a very general
message passing architecture. And so the
observation was just a basic
architecture like transformers plus more
data and scale actually ultimately beats
more kind of handcrafted methods. That's
been the trend we've seen in AI for over
70 years. Now the link here is that also
applies to things that we're building on
top of LLMs. So it also applies at the
kind of AI engineering layer. This is
one of the biggest lessons I've learned
in building LLM applications because
here's the problem. Just as the model
layer, we're building on top of
exponentially, we're designing
architectures on top of kind of
exponentially rising or increasing
compute, we're also now building
applications on top of exponentially
improving LLMs or models. And so what
you build today and the assumptions
baked into your architecture in terms of
the whatever app you're building will
not be correct in 6 months when a new
model is out that's much much better.
And I saw this play out in my own work
on this project called Open Deep
Research and that's what I talk about in
the blog and I just cover a year of
working on this project. It's basically
a fully open source deep research agent.
I started it as a very simple workflow
back in 2024 because basically tool
calling with LMS was weak. Actually
didn't really perform that well. I
didn't want to and we'll talk about what
agents are versus workflows a bit later.
Suffice it to say, I started with a
particular architecture that was not
using an agent and over time as tool
calling get much better. I basically had
to rearchitect open research three or
four times to keep up with increasingly
improving models. I talked to Manis
about this in a webinar we did about two
weeks ago. They mentioned that Manis is
one of the most popular kind of general
purpose agent products out there today.
It's based in Singapore. They actually
rearchitected Manis five times since
launching in March. So what you're
seeing across the industry is building
on top of this new primitive that's
getting much much better all the time
forced you to continually re reassess
your assumptions and rebuild your
applications. Another great example of
this Boris Churnney from Claude Code
mentioned in passing in one of his talks
like in the Q&A the kind of secret sauce
so forth the kind of secret sauce of
cloud code is he said 70% model 30% kind
of scaffolding or harness right they
have an agent harness that they they use
with cloud code and the followup
question was okay but over time as the
model gets better does that mean all
your work on the harness is like
irrelevant he said yeah he said yeah
that is the case so what's happening is
over time models get better and you're
having to strip away structure, remove
assumptions and make your harness or
your system simpler and adapt to the
models. And so this is something that's
very hard. You can't just have a fixed
architecture, a fixed scaffold, fixed
harness and be done because you're
building on top of something that's
always improving. I think this is one of
the biggest lessons that is tricky and
we'll I think we'll talk about a little
bit more later. But that's one of the
biggest things I've observed in working
with LM. It's embracing change, being
willing to rearchitect your system. And
the nice thing is with LMS being
extremely strong at for example code
assist cloud code cursor devon it's very
easy to rebuild things but embracing
that fact that look you're building on
top of a kind of primitive that's always
getting better. You have to constantly
rearchitect your application and you
can't be shy about that I think is one
of the most disorienting things though
for people who are starting to work with
LMS for the first time.
>> Would you mind just clarifying what you
mean by a harness as well?
>> Yeah. Right. So when you build an agent
and we'll talk about kind of agents and
workflows in more detail, but basically
when you build an agent, you have the
LM. So LM's exposed through some kind of
SDK for example, you could be using
cloud, you could be using OpenAI, but
when you build an agent as an example,
you're taking that LLM SDK, you're
binding a bunch of tools to it, and
you're basically allowing that LM to
call tools. So when it calls a tool, it
basically just produces a structured
output that adheres to whatever tool you
provided. Let's say it's a search tool.
The surf tool takes a single parameter
query. The LM will produce a function
call or tool call that just has like
query and then whatever query parameters
that you want. That's all tool call is.
You need something that actually
executes that tool call. And that's kind
of where the harness comes in. So the
harness will actually do that and other
things. We'll talk about context
engineering in a bit. But the harness is
the thing that actually says, okay, the
LM actually made this tool call. Okay,
I'll go ahead and run that tool. I'll
take that tool result, package it as a
message, add it to, for example,
typically you're having you have a
message list that's growing that you're
passing back to the LM every turn of
your agent. So managing that message
list, managing tool execution, packaging
up the tool results as messages, passing
them back to the LM. That's the harness.
And we'll talk about in a little bit,
but actually it handles more than it
typically handles some kind of logic or
what we might call context engineering.
And cloud code, for example, has a very
interesting harness. You actually can
see when it's running, you can see the
tool calls it's making. Like if you ever
work with cloud code, you can see like
it'll run bash tool, it'll run different
search tools. You actually see that in
the trace as it's running. So that's the
harness is doing all that under the hood
and passing those results to the model
and models reasoning, making additional
tool calls and so forth.
>> A few times you've used words like
workflows and pipelines and agents. Can
we unpack those a little bit? What do
those actually mean? How do they fit
together and where are they in the
generative kind of hype cycle?
>> Yeah, this is a really good one to
clarify. So, I've said the word agent a
few times. I might have said the word
workflows. We should break all these
things down carefully. So, actually
probably the best blog post I'm sure
sure we'll put in the show notes is from
anthropic on this came out late last
year. It's called building effective
agents. So, they define workflows as
systems where LMS and tools are
orchestrated through predefined code
paths. It follows a predetermined
sequence and you can have LM calls
embedded in that sequence. But you have
an application that goes A to B to C to
D. Every time you run it, step C could
be an LM call to do something. That's a
workflow. An agent is a bit different.
An agent is a system where an LM they
say dynamically direct its own processes
and tool usage, maning control over how
it accomplishes tasks. So what does this
actually mean in practice? All it means
is I have an LLM. I have some set of
tools. I bind the tools to the LM. Let's
say I have tools A, B, C, D. The LM can
call those tools in any order it wants
to solve the problem. It can call B, C,
D. Whereas in a workflow, you lay out
very precisely the steps A, B, C, D. So
that's the key difference. Workflows
follow some predefined set of steps that
I lay out as a developer that can
involve LM calls, whereas agents allow
an LM to call tools autonomously in a
loop in any order they see fit. And I
think the key point to make is the
difference is autonomy. Agents are
really good for tasks that you can't
really enumerate ahead of time. Research
is a classic. That's why deep research
is one of the seminal agent products.
Research is open-ended. The next step is
conditioned on the prior one. I'm going
to do a search, get some results, reason
about the results, and do another
search. Whereas tests, I want to run
this test suite is a classic workflow
type problem. Every time like a a PR is
put up, I want to run these five tests.
That's more of a workflow thing.
>> You so it sounds like you advocate for
structured workflows. they're simpler in
certain kinds of cases like how do you
think about what kinds of problems are
better solved by those versus more
agentic approaches.
>> Yeah. So this is one of the classic this
is one of the classics. Okay. So when to
use workflows when to use agents. I'll
share some nice documentation in the
show notes on this but there's a few
different resources I like. One in
particular is a talk given by a guy from
Shopify who developed what they call
roast. So roast is a framework built
internally at Shopify for for laying out
workflows. It's very similar to a
framework that we have at Langchain
called Langraph. Langraph is an
extremely popular framework for building
agents or workflows. But I really like
the Shopify example because it really
exemplifies a lot of the rationale and
reasons why we build langraph. Workflows
are great when you have something when
you have problems that have predefined
predictable steps. Migrating a legacy
codebase, running some set of tests. The
Shopify talk talks mentions a lot of
those are some of the things that they
use to motivate roast very well-
definfined predictable steps. Two is
consistent and repeatability. So when
you need deterministic behavior and
clear oversight like testing is another
great example of that with every PR you
want these end tests run and I guess
known sequences steps A B C D that's
really where workflows shine and then
agents are good for anything that's
requires ongoing adaptation debugging
iteration research is a classic coding
is another good one that's why coding
agents are so popular solving problems
with coding often are iterative you try
one solution you might run through a set
of tests fail you try again so more
open-ended adaptable problems are much
better for agents research coding
predictable well- definfined steps
migrations tests much better as
workflows and one nuance is actually you
can embed agents in workflows so you can
have a workflow of end steps and one of
those steps could be calling an agent to
do a thing and in fact Shopify talks
about that a lot in their roast talk
which I'll be sure to link so I think
it's an it's subtle point that they
actually can play together but it is
also true that lots of problems that
people want to solve with agents you
could absolutely solve just by laying
out a workflow.
>> The other subtle point that I think is
worth expanding on slightly cuz it trips
up a lot of people is that it's not
agent or or not. And when we talk about
workflows, a lot of people will use the
term agent or agentic to describe that.
And of course, Anthropic makes that
clear in the blog post you you mentioned
from last December, which is there's a
spectrum of agency where maybe you've
got an LLM and you're adding some
memory, couple of tool calls, retrieval,
and then you're starting to build
workflows. So it isn't an on switch or
an off switch. The other thing I think
worth mentioning, I think you're
speaking to this already, is having high
agency in your software in terms of
having an agent with incredibly high
agency works very well when you have
relatively strong supervision as well.
So a human in the loop with it who can
guide it and train it and have
conversations with it and maybe even god
forbid check the code it's written for
example.
>> So I think this is an interesting point.
Part of the reason why a lot of people
were hesitant to build agents prior to,
for example, this year is there's a few
different reasons. One being that tool
calling was not as reliable. But as a
consequence, you did have to you had to
babysit agents much more. You had to
check their work very carefully. They
get caught in what you might call like a
tool calling loop. So continually trying
to call the same tool many times, that
just burns tokens needlessly. So, as LMS
have gotten better at tool calling, it's
become more and more feasible to build
agents that are actually effective that
don't fall into these kind of common
traps and failure modes as you mentioned
that require a bit less babysitting. Now
I do want to mention it is still true
though that agents because they can
autonomously call tools do pair very
well with human in the loop and there
may be certain tools for example with
cloud code you can basically you can run
it in safe mode where basically you
approve certain tool calls and that's
true across many different aentic
systems that's a whole topic we get into
but sandboxing agents is often very
important and so forth so because
they're running autonomously you do
indeed have to be quite careful both
about token usage and them spinning off
into kind of very kind of long sequences
of tool calls and also security. So
making sure they don't make tool calls
that you don't want them to make, hit
sensitive systems, delete things and so
forth. So it is absolutely true that
because agents have higher autonomy or
agency, we often have to be a little bit
more careful about what they're doing
and sandbox them appropriately.
>> Absolutely. And Anthropic actually made
that very clear when they first released
their first prototype for cla code. I
think they were like please do sandbox
this. This is highly experimental. Yes.
>> And then like earlier this year we saw
even people like Steve Joerger say on
Twitter, I can finally talk about this
now. I deleted a production database by
Vibe Coding. And yes,
>> the oppos the opposite of safe mode.
Cursor used to call it YOLO mode. They
don't anymore. They've changed the name.
But you actually spoke to a really
interesting point which is models are
getting significantly better. So can you
tell us a bit about how
>> the improvement or more affordances of
models have made building agents or made
agents more reliable at what they do.
>> You know, it's so there's a few
different interesting threads here. So,
one is I'll link this in the notes, but
so Meter publishes a kind of a
leaderboard or or kind of an evaluation
that measures the length of tasks that
LMS can accomplish. And I have to go
back and check. I believe it's doubling
every seven months. So, this is like the
length of human equivalent work. So it's
something like agents can at a 50%
success rate accomplish tasks that take
a human two hours today or something.
There's a bunch of different models of
value. But the point is the the kind of
autonomy level of LMS is doubling every
seven months. It's one of the
interesting scaling laws of track. So
that's a consequence of models getting
better at tool calling largely that
getting better at tool calling allowing
them to perform longer horizon tasks. So
it's really comes down to the fact that
the models are indeed getting quite a
bit better at instruction following and
tool calling. And also I would note you
see this with cloud code they're getting
better at adapting. So for example if
they do make an error for example they
format a tool calling correctly they can
see that for example that trace and they
can correct. So selfcorrection is
another very important point. So it's
really all these things coming together
at the end of the day models getting
much better at tool calling allowing for
longer horizon tasks. That's really the
key driver here
>> and that's something the self-healing is
something anthropic has published quite
a bit about in their building their
multi-agent research system and we do
have a bunch of other topics to get to
but something you've actually been
speaking to implicitly is the rise of
background agents right so as the models
get better at doing tool calls the
ability for us to send them off for
longer and so I'm wondering if you can
tell us just what you're seeing with
respect to this burgeoning field of
background agents
>> it's funny at least at Lang we call them
ambient agents We have a whole course on
it. So I did a whole course on building
ambient agents in Langraph. And the use
case I built in the course that you can
build up to is an agent that will run
your email. And so it just runs
autonomously every night on a cron and
it'll process all your emails. Actually,
sorry, it's not every night. You can run
it. It actually pings every 10 minutes.
So it's running in the background every
10 minutes constantly. You can configure
that any way you want though. You could
have it run once a day overnight. For
me, it's every 10 minutes. And it's
constantly kind of monitoring your
emails. It pulls them in. It triages
them. It decides which ones to respond
to, which not to respond to, and it'll
produce responses and cue them all up
for you. You approve them through a
little interface we built, and it'll
fire them all off. So, Harrison, our
CEO, actually uses it. I don't cuz I
don't get that many emails. But, so
that's a good example of an ambient
agent. I think it's a great point. It's
a very good emerging form factor. Codeex
is a great example for code. Just kick
it off. It runs async, does stuff for
you. It makes a lot of sense. The catch
I would say is that in the context of
coding it can actually increase the
burden the review burden. So for example
you have to really trust the system to
do a bunch of work autonomously and come
back to you after some period of time.
If it spins off on a task for a long
period of time and it's on the wrong
track you get this big body of work done
at the end and you say oof. So designing
the right kind of guard rails or human
loop. Oh it's stuck. How do you check
it? How do you prove what it's doing? Is
a little bit tricky still. Like with my
email agent, I have a few different
gates. Like basically, it pings me if it
needs to ask a question. It pings me
when it's going to prepare a response,
like an email that it's drafted it. And
it'll ping me if it just decides this is
worth ignoring to let me confirm. Long
story short, when you're working these
async or ambient or background agents,
you do have to be careful to design the
system such that it has the right kind
of human loop checkpoints because if it
just goes off and does a bunch of work
behind the scenes, that can be
problematic for the obvious reasons. So
that's a little bit of the trick to
these async ambient agents. I personally
for code I actually do use cloud code
synchronously mostly. That's just me and
I do use async agents for things like
email or at least I've built it and I
used it for a while but Harrison
continues to use it. I just don't get
enough email to justify it. But I do
think with async or ambient agents being
very thoughtful about how you set up
human the loop is very important and how
much trust you can build in the system.
I'll make a small note here that in the
email example, a very important aspect
of ambient I think is memory because you
want them to handle long horizon tasks
autonomously. So ideally, they remember
your preferences because you're endowing
them with like longer horizon work. So
in my little email example, I actually
have a memory system. So every time I
give it feedback, it records my feedback
and bakes it into this little kind of in
this little long-term memory which is
just stored in like a very simple set of
files and those files are updated
constantly as I give it feedback. So it
gets smarter and smarter over time. So
it runs autonomously but it's also
learning my preferences. So it' be very
annoying if these systems run
autonomously but don't learn from our
feedback and just keep take making the
same mistakes over and over. I think
that's another tricky thing about
autonomous agents though. Ideally they
have some form of memory so they can
adapt and learn. zooming kind of one
click out on that actually. I think the
context you feed into the agent is
obviously so critical to making the
agent work correctly and you've actually
written a lot about this term kind of
context engineering. I'd love for you as
a domain expert to maybe define that
term and help us think through why it's
important and how leaders should think
about it.
>> Yeah, this so this is actually a very
important kind of term that's emerged in
the last couple months. So the way I
think about it is let's say I build an
agent. I take an LLM, I bind some tools.
Let's say it's a deep research agent. I
actually made this mistake, so I'll walk
through it exactly. You bind a bunch of
search tools. You have it run. It does a
search, returns the results. Those
results are appended to a message list.
You pass messages back to the LM, makes
another search tool call. Same thing
that happens five or six times. By the
end, that message list could be quite
big. depending upon how many s how how
large or how many tokens are in each
search result. You could be talking in
my case hundreds of thousands of tokens.
So extremely tokenheavy if you're just
doing this kind of naive tool calling in
a loop thing which is a base case agent
and that's extremely expensive and slow.
So actually being very thoughtful about
the context you basically feed to your
LLM is important not just for latency
and cost but also Chroma I'll link this
in the show notes put out a really nice
report on what they call context rot. So
basically as context gets longer
performance degrades and anthropic
mentions it recently in a nice post they
have on context engineering and they
mentioned it the attention mechanism
kind of starts to degrade with respect
to context length. So the point is
agents in a naive form use a lot of
tokens if it's just tool calling a loop
depending on the tool calls you're using
often times for example with a search
tool it's pretty tokenheavy so they're
very token hungry consider the fact that
manis typically calls 50 tools per run
it's a lot of tokens and throbbing
mentions production agents can call
hundreds of tools or have hundreds of
turns so it's costly it's slow and it
can degrade quality so that's why we
talked about agents and agent harnesses.
Often agent harnesses have some
mechanisms to manage this and a few of
the trends I've observed are one context
reduction. There's a few interesting
tricks here. Some of those intuitive are
basically compacting older tool calls.
So imagine you've called a tool then you
call another another tool. By like your
fourth turn, you can compact tool call
one. You don't have to keep it in the
message history. Manis does this. It's a
good idea. I've done it as well. Another
thing if you use clock code is
summarization. So basically once you
start to fill up the context window of
the LM produce a summary of the entire
message history and that basically
compresses all those tokens into a much
shorter form and you can move forward.
So context reduction pruning or
compacting older tool results and
trajectory summarization are two tricks
we've seen uh crossclaw code and manis
I've used them as well. Imagine if
you're using like the latest sonnet four
or five models with a million tokens.
You're saying you don't even want to use
those million tokens. You only want to
use fraction of them because of context
raw because you the agent gets confused
even if you start to use the whole
context length. This is a very subtle
good question and it's actually under
reportported. So it is true that these
models have some context window. For
example, a million tokens for the latest
four five models. That doesn't mean that
performance will be of high quality
through that entire context window and
you can have degradation all sorts of
non-obvious ways with respect to context
length. Manis the mana cso and I chatted
about this last week in our webinar and
he mentioned that often the effective
context window for these LM is actually
quite a bit lower than this kind of stay
the technical one. So something to be
very careful of. Just because the
context window is a million tokens
doesn't mean you're going to get
necessarily high quality instruction
following throughout all those instru
all throughout that entire kind of
context length and the failure modes can
be not obvious and subtle as noted in
that chroma report on context rod. So I
think it's worth being careful and even
if you have a very large context window
being judicious about how it's used and
Anthropics come out and said as much.
They have a very nice actually last week
put out a very nice white paper or blog
post on context engineering. Their new
SDK has a bunch of updates that actually
incorporate some of these ideas in
particular. It actually has basically a
compaction of older tool calls
automatically built into what some of
the new models in the SDK. So they're
actually employing this idea. Exactly.
And as noted, cloud code indeed uses
summarization.
>> For those wanting to dive even a bit
deeper, it definitely doesn't help with
context, Rob, but there are tools like
prompt caching that can help with cost
and latency when having long context as
well.
>> Okay, so that's an important point and a
good one. So prompt caching is indeed
useful. That helps with cost and
latency. It doesn't help with context
rot.
>> Yes.
>> Because if it's if you're using 100,000
tokens, even if it's cash, it's still
100,000 tokens. So Manis actually uses
caching very extensively but they still
perform context reduction through
pruning and summarization. So we
actually do both.
>> Yeah. The other thing I've seen and this
is an anecdotal in in work I've done and
from people friends who work in the
space is if something's super super
important a really important piece of
information including it at the start of
the prompt and the end of the prompt can
seem to increase performance anyway.
Yeah, that's actually
>> that is true and actually so I've done
that quite a bit. So basically moving
instructions to for example yeah the
most recent message to reinforce
something and that's part of this. So
basically if you're reducing context in
effective way the overall message list
is managed pretty appropriately because
what's happening is you're pruning or
compacting older messages particularly
the older tool calls and you're doing
summarization. So basically your message
list will be less much less tokenheavy
versus if you did not do those things.
But there's a related so there's
actually three there's three big ideas
here. One is we just talked about
reducing context. The second one is
offloading. The third is isolating. I'll
talk about those briefly. So offloading
actually means for example manis uses a
file system. And for all tool results
it's tied to the pruning thing. You
actually save the full tool message to a
file system. So it's preserved. So then
when you do that compaction, you always
have reference to the actual file if you
ever need it again. That's it. So this
idea of offloading is a really good idea
and I use that as well. I think it's a
very effective way to do it. The other
small point I'll make with respect to
offloading is actually offloading a lot
of your functions or tools. So this is a
very subtle one, but we're seeing this
more and more. Instead of giving your
agent like 100 tools, which can be it
uses a lot of tokens because you have to
include all those instructions in your
system prompt like you have 100 tools,
here's how to use them all. Instead, for
example, what Mattis does and I've used
this quite a bit as well. Use only a few
a small number of kind of atomic tools
like less than 20 file system bash tool
and basically allow your agent to use a
computer to for example run scripts that
can expand its action space hugely
without bloating function calls. For
example, this is a good take that man
mentioned to me most recently. It's top
of mind for all MCP tools. I think we'll
talk about MCP in a minute here. Instead
of binding all those to the model and
having those all live in the system
prompt, they just have a CLI that the
agent can call through a bash tool to
run any of those MCP tools. So I want to
make sure I make this clear. The main
idea is instead of binding a huge number
of different tools to the model
directly, instead bind a small number of
tools like a file system tool, a bash
tool and let the model or agent for
example use the bash tool to can execute
commands to do many other things. So the
action space can be huge even though it
is only making two or three different
tool calls. That's the key insights and
it's a very good one of cloud code.
Think about cloud code. How many
different tools is actually calling? I
mean, I've used it huge amount since
last February. Search, bash, yeah, web
search. I can't think of a huge number
more than that. Very simple that can do
a huge amount because if you just give
it access to a computer, it's extremely
powerful. So, that's a good insight.
Basically, the action space can be
expanded hugely if you give agents
access to a computer effectively. That
that's kind of the key insight. Second
idea there an offloading from like the
system function calls and the Asian
instructions to for example just calling
them directly from for example a
terminal. The final I'll mention briefly
is context isolation. This one's a
little bit more clear but basically when
you have a task that for example is
tokenheavy. You can also offload it to a
sub aent. Let that sub aent perform that
task and then just return some result to
the main agent. We see this a lot. I've
used it extensively indeed research. uh
Anthropic uses this in their multi-agent
researcher. Manis uses it. So very
common approach context isolation
through multi- aent. It's very
intuitive. So those are the three big
ideas reduce offload isolate with a
number of examples from claw code manis
and some my own work on open research.
>> Okay. Another acronym came up there the
MCP lang chain has been really big part
of the emerging ecosystem of LLM
infrastructure and now we're seeing
protocols like MCP become pretty
popular. Can you talk a little about how
this stuff fits together and like what
we all should know?
>> Yeah, this is a really good one. There's
a really good talk from John Welsh at
Enthropic at this year's AI engineer
summit that talked about the origin of
MCP inside enthropy. It's a good
motivating kind of story to how to for
kind of how to think about this. MCP
stands for model context protocol. So,
as we talked about before, models got
really good at tool calling sometime mid
to late last year. And when that
happened, this is internal entropic.
People started writing all sorts of
tools without much coordination. So
there's lots of duplication, many custom
endpoints for different use cases and
all these inconsistent interfaces
basically confused developers, duplicate
functionality, create maintenance
challenges and so forth. The model pro
protocol emerged internally as a
standard protocol to address this
problem and it was open source and
basically is a protocol that allows you
connect tools context prompts to
different LLM applications or it's a
client server model. So it's basically
an MCP server and a client application.
The client in a tangible case like for
me in my day-to-day work I have an MCP
server that services langraph
documentation. I work with Langchain. We
use Langraph. Lang graph is our open
source kind of agent and workflow
framework. I want to write a lot of
Langraph code. I have a little MCP
server that connects to Langraph docs.
That's all it does. But I exposed it as
MCP server. So I could connect it
through the same server to cloud code to
cursor to anthropic cloud desktop app.
So it's like a universal connector
protocol to connect tools, context or
prompts to different applications.
That's really all it is. Now the bigger
picture here that I think is interesting
is relates a little bit more to the
broader ecosystem and lang chain
frameworks. So one of the points that
John made is that standardization
is often very beneficial particularly in
large organizations. This gets a little
bit to maybe some of your audience but
we've seen this a lot with line chain
langraph. The reason why certain
frameworks like lang chain lang graph
protocols like MCP are popular is
because standards are helpful. If you
have large work with many different
people a standard set of tooling is very
beneficial. And that was one of the
reasons why he argued MCP took really
took hold with anthropic is because a
standard was very useful for a lot of
different reasons internally in terms of
kind of author o and security and
consistent documentation and onboarding
and so forth. And we've actually found
that's actually one of the main reasons
people enjoy for example Langraph.
Langraph is a framework for building
agents, building workflows. It's very
popular. It's well supported. There's
good documentation. And so for example,
many organizations building agents on
board to Langraph because it's just a
standard set of low-level tools and
primitives that you can use to build
agents and everyone's speaking roughly
the same language and it works
seamlessly with MCP. So you can build a
langraph agent, you can connect to
tools, you can use MCP to connect to it.
So that langraph agent is basic little
is a little MCP client. You can
connected to MCP servers, no problem. So
they play well together. But I think it
the interesting point though is about
this notion of standardization and why
actually in larger orgs standards are
important and that's what motivates
certain frameworks protocols for kind of
taking hold is my view on it. So now
we've [clears throat] been talking
around this, but I want to talk about
evaluation and in particular like how
people who work a lot on eval
cuz it seems niche, right? Whereas a lot
of the time it's figuring out if your
product works and does what what you
want it to do. And you mentioned that
Manis has rearchitected five times this
year or something along those lines.
>> That's right. And if there is a constant
need and push for re-engineering as
models improve and new models come out,
how can you test if your system is
future proof and how do you think about
evaluation and just making sure it
works?
>> Yeah, so that's actually two maybe two
different interesting threads there. So
one observation is kind of this idea of
you have your system, how do I know it's
going to be resistant to newer and
improving models? So one of the kind of
problems or kind of one of the
challenges that can happen is that I
have some architecture it works very
well with models today and so if models
get better my architecture limits
further improvement of my application.
This can happen for lots of different
reasons and that's one of the major
predictions of the better lesson is
basically the structure we add to our
applications today limits their progress
in the future or limits their
improvement in the future. You can test
this basically by this is a good take
that Manis mentioned that I actually
really like. um you can actually test
against different model capacities even
today. So take your take your system
evaluate it against for example a low
capacity model a midcapacity and then
state-of-the-art and make sure
performance goes up. If performance goes
up with across capacities you can tell
that your harness or your system is
future proof in that sense. So I think
that's point one on eval futurep
proofing your system but point two I
think is interesting and broader broadly
speaking talking to manis talk to a lot
of anthropic people internally as well a
lot of the larger static benchmarks
become saturated very quickly so a lot
of the evaluation for example cla code
they've spoken about this publicly a lot
of it is actually just dog fooding and
direct user feedback in app man
mentioned the same thing a lot of a lot
of their evals are born from direct
inapp app user feedback. So getting your
products out there, having the ability
to capture feedback in app and roll
those into eval sets is often what
people are doing in practice. At
Langchain, we have Langmith. It's a very
featurerich set of evaluation tools. We
have lots of nice tooling to capture
feedback from traces at the eval sets.
And so there's lots of support for that.
But I think that's kind of what we're
seeing is often times people are
attempting to ship product, capture user
feedback, roll that feedback into
evaluation sets is kind of the approach
that I'm seeing more and more which I
think is obvious. I think the only
subtle point is often times these kind
of larger static benchmarks get
saturated very quickly. So you need to
be constantly be surfacing new failure
cases from users rather than relying on
kind of these big fix kind of data sets.
Manis said they they moved away from for
example I think it was like Gaia and
some these other big like QA benchmarks.
I said they saturate relatively quickly
on those.
>> And just to be clear, th this is amazing
uh because relying on user feedback is
wonderful in these cases. But if you're
working in a regulated space and serving
a conversational agent to financial
customers or people coming to an online
pharmacy or something like that, you
can't actually do this.
>> Yeah, that's true. May depend on your
application. So ideally are operating a
domain where you actually can get some
degree of user feedback to assess the
quality of your app. I'm not actually
sure if that's feasible in all domains,
but ideally that would be the case.
>> Yeah. Actually, in your mind, it's
interesting to think through what good
evaluation really requires in this space
because where like the outputs, right,
are so non-deterministic and context
driven. It sounds like there's kind of
these buckets of like user feedback,
large scale eval, and maybe there's a
bit of interchange between the two. Is
it even possible in your mind to build
great agents without sprinkling at least
of those two pieces? Yeah, for every
agent I've built and for most of the
popular production agents, talking to
Manis, talking to Claude folks, eval are
certainly being used across the board.
They are obviously very useful. I think
that the catch is there's been some good
blog posts on this. I'll link a few in
the show notes. It's always just very
important to actually look at your data
and not rely strictly on eval. And I
think there is, we've seen this quite a
bit internally. There's an emphasis on
just dog fooding, getting your
applications to the hands of users,
looking at the raw traces, looking at
user feedback directly rather than
relying on large static eval sets
because the models are changing so fast.
And this is one of the interesting that
came up from talking to from talking to
cloud folks. A lot of cloud code was
really driven by internal dog fooding.
So basically being very aggressive about
shipping updates, dog fooding
internally, collecting feedback very
rapidly, looking at traces and updating
it in that manner. That does seem to be
a common mode of evaluation that we're
seeing across LM applications. And I
think just looking at the raw data,
having a good tracing system in place is
like table stakes. I think that having
high quality evaluation sets is
beneficial. And Hamel Hussein, I'll link
some of his stuff, has posted a lot of
good things on that particular topic.
But I I do think the case and Hamel's
mentioned this quite a bit just setting
up high quality tracing looking at your
data and being very aggressive about
that and shipping at least dog fooding
internally very aggressively is where
you start absolutely and I think it's
very important of course to keep an eye
on your high level business like
evaluation are your metrics you think
you want met actually being met but let
that guide development and I think a
gotcha for a lot of people beginning
building this type of software is
focusing focusing on the generative part
and just thinking about retrieval a lot
of the time it isn't the generative part
which is the issue it's your retrieval
so your evals will guide focusing on on
on retrieval for example right
>> yeah I think a related point there is
actually what I found is when you're
bling out these systems it can be very
beneficial to set up evals for
subcomponents rags retrieval or rag is a
good example if you've retrieval system
you're exactly right that the quality of
the ultimate output is dependent upon
the retrieval itself and then the
generation so you're taking your
retrieving from a vector store or
database, you're passing that to an LM.
The LM's producing an output. You can
actually do an evaluation on the
retrieval itself, and that can be very
beneficial. In my little email app
example, I actually I have eval
component of that. I have an eval for
like just a triage step. I have an eval
for the ultimate responses. I have an
eval for the tool calls. And so, that's
actually a good point. Like when you're
laying out these applications, you can
have eval subcomponents to make sure
that these smaller pieces are working as
expected. And for your ultimate output
maybe for that you of course you of
course maybe have an eval set or two but
you're also doing very aggressive dog
fooding. You're getting it out there
using user feedback to kind of update
your data sets aggressively. So it is
true that in development typically with
these applications you can set up
evaluations or subcomponents just to
make sure they're working well as you
build out your system. You have your
whole system then you ship it. Then
you're more a little bit into like kind
of guard railing with some like online
evals just making sure that you're not
seeing egregiously wrong outputs. So you
might be doing that and then you might
also be just collecting user feedback
directly rolling bad examples into data
sets more on the fly.
>> Totally. So to wrap up, a lot of our
listeners and viewers are AI data
leaders, ML leaders. So I'm wondering
for leaders trying to make sense of the
space, what do you wish more engineering
managers or CTOs understood about how
Geni Gen AI systems actually get built?
>> Yeah. Yeah. Maybe I'll walk through a
few kind of summary principles that
hopefully hit a lot what we talked
about. So I think one, keep things
simple. Use just prompt engineering if
you can get away with it. If you need a
little bit more complexity, then bump up
to a workflow. If you actually can't get
there with a workflow, then you consider
an agent. If the problem is truly more
open-ended, an agent though with try to
minimize the tools, keep it very simple.
If just a single agent can't get you
there, then think about contact
engineering. You could, for example,
offload to multi- aent through if you
needed context isolation for more like
heavy duty tasks. And finally, if all
those things are insufficient, you might
think about fine-tuning or training
models, but really only after all the
others are exhausted. So, I think that's
point one, keep things simple. And I
think it's tricky because you often
might hear, for example, on Twitter or
in the timeline people talking about,
oh, reinforce and finetuning or building
agents and maybe your problem does not
need that and don't increase the
complexity arbitrarily. The second point
is like the bitter lesson thing is
building for rapid model improvements.
So recognize what you build today will
have to be rearchitected kind of
aggressively over time. The manis
example five times in since March my
example of OBD research I rearchitected
that three or four times in a year. You
have to bake in the fact the models are
getting much better and whatever little
kind of crutches you have in your
application today to make it work go
away as that model improves. That's
exactly what is predicted by the better
lesson and you have to be aggressive
about removing those kind of assumptions
or structure as the models get better.
Otherwise you might bottleneck your
performance. So that's a very important
thing to think about and keep in mind
and relate to that. Don't be afraid to
rebuild. Manis rebuilt five times. Cloud
code is constantly rebuilding. I rebuilt
OpenD research three or four times in a
year. So you have to embrace that in
this new era of LLMs. And I also think a
subtlety is that the cost of rebuilding
is much lower with code models. So it's
much faster to rearchitect things.
Another subtle point I'll make is things
that don't work today will work
tomorrow. So cursor is a great example.
this cursor did not work well until claw
35 sonnet and then suddenly the product
experience was unlocked and it obviously
the rest is history. So actually don't
be shy or afraid or whatnot if your
product doesn't quite work yet because
for example with model deficiency
that'll very quickly get removed as
models get better and I think curs is a
great example of that. Yeah and maybe
the final point is just be wary of
rushing to train models. It can be
tempting. It can be really charismatic
to think about applying fine-tuning for
your domain. But often times these the
frontier models are getting so good so
quickly. You can take all this time to
collect a data set, train a model, and
then actually you get bitter lesson
because the frontier model kind of
encapsulates the capability that you
fine-tune for. Then you waste all that
time. I'll give you an example of that
very specifically. Two years ago,
structured outputs weren't great from
frontier models and people were doing
fine-tuning for structured outputs. And
yeah, it's all relevant today for the
vast majority of use cases is irrelevant
because the LM providers have gotten
extremely good at structured outputs and
complex nested schemas and JSON mode and
so forth. So just an example. So keep it
simple. Build for rapid model
improvement. Don't be scared to rebuild.
Things that work today won't work
tomorrow. Don't rush to train models. I
think are like the five things that I
would leave you with.
>> Fantastic. What wonderful lessons both
from all your time building, but
everything you've seen happen in the
space working online chain as well.
Thank you for bringing us your wisdom
and expertise and for your time as well,
Lance. This has been super fun.
>> Great to be here. Great to see Duncan
again and great to meet you, Hugo.
Hopefully I make it up to Sydney
sometime and we can get in the water.
>> Go for a surf together.
>> Yeah.
>> Thanks so much for listening to High
Signal brought to you by Delfina. If you
enjoyed this episode, don't forget to
sign up for our newsletter, follow us on
YouTube, and share the podcast with your
friends and colleagues. Like and
subscribe on YouTube and give us five
stars and a review on iTunes and
Spotify. This will help us bring you
more of the conversations you love. All
the links are in the show notes. We'll
catch you next time.