So for episode 4, I got Tanya Dixit on the podcast, a forward deployed engineer at Google, to sort out which half is right (spoiler: neither, entirely). Yes, also called Tanya. No, we did not plan that but the transcript was a nightmare to edit.

Tanya’s route here was not a tidy straight line. She started out building satellite hardware in embedded systems, back before deep learning was cool enough to have its own hoodie. Then the math from her undergrad started clicking, she did a nano-degree, and somehow that spiralled into a master’s from ANU (university medal, no big deal) and a career that’s touched everything from enterprise AI to bushfire risk modelling. By the time she landed on forward deployed engineering, she’d basically done a full lap of the ML lifecycle. Build it, prove it, deploy it, fix it, repeat.

This is why the conversation doesn’t stay in the shallow end. We get into what her actual week looks like, and the coding-to-customer-calls ratio is not what most people guess. If you assume “forward deployed” is code for “technical enough to be in the room but not enough to be at the keyboard”... you’re in for a surprise.

We also get properly into the thing everyone eyeing this career path wants to know: where does an FDE’s job stop and the product team’s job start? Tanya draws that line with far more precision than I expected going in, and her answer for why companies are suddenly hiring for this role like it’s going out of fashion made a few things click for me. Turns out standard SaaS playbooks and AI products are not playing the same game at all.

Then there’s the bit I haven’t stopped thinking about since we recorded. As models get better and coding gets, in her words, increasingly commoditised, does that mean the industry just needs fewer FDEs? She doesn’t reach for the easy optimist answer, and she doesn’t reach for the easy doom answer either. What she gives instead is a proper mental model for which parts of this job survive and which don’t... and yes, we did end up in recursive self-improvement territory, because apparently no 2026 tech conversation is complete without it.

If you're building AI products, working anywhere near enterprise customers, or just trying to work out where the humans still fit in this whole picture, do yourself a favour and watch the full thing rather than skimming for quotes.

The full episode is live now with audio and video. Go watch it, and if Tanya’s take here makes you want more, she’s finally starting to write on Substack too.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Two Fridays ago, our agent checked the weather. Last week, we gave it a plan and taught the harness to tap it on the shoulder when it wandered. This week, the same loop gets dropped into a codebase.

In the final part of Inside the Harness, we see how a change of tools creates a coding agent, then deal with what happens when its context and capabilities no longer fit. The loop stays put. The harness grows around it.

This also concludes our Friday scheduling experiment. Normal service resumes next issue. But before we dive in…

Leave the harness exactly where it is. Swap the tools, point it at a codebase, and the same loop becomes a coding agent.

Same harness, new tools

The experiment here is deliberately simple: keep the harness and its system prompt as they are, and change only the tools. If the harness skeleton is genuinely generic, that one swap should be enough to turn a research agent into a coding agent.

The system prompt, therefore, stays almost untouched:

Only the toolset changes. Out go the research tools; in come three tools that let the agent work directly with a codebase, defined here in JSON:

That’s it! read_file inspects the codebase, write_file changes it, and bash runs commands. Nothing else in the loop moves.

How this fixes a bug

Say the user asks the agent to fix a failing test. The run looks roughly like this:

That’s a complete code-change loop: run the test, inspect the source, make the change, then verify it. Underneath, it is still ReAct—think → tool use → tool result → think. No new control flow required.

Drawn out:

Three tools and one loop are enough for a straightforward bug fix. For something larger, such as a feature spanning several files, add the Plan-then-Act layer from Part 2. The job changes; the underlying structure does not.

That is the point: the harness skeleton is generic, while the tools determine the agent’s shape. Replace web_search and web_fetch with read_file, write_file, and bash, and a research agent becomes a coding agent.

Speaking of context cost — some tools’ return values are naturally huge. A bash build can dump thousands of lines of stdout. A read_file on a real file is hundreds of lines easy. A web_fetch against a real article is tens of thousands of tokens in one shot. Stuff all of that into context as-is and ten turns later the window pops.

For less hype and more engineering, pull up a chair.

Subscribe now

Why does write_file support appending?

It is less about convenience than context. Without an append option, adding one line to a 1,000-line file could mean sending the entire file through the tool call again—and leaving all 1,000 lines sitting in the context window. Incremental writes avoid paying that cost for a tiny change.

The problem gets worse with tool results. A build can produce thousands of lines of output, while reading a real file can consume hundreds more. Keep feeding all of that back into the loop, and even a generous context window begins to look rather less generous.

The harness now needs somewhere else to put it. That brings us to context offloading.

Offloading: When context won’t fit

Every tool result goes back into the message history. A repository-wide search or noisy build can add thousands of tokens in a single call—useful in the moment, dead weight a few turns later.

Even a 200k-token context window begins to look rather less generous during a serious coding task. As it fills, the agent can lose track of earlier details, contradict itself, or simply run out of room.

The harness layer’s first answer is context offloading.

How offloading works

The rule is simple: when a tool result exceeds a set size, the harness writes the raw content to disk and leaves a path in the context. If the agent needs the details later, it retrieves them with read_file.

Suppose a bash call produces 1,000 tokens. The tool result the agent receives might look like this:

The 1,000 tokens never enter the agent’s context. The harness intercepts the output and replaces it with a roughly 30-token pointer.

From the agent’s perspective, nothing else changes. It still calls bash, receives a tool result, and decides whether it needs to read the file. No new tool. No change to the ReAct loop. Just a smaller payload.

Offloading is not compression

It is easy to confuse offloading with context compression. Both make room in the context window, but they do it differently:

They diverge on three axes:

• Trigger timing: Offloading happens the moment the tool_result is born — the harness sees the return is over threshold and externalizes it on the spot. Compression happens the moment context is about to blow — only when occupancy is near the ceiling does the harness go back, scan history, and summarize.

• Lossy vs. lossless: Offloading is lossless and recoverable — not a byte of the original is lost; only the storage location changed, and the agent gets it back verbatim with read_file. Compression is lossy — summarize crushes 10 turns of conversation into a 200-word summary, and the dropped info is gone for good.

• Layer of abstraction: Offloading lives at the mechanism layer, handling hot/cold tiering of individual tool_results. Compression lives at the last-resort layer, packing whole stretches of history into archives. One is like the OS’s paging scheduler; the other is like tarring up old logs.

The framing I like is this: offloading is hot/cold tiered storage; Compression is archival compression. The former goes first because it’s lossless and cheap. The latter is the fallback because it’s lossy but life-saving.

That is offloading’s place in the harness: when tool results are too large, move them instead of shrinking them.

But fitting the agent’s working data solves only half the problem. Its capabilities may not fit either. That’s where Skill comes in.

Skill: Capability organization at the harness layer

So far, the harness has been helping one agent complete one task. Production introduces a different problem: the same agent may need dozens—or hundreds—of specialist capabilities. It might write Python one moment and diagnose a Kubernetes alert the next, each with a different set of instructions and pitfalls.

Loading all that guidance into the system prompt does not scale. If one Skill takes 3,000 tokens, fifty of them consume 150,000 tokens before the task has even begun. Even if they fit, the model must reason through a wall of mostly irrelevant instructions on every turn.

The harness’s answer to this capability explosion is Skill: make every capability available without loading all of them into context at once.

Load only the index at startup

Skill uses a simple form of progressive loading. Each Skill lives in a folder containing a SKILL.md file. At startup, the harness reads only the file’s frontmatter—a small metadata header containing its name and description—and adds that information to the system prompt’s <skills> index.

The full instructions remain on disk until the agent decides it needs them.

Here is the <skill-system> block from Claude Code:

What the prompt leaves out

Two absences matter:

• No new tool. The harness does not need an invoke_skill() or load_skill() function. When the model selects a Skill, it uses the coding agent’s existing read_file tool to load the relevant SKILL.md. Skill simply reuses the filesystem.

• No Skill body. The index contains only a name, description, and path. The full instructions stay on disk until they are needed.

That is progressive disclosure at the harness layer: keep the map in context, then load the instructions on demand.

Why not retrieve the right Skill?

Why not treat Skill selection as a search problem? Use BM25 or embeddings to match the user’s request against every Skill description, load the top results, and move on.

That can work—but if retrieval happens only once, the candidate set is fixed before the agent fully understands the task.

“Debug the alert that just fired” might begin with checking logs. Those logs may point towards a recent deployment, and resolving the incident may eventually require a company-specific postmortem. Each Skill becomes relevant at a different point in the reasoning chain.

Keeping the compact <skills> index in the system prompt lets the model reconsider its options on every ReAct turn. Retrieval could also be rerun each time, so the real choice is not search versus reasoning. It is one-shot selection versus selection that evolves with the task.

The full sequence

Startup, selection, and loading look like this:

The one line in this sequence diagram worth observing is the read_file call in the middle. It uses the same generic tool the Coding Agent already had. No new abstraction. Harness support for Skill costs zero net new tool surface.

Skill on the harness map

Skill does not change the ReAct loop or add another tool. It simply separates what the agent should always see from what it should load only when needed.

That small act of deferral is what lets an agent carry a hundred capabilities without dragging a hundred playbooks through every turn. With that, the harness map is complete.

Closing the loop on Inside the Harness

Over the past three Fridays, we’ve taken the agent harness apart together, one mechanism at a time. If you’ve been with us since that first weather forecast, thank you for following the loop all the way through.

By now, a harness should look less like black-box middleware and more like a stack of small mechanisms:

• ReAct makes the agent move. It supplies the basic rhythm: reason, act, observe, repeat.

• Plan-then-Act gives it direction. The agent works from an explicit plan instead of chasing whatever catches its attention next.

• Nudge keeps it on track. The harness brings the plan back into focus when it begins to fade.

• Offloading protects the context window. Oversized results move elsewhere without being permanently discarded.

• Skill makes capabilities manageable. The index stays visible; the full instructions appear only when needed.

Each layer answers a failure exposed by the one beneath it. Each does one job, and you can tune each without rebuilding the entire system.

That is the larger point of harness engineering. The ReAct loop makes an agent run. The harness is what gives it a chance of surviving contact with reality.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

The more conversations we have with people building these systems, the more useful material we find that cannot be contained within a single weekly issue. Rather than leave those sessions sitting in an archive or force them into our usual publishing cadence, we’ve decided to make a little more room in the week for them.

Between our regular issues, we’ll occasionally share technical sessions that deserve more than a forgotten recording link. A couple of weeks ago, we brought you Nicole Königstein’s webinar on building AI agent harnesses for finance. This time, it’s Imran Ahmad’s free webinar on the architecture of Agentic RAG.

Imran is a data scientist, AI architect, visiting professor, and the author of 30 Agents Every AI Engineer Must Build. His work looks beyond agent demos to the decisions that matter in production: when to add agency, where to place evaluation, and whether another loop is worth the extra cost and latency.

We’ve also put together a free take-home pack with Imran’s complete slide deck, a practical guide to 10 production agent patterns, and a one-page map of 30 agent architectures. You’ll find it later in the article.

Enough preamble. Let’s get into it!

Traditional RAG has an awkward habit: it keeps going.

Ask it about an internal company acronym, and it retrieves the closest available chunks. Misspell that acronym, and it still retrieves the closest available chunks. Those chunks are passed to the model, and the model produces an answer that may sound perfectly credible—even when the information needed to answer the question was never found.

Every component completed its assigned task. The system still failed.

That is the problem Imran explores in this free Agentic Engineering webinar on the architecture of agentic RAG.

His argument is not that conventional RAG is broken, nor that every retrieval pipeline now needs an agent. It is that traditional RAG is largely a one-way street. Once retrieval begins, there is often no meaningful point at which the system can inspect what it found, recognise that it is insufficient, and change course.

Agentic RAG introduces that missing loop.

RAG is really automated context generation

Large language models know a great deal, but they do not know everything about your organisation.

They may not know your internal terminology, current policies, proprietary research, or the particular meaning your company has assigned to an acronym. RAG addresses this by finding relevant information in an external knowledge source and adding it to the model’s prompt.

In other words, RAG automates context generation.

The familiar pipeline looks something like this: documents are divided into chunks, converted into embeddings, and stored in a vector database. When someone asks a question, the system searches for similar chunks, places them in the prompt, and asks the model to generate an answer from that context.

It is a useful architecture. There is a reason it became one of the first patterns companies reached for when building applications with large language models.

But its simplicity relies on several assumptions: that the user’s question is also a good search query, that the first retrieval will find the right evidence, that similarity means relevance, and that adding more context will improve the answer.

None of those assumptions is reliably true.

The problem with a one-way pipeline

Consider a user asking about an internal tool but misspelling its name.

The vector search does not necessarily respond with “I can’t find that.” It returns whichever chunks happen to be mathematically closest. A reranker may rearrange those chunks, but it is still ranking a weak set of results. The model then receives that material and does what it was designed to do: generate a plausible response.

The failure began before generation. The model was handed evidence that never supported the question.

Traditional RAG has no natural recovery mechanism here. It retrieves once and moves forward. It does not necessarily ask whether the query was misunderstood, whether a different source should be searched, or whether the available evidence is strong enough to justify an answer.

Adding an agent changes the shape of that pipeline. Retrieval can become a loop:

Plan

↓

Retrieve

↓

Evaluate

↓

Adapt

↓

Try again

That sounds like a small architectural change. In practice, however, it changes where the system is allowed to exercise judgement.

Add agency where a specific failure needs it

“Agentic RAG” can make it sound as though the entire system must suddenly become autonomous. That is not the useful way to think about it.

Agency can be introduced at different points in the pipeline, depending on the failure you are trying to correct.

At the retrieval stage, an agent can interpret the user’s request, break a complicated question into smaller searches, rewrite a weak query, or choose between available retrieval tools.

After retrieval, it can examine the evidence and ask whether the results are truly relevant. If important information is missing, it can search again rather than blindly passing weak context downstream.

At the generation stage, another check can compare the final answer with the retrieved evidence, add citations, or reject claims that the sources do not support.

You may not need all three.

If retrieval is the main weakness, putting another agent around generation could add cost without fixing the actual problem. If the answer is poorly grounded despite strong retrieval, changing the search strategy may not help.

The important question is not, “How do we make this RAG pipeline agentic?”

It is, “Where does our existing pipeline fail, and would adding judgement at that point help it recover?”

Imran’s own course materials put the principle neatly: “Only add agency when you can name the failure it fixes.” The accompanying Agentic RAG course hub maps different failure modes to patterns including adaptive retrieval, corrective RAG, GraphRAG and groundedness checks.

A useful system needs the courage to say “I don’t know”

One of the most valuable behaviours in a knowledge system may also be one of the least impressive in a demo: refusing to answer.

If the retrieved passages do not contain the requested information, a reliable system should be able to say so. It might ask the user to clarify an acronym, try another source or explain that it could not find enough evidence.

That is much more useful than producing a polished answer from irrelevant material.

An evaluator inside the retrieval loop can provide that gate. It can score the evidence, check whether it covers the question, and decide whether the system should proceed, search again, or stop.

Of course, adding an evaluator does not make the system correct. The evaluator can make a poor judgement too. Its decisions need test cases, thresholds, and traces of their own.

The improvement is architectural rather than magical: the system now has a place where failure can be noticed.

More context is not always better context

Another assumption behind basic RAG is that retrieving more information gives the model a better chance of answering accurately.

Initially, it often does. But the relationship does not continue indefinitely.

As more chunks enter the prompt, relevant evidence competes with irrelevant material. Different sources may contradict each other. A precise but lengthy passage can pull the model away from the user’s original question. Imran describes this as a signal-and-noise problem: the goal is not to maximise context but to preserve the right signal.

Large context windows do not remove this problem. Being able to place the equivalent of several books into a prompt does not mean the model will identify the right paragraph, weigh every source correctly, or remain focused on the original task.

This is where an agent can help curate context rather than merely accumulate it. It can search for missing evidence, reject weak results, and recognise contradictions before generation.

But again, that judgement has a price.

Retrieval should reflect the question being asked

A short factual question and a request to explain a complete block of code do not require the same kind of context.

With a fixed RAG pipeline, they may still be treated in much the same way. The same chunk size, overlap, retrieval algorithm, and number of results are used regardless of the question.

An agentic retrieval layer can make the process more adaptive. Depending on the tools and indexes available, it can choose a retrieval strategy, change the query, search multiple sources, or gather a wider section of material when the answer depends on information spread across several passages.

For a compound question, it might separate the request into several searches and confirm that each part has been answered. For a vague query, it might rewrite the search or ask the user for clarification. For a technical question, it may retrieve a larger, more coherent unit rather than an isolated fragment.

The underlying trade-offs do not disappear. Larger chunks carry more context but reduce precision. Smaller chunks may retrieve precisely while cutting away something essential. Semantic chunking can preserve meaning, but it introduces more computation and complexity.

Agency allows the system to make choices within those trade-offs. It does not abolish them.

If the system can change course, you need to see where it went

A fixed pipeline is relatively easy to follow. The query went in, a set of chunks came back, and the model generated an answer.

An agentic system may rewrite the query, call more than one retriever, reject the first result, alter its plan, and try again. That adaptability is the point—but it also makes the system harder to inspect.

Imran recommends a provenance layer that records the path the system took. What did it search for? Which evidence did it retrieve? Why was one result accepted and another rejected? Which tool did it choose? Where did the final claim come from?

Without those traces, it becomes difficult to distinguish a retrieval failure from an evaluation failure or a generation failure. The system may be more capable, but the team responsible for it will know less about why it behaved as it did.

Observability is therefore not an optional dashboard added after the architecture is complete. It is part of what makes an adaptive retrieval loop governable.

Agency sends a bill

Each planning step, evaluation, and retry may require another model call. That adds latency, token usage, and another possible point of failure.

A system that runs three searches and evaluates each one may retrieve better evidence than a one-shot pipeline. It may also take longer and cost considerably more.

Imran’s advice is refreshingly blunt: “Don’t hire a PhD for the job of a receptionist.”

A small, focused model may be enough to classify a query, choose a retriever, or judge whether two passages are relevant. Some of these decisions can be handled by local or quantised models. Others may not require an LLM at all.

The most sophisticated model should not automatically be placed inside every part of the loop. The model—and the loop itself—should be proportionate to the decision being made.

This is where agentic RAG becomes an engineering problem rather than a naming exercise. The team has to decide where another round of reasoning is worth its cost.

The goal is not maximum agency

The choice is not between an outdated RAG pipeline and a fully autonomous knowledge agent.

There is a spectrum between them.

A conventional pipeline may be entirely sufficient for a narrow, predictable knowledge base. A retrieval evaluator may solve the most serious failure without changing anything else. A more complex investigation might justify query decomposition, multiple retrieval tools, and several rounds of evidence gathering.

The right architecture is the smallest one that can recognise and recover from the failures that matter.

That is the real case for agentic RAG. It does not simply retrieve more information or attach an agent to a familiar acronym. It gives the system a chance to notice that the evidence is weak before fluent language turns that weakness into an answer.

Your webinar companion pack

• Explore the hands-on labs and supporting material on GitHub.

• Download The Agent Engineer’s Field Guide for 10 production agent patterns, including when to use them, what tends to break first, and how to evaluate them.

• Download 30 Agents at a Glance for a one-page map of all 30 architectures, organised from foundational patterns to specialised and domain-specific systems.

• Download Imran’s slide deck to revisit the architectures, retrieval loops, and implementation patterns later.

• Continue through the patterns in Imran’s book, 30 Agents Every AI Engineer Must Build.

For less hype and more engineering, pull up a chair.

Subscribe now

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Throw a complex question at pure multi-turn ReAct, and a structural flaw shows up — the model walks step by step with no global view. Hit it with a research question that needs five or six angles assembled together, and it’ll grab the most obvious two or three in the first few steps and call it done. It doesn’t even know which angles it skipped.

The fix is crude but effective: make it write the plan down first, then execute.

A new tool: write_todos

In the Deep Research harness, we add one new tool: write_todos. It does nothing! It literally just writes a TODO list into the agent’s state. The flip switch is one line in the system prompt:

You’re a helpful agent.

Look at the line inside <task>: Use \write_todos `to make a plan and break down tasks.` That single sentence flips the model from “see one step, take one step” to “plan first, then act.”

The tool’s schema is dirt simple:

The new behavior

Add this one tool and the loop visibly changes shape. Take a sample question: “research the current state of Chinese EV exports to Europe in 2026.” The transcript now looks like:

The biggest difference vs. pure multi-turn ReAct is that write_todos call right at the top — it pins down the exam outline upfront. If execution skips a dimension, there’s an unchecked pending item sitting on the plan. The model comes back and picks it up.

Building agents that need to work beyond the demo? Join us for the engineering that holds up in production.

Subscribe now

Side-by-side helps. Pure multi-turn ReAct first:

Each step only looks at the current tool_result. No looking back, no looking ahead. Now Plan-then-Act:

What’s new: a plan at the start, and a state-update after each step. The loop itself didn’t change. We just added one tool and one line of system prompt.

That’s the Plan-then-Act skeleton. In practice, you’ll run into two small pitfalls:

• First, on unfamiliar topics, the plan goes off the rails. The model has only a vague grasp of the field. The four sub-tasks it lists from that half-baked understanding might miss the most important dimension from the start.

• Second, mid-run, the model forgets to update todos. It finishes task two but doesn’t call write_todos to mark it completed, and the new sub-task it discovered along the way never makes it onto the list. The plan quietly decays.

Two small patches, one for each.

Patch A: Brief yourself before planning

The first pitfall is easy to grok — the model can’t plan something it doesn’t recognize.

Say the user throws a niche term at it, like “research how pp-ocr performs on long-tail Chinese classical text.” The model has only a fuzzy notion of pp-ocr (”Baidu’s open-source OCR engine, I think?”). If it goes straight to write_todos, it’ll probably list four lukewarm items: figure out what it is, check performance, list pros and cons, summarize. The dimensions that actually matter (its multilingual branches, version history, benchmark against a specific competitor) never get touched.

One-time briefing

The patch is light. In the system prompt, explicitly allow the agent one web_search before planning, purely to build basic familiarity with the topic:

Two key phrases here: one-time and before planning. This isn’t a blanket “search whenever you want” — it’s a hard-coded protocol in the system prompt: before you plan, you can do exactly one web_search to brief yourself.

With that briefing, the model searches “pp-ocr” once, skims the first few snippets, learns it’s a PaddlePaddle-family OCR model with v1 through v5, multilingual branches for Chinese / English / Japanese / Korean, server-side and lightweight variants… then writes the todos. The plan turns from “look up what it is” (useless) into “compare v4 vs v5 recognition accuracy on handwritten classical text” (actionable).

That patches the input side of planning. The next pitfall is on the output side — the model forgets to update its own todos.

Patch B: Nudge after every step

The second pitfall — the model forgets to maintain its todos.

You’ve seen this. The opening plan lists four neat items. Item one gets done; the model jumps to item two without updating anything. Halfway through item three, it realizes it needs a new sub-task, but instead of appending it to the list, it just searches. By the end of the run, the plan state looks the same as it did after step one. Updated once, never again.

Putting “remember to update todos” in the system prompt isn’t enough to fix this. The prompt is a one-shot backdrop. After a few turns, the model’s attention weights have shifted to the more recent tool_results. The backdrop’s reminders fade.

Mechanism: nudge

The harness’s solution is called a nudge — after every tool_result, when the todo list is non-empty, the harness hard-codes an extra reminder onto the context for the model:

This isn’t a user message, and it isn’t an assistant message the model generated. It’s a system-reminder spliced in by the harness right after the tool_result. From the model’s perspective: every time it finishes a step and is about to plan its next move, this line is in its face.

In sequence diagram form:

Two harness actions worth noticing: one, pass the tool_result back to the LLM as-is; two, when “current todo list is non-empty,” also append a system-reminder. Both arrive in the same message. From the model’s view, it looks as if the tool return came bundled with a built-in nudge.

And the nudge fires on every tool_result. Not just at the start, not every few rounds. As long as the plan isn’t done, it keeps firing. The model’s attention gets pulled back to “what’s my plan progress?” again and again, and the probability of forgetting to update drops sharply.

Patch A covers the input side of planning. Patch B covers the output side. Together with the write_todos tool itself, the harness for Deep Research is done — from single-turn ReAct, to multi-turn loop, to Plan-then-Act with two patches. That’s all of it.

But what does the same skeleton look like with a different toolset? That’s the crucial question we’ll answer in Part 3, where we cut to coding… then follow the loop until its tool results and capabilities no longer fit.

Editor’s note

Agent harnesses have acquired a remarkable amount of jargon for something often built around a simple ReAct loop: think, call a tool, inspect the result, repeat.

Over the next three issues, we’re taking that loop apart and watching the harness grow around it as the problems get harder. Part 1 strips the ReAct loop back to its basics. Part 2 adds planning and nudges to help keep it on course. Part 3 turns the same loop into a coding agent, then tackles what happens when its context and capabilities no longer fit.

We’re publishing all three parts on Fridays to see whether a technical deep dive fares better once the midweek inbox stampede has passed. After Part 3, we’ll return to our usual schedule.

Crack open any agent thread these days, and the buzzwords fly: Multi-Agent, Deep Research, Skill, Sub-Agent, Orchestrator… each one more abstract and more intimidating than the last. But peel the fancy wrapping off, layer by layer, and what’s actually running underneath is the same loop: the model thinks, calls a tool, looks at the result, thinks again. A bare-bones ReAct loop.

The staircase looks roughly like this:

Each step adds almost nothing to the last. Multi-turn just lets the loop spin more.

Plan gives the loop a persistent TODO list. Coding Agent swaps web_search for read_file and bash. Offloading and Skill are two different compromises for when stuff stops fitting in the context window. The point of internalising a harness isn’t memorising each level’s name but rather seeing clearly what problem each level solves that the level below it didn’t.

That’s the path this series takes. We start with the weather forecast.

Single-turn ReAct loop: Weather forecast

Picture the smallest possible scenario: the user asks, “is it cold in Beijing today?” We give the model a single tool: get_weather(city).

The model takes one look at the question and makes one decision: “I can’t answer this on my own, I need to look it up.” So it emits a tool_use: get_weather(city=”Beijing”). The harness picks up that tool_use, actually runs the function, takes the result (say {”temp”: 3, “condition”: “sunny”}), and shoves it back into the message queue as a tool_result. The model takes another look and decides whether to call another tool or answer the user. If it answers, what it emits is plain text (no tool_use) and the loop ends.

That’s one full ReAct round-trip: Reason (decide what to call) → Act (emit tool_call) → Observe (look at tool_result), then either go again or hand back the final answer.

What the harness actually is

Strip it down, and the harness is almost embarrassingly simple:

The harness doesn’t ask “have we searched enough?” or “should we try a different keyword?” The only thing it checks is: does this assistant turn contain a tool_call? If yes, execute. If no, exit. All the thinking is the model’s; the harness is just a courier.

Emitting multiple tool_calls in one turn

The interesting bit comes with the next question: “which is colder, Beijing or Shanghai?”

A competent model isn’t going to look up Beijing first and then Shanghai. It emits two tool_use calls in the same assistant turn:

On the harness side, as long as you wrote run_tools to execute in parallel (the run_tools_parallel in the pseudocode above), both queries fire at once. Two tool_results come back together. Next turn, the model sees them side by side and answers the comparison directly.

This is a property of ReAct that people consistently underrate: a single assistant turn can fan out to any number of tool_calls. When you write your harness, don’t default to “one at a time” — the moment you hit “compare three cities” or “read ten files in parallel,” it’ll be painfully slow.

Is single-turn enough?

For a straightforward weather check, yes. But ask for Beijing’s chance of rain this Friday compared with the same week over the past five years, and one lookup no longer does the job.

The model has to produce the forecast, retrieve the historical data, compare the two, and perhaps go back to fill in anything it missed. It needs a multi-turn loop, with each step shaped by what the previous one returned.

Multi-turn loop: Deep research

Swap get_weather for two more general tools (web_search and web_fetch) and you’ve got the skeleton of Deep Research.

Two tools, but the behaviour space they open up jumps by orders of magnitude. The model can search, pull back a pile of candidate links, pick the most relevant one or two and fetch the full text, realise it’s missing something and search again with a different keyword, fetch the next link. The whole trajectory is meandering, stop-and-go. How many steps, when to pivot — entirely the model’s call.

In a multi-turn run, every tool call depends on what the previous one returned. Search and retrieval interleave, and the trajectory is not known in advance: the agent might converge in three turns or still be missing pieces at fifteen.

The shape of a multi-turn loop

Drawn as a sequence diagram, the shape is clean:

The harness code is unchanged. The loop simply spins more times.

Who decides when the loop stops?

This is the most counterintuitive part of multi-turn ReAct: the LLM decides when to halt, not the harness.

Whether the loop continues or not is the model’s call. If it thinks it doesn’t have enough, it emits another web_search. If it thinks it does, it skips tool_use and just produces text. The harness doesn’t get a vote.

What’s still missing

The multi-turn loop solves “one step isn’t enough.” But it has an obvious flaw: the model is purely reactive — wherever its eyes land, that’s where it goes next. No global view. Easy to wander off. A complex question that really should be researched along five dimensions might trap it on the first dimension for seven or eight searches while the others get forgotten. Or it picks a bad keyword early and turns spinning in the wrong direction, eating tokens and producing nothing.

Pure ReAct can keep moving, but it cannot guarantee that it is going anywhere. It has tactics, but no strategy. And an agent without a strategy is just a rather expensive way to walk in circles.

In Part 2 of this series, we give it a plan.

Agentic Engineering grew out of a desire to create something useful for people trying to make sense of agentic AI and beyond. If you’d like to support that direction, please tell your friends about us.

Share Agentic Engineering

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

A quick programming detour from us this week.

If you regularly read Agentic Engineering, you know our core issues tend to follow a fairly deliberate cadence. This isn’t one of those issues. Yesterday, we hosted a free live session with Nicole Königstein on building AI agent harnesses for finance, and rather than let a useful conversation disappear into the webinar graveyard, we wanted to make the full recording freely available here.

Nicole is an AI researcher working on agentic systems and transformers, an instructor, consultant, and O’Reilly author. She teaches large language models and agentic architectures for organizations including the University of Oxford and O’Reilly Media, and much of her work focuses on the considerably harder problem that begins after an AI prototype works: making the resulting system reliable enough for production.

Enough preamble. Let’s get into the harness.

The conversation around agents has moved quickly from prompts to context, loops, orchestration, and now harnesses. But finance is a particularly unforgiving place to discover that a capable model and a reliable system are two very different things.

Nicole spent the session unpacking what has to exist around the model: coordination, verification, observability, control, model selection, evaluation, and the boundaries between agents. Her central argument is a useful one for anyone building agents, whether or not you work in finance: an agent is the model plus the harness.

And once you start thinking about agents that way, reliability becomes less about finding a smarter model and much more about engineering the system around it.

The video above contains the complete session and Q&A. Below, we’ve turned the central ideas from Nicole’s talk into a written companion for those who would rather read or return to the concepts later.

The AI engineering problem has moved

For a long time, the dominant pattern in machine learning was fairly straightforward: task → model → output. If performance was poor, you improved the model.

Agents change that equation. Once tools, retrieval, memory, multiple agents, and decision points are incorporated into the architecture, the question shifts from “Can the model solve this task?” to “Can the system execute a stable sequence of decisions?”

That shift also explains the move from prompt engineering to context engineering and, now, harness engineering. Prompt engineering optimizes the input. Context engineering optimizes what the model sees. Harness engineering deals with everything that governs how the system actually operates: orchestration, tools, memory, retrieval, verification, governance, security, feedback, and execution loops.

The result is a different operating model: execute → observe → verify → correct → repeat. The model still matters, but it is now one component inside a much larger engineering problem.

Strong agents can still make a weak system

This was perhaps the most important warning in the session. It is tempting to assume that if every individual agent in a multi-agent system performs well, the resulting system should perform well too. It doesn’t necessarily follow.

Nicole points to the compounding reliability problem: when independent components execute sequentially, overall system success depends on the success probabilities across those components. Errors can compound as the chain becomes longer.

Imagine an agentic financial workflow involving ten decisions. Even if the individual components are reasonably capable, failures at one stage can affect everything downstream. Add retries as the default recovery mechanism, and the system may eventually produce an answer, but now you have another problem: cost and latency.

Nicole describes three consequences of trying to brute-force reliability through repeated attempts: more compute, more money, and more latency.

And average latency can hide the real production experience. A system with an acceptable mean response time but poor p95 or p99 latency can still feel broken to a meaningful portion of its users.

Reliability, in other words, cannot simply be retried into existence.

The difficult part often lives between agents

One of the more useful distinctions Nicole makes is between communication and coordination.

Communication is the exchange of messages. Coordination controls what the agents do. That matters because an agent handoff isn’t equivalent to a conventional function call.

When one agent hands something to another, the receiving agent has to interpret the message. A schema can ensure that the message has the right structure, but it cannot guarantee that the meaning survived the handoff intact.

The same applies to delegation. Calling a sub-agent isn’t simply calling another tool. You are introducing another probabilistic inference process into the system. That means the boundaries between agents deserve their own engineering attention.

Nicole’s argument here is particularly important for evaluation: if you evaluate only the final output, you may completely miss a coordination problem occurring within the system.

You need signals around the handoffs themselves. You need telemetry. You need to understand whether agents are communicating correctly, whether coordination is progressing as intended, and where meaning begins to drift.

As Nicole puts it, you can’t improve coordination using output-only feedback.

Maybe the model doesn’t need to change

This leads to a more interesting possibility.

A great deal of AI development still revolves around the model: use a more capable model, fine-tune it, or wait for the next generation. But Nicole’s research explores another direction: keep the model fixed and improve the harness around it.

The adjustable state of the system can move away from model weights and into the surrounding harness. Execution traces and trajectories can be scored, fed back into the system, and used to improve how it behaves.

Nicole took this further in her own open-source work.

Rather than hard-coding every coordination choice before the system has even executed a task, she began experimenting with learned coordination: allowing aspects of the harness to become adaptive, auditable, and transferable.

In her experiments, the benefits were particularly visible on coordination-heavy tasks, where she reported improvements in reliability and quality alongside lower costs. Simpler procedural tasks saw less improvement.

There is an important engineering principle hiding in there: not every task needs the same agent architecture.

Stop giving every job to your strongest model

That principle also applies to model selection. There is an understandable instinct when building agentic systems to reach for the strongest reasoning model available, particularly for high-stakes domains. But stronger and more expensive everywhere does not mean better.

In Nicole’s coding experiments, harder, coordination-heavy tasks benefited from a stronger planner, while simpler stages could be handled by cheaper models. She found that distributing models according to the demands of each stage could reduce token consumption and improve speed. More surprisingly, in her experiments, the resulting code was also judged better than code produced using the top-tier model at maximum effort throughout.

This isn’t an argument for universally choosing smaller models. It is an argument for matching capability to responsibility.

Nicole offered a useful mental model during the Q&A: think about building a multi-agent system the way you would build a team.

If you were staffing the equivalent workflow inside an organization, you probably wouldn’t hire the same person for every role. Some jobs require deep expertise. Others require analysis, verification, classification, retrieval, or straightforward execution.

The skills should be complementary. Agent architectures can be designed the same way.

Coding agents need harnesses too

There was also a useful detour into coding agents.

Nicole described an experience in which an AI coding assistant, used without sufficiently strict instructions and boundaries, altered working agent code and introduced serious bugs and security problems.

Her takeaway was not that coding agents shouldn’t be used. Quite the opposite.

It was that using them effectively requires the same engineering discipline we expect elsewhere in an agentic system: clear instructions, guardrails, skills, hooks, review, and people who understand the system well enough to recognize when generated code is wrong.

There is a broader lesson here. AI can accelerate implementation, but it does not remove the need to know what good implementation looks like.

What this means for financial agents

Finance makes all of these problems harder to ignore.

A research assistant, portfolio-analysis system, financial-document agent, or other agentic workflow may need to retrieve evidence, coordinate multiple reasoning steps, invoke tools, maintain state, validate intermediate results, and operate within clearly defined constraints.

And probabilistic systems bring their own hazards.

During the Q&A, Nicole specifically warned about forward-looking bias in financial applications. Models are extremely good at pattern matching, which can also lead them to derive patterns and make suggestions that are not adequately supported by the underlying facts.

So the problem is not merely whether an LLM knows finance.

The problem is whether the system surrounding it can constrain what it does, verify what it produces, expose how it arrived there, and respond sensibly when something goes wrong.

That is the territory of the harness.

The agent is bigger than the model

Nicole closed the main presentation with a progression worth keeping: executable reasoning → inspectable states → feedback-driven control → learned harness decisions.

It captures a shift we’re increasingly interested in at Agentic Engineering.

The model is becoming one component inside a much larger engineering problem.

The interesting work is moving outward: into the runtime, the context, the tools, the coordination layer, the evaluation system, the boundaries, the feedback loops, and the machinery responsible for keeping the whole thing under control.

That is especially visible in finance, where the cost of a system being mostly right can become very real, very quickly.

And perhaps that’s the simplest way to think about harness engineering: Capability comes from the model. Dependability has to be engineered around it.

The foundation is set. Now, build the thing.

This free webinar covered the architecture. You learnt why a financial agent is not simply a model with a clever prompt, but a system in which the harness controls what the model can access, do, remember, verify, and prove.

Agentic AI for Finance, Nicole’s four-day intensive, is where you build that system.

You’ll connect models to financial data, tools, and memory; engineer verification and governance into the workflow; and see what happens when all those neat boxes on an architecture diagram actually have to work together. The aim is to leave knowing how to build an agent that is useful, observable, and safe to operate.

What you’ll leave with

Alongside the hands-on build experience, you’ll get:

• Four days of live access to the workshop with Nicole

• Downloadable code templates, datasets, and architecture blueprints

• Lifetime access to the workshop recordings

• A free copy of Building AI Agents for Finance to go deeper into the engineering covered during the program

• A certificate on completion, plus practical experience relevant to work across quantitative research, fintech, portfolio management, and financial engineering

Agentic AI for Finance runs across August 29–31 and September 12–13, giving you time between sessions to digest what you’ve built before moving into the next stage.

So if you enjoyed Nicole’s webinar and want to take the next step from understanding the architecture to engineering it yourself, book your spot for Agentic AI for Finance below.

Book your spot now

That’s it for this one. We’ll pick up our regular programming next week.

Until then, don’t just engineer the agent. Engineer the harness.

Tanya D’cruz
Editor-in-Chief

But autonomy is not free. The further a system can go on its own, the more room it has to drift, hallucinate, misinterpret a goal or make a consequential decision before someone notices.

For this episode of the Agentic Engineering podcast, I spoke with Ben Auffarth, Director and Chief Data Officer at Chelsea AI, founder of Chelsea AI Ventures, and author of several bestselling AI and machine learning books, including Generative AI with LangChain, Machine Learning for Time-Series with Python and Artificial Intelligence with Python Cookbook. Ben holds a PhD in computational neuroscience and has built systems across finance, insurance, travel and other regulated environments.

Our conversation centred on a digital-footprint intelligence platform his team is building for estate agents. It is the kind of problem that sounds made for an autonomous agent: search for a person, gather information, assess the evidence and produce a useful profile.

Ben’s team made a different choice. They use LLMs extensively during development (to build datasets, run experiments, and improve the underlying system) but limit their authority when a real user is being evaluated.

That decision led us to a more useful question than simply asking “How autonomous can this agent become?”, which is →

How much autonomy does the system actually need?

What does Neighborhood do, and where does AI fit into the system?

Neighborhood helps estate agents prioritise prospective property buyers. An enquiry may contain little more than a name, email address, telephone number and sometimes an address. The platform enriches those details with publicly available information, tries to verify that it has found the correct person and returns a profile with match scores and source information.

The difficult part is not finding someone with the right name; it is knowing whether the information belongs to this particular person. We use LLMs to help build datasets, extract information and run experiments quickly. At inference time, however, the LLM does not make the final matching or prioritisation decision. Those decisions are handled by more deterministic models and statistical methods. That is what I mean by bounded autonomy.

What tells the system that it has found enough information?

A general search engine may return many possible matches and leave someone to work through them. An LLM-based search tool can return a neat answer with a confidence percentage, but that number is not necessarily calibrated. If a model says it is 70 per cent confident, it does not mean that comparable answers are correct 70 per cent of the time.

We wanted confidence to have a measurable meaning. Neighborhood uses statistical principles, information theory and record-linkage methods to estimate the probability of a match. A common name in a large city begins with more uncertainty, and each additional piece of matching evidence adjusts that probability. The system stops relying on whatever the LLM “feels” and works with confidence that can be compared against data.

Can missing information become a hidden penalty?

Some people have extensive digital footprints, while others share very little or use platforms the system cannot access. The absence of information should not automatically become evidence that someone is suspicious or unlikely to proceed.

We found that a comprehensive profile does not necessarily mean someone is a stronger buyer. We therefore try not to downgrade people heavily when little information is available. More information may help an estate agent ask better questions, but that is a product assumption to test—not a universal truth. Sometimes the most accurate answer is simply that we do not know enough.

Is a more autonomous agent necessarily more advanced?

Autonomy is often presented as a measure of technical maturity. I do not think it is something to pursue for its own sake. The more autonomy we add, the more control we give up and the more variance we introduce.

Agents can look impressive in short demonstrations, but benchmarks involving longer tasks still show very low completion rates. If reliability matters, moving down the autonomy scale can be the better engineering decision. Use the LLM for tasks it performs relatively well, such as extracting information, and place deterministic decisions and constraints around it.

What does meaningful human oversight look like?

Putting a person at the end of a workflow does not guarantee meaningful oversight. If the system is usually correct and its output looks professional, the reviewer may gradually stop examining the evidence and simply approve each recommendation.

In Neighborhood, we avoid asking whether an entire profile “looks right.” Instead, we ask the reviewer to identify a believable anchor, such as a company or location that connects the information to the correct person. They can also say they are unsure. Human oversight should contribute evidence or judgement the system does not possess; otherwise, it is merely rubber-stamping.

How do you evaluate whether the system’s confidence is justified?

A system may retrieve accurate information and still attach it to the wrong person. That is why we expose the evidence behind a profile, our confidence in each piece and its provenance. The estate agent can inspect the original sources and provide feedback, while the final decision remains with them.

The business measure is not whether a profile looks comprehensive. It is whether the system helps estate agents spend their time with the right prospective buyers. Evaluation must cover identity matching, evidence quality, confidence calibration and the usefulness of the eventual recommendation—not only the final score.

Are we confusing a successful demo with a reliable system?

Agent demonstrations usually show examples in which the system finds the right information, follows the right steps and produces a convincing answer. Production introduces ambiguous identities, inaccessible data, regional differences and inputs that look nothing like the demonstration.

At Neighborhood, we used agents and repeated runs to help create a labelled dataset, then combined those outputs with human evaluation. We train and tune more deterministic models against that data and track the results of each experiment. That may be less exciting than showing an agent browsing autonomously, but it gives us something we can measure and reproduce.

What should a business leader establish before deploying an agent?

I would ask for four things: a labelled evaluation set, a baseline to improve upon, an understanding of the cost of a wrong answer and a measurement of response variance. Run the same input several times and see whether the system gives the same answer. If it does not, average performance may conceal an agent that cannot be trusted on an individual decision.

Businesses often want to fund the agent before they fund the dataset. That is backwards. Without representative data, you cannot reliably build, compare or govern the system. In many AI projects, the dataset, not the model, is the real bottleneck.

How much autonomy should engineers give an agent?

There is a perception that because a system uses AI, less engineering is required. In practice, the opposite is often true.

You need to define the outcome clearly, establish a benchmark early, and constrain what the system is allowed to do. Deterministic checks and filters can prevent entire categories of failure. An orchestration framework such as LangGraph can also make the available paths explicit: some decisions can be made by the model, while others remain ordinary conditional logic.

The question is not whether the agent could make another decision autonomously. It is whether that autonomy improves the system enough to justify the additional variance and loss of control.

Ben’s case for bounded autonomy is not an argument against agents. His team uses LLMs to create data, accelerate experimentation, extract information, and improve development. The argument is about placing autonomy where it creates value and removing it where consistency matters more.

That distinction becomes especially important now as agents move from helping people produce content or write code towards influencing decisions about buyers, customers, patients and applicants.

The most advanced system may not be the one permitted to do everything. It may be the one whose builders know exactly where the model is useful, where deterministic software should take over and where a person must still make the call.

Before asking how autonomous an agent can become, businesses should decide what happens when it is confidently wrong.

Less hype, more engineering conversations with the experts.

Pull up a chair.

Subscribe now

Editor’s note

Over the past few weeks, we’ve been talking about context engineering, loops, GraphRAG, and all the machinery that sits around modern AI agents.

This week, we’re deliberately taking a step back.

A year ago, most discussions focused on evaluating models. Today, more teams are discovering that evaluating agents is a very different problem. Once an AI system starts planning, calling tools, retrieving memory, and coordinating multiple steps, checking whether the final answer looks right simply isn’t enough. The execution path matters just as much as the destination.

So before we continue exploring more advanced engineering patterns, it felt worth revisiting one of the foundations.

In this issue, Balamurugan Balakreshnan, who works with enterprise AI teams at NVIDIA, walks through how he thinks about evaluation in agentic systems, why trust has to be engineered from the beginning, and which parts of an agent deserve measuring before production.

One of the biggest challenges I see in getting agentic AI systems into production isn’t the technology itself. It’s trust.

We’ve made enormous progress in building models that can reason, write code, call tools, and orchestrate complex workflows. But once those systems start making decisions on their own, a different question seems to start taking up space: How do we know we can trust what they’re doing?

Of course, the simplest answer is to have humans validate every output. But that will never sustain once you move beyond prototypes. If an agent is handling thousands of requests a day, human review becomes the bottleneck. So we need another way to build confidence.

Trust has to be designed

Models are trained on data that reflects the world at a particular point in time. We can improve their responses by providing fresh context during inference, whether that’s enterprise knowledge, live business data, or current events. That helps, but better context doesn’t automatically make an output trustworthy. Trust still has to be earned through evaluation.

In my experience, the best place to start is during development, not after deployment. Before a model ever reaches production, it should be evaluated for things like correctness, relevance, completeness, grounding, and safety. None of this is easy. Good evaluation depends on good benchmark datasets, domain expertise, and clear success criteria. A coding model won’t be evaluated the same way as a reasoning model, and every organisation will have its own priorities beyond the standard metrics.

And the work doesn’t stop once the model ships.

Production systems need continuous evaluation because real users rarely behave as we expect. The data collected in production often tells you more about your system than anything you learned during testing. As models continue through fine-tuning, post-training optimisation, or other updates, both the data and the evaluation process evolve alongside them.

Why evaluating agents is different

Evaluating a model is one thing. Evaluating an agent is something else entirely.

An agent doesn’t just generate text. It interprets the user’s intent, builds a plan, decides which tools to call, retrieves information, and coordinates multiple actions before producing a result. In many systems, there’s also a planning or supervisor agent deciding which specialist agents, MCP servers, or data sources should be involved. In multi-agent systems, every additional agent introduces another layer of decision-making.

That’s why evaluating the final answer is no longer enough.

We also need to understand how the agent arrived there.

When an agent runs in production, I think about questions like these:

• Was the execution plan appropriate for the user’s request?

• Did it choose the right specialist agents?

• Did it select the right tools or MCP servers?

• Did it retrieve the right information from memory or external knowledge?

• If a judge model was involved, has that model itself been properly evaluated?

These questions matter because agents are making decisions continuously as they execute a workflow. Trust comes not just from the quality of the final answer, but from confidence that the system made good decisions all the way through.

So what should we actually evaluate?

This is the question I get asked most often. My honest and perhaps bland answer is: it depends.

Every application is different. The metrics you care about will vary depending on your domain, the framework you’re using, and what success looks like for your users. There isn’t a universal scorecard for agentic AI.

That said, I’ve found there are a handful of areas that every team should think about.

Task performance

The first question is the simplest one. Did the agent accomplish what it was asked to do in the first place?

That sounds obvious, but it’s surprisingly easy to focus on intermediate steps and forget the end goal. At a minimum, I’d want to understand things like:

• Goal achievement

• Subtask completion rate

• End-to-end success rate

• First-attempt success rate

• Human evaluation

Reasoning quality

A correct answer isn’t always the result of good reasoning. Sometimes an agent simply gets lucky.

Because agents plan before they act, it’s important to evaluate the quality of that planning as well. Questions worth asking include: Was the plan coherent? Did it retain the right context? Could it recover when something went wrong? Did it follow instructions, or did it drift off course?

Some useful metrics here include:

• Plan coherence

• Step accuracy

• Self-correction ability

• Hallucination rate

• Context retention

• Decision quality

• Instruction adherence

Tool selection and orchestration

Most production agents don’t work alone. They’re constantly interacting with APIs, databases, MCP servers, and external tools.

Choosing the wrong tool can be just as damaging as producing the wrong answer, so it’s important to evaluate how well the agent orchestrates the systems around it.

Some useful measures include:

• Tool call accuracy

• Tool selection precision

• Parameter correctness

• API call success rate

• Error recovery rate

• Orchestration efficiency

• Unnecessary tool invocation rate

Safety and reliability

No production system is complete without guardrails.

Agents should stay within their intended scope, handle failures gracefully, and avoid actions that could compromise users or data. Depending on your application, you might look at:

• Refusal accuracy

• Guardrail compliance

• Harmful action rate

• Scope creep

• Failure mode coverage

• Graceful degradation

• Uptime and availability

• Data leakage incidents

Cost and efficiency

Finally, there’s the operational side of the equation.

An agent that produces excellent results but takes five minutes and thousands of tokens to complete a simple task may not be practical in production. That’s why it’s worth tracking metrics such as:

• Latency per task

• Token usage

• Cost per successful task

• Number of execution steps

• Retry overhead

• Throughput

• Memory efficiency

These aren’t meant to be a checklist that every team must follow. Think of them as a menu rather than a mandate. Different frameworks expose different signals, and different applications will naturally prioritise different measures.

The important thing is to be deliberate about what you’re evaluating. If you only measure the final response, you’ll miss most of what determines whether an agent is actually reliable.

For me, that’s the biggest shift agentic AI introduces.

Trust doesn’t come from a good answer alone. It comes from understanding how that answer was produced. That means evaluating the planning process, the decisions made along the way, the tools that were selected, the knowledge that was retrieved, and the actions the agent ultimately took.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

How does the system know that it is getting closer to the goal rather than repeating itself? Who decides whether the result is correct? What prevents the agent from changing the test until its own work passes? And if the model can continue taking actions, spending tokens, and modifying files, what finally makes it stop?

For this episode of the Agentic Engineering podcast, I spoke with Rick Hightower, a senior AI engineer and certified Claude architect who has spent more than 20 years building software across companies including Apple, Capital One, and the NFL. Rick now focuses on the less-visible parts of agentic systems: harnesses, context, state, verification, and loops.

Rick began working with AI before today’s command-line agents and coding harnesses were available. The models could suggest what to do, but he still had to move the output into the development environment, run the tests, inspect the result, and decide what should happen next. As he puts it, he was the loop.

Today, harnesses can take on more of that responsibility. But before we remove the person who was checking the work and deciding when to continue, we need to understand what will replace them.

So we began with the most basic question: what actually turns a series of prompts into a loop?

What turns a series of prompts into a loop?

Before command-line agents and coding harnesses existed, I was the loop. I would ask ChatGPT or Claude to research an approach or help port code, move the result into the development environment, run it, inspect what happened and then decide what to try next.

A real agent loop automates that feedback process. It has an expected outcome, a way to verify the result, and a mechanism for returning useful feedback when the result fails. It continues until the work meets the stated criteria or the system determines that it cannot complete the task with the information and tools available.

I use Mermaid diagrams as a simple example. The agent can generate a diagram, lint the syntax, and correct any errors. Once it renders, another model can inspect the image and compare it with the original requirements. The first check tells us whether the diagram is valid. The second tells us whether it actually represents what we asked for.

Without verification and feedback, we do not really have a loop. We have a sequence of prompts.

Who verifies the verifier?

If the same agent interprets the requirement, writes the code and creates the test, it may define success in a way that agrees with its own implementation. It may even change the test to make its work pass.

The harness should control what the agent is allowed to modify. Permissions can prevent it from editing protected tests, and hooks can detect when it tries. There may be legitimate reasons to update a test, but that change should be explained and, in higher-risk situations, reviewed by a person.

Wherever possible, I prefer deterministic verification: does the code compile, does the test pass, does the output match the schema, or does the file pass a linter? When that is not possible, I use a separate judge or adversarial subagent.

The judge should receive the requirements and the final output, but not necessarily the full context of the agent that produced it. If both agents share the same reasoning history and assumptions, the judge may inherit the same bias. I think of it as the difference between the person doing the work and the teacher grading it.

How does a loop know when it is stuck?

A loop can continue producing new output without making meaningful progress. It may repeat the same action, revisit an approach that already failed or improve one part of the result while breaking several others.

I have encountered this while tuning systems that extract structured information from documents. The loop may be working towards a target accuracy across several categories. One attempt improves one category but causes the others to fall. The next attempt reverses the change. The system is doing work, but the overall result has stopped improving.

The harness needs an exit condition for that situation. It might stop after four or five attempts without progress, after a fixed number of turns or after reaching a token or cost limit. It should also recognise when it cannot succeed because data is missing, a downstream system is unavailable, or the available tools are not enough.

Ideally, the loop stops because the work is complete. But production systems also need a responsible way to stop when completion is not possible. At that point, the agent should change strategy, ask for help, or hand the problem to a person.

Are we blaming models for failures caused by the harness?

When an agent behaves badly, the model usually receives the blame. But many failures begin elsewhere in the system.

A team adds a feature and changes the prompt. That new feature works, but three older behaviours stop working. A model provider releases an update, the retrieval system starts loading different context, or someone changes the few-shot examples. The output becomes worse, and it looks as though the model has suddenly lost a capability.

The most common problem I have seen is drift. Agent systems have many moving parts, and changing one of them can create regressions somewhere else. That is why teams need a known set of inputs, expected outputs, and a way to grade them after every meaningful change.

I describe the principle as “slow down and go fast.” If you can detect regressions early, you can move quickly without repeatedly breaking the behaviours you already depend on. Otherwise, developing these systems can feel like nailing Jell-O to a wall.

The harness controls the context, tools, permissions, state, verification, and stopping conditions around the model. Before replacing the model or rewriting the prompt, inspect those layers.

Where does memory live when the model has none?

LLMs are stateless, but a useful agent needs to know what has already happened, what remains unresolved, and whether its latest action changed anything.

I separate memory and state into several categories. There is conversational memory, which may be compressed into a summary. There are important facts that should be stored explicitly because summarisation can lose them. There are instructions and procedures in files or agent skills. There is information retrieved from a database or RAG system when the task requires it. And there are plans, tickets, logs, and repository history that allow the work to continue beyond one session.

The goal is not to keep everything in the active context. If the context becomes too large, important information gets buried, and the model begins forgetting things it would normally handle correctly.

I prefer to keep the main agent focused and use subagents for narrow tasks. Once one task is complete, I can clear the context and begin the next one using the durable information stored in the specifications, plans, and tickets.

If you cannot clear the context because the agent would forget everything it has been doing, the important state probably needs to be written down somewhere outside the conversation.

Is a specification more than a very long prompt?

A prompt usually tells the agent what to do next. A specification describes the system, the work that needs to be completed, the constraints, and the conditions that should be true at the end.

That makes the specification the fuel for a long-running loop. I have given agents several tickets, asked them to complete the work, launch the application, visit the relevant pages and compare the result with the agreed mock-ups. The agent can run for hours because it has both a roadmap and a definition of done.

But a long specification is not automatically a good specification. If the plan is incomplete or wrong, the agent may follow it perfectly and still produce the wrong result. Garbage in, garbage out does not disappear because AI is doing the implementation.

For important or complicated plans, I often ask another agent—sometimes a different model, sometimes the same model in an isolated context—to look for missing assumptions and unclear requirements. A second set of eyes usually finds something.

The specification should also be visible to the human stakeholders who understand the intended outcome. An agent can critique a plan, but it cannot recover business context that nobody gave it.

Can the same specification safely drive action and verification?

Using one specification to tell the agent what to build and then using that same specification to judge the result is efficient, but it creates a shared point of failure.

If the specification misses a requirement, both the builder and the verifier may agree that the work is complete. The agent has built exactly what it was asked to build, but the requested system was wrong.

I still use the specification as the basis for verification, but I try to support it with independent evidence. That may include sample inputs from product management, approved mock-ups, existing tests, business rules, or review from an agent that did not participate in writing the implementation.

The agent should also be able to question the specification. If it finds a contradiction or discovers that an acceptance criterion cannot be satisfied, continuing blindly is not useful autonomy. It should report the problem and bring a person back into the loop.

The specification guides the work. It should not become something the system is forbidden to challenge.

How do you verify an autonomous research loop?

Research does not have the same natural completion signal as code. There may always be another source to read, another reference to follow, or another direction to investigate.

The research loops I have built usually begin with an agreed outline and a defined artifact, such as a document, image or presentation. The system can then validate whether the artifact follows the outline, whether the facts have been checked and whether it meets the required grammar, style and quality standards.

In that case, the verification gate is the rubric agreed upon before the research begins. The loop is complete when the artifact meets those requirements—not when the agent has exhausted every possible source.

Some research and experimentation also have external signals. In marketing, an agent can test different advertisements, measure clicks or conversions, and use that result to decide what to try next. AI could run far more of those experiments than a human marketing team could manage manually.

Open-ended research is more difficult. A rubric can tell us whether an artifact meets its requirements, but it cannot prove that no useful information remains undiscovered. We should be honest about that rather than pretending every kind of research can be given a perfectly deterministic finish line.

What did production force you to change your mind about?

Verification is expensive. Once you add adversarial subagents, judges, and feedback loops, a workflow may cost five or ten times more than a single model call.

But the useful comparison is not between a verified system and the cheapest possible workflow. It is between the cost of verification and the cost of delivering the wrong result.

If I can verify something with a compiler, test, or linter, I will use that. When I cannot, I may use a rubric and another model. That increases inference costs, but it also produces much better results. A brand violation, a broken production change, or an incorrect high-stakes answer can cost far more than the additional model calls needed to catch it.

If I had only one hour to design a new loop, I would begin with the verifier and the stopping conditions. What does done look like? What evidence proves it? How many attempts can the agent make? What happens if it stops improving?

I would also define the permissions and boundaries before allowing the loop to run. I would not run an unfamiliar autonomous agent with unrestricted access to the same machine that holds my credentials, personal files, or business systems. Use a sandbox, virtual machine, or managed environment, and give the agent only the tools it genuinely needs.

Rick’s case for loop engineering is not that we should let agents run for as long as possible. It is that agents become more useful when the system around them can recognise success, detect failure and stop them safely when they are no longer making progress.

The model may perform the visible work, but the harness determines the conditions under which that work happens. It supplies the context, preserves the state, controls the tools and decides when a person needs to return.

Before giving an agent a longer task, a larger context window or more powerful tools, the most useful place to begin may be the end: decide what ‘done’ means, how you will recognise it and what the system should do when it cannot get there.

Enjoyed today’s podcast and want more honest engineering conversations?

Pull up a chair ↓

Subscribe now

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

We’ve spent time over the past few weeks talking about prompts, context, harnesses, and loops. But there’s another part of the AI engineering stack that doesn’t get as much attention: the context your coding agents depend on.

This week, Maxim Salnikov explores why agent context deserves to be treated like any other software dependency, and how Agent Package Manager (APM) aims to solve the configuration drift that’s emerging across AI development teams.

If you’d like to go deeper after reading, Maxim has also put together a free interactive guide on Agent Package Manager, which we’ve linked at the end of this article. I’ll leave the floor to him.

You know the feeling, because you’ve felt it before… in a different decade. A new engineer joins, and getting their environment right means a wiki page, a Slack thread, and copying files out of someone’s home directory. Two developers swear they have the same setup, but their tools behave differently, and nobody can say why.

Before npm, pip, and Cargo, that was simply how software felt. Dependencies lived wherever each developer happened to put them — a lib/ folder copied between machines, a README that said “install these seven things first.” It mostly worked, until it didn’t. Package managers ended it by turning scattered files into a declared dependency: one manifest that names what you need, one lockfile that pins exactly what you got, one command that restores it anywhere.

The files that configure your AI coding agents are in that same pre-package-manager state right now.

The context you never treated as code

Call it your agent context: the instructions and coding standards, the reusable prompts, the skills and agent personas, the plugins, the Model Context Protocol (MCP) server configurations. It shapes the code an agent writes, how it reviews a diff, and which external tools it can reach. That makes it a real project dependency — one of the highest-leverage ones you have, because it multiplies across every line the agent touches.

Yet most teams hand-maintain it as loose files scattered across per-tool directories and personal machines, with no declared source of truth. So it drifts. One tool’s instructions say to use the Money value object; another still mentions raw decimals. Three copies of the same review prompt live in three home directories, each checking a different threat model. Neither is wrong. Neither is portable.

A shared README doesn’t save you: documentation describes intent but never restores state, so the drift returns the moment someone edits a local file. This is “works on my machine,” aimed squarely at your agents — and it’s invisible, because nothing broke. The build is green. The agent just quietly got a little less consistent, or a little more dangerous, on one laptop.

Three files and one habit

The fix is not a new methodology. It’s the boring, proven shape that worked the first time. Agent Package Manager (APM), an open-source tool from Microsoft, borrows the manifest-plus-lockfile pattern and points it at agent context. There are three files and one habit:

• apm.yml — the manifest. The single human-authored source of truth, naming the primitives and MCP servers your project depends on, and which tools to target.

• apm.lock.yaml — the lockfile. Machine-generated, never hand-edited, pinning every dependency to an exact source ref and a content hash, so two developers install byte-identical context.

• apm-policy.yml — install-time governance, for when you’re ready for it.

• apm install — the habit. The one command that reads the manifest and materializes the declared context into each tool’s native location.

The move that makes it click is materialization. A manifest is a recipe, not a meal; apm install turns the declaration into the real files each tool reads natively — compiled into .github/ for Copilot, .claude/ for Claude Code, .cursor/ for Cursor — then stays out of the way at runtime. Your three-harness maintenance burden becomes one.

A 60-second quick start

Enough theory. Here is the entire loop against a real, public collection — web-ai-agent-skills, a maintained set of skills for the browser’s built-in AI APIs. Every command below is in the APM docs:

# 1. Install the CLI (macOS/Linux; use the PowerShell one-liner on Windows)

curl -sSL https://aka.ms/apm-unix | sh

# 2. In your project, once:

apm init

# 3. Add a skill as a pinned dependency:

apm install webmaxru/web-ai-agent-skills/skills/prompt-api

That last line is the whole pitch. It resolves the skill straight from its git source, pins it in apm.lock.yaml by commit and content hash, and compiles it into the location your agents already read. Commit the two files, and the next developer who runs apm install gets byte-identical context. The Prompt API skill is a declared dependency now, not a snippet someone pasted from a stale gist — and swapping prompt-api for any other entry works the same way.

What actually changes

Onboarding stops being a ritual and becomes git clone then apm install — the new hire’s agent is configured identically to yours before their first coffee. Drift stops being something you find during a security review: the lockfile makes it impossible, reproducing known-good context down to the bytes. None of this makes your agents smarter. That’s the point — it makes them consistent, the precondition for every other improvement, since you can’t tune a context you can’t reproduce.

The uncomfortable part is that we already learned this lesson once. Agent context deserves the same declaration and review discipline you already demand of package.json or requirements.txt. The tools exist. The only question is how long you keep paying the drift tax before you declare the dependency you already have.

And if authoring that first manifest feels like one more chore, there is a fitting last move: let an agent do it. The Agent Package Manager skill teaches your coding agent to run apm init, add and pin dependencies, validate the manifest, and manage the lockfile — installed, of course, exactly like any other skill:

apm install webmaxru/ai-native-dev/skills/agent-package-manager

Which is the whole idea coming full circle: the package manager for agent context, installed and operated by the agent itself.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

But adding a graph does not remove the difficult questions. How does an agent know when it has gathered enough evidence? What happens when relationships change, memories become unreliable, or the graph itself sends the model down the wrong path? And when is a graph genuinely the right architecture rather than simply the more fashionable one?

For the first episode of the Agentic Engineering podcast, I spoke with David Knickerbocker, founder and intelligence architect at Verdant Intelligence and author of Network Science with Python. David has spent decades working across databases, cybersecurity, open-source intelligence, and network science. His answers return repeatedly to one idea: GraphRAG and agents are still software systems. If we want to trust them, we need to make their evidence visible, test their individual parts, and resist rebuilding systems that already work.

When do you genuinely need GraphRAG?

I don’t see graphs as an inherently more complicated alternative to RAG. Networks already exist in nature, organisations, and social systems; graphs simply represent how things connect. They can feel difficult because fewer teams have experience with network science, but the basic idea—one thing connected to another—is straightforward.

That does not mean every team should replace its RAG system. If you have invested in a mature similarity-based system and it solves the problem well, there is no reason to solve the same problem twice. Use GraphRAG when relationships, paths, and connected evidence are central to the problem, and your team is prepared to work with them. If the existing system works, keep it.

Does a natural-language GraphRAG endpoint remove complexity, or merely hide it?

A natural-language endpoint should hide implementation work without hiding the evidence. I design systems to return the context behind an answer: the source material, the relevant data, and the reasoning behind a predicted relationship. For example, when one of my tools converts text into a graph, it preserves both the original passage and the reason an edge was created.

There will always be some trust involved when an agent uses an external service, just as there is when we use a search engine or API. That trust should develop through repeated, dependable use—not blind faith. Developers may not need to understand the database schema or write Cypher, but they should still receive enough information to verify what the endpoint returned.

How does an investigative agent know when it has investigated enough?

We need to teach agents something my managers used to tell me: good enough is good enough. An agent should have a defined purpose, a limited mandate, and a clear point at which it reports back. An agent checking for concerts does not need to investigate indefinitely; it can check the relevant venues and artists, return the results, and stop.

An open-source intelligence agent may need to search more widely, compare conflicting sources, and detect manipulation, so its definition of “enough” will be different. The stopping condition has to match the task. The important thing is to decide what good enough means before the agent begins, rather than allowing it to keep reasoning and spending without a boundary.

How should GraphRAG represent information that changes over time?

I treat graphs as temporal. Information changes, and the completeness of our information changes with it. Rather than overwriting an old relationship, I would preserve what was claimed, where it came from, and when it was recorded. The history is part of the information.

In my systems, the highest-level object is a claim rather than a fact because the internet contains many competing claims and facts require judgement. A graph should preserve those differences instead of making every relationship look timeless and definitive. Timestamps, sources, and changing versions allow an agent to reason about what was believed at a particular point rather than treating the latest entry as the only truth.

How do we prevent an agent from poisoning its own memory?

Agent memory needs the same quality assurance we expect from any other piece of software. If a weak inference is stored, the system should retain its source and confidence rather than retrieving it later as an established fact. Independent agents can check one another, although using the same model everywhere may reproduce the same blind spots.

The more granular the memory, the more granular the testing needs to be. If you cannot inspect, validate, correct, or test an agent’s memory, you cannot determine whether it is reliable. Agentic work does not remove the need for software engineering; it makes careful testing at the component level even more important.

Can the knowledge graph, GraphRAG platform and agent really evolve separately?

They can evolve separately if their dependencies are explicit, versioned and tested. I give the major components of my systems their own semantic versions so each can be developed, released and, when necessary, rolled forward without rebuilding everything at once. Iteration creates infrastructure; big-bang development usually creates demonstrations.

A major graph-schema change may require the GraphRAG interface to change, just as a database change could break an application long before generative AI existed. But if the agent communicates through a stable natural-language contract, it may continue asking the same questions while the implementation underneath evolves. Separation does not mean independence; it means managing the connections deliberately.

Are we measuring GraphRAG correctly?

Counting relevant retrieved facts is not enough if the answer depends on a complete chain of evidence. A system may retrieve three of four necessary facts and still produce a confident but fundamentally incomplete conclusion. Teams need to test whether the system identified every question it needed to ask and recovered the evidence required to answer each one.

That also means testing the layers separately. Does the language model decompose the original request correctly? Does the GraphRAG endpoint return the right data for each subquestion? Does the agent assemble that evidence into a supported answer? When something is missing, we should identify the layer that lost it instead of labelling the entire system inaccurate.

Is the AI hallucinating, or is the graph misleading it?

I use a phrase in my training: similarity is not sameness. A similarity-based system may retrieve extra information that looks related but does not support the answer. If the model incorporates that noise as evidence, it can produce a hallucinated conclusion.

GraphRAG can fail differently. A query may miss a relationship, fail to traverse a node type, or return an event without its time or source. That is not necessarily the model inventing information; it may be a problem in the data, schema, query, or traversal. The only way to tell is to inspect the evidence path and determine which layer introduced the error.

David’s case for GraphRAG is not that a graph should replace every RAG pipeline. It is almost the opposite: keep the parts that already work, introduce graphs where relationships genuinely matter, and make every new layer earn its place.

That restraint matters as agents become more capable of searching, remembering, and acting without constant direction. More intelligence in the workflow also creates more places for evidence to become incomplete, outdated, or distorted. Reliability, therefore, depends less on whether a system carries the latest label and more on whether its builders can see what it did, test each part, and recognise when it has done enough.

The most useful place to begin may not be a large architectural migration. It may simply be a small experiment, built slowly enough that you can still understand what the agent sees and why it reaches the conclusions it does.

Enjoyed today’s podcast and want more engineering conversations? Pull up a chair.

Subscribe now

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Last week, Ken Huang broke down agentic engineering into three disciplines: harness, context, and loops. But I couldn’t leave that discussion just at that.

Because if you talk to builders, it’s clear that the work is still moving. A couple of years ago, it was all about prompting. Then we got better at tweaking context. But now, we’re starting to move beyond that. Toward loops.

So this week, Mona picks this apart from a practitioner’s lens. I’ll leave the floor to her.

In 2023, everyone wanted to be a prompt engineer. In 2026, the head of Claude Code at Anthropic says he doesn’t prompt Claude anymore. His words: “My job is to write loops.”

That single shift from prompting an agent to designing the loop that prompts it is the biggest change in how professionals work with AI since ChatGPT launched. And most people haven’t caught up yet.

Let me walk you up the ladder, one level at a time.

Level 1

Prompt engineering, the words you send

Prompt engineering is where all of us started. You learn that “write me a blog post” gets mediocre output, but “write a 900-word blog post for enterprise cloud architects, in a direct second-person voice, with a contrarian hook” gets something usable.

Prompt engineering is about how you express the task. Be specific. Give examples. Assign a role. Ask for a format. But it operates on one thing: the message you type into the box.

The limitation is obvious. A prompt is a single instruction to a model that has no idea who you are, what your codebase looks like, or what happened five minutes ago. You can only cram so much into words.

Level 2

Context engineering, everything the model sees

The next realization: the model’s output quality depends less on your clever phrasing and more on what information it has access to when it responds.

Context engineering is about curating everything in the model’s window — the instructions, the documents, the data, the tool definitions, the conversation history, the intermediate results. It asks a different question from prompt engineering:

• Prompt engineering asks: how do I say it?

• Context engineering asks: what should the model know?

This is why retrieval (RAG), memory systems, and structured tool outputs became the center of gravity for enterprise AI teams in 2024–2025. Teams needed to feed the model the right context at the right time, and what was just as important was keeping the wrong context out.

But context engineering has ceilings of its own:

• The window is finite, and attention degrades before it fills.

  Even massive context windows don’t mean equally useful attention across every token. Overload the window, and the model starts missing details buried in the middle. More context is not better context.

• It’s a snapshot, not a conversation with reality.

  Context is assembled before the model responds. Real tasks surprise you mid-execution — a dependency is missing, an API returns something malformed, a test fails. No amount of upfront curation can anticipate what only shows up when you run the thing.

• The model can know everything and still do nothing.

  Context engineering makes the model informed. It doesn’t make it capable. It can’t execute your code, check the result, or fix what broke. Knowledge without the ability to act is a very well-read consultant with no hands.

• Curation doesn’t scale by hand.

  Someone has to decide what goes into the window for every task. Past a certain complexity, assembling the context itself needs to be automated, which is the door the next two levels walk through.

Level 3

Harness engineering, the environment the agent runs in

Then agents arrived, and a prompt plus context stopped being enough.

An agent doesn’t just answer — it acts. It reads files, runs code, calls APIs, and checks its own work. For that, it needs an environment: tools it’s allowed to use, a filesystem it can touch, permissions that constrain it, feedback mechanisms that tell it whether the tests passed.

That environment is the harness. Harness engineering is designing the executable world around a single agent run — the system prompt, the tool set, the memory, the guardrails, the verification steps. If context engineering decides what the model knows, harness engineering decides what the model can do and how it finds out whether it did it well.

If you haven’t subscribed yet, pull up a chair.

Subscribe now

If you’ve used Claude Code, Cursor, or any serious coding agent, you’ve benefited from harness engineering, whether you knew the term or not. The difference between an agent that flails and an agent that ships is usually not the model. It’s the harness.

But even a great harness has a human bottleneck: you. You’re still the one deciding what the agent works on next, reading the output, and firing the next run. One turn after another. The agent is a power tool, and you’re holding it the whole time.

Level 4

Loop engineering, the cycle that drives it all

Loop engineering removes you from the turn-by-turn seat.

Instead of prompting the agent, you build a small system (sometimes a shell script, sometimes a scheduled job, sometimes a few hundred lines of orchestration code) that runs the agent in a repeating cycle:

You define the goal and the stopping condition. The loop does the iterating. It runs on a schedule (including while you sleep) or until the goal is met.

The idea crystallized fast in mid-2026. Peter Steinberger argued that the real skill had moved from prompting agents to designing their loops; Addy Osmani gave the practice its name and structure in an essay the following day; and inside Anthropic, the

Claude Code team was already describing their daily work the same way. The framing that stuck: the harness equips a single agent run — the loop is what keeps running the harness, spawning helper agents, checking results, and feeding itself the next task.

Each level wraps the one below it. The loop runs the harness. The harness carries the context. The context frames the prompt. Nothing on the ladder becomes obsolete. Rather, the leverage keeps moving up.

How to schedule a loop

1. Open your project in Claude Code.

2. Use the /loop command followed by your interval and the prompt or skill you want to run.

The /loop command is the core, with three modes depending on what you provide:

• /loop 5m check the deploy: Runs your prompt on a fixed schedule (Claude converts the interval to a cron expression)

• /loop check the deploy (no interval): Claude picks the delay dynamically after each iteration, between one minute and one hour, waiting less when things are active and more when quiet

• Bare /loop: Runs a built-in maintenance prompt that continues unfinished work, tends the current PR (review comments, failed CI, merge conflicts), and does cleanup passes. You can replace this default with your own loop.md file at project level (.claude/loop.md) or user level (~/.claude/loop.md).

But here’s the part nobody puts in the headline

First: loops multiply whatever you give them. A well-designed loop multiplies a strong engineer. A badly designed one multiplies a bad decision just as fast — with less of you watching. Two people can build the identical loop: one uses it to move faster on work they deeply understand, the other uses it to avoid understanding the work at all. The loop can’t tell the difference. You can.

Second: the unglamorous work is making it stop. Ask practitioners about loops and the first war stories you’ll hear aren’t about architecture: they’re about cost. Autonomous loops that burned through hundreds of dollars overnight. Companies imposing hard monthly caps on agent spend after blowing annual AI budgets in a quarter. The production rule is simple: every loop ships with hard guards (budget limits, iteration caps, verification gates, audit logs) or it doesn’t ship.

Everything in this article assumes one foundational skill: knowing how to build an agent in the first place. The loop is only as good as the agents it orchestrates.

That’s what my latest book, AI Agents on AWS (co-authored with Bunny Kaushik, published by Packt), teaches. You’ll learn how to design agents that reason, use tools, and act autonomously; how to wire up the harness around them — memory, guardrails, verification; and how to take them from a notebook experiment to production on AWS.

Add it to your shelf

When an agent has to serve real users across real channels, with real credentials, tools, permissions, memory, failures, latency, and cost constraints, the problem changes. It stops being only a prompting problem. It becomes an engineering problem.

This is where agentic engineering begins.

Agentic engineering recognizes that an AI agent is not just a model wrapped in an interface. It is a runtime system. It needs boundaries, identity, observability, recovery paths, and a way to explain what happened when something goes wrong. It must be designed like infrastructure, not like a clever script.

Three practices define this shift: harness engineering, context engineering, and loop engineering.

Harness engineering is the discipline of building the runtime around the agent. It includes gateways, tools, permissions, hooks, retries, telemetry, memory, sandboxes, and recovery paths. The harness is what lets an agent act safely in a real system. Without it, autonomy becomes fragility.

Context engineering is the discipline of selecting, shaping, compressing, refreshing, and protecting the information given to the model. Context is no longer just “stuff in the prompt.” It is an operational resource with a budget, lifecycle, security posture, and failure mode.

Loop engineering is the discipline of designing the cycles through which agentic systems act, learn, improve, and produce software. At the runtime level, this means structuring the agent’s observe-decide-act-evaluate cycle with termination conditions, policy gates, retry limits, rollback paths, and human escalation points.

But loop engineering goes further.

One loop is the recursive self-improvement loop. A production agent system should be able to detect weakness, diagnose failure, propose improvement, test that improvement, validate it against policy, and only then adopt it. This does not mean giving agents uncontrolled permission to rewrite themselves. It means creating bounded, observable, auditable improvement cycles where agents can help harden the system without bypassing engineering discipline.

Another loop is the continuous software factory. In agentic engineering, software development itself becomes a loop: requirements become designs, designs become code, code becomes tests, tests become deployment signals, deployment produces telemetry, telemetry informs the next requirement, and the cycle continues. Agents can participate at every stage, but the factory still needs gates, evidence, review, and rollback.

This is one of the central themes of my upcoming book, OpenClaw AI in Production. OpenClaw treats agent behavior as something that must be routed, bounded, observed, and corrected through architecture. The Gateway becomes the control plane. The request pipeline separates context assembly, authorization, directive handling, tool execution, and model invocation. Hooks allow cross-cutting concerns such as telemetry, policy, billing, and remediation to operate outside the core agent logic.

That architecture matters because agent failures rarely appear as simple crashes. A stale memory, a slow tool, a partial outage, a malformed directive, or an over-permissive policy can quietly distort the loop. The agent may keep acting, but each step moves it further from safe and useful behavior.

Production systems need to detect that drift.

This is why observability for agents has to go beyond CPU, latency, and HTTP response codes. A 200 OK from a model provider does not tell us whether the agent used the right tool, respected the right policy, retrieved the right memory, or stopped at the right time. Agentic systems need semantic observability: traces that show the agent’s observable decisions without leaking sensitive data or private reasoning.

The deeper opportunity is self-correction. If an agent platform can detect degraded state, isolate the failing component, reduce privileges, pause risky workflows, retry safely, or route through a fallback path, then we move from passive monitoring to active resilience. If the software factory can learn from its own incidents, tests, deployments, and user feedback, then engineering itself becomes more adaptive.

The future will not be won by prompts alone. It will be built by engineers who understand harnesses, contexts, and loops - and who can turn model intelligence into systems that are secure, observable, resilient, and continuously improving.

OpenClaw AI in Production is my attempt to map that territory in detail: from Gateway-centered architecture and policy enforcement to distributed state, semantic observability, self-correcting stacks, fault injection, high-throughput design, and decentralized deployments.

Agentic engineering as a discipline is still young. But one thing is already clear: building agents that work in demos is very different from building agents that survive, learn, and improve in production.

This article is a glimpse into the ideas behind Ken’s OpenClaw AI in Production. If you’re interested in building AI systems that are resilient by design rather than optimistic by default, the book expands on the engineering principles that separate production platforms from impressive prototypes.

Add it to your shelf

Context complexity exists across five distinct levels, evolving from zero-context basic prompts to highly advanced semantic blueprints. Prompt engineering resides at the shallowest level, treating context as a mere text prefix. Context engineering, conversely, treats context as a structural, programmatic architecture.

To build a robust multi-agent system (MAS), we must transition from linear text parsing to multidimensional semantic structures. We can achieve this by implementing Semantic Role Labeling (SRL) to map complex data relationships natively. By utilizing the Model Context Protocol (MCP), we can define rigorous protocol message formats that specialist agents such as a Researcher, Writer, and Orchestrator use to communicate without ambiguity.

Instead of hoping a black-box model infers the correct workflow, we architect a semantic blueprint that explicitly guides the system’s reasoning process, completely decoupling the immutable enterprise data layer from the probabilistic reasoning layer.

Architecting the glass-box context engine

The antidote to black-box unpredictability is the context engine: a transparent, glass-box architecture of context and reasoning. In this framework, the engine’s core intelligence is compartmentalized into distinct, observable modules: the Planner, the Executor, and the Tracer.

• The Planner: Acts as a meta-controller. It utilizes a discoverable Agent Registry to dynamically route tasks, intelligently switching between semantic search and strict data filtering based on the precise context of the request.

• The Executor: Relies on a Dual RAG architecture, simultaneously processing factual data via a strict Knowledge Base and procedural instructions via a Context Library.

• The Tracer: Allows human operators to parse the ExecutionTrace object to render token metrics, dependency resolutions, and view the internal “thinking” steps of the agent.

Within a finite reaction field, agent decisions must be bounded by plausibility to avoid cognitive dissonance. We enforce these boundaries using operators like the Semantic Switch ($\sigma$) and the Triton Planner Kernel. These tools apply explicit, rule-bound kinematics over the continuous probabilities of the neural field, acting as the constraint satisfaction engines that pure LLMs lack.

Enterprise guardrails: Scaling with Dual RAG

Bringing these systems into production requires moving AI directly to the data. By leveraging architectures of modern enterprise databases with vector and relational support, we can build Sovereign AI systems that are dock-agnostic, portable multi-agent systems directly to immutable enterprise databases.

This paradigm shift enables hyper-contextual capabilities:

• Hybrid querying: Combining strict SQL scalar filters (such as experience levels or salary caps) with semantic vector searches to generate grounded, policy-compliant recommendations.

• Converged Spatial-RAG and GraphRAG: Integrating physical geolocation (Oracle Spatial) and social dimensions (SQL Property Graphs) directly into the vector search pipeline to evaluate meaning, physical proximity, and relationship mappings simultaneously.

• Micro-context engineering: Deploying specific Summarizer agents to proactively manage API costs, reduce context overhead, and actively manage token limits.

• Policy-driven moderation: Implementing a two-stage moderation gatekeeper that flags anomalies, prevents data poisoning, and adapts to real-world legal compliance limits.

Conclusion: From cost center to value multiplier

The debate over how to achieve true agentic reasoning will not be settled by building ever-larger black boxes or writing increasingly convoluted text prompts. The future belongs to those who recognize and engineer around the structural boundaries of artificial intelligence.

By embracing context engineering, we shift AI from an unpredictable stochastic experiment into a verifiable, transparent asset. Through glass-box architectures, Dual RAG pipelines, and the rigorous application of spatial and semantic boundaries, we can finally build human-centered AI systems that are genuinely business-ready. It is time to stop prompting the machine and start engineering the context.

Less hype, more engineering. Pull up a chair.

Subscribe now

AI-assisted development is becoming the new standard. Learn how to use OpenAI Codex effectively in real engineering projects through a live, instructor-led bootcamp.

Date: Saturday, July 25, 2026
Time: 9:00 AM – 3:00 PM EST
Format: Live Virtual Bootcamp
Hosted by: Packt Publishing

What you’ll learn:

• How to use OpenAI Codex as a practical coding partner

• How to plan, build, debug, test, and review code with AI assistance

• How to apply Codex across real-world developer workflows

• How to complete hands-on projects and coding exercises

• How to improve code quality, productivity, and delivery speed using AI-assisted development

How this helps your career:
AI coding tools are becoming part of everyday software development. Learning how to work effectively with Codex can help developers become faster, more confident, and more valuable in AI-enabled engineering teams. This bootcamp helps participants strengthen their practical development workflow, improve productivity, and build skills that are increasingly relevant across modern software roles.

Certificate included:
Participants will receive a Packt certificate, which can be added to their LinkedIn profile, resume, or professional portfolio to showcase their learning and commitment to practical AI-assisted development.

Best suited for:
Technically professionals, developers, software engineers, technical leads, and AI builders who want to use OpenAI Codex to simplify coding workflows and improve engineering productivity.

This bootcamp helps you stay ahead so you can ship more, grow faster, and take on higher-impact work.

Register Here

Rick Spencer, General Manager for Technology and Product at SUSE, has looked hard at how the industry is measuring AI’s effect on engineering. “I consider that garbage vanity metrics,” he says, calling them unhelpful.

His argument for what to track instead is one of the more clarifying things an engineering leader can hear right now, because it separates the numbers that look like progress from the numbers that actually represent it.

Output is cheap; impact is what counts

The core of Spencer’s position is a distinction between output and impact, and it matters because the two come apart precisely when AI enters the picture. AI makes output cheap as the lines of code, pull requests, and token counts all mount up when agents are doing the writing, which makes them exactly the wrong thing to measure if what you care about is value delivered. “We’re really tending away from measurements that measure output and utilization, and we’re trying to focus on impact,” he says. A leaderboard that ranks engineers by how much their agents produced does not tell you who is solving the hardest problems or keeping customers safe. It tells you who is generating the most volume, and in an AI-assisted world, that number is close to meaningless.

There is also a structural reason the standard tooling does not fit SUSE, and it applies to more organizations than it might first appear. Much of the available measurement tooling assumes a particular shape of company. “They really assume you’re a proprietary software company where everyone’s working on a single code base,” Spencer explains, “which is just not how an open-source enterprise works.” His engineers work across hundreds, sometimes thousands, of repositories, where the maintenance work on each one differs enormously. A per-developer comparison across that landscape measures the shape of the work far more than it measures the contribution of the engineer, which is why he treats developer-to-developer comparison as fundamentally low value rather than merely imperfect.

The reporting burden itself is part of his objection, and it is a point leaders setting up AI dashboards should sit with carefully. A measurement regime that requires engineers to generate weekly utilization reports spends the very time it claims to be optimizing. “I’d rather have them working than reporting,” Spencer says. The instrument meant to measure productivity eventually becomes a tax on it.

What SUSE tracks instead

Rejecting vanity metrics only helps if there is something better to put in their place. And Spencer shares how SUSE measures business impact in terms that connect directly to what customers actually receive. “How fast are CVEs being addressed, how fast are patches being backported, how fast are our L3 responses getting closed while maintaining the same NPS score,” he underscores, listing what his teams track. The common thread is that each one is an outcome the customer feels, not an activity the engineer performs. AI has been applied to exactly these areas, so measuring the speed and quality of those outcomes tells you whether the AI is doing anything worth its cost, which is the actual question worth asking.

This shift from output to outcome reframes what a metric is for in the first place. A CVE response time captures whether the organization is keeping customers safe faster than it used to. A backport speed captures whether stable releases are getting their fixes without the manual grind that used to gate them. These numbers move because the underlying work got genuinely better, not because more text was generated, and that is the property that makes them trustworthy. They are also far harder to game, because the only way to improve them is to actually improve the thing the customer depends on.

Give managers visibility, not a leaderboard

None of this means SUSE ignores cost or utilization entirely, and the distinction Spencer draws here is the one that keeps the approach from collapsing into either negligence or surveillance. The company is building dashboards that give engineering managers visibility into their team’s cost and utilization, but the purpose is coaching rather than ranking. The unit of analysis is the team, and the question it answers is diagnostic. Spencer gives the example of a manager with an eight-person team noticing the numbers and asking the right kind of question. “We’re burning a lot of tokens. What are we actually doing that’s burning that many tokens? I’m not sure we’re getting value out of that.” The inverse matters just as much, where purchased seats for a code assistant sit unused, and the manager asks whether there are places the team should be drawing value that it is currently leaving on the table.

The governance side of that picture, including how SUSE keeps agents and their costs inside a boundary it can stand behind, is covered in a companion piece, How SUSE Runs AI Without Losing Control.

The difference between this and a leaderboard is not subtle, and it is the heart of the leadership lesson. A leaderboard exposes individuals and turns measurement into a game engineers play against each other, a game Spencer is explicit has nothing to do with customer value. Team-level cost visibility used for coaching does the opposite. It gives a manager the information to guide the team toward better use of the tools without making any individual engineer feel watched. “We’re really trying to decentralize and allow engineering managers to guide their teams on getting the most value out of the AI,” he says, “without it becoming like a leaderboard game where developers feel like they’re exposed.” The data exists to help the manager help the team, not to rank the team against itself.

The principle holding it together

What makes Spencer’s approach more than a list of preferred numbers is the principle holding it together, which is that measurement should serve the work rather than distort it. Every choice he describes follows from that one idea. Impact comes before output because output is the thing AI inflates. Team-level diagnostics come before individual leaderboards, because the goal is coaching rather than competition. Business outcomes come before activity counts, because outcomes are what customers actually receive. The decentralization to engineering managers reflects the same conviction that the people closest to the work are best placed to judge whether the AI is helping, given the right information and trusted to use it well.

The deeper point for any leader standing up AI measurement is that the easy numbers and the useful numbers are not the same, and AI has widened the gap between them. The figures that are simplest to collect, lines of code, tokens, and per-head utilization, are the ones AI has made least meaningful. The figures that matter, the speed and quality of the outcomes customers depend on, take more thought to define and more care to track. Spencer’s argument is that the effort is the job. “Let’s focus on the impact,” he says, “the business impact, not on the utilization.” For engineering leaders deciding what belongs on a dashboard as agents reshape their teams, that is the distinction worth getting right before the vanity metrics calcify into the way the organization sees itself.

This creates a pervasive dissonance: the expectation of industrial reliability built atop probabilistic generation.

The current noise suggests that if we simply scale the parameters, refine the system prompts, or throw more compute at the problem, true autonomous reasoning will spontaneously emerge.

To cut through this noise, we must confront an uncomfortable reality. Unconstrained probabilistic generation cannot serve as the kinematics for reliable robotic or enterprise execution. If we are to build truly agentic systems, we must move beyond the brittle, zero-context art of prompting and embrace the rigorous, transparent discipline of context engineering.

The limits of the translation lattice

At their core, LLMs do not guarantee structured reasoning across steps. They act as translation lattices operating on lossy text compressions, fundamentally disconnected from the non-verbal, experiential grounding that characterizes human thought. They are magnificent at semantic association but possess no innate understanding of physical or logical bounds. When an agent relies solely on an LLM’s internal weights to reason, it runs headfirst into the reality of the Probability Decay Theorem (PDT), a multiplicative reliability collapse effect. The theorem was derived through multiple real-world implementations by the author.

The PDT governs the multiplicative collapse of chained rules. In single-turn interactions, an LLM might succeed 95% of the time. But in agentic workflows requiring multi-step execution, the probability of success does not simply carry an additive exception load PDT₁. Instead, it collapses under multiplicative chained probability PDT₂. A chain of ten probabilistic bets, each with a 95% success rate, yields an overall reliability that is entirely unacceptable for real-world enterprise operations.

If an autonomous agent executes a sequence of tasks where each step is 95% reliable, the multiplicative collapse of the Probability Decay Theorem dictates that it takes just 14 consecutive steps 0.95¹⁴ ≈ 0.488 for the overall success rate to plummet below 50%, rendering the entire workflow less predictable than a blind coin toss. This shows that after enough chained steps, reliability rapidly degrades below usable thresholds, even when individual steps appear highly accurate.

LLMs operate on statistical associations in text without explicit grounding in physical or logical constraints. This means that mapping raw signals to lexical labels will always incur an irreducible loss due to the polysemy inherent in language. These models are structurally bounded by the imprecision of polysemy. Treating their hallucinations as mere software bugs to be patched by longer text prompts ignores the mathematical reality of the medium.

The embodied reality factor

To understand how to fix this, we must look at the lineage of modern embodied AI, tracing back to industrial automated systems. Physical robotics and real-world supply chains cannot operate on unconstrained probabilistic generation. Instead, algorithmic logic must be ruthlessly translated into the thermodynamic and spatial kinematics of the physical world. This absolute reliance on physical bounds forms the applied foundation for defining hard limitations such as spatial collisions, reaction times, and hardware constraints, which are not errors. They are the foundational constraint forces that dictate the accuracy of a signal. This principle inherently solves the simulation-to-reality (Sim2Real) transfer gap. If we want our digital agents to interact reliably with the real world, we must enforce digital representations of physical limits to ensure their reasoning remains strictly bounded by the reality factor of the environment.

If reliability cannot emerge from probabilistic generation alone, then it has to come from somewhere else. That somewhere else isn’t another prompt. It’s context, engineered as a first-class component of the system.

That’s where we’ll begin in Part 2. See you next Thursday!

His career has moved fluidly between academia and industry, from teaching advanced AI courses at the American University of Beirut to advising teams on turning machine learning ideas into systems that can be trusted.

He is also known for his candid take on the current AI landscape, especially the gap between meaningful engineering and what he often calls AI slop.

In this conversation, Ammar challenges one of the most common assumptions in agent development: that a correct answer is evidence of a successful agent. He explains why reliability lies in the path an agent takes, not just in the result it produces, and why evaluation must evolve from output scoring to a discipline that measures behaviour and trustworthiness in production.

Most teams think they’re evaluating agents, but they’re actually not. Where do you see the biggest illusion of evaluation today?

The biggest illusion is that teams think they are evaluating an agent when they are only evaluating the final answer.

That works well for a chatbot. But an agent is different. It plans, chooses tools, passes arguments, reads observations, retries, stops, and sometimes takes action. A final-answer score hides most of the actual failure surface.

An agent can produce a good-looking answer after calling the wrong tool, wasting ten steps, misreading a tool result, or ignoring a failed call. From the outside, the answer may look acceptable. From a reliability perspective, the run is not acceptable.

So the illusion is: “the answer looked right, therefore the agent worked.” However, what you need to know is whether the path was valid, efficient, grounded, and safe.

You break evaluation into component, trajectory, outcome, and adversarial layers. Where do most teams underinvest, and what failures does that lead to?

Most teams underinvest in trajectory evaluation and adversarial evaluation.

Outcome evaluation is the easiest layer to reach for because the final answer is visible. Component evaluation is also fairly intuitive once tools are involved: did it choose the right tool, did it pass valid arguments, did the plan make sense?

Trajectory evaluation is harder because you need structured traces and assertions over the run itself. But this is where many production failures live: loops, duplicate calls, silent retries, no recovery after a tool failure, unnecessary detours, high latency, high token cost. Two agents can produce the same answer, but one gets there in four clean steps, and the other gets there through an expensive, brittle path. Output scoring treats them as equal. Production does not.

Adversarial evaluation is also underbuilt. Teams may try a few prompt-injection examples manually, but they rarely turn those attacks into a versioned regression suite. That leads to a false sense of safety. A guardrail can look very strong against the examples it was designed for, while still being fragile against slightly different payloads.

Agent failures only show up after deployment. What’s the hardest failure mode to catch early, even with a good evaluation setup?

The hardest failures are the ones that look like successful runs.

A tool returns something plausible but incomplete. The agent takes a reasonable-looking path. The final answer is fluent. No exception is thrown. But the answer is weakly grounded, the evidence is stale, or the agent skipped a recovery step after a bad observation.

These failures are hard because they do not announce themselves as failures. They show up later as retries, edits after the answer, escalations, user abandonment, or quiet loss of trust.

The other hard category is drift. A hosted model changes, a tool schema changes, retrieval content shifts, or user traffic moves into a different distribution. Nothing “breaks” in the traditional software sense, but the agent becomes less reliable. This is why offline evals need to connect to production monitoring. A test suite is necessary, but it is not the whole system.

LLM-as-a-judge is becoming a default pattern. Where does it actually work well, and where does it quietly break?

LLM-as-a-judge works well when the task is bounded, the rubric is explicit, and the judge has the evidence they need. It is useful for rubric-based scoring, regression checks, pairwise comparisons, and multi-dimensional outcome evaluation, especially when you calibrate it against human labels.

The important part is that the judge itself has to be evaluated. I would not trust a judge just because it is an LLM. I would look at correlation with human labels, agreement rates, mean absolute error, and performance by rubric dimension.

Where it quietly breaks is when teams use it as an uncalibrated oracle. Judges often reward verbosity, prefer answers in a certain style, miss missing citations, or give a strong score to an answer that is polished but not grounded. Overall scores can also hide weak dimensions. For example, a judge may be decent on safety or format, but poor on groundedness, which is often the dimension that matters most for a research or retrieval-heavy agent.

So I see LLM judges as useful evaluators, not authorities. They need rubrics, evidence, calibration, and periodic human audit.

If you had to audit an agent system in production with very limited time, what signals or metrics would you look at first to decide if it’s reliable?

Aggregate success rate is often the last metric I look at. I would start with the traces behind the failures and the production signals that users generate when the agent is not working.

The first signals I would inspect are abandonment rate, retry rate, escalation rate, clarification rate, thumbs down, and edit-after-answer rate. Those are often more honest than a dashboard success metric.

Then I would look at trace-level reliability: number of tool calls, duplicate calls, loop-like behaviour, failed tool calls, recovery after failure, latency, and token cost. A reliable agent should not only get the answer right; it should get there through a path that is stable and explainable.

I would also check whether offline evals are tied to production: are failed production examples converted into regression tests? Are adversarial cases versioned? Are judge scores calibrated against human labels? Are there no-go gates for safety, groundedness, cost, latency, and step count?

With limited time, I am looking for one thing: whether the team has connected offline evaluation, online monitoring, and regression gates. If those are disconnected, reliability is usually more assumed than measured.

As organizations continue to explore what AI can and should do, voices like Ammar’s help bring the discussion back to the questions that matter: What problem are we really solving? Can the system be trusted? And are we building something meaningful, or simply adding more noise to an already crowded field?

If you’d like to continue exploring these ideas, Ammar will be speaking at Agent Evals Bootcamp on June 27th, where he will turn these ideas into a hands-on framework for evaluating agents across tool use, planning, trajectories, outcomes, regressions, and adversarial failure modes before deployment.

Of course, this won’t do for us. What’s the point of building custom AI pipelines if they sit locked in a dark room? To make that logic useful, we need to connect it to the real world. We build user authentication, set up secure access controls, write adapters for communication apps like Telegram or Discord, and configure reliable task schedulers.

By combining LangChain with the OpenClaw ecosystem via the LangClaw framework, you can bridge this gap in an afternoon.

Why combine OpenClaw and LangGraph?

OpenClaw began as a configuration-first, out-of-the-box personal AI runtime. It provides developers with a pre-wired ecosystem that includes direct messaging connectors for platforms like WhatsApp and Telegram, persistent memory layers, and native browser automation tools.

Instead of writing code to manage how an agent communicates with a chat application or remembers previous conversations, users simply write markdown files (SKILL.md) to define functions. OpenClaw handles the operational runtime, API routing, and state storage automatically.

While this structure is ideal for standard automation, engineering teams often require deeper programmatic control over their agentic reasoning. They need complex, multi-step state machines, advanced semantic search pipelines, and conditional execution paths.

This is where LangChain excels, and it is exactly why the two systems are paired together.

LangClaw as the glue layer

LangClaw acts as the architectural bridge between these two worlds. It is a declarative, Pythonic framework that compiles complex LangChain reasoning loops directly into an OpenClaw-compatible runtime engine.

Instead of managing external configuration markdown files, you use clean Python decorators to build tools, establish role-based access controls, and schedule autonomous background tasks.

If you’re the kind of person who reads something like this and immediately wants to see the code, we’ve documented a complete LangGraph + OpenClaw build that turns these ideas into a working corporate intelligence agent, covering everything from subagent orchestration and scheduled research tasks to permissions, messaging, and token-efficient utility commands.

Get the playbook here

Let’s build one.

The first step is creating the primary agent and establishing its responsibilities through a system prompt.

import os
from datetime import datetime
from langclaw import Langclaw
from langclaw.gateway.commands import CommandContext
# Initialize the master agent application interface
app = Langclaw(
 system_prompt=(
 “## Corporate Intelligence Agent\n”
 “You are a corporate intelligence analyst. You track market trends “
 “and draft precise outreach sequences based on current events.\n”
 “Delegate deep multi-source research tasks to the web-researcher subagent.”
 ),
)

Tools allow the agent to access capabilities beyond language generation. In this example, the agent can retrieve market intelligence data.

# Custom Tool: Data Extraction Layer
@app.tool()
async def fetch_market_news(sector: str) -> dict:
 “”“Extract breaking market developments for a specific commercial sector.”“”
 # This acts as a standard tool available to your core reasoning models
 return {
 “timestamp”: datetime.now().isoformat(),
 “sector”: sector,
 “headline”: “Warehouse Automation Demands Surge Amid Labor Reshuffling”,
 “body”: “Logistics providers face margin pressures, driving immediate infrastructure upgrades.”
 }

Rather than placing all reasoning responsibilities inside a single agent, LangClaw allows specialised subagents to handle focused tasks.

# Subagent Delegation: Isolated Reasoning Context
app.subagent(
 “web-researcher”,
 description=”Performs deep multi-step web analysis and synthesis.”,
 system_prompt=”You are an investigative researcher. Identify technical pain points and list them.”,
 tools=[”web_fetch”, “web_search”],
 output=”channel”, # Streams output directly back to the active user channel
)

Role-based permissions make it possible to expose different capabilities to different groups of users.

# Role-Based Access Control: Security Gateways
app.role(”premium_analyst”, tools=[”*”])
app.role(”basic_user”, tools=[”fetch_market_news”])

Agents do not need to wait for user prompts. Using cron schedules, they can execute tasks automatically.

# Automated Execution: Background Scheduler
@app.cron(”0 8 * * 1-5”) # Executes Monday through Friday at 8:00 AM
async def automated_morning_brief():
 “”“Scans market trends autonomously and publishes digests to the user feed.”“”
 raw_data = await fetch_market_news(sector=”Logistics”)
 instruction = f”Synthesize this market report for the main channel: {raw_data[’body’]}”
 # Compile runtime structure down to LangGraph for invocation
 compiled_agent = app.compile()
 analysis = await compiled_agent.ainvoke(
 {”messages”: [{”role”: “user”, “content”: instruction}]},
 config={”configurable”: {”subagent”: “web-researcher”}}
 )
 # Broadcast the completed report directly to your active chat channel
 await app.gateway.send_message(
 channel=”management_feed”,
 text=f”🔔 **Automated Market Digest**\n\n{analysis[’messages’][-1].content}”
 )

Not every task requires an LLM. For predictable workflows, slash commands can bypass the model entirely.

# Slash Command: High-Speed Utility Routing
@app.command(”template”, description=”Generate static outreach copy without AI overhead”)
async def outreach_template_cmd(ctx: CommandContext) -> str:
 “”“Returns a structured outreach framework directly, bypassing the LLM entirely.”“”
 args = ctx.message.text.split(’”’)
 client_name = args[1] if len(args) > 1 else “Colleague”
 # Bypassing the LLM completely saves computation costs and provides sub-millisecond responses
 return (
 f”📩 **System Template Generated**\n\n”
 f”Hello {client_name},\n\n”
 f”I noted your team’s footprint in the logistics sector. Given recent automation trends, “
 f”I wanted to connect to share notes on how regional hubs are optimizing their margins.\n\n”
 f”Regards,\n[Internal System]”
 )

Once the components are assembled, launching the runtime is straightforward.

if __name__ == “__main__”:
 app.run()

Once the runtime is launched, the agent becomes more than a collection of functions and workflows. It becomes a persistent service that can receive requests, execute tasks, and communicate results through your chosen channels.

Let’s see how that works.

What happens when the agent goes live

When you invoke app.run(), LangClaw spins up a persistent asynchronous gateway server. It does not wait for a single terminal input; it establishes a permanent network listener.

Interaction happens entirely through your chosen deployment endpoints:

• Production chat channels: By adding tokens for Telegram, Discord, or Slack into your .env configuration file, your script logs into those platforms as a live bot user.

• LLM invocations: When a user types a natural language question into the chat, LangClaw pipes the string into your LangChain model, runs the necessary code tools, and formats the response.

• The slash command route: When a user types /template “Alex”, the gateway identifies the leading slash character. It isolates the instruction, skips the LLM reasoning step, runs the native Python function, and returns the text.

  ---

Hosting your agent for less than £5 per month

Deploying a traditional enterprise application often brings significant server management costs. However, because this framework relies on an event-driven architecture and asynchronous IO libraries, it requires minimal computing resources.

You can host this entire stack for under £5 per month using modern cloud infrastructure.

Virtual Private Servers (VPS)

Platforms like DigitalOcean, Hetzner, or Linode offer entry-level Linux instances with 1GB of RAM and 1 CPU core. Because LangClaw uses a non-blocking asyncio event loop, a single micro-instance can easily manage hundreds of concurrent chat messages and automated cron tasks without breaking a sweat.

Process Management

To ensure your agent runs continuously, you wrap the script using a Linux system process supervisor like systemd or Supervisor. If the server reboots or encounters an unhandled API timeout, the operating system brings the agent back online instantly, providing 99.9% uptime with zero manual maintenance.

We’ve set up an agent, we’ve talked about messaging and about the advantages of full control, about the hosting. Building your automation stack with this methodology yields three immediate operational improvements:

• Zero-token utilities: By routing routine structural actions through slash commands, you bypass LLM token fees entirely. This provides instantaneous responses while eliminating API runtime costs for predictable tasks.

• Decoupled architecture: Your core business logic remains completely separate from your communication channels. If your team decides to migrate from Telegram to Slack, you alter a single environment variable. The underlying tools, subagents, and schedules require no modifications.

• Production readiness: You move from an abstract local script to an enterprise-grade agent. The system manages its own user security roles, schedules its own data retrieval cycles, and operates autonomously around the clock.

Let us know how you get on!