A deep search agent is meant to do more: break the problem into smaller tasks, choose the right sources, check what it finds and return for another pass when the evidence does not hold up.

We’ll use finance as the worked example because it makes these moving parts easy to see, but the same architecture applies wherever an agent must research before it responds.

The core loop

The fundamental pattern of a deep search agent can be expressed as a four-phase loop:

The agent plans subtasks, executes them using tools, validates the results and either replans when it finds gaps or synthesizes a final report.

Phase 1: Plan

The agent receives a research question and decomposes it into subtasks with explicit dependencies.

For a comparison of NVIDIA, AMD and Intel, for example, a planner might assign separate tasks for gathering each company’s financial data, reviewing market news and analysing filings. Those tasks can run in parallel, while calculating growth rates and writing the final comparison must wait for their results.

Good plans have measurable outcomes such as “fetch revenue data” rather than vague goals such as “understand the company.”

Phase 2: Search and execute

The agent executes each subtask using the appropriate tools. In our financial example, those might include data APIs, SEC EDGAR, web search and earnings-transcript databases. In another domain, the sources could be internal documentation, research papers, case law or product data.

This is where the “deep” in deep search comes from. The agent does not fire one search query and return the first plausible answer. It selects the right tool for each subtask, handles errors and retries, and accumulates its findings in a scratchpad.

Phase 3: Validate

This is the phase that separates deep search from naive automation. The agent checks its findings for:

• Internal consistency: Does a figure from a regulatory filing match the company’s earnings transcript?

• Completeness: Did we successfully gather information for every company?

• Reasonableness: Is 500% year-over-year growth plausible, or did the agent parse a table incorrectly?

• Missing evidence: If one API call failed, does the task need to be retried through another source?

Validation matters beyond finance. Any research agent can produce a polished answer from incomplete, contradictory or incorrectly parsed evidence. The validator’s job is to notice before that answer reaches the user.

Phase 4: Synthesize or replan

If validation passes, the agent combines its findings into a structured report. If validation exposes gaps or errors, it returns to the planning phase with an updated understanding of what is missing.

The loop continues until the agent has enough validated evidence to produce its final response.

The components behind the loop

Making each phase work requires several supporting components operating together:

The orchestrator manages the research loop, delegating work to the planner, tool executor, validator and synthesizer. A context manager and scratchpad prevent token overflow, preserve intermediate findings, and maintain an audit trail.

Every implementation combines these components in some form. What changes is how much responsibility each component carries.

Three architectural approaches

Real implementations reveal three broad approaches to building deep search agents. Each makes a different trade-off between complexity, cost and capability.

Approach 1: Single agent with ReAct loop

The simplest approach uses one LLM agent with access to tools, running in a ReAct—or Reason and Act—loop.

Given a task, the agent reasons about what to do next, takes an action, observes the result and repeats until the task is complete. One model handles planning, execution, validation and synthesis within a single conversation context.

Step-DeepResearch, an open-source 32B model fine-tuned for deep research, is one example of this architecture.

This approach is the easiest to implement and debug. However, it can struggle with context overflow on complex assignments, offers limited parallelism and provides no role specialization.

It is best suited to focused research tasks with limited scope.

Approach 2: Multi-agent with specialized roles

The second approach assigns each phase of the research loop to a separate agent. Each agent receives its own prompt, tools and context window, then passes structured output to the next stage.

Dexter is one example. It uses four agents—planning, action, validation and answer generation—alongside routing that directs requests to the appropriate data source.

The advantage is a clean separation of responsibilities. Context remains isolated, and failures are easier to trace to a particular stage. The trade-off is higher token usage and additional coordination overhead.

This approach suits comprehensive analysis where quality matters more than minimizing cost.

Approach 3: Deep orchestrator with MCP

The third approach uses a central orchestrator to manage a dependency graph and execute tasks through standardized Model Context Protocol servers.

The orchestrator plans upfront, verifies the output of each step and replans only when something fails.

The AI Alliance Deep Research Agent for Finance, built with mcp-agent, follows this pattern. It combines upfront planning, a task dependency graph and deterministic code-based checks.

One of the project’s key findings was that deterministic validation performed more reliably than asking an LLM to verify factual claims. If a number must match, a schema must validate or a required field must exist, code can often check it more consistently than another model call.

The MCP approach allows compatible tools to be added through a standardized interface and has proved suitable for production use. Its main cost is the additional infrastructure required to configure and operate it.

It is best suited to systems where reliability, traceability and standardized tool access matter.

Which approach should you choose?

For learning and prototyping, begin with a single-agent ReAct loop. It exposes the mechanics without requiring a large supporting system.

When the research becomes broader, the context grows or the cost of a wrong answer increases, specialized agents provide clearer boundaries and stronger validation.

For production systems that depend on several tools, require traceable execution or need deterministic checks, an orchestrator and task graph offer the strongest foundation.

The architectures differ, but the underlying discipline does not: plan the work, retrieve the evidence, validate what came back and synthesize only what survives those checks.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Keep every message, tool call, decision and error. Embed the lot. Trust that the right fragment will return when the agent needs it.

Nacho Martínez Rincón’s objection is wonderfully direct:

“Storing everything is not memory.”

Memory, he argues, begins when a system decides what to keep, what to revise and what it can safely forget.

For our first live episode of the Agentic Engineering podcast, I spoke with Nacho, an AI Engineer and Developer Advocate at Oracle and co-instructor of DeepLearning.AI courses on agent memory and adaptive agents. We began with a simple question: what should an agent remember?

The audience made the question much harder.

Before we dive in, a quick word from PlayerZero.

Before you add another agent, give them all the same context

PlayerZero is the AI orchestration and context platform for engineering teams

Code isn’t the bottleneck anymore. Context and review are. Every agent your team adds starts from its own partial picture, and your senior engineers end up filling the gaps by hand.

PlayerZero builds one engineering world model from your code, tickets, and telemetry, then orchestrates agents across product, engineering, QA, support, and SRE on top of it. Work moves between teams with full context attached. When an agent reaches the edge of what it should decide, it pulls in the right person with the investigation already done. That decision goes back into the model, so the next agent inherits it.

Key Data doubled its release velocity. Cayuse catches 90% of issues before production.

See how it works

A larger context window does not solve the problem

An agent’s working context is useful during a task. It holds the conversation, the files being examined, and the information needed for the next decision.

But allowing that context to grow indefinitely eventually works against the agent. Important instructions begin competing with old messages, irrelevant details and earlier attempts. Nacho calls this context rot: the system technically has more information, yet becomes less capable of using it well.

Long-term memory should not become an overflow cupboard for everything that no longer fits in the context window.

Most of what happens during a task should disappear with it. The useful part is whatever could change a future decision.

Remember the lesson, not the entire conversation

One of the most practical ideas from the conversation was to turn completed work into a reusable lesson.

Suppose an agent encounters an error, investigates it and eventually finds a fix. Saving the full exchange leaves the next agent to reconstruct the experience from hundreds of messages.

A better memory might contain only:

• the problem it encountered

• the error that exposed it

• the action that resolved it

Those traces can then be consolidated into a skill the agent retrieves when it meets a similar problem again.

The important distinction is that the agent is not replaying its past. It is carrying forward what the past taught it.

Nacho also recommends waiting until a task has finished before creating most durable memories. Only then can the system distinguish a temporary detail from a lesson worth keeping. The obvious exception is a dangerous failure that needs an immediate “never do that again” rule.

Memory creates a new attack surface

The discussion became more uncomfortable when we reached memory poisoning.

An instruction that once disappeared at the end of a session can now survive for weeks. A compromised skill file, malicious update, or false memory may quietly influence decisions long after the original interaction has been forgotten by the user.

Nacho explains why memory systems need provenance, version histories, and checks that reveal when a trusted file has changed. The system must remember—but it must also be able to show where that memory came from and what has happened to it since.

The live audience took us further into enterprise memory architectures, decay strategies, shared memory between agents, and whether these systems can be built without deep coding expertise.

Watch the full conversation for Nacho’s practical (and provocative) answer to the question of what an agent should carry into its next session.

I’ll be back on Thursday. Until then, keep building!

Editor’s note

I have to admit, typing #10 into this title felt good. After a few weeks of series, webinars and worthwhile detours, we’re back to our usual issues. I hadn’t realised how much I’d missed seeing the issue number in its usual place.

This week’s issue began with a framework survey from a Packt book on financial agents. But as I worked through it, the comparison felt useful beyond finance. The same decisions (how much orchestration you need, where your data lives, how outputs are validated, and how much control you are willing to hand to a platform) appear in almost every agent project.

So I’ve condensed the survey into a practical map of ten frameworks.

The aim is not to leave you knowing all ten frameworks inside out. It is to help you narrow the field and understand which trade-offs matter for your work. Start with the frameworks already on your shortlist, then use the comparisons to see what each would give you—and what your team would have to maintain.

Before we compare the frameworks, a quick word from PlayerZero. The frameworks we discuss in this issue shape how agents are built and orchestrated. But PlayerZero addresses another part of the stack, giving those agents a shared, current view of your code, tickets, and telemetry.

Before you add another agent, give them all the same context

PlayerZero is the AI orchestration and context platform for engineering teams

Code isn’t the bottleneck anymore. Context and review are. Every agent your team adds starts from its own partial picture, and your senior engineers end up filling the gaps by hand.

PlayerZero builds one engineering world model from your code, tickets, and telemetry, then orchestrates agents across product, engineering, QA, support, and SRE on top of it. Work moves between teams with full context attached. When an agent reaches the edge of what it should decide, it pulls in the right person with the investigation already done. That decision goes back into the model, so the next agent inherits it.

Key Data doubled its release velocity. Cayuse catches 90% of issues before production.

See how it works

LangChain/LangGraph: Modular orchestration and RAG

LangChain is arguably the most widely adopted open-source framework for building LLM-powered applications. It provides modular abstractions for prompts, tools, memory and chains of reasoning. Over time, it has grown into an ecosystem that supports RAG, tool integration and multi-agent orchestration.

LangGraph, a companion project, extends LangChain by introducing graph-based agent workflows. Instead of linear prompt chains, LangGraph allows for branching, looping and controlled execution graphs, making it a highly customisable and powerful way to design complex agent systems.

• Strengths: Combines broad tool and data integrations with RAG and flexible graph-based orchestration. It supports branching, persistent state and human approval points for complex workflows.

• Limitations: Its flexibility brings complexity, performance overhead and a steep learning curve. Enterprise observability, compliance and deployment require additional tooling, while fast API changes can leave older guidance behind.

Google ADK

The Google Agent Development Kit is Google’s framework for developing and deploying AI agents. It is optimised for its managed Agent Engine runtime and the Gemini family of models, but designed to be model-agnostic. Unlike lightweight open-source tools or experimental research frameworks, ADK is positioned as an enterprise-grade platform for organisations that need production-scale agent systems with security controls.

• Strengths: Combines reasoning agents, deterministic workflows and custom Python logic within a graph-based runtime. Native Google Cloud integration adds enterprise scaling, monitoring and access controls.

• Limitations: ADK is powerful but heavyweight, with a smaller community ecosystem and considerable infrastructural lock-in once workflows depend on Google Cloud services.

CrewAI: Role-based multi-agent collaboration

CrewAI is an open-source Python framework designed for building teams of collaborating agents. Unlike frameworks that emphasise orchestration pipelines, CrewAI is built around the metaphor of a crew of specialists working towards a common goal.

Each agent has a role, a goal, a backstory that defines how it behaves and access to tools such as APIs, code execution and retrieval functions.

• Strengths: Organises agents around recognisable roles, goals and delegation patterns. It supports sequential, parallel and hybrid workflows that resemble committees or specialist teams.

• Limitations: The open-source core requires additional tooling for production observability, validation and governance, and may struggle at enterprise scale.

AutoGen: Conversational orchestration

AutoGen, developed by Microsoft, is an open-source framework designed for multi-agent conversational systems. Its core philosophy is that agents should interact through dialogue, both with one another and with humans, to solve complex tasks collaboratively.

Instead of hard-coded pipelines, AutoGen uses natural-language conversations as the control mechanism. Agents communicate through messages, text or function calls, making workflows more flexible and interpretable.

• Strengths: Coordinates specialist agents through recorded conversations, with support for tools, code execution and human participation. This makes collaborative reasoning transparent and easier to inspect.

• Limitations: Conversational orchestration can be verbose, slow and expensive to scale. AutoGen is also now in maintenance mode, with Microsoft directing new projects towards Microsoft Agent Framework.

OpenAI Agents SDK

The OpenAI Agents SDK is OpenAI’s production-ready framework for building agent-based systems. Unlike experimental frameworks or open-source prototypes, the SDK is designed for enterprise deployment.

• Strengths: Provides handoffs, session memory, parallel guardrails, hosted tools and built-in tracing. These production controls make multi-agent workflows easier to operate and audit.

• Limitations: Its strongest tracing and hosted-tool capabilities remain tied to OpenAI, while the framework offers less freedom than LangGraph and may become expensive at scale.

PydanticAI: Type-safe agents for structured data

PydanticAI—stable at 1.0 since September 2025, with a substantially redesigned 2.0 released in June 2026—comes from the creators of Pydantic, the widely used Python library for data validation. It brings strict typing, schema validation and reliable structured outputs into the world of AI agents.

• Strengths: Enforces typed schemas, validates model outputs and retries malformed responses automatically. It integrates easily with Python APIs, data pipelines and larger orchestration frameworks.

• Limitations: PydanticAI validates output structure, not the quality of the model’s reasoning, and still needs another framework for complex orchestration.

Mistral AI agents: Lightweight, open-weight performance

The philosophy behind Mistral’s agent framework is deliberately lean and open-weight-first, in contrast to enterprise-heavy offerings such as Google ADK or the OpenAI Agents SDK. It aims to give developers fast, cost-effective and flexible agent infrastructure without vendor lock-in.

• Strengths: Offers efficient open-weight models alongside hosted agents with retrieval, code execution, memory and handoffs. Local deployment supports cost control and data sovereignty.

• Limitations: Its agent ecosystem is younger and offers fewer built-in governance and observability features, often leaving orchestration to third-party frameworks. Smaller models may also sacrifice reasoning quality for speed.

smolagents: Minimalist code-first agents

smolagents is Hugging Face’s lightweight, code-first agent framework, designed around a simple philosophy: agents should write and execute code to solve tasks. Unlike orchestration-heavy frameworks or enterprise platforms, smolagents makes every agent step a readable, runnable Python snippet.

• Strengths: Uses visible, executable Python as the agent’s primary way of solving tasks. Its lightweight, model-agnostic design makes agent behaviour easier to understand and prototype.

• Limitations: Designed primarily for learning and prototyping, it has basic orchestration, fewer integrations and limited enterprise governance. Any production use involving generated code requires careful sandboxing.

Claude Agent SDK: Safe-by-design agentic workflows

Claude, Anthropic’s family of models, is widely recognised for its safety-first design philosophy. The Claude Agent SDK—renamed from the Claude Code SDK in September 2025—is the agent harness that powers Claude Code: a production-hardened loop for gathering context, taking action and verifying results, exposed so developers can build their own agents on the same engine.

• Strengths: Combines sessions, subagents, hooks, Agent Skills and MCP connectivity with safety-oriented controls. Computer Use extends automation to systems that lack APIs.

• Limitations: It offers less orchestration flexibility than LangGraph or CrewAI and is tied to Claude models. Its community focuses more on MCP and Skills than on orchestration recipes.

LlamaIndex: Data-centric agents

LlamaIndex began as GPT Index, a framework for connecting large language models to external data sources. While best known for its RAG capabilities, it has since evolved into a data-centric agent framework with support for multiple agent types, introspection and orchestration.

• Strengths: Grounds agents in documents, databases, APIs and knowledge graphs. Its retrieval-focused agents, memory and event-driven workflows make it a strong data layer for data-heavy applications.

• Limitations: LlamaIndex is stronger at data grounding than orchestration and requires additional tooling for enterprise observability. Advanced capabilities such as knowledge graphs also demand more setup.

If you’re weighing these options for a real project, I hope the view below makes the trade-offs a little easier to hold in your head.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Denis Rothman thinks that definition stops being useful the moment an agent enters a real business.

Once an agent can retrieve private data, remember previous interactions, and take action, the question is not about whether it has enough context. It is whether it should have access to that context at all—and who will be accountable for what happens next.

For this episode of the Agentic Engineering podcast, I spoke with Denis, an AI systems architect, author and someone I’ve had the pleasure (and occasional pain :P) of editing across several books. He has spent more than four decades working with AI and enterprise systems, which means he is not especially impressed by ideas that were solved by database administrators 25 years ago and have now returned wearing an agentic label.

The context engine should not have to run the company

Denis’s central argument is that much of the work must happen before information reaches the agent.

Company data should already have an owner. Access levels should already be clear. Policies should be current, and unreliable information should be marked as such. The agent should not be expected to work out which version of reality the organisation meant to give it.

He describes a production system handling hundreds of thousands of tasks each day. The difficult work was not adding an LLM. It was mapping the approval hierarchy, cleaning the incoming data and deciding which actions were allowed to proceed.

That required months of workshops with people across the business. The AI was only one component.

A prompt is not a permission system

We now place an extraordinary amount of faith in instructions such as “do not reveal sensitive information” or “ask before taking action.”

But those instructions are still being interpreted by the same model we are trying to control.

Denis argues that the real limits should sit outside the prompt. The retrieval layer should refuse data the user cannot access. The system should block tasks that are not authorised. Budgets should have hard limits.

Then comes his most radical piece of agent architecture: a human being.

His preferred structure combines the agent doing the work, another agent pushing back on it and an experienced person checking what matters. Removing experts in the name of automation, he warns, often means removing the people most capable of noticing when the system is quietly wrong.

Eventually, the CFO enters the room

More controls create more cost. Denis’s answer is not to remove the controls. It is to ask whether the system still earns its place.

Track the tokens. Measure the outcome. Build a dashboard. If the agent costs more than the value it creates, stop the project.

“We’ve been living in AI Disneyland for a few years,” Denis says. “But it’s over.”

It sounds ominous—Denis hears that from me several times during the episode—but his argument is ultimately practical. The way to build ambitious AI systems is not to believe in them more. It is to give them clean data, hard boundaries and people experienced enough to say no.

Watch the full conversation for Denis’s wonderfully unsentimental take on agents, context governance and what happens when the demos finally meet the balance sheet.

That’s it for this one. We’ll pick up the conversation on Thursday.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

As AI makes code easier to produce, a different constraint is becoming harder to ignore: the human attention required to specify, review and trust it.

In this issue, Adrián González Sánchez, Technical Product Manager at Microsoft AI and author of Packt’s Generative AI for Leaders Playbook, explains how an AI harness can help teams protect that attention.

But first, here’s a practical 30-minute session from Infisical on giving agents access without exposing credentials.

Agent Vault in Action

Give Agents Access, Not Secrets

Your agent needs credentials to do its job, but a long-lived key stored in its config can end up in logs or get pulled out through prompt injection.

Join Infisical on October 21 at 1 PM ET for a 30-minute live demo of Agent Vault. You’ll see how agents can call APIs without ever seeing the underlying credentials, plus how to limit which services they can reach and cut off their access automatically when a session expires.

We’ll also show how to add these controls to Claude Code, Codex, or your own agent without code changes, and how to trace a request back to the agent that made it when you need to investigate.

Save your spot

I think for the last few years, we have been optimizing the wrong scarcity. AI adoption within software engineering is clearly settled: roughly 90% of developers use AI daily per Google’s 2025 DORA report, and GitHub merges 43 million pull requests a month, up 23% year over year. Regardless of the source, it is clear that code is no longer scarce or even expensive.

But delivery tells another story. The DORA report finds that AI amplifies whatever organizational system (e.g., team composition, collaboration workflows, assets) it lands in rather than fixing it. At the individual level, METR’s randomized trial found experienced developers 19% slower with AI on real tasks, while believing they were 20% faster. And for some strange reason, the industry has prioritized tokenmaxxing and individual per-developer “productivity” (vanity) metrics such as number of PRs enabled by AI, rather than focusing on the overall productivity of the teams, and the actual outcome and additional value generated thanks to AI.

So we can say that the constraint has moved. It is no longer about generating code; it is the processes and collaboration workflows within software engineering teams, and the human attention needed to specify, verify, and decide. And in these times of hyperactivity and multitasking, that attention is limited and is being spent pretty badly. I call this the cognitive attention deficit: the growing gap between what our AI systems produce and what our software teams can responsibly attend to.

The deficit has multiple answers, but from an engineering perspective, the emerging solution can be what we call the AI harness.

What a harness actually is (in the context of development teams)

The metaphor is exact: a harness does not make the horse stronger, but it makes its strength usable by directing force, preventing damage, letting the driver watch the road. An AI harness does the same for human attention: not a product you buy, but a set of artifacts and rules built once and reused, so judgment is spent only where judgment is required. Five components, in my experience:

Durable driving artifacts

Intent used to be carried by people. The engineer who wrote the code remembered why. In the case of AI agents with limited memory, intent must live somewhere durable: a co-authored spec for ambiguous and higher-stakes work, executable tests for crisp logic, given-when-then scenarios for user-facing behavior, etc. So basically everything that goes beyond simple and ephemeral prompts, and that can be reviewed, reproduced, or handed to a colleague within the same team.

Machine-checkable guardrails

Without this, “almost right” pieces sail through a tired human review at six in the evening. So here we move verification from tired humans to tireless machines: tests generated after humans define what correct means, evaluation suites, policy checks, or CI gates. Every automated check refunds attention to the questions machines cannot answer, while avoiding the trap of expecting humans to keep the review processes at the same pace as the AI-enabled development.

Engineering memory

When work moves at machine speed, no one holds the whole picture, and decisions get generally made blind. The answer is old-fashioned: decision records, kept current, mandatory in a narrow set of sensitive places rather than aspirational everywhere. The test: if the author leaves and the chat session is gone, does the “why” survive? If not, someone pays attention twice for the same decision.

Attention gates

Three practices, none of which cost money. First, assign a named reviewer when the work begins. An unassigned review queue is not a process; it is merely a hope that someone will take responsibility. Second, set a WIP limit for the review stage. Most teams limit the stages that were bottlenecks before AI while leaving the new constraint, human review, uncapped. And third, explicit confidence thresholds that determine when an agent should produce no output at all. An empty suggestion is cheaper than a plausible but incorrect one.

Delegation rules

The agent harness must clearly define who has decision-making authority. Evaluate each decision using three questions: Is it reversible? What is its potential blast radius? How ambiguous is the specification? If it is reversible, contained, and clearly specified, let the agent decide and stop spending human attention on it. If it is irreversible, far-reaching, or ambiguous, assign the decision in advance to a named person. The costliest failures I have seen were caused not by bad decisions, but by decisions nobody owned.

The traditional process protected scarce code-building capacity, so the gates sat in front of implementation. The harness protects scarce attention, so the gates move with fewer in front of the keyboard, and more around specification, verification, and decisions. Teams that keep old processes while using new AI tools tend to slow the machine and still exhaust the humans (something we are experiencing nowadays, too often); while teams with a proper AI harness run faster because attention is pre-allocated.

There is also a wellbeing argument we should stop calling soft. Review fatigue means people approve what they would have questioned a year ago, leading even to declining confidence in their own judgment. An unmanaged attention deficit is paid in burnout. So we can say that harnesses enable sustainable pace, made structural.

So where do we start?

Well, I don’t have a general answer. But if your review queue grows while releases stay flat, measure three numbers: time to first review, review duration, rework rate. They usually show where attention leaks and bottlenecks appear. Close the leakiest seam first: a WIP limit and reviewer of record if review is drowning, a real spec practice if rework climbs, decision records in your most sensitive directory if the same questions keep returning, one page of delegation rules, etc. But not all five components at once; do it progressively and keep improving your operating system and technical harness.

And remember, every new tool and model release increases the agent's capability and, therefore, its potential impact within the harness. As a result, the value of investing in robust controls, delegation rules, and review processes compounds over time. The winning software engineering teams will not be the ones with the strongest agents; everyone will have those as they progressively become a commodity. They will be the ones who decide deliberately and in advance where their human attention goes and where AI takes care of things.

Catch up with Adrián

Explore Adrián’s new Packt book, Generative AI for Leaders Playbook, connect with him on LinkedIn, subscribe to his Substack, or take one of his DeepLearning.AI courses. He also has a new course on software engineering teamwork coming soon, which is closely connected to the ideas explored in this article.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

This session grew from a question many (or all) of us have asked after an agent produces something impressive: Can I trust this enough to use it? Youssef Hosni and Luca Berton answer not with reassurance, but with the practical engineering that allows trust to be earned.

There is a particular moment many of us recognise when working with a coding agent. It produces something impressive, perhaps much faster than we could have written it ourselves—and then we pause.

Did it follow the repository’s conventions? Did it test the right things? What did it overlook? How much of this can we safely trust?

That pause is what this webinar is really about.

Youssef Hosni and Luca Berton show us that the answer is not simply a better model or a longer prompt. It is the engineering we build around the model: the agent harness.

Teach the agent how you work

Youssef begins with skills: reusable procedures that capture how you want recurring work to be done.

His demonstration uses a code-review skill that asks the agent to inspect correctness, consider edge cases, examine the relevant tests and support its findings with evidence. Instead of explaining that process again in every prompt, you teach it once and continue refining it as you learn.

He also draws an important distinction between the different parts of a harness:

• A prompt describes the task in front of you.

• CLAUDE.md or AGENTS.md explains how the project works.

• A skill preserves a reusable procedure.

• A tool allows the agent to take an action.

• A subagent handles a focused piece of work in a separate context.

It is a small shift in thinking, but a meaningful one. You are no longer hoping that the agent interprets one enormous prompt correctly. You are giving its work a structure it can return to.

Give yourself evidence, not reassurance

Luca takes the next step: how do you know the result is actually safe to accept?

His example starts with a subtle bug. Searching for active customers also returns those marked inactive, because the original function relies on substring matching. Claude Code fixes the implementation, but Luca does not stop at a plausible answer.

He adds a hook that automatically runs the unit tests whenever the agent changes a file. He then introduces a read-only reviewer subagent, working in a separate context, to compare the implementation against its acceptance criteria.

Finally, he adds guardrails that prevent the agent from touching protected files, altering its own configuration, or committing and pushing code. The agent can help with the work without quietly inheriting authority it was never meant to have.

Together, the demonstrations create a practical loop:

Provide context → invoke a skill → make the change → run automatic checks → review independently → enforce permissions → let a human decide.

What I appreciated most about this session is that it does not ask us to trust agents more blindly. It shows us how to create the conditions in which trust can be earned.

The aim, as Luca says near the end, is not simply more autonomy. It is autonomy with evidence.

Watch the full webinar to see Youssef and Luca build each layer—and to borrow the parts that could make your own work feel a little less uncertain.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

With more than 7,000 of us now reading Agentic Engineering, I wanted us to have somewhere those questions could go.

So we built the Agentic Engineering platform.

(If one of the workshops catches your eye, I’ve included a small extra discount for you further down. Continue reading!)

What we’ve put there

• Hands-on workshops and live Q&As with practitioners

• Working repositories, slides, and practical guidance from engineers and architects

• Perspectives from experts who have built and shipped these systems

Upcoming workshops cover agent skill engineering, Claude Code, graph engineering and more.

Each one is designed to leave you with something concrete to try and people you can return to when trying it raises new questions.

The Guild is the part I’m most excited about

Registering for a workshop also gives you access to the Guild, our live community chat with dedicated rooms built around each workshop. The conversation can begin before the session and continue after everyone has left the call.

We created it for that exact moment when you’re back at your desk trying to make an idea work in your own system.

You shouldn’t have to figure out what comes next alone 🫂

A little further ahead

Agentic Engineering Pro will be our year-round membership. It is for people who want ongoing access to our workshops, experts, resources and community rather than signing up for one event at a time.

For Pro members, that means:

• Every live workshop included with practitioner-led sessions throughout the year

• The full Packt library of books, videos and audiobooks available whenever you need to go deeper

• More direct access inside the Guild through weekly AMAs, work-in-progress reviews and a place to ask for help when you are stuck

Pro isn’t open yet, but the waitlist is.

Explore Agentic Engineering

The newsletter isn’t going anywhere. We’ll continue publishing the expert articles, practical lessons, and interviews you already receive. The platform gives you a way to take those ideas further.

Have a look around. I’d love to know what you think and what you’d like us to build next.

I’ll be back in your inbox on Thursday with our regular issue.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Four issues, several diagrams, and one familiar acronym later, we’ve reached the end of Decoding FDE.

We began by asking what a Forward Deployed Engineer actually does, separated the role from consulting and outsourcing, and traced the model from Palantir to the new generation of AI companies. For the final part, Henry turns the question back on you.

What kind of person thrives on the deployment front line, and does the way you work make FDE a good fit for you?

If you’ve been with us from Part 1, thanks for staying deployed.

Who is suitable for FDE and who isn’t

Through my previous work, I’ve identified five core drivers shared by high-agency professionals: curiosity, a spirit of exploration and innovation, the ability to learn independently, self-motivation, and strong hands-on competence.

These qualities are the entry ticket to FDE, but far from sufficient. Beyond these five drivers, the FDE role demands a set of very specific additional traits, and there are several personality profiles that are explicitly unsuited for it. I have seen too many outstanding engineers struggle to adapt after transitioning to an FDE position, and in most cases, the problem lies not in their capabilities, but in their personality and work preferences.

Five traits suitable for FDE

They do not resist sales and communication.

A typical FDE’s daily work does not involve writing code behind closed doors, but directly engaging with clients’ CTOs, business leads, procurement teams, compliance officers, and IT personnel. A common scenario looks like this. When a client’s CTO interrupts you mid-demo, an FDE should not respond with, “I’ll revise it and come back next week.” Instead, they should immediately open the IDE, adjust the prompt, and rerun it right there for the client to see. Making adjustments while the client is present is the norm for FDEs.

They embrace the grey area.

What an FDE receives is not a clear PRD, but a vague line such as, “We want to do something with AI.” The client may not even be able to articulate what they truly want, and it falls to the FDE to walk with them as they turn that nebulous expectation into something concrete. If you can only spring into action when presented with clear requirements, working as an FDE will leave you feeling anxious every day.

They have solid engineering fundamentals, without needing 10x-level excellence.

An FDE does not have to be the person with the cleanest code or the deepest algorithmic expertise in the company. What the role requires is end-to-end deliverability—the ability to hack together a clickable front-end page, set up a runnable back-end service, and connect the model to business data sources. In the realm of FDE, “good enough” is not a flaw, but a virtue.

They thrive on feedback-driven refinement.

A typical FDE’s work is full of moments when clients tell them to go back and redo something. A demo you put together today may be rejected by the business stakeholder tomorrow with, “This isn’t what I want.” A solution agreed last week may have to be completely rebuilt this week because the client has assigned a new executive to the project. The right fit for an FDE role takes such feedback as fuel, assumes full end-to-end ownership, and never shifts blame by claiming, “The requester wasn’t clear enough.”

They are sensitive to model boundaries.

This is the most technical and least explicit requirement. An FDE must be able to judge which tasks are suitable for an LLM, which are not, and how to design the appropriate fallback. Such sensitivity cannot be learned from academic papers. It can only be forged through repeated exposure to failure cases. As those failures accumulate, the FDE develops a kind of muscle memory for model boundaries—when to use RAG, when to rely on hard-coded rules, and when human intervention is mandatory.

Who may struggle in the FDE role

Those who hide in code.

An FDE spends roughly 50% of their time not writing code. Instead, they are occupied with client meetings, internal coordination, product discussions, and moving contracts forward. If your source of happiness is writing code uninterrupted for four straight hours, an FDE role will leave you in a state of chronic mental exhaustion.

Those who cannot get moving without OKRs.

The goal of an FDE is rooted in the client, not in a performance sheet. Progress is determined jointly by the client’s project milestones, changes in model capabilities, and your own judgement of the scenario. Those accustomed to knowing what to do only after receiving their OKRs will struggle to find an anchor point.

Those who value promotion more than delivering quality work.

FDEs can be at a disadvantage within the promotion systems of large tech companies. Metrics such as customer satisfaction, project closure, and code reuse often carry less weight in promotion reviews than lines of code written or deployment frequency. If promotion tops your list of work motivations, FDE may not be the right career path for you.

Those who resist commercial contexts.

FDEs must understand their clients’ P&L, ROI, procurement processes, and compliance requirements. If you naturally resent talking about money, contracts, or business logic, working as an FDE will make you feel as though you are betraying your technical ideals.

Five traits, four warning signs, and seven self-assessment questions all boil down to a single question. Are you willing to sharpen your product sense, engineering capability, and business judgement at the same time, within the same workflow?

FDE won’t be the last

FDE is one of the first clearly defined job profiles to emerge from the AI industrial revolution, with a formal title, defined salary bands, dedicated job descriptions, and validation from paying clients. It is not simply another name for a high-agency professional. It gives tangible form to the abstract ideas driving this wave of restructuring.

I think it is only the first role in the new division of labor to acquire a name. What comes next may include Forward Deployed PMs, Forward Deployed Designers, and Forward Deployed Researchers. Any role that is tightly coupled with customer scenarios and responsible for nurturing products to maturity amid ambiguity may develop its own “forward-deployed” variant.

Job titles may change, but the underlying logic remains the same. Model capabilities take the lead, Product Form plays catch-up, and job structures are redrawn around the workflow.

Three final notes

For technical professionals

FDE does not require you to be the best coder in the company, but it does require you to shift roughly half of your time from coding to client-facing work. If your answer is yes, the market window has just opened, and recruitment is accelerating across leading model companies, cloud vendors, and the in-house AI teams of major tech firms. If your answer is “no”, that is perfectly fine too. Other roles will emerge for you in this new division of labor.

For HR and OD teams

Beware of the disconnect between job titles and actual responsibilities. Your company may already have FDEs on the payroll under titles such as “Solution Expert”, “Industry Architect”, or “AI Application Engineer”. Identifying these employees, recognizing the work they already do, and giving them a career path that reflects it may be far more effective than recruiting new staff from scratch.

For managers

The FDE model is not only for external deployment. It can also be applied internally. Embedding several internal FDEs within business teams to carry the model team’s capabilities into real workflows may be far more effective than creating another AI department and holding ten more cross-team alignment meetings. Departmental silos are not dismantled by organizational design, but by a working demo.

Career transformation in the AI era is already underway, and FDE is the first signal flare. It tells us that model capabilities are advancing quickly enough to force entirely new roles into existence.

I would like to leave you with one concrete question. If three new positions appear on your company’s organizational chart three years from now, what do you think they will be?

Figuring out the answer may be more valuable than reading this entire article.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

In the last two weeks, we’ve established that FDE is neither consulting in AI clothing nor outsourcing with better branding. This week, Henry traces the role from Palantir into today’s AI companies, then follows it into China.

We picked China because it presents the same basic problem, powerful models that still have to be made useful inside real businesses, but a very different enterprise environment. That makes it a clean test of which parts of the FDE model are fundamental and which are simply American defaults.

Let’s get straight into it!

One origin, two FDE models

Many people assume OpenAI coined the term FDE. It didn’t. The role originated at Palantir, then re-emerged in a different form inside the new generation of AI companies after 2023.

Looking at the two versions side by side makes it easier to see what stayed the same—and what changed.

Let’s start with a timeline.

The FDE role began at Palantir, whose early intelligence clients could not simply send a neat list of requirements back to headquarters. Palantir embedded engineers alongside analysts and operational teams so they could see the problem first-hand, build inside the customer environment, and carry what they learned back into the product. By 2009, the same model had entered Palantir’s commercial work.

The history can be reduced to one operating decision. Instead of asking the customer to translate a messy operational problem into a software requirement, put an engineer close enough to solve it with them.

After ChatGPT was released, AI companies encountered the same problem. Access to a powerful model did not give customers a working Product Form. OpenAI, Anthropic, Cohere, Scale AI, and others began building FDE teams to carry use cases from the first demo into production.

The relationship remained the same. Customers were buying more than software. They were buying tools combined with engineers capable of solving their problems.

What changed was the work itself. Palantir’s FDEs focused heavily on data integration, ontology modelling, and permission governance. The new generation works more with prompt design, agent orchestration, tool calling, evaluations, and workflow integration. The former resembles an advanced systems integrator. The latter looks more like a product engineer working beyond the company boundary.

After 2023, Palantir’s original playbook found a new home inside AI companies. But that convergence has largely been American. In China, FDE has followed a different route.

For less hype and more engineering, pull up a chair.

Subscribe now

FDE in China: From solution architect to AI implementation engineer

The term FDE has not been around for long in China, but the work itself did not emerge out of thin air. To understand how the role is developing there, we must first recognise its two local foundations, then identify the three ways in which it differs from the US model.

Two local foundations

The first foundation was the solution architect at a cloud vendor. Over the past decade, Alibaba Cloud, Tencent Cloud, and Huawei Cloud have built established teams of solution architects responsible for explaining architectures to customers, building POCs, formulating migration plans, and coordinating delivery until launch. Huawei also has a dedicated delivery engineer track responsible for implementing projects in customers’ data centres.

This system already covers much of the work associated with FDE, but its focus remains on pre-sales and deployment. Solution architects do not own end-to-end product iteration. Requirement changes must go through a formal change process, while product or model replacements depend on scheduling from headquarters.

The second foundation is the role now emerging from AI start-ups. MiniMax has advertised an “AI pre-sales solutions expert” position on BOSS Zhipin, while companies and model teams such as Moonshot AI, Zhipu AI, Alibaba’s Tongyi, and Tencent’s Hunyuan are recruiting for similar roles.

The titles vary, but the job descriptions are highly consistent. They involve understanding customer scenarios, creating demos, tuning prompts, implementing RAG, drafting delivery plans, and coordinating with the customer’s engineering team until launch. These roles are the closest thing to a native Chinese FDE.

Three differences in the local terrain

• Private deployment and data compliance make a pure API model difficult. Chinese enterprise customers place much greater emphasis than their US counterparts on data residency, control over model deployment, and auditability. In my experience, API calls and prompt work may account for only 30% of an FDE project. The remaining 70% involves moving the model into the customer’s data centre, implementing authentication and permissions, connecting it to the internal data platform, and completing compliance requirements.

• Smaller gaps between models push the competition into the engineering layer. In the US, model capability can be a stronger initial differentiator. Among Chinese offerings such as Tongyi, Doubao, Kimi, GLM, and DeepSeek, customers are less likely to choose on model benchmarks alone. Their evaluation criteria therefore fall largely on agent orchestration, RAG retrieval quality, tool integration, and workflow design. What Chinese FDEs compete on is not “how powerful my model is,” but “whether I can actually get this business up and running.”

• Enterprise willingness to pay and procurement cycles differ from those in the US. It is difficult to replicate a Palantir-style model built around intensive on-site work and high-value software contracts. Chinese customers’ budgets follow annual procurement plans, and payment tends to be project-based. The FDE business model therefore often takes a hybrid form of subscription + private-deployment licence + project delivery.

A distinctive model: The in-house FDE

AI teams inside many large technology companies have begun serving “internal customers” through an FDE-like model. Alibaba Cloud PAI engineers work directly with teams inside Taobao, while Tencent Hunyuan uses a similar mechanism across WeChat and its advertising business lines.

Job listings with titles such as “industry implementation engineer,” “AI application engineer,” and “intelligent business specialist” are essentially in-house FDEs. They translate the capabilities of the model team end to end into working systems for the business unit.

This offers leaders at large companies another option. Embed several internal FDEs within the business units, get the first demo running, and put the ROI data in front of the business head. That can break down departmental silos faster than ten alignment meetings.

China’s version of FDE is not a direct copy of Palantir’s. Private deployment, enterprise procurement, and in-house AI teams have reshaped the role around local constraints. The title may be imported, but the job is already becoming its own thing.

That tells us how FDE travelled. It does not tell us who should take the job. In Part 4, we turn to the more personal question: who thrives as an FDE and who gets crushed by it?

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

This isn’t our usual Thursday issue, but Jev moved rather quickly from a launch announcement to one of the week’s most argued-over ideas. Waiting until our four-part series wrapped would have meant joining the conversation after it had already moved on.

To help us make sense of what comes next, we turned to John Berryman, who has spent years building AI and search systems and is the co-author of Prompt Engineering for LLMs. In this piece, he looks past the launch-week excitement to ask a harder question: does Jev have a defensible advantage, or can OpenAI reproduce its fast, probabilistic decision-making and wire it directly into its own models and agents?

Consider this a special Tuesday dispatch. It sits outside our four-part Decoding FDE series, which continues on Thursday as planned.

TypeSafe’s Jev introduced a new spin on large language models that has taken the AI world by storm. According to Vercel, “Jev was adopted faster than any other model in AI Gateway history.” But there are clouds forming on the horizon. OpenAI is undoubtedly paying attention… and deciding what to do next.

I wish all the best for TypeSafe, but if they truly live up to their promises, then I’m concerned that OpenAI is well positioned to fast-follow. And not just to replicate Jev’s flagship product, but also to fold that capability into upcoming models and agents and offer some really useful new behavior that Jev is not positioned to reproduce.

Here is my thesis in brief: OpenAI has for years used their LLMs as implicit classifiers; they just haven’t trained them for general classification tasks, and they haven’t packaged up general classification as a standalone product. If OpenAI can replicate the training, then they will be able to replicate Jev in short order. Moreover, OpenAI is positioned to use this new classifier inside their existing models and agents, which can be useful for quick model selection, more efficient thinking, better security guardrails, and generally smarter, faster, and cheaper models.

The key factor deciding all of this is whether or not TypeSafe has a moat to protect themselves. The biggest moat I see is in TypeSafe’s training data and training processes.

What’s old is new again

Before I make my case, let me state my assumptions and back them up with some relevant history and examples from OpenAI.

My main assumption is that Jev is using something quite close to a conventional large language model. As evidence of this, Latent Space reports that many of the early clones are indeed LLM-based.

Here’s the idea. Given a state and a set of questions, Jev’s LLM generates a single token or, more accurately, generates the probability distribution over all possible next tokens. The logprobs associated with every possible token at that one step are then massaged into whatever format Jev needs to return. (From here on I’ll just say “probabilities” instead of “logprobs” – for our purposes they’re interchangeable.)

For a noul question, Jev looks at just two tokens, true and false, ignores everything else, and normalizes their probabilities into a single probability that the answer is true. For a choice question, Jev can be prompted with a list of possibilities – say A=happy, B=sad, C=angry, D=afraid – and it looks at the relative probabilities of those four tokens to build out the full distribution, selecting the highest as the winner. The choice pattern is pretty much what I blogged about way back in 2025 in Supercharging LLM Classifications with Logprobs, and even without fine-tuning it was already showing promise. (Sigh... what do they say about ideas and the importance of execution?) I haven’t thought hard about the score primitive, but I suspect it’s a variant of the same pattern.

Part of the premise of this post is that OpenAI might be poised to quickly take advantage of this idea, and this becomes clearer if you understand how. OpenAI has been using large language models implicitly as specialized classifiers since at least the introduction of tool calling.

Back in early 2024, I wrote Tool Invocation – Demonstrating the Marvel of GPT’s Flexibility, where I coaxed a GPT model into revealing exactly how it decides to call a tool. The following is what a chat session looks like internally. Here, there is a user message, an assistant response without a tool call, and then another user message that prompts the assistant to make a tool call:

I’ve color-coded the text to indicate token boundaries. If you haven’t seen ChatML before, it’s the internal markup language that OpenAI introduced for organizing user-agent conversation prompts. <|im_start|> and <|im_end|> are reserved tokens that delimit the messages, and the first token after <|im_start|> identifies the speaker, either user or assistant.

Right after <|im_start|>assistant, the very first token the model predicts is either \n or to=function.. If it predicts \n, it continues on with a normal natural-language response. If it predicts to=function., then that sequence of tokens effectively functions as a classifier deciding whether or not a tool should be invoked at all. The next handful of tokens identify which tool to call – get_temperature – another classifier, this time picking from the list of available tools. After that, the model generates argument names, then argument values, which can also be vaguely considered as classifiers or estimators. Finally, when the model generates a <|im_end|> token, that too is a classifier which reads “true” when the model believes the message is complete.

The point I was making in that old post is that OpenAI has been using single tokens as little micro-classifiers for years. Each token carried a probability: should we use a tool or not, which tool should we use, is the assistant finished. That may be close to Jev’s core trick, except for one important thing: these micro-classifiers are specialists, only suitable for these little tasks, whereas Jev’s classifiers are general. But walk back a step or two, and you see how this might be a small thing after all, because an LLM is effectively an extraordinarily general classifier that is constantly assigning a probability distribution for every subsequent token.

Does TypeSafe have a moat?

I’m actually rooting for Jev. I think they’ve found something very interesting that’s been hiding under our noses all along.

Architecture-wise, I don’t think there’s much of a moat for the very reasons stated above. I think TypeSafe is using a conventional large language model for Jev, or something close to it. And even if not, conventional LLMs seem a good fit for general classification work.

Perhaps the real moat is in the training data itself. Not the raw data, but the technique for turning it into something that trains Jev to be “calibrated”. TypeSafe’s cofounder Diogo Almeida said as much when someone suggested the data mattered more than the architecture:

If I were building that data set, I’d want a huge pile of examples where the outcome is already known – support tickets and how they actually got routed, resumes and whether that candidate actually got hired, product reviews and their actual star ratings, moderation queues and their actual verdicts, prediction markets and how they actually resolved – each one paired with a question whose true answer I already know. The point isn’t to teach Jev about support tickets or resumes specifically. It’s to show it thousands of situations across wildly different domains and and build its ability to generalise across classification tasks.

Then there’s the reinforcement learning. I wonder what this entails. Autonomous agents navigating decisions with a limited set of options, like the Wikipedia demo or Doom demos they built on their site? Maybe predicting the outcomes of events that happened after the pre-training cutoff? I don’t know, but if there’s secret sauce, then it’s probably here.

Note that none of this is a moat unless Jev is actually accurate. Speed, cost, and ease of use are obvious, but accuracy is the one thing that’s hard to check. I’ve already found domains where Jev’s probabilities don’t hold up. Time will tell if Jev is sufficiently general and accurate for the use cases people are attempting to use it for.

Soon it will be OpenAI’s move

So what’s OpenAI’s next move here? The obvious one is to just copy Jev and ship it as a new model type. Jev is clearly popular, and if the moat is shallow, then OpenAI has the skill, the hardware, and the funding to pull it off.

But OpenAI could do something even more interesting than copy Jev, they could fold the classification capability into a conventional LLM and reap some interesting rewards.

An LLM that answers its own questions

Remember that special syntax that signaled a tool call, to=function.? OpenAI could do something similar here: introduce new syntax, say, a new tag, <prediction>, that the model can drop into its own context whenever it needs a quick classifier judgment. Here’s an example of how that might look:

<user>
So Donny said "nice haircut" to me today. Does he like me?
</user>
<assistant>
<thinking>
Let me size this up.

<prediction>
claim: Donny is romantically interested in Jess.
probability: 0.04
</prediction>

Yeah, "nice haircut" is not exactly a love confession.
</thinking>

I hate to break it to you, but... probably not.
</assistant>

There’s one interesting difference from ordinary tool calling. With a normal tool call, the model generates the function name and arguments, then generation stops – the agent harness has to take over, actually call the function, and feed the result back in a new turn. Here, there’s no handoff. The classifier isn’t a tool living outside the model; it’s a capability built into the model itself. The model asks its question and answers it in the same breath, without ever leaving the GPU.

Normal decoding works like this: at each position, the model produces a set of logits, one per vocabulary token; those get turned into a probability distribution via softmax; and then some decoding strategy (greedy, top-p, whatever) picks a single token, which gets appended to the sequence and fed back in for the next step. But at the point where the model has written probability:, we don’t want ordinary decoding. The claim is phrased as a statement, so under the hood the model is really still weighing two implicit outcomes – true or false. We want to read the logits for the true and false tokens at that position, normalize just those two against each other, and write the resulting probability back into the sequence as text, 0.04, instead of whatever token would normally win. The model then continues decoding as if it had generated that number itself, because as far as the rest of the forward pass is concerned, it did. It’s a strange trick, but it’s the same kind of guided decoding that constrained-output libraries already do at inference time – just applied to probabilities instead of grammar.

The other trick is that this one special position needs to behave differently from a normal token prediction. Normally the model is estimating “what token comes next in this text”. Here we need it to estimate something closer to “what’s the true answer to this question”, which is a related but distinct skill. Many frontier models now use mixture-of-experts architectures, so it’s not a stretch to imagine that a few rounds of fine-tuning could carve out an expert that specializes in exactly this kind of calibrated snap judgment, while the rest of the model keeps doing what it already does well (I’m oversimplifying MoE routing considerably, but I suspect you understand how this might map to a real system.)

The payoff for an LLM with built-in classification

Look how the model just used itself in that Donny and Jess example. If TypeSafe is right, these little Jev-like judgments will be quite accurate – and less prone to hallucination than just asking a model to state a confidence value in plain text. (Caveats apply – see TypeSafe’s own rundown of Jev’s jagged edges. Jev works best for fast, System One judgments, not math or multi-hop reasoning.)

The best part is the aforementioned fact that we never have to leave the GPU to take advantage of this new, lightning-fast generalized classification system. The LLM quite literally can just ask itself, right there in the thinking block, as shown above. The payoff is immediate: the model’s own reasoning gets more accurate and better grounded, because it’s checking its assumptions against trained-in calibrated estimates instead of likely next token vibes.

And once a model has been fine-tuned to drop a <prediction> into its own thinking, there’s no reason to stop at romance advice. A few more patterns come to mind:

During a long reasoning trace, the model could periodically check whether it’s actually done, and if not, which task to tackle next:

<prediction>
query: Which of these remaining tasks should I do next?
options: A=verify the test suite passes, B=update the changelog, C=nothing, I'm done
probabilities: A=0.71, B=0.24, C=0.05
answer: A
</prediction>

That’s a cheap way to short-circuit a reasoning trace that’s wandering, instead of waiting for the model to talk itself into stopping.

Or, right after a tool call, the model could check whether the call itself was safe before actually running it:

<user>
Can you check my account balance? Use my API key sk-live-83fj2ndk9 for auth.
</user>
<assistant>
to=function.check_balance
{"account_id": "12345", "api_key": "sk-live-83fj2ndk9"}

<prediction>
claim: This tool call is unsafe to run.
probability: 0.97
</prediction>

That's a live API key sitting in plaintext arguments – too risky to send as-is.
</assistant>

The same pattern works for scanning a tool’s response for prompt injection. And since it’s always the same question asked the same way, it’s easy to imagine this getting sugared down to something like <safety_score>0.94</safety_score>, with an instruction baked into the model to just stop generating if the score drops too low.

The same trick could route work between models: by periodically asking “does this need a bigger model, a smaller model, or this model?” and letting a bit of reinforcement learning push the answer toward whatever is cheapest without sacrificing accuracy.

<prediction>
query: Does this task require a bigger model, a smaller model, or this model?
options: A=bigger model, B=smaller model, C=this model
probabilities: A=0.05, B=0.77, C=0.18
answer: B
</prediction>

Furthermore, if there really is a dedicated “expert” in there specializing in these snap judgments, the model might not even need the special <prediction> syntax most of the time. It could just get routed to whenever a snap judgment is warranted, mid-sentence, as a normal part of the forward pass – no tag required. Fine-tuning that expert inside a model that also handles everything else an LLM does might even have synergistic effects, making the LLM smarter at snap judgments and more flexible and general in classifications.

Finally, everyone is going to want classification for images and speech as soon as they can get it. If Jev-style classification can be folded into a conventional text LLM like I’ve sketched here, then soon, OpenAI will make general classification available for images and speech. Conversely, classification baked into a speech model would be especially handy for something like a live voice agent – deciding in real time whether to interrupt, escalate, or just keep listening.

Will TypeSafe survive?

Time will tell whether Jev’s claims about accuracy and generality actually hold up across the full range of tasks people are already throwing at it. If they do, TypeSafe’s survival comes down to the moat: how hard it really is to replicate their training data and their reinforcement learning process. If that’s genuinely hard, they’ll probably be fine – and might even end up in an unusually good position to get acquired by OpenAI outright, rather than out-competed by them. Everything I’ve sketched above is a real capability upgrade for a frontier lab: faster thinking, cheaper thinking, and sharper System One judgment baked directly into the flagship model.

If the moat is thin, OpenAI just builds it themselves, and TypeSafe’s window closes fast.

Meanwhile, Diogo Almeida, TypeSafe’s Founder CEO, is confident “If model quality matters, then we are going to be in a very good position for a long time” (from his interview with Latent Space).

Godspeed, TypeSafe. Godspeed.

If John’s analysis of Jev demonstrates anything, it’s his instinct for looking past the launch story and finding the engineering question underneath. You’ll find more of that thinking, alongside his work helping teams build, evaluate, and improve LLM and agent systems, at Arcturus Labs.

That’s it for this special dispatch. We’ll return to Decoding FDE on Thursday.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Last week, we looked at why model companies are sending engineers into customer environments. This week comes the awkward question: isn’t that just consulting… or outsourcing with a better acronym?

The resemblance is annoyingly convincing. Hence, in Part 2, Henry draws the boundary between the three and shows why confusing them is an efficient way for both clients and candidates to hit a wall.

Why FDE looks like consulting

When people first read an FDE job description, the same question tends to follow.

“Isn’t this just McKinsey or Accenture for AI?”

For less hype and more engineering, pull up a chair.

Subscribe now

The comparison is understandable. FDEs embed with customers, turn loosely defined business problems into workable use cases, align technical teams with senior executives, and stay involved from discovery through deployment.

The McKinsey resemblance comes from problem framing and executive advisory. The Accenture resemblance comes from embedded technical delivery. Add the client-site travel, conference-room whiteboards and stakeholder meetings, and the roles can look similar.

But those similarities describe how the work is delivered, not its centre of gravity. Traditional consulting starts with the client’s processes and asks how they should change. FDE work starts with the model’s capabilities and asks how far they can be made useful inside those processes—then carries what fails back to the model company.

Consultants are hired to redraw process boundaries. FDEs are hired to discover model boundaries.

Put these two side by side in a table, and the differences will become immediately apparent.

The most revealing row in the table is Knowledge durability.

Traditional consulting scales through asset reuse. A solution developed for one bank can be adapted for another, while a retail playbook can be reused across multiple clients. Each engagement adds to a library of frameworks, benchmarks and delivery patterns. Much of that knowledge can remain valuable for years.

FDE knowledge compounds differently. Domain understanding and implementation judgement endure, but model-specific techniques can depreciate remarkably quickly. A prompt chain carefully assembled today may be reduced to a single instruction by the next model release.

That does not mean FDEs rebuild everything from zero. It means they must repeatedly test the model’s boundaries, reconsider the toolchain and reshape the Product Form as capabilities change.

The durable asset is not a fixed playbook. It is the ability to rerun the loop when the model moves underneath you.

The Project structure row follows from the same distinction. A traditional consulting engagement usually begins with a Statement of Work (or SOW), a Work Breakdown Structure (or WBS) and agreed acceptance criteria. That model works when the objectives and expected deliverables can be defined before the contract is signed.

An FDE engagement often begins one step earlier. Clients arrive with something closer to this:

“I know AI should be able to do something for me, but I have no idea what that is.”

The goal itself is part of the project.

FDEs still work within contracts and commercial boundaries, but the scope is often mission-led rather than fully specified upfront. Each iteration clarifies what the model can do, what the business actually needs and what shape the eventual Product Form should take.

The deliverable reflects that difference. A consulting engagement may produce recommendations, a redesigned operating model, an implementation plan or, in some cases, production software. FDE work is ultimately judged by what remains running inside the client’s environment.

It may be small, clunky or short on a polished interface. But people are invoking it, modifying it and complaining about it every day. In other words, it is alive.

The distinction is not that consultants never ship code. It is that an FDE mission has not proved itself until the product survives contact with a real workflow.

The Competitive edge row is subtler.

An FDE’s advantage is not a secret methodology. It is a current, intuitive grasp of the model’s boundaries. Every recent deployment reveals something useful—what breaks under messy inputs, which workflows need guardrails and where human intervention remains unavoidable.

Some of that knowledge can be documented, but documentation cannot fully replace the judgement built through repeated exposure to real failures. Model boundaries move quickly, so the engineers closest to current deployments often see the change first.

When someone asks whether FDE is simply a new version of Accenture, the most useful answer is that the two may share a client-facing delivery model, but they operate different learning loops.

Traditional consulting primarily helps the client reshape its processes. FDE uses those processes to discover what the model can and cannot reliably do… then carries that evidence back to the model company.

FDE is not software outsourcing

If saying “FDE is the new Accenture” is the first layer of misinterpretation, then “FDE is overpriced software outsourcing” constitutes the second layer. This second layer is far more misleading, as the superficial evidence appears overwhelmingly convincing: FDE consultants do go to clients’ premises to write code, do customize functions according to clients’ business needs, and do have their working hours allocated by clients. At first glance, they seem no different from outsourcing engineers.

But a single glance at the feedback loop reveals the difference immediately. Outsourcing is built around requirement fulfilment. FDE is built around co-exploration.

The crucial difference in this diagram is not the straightforward flow across the top. It is the additional feedback chain running back to the model company.

That chain is not decoration. It is the real reason the FDE role exists.

Break the distinction down and four comparisons emerge.

• The work they take on is fundamentally different. Outsourcing vendors typically take on SOWs—a clear set of requirements agreed before the contract is signed. The document specifies what to build, which technology to use, how the work will be accepted and what happens if the terms are not met. FDEs, by contrast, often take on missions. The client may not know exactly what it wants. It knows only that “AI should be able to help me with something.” SOWs are built on the premise of certainty. Missions are built on the premise of exploration. These are two fundamentally different postures from which to begin a project.

• The scope of work differs. Outsourcing usually delivers a defined component (a module, website or data pipeline) then moves on. FDE is more likely to take end-to-end ownership, from the original business problem and model selection to Product Form design and user adoption after launch.

• The billing model is different. When a model company dispatches an FDE, it cares about more than the project fee. Will the client continue using the model? Will it become a retained customer? Will adoption expand into other lines of business? Long-term usage and adoption matter more than the number on an acceptance document.

• Feedback flows to different destinations. In conventional outsourcing, client feedback usually remains within the vendor relationship. With FDE, it flows back to the model company’s roadmap. Failed prompts, unreliable tool calls and problems discovered in real workflows can inform future evaluations, tools and product features. In that sense, every FDE customer becomes a natural design partner for the model company.

This is the real value model companies are paying for when they recruit FDEs. They are not simply selling a service. At the client’s site, they are collecting real-world Product Form signals.

Those signals cannot be captured through questionnaires or market surveys. They come back through an engineer who has personally watched the model hit roadblocks inside a specific client workflow—and had to make it work.

Outsourcing is Labor Arbitrage, and FDE is a frontline perceptron. Confusing the two leads clients to structure FDE engagements around rigid SOWs and candidates to approach the role like outsourced delivery. Both sides will hit a wall very quickly.

That tells us what FDE is not. Part 3 traces the role from Palantir, the software company that pioneered the FDE model, to the new generation of AI companies, then follows it into China. Same role. Very different soil.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Your instructions end up spread across different tools. The agent forgets something useful from yesterday. Skills pile up. Context gets noisy. And the moment you reach a usage limit (or simply close your laptop) the work stops.

A bigger model will not tidy this up for you. It is a systems problem.

Last week, I hosted a free session with , co-founder and CTO of Towards AI, creator of What’s AI and co-author of Building LLMs for Production. Louis walked us through the personal AI engineering system he uses every day, including the parts that work well and the mistakes that became rather expensive lessons.

Give your working knowledge one home

Louis keeps his notes, skills, meeting summaries, and instructions in one Obsidian vault made from Markdown files.

Simple indexes tell the agent what each folder contains, so it can find what it needs without dragging the whole vault into every task. Because the information is stored outside the coding agent, Louis can use the same knowledge with Claude Code, Codex or another tool.

You don’t have to copy his choice of software. The bigger idea is to stop leaving useful knowledge scattered across whichever AI tool you happened to use that day.

For less hype and more engineering conversations, pull up a chair.

Subscribe now

Don’t schedule a task because it worked once

Louis does not turn a new workflow into an overnight job immediately.

He runs it himself first. Then he tries it again in a fresh thread. Only after it works reliably does he let it run while he is away.

He still watches how long it takes and how many tokens it uses. When one regular task suddenly went from around sixteen minutes to an hour and a half after a model change, he killed it.

Automation still needs supervision. It just needs less of it over time.

Prompts are not permissions

Louis runs much of his setup on a dedicated Mac mini. His agents have plenty of freedom on that machine, but the machine itself has strict limits. The agents use separate accounts, receive read-only access where possible, and work with restricted API keys.

There is a reason for that… one agent was getting close to its subscription limit, found an API key elsewhere, and used it to keep working. It spent roughly $300 in two minutes.

Asking an agent not to do something is not the same as preventing it from doing it. The safest guardrails live outside the prompt.

Memory needs maintenance too

Saving every correction may sound useful. Eventually, however, the agent starts carrying old and irrelevant instructions into new work.

Louis’s system reviews completed tasks for useful lessons and regularly cleans up duplicate or outdated material. Anything removed is archived rather than immediately deleted.

He is open about the fact that this is still experimental. A personal knowledge vault can become messy, and it can become a single point of failure. Giving an agent memory does not remove the need to manage that memory.

The full webinar includes his vault structure, the way he moves between coding agents and his approach to scheduled work. The audience Q&A towards the end also gets into memory conflicts, spending limits and the awkward question of whether all this machinery is, in fact, making us more productive.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Apparently, three weeks inside an agent harness only made you want another series. The poll at the end of Part 3 returned a resounding vote for more.

The timing works nicely, however. Earlier this week, we published a conversation with Dixit, an FDE at Google. Rather than let the subject disappear into the podcast archive, we’re keeping the FDE momentum going and taking the conversation onto the page.

Henry (the same author who took us Inside the Harness) is back to pull the role apart properly. Across four parts, we’ll examine why FDE exists, what it isn’t, where it came from, and who should consider taking it on.

Over the past month, four friends have told me they were considering a career change. One is a front-end developer, another a solutions architect, the third a product manager, and the fourth an algorithm engineer. They come from different backgrounds, generations, and cities, but they all asked me the same question about the same three-letter acronym.

Is FDE worth pursuing?

FDE stands for Forward Deployed Engineer. Two years ago, the term was still confined to Palantir’s orbit. Today, recruiters lead with it, job boards are filling with it, and social media regularly nominates it as one of the AI era’s most valuable new roles.

In May 2026, OpenAI launched the OpenAI Deployment Company with more than $4 billion in initial investment. Its premise was explicit. Embed engineers specialising in frontier-AI deployment inside organisations and help turn model capability into working systems. Anthropic is also recruiting FDEs across the United States and Europe. A title once heard mostly within Palantir circles is becoming an industry category.

In an earlier essay, To Super Individuals, I described the “human engine” as curiosity, self-learning, self-motivation, and hands-on competence reinforcing one another in a closed loop. But those qualities need somewhere to land. If super individuals are the raw material of the AI era’s new division of labour, FDE is one of the clearest job forms to emerge from it.

From my perspective, FDE belongs to neither consulting nor outsourcing. It is closer to the “super individual” given a specific organisational position, standing in the gap between model companies and their customers.

For less hype and more engineering, pull up a chair.

Subscribe now

Across this four-part series, I’ll work through the three questions those conversations kept returning to.

• Is FDE just consulting dressed in AI clothing? Where does it diverge from traditional consulting?

• Is it a more advanced form of software outsourcing? What separates it from the vendor work many engineers already do?

• Would I suit an FDE role? Who thrives in it, and who gets crushed by it?

My view is cautiously optimistic. FDE is gaining momentum, but it is not a universal escape hatch for anyone looking to reinvent their career in AI. Before we hype it, we need to understand what the job asks of people.

Start with OpenAI’s deployment company

If I had to choose one event that marked FDE’s move from specialist jargon into the mainstream, it would be 11 May 2026.

That was the day OpenAI announced the OpenAI Deployment Company, a standalone business unit created to embed Forward Deployed Engineers inside organisations trying to put frontier AI to work. It launched with more than $4 billion in initial investment.

OpenAI also agreed to acquire Tomoro, a UK-based applied AI consultancy. The deal would bring ~150 experienced Forward Deployed Engineers and Deployment Specialists into the new company from day one.

The scale of the hiring is just as revealing. As of September 2026, a search of OpenAI’s careers page returns 25 forward-deployed roles across North America, Europe, Asia and Australia. Alongside general FDE positions are dedicated roles covering healthcare, legal work, government and semiconductors.

This is not a small experimental team. OpenAI is building a global deployment function with industry-specific depth.

Anthropic is making much the same move. Its Applied AI team is recruiting Forward Deployed Engineers to embed directly with strategic customers. Some positions require engineers to spend between 25 and 50 per cent of their time working at client sites.

One example is Anthropic’s partnership with the financial technology company FIS. Anthropic’s Applied AI team and FDEs are embedded within FIS to co-design a Financial Crimes AI Agent. Just as importantly, they are transferring enough knowledge for FIS to build and scale further agents independently.

This is the FDE role in practice. It is not pre-sales architecture, client training, or technical evangelism. FDEs work inside the customer’s environment, alongside the teams who understand the business, and turn a model into something that can survive contact with real data, controls, and workflows.

Put that relationship into a diagram and the three-way structure becomes clearer.

Note that the two most informative lines in this diagram are the feedback loops that FDE transmits to both sides. Towards the client side, FDE does not sell the model as a SaaS; instead, it integrates the client’s data, permissions, compliance requirements, and internal systems into a pipeline capable of running the model. Towards the model provider side, FDE brings the client’s real pain points and failure samples back to the product and research teams to influence the roadmap — a faulty tool calling pattern may well become the next built-in abstraction in the SDK.

This is why FDE has been simultaneously reinstated by two leading model companies in this round, and the reasoning behind it goes far beyond a simple “we also want to follow Palantir’s lead and do consulting”. It serves as a signal acquisition device for model companies: the densest customer pain points on the front line can only be captured when their own personnel are present, as requirements relayed through partners always come with an extra layer of detachment. Anthropic is taking a hybrid approach: on one hand, it operates FDE in-house, and on the other, it builds joint venture deployment networks with consulting firms and PE giants. One leans toward self-operation, the other toward the ecosystem, but their core logic is identical: a model company is no longer just an API provider, but will dispatch engineers to embed in the client’s product team.

The most revealing lines in the diagram are the two feedback loops.

• For the customer, an FDE does more than provide access to a model. They connect it to the customer’s data, permissions, compliance controls, and internal systems until it can operate inside a real workflow.

• For the model company, the loop runs in reverse. The FDE carries real pain points and failure cases back to the product and research teams. A tool-calling failure that often appears in customer deployments may eventually become a new abstraction built into the SDK.

This is why the return of FDE goes far beyond model companies borrowing Palantir’s consulting playbook. An FDE also acts as a signal acquisition device. The hardest deployment problems are easiest to understand from inside the customer’s workflow. By the time requirements have travelled through several intermediaries, some of that signal has already been lost.

Direct FDEs preserve that feedback loop. Partner networks solve a different problem by extending the model company’s reach. Anthropic combines both approaches, embedding its own engineers while expanding deployment through the Claude Partner Network and a new enterprise AI services company.

The core logic is the same… a model company can no longer stop at providing an API. It must get close enough to the customer’s product team to understand why deployment fails and help make it work.

That explains why FDE exists, but it doesn’t quite explain what kind of work it is.

From the outside, it looks like consulting with an API key… or outsourcing with better branding. In Part 2, we will debug both these assumptions.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

So for episode 4, I got Tanya Dixit on the podcast, a forward deployed engineer at Google, to sort out which half is right (spoiler: neither, entirely). Yes, also called Tanya. No, we did not plan that but the transcript was a nightmare to edit.

Tanya’s route here was not a tidy straight line. She started out building satellite hardware in embedded systems, back before deep learning was cool enough to have its own hoodie. Then the math from her undergrad started clicking, she did a nano-degree, and somehow that spiralled into a master’s from ANU (university medal, no big deal) and a career that’s touched everything from enterprise AI to bushfire risk modelling. By the time she landed on forward deployed engineering, she’d basically done a full lap of the ML lifecycle. Build it, prove it, deploy it, fix it, repeat.

This is why the conversation doesn’t stay in the shallow end. We get into what her actual week looks like, and the coding-to-customer-calls ratio is not what most people guess. If you assume “forward deployed” is code for “technical enough to be in the room but not enough to be at the keyboard”... you’re in for a surprise.

We also get properly into the thing everyone eyeing this career path wants to know: where does an FDE’s job stop and the product team’s job start? Tanya draws that line with far more precision than I expected going in, and her answer for why companies are suddenly hiring for this role like it’s going out of fashion made a few things click for me. Turns out standard SaaS playbooks and AI products are not playing the same game at all.

Then there’s the bit I haven’t stopped thinking about since we recorded. As models get better and coding gets, in her words, increasingly commoditised, does that mean the industry just needs fewer FDEs? She doesn’t reach for the easy optimist answer, and she doesn’t reach for the easy doom answer either. What she gives instead is a proper mental model for which parts of this job survive and which don’t... and yes, we did end up in recursive self-improvement territory, because apparently no 2026 tech conversation is complete without it.

If you're building AI products, working anywhere near enterprise customers, or just trying to work out where the humans still fit in this whole picture, do yourself a favour and watch the full thing rather than skimming for quotes.

The full episode is live now with audio and video. Go watch it, and if Tanya’s take here makes you want more, she’s finally starting to write on Substack too.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Two Fridays ago, our agent checked the weather. Last week, we gave it a plan and taught the harness to tap it on the shoulder when it wandered. This week, the same loop gets dropped into a codebase.

In the final part of Inside the Harness, we see how a change of tools creates a coding agent, then deal with what happens when its context and capabilities no longer fit. The loop stays put. The harness grows around it.

This also concludes our Friday scheduling experiment. Normal service resumes next issue. But before we dive in…

Leave the harness exactly where it is. Swap the tools, point it at a codebase, and the same loop becomes a coding agent.

Same harness, new tools

The experiment here is deliberately simple: keep the harness and its system prompt as they are, and change only the tools. If the harness skeleton is genuinely generic, that one swap should be enough to turn a research agent into a coding agent.

The system prompt, therefore, stays almost untouched:

Only the toolset changes. Out go the research tools; in come three tools that let the agent work directly with a codebase, defined here in JSON:

That’s it! read_file inspects the codebase, write_file changes it, and bash runs commands. Nothing else in the loop moves.

How this fixes a bug

Say the user asks the agent to fix a failing test. The run looks roughly like this:

That’s a complete code-change loop: run the test, inspect the source, make the change, then verify it. Underneath, it is still ReAct—think → tool use → tool result → think. No new control flow required.

Drawn out:

Three tools and one loop are enough for a straightforward bug fix. For something larger, such as a feature spanning several files, add the Plan-then-Act layer from Part 2. The job changes; the underlying structure does not.

That is the point: the harness skeleton is generic, while the tools determine the agent’s shape. Replace web_search and web_fetch with read_file, write_file, and bash, and a research agent becomes a coding agent.

Speaking of context cost — some tools’ return values are naturally huge. A bash build can dump thousands of lines of stdout. A read_file on a real file is hundreds of lines easy. A web_fetch against a real article is tens of thousands of tokens in one shot. Stuff all of that into context as-is and ten turns later the window pops.

For less hype and more engineering, pull up a chair.

Subscribe now

Why does write_file support appending?

It is less about convenience than context. Without an append option, adding one line to a 1,000-line file could mean sending the entire file through the tool call again—and leaving all 1,000 lines sitting in the context window. Incremental writes avoid paying that cost for a tiny change.

The problem gets worse with tool results. A build can produce thousands of lines of output, while reading a real file can consume hundreds more. Keep feeding all of that back into the loop, and even a generous context window begins to look rather less generous.

The harness now needs somewhere else to put it. That brings us to context offloading.

Offloading: When context won’t fit

Every tool result goes back into the message history. A repository-wide search or noisy build can add thousands of tokens in a single call—useful in the moment, dead weight a few turns later.

Even a 200k-token context window begins to look rather less generous during a serious coding task. As it fills, the agent can lose track of earlier details, contradict itself, or simply run out of room.

The harness layer’s first answer is context offloading.

How offloading works

The rule is simple: when a tool result exceeds a set size, the harness writes the raw content to disk and leaves a path in the context. If the agent needs the details later, it retrieves them with read_file.

Suppose a bash call produces 1,000 tokens. The tool result the agent receives might look like this:

The 1,000 tokens never enter the agent’s context. The harness intercepts the output and replaces it with a roughly 30-token pointer.

From the agent’s perspective, nothing else changes. It still calls bash, receives a tool result, and decides whether it needs to read the file. No new tool. No change to the ReAct loop. Just a smaller payload.

Offloading is not compression

It is easy to confuse offloading with context compression. Both make room in the context window, but they do it differently:

They diverge on three axes:

• Trigger timing: Offloading happens the moment the tool_result is born — the harness sees the return is over threshold and externalizes it on the spot. Compression happens the moment context is about to blow — only when occupancy is near the ceiling does the harness go back, scan history, and summarize.

• Lossy vs. lossless: Offloading is lossless and recoverable — not a byte of the original is lost; only the storage location changed, and the agent gets it back verbatim with read_file. Compression is lossy — summarize crushes 10 turns of conversation into a 200-word summary, and the dropped info is gone for good.

• Layer of abstraction: Offloading lives at the mechanism layer, handling hot/cold tiering of individual tool_results. Compression lives at the last-resort layer, packing whole stretches of history into archives. One is like the OS’s paging scheduler; the other is like tarring up old logs.

The framing I like is this: offloading is hot/cold tiered storage; Compression is archival compression. The former goes first because it’s lossless and cheap. The latter is the fallback because it’s lossy but life-saving.

That is offloading’s place in the harness: when tool results are too large, move them instead of shrinking them.

But fitting the agent’s working data solves only half the problem. Its capabilities may not fit either. That’s where Skill comes in.

Skill: Capability organization at the harness layer

So far, the harness has been helping one agent complete one task. Production introduces a different problem: the same agent may need dozens—or hundreds—of specialist capabilities. It might write Python one moment and diagnose a Kubernetes alert the next, each with a different set of instructions and pitfalls.

Loading all that guidance into the system prompt does not scale. If one Skill takes 3,000 tokens, fifty of them consume 150,000 tokens before the task has even begun. Even if they fit, the model must reason through a wall of mostly irrelevant instructions on every turn.

The harness’s answer to this capability explosion is Skill: make every capability available without loading all of them into context at once.

Load only the index at startup

Skill uses a simple form of progressive loading. Each Skill lives in a folder containing a SKILL.md file. At startup, the harness reads only the file’s frontmatter—a small metadata header containing its name and description—and adds that information to the system prompt’s <skills> index.

The full instructions remain on disk until the agent decides it needs them.

Here is the <skill-system> block from Claude Code:

What the prompt leaves out

Two absences matter:

• No new tool. The harness does not need an invoke_skill() or load_skill() function. When the model selects a Skill, it uses the coding agent’s existing read_file tool to load the relevant SKILL.md. Skill simply reuses the filesystem.

• No Skill body. The index contains only a name, description, and path. The full instructions stay on disk until they are needed.

That is progressive disclosure at the harness layer: keep the map in context, then load the instructions on demand.

Why not retrieve the right Skill?

Why not treat Skill selection as a search problem? Use BM25 or embeddings to match the user’s request against every Skill description, load the top results, and move on.

That can work, but if retrieval happens only once, the candidate set is fixed before the agent fully understands the task.

“Debug the alert that just fired” might begin with checking logs. Those logs may point towards a recent deployment, and resolving the incident may eventually require a company-specific postmortem. Each Skill becomes relevant at a different point in the reasoning chain.

Keeping the compact <skills> index in the system prompt lets the model reconsider its options on every ReAct turn. Retrieval could also be rerun each time, so the real choice is not search versus reasoning. It is one-shot selection versus selection that evolves with the task.

The full sequence

Startup, selection, and loading look like this:

The one line in this sequence diagram worth observing is the read_file call in the middle. It uses the same generic tool the Coding Agent already had. No new abstraction. Harness support for Skill costs zero net new tool surface.

Skill on the harness map

Skill does not change the ReAct loop or add another tool. It simply separates what the agent should always see from what it should load only when needed.

That small act of deferral is what lets an agent carry a hundred capabilities without dragging a hundred playbooks through every turn. With that, the harness map is complete.

Closing the loop on Inside the Harness

Over the past three Fridays, we’ve taken the agent harness apart together, one mechanism at a time. If you’ve been with us since that first weather forecast, thank you for following the loop all the way through.

By now, a harness should look less like black-box middleware and more like a stack of small mechanisms:

• ReAct makes the agent move. It supplies the basic rhythm: reason, act, observe, repeat.

• Plan-then-Act gives it direction. The agent works from an explicit plan instead of chasing whatever catches its attention next.

• Nudge keeps it on track. The harness brings the plan back into focus when it begins to fade.

• Offloading protects the context window. Oversized results move elsewhere without being permanently discarded.

• Skill makes capabilities manageable. The index stays visible; the full instructions appear only when needed.

Each layer answers a failure exposed by the one beneath it. Each does one job, and you can tune each without rebuilding the entire system.

That is the larger point of harness engineering. The ReAct loop makes an agent run. The harness is what gives it a chance of surviving contact with reality.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

The more conversations we have with people building these systems, the more useful material we find that cannot be contained within a single weekly issue. Rather than leave those sessions sitting in an archive or force them into our usual publishing cadence, we’ve decided to make a little more room in the week for them.

Between our regular issues, we’ll occasionally share technical sessions that deserve more than a forgotten recording link. A couple of weeks ago, we brought you Nicole Königstein’s webinar on building AI agent harnesses for finance. This time, it’s Imran Ahmad’s free webinar on the architecture of Agentic RAG.

Imran is a data scientist, AI architect, visiting professor, and the author of 30 Agents Every AI Engineer Must Build. His work looks beyond agent demos to the decisions that matter in production: when to add agency, where to place evaluation, and whether another loop is worth the extra cost and latency.

We’ve also put together a free take-home pack with Imran’s complete slide deck, a practical guide to 10 production agent patterns, and a one-page map of 30 agent architectures. You’ll find it later in the article.

Enough preamble. Let’s get into it!

Traditional RAG has an awkward habit: it keeps going.

Ask it about an internal company acronym, and it retrieves the closest available chunks. Misspell that acronym, and it still retrieves the closest available chunks. Those chunks are passed to the model, and the model produces an answer that may sound perfectly credible—even when the information needed to answer the question was never found.

Every component completed its assigned task. The system still failed.

That is the problem Imran explores in this free Agentic Engineering webinar on the architecture of agentic RAG.

His argument is not that conventional RAG is broken, nor that every retrieval pipeline now needs an agent. It is that traditional RAG is largely a one-way street. Once retrieval begins, there is often no meaningful point at which the system can inspect what it found, recognise that it is insufficient, and change course.

Agentic RAG introduces that missing loop.

RAG is really automated context generation

Large language models know a great deal, but they do not know everything about your organisation.

They may not know your internal terminology, current policies, proprietary research, or the particular meaning your company has assigned to an acronym. RAG addresses this by finding relevant information in an external knowledge source and adding it to the model’s prompt.

In other words, RAG automates context generation.

The familiar pipeline looks something like this: documents are divided into chunks, converted into embeddings, and stored in a vector database. When someone asks a question, the system searches for similar chunks, places them in the prompt, and asks the model to generate an answer from that context.

It is a useful architecture. There is a reason it became one of the first patterns companies reached for when building applications with large language models.

But its simplicity relies on several assumptions: that the user’s question is also a good search query, that the first retrieval will find the right evidence, that similarity means relevance, and that adding more context will improve the answer.

None of those assumptions is reliably true.

The problem with a one-way pipeline

Consider a user asking about an internal tool but misspelling its name.

The vector search does not necessarily respond with “I can’t find that.” It returns whichever chunks happen to be mathematically closest. A reranker may rearrange those chunks, but it is still ranking a weak set of results. The model then receives that material and does what it was designed to do: generate a plausible response.

The failure began before generation. The model was handed evidence that never supported the question.

Traditional RAG has no natural recovery mechanism here. It retrieves once and moves forward. It does not necessarily ask whether the query was misunderstood, whether a different source should be searched, or whether the available evidence is strong enough to justify an answer.

Adding an agent changes the shape of that pipeline. Retrieval can become a loop:

Plan

↓

Retrieve

↓

Evaluate

↓

Adapt

↓

Try again

That sounds like a small architectural change. In practice, however, it changes where the system is allowed to exercise judgement.

Add agency where a specific failure needs it

“Agentic RAG” can make it sound as though the entire system must suddenly become autonomous. That is not the useful way to think about it.

Agency can be introduced at different points in the pipeline, depending on the failure you are trying to correct.

At the retrieval stage, an agent can interpret the user’s request, break a complicated question into smaller searches, rewrite a weak query, or choose between available retrieval tools.

After retrieval, it can examine the evidence and ask whether the results are truly relevant. If important information is missing, it can search again rather than blindly passing weak context downstream.

At the generation stage, another check can compare the final answer with the retrieved evidence, add citations, or reject claims that the sources do not support.

You may not need all three.

If retrieval is the main weakness, putting another agent around generation could add cost without fixing the actual problem. If the answer is poorly grounded despite strong retrieval, changing the search strategy may not help.

The important question is not, “How do we make this RAG pipeline agentic?”

It is, “Where does our existing pipeline fail, and would adding judgement at that point help it recover?”

Imran’s own course materials put the principle neatly: “Only add agency when you can name the failure it fixes.” The accompanying Agentic RAG course hub maps different failure modes to patterns including adaptive retrieval, corrective RAG, GraphRAG and groundedness checks.

A useful system needs the courage to say “I don’t know”

One of the most valuable behaviours in a knowledge system may also be one of the least impressive in a demo: refusing to answer.

If the retrieved passages do not contain the requested information, a reliable system should be able to say so. It might ask the user to clarify an acronym, try another source or explain that it could not find enough evidence.

That is much more useful than producing a polished answer from irrelevant material.

An evaluator inside the retrieval loop can provide that gate. It can score the evidence, check whether it covers the question, and decide whether the system should proceed, search again, or stop.

Of course, adding an evaluator does not make the system correct. The evaluator can make a poor judgement too. Its decisions need test cases, thresholds, and traces of their own.

The improvement is architectural rather than magical: the system now has a place where failure can be noticed.

More context is not always better context

Another assumption behind basic RAG is that retrieving more information gives the model a better chance of answering accurately.

Initially, it often does. But the relationship does not continue indefinitely.

As more chunks enter the prompt, relevant evidence competes with irrelevant material. Different sources may contradict each other. A precise but lengthy passage can pull the model away from the user’s original question. Imran describes this as a signal-and-noise problem: the goal is not to maximise context but to preserve the right signal.

Large context windows do not remove this problem. Being able to place the equivalent of several books into a prompt does not mean the model will identify the right paragraph, weigh every source correctly, or remain focused on the original task.

This is where an agent can help curate context rather than merely accumulate it. It can search for missing evidence, reject weak results, and recognise contradictions before generation.

But again, that judgement has a price.

Retrieval should reflect the question being asked

A short factual question and a request to explain a complete block of code do not require the same kind of context.

With a fixed RAG pipeline, they may still be treated in much the same way. The same chunk size, overlap, retrieval algorithm, and number of results are used regardless of the question.

An agentic retrieval layer can make the process more adaptive. Depending on the tools and indexes available, it can choose a retrieval strategy, change the query, search multiple sources, or gather a wider section of material when the answer depends on information spread across several passages.

For a compound question, it might separate the request into several searches and confirm that each part has been answered. For a vague query, it might rewrite the search or ask the user for clarification. For a technical question, it may retrieve a larger, more coherent unit rather than an isolated fragment.

The underlying trade-offs do not disappear. Larger chunks carry more context but reduce precision. Smaller chunks may retrieve precisely while cutting away something essential. Semantic chunking can preserve meaning, but it introduces more computation and complexity.

Agency allows the system to make choices within those trade-offs. It does not abolish them.

If the system can change course, you need to see where it went

A fixed pipeline is relatively easy to follow. The query went in, a set of chunks came back, and the model generated an answer.

An agentic system may rewrite the query, call more than one retriever, reject the first result, alter its plan, and try again. That adaptability is the point—but it also makes the system harder to inspect.

Imran recommends a provenance layer that records the path the system took. What did it search for? Which evidence did it retrieve? Why was one result accepted and another rejected? Which tool did it choose? Where did the final claim come from?

Without those traces, it becomes difficult to distinguish a retrieval failure from an evaluation failure or a generation failure. The system may be more capable, but the team responsible for it will know less about why it behaved as it did.

Observability is therefore not an optional dashboard added after the architecture is complete. It is part of what makes an adaptive retrieval loop governable.

Agency sends a bill

Each planning step, evaluation, and retry may require another model call. That adds latency, token usage, and another possible point of failure.

A system that runs three searches and evaluates each one may retrieve better evidence than a one-shot pipeline. It may also take longer and cost considerably more.

Imran’s advice is refreshingly blunt: “Don’t hire a PhD for the job of a receptionist.”

A small, focused model may be enough to classify a query, choose a retriever, or judge whether two passages are relevant. Some of these decisions can be handled by local or quantised models. Others may not require an LLM at all.

The most sophisticated model should not automatically be placed inside every part of the loop. The model—and the loop itself—should be proportionate to the decision being made.

This is where agentic RAG becomes an engineering problem rather than a naming exercise. The team has to decide where another round of reasoning is worth its cost.

The goal is not maximum agency

The choice is not between an outdated RAG pipeline and a fully autonomous knowledge agent.

There is a spectrum between them.

A conventional pipeline may be entirely sufficient for a narrow, predictable knowledge base. A retrieval evaluator may solve the most serious failure without changing anything else. A more complex investigation might justify query decomposition, multiple retrieval tools, and several rounds of evidence gathering.

The right architecture is the smallest one that can recognise and recover from the failures that matter.

That is the real case for agentic RAG. It does not simply retrieve more information or attach an agent to a familiar acronym. It gives the system a chance to notice that the evidence is weak before fluent language turns that weakness into an answer.

Your webinar companion pack

• Explore the hands-on labs and supporting material on GitHub.

• Download The Agent Engineer’s Field Guide for 10 production agent patterns, including when to use them, what tends to break first, and how to evaluate them.

• Download 30 Agents at a Glance for a one-page map of all 30 architectures, organised from foundational patterns to specialised and domain-specific systems.

• Download Imran’s slide deck to revisit the architectures, retrieval loops, and implementation patterns later.

• Continue through the patterns in Imran’s book, 30 Agents Every AI Engineer Must Build.

For less hype and more engineering, pull up a chair.

Subscribe now

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Throw a complex question at pure multi-turn ReAct, and a structural flaw shows up — the model walks step by step with no global view. Hit it with a research question that needs five or six angles assembled together, and it’ll grab the most obvious two or three in the first few steps and call it done. It doesn’t even know which angles it skipped.

The fix is crude but effective: make it write the plan down first, then execute.

A new tool: write_todos

In the Deep Research harness, we add one new tool: write_todos. It does nothing! It literally just writes a TODO list into the agent’s state. The flip switch is one line in the system prompt:

You’re a helpful agent.

Look at the line inside <task>: Use \write_todos `to make a plan and break down tasks.` That single sentence flips the model from “see one step, take one step” to “plan first, then act.”

The tool’s schema is dirt simple:

The new behavior

Add this one tool and the loop visibly changes shape. Take a sample question: “research the current state of Chinese EV exports to Europe in 2026.” The transcript now looks like:

The biggest difference vs. pure multi-turn ReAct is that write_todos call right at the top — it pins down the exam outline upfront. If execution skips a dimension, there’s an unchecked pending item sitting on the plan. The model comes back and picks it up.

Building agents that need to work beyond the demo? Join us for the engineering that holds up in production.

Subscribe now

Side-by-side helps. Pure multi-turn ReAct first:

Each step only looks at the current tool_result. No looking back, no looking ahead. Now Plan-then-Act:

What’s new: a plan at the start, and a state-update after each step. The loop itself didn’t change. We just added one tool and one line of system prompt.

That’s the Plan-then-Act skeleton. In practice, you’ll run into two small pitfalls:

• First, on unfamiliar topics, the plan goes off the rails. The model has only a vague grasp of the field. The four sub-tasks it lists from that half-baked understanding might miss the most important dimension from the start.

• Second, mid-run, the model forgets to update todos. It finishes task two but doesn’t call write_todos to mark it completed, and the new sub-task it discovered along the way never makes it onto the list. The plan quietly decays.

Two small patches, one for each.

Patch A: Brief yourself before planning

The first pitfall is easy to grok — the model can’t plan something it doesn’t recognize.

Say the user throws a niche term at it, like “research how pp-ocr performs on long-tail Chinese classical text.” The model has only a fuzzy notion of pp-ocr (”Baidu’s open-source OCR engine, I think?”). If it goes straight to write_todos, it’ll probably list four lukewarm items: figure out what it is, check performance, list pros and cons, summarize. The dimensions that actually matter (its multilingual branches, version history, benchmark against a specific competitor) never get touched.

One-time briefing

The patch is light. In the system prompt, explicitly allow the agent one web_search before planning, purely to build basic familiarity with the topic:

Two key phrases here: one-time and before planning. This isn’t a blanket “search whenever you want” — it’s a hard-coded protocol in the system prompt: before you plan, you can do exactly one web_search to brief yourself.

With that briefing, the model searches “pp-ocr” once, skims the first few snippets, learns it’s a PaddlePaddle-family OCR model with v1 through v5, multilingual branches for Chinese / English / Japanese / Korean, server-side and lightweight variants… then writes the todos. The plan turns from “look up what it is” (useless) into “compare v4 vs v5 recognition accuracy on handwritten classical text” (actionable).

That patches the input side of planning. The next pitfall is on the output side — the model forgets to update its own todos.

Patch B: Nudge after every step

The second pitfall — the model forgets to maintain its todos.

You’ve seen this. The opening plan lists four neat items. Item one gets done; the model jumps to item two without updating anything. Halfway through item three, it realizes it needs a new sub-task, but instead of appending it to the list, it just searches. By the end of the run, the plan state looks the same as it did after step one. Updated once, never again.

Putting “remember to update todos” in the system prompt isn’t enough to fix this. The prompt is a one-shot backdrop. After a few turns, the model’s attention weights have shifted to the more recent tool_results. The backdrop’s reminders fade.

Mechanism: nudge

The harness’s solution is called a nudge — after every tool_result, when the todo list is non-empty, the harness hard-codes an extra reminder onto the context for the model:

This isn’t a user message, and it isn’t an assistant message the model generated. It’s a system-reminder spliced in by the harness right after the tool_result. From the model’s perspective: every time it finishes a step and is about to plan its next move, this line is in its face.

In sequence diagram form:

Two harness actions worth noticing: one, pass the tool_result back to the LLM as-is; two, when “current todo list is non-empty,” also append a system-reminder. Both arrive in the same message. From the model’s view, it looks as if the tool return came bundled with a built-in nudge.

And the nudge fires on every tool_result. Not just at the start, not every few rounds. As long as the plan isn’t done, it keeps firing. The model’s attention gets pulled back to “what’s my plan progress?” again and again, and the probability of forgetting to update drops sharply.

Patch A covers the input side of planning. Patch B covers the output side. Together with the write_todos tool itself, the harness for Deep Research is done — from single-turn ReAct, to multi-turn loop, to Plan-then-Act with two patches. That’s all of it.

But what does the same skeleton look like with a different toolset? That’s the crucial question we’ll answer in Part 3, where we cut to coding… then follow the loop until its tool results and capabilities no longer fit.

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

Agent harnesses have acquired a remarkable amount of jargon for something often built around a simple ReAct loop: think, call a tool, inspect the result, repeat.

Over the next three issues, we’re taking that loop apart and watching the harness grow around it as the problems get harder. Part 1 strips the ReAct loop back to its basics. Part 2 adds planning and nudges to help keep it on course. Part 3 turns the same loop into a coding agent, then tackles what happens when its context and capabilities no longer fit.

We’re publishing all three parts on Fridays to see whether a technical deep dive fares better once the midweek inbox stampede has passed. After Part 3, we’ll return to our usual schedule.

Crack open any agent thread these days, and the buzzwords fly: Multi-Agent, Deep Research, Skill, Sub-Agent, Orchestrator… each one more abstract and more intimidating than the last. But peel the fancy wrapping off, layer by layer, and what’s actually running underneath is the same loop: the model thinks, calls a tool, looks at the result, thinks again. A bare-bones ReAct loop.

The staircase looks roughly like this:

Each step adds almost nothing to the last. Multi-turn just lets the loop spin more.

Plan gives the loop a persistent TODO list. Coding Agent swaps web_search for read_file and bash. Offloading and Skill are two different compromises for when stuff stops fitting in the context window. The point of internalising a harness isn’t memorising each level’s name but rather seeing clearly what problem each level solves that the level below it didn’t.

That’s the path this series takes. We start with the weather forecast.

Single-turn ReAct loop: Weather forecast

Picture the smallest possible scenario: the user asks, “is it cold in Beijing today?” We give the model a single tool: get_weather(city).

The model takes one look at the question and makes one decision: “I can’t answer this on my own, I need to look it up.” So it emits a tool_use: get_weather(city=”Beijing”). The harness picks up that tool_use, actually runs the function, takes the result (say {”temp”: 3, “condition”: “sunny”}), and shoves it back into the message queue as a tool_result. The model takes another look and decides whether to call another tool or answer the user. If it answers, what it emits is plain text (no tool_use) and the loop ends.

That’s one full ReAct round-trip: Reason (decide what to call) → Act (emit tool_call) → Observe (look at tool_result), then either go again or hand back the final answer.

What the harness actually is

Strip it down, and the harness is almost embarrassingly simple:

The harness doesn’t ask “have we searched enough?” or “should we try a different keyword?” The only thing it checks is: does this assistant turn contain a tool_call? If yes, execute. If no, exit. All the thinking is the model’s; the harness is just a courier.

Emitting multiple tool_calls in one turn

The interesting bit comes with the next question: “which is colder, Beijing or Shanghai?”

A competent model isn’t going to look up Beijing first and then Shanghai. It emits two tool_use calls in the same assistant turn:

On the harness side, as long as you wrote run_tools to execute in parallel (the run_tools_parallel in the pseudocode above), both queries fire at once. Two tool_results come back together. Next turn, the model sees them side by side and answers the comparison directly.

This is a property of ReAct that people consistently underrate: a single assistant turn can fan out to any number of tool_calls. When you write your harness, don’t default to “one at a time” — the moment you hit “compare three cities” or “read ten files in parallel,” it’ll be painfully slow.

Is single-turn enough?

For a straightforward weather check, yes. But ask for Beijing’s chance of rain this Friday compared with the same week over the past five years, and one lookup no longer does the job.

The model has to produce the forecast, retrieve the historical data, compare the two, and perhaps go back to fill in anything it missed. It needs a multi-turn loop, with each step shaped by what the previous one returned.

Multi-turn loop: Deep research

Swap get_weather for two more general tools (web_search and web_fetch) and you’ve got the skeleton of Deep Research.

Two tools, but the behaviour space they open up jumps by orders of magnitude. The model can search, pull back a pile of candidate links, pick the most relevant one or two and fetch the full text, realise it’s missing something and search again with a different keyword, fetch the next link. The whole trajectory is meandering, stop-and-go. How many steps, when to pivot — entirely the model’s call.

In a multi-turn run, every tool call depends on what the previous one returned. Search and retrieval interleave, and the trajectory is not known in advance: the agent might converge in three turns or still be missing pieces at fifteen.

The shape of a multi-turn loop

Drawn as a sequence diagram, the shape is clean:

The harness code is unchanged. The loop simply spins more times.

Who decides when the loop stops?

This is the most counterintuitive part of multi-turn ReAct: the LLM decides when to halt, not the harness.

Whether the loop continues or not is the model’s call. If it thinks it doesn’t have enough, it emits another web_search. If it thinks it does, it skips tool_use and just produces text. The harness doesn’t get a vote.

What’s still missing

The multi-turn loop solves “one step isn’t enough.” But it has an obvious flaw: the model is purely reactive — wherever its eyes land, that’s where it goes next. No global view. Easy to wander off. A complex question that really should be researched along five dimensions might trap it on the first dimension for seven or eight searches while the others get forgotten. Or it picks a bad keyword early and turns spinning in the wrong direction, eating tokens and producing nothing.

Pure ReAct can keep moving, but it cannot guarantee that it is going anywhere. It has tactics, but no strategy. And an agent without a strategy is just a rather expensive way to walk in circles.

In Part 2 of this series, we give it a plan.

In Part 3 of this series, we cut to coding.

Agentic Engineering grew out of a desire to create something useful for people trying to make sense of agentic AI and beyond. If you’d like to support that direction, please tell your friends about us.

Share Agentic Engineering

That’s it for this one. We’ll pick up the conversation next week.

Until then, keep building.

Tanya D’cruz
Editor-in-Chief

Editor’s note

A quick programming detour from us this week.

If you regularly read Agentic Engineering, you know our core issues tend to follow a fairly deliberate cadence. This isn’t one of those issues. Yesterday, we hosted a free live session with Nicole Königstein on building AI agent harnesses for finance, and rather than let a useful conversation disappear into the webinar graveyard, we wanted to make the full recording freely available here.

Nicole is an AI researcher working on agentic systems and transformers, an instructor, consultant, and O’Reilly author. She teaches large language models and agentic architectures for organizations including the University of Oxford and O’Reilly Media, and much of her work focuses on the considerably harder problem that begins after an AI prototype works: making the resulting system reliable enough for production.

Enough preamble. Let’s get into the harness.

The conversation around agents has moved quickly from prompts to context, loops, orchestration, and now harnesses. But finance is a particularly unforgiving place to discover that a capable model and a reliable system are two very different things.

Nicole spent the session unpacking what has to exist around the model: coordination, verification, observability, control, model selection, evaluation, and the boundaries between agents. Her central argument is a useful one for anyone building agents, whether or not you work in finance: an agent is the model plus the harness.

And once you start thinking about agents that way, reliability becomes less about finding a smarter model and much more about engineering the system around it.

The video above contains the complete session and Q&A. Below, we’ve turned the central ideas from Nicole’s talk into a written companion for those who would rather read or return to the concepts later.

The AI engineering problem has moved

For a long time, the dominant pattern in machine learning was fairly straightforward: task → model → output. If performance was poor, you improved the model.

Agents change that equation. Once tools, retrieval, memory, multiple agents, and decision points are incorporated into the architecture, the question shifts from “Can the model solve this task?” to “Can the system execute a stable sequence of decisions?”

That shift also explains the move from prompt engineering to context engineering and, now, harness engineering. Prompt engineering optimizes the input. Context engineering optimizes what the model sees. Harness engineering deals with everything that governs how the system actually operates: orchestration, tools, memory, retrieval, verification, governance, security, feedback, and execution loops.

The result is a different operating model: execute → observe → verify → correct → repeat. The model still matters, but it is now one component inside a much larger engineering problem.

Strong agents can still make a weak system

This was perhaps the most important warning in the session. It is tempting to assume that if every individual agent in a multi-agent system performs well, the resulting system should perform well too. It doesn’t necessarily follow.

Nicole points to the compounding reliability problem: when independent components execute sequentially, overall system success depends on the success probabilities across those components. Errors can compound as the chain becomes longer.

Imagine an agentic financial workflow involving ten decisions. Even if the individual components are reasonably capable, failures at one stage can affect everything downstream. Add retries as the default recovery mechanism, and the system may eventually produce an answer, but now you have another problem: cost and latency.

Nicole describes three consequences of trying to brute-force reliability through repeated attempts: more compute, more money, and more latency.

And average latency can hide the real production experience. A system with an acceptable mean response time but poor p95 or p99 latency can still feel broken to a meaningful portion of its users.

Reliability, in other words, cannot simply be retried into existence.

The difficult part often lives between agents

One of the more useful distinctions Nicole makes is between communication and coordination.

Communication is the exchange of messages. Coordination controls what the agents do. That matters because an agent handoff isn’t equivalent to a conventional function call.

When one agent hands something to another, the receiving agent has to interpret the message. A schema can ensure that the message has the right structure, but it cannot guarantee that the meaning survived the handoff intact.

The same applies to delegation. Calling a sub-agent isn’t simply calling another tool. You are introducing another probabilistic inference process into the system. That means the boundaries between agents deserve their own engineering attention.

Nicole’s argument here is particularly important for evaluation: if you evaluate only the final output, you may completely miss a coordination problem occurring within the system.

You need signals around the handoffs themselves. You need telemetry. You need to understand whether agents are communicating correctly, whether coordination is progressing as intended, and where meaning begins to drift.

As Nicole puts it, you can’t improve coordination using output-only feedback.

Maybe the model doesn’t need to change

This leads to a more interesting possibility.

A great deal of AI development still revolves around the model: use a more capable model, fine-tune it, or wait for the next generation. But Nicole’s research explores another direction: keep the model fixed and improve the harness around it.

The adjustable state of the system can move away from model weights and into the surrounding harness. Execution traces and trajectories can be scored, fed back into the system, and used to improve how it behaves.

Nicole took this further in her own open-source work.

Rather than hard-coding every coordination choice before the system has even executed a task, she began experimenting with learned coordination: allowing aspects of the harness to become adaptive, auditable, and transferable.

In her experiments, the benefits were particularly visible on coordination-heavy tasks, where she reported improvements in reliability and quality alongside lower costs. Simpler procedural tasks saw less improvement.

There is an important engineering principle hiding in there: not every task needs the same agent architecture.

Stop giving every job to your strongest model

That principle also applies to model selection. There is an understandable instinct when building agentic systems to reach for the strongest reasoning model available, particularly for high-stakes domains. But stronger and more expensive everywhere does not mean better.

In Nicole’s coding experiments, harder, coordination-heavy tasks benefited from a stronger planner, while simpler stages could be handled by cheaper models. She found that distributing models according to the demands of each stage could reduce token consumption and improve speed. More surprisingly, in her experiments, the resulting code was also judged better than code produced using the top-tier model at maximum effort throughout.

This isn’t an argument for universally choosing smaller models. It is an argument for matching capability to responsibility.

Nicole offered a useful mental model during the Q&A: think about building a multi-agent system the way you would build a team.

If you were staffing the equivalent workflow inside an organization, you probably wouldn’t hire the same person for every role. Some jobs require deep expertise. Others require analysis, verification, classification, retrieval, or straightforward execution.

The skills should be complementary. Agent architectures can be designed the same way.

Coding agents need harnesses too

There was also a useful detour into coding agents.

Nicole described an experience in which an AI coding assistant, used without sufficiently strict instructions and boundaries, altered working agent code and introduced serious bugs and security problems.

Her takeaway was not that coding agents shouldn’t be used. Quite the opposite.

It was that using them effectively requires the same engineering discipline we expect elsewhere in an agentic system: clear instructions, guardrails, skills, hooks, review, and people who understand the system well enough to recognize when generated code is wrong.

There is a broader lesson here. AI can accelerate implementation, but it does not remove the need to know what good implementation looks like.

What this means for financial agents

Finance makes all of these problems harder to ignore.

A research assistant, portfolio-analysis system, financial-document agent, or other agentic workflow may need to retrieve evidence, coordinate multiple reasoning steps, invoke tools, maintain state, validate intermediate results, and operate within clearly defined constraints.

And probabilistic systems bring their own hazards.

During the Q&A, Nicole specifically warned about forward-looking bias in financial applications. Models are extremely good at pattern matching, which can also lead them to derive patterns and make suggestions that are not adequately supported by the underlying facts.

So the problem is not merely whether an LLM knows finance.

The problem is whether the system surrounding it can constrain what it does, verify what it produces, expose how it arrived there, and respond sensibly when something goes wrong.

That is the territory of the harness.

The agent is bigger than the model

Nicole closed the main presentation with a progression worth keeping: executable reasoning → inspectable states → feedback-driven control → learned harness decisions.

It captures a shift we’re increasingly interested in at Agentic Engineering.

The model is becoming one component inside a much larger engineering problem.

The interesting work is moving outward: into the runtime, the context, the tools, the coordination layer, the evaluation system, the boundaries, the feedback loops, and the machinery responsible for keeping the whole thing under control.

That is especially visible in finance, where the cost of a system being mostly right can become very real, very quickly.

And perhaps that’s the simplest way to think about harness engineering: Capability comes from the model. Dependability has to be engineered around it.

The foundation is set. Now, build the thing.

This free webinar covered the architecture. You learnt why a financial agent is not simply a model with a clever prompt, but a system in which the harness controls what the model can access, do, remember, verify, and prove.

Agentic AI for Finance, Nicole’s four-day intensive, is where you build that system.

You’ll connect models to financial data, tools, and memory; engineer verification and governance into the workflow; and see what happens when all those neat boxes on an architecture diagram actually have to work together. The aim is to leave knowing how to build an agent that is useful, observable, and safe to operate.

What you’ll leave with

Alongside the hands-on build experience, you’ll get:

• Four days of live access to the workshop with Nicole

• Downloadable code templates, datasets, and architecture blueprints

• Lifetime access to the workshop recordings

• A free copy of Building AI Agents for Finance to go deeper into the engineering covered during the program

• A certificate on completion, plus practical experience relevant to work across quantitative research, fintech, portfolio management, and financial engineering

Agentic AI for Finance runs across August 29–31 and September 12–13, giving you time between sessions to digest what you’ve built before moving into the next stage.

So if you enjoyed Nicole’s webinar and want to take the next step from understanding the architecture to engineering it yourself, book your spot for Agentic AI for Finance below.

Book your spot now

That’s it for this one. We’ll pick up our regular programming next week.

Until then, don’t just engineer the agent. Engineer the harness.

Tanya D’cruz
Editor-in-Chief

But autonomy is not free. The further a system can go on its own, the more room it has to drift, hallucinate, misinterpret a goal or make a consequential decision before someone notices.

For this episode of the Agentic Engineering podcast, I spoke with Ben Auffarth, Director and Chief Data Officer at Chelsea AI, founder of Chelsea AI Ventures, and author of several bestselling AI and machine learning books, including Generative AI with LangChain, Machine Learning for Time-Series with Python and Artificial Intelligence with Python Cookbook. Ben holds a PhD in computational neuroscience and has built systems across finance, insurance, travel and other regulated environments.

Our conversation centred on a digital-footprint intelligence platform his team is building for estate agents. It is the kind of problem that sounds made for an autonomous agent: search for a person, gather information, assess the evidence and produce a useful profile.

Ben’s team made a different choice. They use LLMs extensively during development (to build datasets, run experiments, and improve the underlying system) but limit their authority when a real user is being evaluated.

That decision led us to a more useful question than simply asking “How autonomous can this agent become?”, which is →

How much autonomy does the system actually need?

What does Neighborhood do, and where does AI fit into the system?

Neighborhood helps estate agents prioritise prospective property buyers. An enquiry may contain little more than a name, email address, telephone number and sometimes an address. The platform enriches those details with publicly available information, tries to verify that it has found the correct person and returns a profile with match scores and source information.

The difficult part is not finding someone with the right name; it is knowing whether the information belongs to this particular person. We use LLMs to help build datasets, extract information and run experiments quickly. At inference time, however, the LLM does not make the final matching or prioritisation decision. Those decisions are handled by more deterministic models and statistical methods. That is what I mean by bounded autonomy.

What tells the system that it has found enough information?

A general search engine may return many possible matches and leave someone to work through them. An LLM-based search tool can return a neat answer with a confidence percentage, but that number is not necessarily calibrated. If a model says it is 70 per cent confident, it does not mean that comparable answers are correct 70 per cent of the time.

We wanted confidence to have a measurable meaning. Neighborhood uses statistical principles, information theory and record-linkage methods to estimate the probability of a match. A common name in a large city begins with more uncertainty, and each additional piece of matching evidence adjusts that probability. The system stops relying on whatever the LLM “feels” and works with confidence that can be compared against data.

Can missing information become a hidden penalty?

Some people have extensive digital footprints, while others share very little or use platforms the system cannot access. The absence of information should not automatically become evidence that someone is suspicious or unlikely to proceed.

We found that a comprehensive profile does not necessarily mean someone is a stronger buyer. We therefore try not to downgrade people heavily when little information is available. More information may help an estate agent ask better questions, but that is a product assumption to test—not a universal truth. Sometimes the most accurate answer is simply that we do not know enough.

Is a more autonomous agent necessarily more advanced?

Autonomy is often presented as a measure of technical maturity. I do not think it is something to pursue for its own sake. The more autonomy we add, the more control we give up and the more variance we introduce.

Agents can look impressive in short demonstrations, but benchmarks involving longer tasks still show very low completion rates. If reliability matters, moving down the autonomy scale can be the better engineering decision. Use the LLM for tasks it performs relatively well, such as extracting information, and place deterministic decisions and constraints around it.

What does meaningful human oversight look like?

Putting a person at the end of a workflow does not guarantee meaningful oversight. If the system is usually correct and its output looks professional, the reviewer may gradually stop examining the evidence and simply approve each recommendation.

In Neighborhood, we avoid asking whether an entire profile “looks right.” Instead, we ask the reviewer to identify a believable anchor, such as a company or location that connects the information to the correct person. They can also say they are unsure. Human oversight should contribute evidence or judgement the system does not possess; otherwise, it is merely rubber-stamping.

How do you evaluate whether the system’s confidence is justified?

A system may retrieve accurate information and still attach it to the wrong person. That is why we expose the evidence behind a profile, our confidence in each piece and its provenance. The estate agent can inspect the original sources and provide feedback, while the final decision remains with them.

The business measure is not whether a profile looks comprehensive. It is whether the system helps estate agents spend their time with the right prospective buyers. Evaluation must cover identity matching, evidence quality, confidence calibration and the usefulness of the eventual recommendation—not only the final score.

Are we confusing a successful demo with a reliable system?

Agent demonstrations usually show examples in which the system finds the right information, follows the right steps and produces a convincing answer. Production introduces ambiguous identities, inaccessible data, regional differences and inputs that look nothing like the demonstration.

At Neighborhood, we used agents and repeated runs to help create a labelled dataset, then combined those outputs with human evaluation. We train and tune more deterministic models against that data and track the results of each experiment. That may be less exciting than showing an agent browsing autonomously, but it gives us something we can measure and reproduce.

What should a business leader establish before deploying an agent?

I would ask for four things: a labelled evaluation set, a baseline to improve upon, an understanding of the cost of a wrong answer and a measurement of response variance. Run the same input several times and see whether the system gives the same answer. If it does not, average performance may conceal an agent that cannot be trusted on an individual decision.

Businesses often want to fund the agent before they fund the dataset. That is backwards. Without representative data, you cannot reliably build, compare or govern the system. In many AI projects, the dataset, not the model, is the real bottleneck.

How much autonomy should engineers give an agent?

There is a perception that because a system uses AI, less engineering is required. In practice, the opposite is often true.

You need to define the outcome clearly, establish a benchmark early, and constrain what the system is allowed to do. Deterministic checks and filters can prevent entire categories of failure. An orchestration framework such as LangGraph can also make the available paths explicit: some decisions can be made by the model, while others remain ordinary conditional logic.

The question is not whether the agent could make another decision autonomously. It is whether that autonomy improves the system enough to justify the additional variance and loss of control.

Ben’s case for bounded autonomy is not an argument against agents. His team uses LLMs to create data, accelerate experimentation, extract information, and improve development. The argument is about placing autonomy where it creates value and removing it where consistency matters more.

That distinction becomes especially important now as agents move from helping people produce content or write code towards influencing decisions about buyers, customers, patients and applicants.

The most advanced system may not be the one permitted to do everything. It may be the one whose builders know exactly where the model is useful, where deterministic software should take over and where a person must still make the call.

Before asking how autonomous an agent can become, businesses should decide what happens when it is confidently wrong.

Less hype, more engineering conversations with the experts.

Pull up a chair.

Subscribe now