Editor’s note
A quick programming detour from us this week.
If you regularly read Agentic Engineering, you know our core issues tend to follow a fairly deliberate cadence. This isn’t one of those issues. Yesterday, we hosted a free live session with Nicole Königstein on building AI agent harnesses for finance, and rather than let a useful conversation disappear into the webinar graveyard, we wanted to make the full recording freely available here.
Nicole is an AI researcher working on agentic systems and transformers, an instructor, consultant, and O’Reilly author. She teaches large language models and agentic architectures for organizations including the University of Oxford and O’Reilly Media, and much of her work focuses on the considerably harder problem that begins after an AI prototype works: making the resulting system reliable enough for production.
Enough preamble. Let’s get into the harness.
The conversation around agents has moved quickly from prompts to context, loops, orchestration, and now harnesses. But finance is a particularly unforgiving place to discover that a capable model and a reliable system are two very different things.
Nicole spent the session unpacking what has to exist around the model: coordination, verification, observability, control, model selection, evaluation, and the boundaries between agents. Her central argument is a useful one for anyone building agents, whether or not you work in finance: an agent is the model plus the harness.
And once you start thinking about agents that way, reliability becomes less about finding a smarter model and much more about engineering the system around it.
The video above contains the complete session and Q&A. Below, we’ve turned the central ideas from Nicole’s talk into a written companion for those who would rather read or return to the concepts later.
Want to go from understanding the harness to engineering one?
This free webinar introduced the architecture behind reliable AI agents for finance. We see how the model, harness, coordination layer, and reliability mechanisms fit together.
On August 29, Nicole will take that foundation into Agentic AI for Finance, our four-day intensive workshop where you’ll move from understanding those systems to building them hands-on using real financial workflows.
We’ve broken down what you’ll build, what you’ll take away, and the other benefits of joining the intensive at the end of this article, so keep reading. Or, if you’re already convinced and would rather skip the rest of our pitch, you can book your place right away.
The AI engineering problem has moved
For a long time, the dominant pattern in machine learning was fairly straightforward: task → model → output. If performance was poor, you improved the model.
Agents change that equation. Once tools, retrieval, memory, multiple agents, and decision points are incorporated into the architecture, the question shifts from “Can the model solve this task?” to “Can the system execute a stable sequence of decisions?”
That shift also explains the move from prompt engineering to context engineering and, now, harness engineering. Prompt engineering optimizes the input. Context engineering optimizes what the model sees. Harness engineering deals with everything that governs how the system actually operates: orchestration, tools, memory, retrieval, verification, governance, security, feedback, and execution loops.
The result is a different operating model: execute → observe → verify → correct → repeat. The model still matters, but it is now one component inside a much larger engineering problem.
Strong agents can still make a weak system
This was perhaps the most important warning in the session. It is tempting to assume that if every individual agent in a multi-agent system performs well, the resulting system should perform well too. It doesn’t necessarily follow.
Nicole points to the compounding reliability problem: when independent components execute sequentially, overall system success depends on the success probabilities across those components. Errors can compound as the chain becomes longer.
Imagine an agentic financial workflow involving ten decisions. Even if the individual components are reasonably capable, failures at one stage can affect everything downstream. Add retries as the default recovery mechanism, and the system may eventually produce an answer, but now you have another problem: cost and latency.
Nicole describes three consequences of trying to brute-force reliability through repeated attempts: more compute, more money, and more latency.
And average latency can hide the real production experience. A system with an acceptable mean response time but poor p95 or p99 latency can still feel broken to a meaningful portion of its users.
Reliability, in other words, cannot simply be retried into existence.
The difficult part often lives between agents
One of the more useful distinctions Nicole makes is between communication and coordination.
Communication is the exchange of messages. Coordination controls what the agents do. That matters because an agent handoff isn’t equivalent to a conventional function call.
When one agent hands something to another, the receiving agent has to interpret the message. A schema can ensure that the message has the right structure, but it cannot guarantee that the meaning survived the handoff intact.
The same applies to delegation. Calling a sub-agent isn’t simply calling another tool. You are introducing another probabilistic inference process into the system. That means the boundaries between agents deserve their own engineering attention.
Nicole’s argument here is particularly important for evaluation: if you evaluate only the final output, you may completely miss a coordination problem occurring within the system.
You need signals around the handoffs themselves. You need telemetry. You need to understand whether agents are communicating correctly, whether coordination is progressing as intended, and where meaning begins to drift.
As Nicole puts it, you can’t improve coordination using output-only feedback.
Maybe the model doesn’t need to change
This leads to a more interesting possibility.
A great deal of AI development still revolves around the model: use a more capable model, fine-tune it, or wait for the next generation. But Nicole’s research explores another direction: keep the model fixed and improve the harness around it.
The adjustable state of the system can move away from model weights and into the surrounding harness. Execution traces and trajectories can be scored, fed back into the system, and used to improve how it behaves.
Nicole took this further in her own open-source work.
Rather than hard-coding every coordination choice before the system has even executed a task, she began experimenting with learned coordination: allowing aspects of the harness to become adaptive, auditable, and transferable.
In her experiments, the benefits were particularly visible on coordination-heavy tasks, where she reported improvements in reliability and quality alongside lower costs. Simpler procedural tasks saw less improvement.
There is an important engineering principle hiding in there: not every task needs the same agent architecture.
Stop giving every job to your strongest model
That principle also applies to model selection. There is an understandable instinct when building agentic systems to reach for the strongest reasoning model available, particularly for high-stakes domains. But stronger and more expensive everywhere does not mean better.
In Nicole’s coding experiments, harder, coordination-heavy tasks benefited from a stronger planner, while simpler stages could be handled by cheaper models. She found that distributing models according to the demands of each stage could reduce token consumption and improve speed. More surprisingly, in her experiments, the resulting code was also judged better than code produced using the top-tier model at maximum effort throughout.
This isn’t an argument for universally choosing smaller models. It is an argument for matching capability to responsibility.
Nicole offered a useful mental model during the Q&A: think about building a multi-agent system the way you would build a team.
If you were staffing the equivalent workflow inside an organization, you probably wouldn’t hire the same person for every role. Some jobs require deep expertise. Others require analysis, verification, classification, retrieval, or straightforward execution.
The skills should be complementary. Agent architectures can be designed the same way.
Coding agents need harnesses too
There was also a useful detour into coding agents.
Nicole described an experience in which an AI coding assistant, used without sufficiently strict instructions and boundaries, altered working agent code and introduced serious bugs and security problems.
Her takeaway was not that coding agents shouldn’t be used. Quite the opposite.
It was that using them effectively requires the same engineering discipline we expect elsewhere in an agentic system: clear instructions, guardrails, skills, hooks, review, and people who understand the system well enough to recognize when generated code is wrong.
There is a broader lesson here. AI can accelerate implementation, but it does not remove the need to know what good implementation looks like.
What this means for financial agents
Finance makes all of these problems harder to ignore.
A research assistant, portfolio-analysis system, financial-document agent, or other agentic workflow may need to retrieve evidence, coordinate multiple reasoning steps, invoke tools, maintain state, validate intermediate results, and operate within clearly defined constraints.
And probabilistic systems bring their own hazards.
During the Q&A, Nicole specifically warned about forward-looking bias in financial applications. Models are extremely good at pattern matching, which can also lead them to derive patterns and make suggestions that are not adequately supported by the underlying facts.
So the problem is not merely whether an LLM knows finance.
The problem is whether the system surrounding it can constrain what it does, verify what it produces, expose how it arrived there, and respond sensibly when something goes wrong.
That is the territory of the harness.
The agent is bigger than the model
Nicole closed the main presentation with a progression worth keeping: executable reasoning → inspectable states → feedback-driven control → learned harness decisions.
It captures a shift we’re increasingly interested in at Agentic Engineering.
The model is becoming one component inside a much larger engineering problem.
The interesting work is moving outward: into the runtime, the context, the tools, the coordination layer, the evaluation system, the boundaries, the feedback loops, and the machinery responsible for keeping the whole thing under control.
That is especially visible in finance, where the cost of a system being mostly right can become very real, very quickly.
And perhaps that’s the simplest way to think about harness engineering: Capability comes from the model. Dependability has to be engineered around it.
The foundation is set. Now, build the thing.
This free webinar covered the architecture. You learnt why a financial agent is not simply a model with a clever prompt, but a system in which the harness controls what the model can access, do, remember, verify, and prove.
Agentic AI for Finance, Nicole’s four-day intensive, is where you build that system.
You’ll connect models to financial data, tools, and memory; engineer verification and governance into the workflow; and see what happens when all those neat boxes on an architecture diagram actually have to work together. The aim is to leave knowing how to build an agent that is useful, observable, and safe to operate.
What you’ll leave with
Alongside the hands-on build experience, you’ll get:
Four days of live access to the workshop with Nicole
Downloadable code templates, datasets, and architecture blueprints
Lifetime access to the workshop recordings
A free copy of Building AI Agents for Finance to go deeper into the engineering covered during the program
A certificate on completion, plus practical experience relevant to work across quantitative research, fintech, portfolio management, and financial engineering
Agentic AI for Finance runs across August 29–31 and September 12–13, giving you time between sessions to digest what you’ve built before moving into the next stage.
So if you enjoyed Nicole’s webinar and want to take the next step from understanding the architecture to engineering it yourself, book your spot for Agentic AI for Finance below.
That’s it for this one. We’ll pick up our regular programming next week.
Until then, don’t just engineer the agent. Engineer the harness.
Tanya D’cruz
Editor-in-Chief
