AI agents are usually placed on an imaginary ladder. At the bottom, they assist. A little higher, they complete tasks. Eventually, they plan, act, learn and operate without us. Every step towards autonomy is treated as progress.
But autonomy is not free. The further a system can go on its own, the more room it has to drift, hallucinate, misinterpret a goal or make a consequential decision before someone notices.
For this episode of the Agentic Engineering podcast, I spoke with Ben Auffarth, Director and Chief Data Officer at Chelsea AI, founder of Chelsea AI Ventures, and author of several bestselling AI and machine learning books, including Generative AI with LangChain, Machine Learning for Time-Series with Python and Artificial Intelligence with Python Cookbook. Ben holds a PhD in computational neuroscience and has built systems across finance, insurance, travel and other regulated environments.
Our conversation centred on a digital-footprint intelligence platform his team is building for estate agents. It is the kind of problem that sounds made for an autonomous agent: search for a person, gather information, assess the evidence and produce a useful profile.
Ben’s team made a different choice. They use LLMs extensively during development (to build datasets, run experiments, and improve the underlying system) but limit their authority when a real user is being evaluated.
That decision led us to a more useful question than simply asking “How autonomous can this agent become?”, which is →
How much autonomy does the system actually need?
What does Neighborhood do, and where does AI fit into the system?
Neighborhood helps estate agents prioritise prospective property buyers. An enquiry may contain little more than a name, email address, telephone number and sometimes an address. The platform enriches those details with publicly available information, tries to verify that it has found the correct person and returns a profile with match scores and source information.
The difficult part is not finding someone with the right name; it is knowing whether the information belongs to this particular person. We use LLMs to help build datasets, extract information and run experiments quickly. At inference time, however, the LLM does not make the final matching or prioritisation decision. Those decisions are handled by more deterministic models and statistical methods. That is what I mean by bounded autonomy.
What tells the system that it has found enough information?
A general search engine may return many possible matches and leave someone to work through them. An LLM-based search tool can return a neat answer with a confidence percentage, but that number is not necessarily calibrated. If a model says it is 70 per cent confident, it does not mean that comparable answers are correct 70 per cent of the time.
We wanted confidence to have a measurable meaning. Neighborhood uses statistical principles, information theory and record-linkage methods to estimate the probability of a match. A common name in a large city begins with more uncertainty, and each additional piece of matching evidence adjusts that probability. The system stops relying on whatever the LLM “feels” and works with confidence that can be compared against data.
Can missing information become a hidden penalty?
Some people have extensive digital footprints, while others share very little or use platforms the system cannot access. The absence of information should not automatically become evidence that someone is suspicious or unlikely to proceed.
We found that a comprehensive profile does not necessarily mean someone is a stronger buyer. We therefore try not to downgrade people heavily when little information is available. More information may help an estate agent ask better questions, but that is a product assumption to test—not a universal truth. Sometimes the most accurate answer is simply that we do not know enough.
Is a more autonomous agent necessarily more advanced?
Autonomy is often presented as a measure of technical maturity. I do not think it is something to pursue for its own sake. The more autonomy we add, the more control we give up and the more variance we introduce.
Agents can look impressive in short demonstrations, but benchmarks involving longer tasks still show very low completion rates. If reliability matters, moving down the autonomy scale can be the better engineering decision. Use the LLM for tasks it performs relatively well, such as extracting information, and place deterministic decisions and constraints around it.
If you’re enjoying Ben’s approach to building AI systems that can actually be measured, he’s leading a hands-on workshop on August 29.
What does meaningful human oversight look like?
Putting a person at the end of a workflow does not guarantee meaningful oversight. If the system is usually correct and its output looks professional, the reviewer may gradually stop examining the evidence and simply approve each recommendation.
In Neighborhood, we avoid asking whether an entire profile “looks right.” Instead, we ask the reviewer to identify a believable anchor, such as a company or location that connects the information to the correct person. They can also say they are unsure. Human oversight should contribute evidence or judgement the system does not possess; otherwise, it is merely rubber-stamping.
How do you evaluate whether the system’s confidence is justified?
A system may retrieve accurate information and still attach it to the wrong person. That is why we expose the evidence behind a profile, our confidence in each piece and its provenance. The estate agent can inspect the original sources and provide feedback, while the final decision remains with them.
The business measure is not whether a profile looks comprehensive. It is whether the system helps estate agents spend their time with the right prospective buyers. Evaluation must cover identity matching, evidence quality, confidence calibration and the usefulness of the eventual recommendation—not only the final score.
Are we confusing a successful demo with a reliable system?
Agent demonstrations usually show examples in which the system finds the right information, follows the right steps and produces a convincing answer. Production introduces ambiguous identities, inaccessible data, regional differences and inputs that look nothing like the demonstration.
At Neighborhood, we used agents and repeated runs to help create a labelled dataset, then combined those outputs with human evaluation. We train and tune more deterministic models against that data and track the results of each experiment. That may be less exciting than showing an agent browsing autonomously, but it gives us something we can measure and reproduce.
What should a business leader establish before deploying an agent?
I would ask for four things: a labelled evaluation set, a baseline to improve upon, an understanding of the cost of a wrong answer and a measurement of response variance. Run the same input several times and see whether the system gives the same answer. If it does not, average performance may conceal an agent that cannot be trusted on an individual decision.
Businesses often want to fund the agent before they fund the dataset. That is backwards. Without representative data, you cannot reliably build, compare or govern the system. In many AI projects, the dataset, not the model, is the real bottleneck.
How much autonomy should engineers give an agent?
There is a perception that because a system uses AI, less engineering is required. In practice, the opposite is often true.
You need to define the outcome clearly, establish a benchmark early, and constrain what the system is allowed to do. Deterministic checks and filters can prevent entire categories of failure. An orchestration framework such as LangGraph can also make the available paths explicit: some decisions can be made by the model, while others remain ordinary conditional logic.
The question is not whether the agent could make another decision autonomously. It is whether that autonomy improves the system enough to justify the additional variance and loss of control.
Ben’s case for bounded autonomy is not an argument against agents. His team uses LLMs to create data, accelerate experimentation, extract information, and improve development. The argument is about placing autonomy where it creates value and removing it where consistency matters more.
That distinction becomes especially important now as agents move from helping people produce content or write code towards influencing decisions about buyers, customers, patients and applicants.
The most advanced system may not be the one permitted to do everything. It may be the one whose builders know exactly where the model is useful, where deterministic software should take over and where a person must still make the call.
Before asking how autonomous an agent can become, businesses should decide what happens when it is confidently wrong.
Less hype, more engineering conversations with the experts.
Pull up a chair.
