Editor’s note
The more conversations we have with people building these systems, the more useful material we find that cannot be contained within a single weekly issue. Rather than leave those sessions sitting in an archive or force them into our usual publishing cadence, we’ve decided to make a little more room in the week for them.
Between our regular issues, we’ll occasionally share technical sessions that deserve more than a forgotten recording link. A couple of weeks ago, we brought you Nicole Königstein’s webinar on building AI agent harnesses for finance. This time, it’s Imran Ahmad’s free webinar on the architecture of Agentic RAG.
Imran is a data scientist, AI architect, visiting professor, and the author of 30 Agents Every AI Engineer Must Build. His work looks beyond agent demos to the decisions that matter in production: when to add agency, where to place evaluation, and whether another loop is worth the extra cost and latency.
We’ve also put together a free take-home pack with Imran’s complete slide deck, a practical guide to 10 production agent patterns, and a one-page map of 30 agent architectures. You’ll find it later in the article.
Enough preamble. Let’s get into it!
Traditional RAG has an awkward habit: it keeps going.
Ask it about an internal company acronym, and it retrieves the closest available chunks. Misspell that acronym, and it still retrieves the closest available chunks. Those chunks are passed to the model, and the model produces an answer that may sound perfectly credible—even when the information needed to answer the question was never found.
Every component completed its assigned task. The system still failed.
That is the problem Imran explores in this free Agentic Engineering webinar on the architecture of agentic RAG.
His argument is not that conventional RAG is broken, nor that every retrieval pipeline now needs an agent. It is that traditional RAG is largely a one-way street. Once retrieval begins, there is often no meaningful point at which the system can inspect what it found, recognise that it is insufficient, and change course.
Agentic RAG introduces that missing loop.
RAG is really automated context generation
Large language models know a great deal, but they do not know everything about your organisation.
They may not know your internal terminology, current policies, proprietary research, or the particular meaning your company has assigned to an acronym. RAG addresses this by finding relevant information in an external knowledge source and adding it to the model’s prompt.
In other words, RAG automates context generation.
The familiar pipeline looks something like this: documents are divided into chunks, converted into embeddings, and stored in a vector database. When someone asks a question, the system searches for similar chunks, places them in the prompt, and asks the model to generate an answer from that context.
It is a useful architecture. There is a reason it became one of the first patterns companies reached for when building applications with large language models.
But its simplicity relies on several assumptions: that the user’s question is also a good search query, that the first retrieval will find the right evidence, that similarity means relevance, and that adding more context will improve the answer.
None of those assumptions is reliably true.
The problem with a one-way pipeline
Consider a user asking about an internal tool but misspelling its name.
The vector search does not necessarily respond with “I can’t find that.” It returns whichever chunks happen to be mathematically closest. A reranker may rearrange those chunks, but it is still ranking a weak set of results. The model then receives that material and does what it was designed to do: generate a plausible response.
The failure began before generation. The model was handed evidence that never supported the question.
Traditional RAG has no natural recovery mechanism here. It retrieves once and moves forward. It does not necessarily ask whether the query was misunderstood, whether a different source should be searched, or whether the available evidence is strong enough to justify an answer.
Adding an agent changes the shape of that pipeline. Retrieval can become a loop:
Plan
↓
Retrieve
↓
Evaluate
↓
Adapt
↓
Try again
That sounds like a small architectural change. In practice, however, it changes where the system is allowed to exercise judgement.
A little something for being here
If you’d rather build the patterns than just read about them, Imran is teaching a live, six-hour workshop on September 12.
In 10 Essential AI Agents Every Engineer Must Build, you’ll build 10 working agents spanning RAG, tool use, fact-checking, multi-agent orchestration, multimodal AI, and autonomous planning. You’ll leave with the complete code repository, reusable architectures, and a complimentary copy of 30 Agents Every AI Engineer Must Build.
Tickets are normally $169.99. Agentic Engineering subscribers get 20% off with the private code AE20.
Add agency where a specific failure needs it
“Agentic RAG” can make it sound as though the entire system must suddenly become autonomous. That is not the useful way to think about it.
Agency can be introduced at different points in the pipeline, depending on the failure you are trying to correct.
At the retrieval stage, an agent can interpret the user’s request, break a complicated question into smaller searches, rewrite a weak query, or choose between available retrieval tools.
After retrieval, it can examine the evidence and ask whether the results are truly relevant. If important information is missing, it can search again rather than blindly passing weak context downstream.
At the generation stage, another check can compare the final answer with the retrieved evidence, add citations, or reject claims that the sources do not support.
You may not need all three.
If retrieval is the main weakness, putting another agent around generation could add cost without fixing the actual problem. If the answer is poorly grounded despite strong retrieval, changing the search strategy may not help.
The important question is not, “How do we make this RAG pipeline agentic?”
It is, “Where does our existing pipeline fail, and would adding judgement at that point help it recover?”
Imran’s own course materials put the principle neatly: “Only add agency when you can name the failure it fixes.” The accompanying Agentic RAG course hub maps different failure modes to patterns including adaptive retrieval, corrective RAG, GraphRAG and groundedness checks.
A useful system needs the courage to say “I don’t know”
One of the most valuable behaviours in a knowledge system may also be one of the least impressive in a demo: refusing to answer.
If the retrieved passages do not contain the requested information, a reliable system should be able to say so. It might ask the user to clarify an acronym, try another source or explain that it could not find enough evidence.
That is much more useful than producing a polished answer from irrelevant material.
An evaluator inside the retrieval loop can provide that gate. It can score the evidence, check whether it covers the question, and decide whether the system should proceed, search again, or stop.
Of course, adding an evaluator does not make the system correct. The evaluator can make a poor judgement too. Its decisions need test cases, thresholds, and traces of their own.
The improvement is architectural rather than magical: the system now has a place where failure can be noticed.
More context is not always better context
Another assumption behind basic RAG is that retrieving more information gives the model a better chance of answering accurately.
Initially, it often does. But the relationship does not continue indefinitely.
As more chunks enter the prompt, relevant evidence competes with irrelevant material. Different sources may contradict each other. A precise but lengthy passage can pull the model away from the user’s original question. Imran describes this as a signal-and-noise problem: the goal is not to maximise context but to preserve the right signal.
Large context windows do not remove this problem. Being able to place the equivalent of several books into a prompt does not mean the model will identify the right paragraph, weigh every source correctly, or remain focused on the original task.
This is where an agent can help curate context rather than merely accumulate it. It can search for missing evidence, reject weak results, and recognise contradictions before generation.
But again, that judgement has a price.
Retrieval should reflect the question being asked
A short factual question and a request to explain a complete block of code do not require the same kind of context.
With a fixed RAG pipeline, they may still be treated in much the same way. The same chunk size, overlap, retrieval algorithm, and number of results are used regardless of the question.
An agentic retrieval layer can make the process more adaptive. Depending on the tools and indexes available, it can choose a retrieval strategy, change the query, search multiple sources, or gather a wider section of material when the answer depends on information spread across several passages.
For a compound question, it might separate the request into several searches and confirm that each part has been answered. For a vague query, it might rewrite the search or ask the user for clarification. For a technical question, it may retrieve a larger, more coherent unit rather than an isolated fragment.
The underlying trade-offs do not disappear. Larger chunks carry more context but reduce precision. Smaller chunks may retrieve precisely while cutting away something essential. Semantic chunking can preserve meaning, but it introduces more computation and complexity.
Agency allows the system to make choices within those trade-offs. It does not abolish them.
If the system can change course, you need to see where it went
A fixed pipeline is relatively easy to follow. The query went in, a set of chunks came back, and the model generated an answer.
An agentic system may rewrite the query, call more than one retriever, reject the first result, alter its plan, and try again. That adaptability is the point—but it also makes the system harder to inspect.
Imran recommends a provenance layer that records the path the system took. What did it search for? Which evidence did it retrieve? Why was one result accepted and another rejected? Which tool did it choose? Where did the final claim come from?
Without those traces, it becomes difficult to distinguish a retrieval failure from an evaluation failure or a generation failure. The system may be more capable, but the team responsible for it will know less about why it behaved as it did.
Observability is therefore not an optional dashboard added after the architecture is complete. It is part of what makes an adaptive retrieval loop governable.
Agency sends a bill
Each planning step, evaluation, and retry may require another model call. That adds latency, token usage, and another possible point of failure.
A system that runs three searches and evaluates each one may retrieve better evidence than a one-shot pipeline. It may also take longer and cost considerably more.
Imran’s advice is refreshingly blunt: “Don’t hire a PhD for the job of a receptionist.”
A small, focused model may be enough to classify a query, choose a retriever, or judge whether two passages are relevant. Some of these decisions can be handled by local or quantised models. Others may not require an LLM at all.
The most sophisticated model should not automatically be placed inside every part of the loop. The model—and the loop itself—should be proportionate to the decision being made.
This is where agentic RAG becomes an engineering problem rather than a naming exercise. The team has to decide where another round of reasoning is worth its cost.
Still thinking about it?
You could be building these systems with Imran on September 12.
Agentic Engineering subscribers can take 20% off the workshop with code AE20 if ten working agents sound more useful than ten more bookmarked tutorials.
The goal is not maximum agency
The choice is not between an outdated RAG pipeline and a fully autonomous knowledge agent.
There is a spectrum between them.
A conventional pipeline may be entirely sufficient for a narrow, predictable knowledge base. A retrieval evaluator may solve the most serious failure without changing anything else. A more complex investigation might justify query decomposition, multiple retrieval tools, and several rounds of evidence gathering.
The right architecture is the smallest one that can recognise and recover from the failures that matter.
That is the real case for agentic RAG. It does not simply retrieve more information or attach an agent to a familiar acronym. It gives the system a chance to notice that the evidence is weak before fluent language turns that weakness into an answer.
Your webinar companion pack
Explore the hands-on labs and supporting material on GitHub.
Download The Agent Engineer’s Field Guide for 10 production agent patterns, including when to use them, what tends to break first, and how to evaluate them.
Download 30 Agents at a Glance for a one-page map of all 30 architectures, organised from foundational patterns to specialised and domain-specific systems.
Download Imran’s slide deck to revisit the architectures, retrieval loops, and implementation patterns later.
Continue through the patterns in Imran’s book, 30 Agents Every AI Engineer Must Build.
For less hype and more engineering, pull up a chair.
That’s it for this one. We’ll pick up the conversation next week.
Until then, keep building.
Tanya D’cruz
Editor-in-Chief
