RAG & Agent Strategy: Complex System Design

RAG and Agent are the two directions where AI PMs are most easily led astray by buzzwords. Many roadmaps open with "build an Agent" or "add RAG," but the user task hasn't even been defined yet. Result: lots of architecture terminology, very little product value.

This page isn't about implementation details. It's about judging from a PM perspective: when to use RAG, when to use Agent, and when neither should be rushed.

---

Bottom Line: Ask About the User Task First, Then Choose System Form

A more stable decision sequence:

1. Does the user need "more accurate answers" or "more complex execution"
2. Are you lacking knowledge access capability or task orchestration capability
3. Are risk and cost worth introducing a more complex system

If these three steps aren't thought through first, teams easily mistake technical complexity for product progress.

---

What Problems Are Better Suited for RAG

RAG's core value isn't making answers "smarter." It's making answers more grounded.

It fits better when:

If the problem is fundamentally "the model doesn't know this material," RAG is usually the right direction.

---

What Problems Are Better Suited for Agent

Agent's core value isn't being "more human-like." It's being able to execute multi-step tasks.

It fits better when:

If the task is fundamentally "do A first, then judge, then do B," Agent starts to make sense.

---

When You Shouldn't Rush to Add RAG

Many cases of "RAG performing poorly" aren't actually model problems. They're knowledge base governance failures.

---

When You Shouldn't Casually Add Agent

Agent isn't an upgraded chat box. Many products actually only need a clear flow + a few tool calls, not a full agent loop.

---

A More Practical Decision Framework

Start with this table:

What PMs should avoid most is choosing the most complex solution because "it sounds more advanced."

---

RAG PM Focus Points: Beyond Retrieval Accuracy

More important questions to watch:

Whether RAG works well depends heavily on knowledge operations, not just per-retrieval metrics.

---

Agent PM Focus Point: Controllability

The question Agents should face isn't "is it cool" but:

1. Will it call tools it shouldn't call
2. Will it keep executing based on wrong premises
3. Is each step observable
4. Can it abort or escalate to human on failure

If these can't be answered clearly, the Agent approach usually isn't mature enough.

---

When RAG + Agent Ship Together, Layer Them First

A more stable approach isn't building one "big comprehensive Agent" at once. Break it into:

knowledge layer -> retrieval layer -> decision layer -> action layer

This way you can distinguish:

• Was it a retrieval error
• Was it a judgment error
• Was it a tool execution error

Once systems get complex, the worst thing is having nobody know which layer the error happened in.

---

Most Overlooked Costs

RAG and Agent costs aren't limited to API bills.

They also include:

• Document governance costs
• Eval and monitoring costs
• Prompt / workflow maintenance costs
• Bad case handling costs

If PMs only budget "model call costs," they'll typically underestimate significantly.

---

A Sufficient Strategy Review Question Set

Before discussing RAG or Agent, have the team answer:

1. What's the user task exactly
2. What's the worst-case failure consequence
3. Why isn't a simpler workflow sufficient
4. How will quality be monitored post-launch
5. Which layer can you roll back to when issues arise

If these 5 questions can't be answered clearly, hold off on drawing complex architecture diagrams.

---

Practice

Take your most-wanted AI feature. First determine which category it resembles:

1. Knowledge-type problem
2. Multi-step execution problem
3. Both
4. Actually just regular automation

Get the classification right, and the system design direction usually won't be too far off.