Zucchetti LLM Course — Lecture 7

Agentic RAG
& Autonomous Systems

When the LLM becomes the decision-maker

Kamyar Zeinalipour

Today’s Agenda

The Autonomy Problem

Some tasks can’t be hardcoded — you don’t know the steps in advance

The Capability Stack

Each lecture adds a layer of intelligence to your retrieval system

What Makes a System “Agentic”?

Four capabilities that separate agents from pipelines

What Is Function Calling?

The LLM does NOT execute code. It writes a request. Your code executes it.

The Function Calling Flow

Five steps — the LLM never touches the internet

Step 1: Defining a Tool

You describe your tools in JSON Schema — the LLM reads this to decide what to call

{
  "name": "get_weather",
  "description": "Get current weather for a city",
  "parameters": {
    "type": "object",
    "properties": {
      "city":  {"type": "string", "description": "City name, e.g. 'Rome'"},
      "units": {"type": "string", "enum": ["celsius", "fahrenheit"],
                "description": "Temperature units"}
    },
    "required": ["city"]
  }
}

Step 2: The LLM Decides

From natural language to structured JSON — automatically

{
  "name": "get_weather",
  "arguments": {"city": "Rome", "units": "celsius"}
}

Step 3: Execute & Return

Your code runs the real API — the LLM never touched the internet

Tool Definition Best Practices

The “intern test”: if an intern can’t use your tool from the description, the LLM can’t either

Parallel Function Calling

The LLM generates multiple tool calls at once — when they’re independent

[
  {"name":"get_weather", "args":{"city":"Rome"}},
  {"name":"get_weather", "args":{"city":"Paris"}}
]

Multi-Turn Tool Calling

The LLM chains tools automatically — each result shapes the next decision

The Toolformer Insight NeurIPS 2023 Oral

LLMs can learn to decide when to call tools — not just how

Schick et al. (2023) — “Toolformer: Language Models Can Teach Themselves to Use Tools” — NeurIPS 2023 (Oral)

Common Pitfalls

Five failure modes you’ll hit in production

Function Calling — Summary

From L3 Chaining to L7 ReAct

Same pattern, fundamentally different control

From L5 Decomposition to L7 ReAct

Both break queries into steps — but when?

The Problem with Chain-of-Thought

CoT reasons but can’t verify — it thinks, it doesn’t know

The ReAct Loop ICLR 2023

Thought → Action → Observation — repeat until done

The LLM generates Thought and Action. The Observation comes from the tool. The loop continues until a finish() action.

Yao et al. (2023) — “ReAct: Synergizing Reasoning and Acting in Language Models” — ICLR 2023

ReAct — Worked Example (Part 1)

“What awards has the director of Inception won?”

continued on next slide →

ReAct — Worked Example (Part 2)

ReAct vs Chain-of-Thought

Same question, different approaches

ReAct — HR Domain Example

“Is Mario eligible for the senior developer promotion?”

When ReAct Fails

Four failure modes and their fixes

Self-RAG: The Model Decides ICLR 2024 Oral

Reflection tokens let the model critique its own generation

Asai et al. (2024) — “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection” — ICLR 2024 (Oral, top 1%)

Self-RAG — Worked Example

The model retrieves only when it needs to

Function Calling vs ReAct

When to use which — they’re complementary, not competing

☕ Break — 10 minutes

When we come back: Agentic RAG Architectures

So far: tools (Section 2) + reasoning (Section 3).
Next: put them together into a self-correcting RAG system.

Naive RAG vs Agentic RAG

The Agentic RAG Pipeline

Full architecture — color-coded by origin

If Retrieval Grader says ❌ → rewrite query → loop back. If Answer Grader says ❌ → retry.

Component 1: Query Analyzer

L5 routing always retrieves. The Query Analyzer adds a new option: don’t retrieve at all.

Component 2: Smart Retriever

L5 gave you the tools. This agent orchestrates them.

Component 3: Retrieval Grader

After retrieval: is this context actually relevant?

Component 4: Answer Grader

After generation: is my answer actually good?

Adaptive RAG NAACL 2024

Route by query complexity — save cost on simple questions

A small classifier routes before the expensive pipeline starts.

Jeong et al. (2024) — “Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity” — NAACL 2024

Corrective RAG — The Problem

What happens when retrieval returns garbage?

Corrective RAG — The Solution arXiv 2024

Three-way grading: Correct, Ambiguous, or Incorrect

Yan et al. (2024) — “Corrective Retrieval Augmented Generation” — arXiv 2024

CRAG — Worked Example

“What is Zucchetti’s remote work policy?”

Building It: Architecture Choices

How do you implement an agentic RAG system?

The Full Picture

You’re not starting from scratch — you’re adding intelligence to what you already built

Why Multi-Agent?

Three reasons to split work across agents

Pattern 1: Sequential (Pipeline)

Agents pass work linearly — each has focused context

Pattern 2: Hierarchical (Supervisor)

A manager agent routes sub-tasks to specialists

Supervisor — Worked Example

“Write a report on our Q3 sales performance”

Guardrails & Safety

“Agents without guardrails are interns with root access”

Production Considerations

Four pillars for production-grade agents

The Agent Framework Landscape

When NOT to Use Agents

Don’t use a chainsaw to cut butter

The Cost of Agency

Approximate figures — vary by model, provider, and implementation

Key Takeaways

The Full Retrieval Stack

Four levels of retrieval sophistication — you now have all of them

Choosing Your Level

A decision flowchart for your next RAG project

What’s Next

Questions?

Thank you! See you next session.

Kamyar Zeinalipour — Zucchetti LLM Course