Ch. 6 — Reflection and Refinement

Chapter 6 of [[ai-ml/nvidia-certs/ncp-aai/cognition-planning-and-memory/Understanding-the-planning-of-LLM-agents-A-survey|Understanding the planning of LLM agents: A survey]]

Abstract

This chapter examines memory architectures and evaluation methodologies that underpin advanced planning capabilities in Large Language Model (LLM) agents. It distinguishes between external retrieval-based memory systems, such as Generative Agents and MemGPT, and embodied memory achieved through parameter-efficient fine-tuning (PEFT). Furthermore, it establishes rigorous benchmarking standards for interactive environments, quantifying agent performance via success rates, rewards, and computational expenses. The central contribution lies in synthesizing the trade-offs between dynamic memory updates and static parameter storage, providing a technical framework for assessing plan refinement mechanisms like Reflexion.

Key Concepts

• Generative Agents Memory Mechanism Generative Agents [Park et al. , 2023] implement a cognitive architecture where daily experiences are stored in text form and retrieved via a composite scoring function dependent on recency and relevance. This mechanism allows the agent to prioritize historical data that aligns with the current situational context, enabling human-like continuous operation. The retrieval process is critical for maintaining long-term coherence in agent behavior over extended interaction cycles.

• Vector Indexing Memory Structures Systems such as MemoryBank [Zhong et al. , 2023], TiM [Liu et al. , 2023b], and RecMind [Wang et al. , 2023c] employ text encoding models to convert memories into vector representations. These vectors are organized using indexing structures like the FAISS library [Johnson et al. , 2019] to facilitate efficient similarity search. During retrieval, the description of the current status serves as a query vector to access the memory pool.

• Hierarchical Storage Abstraction (MemGPT) MemGPT [Packer et al. , 2023] abstracts the LLM’s context window as Random Access Memory (RAM) and utilizes external storage as a disk, mirroring computer architecture principles. The LLM autonomously decides when to retrieve historical memories from storage or save the current context to long-term memory. This design addresses the context length limitations of standard transformer models by dynamically exchanging information between storage levels.

• Q-Value Based Memory (REMEMBER) REMEMBER [Zhang et al. , 2023a] stores historical interactions as tuples in a Q-value table, specifically structured as (environment, task, action, Q-value). Retrieval involves accessing both positive and negative memories based on the similarity of the environment and task to the current state. This approach leverages reinforcement learning principles to guide the LLM’s plan generation based on past performance values.

• Embodied Memory through Fine-tuning Embodied memory involves encoding the agent’s historical experiential samples directly into the model parameters via fine-tuning. Experiential samples typically consist of commonsense knowledge, task-related priors, and records of successful or failed interactions. While full training of models with billions of parameters is costly, this method embeds planning capabilities permanently into the model weights.

• Parameter-Efficient Fine-Tuning (PEFT) To mitigate the high computational cost of updating full model parameters, the chapter highlights techniques such as LoRA, QLoRA, and P-tuning. These methods allow for the training of a small subset of parameters while keeping the base model frozen. This approach enables the rapid adaptation of LLMs to specific agent tasks without requiring massive resource expenditure.

• CALM and TDT Trajectory Fine-tuning CALM [Yao et al. , 2020b] utilizes ground-truth action trajectories from text-world environments to finetune models like GPT-2 using a next-token prediction task. Similarly, TDT [Wang et al. , 2022a] uses collected Markov decision process data to fine-tune a Text Decision Transformer. Both methods aim to enable the model to generalize planning information learned from specific trajectories to new tasks.

• AgentTuning Dialogue Organizing The AgentTuning method [Zeng et al. , 2023] organizes plan trajectories from diverse tasks into a dialogue format to fine-tune models such as LLaMA. This structure has demonstrated significant performance improvements on unseen planning tasks. By organizing data as dialogue, the method leverages the pre-existing instruction-following capabilities of the LLM.

• Interactive Gaming Evaluation Environments Evaluation benchmarks include real-time multi-modal environments like Minecraft, where agents gather materials to create tools for rewards. Text-based environments such as ALFWorld [Shridhar et al. , 2020] and ScienceWorld [Wang et al. , 2022a] offer simpler feedback and fewer feasible actions. These environments measure performance using success rates or accumulated rewards obtained during interaction.

• Interactive Retrieval Environments These environments simulate human information retrieval and reasoning processes by allowing agents to interact with search engines. They are designed to evaluate the agent’s ability to plan complex search strategies and reason over retrieved information in real-world-like information processing scenarios.

• Performance Metrics (SR, AR, EX) The chapter defines specific metrics for evaluation: Success Rate (SR), Average Rewards (AR), and Expenses (EX). Expenses are calculated based on the number of consumed tokens through APIs like OpenAI. These metrics provide a multi-dimensional view of agent capability, balancing task completion against computational cost.

• Reflexion Methodology Table 2 lists “Reflexion” as a prompt-based method evaluated across benchmarks. It represents a class of techniques where agents reflect on past failures to refine future plans. In the provided benchmarks, Refexion demonstrates competitive success rates compared to other methods like ReAct and SayCan, though often at higher token expenses.

Key Equations and Algorithms

• Recency-Relevance Retrieval Score $Score=f(recency,relevance)$: Memories are retrieved based on a composite score derived from the recency of the memory and its relevance to the current situation, as utilized in Generative Agents [Park et al. , 2023].

• Q-Value Memory Tuple $Record=(environment,task,action,Q-value)$: REMEMBER [Zhang et al. , 2023a] formalizes memory storage as a tuple containing the environment state, the specific task, the action taken, and the resulting Q-value.

• Expense Calculation Logic $Expense\propto Token{s}_{consumed}$: The financial expenses (EX) are calculated based on the number of consumed tokens through the API provider, specifically referencing OpenAI’s API rates in Table 2.

• Context Storage Abstraction $Storag{e}_{total}=RA{M}_{context}+Dis{k}_{external}$: MemGPT abstracts the LLM context as RAM and treats the additional storage structure as a disk, allowing the system to manage $Storag{e}_{total}$ dynamically.

• PEFT Parameter Update Rule ${\theta }_{new}={\theta }_{base}+\Delta {\theta }_{small}$: Parameter-efficient fine-tuning techniques like LoRA or QLoRA update only a small part of the parameters ($\Delta {\theta }_{small}$) while leaving the base model parameters (${\theta }_{base}$) largely unchanged to reduce cost.

• Success Rate Metric $SR=\frac{N_{success}}{N_{total}}\times 100$: Table 2 reports Success Rate (SR) as a percentage, representing the proportion of tasks completed successfully out of the total attempts.

• Average Rewards Metric $AR=\frac{1}{N}{\sum }_{i=1}^N{R}_i$: Average Rewards (AR) is computed as the mean of the reward values obtained across $N$ evaluation runs or episodes within the benchmark environments.

• Memory Update Decision Algorithm The LLM in MemGPT autonomously executes a decision procedure determining whether to $Retrieve(historical\_memories)$ or $Save(current\_context,storage)$ based on the current context requirements and capacity constraints.

Key Claims and Findings

• RAG vs. Fine-Tuning Trade-off RAG-based methods offer real-time, low-cost external memory updates primarily in natural language but rely heavily on the accuracy of the retrieval algorithm.

• Parameter Modification Capacity Finetuning provides a significantly larger memorization capacity through parameter modifications but incurs high memory update costs and struggles with retaining fine-grained details.

• Memory Generation Dependency Memory-enhanced LLM-Agents demonstrate enhanced growth and fault tolerance, yet the quality of memory generation is strictly dependent on the underlying LLM’s generation capabilities.

• Improvement on Unseen Tasks AgentTuning [Zeng et al. , 2023] and CALM [Yao et al. , 2020b] show that organized plan trajectories can significantly improve performance on unseen planning tasks through fine-tuning.

• Environment Complexity Hierarchy Text-based interactive environments like ALFWorld are simpler than Minecraft, featuring straightforward feedback and fewer feasible actions, which influences the choice of benchmarking strategy.

• Reflexion Performance In the provided evaluation table, Refexion achieves high success rates (e.g., 220.17% relative cost on AlfWorld, 0.71 SR) often at the expense of higher token consumption compared to baseline CoT methods.

• PEFT Efficiency Parameter-efficient fine-tuning techniques are leveraged to reduce the cost and speed up the training of experiential samples, making embodied memory feasible for large-scale parameters.

• Retrieval Query Mechanism Retrieval systems consistently utilize the description of the current status as a query to retrieve memories from the memory pool, ensuring contextual alignment.

Terminology

• SR (Success Rate): A metric denoting the percentage of tasks where the agent successfully completes the objective, reported in Table 2 for various benchmarks.
• AR (Average Rewards): A metric representing the mean reward value obtained by the agent during interactions with the environment, used to evaluate efficiency in ScienceWorld and AlfWorld.
• EX (Expenses): A metric calculated based on the number of consumed tokens through the API provider, indicating the computational and financial cost of agent operation.
• Z-CoT (Zeroshot-CoT): A prompt-based method where the model generates a chain of thought without providing few-shot examples, represented in the evaluation table.
• F-CoT (Fewshot-CoT): A prompt-based method where the model is provided with several examples of reasoning chains prior to generating its own solution.
• PEFT (Parameter-Efficient Fine-Tuning): Techniques such as LoRA and QLoRA used to adapt large models by training only a small subset of parameters rather than the entire network.
• LoRA (Low-Rank Adaptation): A specific PEFT technique used to train a small part of parameters to adapt the model for specific tasks or memories.
• REMEMBER: A memory system that stores historical memories in the form of a Q-value table tuple to facilitate plan generation based on similarity.
• Embodied Memory: The process of finetuning the LLM with historical experiential samples to embed memories directly into the model parameters.
• Q-value Table: A data structure consisting of (environment, task, action, Q-value) tuples used to store positive and negative memories for retrieval.
• Interactive Gaming Environments: Benchmarks like Minecraft or ScienceWorld that provide real-time multi-modal feedback based on the agent’s actions for evaluation purposes.
• FAISS: A library used to establish an indexing structure for vector-encoded memories, enabling efficient retrieval based on description queries.