Ch. 5 — External Planner-Aided Planning

Chapter 5 of [[ai-ml/nvidia-certs/ncp-aai/cognition-planning-and-memory/Understanding-the-planning-of-LLM-agents-A-survey|Understanding the planning of LLM agents: A survey]]

Abstract

This chapter explores advanced mechanisms for enhancing Large Language Model (LLM) agents through external planning assistance, iterative reflection, and memory augmentation. It details the integration of neural planners and dual-process cognitive theories to optimize action selection and reasoning efficiency. Furthermore, it analyzes reflection strategies that mitigate hallucinations and the deployment of Retrieval Augmented Generation (RAG) to expand contextual knowledge. These techniques collectively address the limitations of autonomous planning in complex environments, paving the way for more robust and stable artificial intelligence systems.

Key Concepts

• CALM Neural Planning Framework: CALM represents an early integration of language models with reinforcement learning-based neural planners. In this architecture, a language model processes textual environmental information to generate candidate actions as priors. Subsequently, a DRRN policy network is employed to re-rank these candidate actions, ultimately selecting the optimal action for execution based on the prioritized list.

• SwiftSage Dual-Process Architecture: SwiftSage leverages dual-process theory from cognitive psychology to divide planning into slow and fast thinking subsystems. The fast-thinking process utilizes a DT model trained through imitation learning to generate rapid plans, while the slow-thinking process engages the LLM for complex reasoning upon error detection. This hybrid approach balances efficiency with rational deliberation, ensuring optimal performance across varying task complexities.

• External Planner Support Role: Within external planning strategies, the LLM primarily functions in a supportive role rather than acting as the sole decision-maker. Its critical functions involve parsing textual feedback and providing additional reasoning information to assist in planning, particularly when addressing complex problems. This division of labor enhances the system’s theoretical completeness and stability by combining statistical AI with symbolic reasoning.

• Iterative Reflection and Refinement: Reflection and refinement are indispensable components designed to enhance the fault tolerance and error correction capabilities of LLM-Agent planning. These mechanisms allow agents to correct errors and break out of “thought loops” that arise due to limited feedback or hallucination issues during complex problem solving. By summarizing failures, agents improve subsequent planning attempts through accumulated experience.

• Self-Refine Mechanism: The Self-refine strategy utilizes an iterative process of generation, feedback, and refinement to improve output quality. After each plan generation, the LLM generates feedback for the plan, facilitating adjustments based on the feedback within the same context. This process creates a closed loop where the model continuously optimizes its own reasoning without requiring external tool intervention.

• CRITIC External Validation: CRITIC employs external tools like Knowledge Bases and Search Engines to validate LLM-generated actions against factual information. It then leverages external knowledge for self-correction, significantly reducing factual errors that typically plague LLM reasoning. This integration ensures that the agent’s plan is grounded in verifiable data rather than internal model parameters alone.

• RAG-based Memory Retrieval: Retrieval Augmented Generation techniques are utilized to aid text generation with retrieved information from a memory store. For LLM agents, past experiences are stored in additional storage and retrieved during task planning to inform current decisions. This method enhances the LLM with the latest knowledge and context, preventing information stagnation.

• Memory-Augumented Planning Strategies: For agents, memory serves as a crucial pathway to enhance planning capabilities and the potential for growth over time. There are currently two major approaches to enhance planning abilities through memory, specifically RAG-based memory and embodied memory. These mechanisms allow the agent to maintain continuity across multiple tasks and learning episodes.

Key Equations and Algorithms

• CALM Action Selection Algorithm: The procedure involves generating candidate actions via a language model, then re-ranking them using a DRRN policy network. The computational process prioritizes actions based on environmental textual information before final selection, optimizing the policy through neural re-ranking.

• SwiftSage Process Switching Logic: The algorithm initiates with a fast-thinking DT model for rapid plan generation. If errors occur during execution, the system switches to a slow-thinking LLM process for detailed reasoning, creating a conditional branching logic based on execution feedback.

• Self-Refine Iterative Loop: This algorithmic process cycles through three stages: generation of the initial plan, generation of feedback on that plan, and refinement of the plan based on the feedback. The loop terminates when the LLM determines no further adjustments are necessary or a maximum iteration count is reached.

• Reflexion Evaluation Procedure: Extending ReAct, this procedure incorporates an evaluator to assess the full trajectory of the agent’s actions. Upon error detection, the LLM generates self-reflections which are stored as memory for future reference, aiding in error correction without explicit parameter updates.

• InteRecAgent ReChain Mechanism: An LLM is used to evaluate the response and tool-using plan generated by an interactive recommendation agent. The system summarizes feedback on errors and decides whether to restart the planning process, creating a chain of verification before action commitment.

• RAG-based Memory Retrieval Logic: The core procedure involves retrieving task-relevant experiences from non-volatile storage during task planning phases. These retrieved memories are appended to the context window, allowing the LLM to condition its generation on historical data points.

Key Claims and Findings

• The enhancement of LLM’s capabilities in code generation empowers the potential to deal with more general tasks for symbolic artificial intelligence.

• A significant drawback of traditional symbolic AI systems lies in the complexity and heavy reliance on human experts in constructing symbolic models.

• The combination of statistical AI with LLM is poised to become a major trend in the future development of artificial intelligence.

• Reflection strategies significantly reduce factual errors and help agents get unstuck from “thought loops” during complex problem solving.

• In LLM agents, policy updates occur through self-reflection via textual feedback rather than modifying model parameters as in deep reinforcement learning.

• The convergence of textual form of updates currently lacks a guaranteed proof regarding whether continual reflection leads the agent to a specified goal.

• Past experiences stored in memory can be retrieved when needed to enhance task planning performance.

Terminology

• DRRN Policy Network: A neural network component used within the CALM framework to re-rank candidate actions generated by a language model.

• Slow-Thinking Process: A planning phase involving complex reasoning and rational deliberation handled by the LLM, activated when fast thinking fails.

• Fast-Thinking Process: An instinctive response mechanism resembling long-term training, utilized by the SwiftSage architecture for rapid plan generation.

• DT Model: A model trained through imitation learning that serves as the fast-thinking model within the SwiftSage dual-process framework.

• Symbolic Artificial Intelligence: A paradigm characterized by theoretical completeness, stability, and interpretability, which LLMs assist in constructing more rapidly.

• Thought Loops: A state where LLM-Agents get stuck during planning due to limited feedback or insufficient reasoning abilities for complex problems.

• Self-Reflection: The mechanism by which an LLM generates self-reflections upon error detection to aid in future error correction.

• ReChain Mechanism: A self-correction mechanism employed by InteRecAgent to evaluate responses and summarize feedback on errors before restarting planning.

• RAG-based Memory: A memory architecture based on Retrieval Augmented Generation that stores experiences for retrieval during task planning.

• Embodied Memory: One of the two major memory approaches to enhance planning abilities in LLM-Agents, alongside RAG-based memory.