Ch. 2 — 0: Foundational Concepts

Chapter 2 of NVIDIA DLI: Building Agentic AI Applications with LLMs

Abstract

This chapter establishes the theoretical boundaries and operational constraints inherent to Large Language Model (LLM) systems, fundamentally arguing that model behavior is dictated by training priors rather than prompt instructions. It introduces the “Fundamental Inequality” of LLM capabilities, demonstrating that input capacity significantly exceeds output capacity, a disparity that necessitates architectural strategies such as canvasing. Furthermore, the text analyzes the limitations of prompting as a control mechanism and defines the mechanics of emerging reasoning models. These foundational concepts are critical for designing agentic systems that align with the statistical realities of neural networks rather than attempting to override learned parameter distributions.

Key Concepts

• Training Priors Hierarchy: This concept defines a ranked ordering of influence on model behavior, ranging from Base Model Training as the strongest determinant down to the User Prompt as the weakest. The hierarchy is determined by the volume of parameter updates and exposure duration, where pre-training utilizes billions of tokens over months, while prompting provides only context without parameter modification. Consequently, prompting functions as a retrieval mechanism for existing capabilities rather than a method to instill new knowledge.
• Parameter Update vs. Context Weighting: This principle explains why training priors dominate over prompt instructions when conflicts arise. Training involves billions of parameter updates across diverse examples, permanently embedding statistical patterns, whereas prompting merely adjusts attention weights within a fixed parameter set. As a result, the model generates outputs by following these embedded statistical patterns, making it impossible to “convince” a model to act outside its training distribution.
• Leaky Abstractions in AI Systems: Derived from Spolsky’s law, this concept posits that LLM-based systems inevitably expose underlying complexity, forcing developers to understand implementation details. In the context of LLMs, abstractions leak through tokenization effects, context window limitations, and sensitivity to prompt wording. Users must account for distribution dependencies and output length constraints to prevent system failures that a perfect abstraction would hide.
• The Three Leaky Agents: The text categorizes agent specializations into Conversational, Reasoning, and Coding agents, each offering specific strengths alongside inherent leakage. Conversational agents suffer from context drift, Reasoning agents imitate logic without true computation, and Coding agents over-apply computational patterns to natural language. Effective system design requires combining these abstractions to mitigate individual weaknesses rather than relying on a single approach.
• The Fundamental Inequality of Capacity: This is the core relationship defining LLM operational limits, stating that Input Capacity is much greater than Understanding Capacity, which is in turn much greater than Output Capacity ($C_{input}\gg C_{understanding}\gg C_{output}$). This asymmetry arises because generating tokens is computationally expensive and quality degrades with length, while consuming text remains relatively efficient. This inequality drives the design choice of inputting full context but iterating on small output sections.
• Canvasing Strategy: Derived from the Fundamental Inequality, canvasing is a technique that exploits the asymmetry between input and output capabilities. It involves providing extensive document context as input while requesting modifications to only small sections of text to stay within high-quality output limits. This iterative process allows for the construction of long-form high-quality content without overwhelming the model’s generation constraints.
• Explicit Reasoning Patterns: This concept describes the mechanism by which reasoning models (e.g., DeepSeek-R1, OpenAI o3) are trained to output step-by-step logic before final answers. These models utilize tags to structure their internal processing, relying on pattern matching on reasoning examples rather than performing novel logical computations. The training data specifically includes step-by-step examples to condition the model’s probability distribution towards explicit deliberation.
• Reward Model Guidance in Training: A critical component of reasoning model optimization, this involves using a separate classifier to evaluate the quality of generated reasoning during the training phase. The model receives rewards for clear logic and penalties for inconsistencies, leveraging the asymmetry that it is easier to critique reasoning than to generate it. This guidance shapes the model’s priors to favor structured problem-solving pathways.
• Mixture-of-Experts (MoE) Activation: In reasoning models, MoE architecture allows different model components to specialize in distinct tasks, such as reasoning tokens versus answer tokens. This specialization means that specific internal pathways activate depending on whether the model is processing the thought trace or the final response. It enhances the model’s ability to handle complex reasoning steps without degrading its standard response capabilities.
• Prompt Sensitivity and Context Drift: This concept highlights the fragility of LLM systems where small wording changes in the prompt can cause large behavior changes. In conversational agents, this manifests as easy derailment or context drift over long interactions. Understanding this sensitivity is required to design robust systems that maintain coherence despite the leaky nature of the context window and prompt interface.

Key Equations and Algorithms

• The Fundamental Inequality of LLM Capabilities $S_{input}\gg S_{understanding}\gg S_{output}$ Where $S$ represents the capacity space (in tokens or semantic complexity). This expression quantifies the observation that an LLM can process significantly more input data than it can understand deeply, and can understand more than it can reliably output without quality degradation. It mathematically formalizes the constraint that motivates the canvasing technique.

• Training Prior Strength Hierarchy $W_{pre}>W_{SFT}>W_{RLHF}>W_{system}>W_{few}>W_{zero}$ Here $W$ denotes the weight of influence on model behavior, with subscripts representing Pre-training, Supervised Fine-Tuning, Reinforcement Learning from Human Feedback, System Prompts, Few-Shot Examples, and Zero-Shot User Prompts. This weighted ordering indicates the diminishing return of influence as one moves from parameter-altering training phases to context-only prompting phases.

• Reasoning Model Training Objective $P(\text{Reasoning})\leftarrow \text{RewardModel}(\text{Logic},\text{Inconsistency})$ This algorithmic relation describes the training loop where the probability of generating reasoning steps $P(\text{Reasoning})$ is adjusted based on feedback from a Reward Model. The model is trained to maximize rewards for clear logic while penalizing inconsistencies, effectively learning to emulate a critique process within its generation stream.

Key Claims and Findings

• Training priors consistently dominate over prompting instructions when the two are in conflict, because training updates billions of parameters while prompting only adjusts attention weights within a static model.
• Prompting is strictly limited to invoking existing capabilities embedded in the model and cannot create new knowledge or behaviors that were not learned during the training phase.
• LLM systems function as leaky abstractions that fail to hide complexity, requiring developers to manage failures related to tokenization, context limits, and prompt sensitivity rather than assuming perfect intelligence.
• The output capacity of modern LLMs is the most constrained dimension of their capability, primarily due to cost efficiency constraints and the increased probability of hallucinations during long generations.
• Reasoning models do not perform true logical computation but rather utilize learned patterns of explicit step-by-step reasoning to improve performance on multi-step logical problems.
• Designing effective agentic systems requires combining multiple leaky abstractions to cover each other’s weaknesses rather than relying on a single approach to handle all task types.

Terminology

• Training Priors: The statistical patterns, associations, and default behaviors learned by an LLM during its training phase, represented by the billions of parameter updates exposed to training data.
• Base Model Training: The initial pre-training phase involving exposure to billions of tokens over months of compute, which defines the core capabilities and creates the strongest behavioral priors.
• System Prompt: A persistent context provided across a conversation that exerts moderate influence on model behavior, ranking higher in the strength hierarchy than user prompts.
• Few-Shot Examples: Immediate context examples provided in the prompt to show patterns to the model, exerting a moderate-weak influence compared to training data.
• Leaky Abstraction: A system interface that fails to completely hide underlying complexity, forcing users to understand implementation details such as tokenization or context limits to use it effectively.
• Canvasing: An architectural strategy that exploits the input-output capability inequality by providing full document context but requesting modifications to small sections iteratively.
• Reasoning Models: LLMs specifically trained to generate explicit reasoning steps before providing final answers, often utilizing structured formats like tags to delineate thought processes.
• Out-of-Distribution (OOD) Inputs: Input data that deviates from the statistical patterns present in the training set, causing failure modes such as degraded performance or hallucinations.
• Context Window Limits: The maximum number of tokens a model can process simultaneously, which leads to forgetting or performance degradation when exceeded in long conversations.
• Attention Limits: The mechanism constraints within Transformer models that strain during very long generation sequences, contributing to the limitation on output quality and length.