Are Large Language Models In-Context Graph Learners?

Abstract

This work investigates the ability of off-the-shelf Large Language Models (LLMs) to perform graph learning tasks, specifically node classification on text-attributed graphs, via In-Context Learning (ICL) without fine-tuning. The authors establish a theoretical equivalence between message-passing Graph Neural Networks (GNNs) and Retrieval-Augmented Generation (RAG), showing that GNN propagation can be interpreted as recursive retrieval over the graph topology. Leveraging this insight, they propose three graph-guided RAG frameworks: QUERYRAG, which retrieves neighbor node features; LABELRAG, which retrieves neighbor labels; and FEWSHOTRAG, which retrieves feature-label pairs. Experiments demonstrate that standard ICL fails on graph tasks, but the proposed frameworks significantly improve LLM reasoning, with LABELRAG and FEWSHOTRAG enabling LLMs to match or exceed the accuracy of supervised GNNs and specialized fine-tuned graph LLMs.

Key Concepts

• In-context learning capabilities of LLMs applied to non-Euclidean graph-structured data.
• Message-passing Graph Neural Networks (GNNs) conceptualized as recursive Retrieval-Augmented Generation (RAG) processes.
• Graph-guided Retrieval-Augmented Generation strategies that utilize graph topology rather than external text corpora.
• QUERYRAG, LABELRAG, and FEWSHOTRAG as distinct retrieval-augmentation architectures for LLMs.
• Node classification on text-attributed graphs using one-hop neighbor information as inherent context.
• Performance parity between retrieval-augmented in-context LLMs and supervised/finetuned graph baselines.

Key Equations and Algorithms

• GNN Message-Passing Mechanism: $h_v^{l+1}=\text{UPD}(h_v^l,\text{AGG}(\{h_u^l|u\in \mathcal{N}(v)\}))$, describing how node representations are updated by aggregating messages from local neighbors.
• QUERYRAG Retrieval: $r_i=\text{LLM}(q_i,\{q_j|v_j\in \mathcal{N}(v_i)\})$, augmenting the LLM query with the textual attributes of immediate graph neighbors.
• LABELRAG Retrieval: $r_i=\text{LLM}(q_i,\{r_j|v_j\in \mathcal{N}(v_i)\})$, augmenting the query with ground-truth labels of neighbors, inspired by label propagation principles.
• FEWSHOTRAG Retrieval: $r_i=\text{LLM}(q_i,\{(q_j,r_j)|v_j\in \mathcal{N}(v_i)\})$, providing pairs of neighbor queries and labels to emulate few-shot in-context learning over graph structure.
• Label Propagation: $v_i^{t+1}=\alpha \text{softmax}(A{v}_i^t)+(1-\alpha )v_i^0$, the iterative update mechanism for node labels that motivates the LABELRAG framework design.

Key Claims and Findings

• Off-the-shelf LLMs are not inherently effective as in-context graph learners; zero-shot and standard few-shot performance often lags behind supervised MLPs, and few-shot examples can sometimes degrade performance.
• Message-passing GNNs perform recursive RAG over the graph structure, and utilizing graph connectivity for retrieval significantly enhances LLM performance compared to text-only retrieval methods.
• LABELRAG provides substantial accuracy gains by explicitly incorporating neighbor label correlations, frequently matching or surpassing supervised MLP baselines.
• FEWSHOTRAG enables general-purpose LLMs to match or exceed the node classification accuracy of state-of-the-art supervised GNNs (e.g., GCN, GAT) and fine-tuned graph LLMs on multiple datasets.
• Graph-based retrieval consistently outperforms random and text-based retrieval mechanisms, confirming that structural topology is a critical signal for LLM reasoning on graphs.
• LLM performance plateaus or declines when the number of retrieved neighbors $K$ becomes excessive (e.g., $K>3$), highlighting sensitivity to input length and noise in retrieved context.

Terminology

• QUERYRAG: A retrieval framework that augments an LLM query with the textual attributes of neighboring nodes in a graph structure.
• LABELRAG: A retrieval framework that augments an LLM query with the classification labels of neighboring nodes, leveraging label propagation concepts.
• FEWSHOTRAG: A framework that provides pairs of neighbor queries and labels as few-shot context to the LLM to guide reasoning.
• Graph-guided RAG: A retrieval strategy where the graph topology defines the set of retrieved contexts rather than a text similarity search over an external corpus.
• Text-attributed graph: A graph structure where nodes contain rich textual metadata that serves as the primary feature representation for learning tasks.
• Homophily: The assumption that connected nodes share similar attributes or labels; noted as a limitation of the proposed methods when applied to heterophilic graphs.

Connections to Existing Wiki Pages

• sec-08-rag
• how-to-make-your-llm-more-accurate-with-rag-and-fine-tuning
• ch-07-llm-limitations-and-context-management
• index