LLM Glossary —
every word AI uses.

The mechanism that lets each token influence every other token when computing representations. Self-attention is the core building block of transformer models — it's why LLMs can relate 'it' to its antecedent across hundreds of tokens.

An LLM configured to take actions — calling tools, browsing the web, writing files — and loop until a goal is achieved. Agents observe results and plan next steps autonomously rather than producing a single static response.

The repeating cycle of: reason → pick action → execute tool → observe result. Continues until the task is complete or a stopping condition is met. The core execution model for autonomous AI agents.

The research and engineering effort to make a model's behavior match human values and intentions. Misalignment means the model optimizes for what it was trained on, not what you actually want.

The generation mode where each new token is predicted from all previous tokens. Modern LLMs are autoregressive — they cannot look ahead during generation. One token at a time, left to right.

Previously mentioned. LLMs use this to refer back to something earlier in a response: 'the aforementioned constraints'. Formal, often unnecessary — 'the above' or 'this' usually works fine.

Although; even though. LLMs often reach for 'albeit' to introduce a concession: 'a useful approach, albeit one with tradeoffs'. A more formal alternative to 'even though' or 'though'.

To express clearly and coherently. LLMs use it as a verb ('let me articulate the key points') and as an adjective ('an articulate explanation'). Signals precision — that the output is intentionally structured.

To find out or determine definitively. LLMs use it in reasoning: 'to ascertain whether this is true…'. Stronger than 'find out' — implies investigation rather than guessing.

A generation strategy that keeps k candidate sequences ('beams') in parallel, scoring and pruning them at each step. Produces more coherent text than greedy decoding at the cost of speed and memory.

A standardized test used to compare model capability. Examples: MMLU (knowledge breadth), HumanEval (code), MATH (math reasoning), SWE-bench (software engineering), GPQA (expert-level science).

Returning to an earlier point in a reasoning chain to revise it. In thinking traces: 'backtracking here — my earlier assumption was flawed'. Signals intellectual honesty — the model has caught its own error and is correcting course.

Splitting into multiple parallel paths or possibilities. LLMs use it when exploring alternatives: 'branching into two approaches here'. Borrowed from tree structures — each branch is a distinct direction the reasoning could go.

A prompting technique where the model reasons step-by-step before giving a final answer. Adding 'let's think step by step' measurably improves accuracy on multi-step problems. Extended thinking is an automated, hidden form of CoT.

Anthropic's training technique where models are guided by a written set of principles (a 'constitution'). The model critiques and revises its own outputs against the constitution — reducing reliance on human feedback for every response.

When a model loses coherent access to early parts of a long conversation. The information is technically in the window, but the model's behavior suggests it's no longer attending to it effectively — especially common past 100K tokens.

The maximum number of tokens a model can process in one request — input plus output combined. Claude 3.5: 200K. GPT-4o: 128K. Gemini 1.5 Pro: 1M. Larger windows allow longer documents and conversations without truncation.

A synonym for hallucination borrowed from neuropsychology. The model generates confident, plausible-sounding but factually incorrect content — not intentionally deceptive, just statistically probable noise.

To cause or accelerate a process. LLMs use it metaphorically: 'this insight can catalyze better decision-making'. Borrowed from chemistry — a catalyst speeds up a reaction without being consumed.

Complete; covering all aspects. One of the most overused LLM words — 'a comprehensive overview', 'comprehensive analysis'. Often appears where 'full' or 'complete' would be cleaner.

To think carefully about something. LLMs reach for it when describing deliberate reasoning: 'let us contemplate the implications'. More contemplative (!) than 'consider' or 'think about'.

Falling or flowing in sequence, each triggering the next. LLMs use it for chain reactions: 'cascading failures', 'cascading effects'. Borrowed from waterfalls — implies a sequence that, once started, is hard to stop. Common in reasoning about dependencies.

Carving away to reveal or refine an essential form. LLMs use it metaphorically in editing and distillation: 'chiseling the argument down to its core'. Implies sculpture — the final form was already there, excess just needed removing.

Returning to an earlier point after exploring a detour. Very common in thinking traces: 'circling back to the original constraint…'. Implies the model took a tangent and is now re-anchoring. Honest signal that the reasoning was non-linear.

Gradually coming together to form a unified whole. LLMs use it when separate threads of reasoning converge: 'these ideas are coalescing into a clear picture'. Implies a natural, organic merging — not forced assembly.

Moving toward a common point from different directions. LLMs use it when multiple lines of evidence or reasoning arrive at the same conclusion: 'the evidence is converging on a single explanation'. Contrast with diverging.

Training a smaller 'student' model to mimic the outputs of a larger 'teacher' model. The student learns the teacher's full probability distributions, not just correct/incorrect labels — preserving more nuance than supervised fine-tuning alone.

To arrive at a conclusion through logical reasoning from premises. LLMs use it to signal inferential steps: 'from this, we can deduce that…'. Stricter than 'infer' — deduction is logically guaranteed if premises hold.

To describe or portray something precisely. LLMs use it when drawing boundaries or laying out structure: 'let me delineate the three main approaches'. Implies more precision than 'describe' or 'outline'.

To search deeply into a subject. Perhaps the single most stereotypically LLM word — 'let's delve into this topic'. Arguably the clearest signal that text was written by an AI. 'Let's explore' or 'let's look at' are more natural alternatives.

To perceive or recognize something clearly, especially something subtle. LLMs use it to signal careful observation: 'we can discern a pattern here'. Stronger than 'see' — implies that the recognition required effort.

Extracting the essential essence by removing what is impure or unnecessary. LLMs use it for summarization and core extraction: 'distilling the key insight', 'distilling decades of research into one principle'. Implies the output is purer and more concentrated than the input.

Spreading apart from a common point into different directions. LLMs use it when exploring multiple possibilities: 'the approaches diverge here'. Contrast with converging. Common in reasoning traces when the model is opening up the solution space.

A dense numeric vector representing text in semantic space. Similar meanings produce similar vectors — the foundation of semantic search and RAG. A sentence becomes a point in ~1,536-dimensional space; arithmetic on embeddings encodes meaning.

A mode where the model uses a hidden scratchpad to reason before producing its visible response. Claude 3.7+ supports this. Thinking tokens are invisible in the output but still cost tokens and significantly improve accuracy on hard problems.

To make something clear; to explain. A distinctly formal synonym for 'explain' that LLMs reach for often: 'allow me to elucidate'. Preferred in academic writing; can sound stilted in conversational contexts.

To be a concrete expression of an abstract quality. LLMs use it to connect abstract principles to specific examples: 'this design embodies simplicity'. More vivid than 'represent' — implies the thing is the living form of the quality.

To express the essential features of something in a short form. LLMs use it when summarizing: 'to encapsulate the argument…'. Slightly more precise than 'summarize' — implies capturing the core without losing fidelity.

An earnest attempt to achieve a goal. LLMs use both the noun ('a noble endeavor') and verb ('I will endeavor to explain'). Formal and slightly archaic — 'try' or 'attempt' is almost always clearer.

To list items one by one. LLMs use it to signal structured output: 'let me enumerate the steps'. More explicit than 'list' — implies a complete, ordered accounting rather than a casual rundown.

To explain or develop in detail. LLMs use it when elaborating at length: 'allow me to expound on this'. Heavier than 'explain' — implies comprehensive treatment of a subject.

To extend known data or reasoning into unknown territory. LLMs use it in analytical contexts: 'extrapolating from these results…'. Implies going beyond the given evidence — which is either insight or speculation depending on the leap.

Providing 2–10 examples in the prompt before asking the model to complete a task. The examples implicitly teach output format, reasoning style, or domain knowledge — without any weight updates to the model.

Further training a pre-trained model on a specific dataset to adapt it for a domain, task, or style. Updates model weights — more powerful than prompting for consistent behavior, but requires quality labeled data.

The ability for an LLM to request execution of defined functions (tools) and receive the result in structured JSON. The model decides when to call a tool; your code decides whether to execute it. Foundation of agentic systems.

To make an action or process easier. A perennial LLM favorite: 'this approach facilitates better outcomes'. Often filler — 'helps' or 'enables' says the same thing with fewer syllables.

Immediately; without delay. LLMs sometimes reach for this archaic adverb: 'I will address this forthwith'. Extremely formal — most natural contexts prefer 'right away', 'now', or 'immediately'.

To encourage the development of something. LLMs use it in positive framing: 'this fosters collaboration', 'foster a culture of…'. Implies gentle, ongoing nurturing rather than a one-time push.

In addition to what has already been said. One of the LLM's most-used connectors — appears constantly in multi-point explanations. Slightly more formal than 'also' or 'additionally'; signals the continuation of an argument.

Connecting model outputs to verifiable, real-world sources — via RAG, web search, or tool results. Grounded answers cite sources; ungrounded answers rely on parametric memory and are more prone to hallucination.

Selecting the single highest-probability token at every generation step. Deterministic and fast, but prone to repetition and stuck local optima. Temperature > 0 adds randomness; beam search explores multiple paths.

Moving in strong, sudden bursts — like wind. LLMs use it metaphorically to describe forces or ideas sweeping through something powerfully: 'gusting winds of change', 'a gusting tide of innovation'. More atmospheric and vivid than 'strong' or 'surging'.

When a model generates confident-sounding but factually incorrect output. Common causes: the fact wasn't in training data, the model overgeneralizes, or conflicting training examples. Retrieval-augmented generation is the main mitigation.

To control and use a resource for maximum effect. LLMs use it extensively: 'harness the power of AI', 'harness this capability'. A very common LLM construction — implies taming something powerful and directing it productively.

From this point onward. A formal temporal marker LLMs use when establishing a new state or rule: 'henceforth, we will refer to this as…'. Archaic in casual writing — 'from now on' is more natural.

Before now; up to this point. LLMs use it to contrast a new state with the past: 'heretofore unexplored territory'. Distinctly formal — most readers encounter it primarily in AI-written text.

Considering the whole system rather than its individual parts. LLMs use it to signal broad, systems-level thinking: 'a holistic approach to the problem'. Often appears where 'complete' or 'overall' would be more direct.

The model's ability to learn a new task from examples placed in the prompt — no weight updates required. The 'learning' happens entirely in the forward pass. Few-shot and zero-shot prompting are both forms of in-context learning.

Running a trained model to generate output. Distinct from training (which updates weights). Inference is what happens when you call the API — your input goes in, tokens come out. Also called 'serving' at scale.

Fine-tuning a base model on (instruction, response) pairs so it follows natural language commands. Transforms a raw language model (next-token predictor) into a helpful assistant that responds to requests.

To make something clear or shed light on it. LLMs use it metaphorically: 'this example illuminates the concept'. A vivid alternative to 'clarify' or 'explain' — implies that understanding was obscured and is now made visible.

To the extent that; in so far as. A formal subordinating conjunction: 'inasmuch as this is true…'. Rarely used in everyday speech — its appearance in LLM output often signals formal register or legal-style reasoning.

To derive a conclusion from evidence or reasoning. LLMs use it to signal logical steps: 'from this we can infer…'. Distinguished from 'imply' — a speaker implies; a listener infers. Looser than deduce — inference can be probabilistic.

To the degree or extent that. LLMs use it to qualify scope: 'insofar as this applies to…'. Always followed by 'as' — 'insofar as X is true, Y follows'. Formal; 'to the extent that' is a cleaner alternative for most writing.

Very detailed and complicated. LLMs reach for it to signal complexity: 'an intricate web of dependencies', 'intricate mechanisms'. Implies beauty-in-complexity rather than mere complication.

Key-value cache. Stores intermediate attention computations from previous tokens to avoid recomputing them. Critical for fast inference on long contexts. Prompt caching in APIs is a user-facing version — repeated prefixes cost ~10% on a cache hit.

The high-dimensional vector space where embeddings live. Similar concepts cluster together; analogies exist as geometric relationships (king − man + woman ≈ queen). The model 'thinks' in latent space before projecting to token probabilities.

The empirically observed tendency of LLMs to better attend to content at the start and end of a long context than in the middle. Relevant documents placed in the middle of a 100K-token prompt are more likely to be ignored or misremembered.

To use something as a resource for maximum advantage. A top-tier LLM verb: 'leverage this capability', 'leverage existing infrastructure'. Extremely common in business-adjacent LLM output — use 'use' when you mean 'use'.

An open standard (by Anthropic, 2024) for connecting AI models to external tools, data sources, and services. Think USB-C for LLMs — one protocol that works with any compliant tool or host application. Enables tool discovery and invocation.

A model that processes multiple input types — text, images, audio, video — in a unified architecture. GPT-4o, Claude 3.5, and Gemini 1.5 are multimodal. A shared representation space lets the model reason across modalities.

Showing great attention to detail; very careful and precise. LLMs use it to characterize thorough work: 'meticulous analysis', 'meticulous craftsmanship'. Implies both care and accuracy — not just thoroughness, but correctness.

To reduce the severity, seriousness, or painfulness of something. LLMs use it in risk and tradeoff discussions: 'mitigate these risks', 'mitigate the impact'. Precise — it means lessen, not eliminate. 'Minimize' implies trying for zero; 'mitigate' does not.

In addition; besides that. LLMs use it to extend a line of reasoning: 'moreover, this approach has the advantage of…'. Slightly stronger than 'furthermore' — implies the additional point strengthens, not just adds to, the previous one.

Having many different and complex aspects. LLMs reach for it when acknowledging complexity: 'a multifaceted problem', 'multifaceted approach'. Implies rich dimensionality — more vivid than 'complex' or 'complicated'.

A countless or extremely large number. LLMs use it both as adjective ('myriad options') and noun ('a myriad of possibilities'). Signals abundance — but is so common in LLM output that it can feel like filler. 'Many' often works just as well.

Also called Top-p sampling. Picks the next token from the smallest set of candidates whose combined probability exceeds p. More adaptive than top-k across distributions. Top-p=0.95 is a common default for creative generation.

To plan a route through a complex environment. LLMs use it almost always metaphorically: 'navigate the complexity', 'navigate these tradeoffs'. Implies agency and skill — you don't just pass through something, you direct your path through it.

In spite of that; notwithstanding. LLMs use it to signal a contrast or concession: 'nevertheless, the approach has merit'. Heavier than 'however' or 'but' — implies overcoming a real objection, not just noting a qualification.

Despite; in spite of. LLMs use it as both preposition ('notwithstanding these challenges') and adverb ('notwithstanding, the conclusion holds'). Very formal — mainly appears in legal writing and AI output.

The learnable numerical values (weights and biases) that constitute a model's 'knowledge'. Capability roughly scales with count: GPT-4 ~1.8T, Claude 3 Opus ~200B, Llama 3 8B. More parameters = more capacity, not guaranteed quality.

A statistical measure of how surprised a model is by a text sequence. Lower perplexity = better language model. Used to compare models on identical test sets. Not directly correlated with downstream task quality or user preference.

The large-scale initial training phase where a model learns language from massive corpora (Common Crawl, books, code). Establishes base capabilities and world knowledge. Everything after this — instruction tuning, RLHF — is refinement.

The full input sent to an LLM in one request: system prompt + conversation history + current user message. The model treats everything in the prompt as prior context when predicting the next token.

Reusing previously computed KV-cache states for identical prompt prefixes. Available from Anthropic, OpenAI, and Google. Repeated system prompts cost ~10% of the original on a cache hit — major savings for high-volume or long-system-prompt applications.

An attack where malicious content in the model's context overrides its original instructions. E.g. a webpage with hidden text telling the model to ignore system instructions. A critical security risk in agentic systems that process untrusted data.

More important than anything else. LLMs use it to signal top priority: 'of paramount importance', 'paramount concern'. Superlative — if everything is paramount, nothing is. Usually replaceable with 'most important' or 'critical'.

To walk through or around; to traverse at a leisurely pace. In LLM reasoning, sometimes used metaphorically: 'perambulating through the options'. A notably unusual choice — its presence often signals a model reaching for formal or literary vocabulary.

To spread throughout something. LLMs use it to describe pervasive influence: 'this principle permeates the entire codebase'. Implies deep, thorough penetration — not just spread on the surface.

Of crucial importance; a decisive turning point. LLMs use it frequently: 'a pivotal moment', 'pivotal role'. From 'pivot' — suggests something the rest of the situation turns on. Often interchangeable with 'key' or 'crucial', but more dramatic.

An excess or overabundance of something. LLMs use it loosely to mean 'many': 'a plethora of options'. Technically implies too many — more than is useful. In practice, used as a fancier 'many'. The distinction is mostly lost in modern usage.

To assume as true for the purpose of reasoning; to put forward as a theory. LLMs use it to introduce hypotheses: 'we can postulate that…'. More formal than 'assume' or 'suppose' — implies a reasoned starting point, not an arbitrary guess.

Filtering slowly through; developing gradually. LLMs use it for ideas that need time to form: 'the solution is percolating', 'letting this percolate before committing'. Borrowed from coffee — the idea is brewing slowly, becoming stronger and clearer.

Reinforcement Learning from Human Feedback. Humans rank model outputs; a reward model learns their preferences; the LLM is fine-tuned via RL to maximize the reward. Used by GPT-4, Claude, and Gemini to align behavior.

Augmenting LLM generation with relevant documents retrieved from an external database at query time. The model only sees what you retrieve — no memorization of dynamic data needed. Core technique for knowledge-grounded AI applications.

The step-by-step thinking process a model produces before its final answer. Either visible (chain-of-thought in the output) or hidden (extended thinking). Longer traces generally improve accuracy on complex, multi-step problems.

A second-pass scoring step after retrieval. The retriever returns many candidates; the reranker (often a smaller cross-encoder model) scores each for relevance to the query and selects the top N to pass to the LLM.

Strong and effective in a wide range of conditions. A very common LLM adjective: 'a robust solution', 'robust framework'. Implies resilience and reliability under stress. Often used where 'solid', 'strong', or 'reliable' would be more direct.

The mechanism inside a transformer where each token computes attention scores against all others and uses them to build its contextual representation. Scales as O(n²) with sequence length — why long contexts are computationally expensive.

Finding documents by meaning (via embedding similarity) rather than keyword overlap. 'Best local restaurants' returns results for 'great nearby dining spots' even without the exact words. Contrast with BM25/TF-IDF keyword search.

Training on labeled (input, output) pairs using standard cross-entropy loss. The simplest fine-tuning form. Often the first alignment step — teach the model the right format before optimizing for quality with RLHF.

Delivering model output token-by-token as it's generated, rather than waiting for the complete response. Dramatically improves perceived latency. Most LLM APIs support Server-Sent Events (SSE) for streaming output.

Instructions given to the model before the user's message, typically via a distinct role in the API. Defines persona, constraints, output format, and task context. Usually invisible to end users in production applications.

To make a process more efficient by removing unnecessary steps. LLMs use it in process and product contexts: 'streamline the workflow', 'streamline development'. Implies making something sleeker — fewer steps, same or better outcome.

Coming after something in time or order. LLMs use it as a temporal connector: 'subsequently, the process completes…'. More formal than 'then' or 'after that' — signals a deliberate sequencing of events.

To combine separate elements into a coherent whole. LLMs use it for integration tasks: 'synthesize these findings', 'synthesize information from multiple sources'. Implies active combination — creating something unified from disparate parts.

Building a temporary structure to support construction of something larger. LLMs use it for intermediate reasoning steps: 'scaffolding the argument before the conclusion', 'this scaffolding helps build toward the answer'. Implies the structure itself may not appear in the final output.

Cooking gently just below boiling — developing slowly without breaking surface. LLMs use it for tension or potential building quietly: 'a simmering contradiction in the logic', 'the complexity has been simmering throughout'. Implies something about to surface.

Moving in tightening or widening coils — either converging inward or expanding outward. LLMs use it for recursive reasoning or escalating complexity: 'spiraling into edge cases', 'thoughts spiraling outward'. Context determines whether spiraling is productive or out of control.

Gaining perspective by temporarily disengaging from the detail. Very common in thinking traces: 'stepping back — what's the actual goal here?'. Signals the model is zooming out from a micro-problem to re-examine the macro context.

Moving forward powerfully and suddenly. LLMs use it for sudden increases or forward momentum: 'confidence surging here', 'the argument surges toward a conclusion'. More dramatic than 'increasing' — implies a wave of force.

A hyperparameter (0–2) controlling output randomness. 0 = always pick the most likely token (deterministic). 1 = sample proportionally from probabilities. Higher = more creative and varied; lower = more focused and repeatable.

The smallest unit of text an LLM processes. ~4 characters or ¾ of a word in English. 'Cat' is typically one token; a word like 'unprecedented' may be three. All pricing, context limits, and generation speed are measured in tokens.

Converting text into tokens before passing it to a model. Each model family uses its own tokenizer algorithm (BPE, SentencePiece, etc.). The same text produces different token counts in different models — always check with the target tokenizer.

The ability of an LLM to call external functions, APIs, or services and incorporate the results into its response. Synonymous with function calling. Enables agents to access real-time data, execute code, search the web, or interact with databases.

A sampling strategy that restricts the next token choice to the k highest-probability candidates. Top-k=50 means the model only samples from 50 possible next tokens — preventing extremely improbable choices while allowing variation.

Also called nucleus sampling. Restricts sampling to the smallest set of tokens whose combined probability exceeds p. Top-p=0.95 covers 95% of the probability mass. More adaptive than top-k across different vocabulary distributions.

The neural network architecture underlying virtually all modern LLMs. Introduced in 'Attention is All You Need' (Vaswani et al., 2017). Key innovation: replace recurrence with self-attention, enabling massive parallelization during training.

A complex combination of elements forming a unified whole. LLMs use it almost exclusively as a metaphor: 'a rich tapestry of ideas', 'the tapestry of human experience'. A strong signal of LLM-generated prose — human writers reach for it less often.

Causing a major change or fundamental shift. LLMs use it in superlative framing: 'transformative technology', 'transformative impact'. Implies deep, lasting change — not just improvement, but a shift in kind. Often over-applied to incremental changes.

Passing carefully through a narrow or complex path. LLMs use it for navigation through constraints: 'threading through the edge cases', 'threading a path between competing requirements'. Implies precision — the path is tight and must be followed exactly.

Following a path carefully to understand its origin or course. LLMs use it in debugging and causal analysis: 'tracing the error back to its source', 'tracing the logic forward'. Implies deliberate, step-by-step path following rather than jumping to conclusions.

Moving across or through a space systematically. LLMs use it in technical contexts (traversing a tree, a graph) and metaphorically: 'traversing the problem space', 'traversing possibilities'. More methodical than 'walking through' — implies complete coverage.

To emphasize or draw attention to something. LLMs use it as a verb: 'this underscores the importance of…'. More vivid than 'emphasize' or 'highlight' — implies drawing a line under a key point. Appears very frequently in analytical LLM output.

Discovering something hidden by digging it out. LLMs use it for revealing non-obvious insights: 'unearthing the root cause', 'unearthing assumptions buried in the question'. Implies active excavation — the thing was there all along, just not visible.

Breaking open a dense concept to examine its contents. One of the most common LLM reasoning-trace words: 'let me unpack this', 'unpacking the assumption'. Implies a container holding multiple components that need to be laid out and examined individually.

The learnable numerical parameters of a neural network — the model's encoded knowledge. 'Loading a model' means loading its weights into GPU memory. Training updates weights via gradient descent; inference reads them without changing them.

In which; in what. A formal relative pronoun LLMs use in technical or legal-adjacent writing: 'a scenario wherein both conditions hold'. Very rarely used in spoken English — its presence in output often signals formal or over-engineered language.

A hard stop in a reasoning trace — the model catching an error or contradiction: 'wait, that contradicts what I said earlier'. The single most reliable indicator of genuine self-correction in LLM thinking. When 'wait' appears, a real reconsideration is underway.

Asking a model to complete a task with no examples in the prompt — relying entirely on its pre-trained knowledge and instruction-following ability. Modern large models are surprisingly capable zero-shot on well-defined tasks.