AI ABC: 26 Things to Know Before You Benchmark an LLM

People keep benchmarking AI systems with the confidence of physicists and the experimental design of someone timing a microwave.

So here is the AI ABC: 26 concepts, misconceptions, and failure modes worth understanding before drawing conclusions from model size, VRAM usage, benchmark scores, RAG demos, or a single response that "looked pretty good."

This is not a glossary. It is mostly a list of ways perfectly reasonable technical words get turned into complete nonsense.

A — Accuracy is task-specific#

Common misconception: A model with a higher benchmark score is simply "better."

What actually matters: Accuracy only has meaning relative to a particular task, dataset, metric, and evaluation procedure. A model can outperform another on coding and lose badly on retrieval, summarization, instruction following, or domain-specific work.

B — Benchmark ≠ reality#

Common misconception: A benchmark score tells us how well the model will perform in production.

What actually matters: A benchmark measures performance under its own experimental conditions. Production introduces different inputs, distributions, latency constraints, context lengths, users, integrations, and failure modes.

C — Context is not knowledge#

Common misconception: A model with a 128k-token context window effectively "knows" everything placed inside it.

What actually matters: Context is temporary input available during inference. Models may fail to retrieve or use information buried in long contexts, and useful context length can be substantially smaller than the advertised maximum.

D — Dataset defines the experiment#

Common misconception: "Test the models on RAG" is a sufficiently defined task.

What actually matters: Without a specified corpus, query set, relevance labels, expected answers, and evaluation method, there is no reproducible experiment. There is merely a demo.

E — Evaluation requires a metric#

Common misconception: We can compare models by reading several outputs and deciding which one "looks better."

What actually matters: The metric must correspond to the property being tested: correctness, retrieval recall, factuality, latency, exact match, human preference, cost, or something else. Different metrics answer different questions.

F — FLOPs are not memory#

Common misconception: A model using less memory must also be faster.

What actually matters: Memory capacity and compute throughput are separate constraints. Two models with similar memory requirements can require very different amounts of computation.

G — GPU fit does not imply GPU performance#

Common misconception: If the model fits on a GPU, that GPU is suitable for running it.

What actually matters: Usable inference depends on compute throughput, memory bandwidth, precision, kernel support, batch size, context length, runtime implementation, and desired latency.

H — Hallucination is not a special failure mode#

Common misconception: Models normally retrieve facts and occasionally malfunction by "hallucinating."

What actually matters: A language model generates likely continuations. Factual correctness is an additional property that must be established through training, context, retrieval, tools, or verification.

I — Inference has phases#

Common misconception: "Tokens per second" fully describes model performance.

What actually matters: Prompt processing and autoregressive generation behave differently. Time to first token, prompt-processing throughput, generation throughput, and end-to-end latency should be measured separately.

J — Judge models need judging too#

Common misconception: Using another LLM to score answers makes evaluation objective.

What actually matters: LLM judges have biases, preferences, blind spots, and model-dependent failure modes. Their agreement with human or task-specific ground truth needs validation.

K — KV cache consumes memory#

Common misconception: If the model weights fit in VRAM, the model fits.

What actually matters: Autoregressive inference stores keys and values for previous tokens. KV-cache memory grows with sequence length, batch size, number of layers, KV heads, head dimensions, and representation precision.

L — Latency ≠ throughput#

Common misconception: A server generating many tokens per second means individual users will receive fast responses.

What actually matters: Throughput describes total work completed; latency describes how long one request waits. Batching can improve throughput while making individual requests slower.

M — Model size is not capability#

Common misconception: More parameters means a better model.

What actually matters: Architecture, training data, training compute, optimization, distillation, post-training, tool use, retrieval, and task distribution all matter. Parameter count is one variable, not a quality score.

N — Numbers need baselines#

Common misconception: "This model answered in 12 seconds" is useful benchmark data.

What actually matters: A result needs hardware, runtime, precision, model version, prompt length, output length, context length, batch size, sampling parameters, warm/cold state, and a baseline for comparison.

O — Output quality is multidimensional#

Common misconception: There is one thing called "answer quality."

What actually matters: Correctness, completeness, factuality, instruction following, formatting, reasoning reliability, style, safety, and usefulness can change independently.

P — Prompting is part of the experiment#

Common misconception: The prompt is merely an interface to the model.

What actually matters: Prompt structure, examples, instructions, system messages, tool definitions, and output constraints can substantially affect results. A benchmark must control or document them.

Q — Quantization is a trade-off#

Common misconception: A 4-bit model is basically the same model but smaller.

What actually matters: Quantization changes numerical representation and can affect speed, memory use, accuracy, supported kernels, hardware compatibility, and sometimes particular capabilities disproportionately.

R — RAG is a pipeline#

Common misconception: "Add RAG" means connect the model to some documents.

What actually matters: Retrieval quality depends on document preparation, chunking, embedding model, indexing, query formulation, retrieval depth, filtering, reranking, context construction, and generation. Each component can fail independently.

S — Sampling changes the result#

Common misconception: Asking the same model the same question tests the same system.

What actually matters: Temperature, top-p, seed handling, decoding strategy, repetition penalties, and implementation details affect generated outputs and reproducibility.

T — Tokens are not words#

Common misconception: A 10,000-word document consumes roughly 10,000 context units.

What actually matters: Models process tokens, not words. Tokenization differs between models and languages, affecting context capacity, memory usage, cost, and inference time.

U — Use case comes first#

Common misconception: We should benchmark several models and then decide what they are useful for.

What actually matters: Define the actual workload first: input shape, required output, acceptable errors, latency, concurrency, privacy requirements, cost limits, and integration constraints. Then test models against it.

V — VRAM is not speed#

Common misconception: More VRAM means faster inference, or a model fitting in VRAM means it will run well.

What actually matters: VRAM primarily determines what can be stored. Performance depends on compute throughput, memory bandwidth, architecture, precision, kernels, batching, context length, and runtime efficiency.

This one deserves repetition because apparently it has to be tattooed onto people.

W — Warm-up changes timings#

Common misconception: The first execution time is representative performance.

What actually matters: Model loading, memory allocation, kernel compilation, caching, filesystem effects, and runtime initialization can make first-run timings dramatically different from steady-state performance.

X — "X times faster" requires a denominator#

Common misconception: "Model A is 3× faster than Model B" is a complete statement.

What actually matters: Faster at what? Prompt processing? Generation? Batch throughput? Which hardware, precision, runtime, context length, batch size, and output length? Without those, the multiplier is marketing, not measurement.

Y — Your benchmark can be wrong#

Common misconception: A reproducible test automatically produces a meaningful result.

What actually matters: You can reproducibly measure the wrong thing. Dataset leakage, bad metrics, unrealistic prompts, insufficient sample sizes, uncontrolled variables, and confounding factors can produce precise but meaningless numbers.

Z — Zero-shot is one configuration#

Common misconception: A model failing a plain prompt proves it cannot perform the task.

What actually matters: Zero-shot, few-shot, retrieval-augmented, tool-assisted, constrained, and fine-tuned systems are different operating configurations. Evaluate the configuration you actually intend to deploy.