How LLM Inference Works: A Practical Guide to Serving and Optimizing Large Language Models

Large Language Models (LLM) inference is becoming an art of its own.

Companies are increasingly looking for LLM inference and MLOps engineers who can serve models faster, use less hardware, and reduce inference costs. 

Whether you are running models locally, deploying an inference server, or experimenting with open-source LLMs, understanding how inference works can make a huge difference.

Behind efficient LLM inference is an entire stack of techniques, including quantization, KV-cache optimization, continuous batching, optimized attention kernels, speculative decoding, and hardware-aware tuning. 

Many modern inference frameworks enable some of these automatically, which means you may already be benefiting from them without fully understanding what is happening under the hood.

In this guide, I will explore the fundamentals of LLM inference, the most important performance metrics, popular serving frameworks, and practical techniques for making models faster, more memory-efficient, and cheaper to run.

What is LLM Inference?

We will start with the basics: what inference is, how inference works, how performance is measured, and why model architecture matters.

LLM inference is what happens every time you send a prompt to a language model and wait for a response. The model processes the input through its layers and then generates the output token by token.

This is different from training. During training, the model learns from data and updates its weights. During inference, those weights are fixed, and the trained model is simply used to generate predictions.

Inference is where the vast majority of LLM compute cost occurs in production. 

A model might be trained once, but it handles millions of requests. That makes inference efficiency one of the most consequential engineering problems in applied AI right now.

Understanding how inference works at a technical level matters because the bottlenecks are non-obvious. 

Faster hardware does not always mean faster responses. Larger models are not always slower. The architecture of the model, the length of the input, the batch size, and the serving infrastructure all interact in ways that are worth understanding before you start optimizing.

How does LLM inference work?

For most decoder-based LLMs, inference consists of two main phases: prefill and decode.

• During prefill, the model processes all tokens in the input prompt and creates the initial key-value, or KV, cache. Since many input tokens can be processed in parallel, this stage is generally compute-intensive.
• The decode stage begins when the model starts producing output. LLMs are usually autoregressive, meaning each new token depends on previously generated tokens. The model repeatedly performs another forward pass to generate the next token.

At smaller batch sizes, decoding is often limited more by memory bandwidth than raw GPU compute because model weights and cached attention states must continually be read from memory.

This distinction matters because an optimization that improves prefill performance may not necessarily improve decoding performance.

How do we measure LLM inference?

There is no single metric that tells us whether an inference server is fast, efficient, or cost-effective. These metrics help identify bottlenecks, compare serving setups, and determine whether an optimization is actually improving performance.

Some of the most important metrics include:

• Time to First Token (TTFT): How long it takes before the first token appears.
• Inter-Token Latency (ITL): The delay between generated tokens.
• Time per Output Token (TPOT): The average time required to generate each output token.
• Tokens per second: How quickly tokens are processed or generated.
• Throughput: The total number of tokens or requests processed over time.
• Concurrency: How many requests the server can handle efficiently at once.
• P50/P95/P99 latency: Shows typical performance as well as slower edge cases.
• Goodput: Useful throughput that still meets the required latency target.
• GPU utilization: How effectively the available hardware is being used.
• Cache hit rate: The percentage of input tokens or prefixes that can reuse previously computed KV-cache data instead of being processed again.
• Input cost: The cost of processing prompt and context tokens.
• Output cost: The cost of generating new tokens, which is often more computationally expensive.
• Cost per request/token: The overall cost of serving the workload.

Cache hit rate can be especially important in production. Agentic systems often reuse large system prompts, tool definitions, conversation prefixes, or documents. 

If 95% or more of that input can be served from cache, repeated prefill computation can be greatly reduced, improving latency and lowering the effective cost of serving those requests.

These metrics are closely connected. 

Larger batches and higher concurrency can improve throughput and GPU utilization, but may increase latency for individual users. 

Likewise, reducing precision can lower memory usage and hardware requirements, but it may not improve generation speed if the hardware lacks optimized kernels for that format. 

The goal is therefore not to maximize a single metric, but to find the right balance between latency, throughput, utilization, and cost for your workload.

How model architecture affects inference

Inference performance is heavily influenced by the architecture of the model itself.

A dense transformer uses essentially all of its parameters for every token. A Mixture-of-Experts (MoE) model, in contrast, contains many experts but activates only a small subset for each token. 

This can reduce the amount of computation required per token, but it also introduces challenges such as expert routing, load balancing, GPU communication, and expert placement.

The attention architecture also has a major effect on memory usage and inference speed.

Traditional Multi-Head Attention (MHA) stores separate key and value representations for every attention head, which can make the KV cache large. 

Multi-Query Attention (MQA) reduces this memory requirement by allowing multiple query heads to share the same keys and values.

Grouped-Query Attention (GQA) sits between the two, with groups of query heads sharing key-value representations. Newer architectures may use techniques such as Multi-head Latent Attention (MLA), which compresses these representations even further.

This is why two models with similar parameter counts can behave very differently during inference. 

Their architecture can significantly affect compute requirements, KV-cache size, memory usage, and generation speed.

LLM Serving Frameworks

Next, we will look at the most widely used frameworks for running LLMs, from local setups to large-scale production systems.

vLLM

vLLM is one of the most widely used general-purpose inference engines. It supports continuous batching, PagedAttention, chunked prefill, prefix caching, quantization, speculative decoding, optimized attention kernels, and distributed inference.

SGLang

SGLang focuses on high-performance generation and efficient request scheduling. It is particularly useful for agents, reasoning models, repeated context, structured outputs, and other complex generation workloads.

TensorRT-LLM

TensorRT-LLM is NVIDIA's optimized inference stack. It takes advantage of NVIDIA GPU features, specialized CUDA kernels, low-precision formats, distributed execution, KV caching, and advanced batching.

llama.cpp

llama.cpp is particularly useful for local and resource-constrained inference. It supports GGUF quantization, CPU-GPU hybrid inference, and memory-mapped model files, allowing weights to be distributed across GPU memory, system RAM, and, when necessary, disk-backed memory. 

This makes it possible to run models much larger than the available VRAM, although relying heavily on disk can significantly reduce performance.

MLX

MLX is an inference and machine learning framework designed for Apple Silicon. It takes advantage of the unified memory architecture of Macs, making it a useful option for running and experimenting with LLMs locally on M-series devices.

Core LLM Inference Optimizations

From quantization to speculative decoding, this part covers the techniques that make inference faster, cheaper, and more memory-efficient.

Quantization

One of the most common inference optimizations is quantization.

Models are often trained using BF16 or FP16 precision. 

Quantization represents weights, activations, or caches using smaller numerical formats.

Common formats and approaches include:

• FP8
• FP4
• INT8
• INT4
• AWQ
• GPTQ
• GGUF quantization

Reducing precision lowers memory consumption and can reduce memory bandwidth requirements.

However, smaller does not automatically mean faster. Performance depends heavily on whether the target hardware provides efficient kernels for that precision.

The KV cache can also be quantized, which becomes increasingly valuable for long-context and high-concurrency serving.

Attention and Kernel optimizations

Attention is one of the most important operations inside a transformer. It allows the model to determine which tokens in the context are most relevant when generating the next token. 

However, attention can become computationally expensive, especially as the context window grows.

This is where optimized kernels become important. A kernel is a highly optimized piece of code that performs a specific mathematical operation on hardware such as a GPU. Instead of changing how the model works, optimized kernels make those same calculations run faster and use memory more efficiently.

A good example is FlashAttention. Traditional attention can spend a significant amount of time moving large intermediate results between GPU memory and compute units. 

FlashAttention reduces this overhead by processing attention in smaller blocks and keeping more of the computation close to the GPU.

Modern inference engines use several similar optimizations, including:

• FlashAttention
• FlashInfer
• FlashMLA
• Triton kernels
• Fused operations
• Optimized GEMM kernels
• CUDA graphs
• Specialized MoE kernels

The important idea is that these techniques generally do not change the model or its output quality. They optimize how the calculations are executed, helping inference engines generate tokens faster and use the available hardware more efficiently.

KV-Cache optimization

During generation, transformers store keys and values from previous tokens in the KV cache.

Without this cache, the model would need to recompute previous tokens every time it generated another one.

The downside is memory consumption. Long contexts and many simultaneous users can cause the KV cache to occupy a large portion of GPU memory.

Several techniques address this problem.

PagedAttention divides KV-cache memory into manageable blocks instead of requiring large contiguous allocations.

Prefix caching allows requests with identical prefixes to reuse previously computed KV states. This is particularly useful for shared system prompts, agents, and repeated document contexts.

Other approaches include:

• KV-cache quantization
• Cache eviction
• Cache compression
• KV-cache offloading
• Cross-request cache sharing

For large-scale inference systems, KV-cache management can be just as important as model-weight optimization.

Continuous batching and request scheduling

When an inference server receives many requests at the same time, it tries to process them together so the GPU stays busy. This is known as batching.

The problem is that LLM requests rarely finish at the same time. 

One user might generate 50 tokens while another generates 2,000. 

With traditional batching, shorter requests can end up waiting for the longest request in the batch to finish.

Continuous batching solves this by removing completed requests and immediately replacing them with new ones. This keeps the GPU working instead of leaving capacity unused.

Another useful technique is chunked prefill. Instead of processing one very large prompt all at once, the server breaks it into smaller chunks so it does not block other requests that are already generating tokens.

As traffic grows, inference becomes a scheduling problem. The server has to decide which requests to process, how to group them, and how to balance GPU memory, sequence length, throughput, and latency.

Speculative decoding

Traditional autoregressive decoding generates tokens one at a time, which makes generation inherently sequential.

Speculative decoding tries to break this bottleneck by generating several candidate tokens cheaply and then letting the full target model verify them in parallel. 

If several candidates are accepted, generation can advance multiple tokens at once without changing the target model's output distribution.

Several approaches follow this idea:

• Draft-model speculation uses a smaller, faster model to propose tokens for the larger model to verify.
• Medusa adds lightweight prediction heads that propose multiple future tokens from the target model.
• EAGLE-3 uses a lightweight drafter conditioned on features from multiple layers of the target model to generate higher-quality draft tokens.
• MTP (Multi-Token Prediction) uses prediction heads built into compatible models to predict multiple future tokens.
• DFlash and DFlash2 move toward parallel drafting, generating an entire block of candidate tokens in a single forward pass instead of drafting them sequentially.
• DSpark builds on parallel drafting by modeling dependencies between draft tokens and estimating which parts of the proposed block are likely to be accepted.

The basic idea is simple:

Generate several likely future tokens cheaply, then verify them efficiently with the larger model.

Speculative decoding works best when the next tokens are relatively predictable and the acceptance rate is high. 

This makes it especially useful for coding and other sequential tasks, where even a smaller model can often predict what comes next. 

For complex reasoning or highly unpredictable generation, fewer draft tokens may be accepted, reducing the performance benefit.

Memory and Scaling

As models and workloads grow, memory becomes a major constraint. Here, we will explore offloading, multi-GPU inference, disaggregated serving, and long-context optimization.

Offloading and the memory hierarchy

Sometimes the goal is not maximum speed. It is simply getting a model to run on the hardware you have.

Inference systems can use different levels of memory:

GPU HBM → CPU RAM → NVMe storage

GPU memory is the fastest but also the most limited. 

CPU memory offers more capacity at lower speed, while NVMe storage provides even more space but is much slower.

If a model or its KV cache cannot fit entirely in VRAM, inference engines can move part of the workload elsewhere using techniques such as:

• Weight offloading
• CPU-GPU hybrid inference
• KV-cache offloading
• CPU memory
• NVMe or SSD storage

This increases the amount of memory available, but usually comes with a performance cost because data has to move between slower memory layers, often across PCIe.

The basic trade-off is simple: the farther data moves away from GPU memory, the more capacity you gain, but the slower inference usually becomes.

Multi-GPU and distributed inference

Very large models often require multiple accelerators.

Common strategies include:

1. Tensor parallelism: Splits individual model computations across multiple GPUs so they work together on the same layers.
2. Pipeline parallelism: Splits the model's layers across GPUs, with each GPU handling a different portion of the model.
3. Data parallelism: Runs multiple copies of the model across GPUs to process more requests simultaneously.
4. Expert parallelism: Distributes the experts of a Mixture-of-Experts (MoE) model across different GPUs.
5. Context parallelism: Splits long input sequences across multiple GPUs to reduce the memory and computation required on each device.

Once inference spans multiple GPUs, communication becomes an important bottleneck. 

Technologies such as NVLink, NVSwitch, and high-speed networking can therefore have a substantial effect on performance.

Disaggregated prefill and decode

Prefill and decode behave very differently during inference. Prefill is usually compute-intensive, as it processes many prompt tokens in parallel, while decode is often memory-bandwidth-intensive, generating tokens one at a time.

Instead of making both stages compete for the same GPUs, larger serving systems can separate them:

Prompt → Prefill Worker → KV Cache Transfer → Decode Worker → Response

Prefill workers process the prompt and build the KV cache, which is then transferred to dedicated decode workers that generate the response. This allows each stage to use different hardware, batching, and parallelism strategies.

The main advantage is that prefill and decode can be optimized and scaled independently, helping control time to first token (TTFT) and inter-token latency (ITL), especially for long prompts and high-traffic workloads.

Long-context inference

Long context windows create a different set of inference challenges. As the input grows, prefill takes more computation, while the KV cache grows larger, consuming GPU memory and making decoding more expensive.

Several techniques can help:

• Prefix caching: Reuses the KV cache when requests share the same prompt or context, avoiding repeated prefill computation.
• KV-cache quantization and compression: Reduces the memory required to store previous tokens.
• Sliding-window or sparse attention: Limits how much of the context the model attends to, when supported by the model architecture.
• Context parallelism: Splits a long sequence or its KV cache across multiple GPUs, helping both long-context prefill and decoding.
• KV-cache transfer and offloading: Moves cached states between GPUs, CPU memory, or storage when GPU memory becomes limited.

The important point is that supporting a one-million-token context window does not mean serving one million tokens is fast, cheap, or practical. 

Longer contexts increase memory usage, time to first token, and overall inference cost.

Specialized Inference Workloads

Not every model behaves the same at inference time, so we will examine the unique challenges of MoE models, structured generation, and reasoning workloads.

Mixture-of-Experts inference

Mixture-of-Experts (MoE) models behave differently from dense models. Instead of using every parameter for every token, a router selects only a small number of experts to process each token. 

This reduces computation, but the full set of experts still needs to be stored somewhere across the serving hardware.

Efficient MoE inference, therefore, depends on:

• Expert routing: Choosing which experts process each token.
• Expert placement: Deciding where experts are stored across GPUs.
• Expert parallelism: Distributing different experts across multiple GPUs.
• Load balancing: Preventing popular experts from overloading certain GPUs.
• All-to-all communication: Efficiently moving tokens to their selected experts and returning the results.
• Optimized MoE kernels: Speeding up expert computation and communication.

At scale, communication can become a major bottleneck, which is why technologies such as DeepEP provide specialized high-throughput and low-latency kernels for moving tokens between experts.

The key distinction is that total parameters determine how much model capacity must be stored, while active parameters give a better idea of how much computation is performed for each token.

Structured and constrained decoding

Modern LLM applications often need something more predictable than unrestricted text. Agents, for example, may need to generate valid JSON, tool arguments, SQL, or responses that follow a predefined schema.

Constrained decoding controls which tokens the model is allowed to generate at each step, so the final output follows a required structure. Modern inference engines such as vLLM support constraints, including:

• JSON Schema
• Regular expressions
• Grammars
• Tool and function schemas

This is usually more reliable than generating unrestricted text and trying to repair it afterward. However, constrained decoding guarantees the structure, not necessarily the correctness of the information inside it.

Reasoning model inference

Reasoning models create a very different inference workload because they may generate thousands of intermediate reasoning tokens before producing the final answer. 

Unlike long-context workloads, where most of the cost may come from processing a large input, reasoning workloads can remain expensive throughout generation because the decode phase itself becomes very long.

Long reasoning traces affect:

• Decode latency
• KV-cache memory
• Request scheduling
• Batch efficiency
• Throughput
• Cost

Another challenge is that reasoning length is often difficult to predict. 

One request may finish quickly, while another continues generating thousands of tokens. This makes batching, scheduling, and KV-cache allocation harder when many users are being served at the same time.

As a result, serving a chatbot that generates a few hundred tokens and a reasoning model that may generate 10,000 tokens can require very different inference, memory, and scheduling strategies.

Hardware-Aware LLM Inference

Software optimization is only part of LLM inference. The hardware underneath it can have just as much impact on speed, memory capacity, scalability, and cost.

Important hardware characteristics include:

• GPU compute performance: Determines how quickly mathematical operations can be performed.
• Memory bandwidth: Determines how quickly model weights and KV-cache data can move between GPU memory and compute units.
• HBM capacity: Determines how much of the model, KV cache, and active requests can stay directly on the GPU.
• CPU memory: Provides additional capacity when models or caches need to be offloaded.
• PCIe bandwidth: Affects how quickly data can move between the CPU and GPUs.
• GPU-to-GPU interconnects: Technologies such as NVLink allow GPUs to exchange data quickly during multi-GPU inference.
• Network bandwidth: Becomes important when inference is distributed across multiple servers.

A useful concept for understanding hardware performance is arithmetic intensity, which describes how much computation is performed relative to how much data must be moved. Workloads with high arithmetic intensity tend to be compute-bound, while workloads with low arithmetic intensity are often limited by memory bandwidth.

During prefill, many prompt tokens can be processed in parallel, allowing the hardware to make better use of its compute resources. During decode, however, tokens are generated sequentially, and the system repeatedly accesses model weights and KV-cache data. This makes memory bandwidth especially important for generation speed.

This is why memory capacity and memory bandwidth are two different things. For example, both the NVIDIA DGX Spark and an Apple M4 Max system can provide up to 128 GB of unified memory. However, DGX Spark provides 273 GB/s of memory bandwidth, while M4 Max reaches up to 546 GB/s. So even though both systems may be able to fit the same large model in memory, their inference performance can behave very differently.

The same principle applies when comparing GPUs and local AI machines: fitting the model solves the capacity problem, not necessarily the speed problem. 

For LLM inference, compute performance, memory capacity, memory bandwidth, and interconnect speed all need to be considered together.

Choosing the Right LLM Inference Optimization

With so many inference techniques available, the key is to first identify what is actually limiting your system. 

A model that does not fit in VRAM needs a very different optimization from a server struggling with high latency or low throughput.

The important point is that there is no single best optimization. 

Start by measuring metrics such as TTFT, token generation speed, throughput, GPU utilization, and memory usage. Then apply the technique that targets the actual bottleneck and benchmark again.

For example, quantization can help when memory is limited, continuous batching can improve throughput, prefix caching can reduce repeated prefill work, and speculative decoding can improve generation speed when future tokens are predictable.

The goal is not simply to maximize tokens per second, but to find the best balance between latency, throughput, memory usage, model quality, hardware, and cost.

Final Thoughts

Running an LLM locally and getting a response from llama.cpp is a great starting point, but it does not automatically mean you have built an efficient inference system.

For personal projects, experimentation, and smaller workloads, a simple local setup may be all you need. 

But once you start serving many users, handling long contexts, or building complex coding and agentic systems, the requirements change quickly. 

You need an inference stack that can process many requests simultaneously, maintain good generation speed, manage memory efficiently, and stay responsive even as workloads become more demanding.

This is where concepts such as continuous batching, prefix caching, KV-cache management, quantization, speculative decoding, optimized kernels, parallelism, and request scheduling start to matter. 

You also need to understand metrics such as TTFT, throughput, latency, concurrency, GPU utilization, and cost per token instead of judging performance only by tokens per second.

There is also no universal configuration that works for every model or workload. A technique that works well for coding may not provide the same benefit for reasoning. 

An optimization that improves throughput may increase latency. Quantization can significantly reduce memory requirements, but more aggressive quantization may also reduce model quality. 

Ultimately, good LLM inference engineering is about balance. The goal is to make the best use of the hardware you already have while balancing speed, memory, throughput, latency, model quality, and cost.

That is why learning inference matters. Instead of relying entirely on default settings or simply adding more GPUs, you can understand where the bottleneck is, choose the right optimization, and build a serving system that is actually designed for your workload. 

To learn more about LLM inference, I recommend checking out our webinar on Understanding LLM Inference.