When I first tried to run a 7B model on a consumer RTX 3060, the token rate stalled at about 5 tokens / s. The model fit comfortably in 12 GiB, so I assumed the bottleneck was RAM or the CUDA kernel. A quick nvidia-smi check showed the GPU was at 99 % memory bandwidth utilization while compute hovered around 30 %. That was the moment I realized the real limiter was VRAM bandwidth, not capacity.

What VRAM bandwidth actually measures

VRAM bandwidth is the amount of data the GPU can move between its memory and the compute cores per second. In the context of transformer inference, each token requires a few gigabytes of weight reads, activation writes, and intermediate matrix multiplications. The larger the model, the more weight data must be streamed in each forward pass.

  • weight reads, pulling the parameter matrices from VRAM into the tensor cores.
  • activation writes, storing the intermediate hidden states for the next layer.
  • matrix mul, the core compute that still needs data from memory.

If the memory bus cannot supply data fast enough, the cores sit idle, and token generation slows down.

Why size-only arguments mislead

Most tutorials tell you to check "does the model fit in VRAM?" That is a necessary first step, but it hides the second, more subtle requirement: the bandwidth must sustain the data-flow of each layer. A 12 GiB card can hold a 6B model, yet if the bus is 200 GB/s while a similar-capacity card offers 500 GB/s, the slower card will produce fewer tokens per second.

I once swapped a 4 GB-wide memory card for a 2 GB-wide one on the same GPU chassis. The model still fit, but token speed dropped by 40 % because the narrower bus halved the effective bandwidth.

Measuring the real-world bandwidth usage

The easiest way to see the bottleneck is to run nvprof or the newer nsight profiler while generating text. Look for the Memory Throughput metric. If you see numbers close to the card's rated bandwidth, you are memory-bound.

nsight-systems profile --trace=cuda,osrt --stats true python generate.py
  • nsight-systems launches the profiler.
  • --trace=cuda,osrt captures CUDA memory activity and OS runtime.
  • The generated report shows Device Memory Throughput in GB/s.

When the throughput line sits at 95 % of the card's spec, any further speed-up must come from reducing memory traffic, not adding more cores.

Practical ways to reduce bandwidth pressure

1. Quantize the weights

Switching from FP16 to INT8 or even INT4 cuts the weight size by half or three-quarters. The model still fits in VRAM, but each forward pass now reads far less data. Libraries like bitsandbytes make this a one-line change.

import bitsandbytes as bnb
model = bnb.nn.Int8Params.from_pretrained('model_name')
  • Int8Params.from_pretrained loads a quantized checkpoint.
  • The rest of the code stays unchanged.

2. Use FlashAttention or xFormers kernels

These kernels fuse the attention softmax and dropout steps, reducing the number of intermediate writes. The memory traffic drops dramatically, and token speed often jumps 1.5× on the same hardware.

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('model', attn_implementation='flash_attention_2')
  • attn_implementation='flash_attention_2' tells HuggingFace to use the fused kernel.
  • No manual kernel compilation needed for recent PyTorch versions.

3. Cache KV-states wisely

When generating long sequences, the key-value cache grows linearly. If you keep the full cache, each new token forces the GPU to read an ever-larger block of memory. Truncating the cache after a reasonable window (e.g., 1024 tokens) keeps the memory traffic constant.

outputs = model.generate(input_ids, max_new_tokens=500, attention_mask=mask, use_cache=True, cache_implementation='sliding_window', window_size=1024)
  • cache_implementation='sliding_window' enables a rolling cache.
  • window_size=1024 caps the memory needed for KV-states.

4. Offload non-critical layers to CPU

Some layers, like the final layer-norm, are cheap compute-wise but still require weight reads. Offloading them to the CPU frees a fraction of the bandwidth for the heavier attention layers.

model.lm_head.to('cpu')
  • Moves the language-model head to system RAM.
  • The rest of the model stays on the GPU, preserving speed.

When bandwidth isn’t the problem

If profiling shows only 30-40 % memory utilization while compute is at 90 %, you are compute-bound. In that case, upgrading to a GPU with more tensor cores (e.g., moving from RTX 3060 to RTX 4090) will help, but the bandwidth tip still applies: a GPU with both high compute and high bandwidth gives the best token rate.

Common mistake that kills performance

A frequent pitfall is enabling torch.backends.cudnn.benchmark = True for transformer inference. While it helps image models, it forces cuDNN to run a series of kernel autotuning passes that flood the memory bus before the real work starts. Turn it off for LLM generation.

import torch
torch.backends.cudnn.benchmark = False
  • Disables the autotuning overhead.
  • Keeps the memory bus free for the actual forward passes.

How to estimate token throughput from bandwidth

A rough rule of thumb: each token of a decoder-only transformer reads roughly 2 × model_size bytes of weights (once for QKV, once for output projection). For a 6B-parameter model in FP16, that is about 12 GB per token. Divide the card's sustained bandwidth (e.g., 400 GB/s) by 12 GB to get ~33 tokens / s as a theoretical ceiling.

If you measure 20 tokens / s, you are at ~60 % of the ceiling, a good sign you are close to the bandwidth limit. If you see only 5 tokens / s, something else (perhaps a Python-level bottleneck) is still dragging you down.

Choosing the right GPU for local LLM work

| GPU | VRAM | Bandwidth | Approx. token ceiling (6B FP16) | |-----|------|-----------|---------------------------------| | RTX 3060 | 12 GiB | 360 GB/s | ~30 t/s | | RTX 3080 | 10 GiB | 760 GB/s | ~63 t/s | | RTX 4090 | 24 GiB | 1000 GB/s | ~83 t/s |

The table shows that even a modest jump in bandwidth can double token speed, assuming the model fits.

FAQ

How can I tell if my model is memory-bound without a profiler?

Run a short generation loop and watch nvidia-smi. If the Memory-Util column hovers near 100 % while GPU-Util stays below 50 %, you are memory-bound.

Does quantizing to INT4 always improve token speed?

Usually, yes, because it halves the weight traffic. However, INT4 kernels are newer and may not be fully optimized on older GPUs, so test on your hardware before committing.

Will increasing batch size help token throughput?

For single-prompt generation, batch size >1 adds overhead. For many independent prompts, batching can amortize memory reads across tokens, effectively increasing overall throughput.

Wrapping up

If you’re chasing higher token rates on a consumer GPU, stop obsessing over the number of cores and start looking at the memory bus. Quantize, use fused kernels, trim the KV-cache, and keep an eye on actual bandwidth usage. Those tweaks squeeze more tokens out of the same silicon without spending a fortune on new hardware.