Take one 8B model and one 24GB card. It runs out of memory in three different situations:

Three crashes, one model, three different causes. This post splits GPU memory into four buckets (weights, gradients and optimizer state, activations, and the KV cache) so you can work out which one is full before you reach for a fix.

A lever is a setting you change to save memory, such as loading the model in 4-bit, using LoRA, turning on gradient checkpointing, or shortening the prompt. Estimate the buckets first, then pick the lever that shrinks the full one.

A parameter is one learned number. During training, the model starts with random values and adjusts each one, over and over, until its predictions improve; those adjusted values are what the model "knows". Think of them as billions of tiny dials, each set to one value, that together turn your input into an output. The model has about 8 billion of them, and each is stored in some dtype (data type), which fixes how many bytes it takes. Weight memory is parameters times bytes per parameter:

The 4-bit row has a range because of overhead. Quantized weights carry scale factors, and some layers (usually the embeddings and the output head) often stay in higher precision. Budget 4.5GB, not 4.

Two ideas hide in that table, and they are not the same.

Precision is a dtype choice at load time. In the table, the fp32 and fp16 / bf16 rows are precision: they are all floating-point numbers, and going from fp32 to fp16 or bf16 just gives each one fewer bits.

Quantization is the int8 and 4-bit rows. It replaces floating-point weights with small integer codes plus a scale factor (and sometimes an offset) per block of weights. To use a weight, the runtime multiplies the code back by its scale. You store 4 bits and compute with a reconstructed value. You will see this under names like GPTQ, AWQ and GGUF, which are different ways of doing it; for memory they all do the same job: they shrink the weights to about 4 bits each, so the 8B model needs 4 to 5GB instead of 16GB.

Fewer bits means a rougher copy of the original numbers, so the model's answers can get slightly worse. That is the "cost" here: lost answer quality, not money. In practice, for many tasks, going from 16-bit to 8-bit barely changes the answers, and 4-bit loses a little more, but the loss depends on the task and the method, so test on your own evaluation. This post does not rank them.

At this point the 8B model's weights fit on the card at 16-bit (16GB of 24GB). Serving needs only these weights plus a small cache, so short prompts fit with 8GB to spare. Training adds more buckets on top of the 16GB, and that is where it runs out.

Training needs more than the weights. For each trainable parameter, you also keep a gradient (how much to change it) and optimizer state (running statistics the optimizer uses to decide the step). The Adam optimizer, the most common choice for training language models, keeps two numbers per parameter.

With fp32 everything, the cost per trainable parameter is:

That is roughly 16 bytes per trainable parameter.

We stay with fp32 for training because fp16 on its own is often unstable: the tiny updates can get rounded away and training stalls or breaks. (Mixed precision, which is the usual workaround, ends up at about the same 16 bytes.)

For full fine-tuning of the 8B model, 8B × 16 bytes is about 128GB. The card has 24. It dies on step one because of this bucket. Loading the weights in 4-bit would save about 28GB, but the gradients and optimizer state (roughly 96GB) are still needed for every weight you train, so full fine-tuning cannot fit this way.

Mixed precision, 8-bit optimizers and paged optimizers all change the 16 bytes per parameter. Treat it as the baseline to reason from, not a law.

LoRA changes which parameters are trainable. It freezes the base (its weights never change) and trains small extra matrices, called adapters, added to chosen layers. A frozen weight needs no gradient and no optimizer state, so you pay the 16 bytes only on the adapters. Each layer's weights are matrices that turn a vector of numbers going in (the input size, 4,096 here) into a vector coming out (the output size, 4,096 for most of them). Instead of retraining a full 4,096 × 4,096 matrix, LoRA adds two thin ones: one that squeezes the input down to a small width, and one that expands it back to the output size. That small width is the rank, usually a number like 8 or 16; a higher rank means a more flexible adapter but more parameters. So for each adapted matrix, the added parameters are rank × (input size + output size).

Take rank 16 on the four attention projections (q, k, v, o) in each of 32 layers. With the example shape, that is about 13.6 million trainable parameters, or roughly 0.22GB at 16 bytes each. Doubling the rank doubles it. It stays small next to 128GB.

QLoRA keeps the same adapters and stores the frozen base in 4-bit. It is a bucket-1 change (weights drop from 16GB to about 4.5GB) and a bucket-2 change (gradients and optimizer state exist only for the adapters) together. For a comparison of how these methods differ in quality, see our earlier post, Parameter-efficient fine-tuning compared: what LoRA and its relatives trade off.

Activations are the intermediate outputs each layer produces as your input flows through the model (the forward pass). Training keeps all of them in memory because the backward pass, which computes the gradients, needs them to work out how to change each weight.

This is the bucket that surprises people on LoRA and QLoRA runs. The base is frozen and has no gradients, but gradients still flow through it to reach the adapters in earlier layers, so its activations must still be stored. Freezing weights does not free them.

Activation memory scales with batch size times sequence length (the number of tokens in flight), times layers, times hidden size. Batch size is how many examples you train on at once, and sequence length is how many tokens each example has. So double either one and activation memory doubles: 2,048 tokens cost about 10.7GB (see the table later in the post), and 4,096 tokens would cost about 21.5GB. This is why the 4-bit fine-tune above trained and then died when you raised the sequence length: bucket 1 had shrunk, bucket 3 had not, and you grew bucket 3.

Gradient checkpointing trades compute for this bucket. Normally every layer's activations stay in memory until the backward pass. With checkpointing, you keep only each layer's input, and when the backward pass needs the rest, the model recomputes them. The diagram above assumes about 20 stored tensors per layer (an illustrative figure, not a measurement), and checkpointing cuts that to about 1. The price is that the forward work is partly done twice, which commonly slows training by something like a third, though it varies. It shrinks bucket 3 and nothing else.

At inference (when the model is only generating answers, not learning) there are no gradients, no optimizer, and no stored activations (each layer's output is discarded once the next layer has used it), but a new bucket appears. To generate each new token, the model looks back at every earlier token in the prompt and its own output so far, using two numbers it computes for each token, a key and a value. Recomputing those for the whole history at every step would be slow, so the model stores them once: that store is the KV cache.

All you need to know for now is that every token adds the same fixed amount to the cache. For our 8B model in 16-bit that is about 128 KiB (0.13MB) per token, and nothing is removed as the conversation goes on. So the cache grows in a straight line: twice the tokens, twice the cache. For one sequence:

It also grows with the batch, meaning how many requests the GPU serves at the same time (say, four users chatting with the model at once). Each one has its own cache, so four 32,768-token sequences need 4 × 4.3GB, about 17.2GB. That is more than the 16GB the weights take.

This is the long-context crash. The model did not change between the short prompt and the 32k prompt, and the weights stayed at 16GB (bucket 1). What grew was the cache (bucket 4), from under 1GB to 4.3GB per sequence, and with several sequences at once it filled the rest of the card. Shrinking the weights further would not fix it, because the cache is now the bigger bucket.

Where the 128 KiB comes from (keys, values and attention heads), and the ways to shrink the cache, are a deep dive for a later post. Here you only need to know that the cache is its own bucket and that it grows with tokens and with parallel requests.

Here is the whole accounting for the example, against the 24GB line. Training rows use a batch of 1 at 2,048 tokens, about 10.7GB of activations without checkpointing and about 0.5GB with it. Inference rows use the 16-bit KV cache. Figures are in GB, rounded, and exclude the CUDA context, logits buffers and fragmentation, which add a few GB in practice.

In the full fine-tune row, the 128GB from bucket 2's formula (16 bytes × 8B) includes the weights themselves, so the table splits it into 32GB for weights (bucket 1) and 96GB for gradients and optimizer state (bucket 2).

Read the table one column at a time, and the lever-to-bucket map starts to make sense:

These figures are arithmetic, not measurements from a profiler. Use them to decide what to try, then confirm with torch.cuda.max_memory_allocated(), which reports the peak GPU memory your code actually used during a real training or inference step.