Fine-Tuning and Efficient Models

Why adapt a model

The most capable language models are one API call away. Yet a large share of real AI products run on something else: an open model whose weights someone has changed, then squeezed until it fits on a single GPU, a laptop or a phone.

What does it mean to adapt a model?

A pretrained model is a general-purpose starting point. Adapting it means changing the model itself, not just the prompt, and it comes in two flavours that this lesson covers in turn:

• Specialising it: continuing training on your own examples so it does one job the way you want (fine-tuning, and its cheap cousin LoRA).
• Shrinking it: storing its numbers in fewer bits, deleting the ones that matter least, or training a smaller model to imitate it (quantization, pruning and distillation).

The two are usually combined. The typical recipe in 2026 is: start from an open model, fine-tune a small adapter, quantize, ship.

Why not just prompt a frontier model?

Often you should. But there are five recurring reasons teams choose otherwise.

• Cost at volume. API prices are per token. A task that runs millions of times a day, such as classifying support tickets or extracting fields from invoices, can cost far less on a small model you host yourself.
• Latency. A 3-billion-parameter model on local hardware answers before a network round trip to a data centre has finished.
• Privacy and control. Medical notes, legal documents and source code may not be allowed to leave the building. Your own weights also never change underneath you when a provider updates its model.
• On-device and offline. Phones, cars and factory equipment cannot always reach the internet.
• Narrow tasks. A small model trained on one task can beat a large generalist at that task. In LoRA Land (2024), 310 small open models fine-tuned with 4-bit LoRA beat GPT-4 by 10 points on average across 31 narrow tasks, while costing a fraction as much to serve.

What does this look like in practice?

Three shipping examples from 2025 show both halves of the lesson at work.

• Apple's on-device model has about 3 billion parameters, compressed with 2-bit quantization-aware training so it runs on an iPhone. Developers can specialise it for their app by training small LoRA adapters through Apple's Foundation Models framework.
• OpenAI's open-weights gpt-oss-20b stores its mixture-of-experts weights in a 4-bit format so that the 21-billion-parameter model runs within 16 GB of memory, the size of a well-equipped laptop.
• Google's Gemma 3n models have 5 and 8 billion raw parameters but are engineered to run in as little as 2 and 3 GB of memory on phones.

Key takeaways

• Adapting a model means changing its weights: specialising it (fine-tuning) or shrinking it (quantization, pruning, distillation).
• Cost at volume, latency, privacy, offline use and narrow tasks are the usual reasons to adapt an open model instead of prompting a frontier one.
• Shipping systems combine both: Apple’s 3B on-device model uses 2-bit weights plus per-app LoRA adapters.

Prompt, retrieve or fine-tune?

There are four ways to make a model better at your task, and they differ in cost by orders of magnitude. The rule is simple: climb only when the rung below has failed, and know what kind of failure sends you up.

What does each rung actually change?

The key distinction is between knowledge and behaviour. Retrieval changes what the model knows at answer time by putting facts in front of it. Fine-tuning changes how it behaves: its format, tone, level of detail, the labels it uses, the steps it follows.

Fine-tuning is a surprisingly poor way to teach facts. Ovadia and colleagues (2023) compared the two directly and found that retrieval consistently beat fine-tuning at injecting knowledge, whether the facts were already partly known to the model or entirely new. Facts learned into weights are also hard to update, impossible to cite and easy to blur with similar facts.

How do you decide which rung you need?

Look at how the current system fails on a set of real examples, then match the failure to the fix:

The third case is the most common reason to fine-tune, and it has a neat trick built in: use the big model, with its long and careful prompt, to produce the training examples, check them, and fine-tune a small model on the result. That is distillation by another name, and we will return to it.

Why does the order matter?

Each rung up adds a training pipeline to maintain, data to curate, a model to evaluate and host, and new ways to break things. A prompt can be edited in a minute; a fine-tuned model has to be retrained when the base model or the task changes. Teams that skip straight to fine-tuning often discover that a few good examples in the prompt would have done the job.

Key takeaways

• Climb the ladder in order: prompting, then retrieval, then fine-tuning, and almost never pretraining.
• Retrieval adds knowledge; fine-tuning changes behaviour. Fine-tuning is a poor way to teach facts.
• The most common good reason to fine-tune is cost: making a small model do reliably what a large prompted model already does.

Full fine-tuning and forgetting

Full fine-tuning is the obvious approach: keep training the whole network on your examples. It works, and for an 8-billion-parameter model it needs eight times more GPU memory than simply running the model.

What is full fine-tuning?

It is ordinary gradient descent, started from pretrained weights instead of random ones. The loss is the same next-token prediction loss used in pretraining, computed on your examples; for a chat model it is usually applied only to the tokens of the assistant's replies, so the model learns to produce answers rather than to predict questions. The learning rate is much smaller than in pretraining and training lasts a few passes over the data, because the goal is to nudge a good model, not rebuild it.

Why is it so expensive?

Because training has to keep far more than the weights in memory. With the standard recipe, mixed-precision training with the Adam optimizer, each parameter needs 16 bytes:

Per parameter: bf16 weights and gradients, plus an fp32 master copy of the weights and Adam's two running averages (momentum and variance).

For Llama 3.1 8B that is 8.03 billion × 16 ≈ 128 GB before a single activation is stored, more than the 80 GB of one H100. The model itself needs only 16 GB to run. And every fine-tuned copy is another full-size checkpoint to store and serve.

Full fine-tuning of the 8B model needs two data-centre GPUs, while LoRA and QLoRA both fit on a single card. For the 70B model the gap is decisive: over a terabyte for full fine-tuning, about 145 GB for LoRA and about 40 GB for QLoRA. Notice what dominates full fine-tuning: the optimizer states, not the weights. That is the first clue for the next section. If most of the memory goes on bookkeeping for parameters being trained, train fewer parameters.

Catastrophic forgetting

The second cost is subtler. Gradient descent on new data has no reason to preserve what the network learned before, and neural networks are known to overwrite old skills abruptly when trained on something new: catastrophic forgetting. A model fine-tuned hard on legal contracts may become worse at arithmetic, at following unrelated instructions, or at refusing harmful requests.

With no protection, accuracy on task A collapses from about 99% to well below chance within a hundred epochs, even though the two tasks live in different halves of the plane and the network has room for both. The network simply extends task B's rule everywhere. Anchoring the weights to their old values (a simplified version of Kirkpatrick's elastic weight consolidation, which weights the anchor by how important each parameter was) slows forgetting but, when strong, also stops the network learning B. Replaying a slice of the old data is crude and very effective, which is why practitioners mix general-purpose instruction data into fine-tuning sets.

There is a structural fix too. Biderman and colleagues (2024) found that LoRA, the method of the next section, learns less of a hard new domain than full fine-tuning but also forgets less of everything else. A small update can only move the model so far.

Key takeaways

• Full fine-tuning with Adam needs about 16 bytes per parameter: an 8B model needs about 128 GB before activations, versus 16 GB to run it.
• Optimizer states, not weights, dominate the memory, which is why training fewer parameters helps so much.
• Fine-tuning can erase earlier abilities (catastrophic forgetting); replaying old data, anchoring weights and small updates all reduce it.

LoRA: low-rank updates

LoRA rests on a bet about geometry: the change a model needs for a new task, though written as a huge matrix, only really moves in a few directions. If that is true, you can learn the change as a product of two thin matrices and never touch the original weights.

What is LoRA?

Low-Rank Adaptation, introduced by Hu and colleagues (2021), freezes every pretrained weight matrix W₀ and learns an update in factored form, ΔW = BA. If W₀ is d × k, then B is d × r and A is r × k, with a rank r that is tiny compared with d: typically 8 to 64 when d is in the thousands.

The adapted layer. α is a fixed scaling constant; dividing by r keeps the update's size stable when you change the rank.

How does it work?

• Start where the model already is. A is initialised randomly and B at zero, so BA = 0 and training begins exactly at the pretrained model.
• Train only A and B. Gradients and Adam states exist only for the adapters, which is where the memory saving comes from. Hu and colleagues fine-tuned GPT-3 175B with 10,000 times fewer trainable parameters and a third of the GPU memory, and each task's checkpoint shrank from 350 GB to about 35 MB.
• Merge or swap. After training you can add BA into W₀ and serve a normal model with no extra latency. Or keep adapters separate and swap them per request: one base model in memory, hundreds of specialised adapters beside it. That is how Apple ships per-feature adapters, and how multi-LoRA servers host many customers on one GPU.

Why would the update be low-rank?

Any matrix can be split by the singular value decomposition into rank-one layers, ordered by importance (the singular values). If the importance falls off quickly, the first few layers carry nearly everything. Hu and colleagues found that for GPT-3 even r = 1 or 2 on the attention query and value matrices performed competitively with larger ranks. Adapting to one task needs few new directions; the rich machinery is already in W₀.

For the update-like matrix, rank 3 captures 96% of the energy (the sum of squared singular values) and rank 5 about 99%, using 192 or 320 numbers instead of 1,024. A random matrix has a flat spectrum: every direction matters about equally, and rank 4 captures only 39% of it. LoRA works because real task updates look like the first matrix, not the second. The Eckart–Young theorem guarantees the SVD truncation is the best rank-r approximation possible, so no clever factorisation could do better.

The bet does not always pay off. Biderman and colleagues measured the updates of full fine-tuning on code and maths and found them to be of much higher rank than typical LoRA settings. For large shifts, such as teaching a model a new programming language, LoRA lags behind full fine-tuning. For the far more common job of shaping behaviour, it is usually close, far cheaper, and it forgets less.

Key takeaways

• LoRA freezes W₀ and learns ΔW = BA with small rank r, so only a tiny fraction of parameters need gradients and optimizer states.
• B starts at zero, adapters can be merged for free inference or swapped per request, and each task costs megabytes instead of gigabytes.
• It works because task updates have fast-decaying singular values; for large domain shifts full fine-tuning still learns more.

QLoRA and other adapters

LoRA removed the optimizer states. The frozen weights were still stored in 16 bits, so a 65-billion-parameter model needed 130 GB just to sit in memory. QLoRA stored them in 4.

What is QLoRA?

QLoRA (Dettmers and colleagues, 2023) keeps the base model frozen in a 4-bit format and trains 16-bit LoRA adapters on top. During each forward and backward pass the 4-bit weights are unpacked to 16 bits block by block, used, and discarded, so gradients flow through the quantized model into the adapters. It cut the memory to fine-tune a 65B model from more than 780 GB to under 48 GB: one GPU.

How does it keep quality?

• NF4 (4-bit NormalFloat). Trained weights are roughly bell-shaped, so instead of 16 evenly spaced levels NF4 places them at the quantiles of a normal distribution: dense near zero where most weights are, sparse in the tails.
• Double quantization. Each block of 64 weights needs a scale. Quantizing those scales too saves about 0.37 bits per parameter, roughly 3 GB on a 65B model.
• Paged optimizers. Occasional memory spikes from long sequences are paged out to CPU memory instead of crashing the run.

The paper's Guanaco models, fine-tuned for 24 hours on a single GPU, reached 99.3% of ChatGPT's performance level on the Vicuna benchmark, as judged at the time. QLoRA is a big reason fine-tuning moved from labs to laptops and single rented GPUs.

The wider parameter-efficient family

What goes into the training data

The method decides the cost; the data decides the result. Two kinds of fine-tuning cover most uses, and the Large Language Models lesson explains both in depth:

• Supervised fine-tuning on demonstrations: an input and the exact output you want. This is what you do when you teach a model your format, labels or house style.
• Preference tuning on pairs of a better and a worse answer, usually with DPO. Useful when “better” is easier to judge than to write down.

A few hundred to a few thousand clean, consistent examples often beat tens of thousands of noisy ones. The examples must look like what the model will see in production, down to the prompt template, and a held-out slice must be kept for evaluation before any training starts.

Key takeaways

• QLoRA trains LoRA adapters through a frozen 4-bit (NF4) base model, putting 65B fine-tuning on one 48 GB GPU.
• Adapters and prefix tuning are older parameter-efficient methods; LoRA won because it adds no inference cost once merged.
• Supervised fine-tuning teaches from demonstrations, preference tuning from comparisons; data quality matters more than method choice.

Quantization

When a language model writes a token, it reads every one of its weights from memory. For most models that reading, not the arithmetic, is the bottleneck. Store each weight in 4 bits instead of 16 and there is a quarter as much to read.

What is quantization?

Quantization stores numbers with fewer bits. Training uses floating-point formats with a sign, an exponent (how big) and a mantissa (how precise). Inference can often get away with far less: 8-bit or 4-bit integers, each paired with a shared scale that says what one step is worth.

How does it work?

The simplest scheme is absmax quantization. Take a group of weights, find the largest magnitude, and map the range [−max, +max] onto the available integer codes. Each weight becomes the nearest code; to use it, multiply back by the scale.

Symmetric b-bit quantization of a group of weights. Rounding error per weight is at most half a step, s/2.

Two lessons fall out. First, every bit you remove roughly doubles the error (about 6 dB of signal-to-noise per bit): at 8 bits the error is under 1%, at 4 bits about 15%. Second, one large value ruins the grid for everyone. With six outliers and a single scale, 4-bit quantization rounds about two thirds of all values to zero and the error triples. Giving each block of 64 its own scale confines the damage to the few blocks that contain an outlier, for a cost of a quarter of a bit per value. Nearly every modern LLM quantizer works in small blocks for exactly this reason.

The outlier problem in real LLMs

This is not a toy concern. Dettmers and colleagues (2022) found that once transformers pass about 6.7 billion parameters, a small number of hidden-state dimensions start carrying values up to 20 times larger than the rest, consistently across layers, and that naive 8-bit quantization of these activations wrecked the models. Their LLM.int8() method splits each matrix multiplication: the handful of outlier dimensions stay in 16 bits, and everything else runs in 8. The result matched 16-bit quality on models up to 175B.

Since then the toolbox has grown:

• Smarter rounding. GPTQ quantizes weights one column at a time and adjusts the remaining columns to cancel the error just introduced, using second-order information from a little calibration data. It reaches 3 to 4 bits with small losses and quantized 175B-parameter models in about four GPU hours.
• Quantization-aware training. Simulate the rounding during training so the model learns weights that survive it. Apple uses this to get its on-device model down to 2 bits per weight.
• Low-precision formats in hardware. Recent GPUs compute directly in 8-bit and 4-bit floating point with small shared scales per block. gpt-oss ships its expert weights in one such format, MXFP4, so the 117B model fits on a single 80 GB GPU.

Why does it matter?

Because it decides what hardware a model needs. Just holding the weights (before any KV cache) takes:

Teal marks what fits a 24 GB consumer GPU. Going from 16 to 4 bits moves an 8B model from a data-centre card to a laptop, and a 70B model from two GPUs to one. As for quality: 8-bit weights are close to lossless for most models, well-made 4-bit versions lose a little, and 3 bits and below usually need quantization-aware training or careful methods to stay useful. Always measure on your own task; averages hide which abilities break first.

Key takeaways

• Quantization stores weights as small integers plus shared scales; generating text is memory-bound, so fewer bytes means faster, cheaper tokens.
• Each bit removed roughly doubles rounding error, and a single outlier stretches the grid, which is why quantizers use small blocks with their own scales.
• LLM.int8(), GPTQ, quantization-aware training and hardware formats like MXFP4 make 8-bit near lossless and 4-bit practical.

Distillation and pruning

Quantization keeps the same network and stores it more cheaply. The other two routes change the network: train a smaller one to imitate it, or cut pieces out.

What is knowledge distillation?

A large teacher model is trained first. A small student is then trained to match the teacher's output probabilities, not just the correct labels. Hinton, Vinyals and Dean (2015) pointed out that those probabilities carry information the labels throw away: a teacher that says an image of a 2 is 95% “2”, 3% “3” and 1% “7” is telling the student which digits this one resembles. They called it dark knowledge.

How does temperature help?

A confident teacher puts almost all its probability on one class, so the interesting ratios between wrong answers are buried in numbers like 0.0003. Dividing the logits by a temperature T > 1 before the softmax flattens the distribution and brings those ratios up to a size that the training signal can feel. The student is trained on a mix of the softened teacher distribution and the true labels.

The distillation loss. p_T is the softmax of logits divided by T. The T² factor keeps the soft-target gradients on the same scale as T changes.

At T = 1 the wrong answers share under 5% of the probability, most of it on 3 and 7. At T = 4 they hold over 60%, and the student is now clearly told that 3 is far more plausible than 4. Push T towards 20 and every class looks alike: the signal is gone again. Hinton and colleagues found the best temperature depends on the student's size; very small students did best with moderate values of about 2.5 to 4.

For language models, distillation often takes a simpler form: the teacher writes training data and the student is fine-tuned on it. DeepSeek released small models distilled this way from R1 (see the Large Language Models lesson), and it is the pattern behind the “fine-tune a small model to do what a prompted large one does” strategy from the start of this lesson.

Pruning and sparsity

Pruning deletes weights. The simplest rule, remove those with the smallest magnitude, works surprisingly well, especially if the network is retrained afterwards. For LLMs, retraining is expensive, so one-shot methods matter: SparseGPT pruned 175-billion-parameter models to 50–60% sparsity in under four and a half hours, without retraining, at a negligible cost in perplexity.

The catch is that zeros only save time if the hardware can skip them. The pattern of sparsity decides that:

Structured pruning of whole neurons, heads or layers, followed by distillation from the original model to repair the damage, has become a standard way to derive a family of smaller models from one large one.

Where does mixture of experts fit?

Mixture of experts is sparsity by design rather than by pruning. Each token is routed to a few of many expert sub-networks, so a model can hold a huge number of parameters while doing the arithmetic of a small one per token: gpt-oss-120b uses 5.1B of its 117B parameters for each token. It saves compute, not memory, which is why MoE models pair so naturally with 4-bit weights. The LLM lesson compares dense and MoE costs in detail.

Key takeaways

• Distillation trains a small student on a teacher’s temperature-softened probabilities, which carry more information than hard labels.
• Pruning removes low-importance weights; it only speeds inference when the sparsity has a structure the hardware can exploit.
• Mixture of experts is conditional computation: many parameters stored, few used per token, saving compute rather than memory.

Evaluation, risks and on-device

A fine-tuned, quantized model can look better on the metric you trained for and be worse in ways nobody measured. Evaluation and safety are not the last step of adaptation; they decide whether it was worth doing.

What can go wrong?

• Safety regressions. Qi and colleagues (2023) fine-tuned GPT-3.5 Turbo on just 10 harmful examples, costing under $0.20 through the public fine-tuning API, and largely removed its safety behaviour. More worrying for honest users, fine-tuning on entirely benign data also measurably weakened the model's refusals. Safety training is a thin layer, and fine-tuning can wear it away.
• Overfitting to a small dataset. With a few hundred examples, a model can memorise phrasings, become repetitive, and lose the variety and general ability that made the base model worth adapting.
• Forgetting. Skills unrelated to your task can degrade, as the forgetting demo showed. You only find out if you test for them.
• Compression damage in the tails. Averages hide it. A quantized model may keep its benchmark score and still fail on long documents, rare languages or careful arithmetic.

How should you evaluate an adapted model?

1. Build the evaluation set before training, from real cases, and never train on it.
2. Measure the baseline you are trying to beat: the base model with your best prompt, and a frontier model if that is the alternative.
3. Keep a regression suite of general abilities (instruction following, reasoning, other languages) and run it on every checkpoint.
4. Re-run safety evaluations after fine-tuning, using harmful-request benchmarks relevant to your product.
5. Evaluate the model you will actually ship: quantized, with the real prompt template and decoding settings.

Why does on-device matter so much now?

Everything in this lesson converges on small devices. Apple's 2025 on-device model combines about 3 billion parameters, 2-bit quantization-aware training and developer-trainable LoRA adapters. Gemma 3n is engineered to run in 2 to 3 GB of phone memory. gpt-oss-20b brings a reasoning model to a 16 GB laptop. For users, that means features that work offline and data that never leaves the device; for developers, it means a model per task at close to zero marginal cost.

The trade-off does not disappear. A phone-sized model is weaker than a frontier model at open-ended reasoning and knowledge, which is why shipping systems are increasingly hybrid: a small local model for the frequent, simple, private requests, and a large remote one when the task demands it.

Key takeaways

• Fine-tuning can erode safety training, even on benign data: re-run safety evaluations on every adapted model.
• Evaluate against a prompted baseline, keep a regression suite of general skills, and test the exact quantized model you ship.
• On-device models combine small size, low-bit weights and adapters, and are usually paired with a larger remote model for hard requests.

Check your understanding

Seven situations you could meet when adapting or shrinking a model. Each asks what you would actually do.

A company wants its support assistant to answer questions about a product catalogue that changes every week. A team member proposes fine-tuning an open model on the catalogue each week. What would you recommend?

References

The primary sources behind this lesson. The memory calculator uses the published Llama 3 architectures and the byte accounting from the ZeRO and QLoRA papers; every demo runs its computation live in your browser. For how base models are pretrained and aligned in the first place, see Large Language Models.

Sources

1. [1]

   LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report(opens in a new tab)

   Zhao, J. et al. (Predibase), 2024

   310 models: 10 base models × 31 tasks, fine-tuned with 4-bit LoRA. On average they beat their base models by 34 points and GPT-4 by 10 points on those narrow tasks.

2. [2]

   Apple Intelligence Foundation Language Models: Tech Report 2025(opens in a new tab)

   Apple Machine Learning Research, 2025

   Describes Apple’s roughly 3B-parameter on-device model, compressed with 2-bit quantization-aware training and KV-cache sharing, and the Foundation Models framework that lets developers train LoRA adapters for it.

3. [3]

   gpt-oss-120b & gpt-oss-20b Model Card(opens in a new tab)

   OpenAI, 2025

   Two open-weight mixture-of-experts reasoning models under Apache 2.0: 117B parameters (5.1B active) and 21B (3.6B active), with MoE weights stored in the 4-bit MXFP4 format so they run on one 80 GB GPU or in 16 GB of memory.

4. [4]

   Introducing Gemma 3n: The developer guide(opens in a new tab)

   Google Developers Blog, 2025

   Gemma 3n E2B and E4B have 5B and 8B raw parameters but, through per-layer embeddings and the MatFormer architecture, run with memory footprints comparable to 2B and 4B models: as little as 2 GB and 3 GB.

5. [5]

   Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs(opens in a new tab)

   Ovadia, O., Brief, M., Mishaeli, M., Elisha, O., 2023

   Finds that retrieval-augmented generation consistently outperforms unsupervised fine-tuning for injecting both existing and genuinely new knowledge into LLMs.

6. [6]

   ZeRO: Memory Optimizations Toward Training Trillion Parameter Models(opens in a new tab)

   Rajbhandari, S., Rasley, J., Ruwase, O., He, Y., 2019

   Breaks down training memory: mixed-precision Adam needs 16 bytes per parameter (fp16 weights and gradients plus fp32 master weights, momentum and variance) before activations.

7. [7]

   Overcoming catastrophic forgetting in neural networks(opens in a new tab)

   Kirkpatrick, J. et al., 2017

   Elastic weight consolidation: slow down learning on weights that mattered for earlier tasks, using a quadratic penalty weighted by the Fisher information. PNAS 114(13).

8. [8]

   LoRA Learns Less and Forgets Less(opens in a new tab)

   Biderman, D. et al., 2024

   On code and maths, LoRA underperforms full fine-tuning on the target domain but better preserves the base model’s abilities elsewhere; full fine-tuning learns weight updates of much higher rank than typical LoRA configurations. TMLR 2024.

9. [9]

   LoRA: Low-Rank Adaptation of Large Language Models(opens in a new tab)

   Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., 2021

   Freezes the pretrained weights and learns a low-rank update BA for chosen matrices. On GPT-3 175B it cut trainable parameters by 10,000× and GPU memory by 3× versus full fine-tuning with Adam, with no added inference latency. ICLR 2022.

10. [10]

    QLoRA: Efficient Finetuning of Quantized LLMs(opens in a new tab)

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L., 2023

    Backpropagates through a frozen 4-bit (NF4) model into LoRA adapters, with double quantization and paged optimizers, cutting the memory to fine-tune a 65B model from over 780 GB to under 48 GB. NeurIPS 2023.

11. [11]

    Parameter-Efficient Transfer Learning for NLP(opens in a new tab)

    Houlsby, N. et al., 2019

    Introduces adapter modules: small bottleneck layers inserted into each transformer block. On GLUE, adapters came within 0.4% of full fine-tuning while training 3.6% of the parameters per task. ICML 2019.

12. [12]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale(opens in a new tab)

    Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L., 2022

    Shows that large-magnitude outlier features emerge in transformers at around 6.7B parameters and break naive 8-bit quantization; a mixed-precision decomposition keeps those dimensions in 16 bits. NeurIPS 2022.

13. [13]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers(opens in a new tab)

    Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D., 2022

    One-shot 3- to 4-bit weight quantization using approximate second-order information to compensate for rounding error; quantizes 175B-parameter models in about four GPU hours. ICLR 2023.

14. [14]

    Distilling the Knowledge in a Neural Network(opens in a new tab)

    Hinton, G., Vinyals, O., Dean, J., 2015

    Trains a small model on the temperature-softened output distribution of a large one (soft targets), transferring the “dark knowledge” in the relative probabilities of wrong answers.

15. [15]

    SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot(opens in a new tab)

    Frantar, E., Alistarh, D., 2023

    Prunes OPT-175B and BLOOM-176B to 50–60% unstructured sparsity in under 4.5 hours without retraining, with negligible increase in perplexity. ICML 2023.

16. [16]

    Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!(opens in a new tab)

    Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., Henderson, P., 2023

    Fine-tuning GPT-3.5 Turbo on just 10 harmful examples, at a cost under $0.20 through the public API, removed its safety guardrails; fine-tuning on benign data also degraded safety. ICLR 2024.