Large Language Models

Why LLMs matter

In 2020, a model that could write a passable paragraph was a research result. Six years later, more than 900 million people a week use ChatGPT alone, and large language models draft code, summarise contracts, tutor students and run multi-step tasks on their own. This lesson explains what is actually inside them.

What is a large language model?

A large language model (LLM) is a neural network trained to do one thing: given some text, predict what comes next. It reads a sequence of tokens (word pieces) and outputs a probability for every token in its vocabulary. Pick one, add it to the text, and ask again. Repeat a few hundred times and you have an essay, a function or an answer.

“Large” refers to two things: the number of learned parameters (billions to around a trillion) and the amount of text used to train them (trillions of tokens). Almost every capability you have seen from Claude, GPT, Gemini, Llama or DeepSeek comes from that simple objective, pushed to enormous scale, and then shaped by a few extra stages of training.

How did we get here so fast?

Three ingredients arrived at once. The transformer (2017) made it possible to train on huge amounts of text in parallel on GPUs. Researchers discovered that performance improves predictably as you scale up, which made it rational to spend hundreds of millions of dollars on a single training run. And a set of fine-tuning techniques turned raw text predictors into assistants that follow instructions.

The use is broad rather than niche. A large study of ChatGPT conversations found that by July 2025 it had been adopted by around 10% of the world's adults, that practical guidance, seeking information and writing made up nearly 80% of conversations, and that non-work messages had grown to more than 70% of use. People treat these systems as a general-purpose tool.

Why should you understand the inside?

Because the failure modes follow directly from the design. A system that predicts plausible text will sometimes produce plausible falsehoods. A system steered by a probability distribution will give different answers to the same question. A system that reads instructions and data in the same stream can be tricked by data that looks like instructions. Knowing the mechanism tells you when to trust the output, how to prompt it, and when a different kind of model is the better tool.

Key takeaways

• An LLM is a next-token predictor: it outputs a probability for every possible next token, and text is generated one token at a time.
• Scale (parameters, data, compute) plus post-training turned that simple objective into general-purpose assistants between 2020 and 2026.
• Its strengths and failures, from fluency to confident errors, follow from how it is built and trained.

Next-token prediction

Every LLM, however capable, generates text the same way: look at everything so far, produce a probability distribution over the next token, choose one, append it, and go round again.

What is a token?

Models do not see letters or words; they see integers, each standing for a chunk of text from a fixed vocabulary, typically 100,000 to 200,000 entries for recent models. Common words are a single token, rare words are split into pieces, and any byte sequence can still be represented, so the model never meets a truly unknown word.

The vocabulary is learned with byte-pair encoding (BPE). Start with individual characters. Count every adjacent pair in the training text, merge the most frequent pair into a new symbol, and repeat. Early merges capture fragments like “th” and “ the”; later ones capture whole words. The demo below runs that algorithm on the first two chapters of Alice in Wonderland.

With no merges every character is its own token. After a few hundred, common words from the book (“ the”, “ Alice”, “ said”) collapse to one token while unfamiliar words stay in pieces. Production tokenisers do the same on far more text and settle around 4 characters per token for English. That is why prices and context limits are quoted in tokens, and why models can struggle with tasks like counting the letters in a word: the letters are hidden inside tokens.

How does the model turn text into a prediction?

Each token is looked up in an embedding table to become a vector. A stack of transformer layers then lets every position gather information from earlier positions through attention, and refine it with feed-forward layers. At the last position, a final layer produces one score (a logit) per vocabulary entry, and the softmax turns scores into probabilities:

The softmax. z is the vector of logits, one per vocabulary token.

Training adjusts the parameters so that the probability assigned to the token that actually came next is as high as possible, averaged over trillions of positions. The loss is the cross-entropy, −log P(actual next token). Its exponential is the perplexity: a perplexity of 10 means the model is, on average, as unsure as if it were choosing among 10 equally likely tokens.

Here is the key insight: to predict the next token of a physics textbook well, it helps to know physics. To predict the next line of a proof, it helps to follow the proof. Next-token prediction on a broad enough corpus rewards learning facts, grammar, style and some amount of reasoning, because all of them reduce the loss.

How is a token chosen?

The model gives a distribution; a decoding strategy turns it into a choice. Always taking the most likely token (greedy decoding) sounds sensible but produces dull, repetitive text. Holtzman and colleagues showed that human writing is regularly less predictable than the model's top choice, and that pure sampling instead goes off the rails because thousands of individually unlikely tokens add up. Three knobs manage that trade-off:

• Temperature divides the logits by T before the softmax. T < 1 sharpens the distribution, T > 1 flattens it, T → 0 is greedy.
• Top-k keeps only the k most likely tokens.
• Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p, so the shortlist shrinks when the model is confident and grows when it is not.

The dashed outline is the model's own probability; the filled bar is what sampling actually uses after temperature and top-p. A tiny model like this one spreads its bets widely, so high temperatures quickly produce nonsense. Real LLMs are much sharper, but the arithmetic of sampling is exactly the same.

Key takeaways

• Text becomes tokens via byte-pair encoding: frequent strings get their own token, rare ones are built from pieces.
• The model outputs logits, the softmax turns them into probabilities, and training minimises cross-entropy on the actual next token.
• Temperature, top-k and top-p decide how that distribution becomes text; greedy is repetitive, unfiltered sampling is erratic.

Pretraining and scaling laws

Pretraining is where almost all the compute goes and almost all the knowledge comes from. Its most surprising discovery is that the results are predictable: loss falls along smooth curves as you add parameters, data and compute.

What happens in pretraining?

A randomly initialised transformer reads a vast, filtered mix of web pages, books, code, scientific papers and other text, predicting each next token and nudging its weights by gradient descent after every batch. The numbers are hard to picture. GPT-3 (2020) had 175 billion parameters and saw 300 billion tokens. Meta's Llama 3 405B (2024) saw 15.6 trillion tokens, roughly fifty times more.

A useful rule of thumb gives the cost: training takes about 6 × N × D floating-point operations, for N parameters and D tokens (2 for the forward pass, 4 for the backward pass, per parameter per token). For Llama 3 405B that is 6 × 405×10⁹ × 15.6×10¹² ≈ 3.8×10²⁵ FLOPs, exactly the figure Meta reports. At a sustained few hundred teraFLOPs per GPU, that is on the order of thirty million GPU-hours.

What are scaling laws?

Kaplan and colleagues at OpenAI (2020) trained hundreds of models and found that test loss follows a power law in each resource, as long as the other two are not the bottleneck: L(N) = (N₀ / N)^0.076. On a log-log plot a power law is a straight line. The exponent is small, which means progress is slow but reliable: every tenfold increase in parameters multiplies the loss by 10^−0.076 ≈ 0.84.

Two years later, DeepMind's Chinchilla paper asked a sharper question: with a fixed compute budget, how should you split it between a bigger model and more data? Their answer was that the two should grow roughly in proportion. Most large models of the time were too big and undertrained. To prove it, they trained Chinchilla, 70 billion parameters on 1.4 trillion tokens, with about the same compute as their 280-billion-parameter Gopher, which saw 300 billion tokens. Chinchilla was better on nearly every benchmark.

Two things stand out. First, the Kaplan lines are straight on log-log axes across the whole range the authors could test, which is why labs felt safe extrapolating. Second, the parametric fit matters. With Hoffmann's published constants, the optimum at Chinchilla's budget sits near 60 tokens per parameter. A 2024 replication re-fitted the same data and found constants that put the optimum at about 20 tokens per parameter, consistent with the paper's other two methods and with the famous “20 tokens per parameter” rule. Fitted laws are models of models: useful, but only as good as the fit.

Do abilities emerge suddenly?

Loss falls smoothly, yet some capabilities seem to switch on abruptly. Wei and colleagues (2022) documented dozens of tasks, such as multi-digit arithmetic, where small models score near zero and larger ones suddenly succeed, and called these emergent abilities. Schaeffer, Miranda and Koyejo (2023) offered a deflating explanation: many of those tasks are scored with all-or-nothing metrics like exact match. If each token of a ten-token answer is right with probability p, the whole answer is right with probability p¹⁰, which stays near zero and then shoots up even when p improves steadily.

Both views have survived. Many “jumps” do shrink under continuous metrics, but some capabilities still matter only when they cross a threshold: a proof that is 90% right is still wrong. The practical lesson is to look at the metric before believing the narrative.

Key takeaways

• Training cost is about 6 × parameters × tokens FLOPs; frontier pretraining runs now reach the order of 10²⁵ FLOPs and beyond.
• Loss follows smooth power laws in scale, and for a fixed budget model size and data should grow together (roughly 20 tokens per parameter).
• Apparent “emergent” jumps are often partly an artefact of all-or-nothing metrics; check how a capability is scored.

From base model to assistant

A freshly pretrained model is not an assistant. Ask it “What is the capital of France?” and it may continue with “What is the capital of Germany?”, because a list of quiz questions is a perfectly plausible continuation. Turning it into something helpful takes post-training.

What is instruction tuning?

Supervised fine-tuning (SFT) continues training on examples of the behaviour you want: a prompt, followed by a high-quality response written by a person or a strong model, formatted as a conversation. The objective is still next-token prediction, just on a curated dataset. The model learns the format of a dialogue, to answer rather than continue, and a default tone.

How does learning from human feedback work?

It is much easier for people to say which of two answers is better than to write the perfect answer. RLHF (reinforcement learning from human feedback) exploits that. In the InstructGPT recipe:

1. Sample several answers to the same prompt and have people rank them.
2. Train a reward model to predict those preferences. It uses the Bradley–Terry model: the probability that answer A beats B is σ(r_A − r_B), where r is the reward score and σ the logistic function.
3. Fine-tune the language model with reinforcement learning (PPO) to produce answers the reward model scores highly, with a penalty for drifting too far from the SFT model so it cannot exploit quirks of the reward model.

The effect was striking: people preferred answers from a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3, a model more than a hundred times larger. Post-training does not add much knowledge; it changes which of the model's latent behaviours comes out.

The RLHF objective: maximise reward while staying close (in KL divergence) to the reference model. β sets how close.

Two important refinements

The DPO loss for a prompt x with preferred answer y_w and rejected answer y_l.

Why does post-training matter so much?

It is where a model's character, safety behaviour and usefulness are set, and where many of its quirks are introduced. Optimising for human approval can teach a model to tell people what they want to hear (sycophancy), or to write long, confident answers because raters liked them. Choosing what to reward is choosing what the model becomes. The newest stage, reinforcement learning on problems with checkable answers, is covered in the reasoning section.

Key takeaways

• A base model continues text; supervised fine-tuning on demonstrations teaches it to answer in a conversational format.
• RLHF trains a reward model on human comparisons and optimises the LLM against it, with a KL penalty to stay near the original.
• Constitutional AI uses written principles and AI feedback; DPO skips the reward model and RL loop by optimising preferences directly.

Context, retrieval and tools

An LLM's weights are frozen after training. Everything it knows about your task, from the instructions to the documents to the results of a web search, has to arrive through one channel: the context window.

What is the context window?

It is the span of tokens the model can attend to at once: the system prompt, the conversation so far, any pasted or retrieved documents, and the answer being written. GPT-3 handled 2,048 tokens. Llama 3.1 handles 128,000, and several frontier models accept a million or more, enough for several novels or a mid-sized codebase.

How does a model learn from its prompt?

The most surprising finding of the GPT-3 paper was in-context learning: show a model a few examples of a task in the prompt and it performs the task on a new input, without any change to its weights. A prompt like this works:

A few-shot prompt. The model infers the pattern (translate English to French) from the examples and continues it.

The ability grew with scale: in that paper, few-shot performance improved much faster with model size than zero-shot performance. The mechanism is still studied, but the practical upshot is clear. A prompt is a program written in examples and instructions, and it is the cheapest way to adapt a model.

Long contexts have a cost. At every generation step the model attends to all earlier tokens, so it caches each layer's keys and values for every token already processed (the KV cache) rather than recomputing them. That cache grows linearly with context length and can outgrow the model itself.

At 128,000 tokens the 70B model needs about 42 GB of cache for one user, on top of 141 GB of weights. This is why long-context requests cost more, why providers cache repeated prompt prefixes, and why architectures such as grouped-query attention (sharing keys and values across heads, as Llama does with 8 KV heads instead of 64) exist.

Why do retrieval and tools matter?

A model's knowledge stops at its training cutoff and is fuzzy about rare facts. Retrieval-augmented generation fixes part of that: search a document store for passages relevant to the question and paste them into the context, so the model answers from evidence it can quote.

Tool use goes further. The model is trained to emit a structured call, such as {"tool": "get_weather", "city": "Leeds"}, instead of prose. The surrounding software runs the tool and appends the result to the context, and the model continues. Put that in a loop, with tools for search, code execution or a browser, and you have an agent. Standards such as the Model Context Protocol let one model connect to many tools the same way.

Long context does not make retrieval obsolete. Studies have repeatedly found that models use information less reliably when it is buried in the middle of a very long prompt, and every extra token costs money and time. Selecting the right few thousand tokens is often better than sending everything.

Key takeaways

• The context window is the model’s only working memory for a task; it has grown from 2K tokens (GPT-3) to 128K and beyond.
• In-context learning lets a model pick up a task from examples in the prompt, with no weight updates.
• Retrieval and tool calls feed fresh, checkable information into the context; long context has real memory and reliability costs.

Reasoning models

For five years, “scaling” meant bigger models and more training data. Since late 2024 there has been a second axis: let the model spend more computation, in the form of more tokens of thinking, on each question.

What is chain-of-thought?

A model generates one token per forward pass, so the amount of computation it can apply before committing to an answer is limited by how many tokens it writes. Wei and colleagues (2022) showed that including worked, step-by-step solutions in the prompt makes large models write out intermediate steps too, and that this sharply improves accuracy on arithmetic and multi-step word problems. Writing the steps gives the model somewhere to put intermediate results: the text becomes its scratchpad.

How are reasoning models trained?

Prompting asks for reasoning; reasoning models are trained for it. OpenAI's o1, announced in September 2024, was trained with reinforcement learning to produce long internal chains of thought before answering, and OpenAI reported that its accuracy kept improving both with more RL training and with more thinking time at inference.

DeepSeek-R1 (January 2025) published a recipe. Start from a strong base model; give it problems whose answers can be checked automatically (maths with a known result, code with unit tests); reward it only for a correct final answer in the right format; and let reinforcement learning (their GRPO algorithm) do the rest. Nobody showed it how to reason. In the R1-Zero experiment, pass@1 on the AIME 2024 maths competition rose from 15.6% to 71.0% during training, and the model's answers grew longer on their own, with spontaneous re-checking (“wait, let me reconsider”). Majority voting over 64 samples lifted it to 86.7%.

Progress since has been fast. In July 2025 reasoning systems from Google DeepMind and OpenAI both reached gold-medal level on the International Mathematical Olympiad. By 2026 every major lab ships reasoning models, usually with a setting that controls how long they think, and long-horizon reasoning is what makes today's coding and research agents workable.

Two ways to buy accuracy with compute at answer time.

When the right answer is the single most likely one, voting drives accuracy towards 100% even if a lone sample is usually wrong, because the wrong answers disagree with each other. When the model has a favourite wrong answer, voting locks it in. Extra compute amplifies whatever the model already believes; it does not create knowledge it lacks. That is also why verifiable rewards were so important for R1: they give the training signal something firmer than the model's own opinion.

Key takeaways

• Writing intermediate steps gives a model more computation per answer; chain-of-thought prompting exploits this.
• Reasoning models (o1, DeepSeek-R1 and successors) learn long reasoning through reinforcement learning on checkable problems.
• Test-time compute is a second scaling axis, but it amplifies what the model already knows; voting cannot fix a consistent mistake.

What LLMs get wrong

A fluent answer is not a correct one. The main ways LLMs fail are not random bugs; they follow from training a model to produce likely, approved-of text from whatever is in its context.

Why do models hallucinate?

A hallucination is a confident statement that is false: an invented citation, a wrong date, a function that does not exist. Pretraining rewards producing plausible text, and for a rare fact the plausible and the true can diverge. Kalai and colleagues (2025) add a second reason it persists: most benchmarks grade answers as simply right or wrong, and under that scoring a model that guesses always scores at least as well as one that says “I don't know”. We have, in effect, trained models to be good test-takers.

With a penalty of t/(1−t) points for a wrong answer, answering only pays when confidence exceeds t. Kalai and colleagues propose stating such thresholds in benchmark instructions so that honesty about uncertainty stops being punished. It is a fix to the incentives, not just to the model.

Other characteristic failures

How are LLMs evaluated?

There is no single score. Labs report a portfolio: knowledge and reasoning benchmarks (graduate-level science questions, competition maths), coding benchmarks built from real GitHub issues, agentic tasks measured by whether the job gets done, and human-preference leaderboards where people vote between two anonymous models. Each has weaknesses. Public test questions leak into training data (contamination), strong models saturate benchmarks within a year or two, and using an LLM as a judge brings its own biases, such as favouring longer answers.

The practical advice for anyone deploying a model: build a small evaluation set from your own real cases, with answers you trust, and measure on that. A model that tops a leaderboard can still be the wrong choice for your task.

Key takeaways

• Hallucinations come from optimising for plausible text, and persist because right-or-wrong grading rewards guessing over abstaining.
• Sycophancy and prompt injection follow from preference training and from mixing instructions with data in one context.
• Benchmarks leak and saturate; evaluate on your own representative cases before trusting a model with a task.

Open models and cost

Frontier capability used to be available only through a few companies' APIs. Since DeepSeek-R1, the gap between the best open-weights models and the best closed ones has narrowed sharply, and much of the contest has shifted to cost.

What is the difference between open and closed models?

How are models made cheaper to run?

Three techniques do most of the work, and most large models now combine them.

• Mixture of experts (MoE). Replace each feed-forward layer with many smaller “expert” networks and a router that sends each token to only a few of them. DeepSeek-V3 has 671 billion parameters but uses only 37 billion per token, and was trained in 2.788 million H800 GPU-hours. Kimi K2.5 (January 2026) pushes this to about a trillion parameters with 32 billion active.
• Quantisation. Store weights in 8 or 4 bits instead of 16. Memory and bandwidth halve or quarter, usually with a small loss in quality. This is what makes a capable model run on a laptop.
• Distillation. Train a small model to imitate a large one's outputs. DeepSeek released six small models distilled from R1 (1.5B to 70B parameters) that inherited much of its reasoning ability.

The mixture-of-experts trick is visible at once: DeepSeek-V3 needs more memory than the dense 405B model but only about a tenth of the arithmetic per token. Quantising to 4 bits cuts the memory by four, which is often the difference between one server and several. None of this changes what the model is; it changes who can afford to run it.

Not every problem needs a text generator

Many real uses of an LLM are really decisions: which queue should this ticket go to, is this document relevant, does this case need review? A generative model answers by writing text, token by token, which you then have to parse, and it does not naturally give you a trustworthy confidence. Decision models take the other route: they read the same messy input and return a choice from a fixed set, with a calibrated probability, in a single pass. TypeSafe's Jev, announced in September 2026, is a prominent new example of this category, positioned as a component inside software rather than a chat assistant.

The distinction is an old one in machine learning (generative versus discriminative models), and it is worth keeping in mind when choosing a tool: use an LLM when the output is language or open-ended; consider a classifier or decision model when the output is a label and the confidence matters.

Key takeaways

• Open-weights models can be run and fine-tuned privately; closed API models are usually strongest and simplest to use.
• Mixture of experts, quantisation and distillation cut the cost of running a model, often by large factors.
• For decisions with a fixed set of answers, a calibrated classifier or decision model can beat free-form text generation.

Check your understanding

Seven scenarios. Each one asks you to apply an idea from the lesson to a situation you could meet in practice.

You have a fixed compute budget for pretraining. Option A: a 280B-parameter model trained on 300B tokens. Option B: a 70B-parameter model trained on 1.4T tokens (about the same compute). Which does the Chinchilla analysis predict will perform better, and why?

References

The primary sources behind this lesson. The scaling-law demo evaluates the equations from Kaplan et al. (2020), Hoffmann et al. (2022) and Besiroglu et al. (2024) directly; the KV-cache and cost demos use architecture figures from the Llama 3, DeepSeek-V3 and Kimi K2.5 releases. For the maths of attention itself, continue to Transformers and Attention.

Sources

1. [1]

   Scaling AI for everyone(opens in a new tab)

   OpenAI, 2026

   OpenAI announcement (February 2026) reporting more than 900 million weekly ChatGPT users and over 50 million subscribers.

2. [2]

   How People Use ChatGPT(opens in a new tab)

   Chatterji, A., Cunningham, T., Deming, D. J., Hitzig, Z., Ong, C., Shan, C. Y., Wadman, K., 2025

   NBER Working Paper 34255. Privacy-preserving analysis of ChatGPT conversations from launch to July 2025.

3. [3]

   Language Models are Few-Shot Learners(opens in a new tab)

   Brown, T. et al., 2020

   The GPT-3 paper: a 175-billion-parameter model trained on 300 billion tokens, and the discovery that it can learn tasks from examples in its prompt.

4. [4]

   The Curious Case of Neural Text Degeneration(opens in a new tab)

   Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y., 2020

   Shows why greedy and beam search produce repetitive text and introduces nucleus (top-p) sampling. ICLR 2020.

5. [5]

   The Llama 3 Herd of Models(opens in a new tab)

   Grattafiori, A. et al. (Llama Team, AI @ Meta), 2024

   Technical report for Llama 3: a 405B dense transformer pretrained on 15.6T tokens with 3.8 × 10^25 FLOPs, with a 128K-token context.

6. [6]

   Scaling Laws for Neural Language Models(opens in a new tab)

   Kaplan, J. et al., 2020

   Finds that language-model loss falls as a smooth power law in parameters, data and compute over many orders of magnitude.

7. [7]

   Training Compute-Optimal Large Language Models(opens in a new tab)

   Hoffmann, J. et al., 2022

   The Chinchilla paper: for a fixed compute budget, parameters and training tokens should grow together; Chinchilla (70B, 1.4T tokens) beats Gopher (280B).

8. [8]

   Chinchilla Scaling: A replication attempt(opens in a new tab)

   Besiroglu, T., Erdil, E., Barnett, M., You, J., 2024

   Re-fits the Chinchilla parametric loss from the paper’s own data and finds constants consistent with roughly 20 tokens per parameter.

9. [9]

   Emergent Abilities of Large Language Models(opens in a new tab)

   Wei, J. et al., 2022

   Catalogues abilities that appear absent in small models and present in large ones. Transactions on Machine Learning Research.

10. [10]

    Are Emergent Abilities of Large Language Models a Mirage?(opens in a new tab)

    Schaeffer, R., Miranda, B., Koyejo, S., 2023

    Argues many apparent emergent jumps come from harsh, discontinuous metrics such as exact match; continuous metrics show smooth improvement. NeurIPS 2023.

11. [11]

    Training language models to follow instructions with human feedback(opens in a new tab)

    Ouyang, L. et al., 2022

    InstructGPT: supervised fine-tuning plus RLHF; people preferred a 1.3B InstructGPT model to the 175B GPT-3.

12. [12]

    Constitutional AI: Harmlessness from AI Feedback(opens in a new tab)

    Bai, Y. et al., 2022

    Anthropic’s method for training a harmless assistant using a written set of principles and AI-generated feedback instead of human harm labels.

13. [13]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model(opens in a new tab)

    Rafailov, R. et al., 2023

    Derives a simple classification-style loss that optimises the RLHF objective directly from preference pairs, without a separate reward model or RL loop.

14. [14]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models(opens in a new tab)

    Wei, J. et al., 2022

    Showing worked reasoning steps in the prompt substantially improves large models on arithmetic and multi-step problems. NeurIPS 2022.

15. [15]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning(opens in a new tab)

    DeepSeek-AI (Guo, D. et al.), 2025

    Reinforcement learning with rule-based rewards teaches a base model long chains of reasoning, self-checking included. A revised version appeared in Nature 645 (2025).

16. [16]

    Why Language Models Hallucinate(opens in a new tab)

    Kalai, A. T., Nachum, O., Vempala, S. S., Zhang, E., 2025

    Argues hallucinations arise like ordinary classification errors and persist because most benchmarks reward guessing over admitting uncertainty.

17. [17]

    Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection(opens in a new tab)

    Greshake, K. et al., 2023

    Demonstrates that instructions hidden in retrieved web pages, emails or documents can hijack LLM-powered applications.

18. [18]

    DeepSeek-V3 Technical Report(opens in a new tab)

    DeepSeek-AI, 2024

    A mixture-of-experts model with 671B total and 37B active parameters per token, trained on 14.8T tokens in 2.788M H800 GPU hours.