Logistic Regression

Yes-or-no answers, with a probability

Will this transaction turn out to be fraud? Will this patient be readmitted within 30 days? Will this user click the advert? A huge share of real prediction problems have a yes-or-no answer, and the model that statisticians, banks and hospitals reach for first is still logistic regression.

What is it?

Logistic regression is a classifier that outputs a probability. It computes a weighted sum of the inputs, exactly like linear regression, then squashes that score through an S-shaped curve, the sigmoid, so the answer always lands between 0 and 1. “P(fraud) = 0.83” is far more useful than a bare “fraud”, because you can decide what to do with an 83% risk.

Despite the name, it is used for classification. The “regression” is historical: it regresses the log-odds of the outcome on the inputs. Joseph Berkson used the logistic curve for dose-response experiments and coined the word “logit”, and David Cox’s 1958 paper established it as a general method for binary outcomes.

How does it work?

Four ideas, one per section of this lesson:

1. Score, then squash. z = w·x + b, then p = σ(z).
2. Log loss scores the probabilities against the true labels, punishing confident mistakes hardest.
3. Gradient descent adjusts w and b to lower the loss. The result is a straight decision boundary.
4. A threshold turns the probability into an action, and choosing it is a business decision, not a mathematical one.

Why does it matter?

Logistic regression is fast, hard to overfit with a handful of features, and its weights can be read and audited, which is why it remains the standard in credit scoring and clinical risk models. It is also the last layer of almost every neural network classifier: a network’s final sigmoid or softmax, trained with log loss, is logistic regression on features the network learned. Understand this model and you understand how deep classifiers produce their probabilities.

Key takeaways

• Logistic regression classifies by producing a probability between 0 and 1, not just a label.
• It is a weighted sum of inputs passed through a sigmoid, trained by minimising log loss.
• It is a workhorse in its own right and the output layer of most neural network classifiers.

The sigmoid: from score to probability

A weighted sum can be any number, from minus a million to plus a million. A probability must sit between 0 and 1. The sigmoid is the bridge.

What is it?

The sigmoid (or logistic function) maps any real number z to a value between 0 and 1. Large positive scores map close to 1, large negative scores close to 0, and a score of exactly 0 maps to one half.

The sigmoid. e is Euler’s number, about 2.718. The model’s score is z = w₁x₁ + w₂x₂ + … + b.

How does it work?

Read the curve backwards and you get the cleanest way to think about the model. If p is the probability, then p / (1 − p) is the odds (0.8 means odds of 4 to 1), and the logarithm of the odds, the logit, is exactly the score z. So logistic regression is a linear model for the log-odds:

Each unit of xᵢ adds wᵢ to the log-odds, which multiplies the odds by e^wᵢ.

That gives every weight a concrete meaning. A weight of 0.7 multiplies the odds by e^0.7 ≈ 2.0 per unit of its feature. A weight of 0 means the feature has no effect. A negative weight lowers the odds.

Why does it matter?

Because the output is a probability, it can be combined with costs, compared across cases and checked against reality: of all the emails given 0.9, about 90% should be spam. Logistic regression trained with log loss tends to be well calibrated in exactly this sense, which is not true of every classifier. The sigmoid also appears inside neural networks as an activation function, and its flat tails are the origin of the vanishing-gradient problem in deep networks.

Key takeaways

• The sigmoid σ(z) = 1 / (1 + e^−z) squashes any score into a probability; σ(0) = 0.5.
• The score z is the log-odds, so each weight adds to the log-odds and multiplies the odds by e^w.
• Probabilities, unlike bare labels, can be combined with costs and checked for calibration.

Log loss: scoring a probability

A weather forecaster who says “100% chance of sun” on a day it pours should lose more credibility than one who said 60%. Log loss is that intuition written as a formula.

What is it?

Log loss, also called binary cross-entropy, scores a predicted probability against the true label. For each example, take the probability the model gave to the correct answer and compute minus its natural logarithm. Average over the dataset.

Mean log loss over n examples. yᵢ is 1 or 0 and pᵢ is the predicted P(y = 1). Only one of the two terms is non-zero for each example.

How does it work?

A perfect, confident prediction (probability 1 on the truth) costs −log 1 = 0. A coin-flip prediction costs −log 0.5 ≈ 0.69. As the probability on the truth heads towards 0, the cost heads towards infinity. That asymmetry is the point: being confidently wrong is catastrophic, being uncertain is merely mediocre.

Minimising log loss is the same as maximum likelihood: choosing the weights under which the observed labels were most probable. That is the statistical justification for this particular loss, and it is how Cox and the textbooks derive it.

Gradient descent settles at a steepness of about 1.12 and a midpoint of about 4.76 hours, with mean log loss 0.316. Try to beat it by hand: you cannot, because log loss is convex and this is its single minimum. Notice that the best curve is not a cliff. The data overlap (a student passed after 3.5 hours and another failed after 6), and a steeper curve would make those two confident mistakes far more expensive.

Why does it matter?

Log loss is the training objective of nearly every classifier in deep learning, under its other name, cross-entropy. Language models are trained with exactly this loss over their vocabulary at every token. It is also a good evaluation metric when you care about the probabilities themselves, not just the labels: two models with the same accuracy can have very different log loss.

Key takeaways

• Log loss charges −log of the probability given to the true label: 0 when certain and right, about 0.69 for a coin flip, unbounded when certain and wrong.
• Minimising it is maximum likelihood, and for logistic regression the loss is convex, with one minimum.
• The same loss, called cross-entropy, trains neural network classifiers and language models.

The decision boundary and training

With two features you can see the whole model at once: a probability for every point of the plane, and a straight line where it crosses one half.

What is it?

The decision boundary is where the model is exactly undecided, P = 0.5. Since σ(z) = 0.5 only when z = 0, the boundary is the set of points where w·x + b = 0. With two features that is a straight line; with three, a flat plane; with more, a hyperplane. On one side the model predicts class 1, on the other class 0, and the probability changes smoothly as you cross it.

How does it work?

Geometry

The weight vector w points at right angles to the boundary, towards class 1. Its length sets how steep the sigmoid is across the plane: a long w means the probability goes from 0.1 to 0.9 over a short distance. The bias b slides the line along w without turning it.

Training

There is no formula for the best weights, so we use gradient descent. The gradient of log loss has a strikingly simple form: for each example, the error (p − y) times its input. Examples the model already gets right with confidence contribute almost nothing; confident mistakes pull hardest.

One step of batch gradient descent on mean log loss. η is the learning rate and pᵢ = σ(w·xᵢ + b). Compare the update for linear regression: it is the same shape.

From zero weights the boundary swings into place within a few dozen steps, then the weights keep lengthening slowly as the model grows more confident. With a learning rate of 0.05 it is still creeping after thousands of steps; at 5 it gets there fastest here, because log loss is convex with bounded curvature. Interesting detail: the lowest log loss does not always mean the highest accuracy. The loss cares about every probability, not just which side of the line a point falls on.

Why does it matter?

Knowing that the boundary is linear tells you when the model will fail and how to fix it: give it better features. It also explains a neural network’s output layer, which draws a linear boundary in the space of features computed by the layers below. And the update rule, error times input, is the seed of backpropagation.

Key takeaways

• The decision boundary is where w·x + b = 0: a straight line (hyperplane) at right angles to the weight vector.
• Gradient descent updates each weight by the average of (p − y) times its input; confident mistakes pull hardest.
• On separable data the weights grow forever, which an L2 penalty prevents.

Thresholds: from probability to decision

The model says P(fraud) = 0.3. Do you block the card? That depends on what a blocked honest customer costs you, compared with a missed fraud. The model cannot know that. You choose it.

What is it?

A threshold turns a probability into a label: predict positive when p ≥ t. The default t = 0.5 picks the more likely class, which minimises the number of mistakes only if every mistake costs the same.

How does it work?

Moving the threshold does not change the model or its probabilities. It only changes which of them count as positive. Lower t and you catch more true positives (higher recall) but flag more negatives by mistake (lower precision). Raise it and the reverse happens. If a false negative costs C_FN and a false positive costs C_FP, and the probabilities are calibrated, the cost-minimising threshold is t = C_FP / (C_FP + C_FN). A miss that is nine times worse than a false alarm gives t = 0.1.

No threshold gets every point right, because the classes overlap: some negatives score higher than some positives. All the threshold does is decide which mistakes you prefer. Plot recall against the false-positive rate for every threshold and you get the ROC curve, which summarises the model independently of any single cut-off.

Why does it matter?

Many real failures of deployed classifiers are threshold failures: a fraud model tuned for accuracy on data where 0.1% of transactions are fraud learns that “never fraud” is 99.9% accurate. Report the probabilities, choose the threshold from the costs, and check precision and recall at that threshold on held-out data.

Key takeaways

• A threshold turns probabilities into labels; 0.5 is only right when both kinds of mistake cost the same.
• Lowering it raises recall and usually lowers precision; raising it does the reverse.
• With calibrated probabilities, the best threshold is C_FP / (C_FP + C_FN).

Using logistic regression well

A few habits separate a logistic regression you can trust from one that quietly misleads.

What are the common extensions?

How do you read the weights?

With standardised features, a weight is the change in log-odds per standard deviation of that feature, and e^w is the odds ratio. Two cautions. First, correlated features share credit unpredictably, so a single weight can be small or even flip sign while the pair matters a lot. Second, a weight is an association learned from the data you had. It says nothing about what would happen if you intervened.

Why does it matter?

Logistic regression is the right first model for most tabular yes-or-no problems: it trains in milliseconds, gives calibrated probabilities, and sets the baseline any fancier model has to beat. When it falls short, the reason is nearly always a boundary that is not linear in your features, which is the gap trees, nearest neighbours and neural networks fill. Jurafsky and Martin’s free textbook is a good next step for the softmax version and its use in language processing.

Key takeaways

• Softmax regression extends the model to many classes; L1 and L2 penalties keep weights in check.
• Weights are log-odds per unit of a feature: compare them only on standardised features, and never read them as causes.
• It is the baseline to beat for tabular classification, and its limit is a linear boundary in the given features.

Check your understanding

Six situations from real projects. Each one asks you to apply an idea, not just recall it.

A hospital model outputs P(sepsis) = 0.30 for a patient. Missing sepsis is far worse than an unnecessary blood test. The default rule flags patients at P ≥ 0.5. What should change?

Next, compare a model that draws no boundary at all in k-Nearest Neighbours, or see logistic units stacked into layers in Neural Networks.

References

Papers and textbooks behind the claims and methods in this lesson.

References

1. [1]

   Application of the Logistic Function to Bio-Assay(opens in a new tab)

   Joseph Berkson, 1944

   Journal of the American Statistical Association 39(227). Proposed the logistic curve for dose-response data and coined the word “logit”.

2. [2]

   The Regression Analysis of Binary Sequences(opens in a new tab)

   David R. Cox, 1958

   Journal of the Royal Statistical Society, Series B 20(2). The paper usually credited with establishing logistic regression for binary outcomes.

3. [3]

   Predicting Good Probabilities with Supervised Learning(opens in a new tab)

   Alexandru Niculescu-Mizil & Rich Caruana, 2005

   ICML 2005. Compares how well different classifiers’ scores work as probabilities; logistic regression is well calibrated out of the box.

4. [4]

   The Elements of Statistical Learning (2nd edition), chapter 4(opens in a new tab)

   Trevor Hastie, Robert Tibshirani & Jerome Friedman, 2009

   Springer. Linear methods for classification, including fitting logistic regression by maximum likelihood. Free PDF from the authors.

5. [5]

   Probabilistic Machine Learning: An Introduction(opens in a new tab)

   Kevin P. Murphy, 2022

   MIT Press. Logistic regression as a probabilistic model, including why the maximum-likelihood weights diverge on separable data.

6. [6]

   Speech and Language Processing (3rd edition draft), chapter on logistic regression(opens in a new tab)

   Daniel Jurafsky & James H. Martin, 2024

   A clear, free textbook treatment: sigmoid, cross-entropy loss, gradient descent, regularisation and softmax regression.