Logistic regression is a machine learning algorithm used for classification problems. The term logistic is derived from the cost function (logistic function) which is a type of sigmoid function known for its characteristic S-shaped curve. A logistic regression model predicts probability values which are mapped to two (binary classification) or more (multiclass classification) classes.
1 = the curve's maximum value
S(z) = output between 0 and 1 (probability estimate)
z = the input
e = base of natural log (also known as Euler's number)
In multiclass classification with logistic regression, a softmax function is used instead of the sigmoid function.
The decision boundary is the acceptable threshold at which a probability can be mapped to a discrete class e.g. pass/fail or vegan/vegetarian/omnivore.
The cost function in logistic regression is more complex than linear regression. For example, mean squared error would yield a non-convex function with many local minimums, making it difficult to optimize with gradient descent. Cross entropy, also called log loss is the cost function used with logistic regression.
Regularization is a technique used to prevent overfitting by penalizing signals that provide too much explanatory power to a single feature. Regularization is extremely important in logistic regression.
Linear regression predictions are continuous (e.g. test scores from 0-100).
Logistic regression predictions classify items where only specific values or classes are allowed (e.g. binary classification or multiclass classification). The model provides a probability score (confidence) with each prediction.