Linear and Logistic Regression

The Model

We seek a straight-line relationship between one input feature $x$ and one output $y$:

where $w$ is the weight (slope) and $b$ is the bias (intercept). Given $m$ training examples ${(x^{(1)}, y^{(1)}), \ldots, (x^{(m)}, y^{(m)})}$, we want to find $w$ and $b$ that minimise the prediction error.

Cost Function (Mean Squared Error)

The cost function measures how far our predictions are from the actual values:

The factor of $\frac{1}{2}$ is a convenience that simplifies the derivative. Our goal is to minimise $J(w, b)$.

Gradient Descent

Gradient descent iteratively updates $w$ and $b$ in the direction of steepest descent:

where $\alpha$ is the learning rate. The partial derivatives are:

Extending to Multiple Features

When we have $n$ features $x_1, x_2, \ldots, x_n$, the model becomes:

In matrix notation, for all $m$ training examples:

where $X$ is the $m \times n$ feature matrix.

The Normal Equation

For linear regression, there is a closed-form solution that gives the optimal weights directly (no iterations needed):

This works when $X^T X$ is invertible. For large datasets or many features, gradient descent is more efficient computationally.

Mean Squared Error (MSE)

Penalises large errors heavily due to squaring.

Mean Absolute Error (MAE)

More robust to outliers than MSE.

R-Squared (Coefficient of Determination)

$R^2 = 1$ means a perfect fit; $R^2 = 0$ means the model is no better than predicting the mean. Values near 1 are good.

The Sigmoid Function

Logistic regression is used for binary classification (two classes). Instead of predicting a continuous value, we predict the probability that an observation belongs to class 1:

The sigmoid function maps any real number to the range $(0, 1)$. If $\hat{y} \geq 0.5$, predict class 1; otherwise, predict class 0.

Cross-Entropy Loss

The cost function for logistic regression is the binary cross-entropy:

When $y = 1$, the loss is $-\log(\hat{y})$, which penalises predictions near 0 heavily. When $y = 0$, the loss is $-\log(1-\hat{y})$, which penalises predictions near 1.

Decision Boundary

The decision boundary is the surface where $\hat{y} = 0.5$, i.e., where $z = \mathbf{w}^T\mathbf{x} + b = 0$. For two features, this is a straight line in the feature space.

Geoscience Applications

Linear regression: Predicting porosity from depth, estimating reservoir pressure from well-test data, computing velocity from offset in seismic refraction surveys.

Logistic regression: Classifying rock types (sandstone vs. shale) from well logs, predicting whether a well will be economic (yes/no), identifying fault presence from seismic attributes.

Beyond Straight Lines

When the relationship between $x$ and $y$ is non-linear, we can extend linear regression by adding polynomial features:

Despite the non-linear relationship with $x$, this is still "linear" in the parameters $w_0, \ldots, w_d$, so we can use the same least-squares machinery.

When to use: When scatter plots reveal curvature (e.g., porosity-permeability cross-plots often follow a power law). Start with $d = 2$ or $d = 3$ and increase cautiously.

Overfitting risk: High-degree polynomials fit the training data very well but oscillate wildly between data points. A polynomial of degree $d = m - 1$ (where $m$ is the number of training points) passes through every point but generalises terribly. Always evaluate on a held-out test set.

Ridge Regression (L2 Regularization)

Ridge regression adds a penalty proportional to the squared magnitude of the weights to the cost function:

The hyperparameter $\lambda > 0$ controls the strength of regularization. Larger $\lambda$ shrinks weights toward zero, producing a simpler model. Ridge regression reduces overfitting and handles multicollinearity.

Lasso Regression (L1 Regularization)

Lasso uses the sum of absolute weights as the penalty:

A key advantage of Lasso over Ridge: Lasso can drive some weights to exactly zero, effectively performing feature selection. If you have 50 well-log features but only 5 are truly relevant, Lasso will zero out the irrelevant ones.

Elastic Net combines L1 and L2: $J + \lambda_1 \sum|w_j| + \lambda_2 \sum w_j^2$. This gets the best of both worlds — sparsity from L1 and stability from L2.

Extending to More Than Two Classes

Binary logistic regression handles two classes. For $K > 2$ classes, we use the softmax function:

Each class $k$ has its own weight vector $\mathbf{w}_k$ and bias $b_k$. The softmax ensures all probabilities sum to 1. The predicted class is $\hat{y} = \arg\max_k P(y = k | \mathbf{x})$.

The loss function generalizes to categorical cross-entropy:

In geoscience, softmax regression classifies well-log data into multiple lithofacies (sandstone, shale, limestone, dolomite, etc.) simultaneously.

Beyond Accuracy: Confusion Matrix and Derived Metrics

Accuracy alone is misleading when classes are imbalanced (e.g., 95% shale, 5% sandstone). A model predicting "shale always" gets 95% accuracy but is useless.

Confusion Matrix: A $K \times K$ table where entry $(i, j)$ counts samples with true class $i$ predicted as class $j$.

Precision (of predicted positives, how many are correct): $\text{Precision} = \frac{TP}{TP + FP}$

Recall (of actual positives, how many are found): $\text{Recall} = \frac{TP}{TP + FN}$

F1-Score (harmonic mean balancing precision and recall): $F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$

In geoscience, recall is often critical: missing a sandstone reservoir (false negative) is more costly than a false alarm. Choose the metric that aligns with the geological decision being made.

Geoscience Applications (Extended)

Reservoir property prediction: Multiple linear regression predicts porosity, permeability, or water saturation from suites of well-log curves (GR, RHOB, NPHI, RT). Polynomial terms capture non-linear responses.

Lithology classification: Softmax logistic regression classifies well-log data into 5-10 lithofacies. Feature engineering (e.g., GR/RHOB ratio) and regularization improve results.

Feature selection for reservoir models: Lasso regression identifies which seismic attributes are most predictive of reservoir thickness, automatically zeroing out uninformative attributes.