Equivalence of MLE and OLS in linear regression

Introduction

A linear regression model can be described using:

This simplifies to the following form on the observed data:

Equation 1

After obtaining the maximum likelihood estimate for the coefficients using the sample, this equation simplifies to:

The objective of this (short) article is to use the assumptions to establish the equivalence of OLS and MLE solutions for linear regression.

Important Model Assumptions

1. True underlying distribution of the errors is Gaussian
2. Expected value of the error term is 0 (known)
3. Variance of the error term is constant with respect to x
4. The ‘lagged’ errors are independent of each other

Full Likelihood

For an observation e from a Gaussian distribution with 0 mean and constant variance, the likelihood is given by:

Given the whole data set of n observations, assuming the residues are realizations of the iid (independent and identically distributed) Gaussian error, the likelihood can be written as:

Since log is a monotonous transformation, the maximum likelihood estimate does not change on log transformation:

Substituting the maximum likelihood estimate:

Removing the constant terms:

Substituting e from equation 1, we get:

Maximizing -z is equivalent to minimizing z, therefore:

All Assumptions

1. Relationship between independent variable and dependent variables is linear
2. True underlying distribution of the error is Gaussian with 0 mean
3. Independent variables do not exhibit high level of multicollinearity
4. No autocorrelation: ‘lagged’ error terms are independent
5. No heteroskedasticity (already used): variance of the error is independent of X and is constant throughout
6. Multivariate normality of independent variables (not required, but helpful) for proving few special properties
7. The independent variables are measured without random error. Therefore, X and x are not random

Additional Resources

1. Equivalence of ANOVA and linear regression
2. Simple physics for an intuitive understanding of linear regression