Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE) is a fundamental method in statistical inference used to estimate the parameters of a statistical model. It is widely used in data science and machine learning because of its theoretical properties and practical applicability. This article dives deep into the principles of MLE, explores its properties, and provides detailed examples of how to apply it to estimate parameters in various statistical models.

Understanding Maximum Likelihood Estimation (MLE)

What is MLE?

Maximum Likelihood Estimation is a method for estimating the parameters of a statistical model by finding the parameter values that maximize the likelihood function. The likelihood function measures how plausible the observed data is given a set of parameter values.

Mathematically, if we have a statistical model with a parameter $\theta$ and a set of observed data $X = \{x_1, x_2, \dots, x_n\}$, the likelihood function is defined as:

$$L(\theta; X) = P(X | \theta)$$

The goal of MLE is to find the parameter value $\hat{\theta}$ that maximizes the likelihood function:

$$\hat{\theta} = \underset{\theta}{\text{argmax}} \, L(\theta; X)$$

Assumptions:

• The likelihood must match the sampling design, dependence structure, censoring, and support of the data. The examples below use independent and identically distributed observations, which lets the joint density factor into a product, but MLE also applies to dependent observations when their joint likelihood is specified.

Likelihood vs. Probability

It’s important to distinguish between likelihood and probability:

• Probability ($P(X | \theta)$): The probability of observing the data $X$ given specific parameter values $\theta$. It is used to predict future data based on the model.

• Likelihood ($L(\theta | X)$): A function of the parameters $\theta$ given the observed data $X$. It represents how plausible the parameters are in explaining the observed data.

While probability is used to predict future data given a model, likelihood is used to infer model parameters from observed data.

Log-Likelihood

To simplify the maximization process, especially when dealing with products of probabilities, it’s common to work with the log-likelihood function. The log-likelihood is the natural logarithm of the likelihood function:

$$\ell(\theta; X) = \log L(\theta; X) = \sum_{i=1}^n \log P(x_i | \theta)$$

Maximizing the log-likelihood function $\ell(\theta; X)$ gives the same result as maximizing the likelihood function $L(\theta; X)$, but with the advantage of turning products into sums, making the calculations easier and numerically more stable.

Properties of Maximum Likelihood Estimators

MLE has several desirable properties that make it a powerful method in statistical inference:

1. Consistency:

   • Under identifiability and other regularity conditions, the MLE converges to the data-generating parameter as the sample size increases. Misspecification or non-identifiability can invalidate this claim.

2. Asymptotic Normality:

   • For large samples, the distribution of the MLE is approximately normal (Gaussian) with a mean equal to the true parameter value and a variance equal to the inverse of the Fisher information. This allows for the construction of confidence intervals and hypothesis tests.

3. Asymptotic Efficiency:

   • Under regularity conditions, the MLE asymptotically attains the information bound. A finite-sample MLE need not be unbiased or minimum variance.

4. Invariance:

   • If $\hat{\theta}$ is the MLE for $\theta$, and $\phi = g(\theta)$ is a function of $\theta$, then the MLE for $\phi$ is $g(\hat{\theta})$. This property makes MLEs easy to work with when transforming parameters.

   Invariance Example:

   Suppose $\hat{\theta}$ is the MLE for $\theta$, and you are interested in estimating $\phi = \theta^2$. According to the invariance property, the MLE for $\phi$ is $\hat{\phi} = \hat{\theta}^2$.

5. Regularity Conditions:

   • These properties require more than differentiability: the model must be correctly specified and identifiable, the true parameter should not be on a problematic boundary, and information must behave regularly. Mixtures, change points, and boundary tests are common exceptions.

Applying MLE to Estimate Parameters

Let’s explore how to apply MLE to estimate parameters in various statistical models through detailed examples.

Example 1: Estimating the Mean of a Normal Distribution

Problem Setup

Suppose you have a set of data points that you believe are drawn from a normal distribution with an unknown mean $\mu$ and a known variance $\sigma^2 = 4$. Your goal is to estimate the mean $\mu$ using MLE.

Step 1: Write Down the Likelihood Function

The likelihood function for the normal distribution is given by:

$$L(\mu; X) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)$$

Given that $\sigma^2$ is known and constant, we can focus on the part of the likelihood that depends on $\mu$:

$$L(\mu; X) = \prod_{i=1}^n \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)$$

Step 2: Simplify Using the Log-Likelihood

Taking the logarithm of the likelihood function to get the log-likelihood:

$$\ell(\mu; X) = \log L(\mu; X) = -\frac{n}{2} \log(2\pi\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2$$

The first term is constant with respect to $\mu$, so we can ignore it when maximizing. Focus on the second term:

$$\ell(\mu; X) = -\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2$$

Step 3: Maximize the Log-Likelihood

To find the value of $\mu$ that maximizes the log-likelihood, take the derivative with respect to $\mu$ and set it to zero:

$$\frac{d\ell(\mu; X)}{d\mu} = \frac{1}{\sigma^2} \sum_{i=1}^n (x_i - \mu) = 0$$

This simplifies to:

$$\sum_{i=1}^n (x_i - \mu) = 0$$

Which further simplifies to:

$$\mu = \frac{1}{n} \sum_{i=1}^n x_i$$

Thus, the MLE for $\mu$ is the sample mean:

$$\hat{\mu} = \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$$

Step 4: Interpretation

The MLE for the mean $\mu$ of a normal distribution with known variance is simply the arithmetic mean of the observed data. This result is intuitive and aligns with our understanding that the sample mean is a good estimator for the population mean.

Example 2: Estimating the Success Probability in a Binomial Distribution

Problem Setup

Suppose you are running an experiment where you flip a coin $n$ times and observe $k$ heads. You want to estimate the probability $p$ of getting heads using MLE.

Step 1: Write Down the Likelihood Function

The likelihood function for a Binomial distribution, where $k$ successes are observed out of $n$ trials, is given by:

$$L(p; X) = \binom{n}{k} p^k (1-p)^{n-k}$$

Step 2: Simplify Using the Log-Likelihood

Taking the logarithm of the likelihood function to get the log-likelihood:

$$\ell(p; X) = \log L(p; X) = \log \binom{n}{k} + k \log(p) + (n-k) \log(1-p)$$

Again, the first term is constant with respect to $p$, so we can focus on the second and third terms:

$$\ell(p; X) = k \log(p) + (n-k) \log(1-p)$$

Step 3: Maximize the Log-Likelihood

To find the value of $p$ that maximizes the log-likelihood, take the derivative with respect to $p$ and set it to zero:

$$\frac{d\ell(p; X)}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0$$

Solving for $p$:

$$\frac{k}{p} = \frac{n-k}{1-p}$$

This simplifies to:

$$p = \frac{k}{n}$$

Thus, the MLE for $p$ is the sample proportion:

$$\hat{p} = \frac{k}{n}$$

Step 4: Interpretation

The MLE for the probability of success $p$ in a binomial distribution is the observed proportion of successes. This result makes intuitive sense and is widely used in estimating probabilities from binary data.

Example 3: Estimating the Rate Parameter of a Poisson Distribution

Problem Setup

Suppose you are observing the number of events that occur in each of $n$ fixed intervals of time, and you believe the number of events in each interval follows a Poisson distribution with an unknown rate parameter $\lambda$. You observe $k$ events in total across all intervals. You want to estimate $\lambda$ using MLE.

Step 1: Write Down the Likelihood Function

The likelihood function for the Poisson distribution, where $x_i$ events are observed in each interval, is given by:

$$L(\lambda; X) = \prod_{i=1}^n \frac{\lambda^{x_i} e^{-\lambda}}{x_i!}$$

Where $x_i$ is the number of events observed in each interval.

Step 2: Simplify Using the Log-Likelihood

Taking the logarithm of the likelihood function to get the log-likelihood:

$$\ell(\lambda; X) = \sum_{i=1}^n \left( x_i \log(\lambda) - \lambda - \log(x_i!) \right)$$

Ignoring the constant term $\sum_{i=1}^n \log(x_i!)$, the log-likelihood simplifies to:

$$\ell(\lambda; X) = \sum_{i=1}^n \left( x_i \log(\lambda) - \lambda \right)$$

Step 3: Maximize the Log-Likelihood

To find the value of $\lambda$ that maximizes the log-likelihood, take the derivative with respect to $\lambda$ and set it to zero:

$$\frac{d\ell(\lambda; X)}{d\lambda} = \sum_{i=1}^n \left( \frac{x_i}{\lambda} - 1 \right) = 0$$

Simplifying this:

$$\sum_{i=1}^n \frac{x_i}{\lambda} = n$$ $$\lambda = \frac{1}{n} \sum_{i=1}^n x_i$$

Thus, the MLE for $\lambda$ is the sample mean of the observed data:

$$\hat{\lambda} = \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$$

Step 4: Interpretation

The MLE for the rate parameter $\lambda$ of a Poisson distribution is the average number of events observed per interval. This makes intuitive sense, as the sample mean is the best estimate of the expected number of events in each interval.

Conclusion

Maximum Likelihood Estimation (MLE) is a powerful and versatile method for estimating the parameters of statistical models. Its consistency, efficiency, and normal approximation are large-sample results that depend on correct specification and regularity conditions.

In this article, we explored the principles of MLE, discussed its important properties, and demonstrated how to apply it to estimate parameters in various statistical models, including the normal, binomial, and Poisson distributions. Through detailed examples, we showed how MLE works in practice and how it can be used to derive meaningful parameter estimates from data.

Understanding and applying MLE is essential for data scientists and statisticians, as it forms the basis for many advanced techniques in machine learning, econometrics, and beyond. By mastering MLE, you can enhance your ability to build and interpret statistical models, leading to better insights and more informed decisions based on data.