MMSE Minimum Mean Square Error Estimator

Last updated on May 4, 2023

The Minimum Mean Square Error (MMSE) estimator is a method commonly used in signal processing and estimation theory. It is used to estimate a signal or parameter of a random process based on observations that are corrupted by noise. The MMSE estimator is designed to minimize the mean square error (MSE) between the estimated signal and the true signal, thus providing a measure of how close the estimate is to the true signal.

In this article, we will explain the MMSE estimator, including its mathematical formulation, assumptions, and applications.

Mathematical Formulation

Suppose we have a random process with an unknown signal or parameter that we wish to estimate. The process is corrupted by noise, and we observe a set of measurements or observations. Let x denote the observations, and let s denote the true signal or parameter.

Our goal is to find an estimator, denoted as s_hat, that minimizes the mean square error (MSE) between the estimate and the true signal or parameter. The MSE is defined as:

MSE = E[(s - s_hat)^2]

where E denotes the expectation operator.

The MMSE estimator is designed to minimize the MSE. To derive the MMSE estimator, we first assume that the observations x are a linear combination of the true signal s and a noise term w, i.e.,

x = s + w

where w is a zero-mean Gaussian random variable with variance sigma^2. This assumption is common in many applications, including communications, control systems, and signal processing.

Next, we define the estimator s_hat as a linear combination of the observations x, i.e.,

s_hat = a^T x

where a is a vector of weights that we need to determine. The goal is to find the vector a that minimizes the MSE.

To do this, we first compute the conditional mean of the true signal s given the observations x, denoted as s_bar. This is given by:

s_bar = E[s | x]

Using Bayes' theorem, we can express s_bar in terms of the probability density functions (PDFs) of s and x, as follows:

s_bar = integral(s * f(s | x) ds)

where f(s | x) is the conditional PDF of s given x.

We can simplify this expression by assuming that the PDFs of s and w are Gaussian. In this case, the conditional PDF of s given x is also Gaussian, with mean mu and variance sigma^2, given by:

mu = E[s | x] = s + (sigma^2 / (sigma^2 + sigma_w^2)) * (x - s) sigma^2 = E[(s - mu)^2 | x] = (sigma^2 * sigma_w^2) / (sigma^2 + sigma_w^2)

where sigma_w^2 is the variance of the noise term w.

Using this expression for mu, we can write the MSE as:

MSE = E[(s - s_hat)^2] = E[(s - a^T x)^2] = E[(s - mu + mu - a^T x)^2] = E[(s - mu)^2] + E[(mu - a^T x)^2] + 2 E[(s - mu)(mu - a^T x)] = sigma^2 + E[(mu - a^T x)^2]

The first term sigma^2 is the variance of the true signal s, and the second term E[(mu - a^T x)^2] is the variance of the estimation error. The third term 2 E[(s - mu)(mu - a^T x)] is zero because s and mu are independent of mu - a^T x.

To minimize the MSE, we need to minimize the second term, i.e., the variance of the estimation error. This can be done by choosing the vector a that satisfies:

a = argmin(E[(mu - a^T x)^2])

This is equivalent to finding the vector a that maximizes the correlation between the conditional mean mu and the observations x. This can be written as:

a = (sigma^2 / (sigma^2 + sigma_w^2)) * R_x_s

where R_x_s is the correlation matrix between the observations x and the true signal s, given by:

R_x_s = E[x s^T] = E[(s + w) s^T] = E[s s^T] + E[w s^T] = R_s + 0 = R_s

where R_s is the autocorrelation matrix of the true signal s.

Substituting the expression for a into the estimator s_hat, we obtain the MMSE estimator:

s_hat = (sigma^2 / (sigma^2 + sigma_w^2)) * R_x_s^T x

This estimator is linear in the observations x and has a simple closed-form expression. It is also optimal in the sense that it minimizes the MSE, given the assumptions we made about the random process.

Assumptions

The MMSE estimator relies on several assumptions about the random process and the noise. These include:

Linearity: The observations x are a linear combination of the true signal s and a noise term w.
Gaussianity: The noise term w is a zero-mean Gaussian random variable with variance sigma^2.
Stationarity: The autocorrelation matrix R_s of the true signal s is time-invariant.
Known parameters: The variance sigma^2 of the noise term w and the autocorrelation matrix R_s of the true signal s are known.

These assumptions are common in many applications, but they may not always hold in practice. In particular, the Gaussianity assumption may be too restrictive, and non-Gaussian noise models may be needed.

Applications

The MMSE estimator has many applications in signal processing and estimation theory. Some examples include:

Channel estimation: In wireless communications, the MMSE estimator is used to estimate the channel response from the received signal, which is corrupted by noise and interference.
Adaptive filtering: The MMSE estimator can be used in adaptive filtering algorithms to estimate the coefficients of a filter that optimally separates a desired signal from the noise.
Image denoising: The MMSE estimator can be used to remove noise from images, by estimating the true image from a noisy observation.
Control systems: The MMSE estimator can be used in control systems to estimate the state of a dynamical system from noisy measurements.
Machine learning: The MMSE estimator can be used in regression problems to estimate a continuous target variable from noisy observations.

Conclusion

The Minimum Mean Square Error (MMSE) estimator is a method used to estimate a signal or parameter of a random process based on observations that are corrupted by noise. The MMSE estimator is designed to minimize the mean square error (MSE) between the estimated signal and the true signal, thus providing a measure of how close the estimate is to the true signal. The MMSE estimator relies on several assumptions about the random process and the noise, and it has many applications in signal processing and estimation theory.