Sign in Subscribe

MMSE minimum mean-squared error

Last updated on 

Minimum mean-squared error (MMSE) is a technique used in signal processing, statistical inference, and other fields to estimate the values of an unknown signal based on a set of noisy measurements. The goal of MMSE is to minimize the mean-squared error (MSE) between the estimated signal and the true signal.

In this article, we will explain the MMSE technique in detail, including its derivation, applications, and limitations. We will also discuss some variations of the MMSE estimator, such as the linear MMSE estimator and the Wiener filter.

Derivation of the MMSE Estimator

The MMSE estimator is derived from the conditional mean of the signal given the measurements. Let us assume that we have a signal x(n) that we want to estimate based on a set of measurements y(n) corrupted by noise v(n):

y(n) = x(n) + v(n)

where n is the index of the sample. We can represent this relationship in matrix form as:

y = Hx + v

where y is an N x 1 vector of measurements, x is an M x 1 vector of the signal we want to estimate, H is an N x M matrix representing the linear relationship between y and x, and v is an N x 1 vector of noise.

We assume that the noise v is additive white Gaussian noise (AWGN) with zero mean and variance σv². This means that the probability density function (PDF) of v is given by:

p(v) = N(0, σv²)

where N(0, σv²) denotes a normal (Gaussian) distribution with mean 0 and variance σv².

We also assume that the signal x is a random variable with mean μx and covariance Σxx. This means that the PDF of x is given by:

p(x) = N(μx, Σxx)

where N(μx, Σxx) denotes a normal distribution with mean μx and covariance Σxx.

Our goal is to estimate the signal x given the measurements y. We can do this by finding the conditional mean of x given y, denoted as ẑ:

ẑ = E[x|y]

where E[·] denotes the expectation operator.

Using Bayes' theorem, we can express the conditional mean ẑ as:

ẑ = argmin_{z} E[||x - z||² | y]

where ||·|| denotes the Euclidean norm.

We can expand the squared norm as:

||x - z||² = xᵀx - 2xᵀz + zᵀz

Taking the expectation of both sides with respect to the PDF of x, we get:

E[||x - z||²] = E[xᵀx] - 2E[xᵀz] + E[zᵀz]

Since the signal x is a random variable, we can express its covariance matrix Σxx as:

Σxx = E[xxᵀ] - μxμxᵀ

where xxᵀ is the outer product of x with itself.

Substituting this expression into the first term of the above equation, we get:

E[xᵀx] = μxᵀμx + tr(Σxx)

where tr(·) denotes the trace operator.

Similarly, we can express the covariance matrix Σzx of the joint distribution of x and z as:

Σzx = E[xzᵀ] - μxμzᵀ

where xzᵀ is the outer product of x and z.

Using the property that E[x] = μx and E [z] = μz, we can express the second term of the above equation as:

E[xᵀz] = μxᵀμz + Σzx

Finally, we can express the third term of the above equation as:

E[zᵀz] = μzᵀμz + tr(Σzz)

where Σzz is the covariance matrix of the estimator z.

Substituting these expressions into the equation for E[||x - z||²], we get:

E[||x - z||²] = tr(Σxx) + μxᵀμx - 2μxᵀμz + μzᵀμz + tr(Σzz) - 2tr(Σzx)

Differentiating this expression with respect to z and setting the derivative to zero, we get:

-2Σzx + 2(μz - μx)Σxx = 0

Solving for z, we get:

z = μx + ΣxxHᵀ(Σvv + HΣxxHᵀ)⁻¹(y - Hμx)

where H is the observation matrix, Σvv is the covariance matrix of the noise v, and ⁻¹ denotes the matrix inverse.

This expression is the MMSE estimator for the signal x given the measurements y. It is a linear estimator that depends on the observation matrix H, the covariance matrices Σvv and Σxx, and the mean μx of the signal x.

Applications of the MMSE Estimator

The MMSE estimator is a widely used technique in signal processing, communication systems, and other fields that involve the estimation of a signal from noisy measurements. Some applications of the MMSE estimator include:

  1. Channel estimation: In wireless communication systems, the MMSE estimator is used to estimate the channel impulse response from the received signal.
  2. Image and video denoising: The MMSE estimator is used to remove noise from images and videos by estimating the original signal from the noisy measurements.
  3. Speech enhancement: The MMSE estimator is used to improve the quality of speech signals by removing noise and other distortions.
  4. System identification: The MMSE estimator is used to estimate the parameters of a system based on the input and output signals.
  5. Radar and sonar processing: The MMSE estimator is used to estimate the range, velocity, and direction of a target from the received radar or sonar signal.

Limitations of the MMSE Estimator

The MMSE estimator has some limitations that should be taken into account when using it:

  1. Model assumptions: The MMSE estimator assumes that the noise is additive white Gaussian noise and that the signal and noise are uncorrelated. If these assumptions are not met, the MMSE estimator may not be optimal.
  2. Nonlinearity: The MMSE estimator is a linear estimator that assumes that the relationship between the measurements and the signal is linear. If this assumption is not met, a nonlinear estimator may be more appropriate.
  3. Computation complexity: The MMSE estimator involves matrix inversion and multiplication, which can be computationally expensive for large matrices.

Variations of the MMSE Estimator

There are several variations of the MMSE estimator that address some of its limitations and improve its performance. Some of these variations include:

  1. Linear MMSE estimator: The linear MMSE estimator is a simplified version of the MMSE estimator that assumes that the noise is zero-mean and uncorrelated with the signal. This estimator is often used in practice due to its simplicity and computational efficiency.
  2. Wiener filter: The Wiener filter is a linear filter that uses the MMSE estimator to remove noise from a signal. The Wiener filter assumes that the noise and signal are stationary and that the power spectral density of the signal and noise are known. The filter minimizes the mean-squared error between the original signal and the filtered signal, subject to the constraint that the filtered signal is a linear combination of the observed signal and the noise.
  3. Kalman filter: The Kalman filter is a recursive estimator that uses a state-space model to estimate the state of a system based on noisy measurements. The filter uses the MMSE estimator to update the state estimate at each time step, and it also estimates the covariance matrix of the state estimate. The Kalman filter is widely used in control systems, navigation, and other applications that involve dynamic systems.
  4. Bayesian MMSE estimator: The Bayesian MMSE estimator extends the MMSE estimator by incorporating prior information about the signal and noise. The estimator uses Bayesian inference to update the prior distribution of the signal based on the measurements, and it computes the posterior mean and covariance matrix of the signal. The Bayesian MMSE estimator is useful when prior information is available and can improve the performance of the estimator.

Conclusion

The MMSE estimator is a powerful technique for estimating a signal from noisy measurements. The estimator provides a rigorous mathematical framework for finding the optimal estimator that minimizes the mean-squared error between the estimator and the true signal. The MMSE estimator has many applications in signal processing, communication systems, and other fields that involve the estimation of a signal from noisy measurements. The estimator has some limitations, such as model assumptions and computation complexity, but there are several variations of the estimator that address these limitations and improve its performance.