Statistics for Data Analyst

Introduction to Efficiency of an Estimator

The efficiency of an estimator is a measure of how well it estimates a population parameter compared to other estimators. It is possible to have more than one unbiased estimator of a parameter. We should have at least one additional criterion for choosing among the unbiased estimator of the parameter. Usually, unbiased estimators are compared in terms of their variances. Thus, the comparison of variances of estimators is described as a comparison of the efficiency of estimators.

Use of Efficiency

The efficiency of an estimator is often used to evaluate an estimator through the following concepts:

• Bias: An estimator is unbiased if its expected value equals the true parameter value ($E[\hat{\theta}]=\theta$). The efficiency of an estimator can be influenced by bias; thus, unbiased estimators are often preferred.
• Variance: Efficiency is commonly assessed by the variance of the estimator. An estimator having a lower variance is considered more efficient. The Cramér-Rao lower bound provides a theoretically lower limit for the variance of unbiased estimators.
• Mean Squared Error (MSE): Efficiency can also be measured using MSE, which combines both variance and bias. MSE is given by: MSE = $Var(\hat{\theta}) + Bias (\hat{\theta})^2$. An estimator with a lower MSE is more efficient.
• Relative Efficiency: The relative efficiency compares the efficiency of two estimators, often expressed as the ratio of their variances: Relative Efficiency = $\frac{Var(\hat{\theta}_2)}{Var(\hat{\theta}_1)}, where $\hat{\theta}_1$ is the estimator being compared, and $\hat{\theta}_2$ is a competitor.

The efficiency of an estimator is stated in relative terms. If say two estimators $\hat{\theta}_1$ and $\hat{\theta}_2$ are unbiased estimators of the same population parameter $\theta$ and the variance of $\hat{\theta}_1$ is less than the variance of $\hat{\theta}_2$ (that is, $Var(\hat{\theta}_1) < Var(\hat{\theta}_2)$ then $\hat{\theta}_1$ is relatively more efficient than $\hat{\theta}_2$. The ration is $E=\frac{Var(\hat{\theta}_2)}{var(\hat{\theta}_1)}$ is a measure of relative efficiency of $\hat{\theta}_1$ with respect to the $\hat{\theta}_2$. If $E>1$, $\hat{\theta}_1$ is said to be more efficient than $\hat{\theta}_2$.

If $\hat{\theta}$ is an unbiased estimator of $\theta$ and $Var(\hat{\theta})$ is minimum compared to any other unbiased estimator for $\theta$, then $\hat{\theta}$ is said to be a minimum variance unbiased estimator for $\theta$.

It is preferable to make efficient comparisons based on the MSE instead of its variance.

\begin{align*}
MSE(\hat{\theta}) & = E(\hat{\theta} – \theta)^2\\
&= E\left[(\hat{\theta} – E(\hat{\theta}) + E(\hat{\theta}) – \theta \right]\\
&= E\left[ \left(\hat{\theta} – E(\hat{\theta})\right) ^2 + \left(E(\hat{\theta})-\hat{\theta}\right)^2 + 2(\hat{\theta}-E(\hat{\theta}))(E(\hat{\theta}) -\theta)\right]\\
&= E[\hat{\theta} – E(\hat{\theta})]^2 + [E(\hat{\theta})-\theta]^2 \\
&= Var(\hat{\theta}) + (Bias)^2
\end{align*}

where $E[\hat{\theta}-E(\hat{\theta})] = E(\hat{\theta}) – E(\hat{\theta})=0$

Question about the Efficiency of an Estimator

Question: Let $X_1, X_2, \cdots, X_n$ be a random sample of size 3 from a population with mean $\mu$ and variance \sigma^2$. Consider the following estimators of mean $\mu$:

\begin{align*}
T_1 &= \frac{X_1+X_2+X_3}{2}\qquad Sample\,\, mean\\
T_2 &- \frac{X_1 + 2X_2 + X_3}{4} \qquad Weighted \,\, mean
\end{align*}

which estimator should be preferred?

Solution

First, we check the unbiasedness of $T_1$ and $T_2.

\begin{align*}
E(T_1) &= \frac{1}{3} E(X_1 + X_2 + X_3)=\mu\\
E(T_2) &= \frac{1}{4}E(X_1+2X_2 + X_4) = \mu
\end{align*}

Therefore, $T_1$ and $T_2$ are unbiased estimators of $\mu$.

For efficiency, let us check the variances of these estimators.

\begin{align*}
Var(T_1) &= Var\left(\frac{X_1 + X_2 + X_3}{3} \right)\\
&= \frac{1}{9} \left(Var(X_1) + Var(X_2) + Var(X_3)\right)\\
&= \frac{1}{9} (\sigma^2 + \sigma^2 + \sigma^2) = \frac{\sigma^2}{3}\\
Var(T_2) &= Var\left(\frac{X_1 + 2X_2 + X_3}{4}\right)\\
&= \frac{1}{16} \left(Var(X_1) + 4Var(X_2) + Var(X_3)\right)\\
&= \frac{1}{16}(\sigma^2 + 4\sigma^2 + \sigma^2) = \frac{3\sigma^2}{8}
\end{align*}

Since $\frac{1}{3} < \frac{3}{8}$, that is, $Var(T_1) < Var(T_2). The $T_1$ is better estimator of $\mu$ than $T_2$.

Reasons to Use Efficiency of an Estimator

1. Optimal Use of Data: An efficient estimator makes the best possible use of the available data, providing more accurate estimates. This is particularly important in research, where the goal is often to make inferences or predictions based on sample data.
2. Reducing Uncertainty: Efficiency reduces the variance of the estimators, leading to more precise estimates. This is essential in fields like medicine, economics, and engineering, where precise measurements can significantly impact decision-making and outcomes.
3. Resource Allocation: In practical applications, using an efficient estimator can lead to savings in money, time, and resources. For example, if an estimator provides a more accurate estimate with less data, it can result in fewer resources needed for data collection.
4. Comparative Evaluation: Comparisons between different estimators help researchers and practitioners choose the best method for their specific context. Understanding efficiency allows one to select estimators that yield reliable results.
5. Statistical Power: Efficient estimators contribute to higher statistical power, which is the probability of correctly rejecting a false null hypothesis. This is particularly important in hypothesis testing and experimental design.
6. Robustness: While efficiency relates mostly to variance and bias, efficient estimators are often more robust to violations of assumptions (e.g., normality) in some contexts, leading to more reliable conclusions.

In summary, the efficiency of an estimator is vital as it directly influences the accuracy, reliability, and practical utility of statistical analyses, ultimately affecting the quality of decision-making based on those analyses.

MCQs Functions and Limits

Packages in R for Data Analysis