Estimates and Estimation

Post about Estimates and Estimation that include point estimation, interval estimation, level of significance, Properties of estimators, and real-life examples about estimate and estimation.

Inferential Statistics Terminology

by

This post is about Inferential Statistics (or statistical inference) and some of its related terminologies. This is a field of statistics that allows us to understand and make predictions about the world around us.

Parameter and Statistic

Any measurable characteristic of a population is called a parameter. For example, the mean of a population is a parameter. OR

Numerical values that describe the characteristics of a whole population are called parameters, commonly presented in Greek Letters.

Any measurable characteristic of a sample is called a statistic. For example, the mean of a sample is a statistic. OR

Numerical measures describing the characteristics of a sample are called statistics, represented by Roman Letters.

Population and Sample

Population: The entire group of individuals, objects, or data points that one is interested in studying. A population under study can be finite or infinite. However, often too large or impractical to study directly.

Sample: A smaller, representative subset of the population. It is used to gain insights about the population without having to study every member. A sample should accurately reflect the characteristics of the population.  

Inference

A Process of drawing conclusions about a population based on the information contained in a sample taken from that population

Estimator

An estimator is a rule (method, formula) that tells how to calculate the value of an estimate based on the measurements contained in a sample. The sample mean is one possible estimator of the population mean $\mu$.

An estimator will be a good estimator in the sense that the distribution of an estimator is concentrated near the value of the parameter.

Estimate

Estimate is a way to use samples. There are many ways to estimate a parameter. Estimates are near to reality (biased or crude). Decisions are very accurate if the estimate is near to reality.

$X_1, X_2, \cdots, X_n$ is a sample and $\overline{X}$ is an estimator. $x_1, x_2, \cdots, x_n$ are sample observation and $\overline{x}=\frac{\Sigma x_i}{n}$ is an estimate.

Estimation

Estimation is the process of finding an estimate or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable.

Statistical Inference (or Inferential Statistics)

Any process (art) of drawing inferences (conclusions) about the population based on limited information contained in a sample taken from the same population is called statistical inference (or inferential statistics). It is difficult to draw an inference about the population because the study of the entire universe (population) is not simple. To get some idea about the characteristics (parameters) of the population, we choose a part of a reasonable size, generally, referred to as a sample (by some appropriate method).

Statistical inference is a powerful set of tools used to conclude a population based on data collected from a sample of that population. It allows us to make informed decisions and predictions about the larger group even when we have not examined every single member.

Why Estimate?

  • Speed: Often, an estimate is faster to get than an exact calculation.
  • Simplicity: It can simplify complex problems.
  • Decision-Making: Estimates help one to make choices when one does not have all the details.
  • Checking: One can use estimates to check if a more precise answer is reasonable.

Why is Statistical Inference Important?

  • Decision-making: It helps us make informed decisions in various fields, such as medicine, business, and social sciences.
  • Research: It is crucial for conducting research and drawing meaningful conclusions from data.
  • Understanding the World: It allows us to understand and make predictions about the world around us.
Inferential Statistics or Statistical Inference

Learn R Programming Language, Learn Statistics and Data Analysis

Properties of a Good Estimator

by

Introduction (Properties of a Good Estimator)

The post is about a comprehensive discussion of the Properties of a Good Estimator. In statistics, an estimator is a function of sample data used to estimate an unknown population parameter. A good estimator is both efficient and unbiased. An estimator is considered as a good estimator if it satisfies the following properties:

  • Unbiasedness
  • Consistency
  • Efficiency
  • Sufficiency
  • Invariance

Let us discuss these properties of a good estimator one by one.

Unbiasedness

An estimator is said to be an unbiased estimator if its expected value (that is mean of its sampling distribution) is equal to its true population parameter value. Let $\hat{\theta}$ be an unbiased estimator of its true population parameter $\theta$ then $\hat{\theta}$. If $E(\hat{\theta}) = E(\theta)$ the estimator ($\hat{\theta}$) will be unbiased. If $E(\hat{\theta})\ne \theta$, then $\hat{\theta}$ will be a biased estimator of $\theta$.

  • If $E(\hat{\theta}) > \theta$, then $\hat{\theta}$ will be positively biased.
  • If $E(\hat{\theta}) < \theta$, then $\hat{\theta}$ will be negatively biased.

Some examples of biased or unbiased estimators are:

  • $\overline{X}$ is an unbiased estimator of $\mu$, that is, $E(\overline{X}) = \mu$
  • $\widetilde{X}$ is also an unbiased estimator when the population is normally distributed, that is, $E(\widetilde{X}) =\mu$
  • Sample variance $S^2$ is biased estimator of $\sigma^2$, that is, $E(S^2)\ne \sigma^2$
  • $\hat{p} = \frac{x}{n}$ is an unbiased estimator of $E(\hat{p})=p$

It means that if the sampling process is repeated many times and calculations about the estimator for each sample are made, the average of these estimates would be very close to the true population parameter.

An unbiased estimator does not systematically overestimate or underestimate the true parameter.

Consistency

An estimator is said to be a consistent estimator if the statistic to be used as an estimator approaches the true population parameter value by increasing the sample size. OR
An estimator $\hat{\theta}$ is called a consistent estimator of $\theta$ if the probability that $\hat{\theta}$ becomes closer and closer to $\theta$, approaches unity with increasing the sample size.

Symbolically, $\hat{\theta}$ is a consistent estimator of the parameter $\theta$ if for any arbitrary small positive quantity $e$ or $\epsilon$.

\begin{align*}
\lim\limits_{n\rightarrow \infty} P\left[|\hat{\theta}-\theta|\le \varepsilon\right] &= 1\\
\lim\limits_{n\rightarrow \infty} P\left[|\hat{\theta}-\theta|> \varepsilon\right] &= 0
\end{align*}

A consistent estimator may or may not be unbiased. The sample mean $\overline{X}=\frac{\Sigma X_i}{n}$ and sample proportion $\hat{p} = \frac{x}{n}$ are unbiased estimators of $\mu$ and $p$, respectively and are also consistent.

It means that as one collects more and more data, the estimator becomes more and more accurate in approximating the true population value.

An efficient estimator is less likely to produce extreme values, making it more reliable.

Efficiency

An unbiased estimator is said to be efficient if the variance of its sampling distribution is smaller than that of the sampling distribution of any other unbiased estimator of the same parameter. Suppose there are two unbiased estimators $T_1$ and $T_2$ of the sample parameter $\theta$, then $T_1$ will be said to be a more efficient estimator compared to the $T_2$ if $Var(T_1) < Var(T_2)$. The relative efficiency of $T_1$ compared to $T_2$ is given by the ration

$$E = \frac{Var(T_2)}{Var(T_1)} > 1$$

Note that when two estimators are biased then MSE is used to compare.

A more efficient estimator has a smaller sampling error, meaning it is less likely to deviate significantly from the true population parameter.

An efficient estimator is less likely to produce extreme values, making it more reliable.

Sufficiency

An estimator is said to be sufficient if the statistic used as an estimator utilizes all the information contained in the sample. Any statistic that is not computed from all values in the sample is not a sufficient estimator. The sample mean $\overline{X}=\frac{\Sigma X}{n}$ and sample proportion $\hat{p} = \frac{x}{n}$ are sufficient estimators of the population mean $\mu$ and population proportion $p$, respectively but the median is not a sufficient estimator because it does not use all the information contained in the sample.

A sufficient estimator provides us with maximum information as it is close to a population which is why, it also measures variability.

A sufficient estimator captures all the useful information from the data without any loss.

A sufficient estimator captures all the useful information from the data.

Invariance (Property of Love)

If the function of the parameter changes, the estimator also changes with some functional applications. This property is known as invariance.

\begin{align}
E(X-\mu)^2 &= \sigma^2 \\
\text{or } \sqrt{E(X-\mu)^2} &= \sigma\\
\text{or } [E(X-\mu)^2]^2 &= (\sigma^2)^2
\end{align}

The property states that if $\hat{\theta}$ is the MLE of $\theta$ then $\tau(\hat{\theta})$ is the MLE of $\tau(\hat{\theta})$ for any function. The Taw ($\tau$) is the general form of any function. for example $\theta=\overline{X}$, $\theta^2=\overline{X}^2$, and $\sqrt{\theta}=\sqrt{\overline{X}}$.

Properties of a Good Estimator

From the above diagrammatic representations, one can visualize the properties of a good estimator as described below.

  • Unbiasedness: The estimator should be centered around the true value.
  • Efficiency: The estimator should have a smaller spread (variance) around the true value.
  • Consistency: As the sample size increases, the estimator should become more accurate.
  • Sufficiency: The estimator should capture all relevant information from the sample.

In summary, regarding the properties of a good estimator, a good estimator is unbiased, efficient, consistent, and ideally sufficient. It should also be robust to outliers and have a low MSE.

Properties of a good estimator

https://rfaqs.com, https://gmstat.com

Best Online Estimation MCQs 1

by

Online Estimation MCQs for Preparation of PPSC and FPSC Statistics Lecturer Post. There are 20 multiple-choice questions covering the topics related to properties of a good estimation (unbiasedness, efficiency, sufficiency, consistency, and invariance), expectation, point estimate, and interval estimate. Let us start with the Online Estimation MCQs Quiz.

Online MCQs about Estimate and Estimation for Preparation of PPSC and FPSC Statistics Lecturer Post

1. A test is said to be the most powerful test of size $\alpha$, if

 
 
 
 

2. What are the main components of a confidence interval?

 
 
 
 

3. If $X_1,X_2,\cdots, X_n$ is the joint density of n random variables, say, $f(X_1, X_2,\cdots, X_n;\theta)$ which is considered to be a function of $\theta$. Then $L(\theta; X_1,X_2,\cdots, X_n)$ is called

 
 
 
 

4. Let $L(\theta;X_1,X_2,\cdots,X_n)$ be the likelihood function for a sample $X_1,X_2,\cdots, X_n$ having joint density $f(x_1,x_2,\cdots,x_n;\theta)$ where ? belong to parameter space. Then a test defined as $\lambda=\lambda_n=\lambda(x_1,x_2,\cdots,x_n)=\frac{Sup_{\theta\varepsilon \Theta_0}L(\theta;x_1,x_2,\cdots,x_n)}{Sup_{\theta\varepsilon \Theta}L(\theta;x_1,x_2,\cdots,x_n)}$

 
 
 
 

5. If the conditional distribution of $X_1, X_2,\cdots,X_n$ given $S=s$, does not depend on $\theta$, for any value of $S=s$, the statistics $S=s(X_1,X_2,\cdots,X_n)$ is called

 
 
 
 

6. Let $X_1,X_2,\cdots,X_n$ be a random sample from a density $f(x|\theta)$, where $\theta$ is a value of the random variable $\Theta$ with known density $g_\Theta(\theta)$. Then the estimator $\tau(\theta)$ with respect to the prior $g_\Theta(\theta)$ is defined as $E[\tau(\theta)|X_1,X_2,\cdots,X_n]$ is called

 
 
 
 

7. A set of jointly sufficient statistics is defined to be minimal sufficient if and only if

 
 
 
 

8. Which of the following assumptions are required to show the consistency, unbiasedness, and efficiency of the OLS estimator?

  1. $E(\mu_t)=0$
  2. $Var(\mu_t)=\sigma^2$
  3. $Cov(\mu_t,\mu_{t-j})=0;t\neq t-j$
  4. $\mu_t \sim N(0,\sigma^2)$
 
 
 
 

9. $Var_\theta (T) \geq \frac{[\tau'(\theta)]^2}{nE[{\frac{\partial}{\partial \theta}log f((X;\theta)}^2]}$, where $T=t(X_1,X_2,\cdots, X_n)$ is an unbiased estimator of $\tau(\theta)$. The above inequality is called

 
 
 
 

10. In statistical inference, the best asymptotically normal estimator is denoted by

 
 
 
 

11. If $Var(T_2) < Var(T_1)$, then $T_2$ is

 
 
 
 

12. For a biased estimator $\hat{\theta}$ of $\theta$, which one is correct

 
 
 
 

13. For two estimators $T_1=t_1(X_1,X_2,\cdots,X_n)$ and $T_2=t_2(X_1,X_2,\cdots,X_n)$ then estimator $t_1$ is defined to be $R_{{t_1}(\theta)}\leq R_{{t_2}(\theta)}$ for all $\theta$ in $\Theta$

 
 
 
 

14. What is the maximum expected difference between a population parameter and a sample estimate?

 
 
 
 

15. Let $Z_1,Z_2,\cdots,Z_n$ be independently and identically distributed random variables, satisfying $E[|Z_t|]<\infty$. Let N be an integer-valued random variable whose value n depends only on the values of the first n $Z_i$s. Suppose $E(N)<\infty$, then $E(Z_1+Z_2+\cdots+Z_n)=E( N)E(Z_i)$ is called

 
 
 
 

16. Which of the following statements describes an interval estimate?

 
 
 
 

17. Let $X_1,X_2,\cdots,X_n$ be a random sample from the density $f(x;\theta)$, where $\theta$ may be vector. If the conditional distribution of $X_1,X_2,\cdots,X_n$ given $S=s$ does not depend on $\theta$ for any value of $s$ of $S$, then statistic is called.

 
 
 
 

18. If $E(\hat{\theta})=\theta$, then $\hat{\theta}$ is said to be

 
 
 
 

19. If $Var(\hat{\theta})\rightarrow 0$ as $n \rightarrow 0$, then $\hat{\theta}$ is said to be

 
 
 
 

20. If $f(x_1,x_2,\cdots,x_n;\theta)=g(\hat{\theta};\theta)h(x_1,x_2,\cdots,x_n)$, then $\hat{\theta}$ is

 
 
 
 

Online Estimation MCQs with Answers

Online Estimation MCQs with Answers
  • If $Var(\hat{\theta})\rightarrow 0$ as $n \rightarrow 0$, then $\hat{\theta}$ is said to be
  • If $E(\hat{\theta})=\theta$, then $\hat{\theta}$ is said to be
  • If $Var(T_2) < Var(T_1)$, then $T_2$ is
  • If $f(x_1,x_2,\cdots,x_n;\theta)=g(\hat{\theta};\theta)h(x_1,x_2,\cdots,x_n)$, then $\hat{\theta}$ is
  • Which of the following assumptions are required to show the consistency, unbiasedness, and efficiency of the OLS estimator?
    i. $E(\mu_t)=0$
    ii. $Var(\mu_t)=\sigma^2$
    iii. $Cov(\mu_t,\mu_{t-j})=0;t\neq t-j$
    iv. $\mu_t \sim N(0,\sigma^2)$
  • For a biased estimator $\hat{\theta}$ of $\theta$, which one is correct
  • A test is said to be the most powerful test of size $\alpha$, if
  • In statistical inference, the best asymptotically normal estimator is denoted by
  • If the conditional distribution of $X_1, X_2,\cdots,X_n$ given $S=s$, does not depend on $\theta$, for any value of $S=s$, the statistics $S=s(X_1,X_2,\cdots,X_n)$ is called
  • A set of jointly sufficient statistics is defined to be minimal sufficient if and only if
  • If $X_1,X_2,\cdots, X_n$ is the joint density of n random variables, say, $f(X_1, X_2,\cdots, X_n;\theta)$ which is considered to be a function of $\theta$. Then $L(\theta; X_1,X_2,\cdots, X_n)$ is called
  • For two estimators $T_1=t_1(X_1,X_2,\cdots,X_n)$ and $T_2=t_2(X_1,X_2,\cdots,X_n)$ then estimator $t_1$ is defined to be $R_{{t_1}(\theta)}\leq R_{{t_2}(\theta)}$ for all $\theta$ in $\Theta$
  • Let $X_1,X_2,\cdots,X_n$ be a random sample from the density $f(x;\theta)$, where $\theta$ may be vector. If the conditional distribution of $X_1,X_2,\cdots,X_n$ given $S=s$ does not depend on $\theta$ for any value of $s$ of $S$, then statistic is called.
  • $Var_\theta (T) \geq \frac{[\tau'(\theta)]^2}{nE[{\frac{\partial}{\partial \theta}log f((X;\theta)}^2]}$, where $T=t(X_1,X_2,\cdots, X_n)$ is an unbiased estimator of $\tau(\theta)$. The above inequality is called
  • Let $X_1,X_2,\cdots,X_n$ be a random sample from a density $f(x|\theta)$, where $\theta$ is a value of the random variable $\Theta$ with known density $g_\Theta(\theta)$. Then the estimator $\tau(\theta)$ with respect to the prior $g_\Theta(\theta)$ is defined as $E[\tau(\theta)|X_1,X_2,\cdots,X_n]$ is called
  • Let $L(\theta;X_1,X_2,\cdots,X_n)$ be the likelihood function for a sample $X_1,X_2,\cdots, X_n$ having joint density $f(x_1,x_2,\cdots,x_n;\theta)$ where ? belong to parameter space. Then a test defined as $\lambda=\lambda_n=\lambda(x_1,x_2,\cdots,x_n)=\frac{Sup_{\theta\varepsilon \Theta_0}L(\theta;x_1,x_2,\cdots,x_n)}{Sup_{\theta\varepsilon \Theta}L(\theta;x_1,x_2,\cdots,x_n)}$
  • Let $Z_1,Z_2,\cdots,Z_n$ be independently and identically distributed random variables, satisfying $E[|Z_t|]<\infty$. Let N be an integer-valued random variable whose value $n$ depends only on the values of the first n $Z_i$s. Suppose $E(N)<\infty$, then $E(Z_1+Z_2+\cdots+Z_n)=E( N)E(Z_i)$ is called
  • What is the maximum expected difference between a population parameter and a sample estimate?
  • Which of the following statements describes an interval estimate?
  • What are the main components of a confidence interval?

R Frequently Asked Questions

Online MCQs Test Preparation Website