# Basic Statistics and Data Analysis

## Principal Component Regression (PCR)

The transformation of original data set into a new set of uncorrelated variables is called principal components.  This kind of transformation ranks the new variables according to their importance (that is, variable are ranked according to the size of their variance and eliminates those of least importance). After transformation, a least square regression on this reduced set of principal components is performed.

Principal Component Regression (PCR) is not scale invariant, therefore, one should scale and center data first. Therefore, given a p-dimensional random vector $x=(x_1, x_2, …, x_p)^t$ with covariance matrix $\sum$ and assume that $\sum$ is positive definite. Let $V=(v_1,v_2, \cdots, v_p)$ be a $(p \times p)$-matrix with orthogonal column vectors that is $v_i^t\, v_i=1$, where $i=1,2, \cdots, p$ and $V^t =V^{-1}$. The linear transformation

\begin{aligned}
z&=V^t x\\
z_i&=v_i^t x
\end{aligned}

The variance of the random variable $z_i$ is
\begin{aligned}
Var(Z_i)&=E[v_i^t\, x\, x^t\,\, v_i]\\
&=v_i^t \sum v_i
\end{aligned}

Maximizing the variance $Var(Z_i)$ under the conditions $v_i^t v_i=1$ with Lagrange gives
$\phi_i=v_i^t \sum v_i -a_i(v_i^t v_i-1)$

Setting the partial derivation to zero, we get
$\frac{\partial \phi_i}{\partial v_i} = 2 \sum v_i – 2a_i v_i=0$

which is
$(\sum – a_i I)v_i=0$

In matrix form
$\sum V= VA$
of
$\sum = VAV^t$

where $A=diag(a_1, a_2, \cdots, a_p)$. This is know as the eigvenvalue problem, $v_i$ are the eigenvectors of $\sum$ and $a_i$ the corresponding eigenvalues such that $a_1 \ge a_2 \cdots \ge a_p$. Since $\sum$ is positive definite, all eigenvalues are real and non-negative numbers.

$z_i$ is named the ith principal component of $x$ and we have
$Cov(z)=V^t Cov(x) V=V^t \sum V=A$

The variance of the ith principal component matches the eigenvalue $a_i$, while the variances are ranked in descending order. This means that, the last principal component will have the smallest variance. The principal components are orthogonal to all the other principal components (they are even uncorrelated) since $A$ is a diagonal matrix.

In following, for regression, we will use $q$, that is,($1\le q \le p$) principal components. The regression model for observed data $X$ and $y$ can then be expressed as

\begin{aligned}
y&=X\beta+\varepsilon\\
&=XVV^t\beta+varepsilon\\
&= Z\theta+\varepsilon
\end{aligned}

with the $n\times q$ matrix of the empirical principal components $Z=XV$ and the new regression coefficients $\theta=V^t \beta$. The solution of the least squares estimation is

\begin{aligned}
\hat{\theta}_k=(z_k^t z_k)^{-1}z_k^ty
\end{aligned}

and $\hat{\theta}=(\theta_1, \cdots, \theta_q)^t$

Since the $z_k$ are orthogonal, the regression is a sum of univariate regressions, that is
$\hat{y}_{PCR}=\sum_{k=1}^q \hat{\theta}_k z_k$

Since $z_k$ are linear combinations of the original $x_j$, the solution in terms of coefficients of the $x_j$ can be expressed as
$\hat{\beta}_{PCR} (q)=\sum_{k=1}^q \hat{\theta}_k v_k=V \hat{\theta}$

Note that if $q=p$, we would get back the usual least squares estimates for the full model. For $q<p$, we get a “reduced” regression.

## Canonical Correlation Analysis

The bivariate correlation analysis measures the strength of relationship between two variables. One may require to find the strength of relationship between two sets of variables. In this case canonical correlation is an appropriate technique for measuring the strength of relationship between two sets of variables. Canonical correlation is appropriate in the same situations where multiple regression would be, but where there are multiple inter-correlated outcome variables. Canonical correlation analysis determines a set of canonical variates, orthogonal linear combinations of the variables within each set that best explain the variability both within and between sets. For example,

• In medical, individuals’ life styles and eating habits may have effect on their different health measures determined by number of health-related variables such as hypertension, weight, anxiety and tension level.
• In business, the marketing manager of a consumer goods firm may be interested in finding the relationship between types of products purchased and consumers’ life styles and personalities.

From above two examples, one set of variables is the predictor or independent while other set of variables is the criterion or dependent set. The objective of canonical correlation analysis is to determine if the predictor set of variables affects the criterion set of variables.

Note that it is not necessary to designate the two sets of variables as the dependent and independent sets. In this case the objective of canonical correlation is to ascertain the relationship between the two sets of variables.

The objective of canonical correlation is similar to that of conducting a principal components analysis on each set of variables. In principal components analysis, the first new axis results in a new variable that accounts for the maximum variance in the data, while in canonical correlation analysis a new axis is identified for each set of variables such that the correlation between the two resulting new variables is maximum.

Canonical correlation analysis can also be considered as data reduction technique as it is possible that only a few canonical variables are needed to adequately represents the association between the two sets of variables. Therefore, an additional objective of canonical correlation is to determine the minimum number of canonical correlations needed to adequately represent the association between two sets of variables.

## Data Collection Methods

There are many methods to collect data, but these methods can be classified in four main methods (sources) of collecting data to use in statistical inference. These are (i) Survey Method (ii) Simulation (iii) Controlled Experiments (iv) Observational Study.

## Survey Method

A very popular and widely used method is the survey, where people with special training go out and record observations of, the number of vehicles, traveling along a road, the acres of fields that farmers are using to grow a particular food crop; the number of house-holds that own more than one motor vehicle, the number of passenger using Metro transport and so on. Here the person making the study has no direct control over generating the data that can be recorded, although the recording methods need care and control.

## Simulation

In Simulation, a computer model for the operation of a (industrial)  system is setup in which an important measurement is a percentage purity of a (chemical) product. A very large number of realizations of the model can be run in order to look for any pattern in the results. Here the success of the approach depends on how well that measurement can be explained by the model and this has to be tested by carrying out at least a small amount of work on the actual system in operation.

## Controlled Experiments

An experiment is possible when the background conditions can be controlled, at least to some extent. For example, we may be interested in choosing the best type of a grass seed to use in sport field.

The first stage of work is to grow all the competing varieties of seed at the same place and make suitable records of their growth and development. The competing varieties should be grown in quite small units close together in the field as in the figure below

This is the controlled experiment as it has certain constraints such as;

i) River on right side
ii) Shadow of trees on left side
iii) There are 3 different varieties (say, v1, v2, v3) and are distributed in 12 units.

In diagram below, much more control of local environmental conditions than there would have been of one variety had been replaced in strip in the shelter of the trees, another close by the river while third one is more exposed in center of the field;

There are 3 experimental units. One is close to stream and other is to trees while third one is between them which is most beneficial than others. It is now our choice where to place any one of them at any of the side.

## Observational Study

Like experiments, observational studies try to understand cause-and-effect relationships. However, unlike experiments, the researcher is not able to control (1) how subjects are assigned to groups and/or (2) which treatments each group receives.

Note that small units of land or plots are called experimental units or simply units.

There is no “right” side for a unit, it depends on the type of the crop, the work that is to be done on it and the measurements that are to be taken. Similarly, the measurements upon which inferences are eventually going to be based are to be taken as accurately as possible. The unit must therefore need not be so large as to make recording very tedious because that leads to errors and inaccuracy. On the other hand, if a unit is very small there is the danger that relatively minor physical errors in recording, can lead to a large percentage errors.

Experimenters and statisticians who collaborate with them, need to gain a good knowledge of their experimental material or units as a research program proceeds.

## Basic Principles of Experimental Design

The basic principles of experimental design are (i) Randomization, (ii) Replication and (iii) Local Control.

1. ## Randomization

Randomization is the corner stone underlying the use of statistical methods in experimental designs.  Randomization is the random process of assigning treatments to the experimental units. The random process implies that every possible allotment of treatments has the same probability. For example, if number of treatment = 3 (say, A, B, and C) and replication = r = 4, then the number of elements = t x r = 3 x 4 = 12 = n. Replication means that each treatment will appear 4 times as r = 4. Let the design is

 A C B C C B A B A C B A

Note from the design elements 1, 7, 9, 12 are reserved for treatment A, element 3, 6, 8 and 11 are reserved for Treatment B and elements 2, 4, 5 and 10 are reserved for Treatment C. P(A)= 4/12, P(B)= 4/12, and P(C)=4/12, meaning that Treatment A, B and C has equal chances of its selection.

2. ## Replication

By replication we means that repetition of the basic experiments. For example, If we need to compare grain yield of two varieties of wheat then each variety is applied to more than one experimental units. The number of times these are applied on experimental units is called their number of replication. It has two important properties:

• It allows the experimenter to obtain an estimate of the experimental error.
• The more replication would provide the increased precision by reducing the standard error (SE) of mean as $s_{\overline{y}}=\tfrac{s}{\sqrt{r}}$, where $s$ is sample standard deviation and $r$ is number of replications. Note that increase in $r$ value $s_{\overline{y}}$ (standard error of $\overline{y}$).
3. ## Local Control

It has been observed that all extraneous source of variation are not removed by randomization and replication, i.e. unable to control extraneous source of variation.
Thus we need to a refinement in the experimental technique. In other words we need to choose a design in such a way that all extraneous source of variation are brought under control. For this purpose we make use of local control, a term referring to the amount of (i) balancing, (ii) blocking and (iii) grouping of experimental units.

Balancing: Balancing means that the treatment should be assigned to the experimental units in such a way that the result is a balanced arrangement of treatment.

Blocking: Blocking means that the like experimental units should be collected together to far relatively homogeneous groups. A block is also a replicate.

The main objective/ purpose of local control is to increase the efficiency of experimental design by decreasing the experimental error.

## Standard Error of Estimate

Standard error (SE) is a statistical term used to measure the accuracy within a sample taken from population of interest. The standard error of the mean measures the variation in the sampling distribution of the sample mean, usually denoted by $\sigma_\overline{x}$ is calculated as

$\sigma_\overline{x}=\frac{\sigma}{\sqrt{n}}$

Drawing (obtaining) different samples from the same population of interest usually results in different values of sample means, indicating that there is a distribution of sampled means having its own mean (average values) and variance. The standard error of the mean is considered as the standard deviation of all those possible sample drawn from the same population.

The size of the standard error is affected by standard deviation of the population and number of observations in a sample called the sample size. The larger the standard deviation of the population ($\sigma$), the larger the standard error will be, indicating that there is more variability in the sample means. However larger the number of observations in a sample smaller will be the standard error of estimate, indicating that there is less variability in the sample means, where by less variability we means that the sample is more representative of the population of interest.

If the sampled population is not very larger, we need to make some adjustment in computing the SE of the sample means. For a finite population, in which total number of objects (observations) is $N$ and the number of objects (observations) in a sample is $n$, then the adjustment will be $\sqrt{\frac{N-n}{N-1}}$. This adjustment is called the finite population correction factor. Then the adjusted standard error will be

$\frac{\sigma}{\sqrt{n}} \sqrt{\frac{N-n}{N-1}}$

The SE is used to:

1. measure the spread of values of statistic about the expected value of that statistic
2. construct confidence intervals
3. test the null hypothesis about population parameter(s)

The standard error is computed from sample statistics. To compute SE for simple random samples, assuming that the size of population ($N$) is at least 20 times larger than that of the sample size ($n$).
\begin{align*}
Sample\, mean, \overline{x} & \Rightarrow SE_{\overline{x}} = \frac{n}{\sqrt{n}}\\
Sample\, proportion, p &\Rightarrow SE_{p} \sqrt{\frac{p(1-p)}{n}}\\
Difference\, b/w \, means, \overline{x}_1 – \overline{x}_2 &\Rightarrow SE_{\overline{x}_1-\overline{x}_2}=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}\\
Difference\, b/w\, proportions, \overline{p}_1-\overline{p}_2 &\Rightarrow SE_{p_1-p_2}=\sqrt{\frac{p_1(1-p_1)}{n_1}+\frac{p_2(1-p_2)}{n_2}}
\end{align*}

The standard error is identical to the standard deviation, except that it uses statistics whereas the standard deviation uses the parameter.