Correlation and Regression Analysis, simple and multiple regression, regression, regression diagnostics, Assumptions of the regression model, model selection criteria, Linear Regression models, Relationship between Variables, Strength, and direction of the relationship
The Akaike Information Criteria/Criterion (AIC) is a method used in statistics and machine learning to compare the relative quality of different models for a given dataset. The AIC method helps in selecting the best model out of a bunch by penalizing models that are overly complex. Akaike Information Criterion provides a means for comparing among models i.e. a tool for model selection.
A too-simple model leads to a large approximation error.
A too-complex model leads to a large estimation error.
AIC is a measure of goodness of fit of a statistical model developed by Hirotsugo Akaike under the name of “an information Criteria (AIC) and published by him in 1974 first time. It is grounded in the concept of information entropy in between bias and variance in model construction or between accuracy and complexity of the model.
The Formula of Akaike Information Criteria
Given a data set, several candidate models can be ranked according to their AIC values. From AIC values one may infer that the top two models are roughly in a tie and the rest far worse.
$$AIC = 2k-ln(L)$$
where $k$ is the number of parameters in the model, and $L$ is the maximized value of the likelihood function for the estimated model.
For a set of candidate models for the data, the preferred model is the one that has a minimum AIC value. AIC estimates relative support for a model, which means that AIC scores by themselves are not very meaningful
Akaike Information Criteria focuses on:
Balances fit and complexity: A model that perfectly fits the data might not be the best because it might be memorizing the data instead of capturing the underlying trend. AIC considers both how well a model fits the data (goodness of fit) and how complex it is (number of variables).
A lower score is better: Models having lower AIC scores are preferred as they achieve a good balance between fitting the data and avoiding overfitting.
Comparison tool: AIC scores are most meaningful when comparing models for the same dataset. The model with the lowest AIC score is considered the best relative to the other models being evaluated.
Summary
The AIC score is a single number and is used as model selection criteria. One cannot interpret the AIC score in isolation. However, one can compare the AIC scores of different model fits to the same data. The model with the lowest AIC is generally considered the best choice.
The AIC is the most useful model selection criterion when there are multiple candidate models to choose from. It works well for larger datasets. However, for smaller datasets, the corrected AIC should be preferred. AIC is not perfect, and there can be situations where it fails to choose the optimal model.
There are many other model selection criteria. For more detail read the article: Model Selection Criteria
Francis Galton (a biometrician) examines the relationship between fathers’ and sons’ height. He analyzed the similarities between the parent and child generation of 700 sweet peas. Galton found that the offspring of tall parents tended to be shorter and offspring of shorter parents tended to be taller. The height of the children depends ($Y$) upon the height of the parents ($X$). In case, there is more than one independent variable (IV), we need multiple regression analysis (MRA), also called multiple linear regression (MLR).
Table of Contents
Multiple Linear Regression Model
The linear regression model (equation) for two independent variables (regressors) is
To fit the MLR equation for two variables, one needs to compute the values of $\hat{\beta}_1, \hat{\beta}_2$, and $\alpha$.
The yellow part of the above formula is the (“sum of the product of 1st independent and dependent variables”) multiplied by the (“sum of the square of 2nd independent variable).
The red part of the above formula is the (“Sum of the product of 2nd independent and dependent variables”) multiplied by the (“sum of the product of two independent variables”).
The green part of the above formula is the (“sum of the square of 1st independent variable”) multiplied by the (“sum of the square of 2nd independent variable”).
The blue part of the above formula is the (“square of the sum of the product of two independent variables”).
The formula for 2nd regression coefficient is
In short, note that the $S$ stands for the sum of squares and the sum of products.
Multiple Linear Regression Example
Consider the following data about two regressors ($X_1, X_2$) and one regressand variable ($Y$).
Independent variables (predictors, regressors): These are the variables that one believes to influence the dependent variable. One can have two or more independent variables in a multiple-regression model.
Dependent variable (outcome, response): This is the variable one is trying to predict or explain using the independent variables.
Linear relationship: The core assumption is that the relationship between the independent variables and dependent variable is linear. This means the dependent variable changes at a constant rate for a unit change in the independent variable, holding all other variables constant.
The main goal of multiple regression analysis is to find a linear equation that best fits the data. The multiple regression analysis also allows one to:
Predict the value of the dependent variable based on the values of the independent variables.
Understand how changes in the independent variables affect the dependent variable while considering the influence of other independent variables.
A simple Linear Regression model is one of the most fundamental techniques in machine learning and statistics. Whether you are a data science newbie or just brushing up on the basics, understanding linear regression is essential.
Table of Contents
Introduction
Frequently, we measure two or more variables on each individual and try to express the nature of the relationship between these variables (for example, in the simple linear regression model and correlation analysis). Using the regression technique, we estimate the relationship of one variable with another by expressing the one in terms of a linear (or more complex) function of another. We also predict the values of one variable in terms of the other. The variables involved in regression and correlation analysis are continuous. In this post, we will learn about the Simple Linear Regression Model.
Functional Relationship Between Variables
We are interested in establishing significant functional relationships between two (or more) variables. For example, the function $Y=f(X)=a+bx$ (read as $Y$ is a function of $X$) establishes a relationship to predict the values of variable $Y$ for the given values of variable $X$. In statistics (biostatistics), the function is a simple linear regression model or the regression equation.
The variable $Y$ is called the dependent (response) variable, and $X$ is called the independent (regressor or explanatory) variable.
In biology, many relationships can be appropriate over only a limited range of values of $X$. Negative values are meaningless in many cases, such as age, height, weight, and body temperature.
The method of linear regression is used to estimate the best-fitting straight line to describe the relationship between variables. The linear regression gives the equation of the straight line that best describes how the outcome of $Y$ increases/decreases with an increase/decrease in the explanatory variable $X$. The equation of the regression line is $$Y=\beta_0 + \beta_1 X,$$ where $\beta_0$ is the intercept (value of $Y$ when $X=0$) and $\beta_1$ is the slope of the line. Both $\beta_0$ and $\beta_1$ are the parameters (or regression coefficients) of the linear equation.
Estimation of Regression Coefficients in Simple Linear Regression Model
The best-fitting line is derived using the method of the \textit{Least Squares} by finding the values of the parameters $\beta_0$ and $\beta_1$ that minimize the sum of the squared vertical distances of the points from the regression line,
The dotted-line (best-fit) line passes through the point ($\overline{X}, \overline{Y}$).
The regression line $Y=\beta_0+\beta_1X$ is fit by the least-squares methods. The regression coefficients $\beta_0$ and $\beta_1$ are both calculated to minimize the sum of squares of the vertical deviations of the points about the regression line. Each deviation equals the difference between the observed value of $Y$ and the estimated value of $Y$ (the corresponding point on the regression.
The following table shows the \textit{body weight} and \textit{plasma volume} of eight healthy men.
Subject
Body Weight (KG)
Plasma Volume (liters)
1
58.0
2.75
2
70.0
2.86
3
74.0
3.37
4
63.5
2.76
5
62.0
2.62
6
70.5
3.49
7
71.0
3.05
8
66.0
3.12
Estimation of Paramters
The parameters $\beta_0$ and $\beta_1$ are estimated using the following formula (for simple linear regression model):
Regression coefficients are sometimes known as “beta-coefficients”. When the slope ($\beta_1=0$, then there is no relationship between $X$ and $Y$ variables. For the data above, the best-fitting straight line describing the relationship between plasma volume with body weight is $$Plasma\, Volume = 0.0857 +0.0436\times Weight$$ Note that the calculated values for $\beta_0$ and $\beta_1$ are estimates of the population values and, therefore, subject to sampling variations.
Real-Life Examples: Simple Linear Regression Models
Real Estate: Predicting House Prices (Estimate home prices based on size to guide buyers and sellers.) Independent Variable ($X$): Size of the house (sq ft) Dependent Variable ($Y$): Price of the house
Education: Predicting Student Scores (Teachers or students can predict likely outcomes based on study habits.) $X$: Hours studied $Y$: Exam scores
Healthcare: Predicting Blood Pressure (Understand how blood pressure tends to rise with age, aiding diagnosis.) $X$: Age of patient $Y$: Systolic blood pressure
Energy: Predicting Electricity Usage (Power companies use this to forecast demand and manage resources) $X$: Temperature (°C or °F) $Y$: Electricity consumption (kWh)
Manufacturing: Predicting Machine Failures $X$: Hours a machine has been in use (Predict maintenance schedules and avoid production delays.) $Y$: Number of breakdowns or wear percentage
Business: Predicting Sales Based on Advertising Spend (Helps businesses decide how much to invest in advertising.) $X$: Advertising expenditure (in $\$$) $Y$: Product sales (in units)
Agriculture: Predicting Crop Yield (Estimate yield based on expected rainfall to plan for food production.) $X$: Amount of rainfall (mm) $Y$: Crop yield (kg per acre)
Finance: Predicting Stock Prices (Although basic, it helps in forecasting trends over time (note: simple linear regression has limits in volatile markets)) $X$: Time (days or months) $Y$: Stock closing price
Transportation: Estimating Fuel Consumption (Predict fuel needs and optimize transportation costs.) $X$: Distance traveled (km) $Y$: Fuel used (liters)
E-commerce: Predicting Customer Spending (Analyze user behavior and optimize website experience for better conversion.) $X$: Time spent on the website $Y$: Amount spent on a purchase
