From Descriptive Statistics to Inferential Statistics: The Complete Guide
Introduction to Statistics
What is Statistics?
We deal with data every day, often without realizing it. Schools keep records of students’ marks, companies track their sales, and hospitals maintain patient records. But collecting data is one thing; understanding what it tells us is another.
This is where statistics becomes useful.
Statistics is the process of collecting, organizing, analyzing, and interpreting data to find useful information and make better decisions.
For example, a teacher who has the marks of 50 students might calculate the average, find the highest and lowest marks, and check how much the scores differ. These numbers give the teacher a clearer picture of the class’s performance without having to look at every result individually.
Simply put, statistics helps us make sense of data rather than just look at numbers.
Why is Statistics Important in Data Science?
In Data Science, having a large amount of data doesn’t automatically mean we understand it. We need statistical methods to find patterns, answer questions, and draw meaningful conclusions.
For example, an online shopping company might notice that some customers stop purchasing products after a few months. By studying their purchase history and shopping habits, a data scientist can look for patterns that might explain this behavior.
Statistics also helps us compare different groups, understand relationships between variables, and determine whether an observed difference is meaningful or could have occurred by chance. It is equally useful when evaluating machine learning models and understanding how reliable their predictions are.
Types of Statistics
Statistics is generally divided into two main branches: Descriptive Statistics and Inferential Statistics. Both help us understand data, but they serve different purposes.
A. Descriptive Statistics
Descriptive statistics is about summarizing the data we already have. Instead of going through every observation individually, we use a few statistical measures to get a clearer picture.
For example, consider the marks of five students:
60, 70, 75, 85, 90
We can calculate their average mark:
Mean = (60+70+75+85+90) / 5 = 76
The average is 76. We could also find the median, identify the highest mark, or measure how much the scores vary. Together, these values help us understand the students’ performance. This is descriptive statistics in practice.
B. Inferential Statistics
What if we want to understand a much larger group but cannot collect data from everyone? Instead, we can study a smaller group and use the results to draw conclusions about the whole population. This is the idea behind inferential statistics.
For example, suppose a university has 5,000 students and wants to estimate how much time they spend studying each day. Rather than asking every student, it selects 200 students and collects their responses.
The university can use this sample to estimate the average study time of all 5,000 students. The estimate may not be exact, since the selected students might not represent everyone equally well. Inferential statistics helps us account for this uncertainty and understand how reliable the estimate is.
Descriptive Statistics vs. Inferential Statistics
In practice, these two branches often work together. We usually start with descriptive statistics to understand our data. When we want to go beyond the observed data and draw conclusions about a larger population, we use inferential statistics.
Before exploring these methods in detail, let’s first understand four basic concepts: population, sample, parameter, and statistic.
Understanding Data — The Foundation
Before applying statistical methods, we need to understand the data we’re working with. Is it collected from an entire group or just a part of it? Does it represent categories or numerical values? And what kind of comparisons can we make using it?
These questions help us choose the right statistical methods. Let’s look at the basic concepts one by one.
Population vs. Sample
In statistics, we often want to study a particular group of people, objects, or observations. This entire group is called the population. However, collecting information from every member can be difficult, so we often study a smaller group called a sample.
For example, suppose a university has 5,000 students and wants to understand their daily study habits. If we collect information from all 5,000 students, we’re studying the population. If we select 200 students and collect their responses, those 200 students form the sample.
A sample should represent the population as closely as possible. If we select only students who regularly study in the library, for example, our results may not reflect the habits of all students.
Parameter vs. Statistic
Population and sample are closely related to two other important terms: parameter and statistic.
A parameter is a numerical value that describes a population, whereas a statistic is a numerical value calculated from a sample.
Continuing with our university example:
- Parameter: The actual average daily study time of all 5,000 students.
- Statistic: The average daily study time calculated from the 200 selected students.
Since collecting data from the entire population may not be practical, we often use a sample statistic to estimate the population parameter.
Remember the difference: a parameter describes the population, while a statistic describes the sample.
Types of Data
Data can be broadly divided into two types: qualitative (categorical) and quantitative (numerical). Understanding this distinction helps us decide how to summarize and analyze our data.
A. Qualitative (Categorical) Data
Qualitative data describes qualities, labels, or groups rather than numerical quantities. For example, gender categories, blood groups, and product types are categorical data.
Categorical data can be divided into the following types.
Nominal data: These are categories with no natural order. For example, blood groups such as A, B, AB, and O. None of these categories is higher or lower than another.
Dichotomous (Binary) data: This is a special type of categorical data with only two possible categories. Examples include Yes/No, Pass/Fail, and True/False.
Ordinal data: These are categories that have a meaningful order, but the differences between categories cannot be measured precisely. For example, customer satisfaction levels such as Poor, Average, Good, and Excellent.
B. Quantitative (Numerical) Data
Quantitative data represents quantities that can be expressed numerically. Examples include age, salary, height, number of products sold, and examination marks.
It has two main types.
Discrete data: These are countable values, usually whole numbers. For example, the number of students in a classroom or the number of products sold in a day. We can have 25 students, but not 25.5 students.
Continuous data: These are measurements that can take any value within a range, including decimal values. For example, a person’s height could be 172.5 cm, and their weight could be 68.4 kg.
The key difference is that discrete data is counted, while continuous data is measured.
Levels of Measurement (Data Scales)
Knowing whether data is categorical or numerical is not always enough. We also need to understand its level of measurement, which tells us whether the values have an order, equal intervals, or a meaningful zero.
There are four levels of measurement: nominal, ordinal, interval, and ratio.
1. Nominal Scale
Used to classify data into categories without any particular order.
Example: Blood groups — A, B, AB, O.
We can count how many observations belong to each category, but we cannot rank the categories.
2. Ordinal Scale
Categories have a meaningful order, but the differences between consecutive levels are not necessarily equal.
Example: Customer ratings — Poor, Fair, Good, Excellent.
We know that Excellent ranks above Good, but we cannot say the difference between them is equal to the difference between Good and Fair.
3. Interval Scale
Values are ordered and have equal intervals, but there is no true zero that represents the complete absence of the quantity.
Example: Temperature measured in Celsius. The difference between 10°C and 20°C is the same as between 20°C and 30°C.
However, 20°C does not mean twice as hot as 10°C because the Celsius scale has an arbitrary zero point.
4. Ratio Scale
Values have equal intervals and a meaningful zero, so both differences and ratios can be interpreted.
Example: Weight, height, distance, and age.
A weight of 60 kg is twice a weight of 30 kg. A value of zero represents the absence of the measured quantity.
These four scales help us understand what operations and comparisons make sense for a particular variable. For example, calculating an average temperature in Celsius is meaningful, while calculating an average blood group is not.
Sampling Techniques
What is Sampling and Why Do We Need It?
In statistics, we often want to understand a large population, but collecting information from every member is not always practical. It can take too much time, cost a lot of money, or require more resources than we have.
Sampling is the process of selecting a smaller group of observations from a population to learn about the larger group.
For example, suppose a university has 5,000 students and wants to know how satisfied they are with its facilities. Instead of asking every student, the university could select 300 students and collect their feedback.
The results can help the university understand overall student satisfaction, provided the selected students represent the larger population reasonably well.
The way we select a sample matters because a poorly selected sample can lead to misleading conclusions. This is why we need to understand the different sampling techniques.
Types of Sampling Techniques
Sampling techniques are broadly divided into two categories: probability sampling and non-probability sampling.
A. Probability Sampling
In probability sampling, each member of the population has a known, non-zero chance of being selected. Selection is based on a random process, which helps reduce selection bias and supports statistical generalization.
There are five common techniques.
1. Simple Random Sampling
Every member of the population has an equal chance of being selected. The selection is made randomly, much like drawing names from a box.
Example: A university randomly selects 300 student IDs from its complete list of 5,000 students.
This method is straightforward, but it requires a suitable list of the population.
2. Stratified Sampling
The population is divided into smaller groups called strata, based on a characteristic such as department, age group, or year of study. A random sample is then selected from each group.
Example: A university divides its students into first-year, second-year, and final-year groups, then randomly selects students from each group.
This helps ensure that important groups are represented in the sample.
3. Cluster Sampling
The population is divided into natural groups called clusters. Some clusters are randomly selected, and the researcher collects data from the selected clusters. In some designs, all members of the selected clusters are included.
Example: A university randomly selects five departments and surveys all students in those departments.
Cluster sampling can reduce the time and cost of collecting data when the population is spread across different locations or groups.
Stratified vs. cluster sampling: Stratified sampling selects members from every stratum, while cluster sampling selects only some clusters.
4. Systematic Sampling
In systematic sampling, we select every k-th member from a list after choosing a random starting point.
For example, if a researcher wants to select 100 students from a list of 1,000, the sampling interval is:
k = 1000 / 100 = 10
After choosing a random starting point, the researcher selects every 10th student.
This method is simple to apply, but it can produce biased results if the list contains a repeating pattern that matches the sampling interval.
5. Multistage Sampling
Multistage sampling involves selecting a sample in multiple stages, often by combining different sampling techniques.
Example: To study students across a state, a researcher might randomly select districts, then schools within those districts, and finally students within those schools.
This approach is useful when studying a large population spread across many locations.
B. Non-probability Sampling
In non-probability sampling, members are selected without using a random process that gives everyone a known chance of selection. The choice may depend on availability, willingness to participate, or the researcher’s judgment.
Five common techniques are described below.
1. Convenience Sampling
Participants are selected because they are easy to reach.
Example: A student collects feedback about a university canteen by asking classmates who are nearby.
This method is quick and inexpensive, but the participants may not represent the entire student population.
2. Voluntary Response Sampling
People choose for themselves whether to participate, usually in response to an open invitation.
Example: A university posts an online survey asking students to rate campus facilities, and anyone interested can respond.
Students with particularly strong opinions may be more likely to participate, which can affect the results.
3. Purposive (Judgmental) Sampling
The researcher deliberately selects participants who have specific knowledge, experience, or characteristics relevant to the study.
Example: A researcher studying cybersecurity practices interviews experienced cybersecurity professionals.
This is useful when specialized knowledge is needed, although the results may not represent the wider population.
4. Quota Sampling
The researcher divides the population into groups and sets a target number of participants for each group. Participants are then selected non-randomly until each target is reached.
Example: A researcher wants feedback from 100 students, with 50 undergraduate and 50 postgraduate students, and recruits available participants until both quotas are filled.
Although quotas ensure that the selected groups meet the required numbers, they do not guarantee that the sample represents each group accurately.
5. Snowball Sampling
Existing participants help the researcher find additional participants by referring people they know who meet the study requirements.
Example: A researcher studying the experiences of freelance data scientists interviews a few freelancers who then introduce other freelancers to the study.
This is useful when the target population is difficult to reach, but participants may refer people with similar backgrounds or experiences.
Probability Sampling vs. Non-probability Sampling
The main difference is how participants are selected, which affects how confidently we can generalize the results.
Neither approach is suitable for every situation. Probability sampling is useful when the goal is to estimate population characteristics, while non-probability sampling can be practical for exploratory research, specialized groups, or studies with limited resources.
Sampling Bias and Representativeness
Selecting a sample does not automatically mean that it represents the population well. The way participants are chosen can influence the results.
Sampling bias occurs when the selection process systematically favors certain members or groups, causing the sample to differ from the population in relevant ways.
For example, if a university wants to measure student satisfaction but collects responses only from students who regularly visit the library, students who rarely use the library may be underrepresented. The results may not reflect the views of all students.
Representativeness refers to how well a sample reflects the important characteristics of the population relevant to the study. A representative sample does not need to match the population in every detail, but it should avoid systematic imbalances that could distort the conclusions.
To improve representativeness, researchers can:
- Use random sampling when appropriate.
- Include relevant groups from across the population.
- Avoid relying only on easily accessible participants.
- Check for underrepresented groups and non-response.
- Choose a suitable sample size for the study.
One important point to remember is that a larger sample does not automatically eliminate sampling bias. Surveying 2,000 people from one department may still give a misleading picture of a university with students from many departments.
A carefully selected sample of suitable size can often provide more useful information than a much larger, biased sample.
Part I — Descriptive Statistics
1. Measures of Central Tendency
Suppose a teacher wants to understand how a class performed in an exam. Looking at every student’s mark individually may not give a quick overview. Instead, the teacher can use a single value to describe the centre of the marks.
This is the idea behind measures of central tendency. They help us identify a central or typical value in a dataset.
The three main measures are mean, median, and mode. Each describes the centre in a different way, and the choice depends on the type of data and how the values are distributed.
A. Mean (Arithmetic Mean)
The mean, commonly called the average, is calculated by adding all the observations and dividing the total by the number of observations.
Here, xi represents each observation and n is the total number of observations.
Example: Consider the marks of five students: 60, 70, 75, 85, 90
The average mark is 76.
The mean is easy to calculate and uses every observation in the dataset. However, it can be affected by unusually high or low values, which we will discuss shortly.
B. Median
The median is the middle value when the data is arranged in ascending or descending order. Unlike the mean, it depends on the position of the values rather than their total.
Consider the marks: 60, 70, 75, 85, 90
There are five observations, so the middle value is 75.
Median=75
When the dataset contains an even number of observations, the median is the average of the two middle values.
For example, consider: 60, 70, 80, 90
The two middle values are 70 and 80.
Median = (70 + 80) / 2 = 75
The median is particularly useful when the data contains extreme values because it is less affected by them than the mean.
C. Mode
The mode is the value that appears most frequently in a dataset.
For example, consider the number of books read by five students in a month: 2, 3, 3, 4, 5
The value 3 appears twice, while the other values appear only once.
Mode = 3
A dataset can have one mode, multiple modes, or no mode if every value occurs equally often.
The mode is also useful for categorical data. For example, if the most frequently purchased phone colour is black, black is the mode. Unlike the mean and median, the mode can be used with nominal categories.
Mean vs. Median vs. Mode
Let’s compare the three measures using the same dataset.
The mean and median are close in this example because the values do not contain an extreme observation. There is no mode because every value occurs only once.
The main difference is how each measure identifies the centre: the mean uses all values, the median uses the middle position, and the mode uses frequency.
Limitations of Mean and Median
Although the mean and median are widely used, neither describes every dataset perfectly.
Limitations of the mean:
- It is sensitive to extreme values.
- It may not represent a typical observation in highly skewed data.
- It cannot be meaningfully calculated for nominal categories such as blood groups.
Limitations of the median:
- It does not reflect the exact size of every observation.
- It does not use the magnitude of all values in the same way the mean does.
- It may not be suitable when further calculations require the arithmetic mean.
For this reason, we should consider the data and the purpose of the analysis before choosing a measure.
How Outliers Affect the Mean and Median
An outlier is a value that lies unusually far from the other observations. It can have a noticeable effect on the mean, while the median is generally more resistant.
Consider these monthly salaries:
₹20,000, ₹22,000, ₹24,000, ₹26,000, ₹28,000
The mean and median are both ₹24,000.
Now suppose one salary changes to ₹1,00,000:
₹20,000, ₹22,000, ₹24,000, ₹28,000, ₹1,00,000
The mean becomes:
Mean = 194000 / 5 = ₹38,800
But the median remains ₹24,000.
The unusually high salary pulls the mean upward, while the median stays at the middle value. In this situation, the median may better describe the typical salary of the group.
Compare the effect of the outlier
Original salaries vs. salaries with one unusually high value.
Mean and median salary — Comparison before and after adding an unusually high salary.
This is why the median is often preferred for salary, house-price, and income data, where a small number of unusually large values can pull the mean away from the typical observation.
When to Use Mean, Median, and Mode
The right measure depends on what the data looks like and what we want to understand.
For example, if we want to calculate average exam performance, the mean may be useful. If we want to describe typical salaries, the median may give a clearer picture. If a store wants to know which shoe size sells most often, the mode is the natural choice.
The key takeaway: Don’t choose a measure just because it is easy to calculate. First understand the data, check for outliers, and decide what you want the result to tell you.
Next, we’ll look at measures of position, including quantiles, quartiles, deciles, percentiles, and the five-number summary.
2. Measures of Position
What are Quantiles?
When we have a dataset, the mean or median can give us an idea of its centre, but sometimes we want to know where a particular value stands compared with the rest of the data.
For example, a student might want to know whether their exam mark is higher than most students’ marks. Similarly, a company might want to understand how an employee’s salary compares with others.
This is where measures of position become useful.
Quantiles are values that divide an ordered dataset into smaller groups containing approximately equal proportions of observations. They help us understand the relative position of values within a dataset.
Depending on how many groups we divide the data into, quantiles can be expressed as quartiles, deciles, or percentiles.
a. Quartiles
Quartiles divide an ordered dataset into four approximately equal parts. There are three quartiles: Q1, Q2, and Q3.
- First quartile (Q1): The 25th percentile. About 25% of the observations lie at or below this value.
- Second quartile (Q2): The 50th percentile. This is the median, with about 50% of the observations at or below it.
- Third quartile (Q3): The 75th percentile. About 75% of the observations lie at or below this value.
Let’s take the following ordered dataset:
10, 20, 30, 40, 50, 60, 70
The median is 40, so Q2 = 40. Using the common method of finding the median of the lower and upper halves while excluding the overall median:
- Q1 = 20
- Q2 = 40
- Q3 = 60
Quartiles are useful for understanding how data is distributed. They are also used to calculate the interquartile range (IQR), which measures the spread of the middle 50% of observations.
IQR = Q3 − Q1
For our example:
IQR = 60 − 20 = 40
b. Deciles
Deciles divide an ordered dataset into 10 approximately equal parts. They are represented by D1,D2,…,D9.
Each decile marks a position corresponding to a 10% increase in the cumulative proportion of the data.
For example:
- D1 corresponds to the 10th percentile.
- D5 corresponds to the 50th percentile, or median.
- D9 corresponds to the 90th percentile.
Suppose a student’s mark is at the 8th decile (D8). This means the mark is at approximately the 80th percentile: around 80% of the observations are at or below that value.
Deciles are useful when we want to divide data into ten groups, such as comparing household incomes or analyzing customer spending levels.
c. Percentiles
Percentiles divide an ordered dataset into 100 parts. They are commonly used to describe the relative position of an observation.
For example, if a student scores at the 90th percentile, their score is at or above approximately 90% of the scores in the reference group.
Some commonly used percentiles are:
- 25th percentile (P25): First quartile (Q1).
- 50th percentile (P50): Median (Q2).
- 75th percentile (P75): Third quartile (Q3).
- 90th percentile (P90): A value at or below which approximately 90% of observations fall.
Percentiles are widely used in exam results, salary comparisons, and child growth charts. For example, the 90th percentile of salaries indicates a salary level that is higher than or equal to approximately 90% of salaries in the comparison group.
Remember that a percentile is a position in a distribution, not the percentage of marks someone received. Scoring at the 90th percentile does not necessarily mean scoring 90% on an exam.
Relationship Between Quantiles, Quartiles, Deciles, and Percentiles
Quartiles, deciles, and percentiles are all types of quantiles. The main difference is the number of parts into which they divide the ordered data.
The measures are related because certain quartiles and deciles correspond to specific percentiles:
The fractional decile labels in this table indicate equivalent positions; the standard deciles themselves are usually numbered D1 through D9.
In practice, the exact numerical values of quartiles and percentiles can differ slightly across software packages because different calculation methods are used, especially for small datasets.
The main idea is simple: quantiles tell us where values sit within an ordered dataset. They help us compare observations, understand the spread of data, and identify unusually low or high values.
3. Measures of Dispersion
Suppose two students score an average of 75 marks in their exams. Does that mean they performed similarly? Not necessarily. One student’s marks might be close to 75 in every subject, while the other’s marks might vary considerably.
This is where measures of dispersion become useful. They tell us how spread out the values are in a dataset and whether the observations tend to stay close to the centre or vary widely.
The main measures of dispersion are range, interquartile range (IQR), mean absolute deviation (MAD), variance, and standard deviation. Each describes the spread of data in a slightly different way.
Let’s use the following dataset to understand them:
2, 4, 6, 8, 10
The mean is 6. We can now examine how far the values spread around it.
a. Range
The range is the simplest measure of dispersion. It is the difference between the largest and smallest values in a dataset.
Range = Maximum − Minimum
For our dataset: Range = 10 − 2 = 8
The range is 8, meaning the difference between the highest and lowest values is 8.
Range is easy to calculate, but it considers only the two extreme values. A single unusually high or low observation can change it significantly.
b. Interquartile Range (IQR)
The interquartile range measures the spread of the middle 50% of a dataset. It is calculated by subtracting the first quartile from the third quartile.
IQR = Q3 − Q1
Using the common median-of-halves method for our dataset, 2, 4, 6, 8, 10, we get:
- Q1 = 3
- Q3 = 9
Therefore, IQR = 9 − 3 = 6
The IQR is 6. Unlike the range, it focuses on the middle half of the data, so it is less affected by extreme values.
The IQR is also used to identify potential outliers. A common rule flags values below Q1 − 1.5 ( IQR ) or above Q3 + 1.5 ( IQR ). These are potential outliers, not necessarily errors.
c. Mean Absolute Deviation (MAD)
Mean absolute deviation measures the average distance of observations from their mean, using absolute values so that positive and negative deviations do not cancel each other out.
Here, xix_ixi represents each observation, xˉ\bar{x}xˉ is the mean, and nnn is the number of observations.
For the dataset 2, 4, 6, 8, 10, the mean is 6.
The absolute deviations from the mean are: 4, 2, 0, 2, 4
Therefore,
The MAD is 2.4, meaning that the average absolute distance from the mean is 2.4.
MAD is relatively easy to interpret because it uses the same units as the original data. It is different from the median absolute deviation, which is another measure sometimes abbreviated as MAD and is more resistant to outliers.
d. Variance
Variance measures how far the observations are spread around the mean. It is calculated by finding each observation’s deviation from the mean, squaring those deviations, and then taking their average.
Squaring the deviations prevents positive and negative differences from cancelling each other out. It also gives greater weight to larger deviations.
For our dataset, 2, 4, 6, 8, 10, the mean is 6.
| Value (x_i) | Deviation (x_i - 6) | Squared deviation |
|---------------|-----------------------|-------------------|
| 2 | -4 | 16 |
| 4 | -2 | 4 |
| 6 | 0 | 0 |
| 8 | 2 | 4 |
| 10 | 4 | 16 |
| Total | | 40 |If these five observations represent the entire population, the variance is:
The population variance is 8. A larger variance generally indicates greater spread around the mean, while a smaller variance indicates that the observations tend to be closer to it.
One thing to remember is that variance is expressed in squared units. For example, if the original data is measured in marks, the variance is measured in marks squared.
e. Standard Deviation
Standard deviation is the square root of variance. It measures how spread out the observations are around the mean, but expresses that spread in the same units as the original data.
For our dataset, the population variance is 8.
So, the population standard deviation is approximately 2.83.
This is often easier to interpret than variance because the result is in the original unit of measurement. For example, if we’re analyzing students’ marks, standard deviation is expressed in marks rather than marks squared.
Standard deviation is widely used in statistics and Data Science to understand variability, compare distributions, and assess how far observations tend to lie from the mean. However, like variance, it is sensitive to extreme values.
Population Variance vs. Sample Variance
When calculating variance, we need to know whether our data represents the entire population or only a sample.
Population variance is used when we have data for every member of the population. We divide the sum of squared deviations by the total number of observations, N.
Sample variance is used when our data comes from a sample and we want to estimate the population variance. We divide by n − 1, rather than nnn. This adjustment is called Bessel’s correction and helps correct the tendency of the unadjusted sample variance to underestimate population variance.
The formulas are:
Here, μ is the population mean, while xˉ is the sample mean.
For our five values, the sum of squared deviations is 40.
- Population variance: 40 / 5 = 8
- Sample variance: 40 / ( 5 − 1 ) = 10
The results differ because the formulas serve different purposes. In practice, when calculating variance from a sample to estimate population variability, we generally use the sample variance formula.
Variance vs. Standard Deviation
Both measures describe variability around the mean, but they express it differently.
For example, if the population variance of exam marks is 8 marks squared, the standard deviation is approximately 2.83 marks.
In short, variance is useful for calculations, while standard deviation is often easier to explain and interpret.
4. Shape of a Distribution
When we analyze a dataset, knowing its mean and standard deviation is useful, but these numbers don’t tell us the whole story. We also need to understand how the values are spread out and where most observations are concentrated.
A data distribution describes how values are spread across the possible range of a variable, including where observations tend to cluster and whether the data has a long tail on either side.
For example, consider the monthly salaries of employees in a company. Most employees might earn salaries within a similar range, while a few senior executives earn much more. This creates a different distribution from one in which salaries are spread evenly across the company.
One way to understand a distribution is to examine its shape. Three important aspects are symmetry, skewness, and kurtosis. They help us understand whether the data is balanced, whether one side stretches farther than the other, and how much the distribution’s tails contain extreme observations.
1. Symmetrical Distribution
A distribution is symmetrical when its left and right sides are approximately mirror images around the centre.
In a perfectly symmetrical distribution, values are balanced on both sides of the centre. For a typical unimodal, symmetric distribution, the mean and median are equal. If the distribution also has a single, well-defined mode at the centre, all three measures coincide.
For example, a dataset of exam marks may be approximately symmetrical if similar numbers of students score below and above the average.
Not every symmetrical distribution has a bell shape. A uniform distribution, for instance, can also be symmetrical. What matters is the balance between the two sides, not the exact shape of the curve.
2. Skewness
Real-world data is not always symmetrical. Sometimes most observations are concentrated on one side, while a smaller number of values stretch out toward the other side. This lack of symmetry is described by skewness.
Skewness tells us the direction and degree of asymmetry in a distribution. It is commonly described as approximately zero, negative, or positive.
A. Zero or Approximately Zero Skewness
A distribution with zero skewness is symmetrical in the sense measured by the skewness statistic. A value close to zero suggests little asymmetry, although it does not guarantee that the distribution is perfectly symmetrical.
For example, exam marks that are distributed fairly evenly around the centre may have skewness close to zero.
B. Negative (Left) Skewness
A negatively skewed distribution has a longer or heavier tail on the left side. Most observations tend to be toward the higher end, while a smaller number of low values stretch the distribution to the left.
For example, an easy exam may result in most students receiving high marks, with a few students scoring much lower.
C. Positive (Right) Skewness
A positively skewed distribution has a longer or heavier tail on the right side. Most observations tend to be toward the lower end, while a few unusually high values extend the distribution to the right.
For example, income data often shows positive skewness because most people earn within a certain range, while a smaller number earn substantially more.
One common mistake is to identify skewness by looking only at where the curve peaks. The important feature is the direction of the tail, not simply the location of the highest point.
Relationship Between Mean, Median, and Mode in Skewed Distributions
The mean, median, and mode can help us understand the shape of a distribution. In many familiar, unimodal distributions, their positions follow a predictable pattern.
These are common patterns, not universal rules. In real datasets, especially those with multiple peaks or unusual shapes, the mean, median, and mode may not follow these exact relationships.
This connection is useful when deciding which measure of central tendency to report. For strongly skewed data, the median often describes the typical observation better than the mean.
3. Kurtosis
Skewness describes asymmetry, but it doesn’t tell us everything about the tails of a distribution. Two distributions can have similar skewness and still differ in how frequently extreme observations occur.
This is where kurtosis becomes useful.
Kurtosis is a statistical measure related to the tails of a distribution and its tendency to produce extreme values. It is often introduced using three categories: mesokurtic, leptokurtic, and platykurtic.
A. Mesokurtic
A mesokurtic distribution has kurtosis similar to that of a normal distribution. The normal distribution has kurtosis 3 and excess kurtosis 0.
Example: The normal distribution is the standard reference for this category.
B. Leptokurtic
A leptokurtic distribution has higher kurtosis than a normal distribution. It has relatively heavier tails, indicating a greater tendency for extreme observations under comparable conditions.
Example: Financial returns can exhibit heavy tails, although their distributions vary.
C. Platykurtic
A platykurtic distribution has lower kurtosis than a normal distribution. Its tails are relatively lighter, meaning extreme observations are less prominent under comparable conditions.
Example: A uniform distribution is a common example of a platykurtic distribution.
An important point: kurtosis is not simply a measure of how tall or flat a curve looks. The height and sharpness of the peak may differ, but kurtosis is particularly concerned with tail behavior and the contribution of extreme observations.
An important point: kurtosis is not simply a measure of how tall or flat a curve looks. The height and sharpness of the peak may differ, but kurtosis is particularly concerned with tail behavior and the contribution of extreme observations.
Kurtosis vs Excess Kurtosis
You may see kurtosis reported in two different forms: ordinary kurtosis and excess kurtosis. The difference is simply the reference value used.
- Kurtosis: For a normal distribution, the value is 3.
- Excess kurtosis: Calculated by subtracting 3 from ordinary kurtosis. A normal distribution therefore has excess kurtosis of 0.
Excess Kurtosis = Kurtosis − 3
This gives us a simple way to interpret the result:
For example, if a dataset has kurtosis of 4.5, its excess kurtosis is:
4.5 − 3 = 1.5
The positive excess kurtosis indicates heavier tails relative to a normal distribution.
Keep in mind that sample estimates of skewness and kurtosis can vary, particularly when the dataset is small or contains extreme observations.
5. Covariance and Correlation
So far, we’ve looked at individual variables and learned how to describe their centre, spread, and distribution. But in Data Science, we often want to understand how two variables behave in relation to each other.
For example, does a student’s study time tend to increase along with their exam marks? Do house prices tend to rise with the size of the house? Do customers who spend more time on an app tend to spend more money?
Covariance and correlation help us understand how two variables change together. Although they are related concepts, they provide different kinds of information.
What Is Covariance?
Covariance measures whether two variables tend to move in the same direction or in opposite directions.
- If both variables tend to increase or decrease together, covariance is positive.
- If one tends to increase while the other decreases, covariance is negative.
- If there is no consistent linear pattern, covariance may be close to zero.
For example, suppose we record the number of hours students study and their exam marks. If students who study more hours generally receive higher marks, the two variables may have positive covariance.
For a population, covariance is calculated as:
Here, X and Y are the two variables, and μX and μY are their population means.
For a sample, we usually divide by n−1n-1n−1 instead of NNN when estimating population covariance.
Positive, Negative, and Zero Covariance
Covariance can indicate the direction in which two variables tend to change.
1. Positive covariance
The variables tend to move in the same direction. When one increases, the other tends to increase as well.
Example: Study hours and exam marks may have positive covariance.
2. Negative covariance
The variables tend to move in opposite directions. When one increases, the other tends to decrease.
Example: As the price of a product increases, the quantity demanded may decrease.
3. Zero or near-zero covariance
The variables show little or no linear co-movement.
Example: Two unrelated variables may have covariance close to zero. However, zero covariance does not always mean the variables are independent or have no relationship.
The important thing to remember is that covariance tells us the direction of co-movement, but its numerical value is not easy to interpret on its own.
Limitations of Covariance
Covariance is useful, but it has a few limitations.
- Its magnitude depends on the units: Changing the units of a variable changes the covariance. For example, measuring height in centimetres instead of metres changes its numerical value.
- It is difficult to compare across datasets: Covariance values from different pairs of variables may not be directly comparable.
- It mainly describes linear co-movement: A covariance of zero does not rule out a nonlinear relationship.
- It does not establish causation: Even if two variables move together, we cannot conclude that one causes the other.
These limitations make correlation useful because it standardizes the measure of linear association.
What Is Correlation?
Correlation measures the direction and strength of the relationship between two variables. Unlike covariance, correlation is standardized, so its value always lies between −1 and +1.
For example, a company might examine the relationship between money spent on advertising and product sales. Correlation helps determine whether higher advertising expenditure tends to be associated with higher sales and how closely the two variables follow a linear pattern.
Correlation is easier to interpret than covariance because it has a fixed range, regardless of the units used to measure the variables.
Pearson Correlation Coefficient
The Pearson correlation coefficient, usually represented by rrr for a sample, measures the strength and direction of the linear relationship between two numerical variables.
Its formula is:
Here:
- Cov(X,Y) is the sample covariance.
- sX is the sample standard deviation of X.
- sY is the sample standard deviation of Y.
Dividing covariance by the two standard deviations removes the effect of the measurement units. This makes the coefficient standardized and easier to compare.
For example, suppose study hours and exam marks have a Pearson correlation of r = 0.85. This indicates a strong positive linear relationship: students who study more hours tend to have higher marks in the observed data.
However, it does not mean that studying explains 85% of the marks or that studying alone causes the higher scores.
Interpreting Correlation Values (−1 to +1)
The value of the Pearson correlation coefficient tells us both the direction and the strength of the linear relationship.
One important point: a correlation of r = 0.8 is not necessarily twice as strong as a correlation of r = 0.4 . Correlation should be interpreted in context, and the scatter plot should be examined for outliers and unusual patterns.
Positive, Negative, and Zero Correlation
You can explore how different patterns of data affect the Pearson correlation coefficient in the interactive example below.
The three main patterns are:
- Positive correlation: The points generally move upward from left to right. For example, house size and house price often have a positive relationship.
- Negative correlation: The points generally move downward from left to right. For example, travel speed and journey time tend to have a negative relationship when the distance is fixed.
- Zero correlation: There is no linear pattern in the points. However, the variables may still be related in a curved or other nonlinear way.
Correlation describes the overall pattern, not necessarily a perfect relationship. Real-world data usually contains variation.
Correlation Does Not Imply Causation
This is one of the most important concepts in statistics.
Correlation does not prove that one variable causes changes in another. Two variables may be related because one influences the other, because a third variable affects both, or simply because of coincidence.
Consider ice cream sales and swimming-related accidents. Both might increase during summer. It would be incorrect to conclude that buying ice cream causes accidents.
A third factor — hot weather — can help explain the pattern. Warmer weather may increase both ice cream consumption and the number of people swimming, creating an association between the two variables.
To investigate whether one variable causes another, we need more than a correlation coefficient. Depending on the question, we may need controlled experiments, randomized studies, or careful analysis of other factors that could explain the relationship.
Part II — Inferential Statistics
Introduction to Probability
Probability is the study of how likely an event is to occur. It helps us understand uncertainty and express the likelihood of an event using a value between 0 and 1.
A probability of 0 means an event is impossible, while a probability of 1 means it is certain to happen. For example, when tossing a fair coin, the probability of getting heads is 0.5, or 50%.
Why is Probability Important in Statistics?
Probability helps us make sense of uncertain situations and draw conclusions from data. In inferential statistics, we use probability to understand how likely a result is to occur by chance and how reliably we can draw conclusions about a larger population from a sample.
For example, a company might use probability to estimate the likelihood that a customer will purchase a product. In Data Science, probability is also used in machine learning predictions, hypothesis testing, and risk analysis.
Important Terms in Probability
To understand probability, we first need to become familiar with a few basic terms.
1. Random Experiment
A random experiment is an action whose exact outcome cannot be predicted with certainty in advance, even though the possible outcomes are known.
For example, rolling a die is a random experiment because we know the result will be between 1 and 6, but we cannot predict the exact number beforehand.
2. Outcome and Sample Space
An outcome is a single possible result of a random experiment. The sample space is the set of all possible outcomes.
For example, when tossing a coin, the possible outcomes are heads and tails. Therefore, the sample space is: S = {H,T}
When rolling a die, the sample space is: S = {1,2,3,4,5,6}
3. Event
An event is a specific outcome or a group of outcomes that we are interested in.
For example, when rolling a die, getting an even number is an event. The event consists of the outcomes: A = {2,4,6}
4. Trial
A trial is one individual performance of a random experiment. For example, tossing a coin once is one trial. If we toss it five times, we have conducted five trials.
Probability Formula
When all possible outcomes are equally likely, we can calculate the probability of an event using the following formula:
For example, when rolling a fair six-sided die, the probability of getting an even number is:
P(Even)=36=0.5P(\text{Even})=\frac{3}{6}=0.5P(Even)=63=0.5
There are three favorable outcomes (2, 4, and 6) out of six possible outcomes. Therefore, the probability of getting an even number is 0.5, or 50%.
This formula applies when all possible outcomes are equally likely. Other methods are needed when outcomes have different probabilities.
Basic Properties and Rules of Probability
Probability follows a few basic rules that help us calculate the likelihood of events.
- Probability lies between 0 and 1: For any event A, 0 ≤ P(A) ≤ 1. An impossible event has probability 0, while a certain event has probability 1.
- Probability of the sample space is 1: Since the sample space includes all possible outcomes, P(S) = 1.
- Probability of an impossible event is 0: For example, getting a 7 when rolling a standard six-sided die has a probability of 0.
- Probabilities of mutually exclusive outcomes covering the sample space sum to 1: For a fair coin, heads and tails are mutually exclusive and cover all possible outcomes. Therefore, P(H)+P(T) = 0.5+0.5 = 1
- The probability of an event and its complement sums to 1: The complement means the event does not occur. Therefore, P(A^c)=1-P(A). If the probability of purchasing a product is 0.7, the probability of not purchasing it is 1 − 0.7 = 0.3
Types of Events
In probability, an event is an outcome or a group of outcomes that we are interested in. Depending on how events occur and how they are related to each other, they can be classified into different types. Understanding these relationships helps us calculate probabilities correctly.
1. Certain (Sure) Event
A certain event is an event that is guaranteed to happen. Its probability is always 1.
For example, when rolling a standard six-sided die, getting a number between 1 and 6 is a certain event.
P(A) = 1
2. Impossible Event
An impossible event is an event that cannot happen. Its probability is always 0.
For example, getting a 7 when rolling a standard six-sided die is impossible.
P(A) = 0
3. Complementary Event
A complementary event represents the situation in which an event does not occur. If A is an event, its complement is written as A^c.
For example, when tossing a coin, if event A is getting heads, then A^c is getting tails.
The probabilities of an event and its complement always add up to 1.
P(A^c) = 1 − P(A)
If the probability of a customer purchasing a product is 0.7, the probability of not purchasing it is 1 − 0.7 = 0.3
4. Independent Events
Two events are independent when the occurrence of one does not affect the probability of the other.
For example, tossing a coin twice gives two independent trials. Getting heads on the first toss does not change the probability of getting heads on the second toss.
For independent events, the multiplication rule is:
If the probability of heads on each toss is 0.5, the probability of getting heads on both tosses is:
P(HH) = 0.5 × 0.5 = 0.25
5. Dependent Events
Two events are dependent when the occurrence of one event affects the probability of the other.
For example, suppose a bag contains 3 red balls and 2 blue balls. If we pick one red ball and do not put it back, the probability of picking another red ball changes because there are fewer balls remaining.
For dependent events, we use conditional probability:
Here, P(B∣A) means the probability of event B occurring given that event A has already occurred.
6. Mutually Exclusive Events
Two events are mutually exclusive if they cannot occur at the same time in a single trial.
For example, when rolling a die, getting a 2 and getting a 5 are mutually exclusive events. A single roll cannot produce both results.
For mutually exclusive events: P(A∩B) = 0
It is important to remember that mutually exclusive events are generally not independent when both have non-zero probabilities, because the occurrence of one rules out the other.
Types of Relationships of Events
1. Addition Rule of Probability
The addition rule helps us calculate the probability that event A, event B, or both occur. It is used to find the probability of their union.
The general formula is: P(A∪B) = P(A) + P(B) − P(A∩B)
We subtract the intersection because outcomes shared by both events would otherwise be counted twice.
For mutually exclusive events, the intersection is zero, so the formula becomes: P(A∪B) = P(A) + P(B)
For example, when rolling a fair die, the probability of getting a 2 or a 5 is:
2. Multiplication Rule of Probability
The multiplication rule helps us calculate the probability that two events occur together.
For any two events, the general formula is: P(A∩B) = P(A) × P(B∣A)
For independent events, the probability of B remains unchanged after A, so we can simplify the formula: P(A∩B) = P(A) × P(B)
For example, when tossing a fair coin twice, the probability of getting heads on both tosses is: P(HH) = 1/2 × 1/2 = 1/4
The key difference is that the addition rule deals with the probability of events occurring as alternatives, while the multiplication rule deals with events occurring together. The appropriate formula depends on whether the events overlap and whether they are independent or dependent.
Types of Probability
In probability, we often want to understand how different events are related. For example, a company may want to know how many customers purchase a product, how many purchase it after seeing an advertisement, or how likely a customer is to buy the product given that they clicked on an advertisement.
Three important concepts help us answer these questions: marginal probability, joint probability, and conditional probability. Bayes’ theorem builds on these concepts to update probabilities when new information becomes available.
1. Marginal Probability
Marginal probability is the probability of a single event occurring, regardless of whether other events occur.
For example, suppose an online store studies 100 customers and finds that 40 purchased a product. The probability that a randomly selected customer purchased the product is:
P(Purchase) = 40/100 = 0.4
Therefore, the marginal probability of a purchase is 0.4, or 40%. We do not need to know whether these customers clicked on an advertisement or visited the website directly.
2. Joint Probability
Joint probability is the probability that two events occur together. It is represented by P(A∩B).
Suppose the same store finds that 30 out of 100 customers both clicked on an advertisement and purchased the product.
Let:
- A = Customer clicked on the advertisement.
- B = Customer purchased the product.
The joint probability is: P(A∩B) = 30/100 = 0.3
Therefore, the probability that a customer both clicked on the advertisement and purchased the product is 30%.
3. Conditional Probability
Conditional probability is the probability of an event occurring given that another event has already occurred. It helps us understand how additional information changes the probability of an event.
The formula is:
Here, P(B∣A) means the probability of event B occurring given that event A has occurred.
Using our example, suppose 50 customers clicked on the advertisement, and 30 of them purchased the product. The probability of a purchase among customers who clicked on the advertisement is:
P(B∣A) = 30/50 = 0.6
So, the conditional probability is 60%.
Notice the difference: the overall purchase rate was 40%, but among customers who clicked on the advertisement, it was 60%. This describes an association; it does not, by itself, prove that the advertisement caused the increase.
4. Independent Events and Conditional Probability
Two events are independent when the occurrence of one does not change the probability of the other.
For independent events, knowing that event A has occurred does not change the probability of event B. Therefore: P(B∣A) = P(B)
For example, tossing a fair coin and rolling a fair die are independent experiments. Getting heads on the coin does not affect the probability of rolling a 6 on the die.
However, if knowing that a customer clicked on an advertisement changes the probability that they purchase a product, those events are not independent in the observed data.
5. Bayes Theorem
Bayes’ theorem helps us update the probability of an event when we receive new information. It uses the initial probability of an event and the probability of observing the evidence under different conditions.
The formula is:
For example, suppose 20% of a store’s customers purchase a particular product. Of those who purchase it, 80% use a discount coupon. Among customers who do not purchase it, 10% still use a coupon on other items.
If a randomly selected customer used a coupon, what is the probability that they purchased the particular product?
Let:
- A = Customer purchased the product.
- B = Customer used a coupon.
We know: P(A) = 0.2, P(B∣A) = 0.8
First, calculate the overall probability of using a coupon:
Now apply Bayes’ theorem:
Therefore, the probability that the customer purchased the particular product, given that they used a coupon, is approximately 66.7%.
This example shows why Bayes’ theorem is useful: it allows us to revise an initial probability using new evidence. In Data Science, the same idea is used in applications such as spam filtering, fraud detection, and medical test interpretation.
Permutations and Combinations
When calculating probability, we often need to count the number of possible arrangements or selections. For example, we might want to know how many ways we can arrange students in a row or select a team from a group of people.
Permutations and combinations help us count these possibilities without listing every option individually. The main difference is whether the order of selection matters.
1. Factorial
A factorial is the product of all positive integers from a given number down to 1. It is represented by the symbol !.
For example: 5! = 5 × 4 × 3 × 2 × 1 = 120
Similarly, 3! = 3 × 2 × 1 = 6. By definition, 0! = 1.
Factorials are used in both permutation and combination formulas to calculate the number of possible arrangements and selections.
2. Permutations — When Order Matters
A permutation is an arrangement of items in which the order matters.
For example, suppose three students — A, B, and C — are competing for first and second place. The arrangements A-B and B-A represent different results because different students finish first.
The formula for selecting and arranging r items from n distinct items is:
Example: How many ways can we select a first-place and second-place winner from 5 students?
Therefore, there are 20 possible arrangements.
3. Combinations — When Order Does Not Matter
A combination is a selection of items in which the order does not matter.
For example, suppose we need to select two students from a group of five to represent a class. Selecting A and B is the same as selecting B and A because the team remains the same.
The formula for selecting r items from n distinct items is:
Example: How many ways can we select 2 students from 5?
Therefore, there are 10 possible combinations.
Applications in Probability
Permutations and combinations are useful when calculating probabilities involving multiple possible arrangements or selections.
For example, suppose a box contains 3 defective products and 7 non-defective products. If we randomly select 2 products without replacement, what is the probability that both are defective?
The total number of ways to select 2 products from 10 is:
The number of ways to select 2 defective products from the 3 defective products is:
Therefore, the probability that both selected products are defective is:
So, the probability is approximately 6.67%.
This example shows how combinations help us calculate probabilities when we select items without considering their order.
Permutations and combinations provide the counting techniques needed for more advanced probability concepts, including random variables and probability distributions.
Random Variables and Expected Value
In probability, we often deal with experiments that can produce different outcomes. A random variable helps us represent these outcomes using numbers. Once we understand random variables, we can calculate the expected value, variance, and standard deviation to understand the average outcome and how much it can vary.
What is a Random Variable?
A random variable is a variable that assigns a numerical value to each possible outcome of a random experiment. It is usually represented by a capital letter, such as X.
For example, when tossing a coin twice, let X represent the number of heads obtained. The possible values of X are 0, 1, and 2.
- No heads: X = 0
- One head: X = 1
- Two heads: X = 2
The value of X depends on the outcome of the experiment.
Discrete Random Variables
A discrete random variable takes distinct, countable values. These values are often whole numbers obtained by counting something.
For example, let X represent the number of defective products in a batch of three products. The possible values are: X ∈ {0,1,2,3}
Other examples include the number of customers visiting a store, the number of heads in coin tosses, and the number of errors in a program.
Continuous Random Variables
A continuous random variable can take any value within a given range, including decimal values. These variables usually represent measurements rather than counts.
For example, the time a customer spends on a website might be 2 minutes, 2.5 minutes, or 2.53 minutes. Similarly, height, weight, temperature, and waiting time are examples of continuous random variables.
The key difference is that discrete variables count distinct outcomes, while continuous variables measure quantities that can take any value within a range.
Expected Value (Mathematical Expectation)
The expected value is the long-run average value we would expect from a random experiment if it were repeated many times under the same conditions. It considers both the possible outcomes and their probabilities.
Get V V Srimannarayana Reddy Tetali’s stories in your inbox
Join Medium for free to get updates from this writer.
For a discrete random variable, the formula is:
This means multiplying each possible value by its probability and adding the results.
Example: Suppose a fair die is rolled. The expected value is:
The expected value is 3.5. This does not mean the die can show 3.5; it means the average result approaches 3.5 over many rolls.
Variance of a Random Variable
Variance measures how spread out the possible values of a random variable are around its expected value. A small variance means the values tend to stay close to the expected value, while a large variance indicates greater variation.
For a discrete random variable, variance can be calculated as:
An equivalent formula is:
Here, μ = E(X) is the expected value.
For a fair die, the expected value is 3.5 and the variance is:
Var(X) = 35/12 ≈ 2.92
Variance is useful for understanding uncertainty and comparing the spread of possible outcomes. However, because deviations are squared, variance is expressed in squared units.
Probability Mass, Density, and Distribution Functions
A random variable can take different values, and each value or range of values has a certain probability. A probability distribution describes how these probabilities are assigned. Depending on whether the random variable is discrete or continuous, we use different functions to represent its distribution.
What is a Probability Distribution?
A probability distribution describes all the possible values of a random variable and the probabilities associated with them.
For example, when rolling a fair six-sided die, the random variable XXX represents the number obtained. Each possible value from 1 to 6 has a probability of 1/61/61/6.
A probability distribution helps us understand which outcomes are possible and how likely they are. It is also the foundation for many statistical methods used in Data Science.
1. Probability Mass Function (PMF)
A Probability Mass Function (PMF) is used for discrete random variables. It gives the probability of each individual value.
For a fair die, the PMF is: P(X = x) = 1 / 6 , x ∈ {1,2,3,4,5,6}
This means the probability of obtaining any particular number is 1/61/61/6.
A valid PMF follows two basic rules:
- Each probability must be between 0 and 1.
- The probabilities of all possible values must add up to 1.
PMFs are useful for countable outcomes, such as the number of defective products, customers visiting a store, or heads obtained in coin tosses.
2. Probability Density Function (PDF)
A Probability Density Function (PDF) is used for continuous random variables. It describes how probability is distributed across possible values.
For example, the time a customer spends on a website or the height of a person can be treated as continuous variables.
Unlike a PMF, a PDF does not give the probability of an exact value. Instead, we calculate the probability that a variable falls within a particular interval by finding the area under the density curve.
For a continuous random variable X:
Here, f(x)f(x)f(x) is the probability density function.
For a continuous variable, the probability of obtaining one exact value is zero:
P( X = a ) = 0
This does not mean that the value is impossible. It means that probability is measured over intervals rather than individual points.
A valid PDF must be non-negative, and the total area under its curve must equal 1.
3. Cumulative Distribution Function (CDF)
A Cumulative Distribution Function (CDF) gives the probability that a random variable is less than or equal to a particular value.
It is defined as:
For example, when rolling a fair die, the probability of obtaining a number less than or equal to 3 is:
So, the CDF at 3 is 0.5, or 50%.
Unlike a PMF or PDF, the CDF can be used for both discrete and continuous random variables. Its value never decreases as x increases, and it ranges from 0 to 1.
Discrete vs. Continuous Probability Distributions
Probability distributions can be classified into two main types based on the random variable.
Discrete probability distributions describe countable values. For example, the number of defective products in a batch can be 0, 1, 2, and so on. Each value has an individual probability that can be calculated using a PMF.
Continuous probability distributions describe measurements that can take any value within a range. Examples include height, temperature, and waiting time. Their probabilities are calculated over intervals using a PDF.
Discrete Probability Distributions
Discrete probability distributions describe the probabilities of the possible values of a discrete random variable. They are useful when we want to model countable outcomes, such as the number of defective products, customer purchases, or successful attempts.
Let’s look at four commonly used discrete probability distributions.
1. Bernoulli Distribution
A Bernoulli distribution describes a random experiment with only two possible outcomes: success or failure. These outcomes are usually represented by 1 and 0.
Assumptions:
- There are only two possible outcomes.
- The probability of success is p.
- The probability of failure is 1 − p.
Probability formula:
Mean and variance:
Example: Suppose a customer has a 70% probability of purchasing a product. Let X=1 if the customer purchases it and X=0 otherwise.
Here, p = 0.7
E(X) = 0.7
Var(X) = 0.7 (0.3) = 0.21
The expected value is 0.7, representing the purchase probability. It does not mean that an individual customer makes 0.7 of a purchase.
2. Binomial Distribution
A binomial distribution models the number of successes in a fixed number of independent trials, where each trial has two possible outcomes and the same probability of success.
Assumptions:
- The number of trials, n, is fixed.
- Each trial has two possible outcomes.
- Trials are independent.
- The probability of success, p, remains constant.
Probability formula:
Mean and variance:
Example: Suppose 10 customers independently visit an online store, and each has a 20% probability of purchasing a product. We want to find the probability that exactly 2 customers make a purchase.
Here, n = 10, k = 2, and p = 0.2
The probability is approximately 30.2%.
The expected number of purchases is 10 (0.2) = 2, and the variance is
10 (0.2) (0.8) = 1.6.
3. Poisson Distribution
A Poisson distribution models the number of times an event occurs within a fixed interval of time, area, distance, or another defined measure, when events occur at a constant average rate independently of one another.
Assumptions:
- We count events within a fixed interval.
- Events occur independently.
- The average rate, λ, is constant.
- In the standard Poisson process, events occur singly rather than in simultaneous groups.
Probability formula:
Here, λ is the expected number of events in the interval, and k is the observed number of events.
Mean and variance:
Example: Suppose an online store receives an average of 3 customer support requests per hour. Assuming a Poisson model is appropriate, what is the probability of receiving exactly 2 requests in an hour?
Here, λ = 3 and k = 2
Therefore, the probability is approximately 22.4%. Both the expected number of requests and the variance are 3.
4. Multinomial Distribution
A multinomial distribution extends the binomial distribution to experiments with more than two possible outcomes. It models the number of times each outcome occurs across a fixed number of independent trials.
Assumptions:
- The number of trials, n, is fixed.
- Each trial produces exactly one outcome from several possible categories.
- Trials are independent.
- The probability of each category remains constant.
Probability formula:
Mean and variance for each category:
Example: Suppose 10 customers independently choose one of three product categories: A, B, or C. Their respective probabilities are 0.5, 0.3, and 0.2.
The expected number of customers choosing each category is:
- Product A: 10(0.5) = 5
- Product B: 10(0.3) = 3
- Product C: 10(0.2) = 2
For Product A, the variance is:
These expected counts describe the average pattern over repeated groups of 10 customers; the actual counts can differ.
Comparing the Four Distributions
The main difference is the type of count being modeled. Bernoulli handles one yes-or-no trial, binomial counts successes across repeated trials, Poisson models event counts over an interval, and multinomial handles multiple possible categories.
Continuous Probability Distributions
Continuous probability distributions describe random variables that can take any value within a given range. Unlike discrete variables, which take specific, countable values, continuous variables can take decimal values as well.
For example, a person’s height could be 172 cm, 172.5 cm, or 172.56 cm. Other examples include delivery time, temperature, product lifetime, and income.
Different continuous probability distributions are used to model different patterns in data. Let’s explore six commonly used distributions and understand when each one is useful.
1. Uniform Distribution
The uniform distribution is used when every value within a specified interval has the same probability density. In other words, no part of the interval is favored over another.
For example, suppose a bus arrives at any time within a 10-minute interval, and its arrival time is equally likely to fall anywhere within that interval. We can model the waiting time using a uniform distribution, assuming arrivals are equally likely throughout those 10 minutes.
The probability density function (PDF) is:
Here, a is the minimum value and b is the maximum value.
The mean and variance are:
When is it used?
Uniform distributions are useful when values are equally likely across a fixed interval. Examples include randomly selecting a time within an interval, generating random numbers within a range, and modeling certain simulation inputs.
However, the uniform distribution should only be used when the equal-likelihood assumption is reasonable.
2. Exponential Distribution
The exponential distribution models the waiting time between events that occur independently at a constant average rate.
For example, a customer support team might receive an average of five requests per hour. The exponential distribution can help model the waiting time until the next request, assuming requests follow a Poisson process with a constant rate.
Its probability density function is:
Here, λ represents the average event rate per unit of time.
The mean and variance are:
For example, if a system receives an average of five requests per hour, then λ = 5. The expected waiting time between requests is:
E(X) = 1 / 5 hour = 12 minutes
When is it used?
The exponential distribution is commonly used to model waiting times, such as the time between customer arrivals, the time between incoming calls, or the lifetime of a component with a constant failure rate.
One important property is that it is memoryless: under the model, the time already spent waiting does not change the distribution of the remaining waiting time.
3. Normal Distribution
The normal distribution is one of the most widely used probability distributions in statistics. It is a symmetric, bell-shaped distribution in which most observations lie near the mean, while fewer observations occur farther away from it.
For example, measurements such as human heights within a reasonably homogeneous population may be approximately normally distributed. Measurement errors can also often be modeled using a normal distribution.
Its probability density function is:
For a normal distribution, the mean, median, and mode are equal.
A useful rule is the 68–95–99.7 rule:
- Approximately 68% of observations fall within one standard deviation of the mean.
- Approximately 95% fall within two standard deviations.
- Approximately 99.7% fall within three standard deviations.
For example, if students’ marks follow an approximately normal distribution with a mean of 70 and a standard deviation of 10, around 68% of the marks would fall between 60 and 80.
When is it used?
The normal distribution is useful for modeling approximately symmetric measurements, measurement errors, and many sampling distributions. It also plays an important role in statistical inference, including confidence intervals and hypothesis testing.
However, not every real-world variable follows a normal distribution. Data with strong skewness, extreme outliers, or natural lower bounds may require a different model.
4. Standard Normal Distribution and Z-Scores
The standard normal distribution is a special type of normal distribution with a mean of 0 and a standard deviation of 1.
Instead of working directly with values from different normal distributions, we can convert observations into a common scale using a Z-score.
A Z-score tells us how many standard deviations an observation lies above or below the mean.
The formula is:
Here, x is the observed value, μ is the mean, and σ is the standard deviation.
For example, suppose a student scores 85 on a test where the class mean is 70 and the standard deviation is 10.
z = (85 − 70) / 10 = 1.5
The student’s Z-score is 1.5, meaning the score is 1.5 standard deviations above the class mean.
Z-scores also help us calculate probabilities using the standard normal distribution. For example, we can estimate the proportion of observations below a particular score or compare scores from tests with different means and standard deviations.
When is it used?
Z-scores are useful for comparing observations measured on different scales, identifying unusually high or low values, and calculating probabilities under a normality assumption. They are also used in hypothesis testing and confidence interval calculations.
One caution: a Z-score can be calculated for any numerical observation, but interpreting it using standard normal probabilities requires an appropriate distributional assumption.
5. Log-normal Distribution
The log-normal distribution is used when a variable is positive and its natural logarithm follows a normal distribution.
Unlike a normal distribution, which is symmetric, a log-normal distribution is typically right-skewed. Most observations may be relatively small, while a smaller number of observations are much larger.
For example, income, property prices, and some biological measurements may show a log-normal pattern. These variables cannot take negative values and may have a long right tail.
If X follows a log-normal distribution, then:
The mean of the original variable is:
Its variance is:
When is it used?
The log-normal distribution is useful for modeling positive, right-skewed data, such as certain income distributions, house prices, financial quantities, and product lifetimes affected by multiplicative factors.
For instance, if house prices in an area are concentrated around lower and middle price ranges but a few properties are extremely expensive, a log-normal distribution may provide a reasonable model.
It is not suitable for every positive, skewed variable, so the fit should be checked against the actual data.
6. Pareto Distribution
The Pareto distribution is used to model situations where a small proportion of observations account for a large proportion of the total. It has a heavy right tail, meaning extremely large values are more likely than they would be under many lighter-tailed distributions.
A familiar example is wealth distribution, where a relatively small number of individuals may hold a large share of total wealth. Similar patterns can sometimes appear in company sizes, website traffic, and sales across products.
One common form of the Pareto probability density function is:
For this version of the distribution, the mean exists when α > 1:
The variance exists when α > 2:
These conditions are important because heavy-tailed data can have a mean or variance that is extremely large or does not exist mathematically.
When is it used?
The Pareto distribution is useful for modeling heavy-tailed data, such as wealth, large insurance claims, certain business revenues, and website traffic. It can also help describe situations related to the Pareto principle, often summarized as the “80/20 rule.”
However, the 80/20 rule is a general observation, not a defining property of every Pareto distribution. The actual proportion depends on the data and the distribution’s parameters.
Sampling Distributions and the Central Limit Theorem
When we work with a population, we often use a sample instead of collecting data from every individual. But a natural question arises: How much can the result of one sample vary from another sample?
Sampling distributions and the Central Limit Theorem help us answer this question.
What is a Sampling Distribution?
A sampling distribution is the distribution of a statistic calculated from many different samples taken from the same population.
For example, suppose a university has 10,000 students. We randomly select 100 students and calculate their average study time.
If we repeat this process many times, each sample will have a slightly different average.
The collection of all these sample averages forms the sampling distribution of the sample mean.
It is important to remember that a sampling distribution is not the distribution of the original individual observations. It describes how a sample statistic behaves across repeated samples.
Sampling Distribution of the Sample Mean
The sampling distribution of the sample mean describes how sample means vary when we repeatedly take samples of the same size from a population.
Suppose the population has:
- Population mean = μ
- Population standard deviation = σ
- Sample size = n
The mean of the sampling distribution of the sample mean is:
This means that, on average, the sample mean is centered around the population mean.
For example, if the average height of a population is 170 cm, different samples might have means such as 168.5 cm, 171 cm, 169.8 cm, and 170.4 cm.
The sample means will vary, but they tend to be centered around 170 cm.
Standard Error
The standard error (SE) measures how much a sample statistic, such as the sample mean, is expected to vary from sample to sample.
For the sample mean:
As the sample size increases, the standard error decreases.
For example, suppose the population standard deviation is 20.
For a sample of 100:
For a sample of 400:
So, larger samples generally give more stable estimates of the population mean.
When the population standard deviation is unknown, which is common in practice, the sample standard deviation ss is often used:
Standard deviation describes the variation among individual observations, while standard error describes the variation of a sample statistic across samples.
Central Limit Theorem (CLT)
The Central Limit Theorem (CLT) is one of the most important ideas in statistics.
It states that when we take sufficiently large random samples from a population, the sampling distribution of the sample mean tends to become approximately normal, even if the original population is not normally distributed.
For example, imagine that individual customer spending is highly skewed because a few customers spend very large amounts.
If we repeatedly take sufficiently large samples and calculate the mean spending for each sample, those sample means will tend to form an approximately normal distribution.
The CLT is important because it allows us to use many statistical methods based on the normal distribution even when the original data is not normally distributed.
The approximation generally improves as the sample size increases, although the required sample size depends on the population’s distribution and other conditions.
Why Sampling Distributions Matter in Inferential Statistics
Sampling distributions are important because inferential statistics is based on samples.
We usually do not know the exact population value. Instead, we calculate a statistic from a sample and use its sampling distribution to understand how reliable that statistic is.
For example, a sample mean can be used to estimate a population mean. The sampling distribution and standard error help us understand how much that estimate might vary from sample to sample.
These ideas form the foundation for important inferential methods such as:
- Confidence intervals
- Hypothesis testing
- Z-tests and t-tests
- Estimating population parameters
In simple terms:
Population → Sample → Sample Statistic → Sampling Distribution → Inference about Population
Part III — Estimation and Hypothesis Testing
Estimation
What is Statistical Estimation?
In statistics, we often want to know something about a population, such as its average income, average height, or proportion of customers who are satisfied.
Since collecting data from the entire population may not be practical, we take a sample and use the sample data to estimate the unknown population value.
For example, if a university has 10,000 students, we may select 200 students and calculate their average study time. We can then use this sample average to estimate the average study time of all 10,000 students.
This process of using sample data to estimate an unknown population value is called statistical estimation.
Population Parameter vs. Sample Estimate
A population parameter is a numerical value that describes the entire population.
A sample estimate is a value calculated from a sample and used to estimate the corresponding population parameter.
For example:
- Population mean μ → population parameter
- Sample mean xˉ → sample estimate
Suppose the actual average study time of all students is unknown. If a sample of 200 students has an average study time of 3.5 hours, then 3.5 hours is the sample estimate of the population mean.
Point Estimation
A point estimate is a single value used to estimate a population parameter.
For example, if the average income of a sample is ₹35,000, we can use ₹35,000 as a point estimate of the population’s average income.
Common examples include:
The main advantage of a point estimate is that it is simple and easy to interpret. However, it does not tell us how uncertain the estimate is. For that, we use interval estimation.
Properties of a Good Estimator
A good estimator should provide reliable estimates of the population parameter. Three important properties are unbiasedness, consistency, and efficiency.
1. Unbiasedness
An estimator is unbiased if its expected value is equal to the true population parameter.
For example, the sample mean is an unbiased estimator of the population mean: E(Xˉ) = μ
This means that if we repeatedly take samples and calculate their means, the average of those sample means will equal the population mean.
2. Consistency
An estimator is consistent if it gets closer to the true population parameter as the sample size increases.
For example, as we collect more observations, the sample mean generally becomes a more reliable estimate of the population mean.
3. Efficiency
When two estimators are unbiased, an estimator is considered more efficient if it has less variability.
In simple terms, a more efficient estimator tends to produce estimates that are more tightly clustered around the true population parameter.
Interval Estimation
A point estimate gives us one value, but a sample estimate is rarely exactly equal to the population parameter.
Interval estimation addresses this uncertainty by providing a range of plausible values for the population parameter.
For example, instead of saying: “The average income is ₹35,000.”
we might say: “ The estimated average income is between ₹33,000 and ₹37,000.”
This range is called an interval estimate.
The most commonly used interval estimate is the confidence interval.
Confidence Intervals
A confidence interval (CI) is a range calculated from sample data that is used to estimate an unknown population parameter.
A general form is: Estimate ± Margin of Error
For example: ₹35,000 ± ₹2,000
gives the interval: [₹33,000, ₹37,000]
The interval gives us a range rather than relying on a single point estimate.
Confidence Level and Significance Level
The confidence level indicates the long-run coverage associated with a confidence interval procedure.
Common confidence levels are:
- 90%
- 95%
- 99%
For example, a 95% confidence interval procedure is designed so that, over many repeated samples, about 95% of the intervals produced by that procedure would contain the true population parameter.
It is important not to interpret this as saying there is a 95% probability that the already-calculated fixed parameter lies inside one particular interval.
The significance level, represented by α\alpha, is commonly related to the confidence level: α = 1 − Confidence Level
Therefore:
- 90% confidence level → α = 0.10
- 95% confidence level → α = 0.05
- 99% confidence level → α = 0.01
These concepts become especially important in hypothesis testing.
Confidence Interval for a Population Mean
When estimating a population mean, the confidence interval depends on whether the population standard deviation is known.
If the population standard deviation σ\sigma is known, a common interval is:
When the population standard deviation is unknown, the t-distribution is generally used:
where ss is the sample standard deviation.
For example, if a sample of students has an average study time of 3.5 hours, a confidence interval can provide a range of plausible values for the average study time of the entire population.
Confidence Interval for a Population Proportion
A population proportion represents the fraction or percentage of a population having a particular characteristic.
For example, suppose 120 out of 200 surveyed customers are satisfied:
The sample proportion is 60%.
A commonly used approximate confidence interval for a population proportion is:
The resulting interval gives a range of plausible values for the population proportion.
Z-Interval vs. t-Interval
Both z-intervals and t-intervals can be used to estimate a population mean, but the appropriate choice depends mainly on whether the population standard deviation is known.
The t-distribution has heavier tails than the standard normal distribution, which accounts for the additional uncertainty from estimating the population standard deviation using the sample.
Factors Affecting Confidence Interval Width
The width of a confidence interval depends mainly on three factors:
1. Sample size
A larger sample generally produces a narrower confidence interval because the standard error decreases.
2. Confidence level
A higher confidence level produces a wider interval.
For example, a 99% confidence interval will generally be wider than a 95% confidence interval calculated from the same data.
3. Variability in the data
Greater variability generally produces a wider interval because the estimate is less precise.
In simple terms:
Larger sample → narrower interval
Higher confidence level → wider interval
Greater variability → wider interval
Confidence intervals help us understand the uncertainty in our estimates. The next step is to move from estimating population parameters to testing whether a claim about a population is supported by the sample data.
Introduction to Hypothesis Testing
After learning estimation and confidence intervals, the next step is to understand how we can test claims about a population using sample data.
For example, a company may claim that a new website design increases the average time customers spend on the website. We can collect sample data and use hypothesis testing to determine whether the evidence supports this claim.
What is Hypothesis Testing?
Hypothesis testing is a statistical method used to evaluate a claim about a population using sample data.
Since we usually cannot observe the entire population, we use a sample and determine whether the observed result provides enough evidence against a particular assumption.
For example, suppose a company claims: “The average delivery time is 30 minutes.”
We can take a sample of deliveries and test whether the data provides enough evidence to question this claim.
Hypothesis testing does not prove that a hypothesis is absolutely true or false. Instead, it helps us decide whether the sample provides sufficient statistical evidence against the null hypothesis.
Why Do We Need Hypothesis Testing?
Sample results naturally vary from one sample to another. Therefore, an observed difference does not automatically mean that a real difference exists in the population.
For example, suppose:
- Old website average conversion rate = 10%
- New website sample conversion rate = 11%
The increase from 10% to 11% might be due to a genuine improvement, or it might simply be due to random variation in the sample.
Hypothesis testing provides a structured way to evaluate this uncertainty.
It is commonly used in:
- A/B testing
- Medical research
- Quality control
- Business experiments
- Comparing groups
- Evaluating changes or interventions
Null Hypothesis and Alternative Hypothesis
Hypothesis testing starts by defining two competing statements.
1. Null Hypothesis (H_0)
The null hypothesis represents the default assumption or a statement of no effect, no difference, or a specified population value.
For example: H_0 : μ = 30
This could represent a claim that the average delivery time is 30 minutes.
2. Alternative Hypothesis (H_1 or H_a)
The alternative hypothesis represents what we are trying to find evidence for.
For example: H_a : μ ≠ 30
This means the average delivery time is different from 30 minutes.
The two hypotheses must be defined before examining the test result.
One-Tailed vs. Two-Tailed Tests
The alternative hypothesis determines whether the test is one-tailed or two-tailed.
1. Two-Tailed Test
A two-tailed test is used when we want to know whether a parameter is different from a particular value.
Here, both values above and below 30 are considered evidence against the null hypothesis.
2. One-Tailed Test
A one-tailed test is used when the research question has a specific direction.
For example, if we want to test whether a new process reduces delivery time:
If we want to test whether it increases delivery time:
The direction of the test should be decided before looking at the results.
Significance Level (α)
The significance level, represented by α, is the threshold used to decide whether the evidence against the null hypothesis is strong enough.
Common choices are: α = 0.05, 0.01, 0.10
For example, if we choose: α = 0.05
we use a 5% significance level.
The significance level is also related to the probability of making a Type I error: rejecting the null hypothesis when it is actually true.
Test Statistic
The test statistic is a value calculated from the sample data.
It measures how far the observed sample result is from what we would expect if the null hypothesis were true.
For example, a test statistic for a population mean might be calculated as:
A large positive or negative value indicates that the observed sample result is far from the value expected under the null hypothesis.
Different statistical tests use different test statistics.
Critical Value and Critical Region
The critical value is a boundary that separates results that are considered sufficiently unusual under the null hypothesis from those that are not.
The critical region, or rejection region, contains the values of the test statistic that lead to rejection of H_0.
For example, in a two-tailed z-test at a 5% significance level, the critical values are approximately:
The critical values depend on the significance level and the type of test.
P-value
The p-value tells us how unusual the observed result, or a more extreme result, would be if the null hypothesis were true.
A small p-value indicates that the observed result would be relatively unusual under H0H_0.
A common decision rule is:
For example, if p = 0.03 and α = 0.05
then p < α, so we reject H_0.
Importantly, the p-value is not the probability that the null hypothesis is true.
Step-by-Step Hypothesis Testing Workflow
A typical hypothesis test follows these steps:
Step 1: Define the hypotheses
Set up H_0 and H_a.
Step 2: Choose the significance level
Usually α=0.05, depending on the context.
Step 3: Select the appropriate statistical test
The choice depends on the data, research question, sample design, and assumptions.
Step 4: Calculate the test statistic
Use the sample data to calculate the appropriate statistic.
Step 5: Calculate the p-value or determine the critical region
Use the relevant probability distribution.
Step 6: Make a decision
Compare the p-value with α, or compare the test statistic with the critical value.
Step 7: Interpret the result in context
State what the result means for the original research question.
A simple way to remember the process is:
Question → Hypotheses → Significance Level → Test → Test Statistic → P-value → Decision → Conclusion
Statistical Significance vs. Practical Significance
A result can be statistically significant without being important in the real world.
For example, suppose a new website increases the average time customers spend on the site by only 2 seconds. With a very large sample, this small difference might produce a very small p-value and therefore be statistically significant.
However, a 2-second increase may have little practical value to the business.
Statistical significance asks: “ Is the observed result unlikely to be explained by random sampling variation under the null hypothesis?”
Practical significance asks: “Is the size of the difference large enough to matter in the real world?”
Therefore, when interpreting a hypothesis test, we should consider both the statistical evidence and the size and practical importance of the effect.
Types of Statistical Tests
Once we understand hypothesis testing, the next step is to know which statistical test to use for a particular problem.
The choice depends on factors such as the type of data, number of groups, whether the observations are independent or paired, and whether the assumptions of a parametric test are reasonably satisfied.
Parametric vs. Non-parametric Tests
Parametric tests generally make assumptions about the population distribution and often involve parameters such as the mean and variance.
Examples include:
- t-tests
- Z-test
- ANOVA
- Pearson correlation
Non-parametric tests make fewer or different distributional assumptions and are often useful for ordinal data, non-normal data, or situations where parametric assumptions are not appropriate.
Examples include:
- Mann–Whitney U test
- Wilcoxon signed-rank test
- Kruskal–Wallis test
- Spearman correlation
Non-parametric does not mean “no assumptions.” These tests still have their own assumptions and requirements.
1. One-Sample t-Test
A one-sample t-test is used to determine whether the mean of a single sample differs from a specified population value.
For example, suppose a company claims that its employees spend an average of 8 hours per day working. We collect data from a sample of employees and test whether the sample provides evidence that the population mean differs from 8 hours.
Common assumptions:
- The outcome variable is numerical.
- Observations are independent.
- The population is approximately normal, especially for small samples.
- There should not be extreme outliers that strongly affect the mean.
2. Independent-Samples t-Test
An independent-samples t-test compares the means of two independent groups.
For example, we might compare the average test scores of students who used two different learning methods.
Common assumptions:
- The outcome variable is numerical.
- There are two independent groups.
- Observations within and between groups are independent.
- The data in each group is approximately normally distributed, particularly for smaller samples.
- The standard version assumes equal population variances; when this assumption is doubtful, Welch’s t-test is commonly used instead.
3. Paired t-Test
A paired t-test is used when two measurements are related or come from the same subjects.
For example, suppose we record the weight of 30 people before and after a fitness program. Since each person’s two measurements belong together, the observations are paired.
The test examines whether the average difference between the paired measurements is significantly different from zero.
Common assumptions:
- The data consists of paired observations.
- The pairs are independent of other pairs.
- The differences between the paired observations are approximately normally distributed.
- The differences should not contain extreme outliers.
4. Z-Test
A Z-test can be used to test a population mean or proportion when the conditions for a normal approximation are appropriate.
For a population mean, a Z-test is commonly associated with situations where the population standard deviation is known.
For example, a manufacturer may want to test whether the average weight of a product is 500 grams when the population standard deviation is known.
Common assumptions:
- Observations are independent.
- The variable is approximately normally distributed, or the sample is sufficiently large for the normal approximation to be reasonable.
- For the classic one-sample mean Z-test, the population standard deviation is known.
5. ANOVA Test
ANOVA (Analysis of Variance) is used to compare the means of three or more groups.
For example, a company may want to compare the average sales generated by employees using three different sales training programs.
ANOVA tests the overall null hypothesis that the group means are equal.
A significant ANOVA result tells us that at least one group mean differs, but it does not tell us which specific groups are different. Additional post-hoc tests can be used for that purpose.
Common assumptions:
- Observations are independent.
- The outcome variable is numerical.
- The data within each group is approximately normally distributed.
- Group variances are reasonably similar for the standard one-way ANOVA.
6. Chi-Square Test
The chi-square test is used with categorical data.
There are two common forms.
a. Chi-Square Test of Independence
It tests whether two categorical variables are associated.
For example, we might test whether gender and product preference are associated.
b. Chi-Square Goodness-of-Fit Test
It tests whether observed category frequencies differ from expected frequencies under a specified distribution.
For example, we might test whether customers choose four product categories equally often.
Common assumptions:
- Data consists of frequency counts.
- Observations are independent.
- Categories are mutually exclusive.
- Expected cell counts should generally be sufficiently large for the usual chi-square approximation.
7. Mann–Whitney U Test
The Mann–Whitney U test is a non-parametric alternative to the independent-samples t-test.
It is used to compare two independent groups, particularly when the outcome is ordinal or when the assumptions needed for a t-test are not appropriate.
For example, we might compare customer satisfaction ratings between two independent stores.
Common assumptions:
- Two independent groups.
- Observations are independent.
- The outcome is at least ordinal.
- For interpreting the test as a difference in distributions or locations, the shapes of the two group distributions should be reasonably similar.
8. Wilcoxon Signed-Rank Test
The Wilcoxon signed-rank test is a non-parametric alternative to the paired t-test.
It is used for paired or matched observations when the assumptions of the paired t-test are not suitable.
For example, we could compare customer satisfaction ratings before and after a service improvement.
Common assumptions:
- Data consists of paired observations.
- Pairs are independent of one another.
- The differences can be meaningfully ranked.
- The distribution of differences is approximately symmetric for the usual interpretation.
9. Kruskal–Wallis Test
The Kruskal–Wallis test is a non-parametric alternative to one-way ANOVA.
It is used to compare three or more independent groups when the assumptions of ANOVA are not appropriate.
For example, we could compare customer satisfaction ratings across three different branches.
Common assumptions:
- There are three or more independent groups.
- Observations are independent.
- The outcome is at least ordinal.
- For interpreting the result as a difference in distributions or locations, the group distributions should have reasonably similar shapes.
A significant result tells us that at least one group differs, but additional analysis is needed to identify which groups differ.
Pearson vs. Spearman Correlation Tests
Both tests examine the relationship between two variables, but they measure different types of association.
1. Pearson Correlation
Pearson correlation measures the strength and direction of a linear relationship between two numerical variables.
For example, we could examine the relationship between hours studied and exam scores.
Common assumptions:
- Variables are numerical.
- Observations are independent.
- The relationship is approximately linear.
- Extreme outliers should be investigated because they can strongly affect the correlation.
2. Spearman Correlation
Spearman correlation measures the strength and direction of a monotonic relationship using the ranks of the observations.
It can be useful for ordinal data or when the relationship is not linear but generally moves in one direction.
For example, we could examine whether higher customer satisfaction rankings are associated with higher product ratings.
Common assumptions:
- Observations are independent.
- Variables can be ranked.
- The relationship should be reasonably monotonic.
- Extreme observations can still influence the result.
Comparing Statistical Tests
The following table provides a quick way to decide which test may be appropriate.
Hypothesis Testing Errors
Hypothesis testing involves making decisions based on sample data. Because we are working with a sample rather than the entire population, there is always a possibility of making an incorrect decision.
Two important types of errors are Type I error and Type II error.
1. Type I Error
A Type I error occurs when we reject the null hypothesis even though it is actually true.
In simple terms, we conclude that there is an effect or difference when there actually isn’t one.
For example, suppose a company tests whether a new training program improves employee performance.
- H_0: The training program does not improve performance.
- H_a: The training program improves performance.
If we reject H_0 and conclude that the training program works when it actually does not, we have made a Type I error.
The probability of making a Type I error is represented by the significance level α.
For example, if α = 0.05, the test is performed with a 5% significance level.
2. Type II Error
A Type II error occurs when we fail to reject the null hypothesis even though the alternative hypothesis is actually true.
In simple terms, we fail to detect an effect or difference that actually exists.
Using the same example, suppose the training program really does improve employee performance, but our sample does not provide enough evidence to reject H_0.
That would be a Type II error.
The probability of a Type II error is represented by: β
An important point is that failing to reject H_0 does not prove that H_0 is true.
It only means that the available sample evidence was not strong enough to reject it.
Statistical Power
Statistical power is the probability that a hypothesis test correctly detects an effect when that effect actually exists.
It is defined as: Power = 1 − β
For example, if: β = 0.20
then: Power = 1 − 0.20 = 0.80
So the test has 80% power to detect the specified effect under the assumed conditions.
Higher statistical power means a lower probability of making a Type II error.
Relationship Between Significance Level, Power, Sample Size, and Effect Size
Several factors affect the ability of a statistical test to detect an effect.
1. Significance Level (α)
Increasing α makes it easier to reject the null hypothesis. This can increase power, but it also increases the probability of a Type I error.
2. Sample Size
A larger sample generally provides more information and reduces sampling variability. This usually increases the power of a test and makes it easier to detect smaller effects.
3. Effect Size
The effect size describes how large or meaningful the difference or relationship is.
Large effects are generally easier to detect than very small effects when the sample size and other conditions are the same.
4. Power
Power increases when the test has a greater ability to distinguish a real effect from random variation.
In general:
Larger sample → higher power
Larger effect → higher power
Higher α → higher power, but also higher Type I error risk
Real-World Example of Hypothesis Testing Errors
Suppose a company introduces a new website design and wants to know whether it increases the conversion rate.
The hypotheses are:
H_0 : The new design does not increase conversion rate
H_a : The new design increases conversion rate
There are four possible situations:
For example, if the new design actually has no effect but the test reports a statistically significant improvement, that is a Type I error.
If the new design actually improves conversion but the test does not find sufficient evidence to reject H0H_0, that is a Type II error.
This is why hypothesis testing results should be interpreted carefully. Rejecting the null hypothesis provides evidence against it, while failing to reject the null hypothesis simply means that the evidence was insufficient to reject it. It does not establish that the null hypothesis is true.
Conclusion
Statistics is not just a collection of formulas. It provides a way to understand data, measure uncertainty, and make better-informed decisions.
Key Takeaways from Descriptive Statistics -
Descriptive statistics helps us understand and summarize the data we already have. Measures such as mean, median, mode, variance, standard deviation, percentiles, correlation, and distributions help us identify the center, spread, relationships, and overall behavior of data.
Key Takeaways from Inferential Statistics -
Inferential statistics allows us to go beyond the observed sample and make estimates or test claims about a larger population.
Concepts such as probability, sampling distributions, confidence intervals, hypothesis testing, statistical tests, errors, and statistical power help us understand uncertainty and make conclusions from sample data.
How Descriptive and Inferential Statistics Work Together -
Descriptive and inferential statistics are not separate worlds. They usually work together.
We can think of the process as:
Collect Data → Describe the Data → Understand Uncertainty → Make Inferences → Make Decisions
We first use descriptive statistics to understand our data. Then, when we want to generalize findings from a sample to a population, inferential statistics helps us quantify uncertainty and evaluate our conclusions.
