What is it?

T-test is popular and basic analysis technique to compare two groups. It is one technique of Group Mean Comparison.

Why educational researchers has used it?

T-test (and Group Mean Comparison) is suitable for the experimental logic. The comparison of intervention and control groups is the fundamental rationale for testing the effect of educational intervention. For example, suppose a teacher wants to validate the impact of a new practice. In that case, they can randomly assign students as much as possible into two groups, apply previous practice and new practice differently (other components are the same), and then measure a characteristic(s) of interest. T-test is used to test whether this effect can be generalized or not.

What should I know?

• The concept of central tendency and variability (descriptive statistics)
• The concept of sampling distribution and sampling error
• Central Limit Theorem (but just some principles)
• The experimental logic (especially randomization and control/intervention)
• Hypothesis testing logic (Type 1 error and power)

Where can I learn?

• EPSY 530 Statistics 1

A more articulation

The logic of inference

We want to know the parameter of the population by using a limited number of sample data. Testing the new literacy teaching practice with all high school students is impossible. Thus, we rely on some mathematical and statistical principles that help us estimate the population’s characteristics by using that of the sample. Usually, these principles are related to the distribution of the target characteristics (mean, median, variance, etc.) or data. The difference between T-test and Z-test for mean comparison is rooted in the difference of distribution they rely on for inferential task.

Does ‘statistically significant’ mean that the intervention (new teaching in the example) is much better?

No. Statistical significance only refers to generalizability. By achieving statistical significance, you can tentatively argue that your data didn’t support the null hypothesis (there is no difference between the two teaching practices). You can accept the alternative hypothesis (there is a difference), but it says nothing about the magnitude of the difference.

Please be careful of this point. The indicator for measuring magnitude is effect size. You can start exploring effect size by searching Cohen’s D.

In a very very strict sense, ‘accept the alternative hypothesis’ is the wrong expression.

It would be better to say, ‘the result did not support the null hypothesis.’ This double-negation language seems weird, but this conservative language is what APA follows. The logic of hypothesis testing may be the first challenge for those who equate statistics to mathematics. Please be careful of the truth claim. As a researcher, we need epistemological humbleness.

Written by YangHyun Kim (ykim39@albany.edu)