Essential Precautions for Using the Chi-Square Test Effectively

The Chi-square test is one of the most widely used statistical tools in psychology research. It helps determine whether two categorical variables are associated – for example, whether there is a link between therapy type and treatment outcome, or between gender and a specific behavioral pattern. But like any statistical method, it comes with conditions that must be met for the results to be valid. Ignoring these conditions doesn’t just produce questionable numbers – it can lead to conclusions that are entirely wrong. Here are the key precautions every researcher should understand before running a Chi-square test.

What the Chi-square test actually does 🔗

Before examining the precautions, it helps to be clear on what the test measures. According to a review published in PMC, the Chi-square test is a non-parametric statistic that can tell you not only whether a significant difference exists, but also which specific categories account for those differences. That makes it unusually informative. However, it only works with categorical (nominal or ordinal) data – it cannot be applied to continuous variables like height or reaction time. And it requires several conditions to be satisfied before the results can be trusted.

Precaution 1: Sample size must be large enough 🔗

This is the most fundamental precaution. The Chi-square test uses a mathematical approximation – it assumes your data is large enough for the chi-squared distribution to accurately describe the test statistic. When samples are very small, the test may lack power to detect meaningful associations, while very large samples can make even trivial differences appear significant. Both directions are problematic.

The expected frequency rule 🔗

The most widely cited requirement concerns the expected frequencies in each cell of your contingency table – not the observed counts, but what you would mathematically expect if there were no relationship between the variables. The expected value in each cell should be 5 or more in at least 80% of cells, and no single cell should have an expected frequency below 1. A practical way to check this: the total sample size should be at least equal to the number of cells in the table multiplied by 5.

A standard rule of thumb credited to Cochran advises against using the Chi-square test when more than 20% of cells have expected frequencies below 5, or when any cell has an expected frequency below 1. Violating this condition means the approximation that underlies the test breaks down, and the resulting p-value can no longer be trusted.

What to do when expected frequencies are too low 🔗

If your expected cell frequencies are too small, you have a few options. The first is to increase your sample size. The second is to combine adjacent categories to raise cell counts – for example, merging two sparsely populated response groups into one, provided doing so makes conceptual sense. The third option is to switch to a different test entirely. Fisher’s exact test always gives an exact p-value regardless of sample size and is the preferred alternative when working with small samples in a 2×2 table.

Historically, Yates’ correction for continuity was another solution – it adjusts the chi-square formula by subtracting 0.5 from the difference between each observed and expected value, reducing the chi-square statistic and making the test more conservative. This correction was designed specifically for 2×2 tables where at least one cell has an expected count below 5. However, its use has declined because it tends to overcorrect, producing p-values that are too conservative and increasing the risk of a Type II error (failing to detect a real effect). The current practical recommendation is: if all expected frequencies are 5 or above, use the standard Chi-square test; if any expected frequency falls below 5, skip Yates’ correction and go directly to Fisher’s exact test.

Precaution 2: Observations must be independent 🔗

The Chi-square test assumes that each data point comes from a different, unrelated participant. Violating this independence assumption can lead to seriously misleading results. Independence means the response of one participant must not influence the response of another.

A common way this assumption gets violated is through repeated-measures designs – where the same participants are measured at two different time points, such as before and after an intervention. If you apply a Chi-square test to pre-test and post-test data from the same group of people, the responses are not independent. When observations are paired in this way, McNemar’s Test is the appropriate alternative. Similarly, if two participants are closely related – such as siblings or partners in a study – their responses may not be truly independent, and the standard Chi-square test may give misleading results.

Ensuring independence usually requires random sampling or careful study design at the data collection stage. It cannot be corrected statistically after the fact.

Precaution 3: Data must be in raw frequencies, not percentages 🔗

This precaution is easy to overlook, especially when data is first summarized and then entered into a contingency table. The Chi-square formula compares observed counts to expected counts. The values in each cell must be raw frequencies – actual counts of cases – not percentages, proportions, or any other transformed form of the data. Entering percentages instead of frequencies will produce an artificially small chi-square statistic and an incorrect p-value.

Additionally, each participant must contribute data to exactly one cell. A person cannot be counted in two different categories simultaneously. If your categories are not mutually exclusive, the test’s logic breaks down entirely.

Precaution 4: Statistical significance is not the same as practical significance 🔗

One of the most important – and most commonly ignored – precautions involves interpreting a significant result correctly. The Chi-square test is very sensitive to sample size; with a large enough sample, even trivially small differences between groups can appear statistically significant. A p-value below 0.05 tells you that the pattern in your data is unlikely to be due to chance. It does not tell you how strong or meaningful the relationship actually is.

This is why effect size measures must accompany any Chi-square result. The Chi-square test should always be paired with an appropriate measure of effect size to show the magnitude of the association. The two most common options are:

• Phi (φ): Used for 2×2 contingency tables. It ranges from 0 (no association) to 1 (perfect association).
• Cramer’s V: Used for tables larger than 2×2. It is calculated from the chi-squared statistic, the sample size, and the smaller of the number of rows or columns minus one, and also ranges from 0 to 1.

A result can be statistically significant without being practically significant – effect size tells you not just whether an effect exists, but how large it is in real terms. Reporting a p-value without an effect size gives an incomplete picture, and in psychology research, that gap can lead to overinterpreting findings that are, in practice, quite modest.

Precaution 5: The Chi-square test cannot establish causality 🔗

This point is worth stating clearly. The Chi-square test can only tell you whether two variables are related – it cannot tell you whether one variable caused the other. A significant Chi-square result means the two categorical variables are associated in your sample. It does not mean that one variable produced or explains the other. Establishing causality requires a different research design – typically an experimental one with controlled conditions and random assignment.

In psychology, this distinction matters enormously. Finding a significant association between, say, social media use and anxiety symptoms does not prove that social media causes anxiety. It simply indicates that a relationship exists and warrants further investigation.

Precaution 6: Avoid too many cells 🔗

While the Chi-square test has no hard limit on the number of categories, creating a very large contingency table introduces practical problems. When a contingency table has more than 20 cells, meeting the expected frequency requirement becomes much harder, and interpreting the results becomes significantly more complex. With many sparse cells, the test may technically pass significance thresholds while being driven by a small corner of the data rather than a genuine overall pattern. Keeping category schemes parsimonious and meaningful reduces this risk.

Putting the precautions together 🔗

Each of these precautions addresses a specific way the Chi-square test can go wrong. Small expected frequencies undermine the mathematical approximation the test relies on. Dependent observations violate its structural logic. Entering percentages instead of counts produces incorrect statistics. Ignoring effect size leads to inflated conclusions. And mistaking association for causation is a conceptual error with real consequences for how findings get applied.

Before running a Chi-square test, it is good practice to build a checklist: Is the data categorical? Are the observations independent? Are expected frequencies adequate in each cell? Are the categories mutually exclusive, and is the data in raw counts? After running the test: Is the result accompanied by an effect size measure? Is the interpretation careful about what the test can and cannot conclude? Working through these questions systematically protects against the most common sources of error in Chi-square analysis.

What do you think? If a study reports a statistically significant Chi-square result but does not include an effect size measure, how much confidence would you place in its conclusions? And when a researcher finds that several cells in their contingency table have expected frequencies below 5, what factors should guide the decision between combining categories and switching to Fisher’s exact test?

References

1. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900058/
2. https://en.wikipedia.org/wiki/Chi-squared_test
3. https://www.quality-control-plan.com/StatGuide/gf-dist_ass_viol.htm
4. https://www.graphpad.com/support/faq/which-test-should-i-use-to-determine-the-statistical-significance-of-a-2×2-contingency-table-chi-square-fishers-exact-test-something-else/
5. https://en.wikipedia.org/wiki/Yates's_correction_for_continuity
6. https://statmate.org/blog/chi-square-vs-fisher-exact
7. https://libguides.library.kent.edu/spss/chisquare
8. https://spss-tutor.com/blogs/top-assumptions-for-running-a-chi-square-test.php
9. https://soc.utah.edu/sociology3112/chi-square.php
10. https://pmc.ncbi.nlm.nih.gov/articles/PMC5426219/
11. https://en.wikipedia.org/wiki/Cram%C3%A9r's_V
12. https://library.soton.ac.uk/effect-size