How does sample size affect effect size

In addition, it seems impossible to compare effects from studies with between-subjects designs and within-subject designs, particularly when it comes to large effects. Figure 2. For Cohen , p. Note that this analysis could only be done for the studies published without pre-registration because studies with pre-registration were too few to be sensibly divided into sub-categories.

Figure 3. The bars contain all effects that were extracted as or could be transformed into a correlation coefficient r. The vertical line is the grand median. The largest effects come from disciplines such as experimental and biological psychology where the use of more reliable instruments and devices is common.

Disciplines such as social and developmental psychology provide markedly smaller effects. Note that, for instance, there is not even an overlap of the confidence intervals of social and biological psychology. This simply means that in terms of effect sizes, we are talking about completely different universes when we talk about psychological research as a whole.

The differences between the sub-disciplines shown in Figure 3 largely match the differences between the results of the studies discussed in the Introduction. By contrast, Richard et al. Effect sizes were smaller the larger the samples see Figures 4 , 5. One obvious explanation for these strong correlations is the publication bias, since effects from large samples have enough statistical power to become significant regardless of their magnitude.

However, a look at Figure 5 reveals that with studies published with pre-registration, hence potentially preventing publication bias, the correlation is indeed smaller but still far from zero. This general correlation between sample size and effect size due to statistical power might also have led to a learning effect: in research areas with larger effects, scientists may have learned that small samples are enough while in research areas with smaller effects, they know that larger samples are needed.

Moreover, studies on social processes or individual differences can be done online with large samples; developmental studies can be done in schools, also providing large samples. By contrast, experimental studies or studies requiring physiological measurement devices are usually done with fewer participants but reveal larger effects.

However, when calculating the correlation between sample size and effect size separately for the nine sub-disciplines, it is still very large in most cases ranging from 0. The relationship between larger effects and the use of more reliable measurement devices might of course also be there within the sub-disciplines but this explanation needs more empirical evidence.

Figure 4. Relationship Loess curve 1 between sample size and effect size r for studies published without pre-registration. Figure 5. Relationship Loess curve between sample size and effect size r for studies published with pre-registration.

Is the year of publication associated with the effect size? For instance, the call for replication studies in recent years together with the decline effect e. As Figure 6 shows, however, there is no correlation between year of publication and size of the effects reported only done for studies published without pre-registration since studies with pre-registration started no earlier than Thus, effect sizes appear to be relatively stable over the decades so that, in principle, nothing speaks against providing fixed guidelines for their interpretation.

Figure 6. Relationship Loess curve between year of publication and effect size r for studies published without pre-registration.

When is an effect small or large? The present results demonstrate that this is not so easy to answer. Hence, it should make sense to look at the distribution of empirical effects that have been published in the past years or so. We have called this the comparison approach. Yet, as shown, this does not seem to be a practicable solution because most published effects are seriously inflated because of the potential biases in analyzing, reporting, and publishing empirical results.

This is what we can verify by the present analysis: The median effect of studies published without pre-registration i. Hence, if we consider the effect size estimates from replication studies or studies published with pre-registration to represent the true population effects we notice that, overall, the published effects are about twice as large.

One reason might be that within-subject designs generally have higher statistical power so that the effect of potential biases might be smaller. The potential biases also seem to have affected the shape of the distribution of the effects: While the distribution of effects published without pre-registration is fairly symmetrical around its median, the distribution of effects published with pre-registration is markedly skewed and contains many more values close to zero.

This is what one would expect given that in confirmatory i. Thus, at least currently, the comparison approach is limited to the interpretation of an effect in the context of published and potentially biased effects only but it fails to provide a comparison with real population effects.

In other words, one can compare the effect of a study with previous effects in the respective area of research but must keep in mind that these past publications provide a biased picture with effects much larger than what holds true for the population. The hope is, of course, that the near future will bring many more studies that adhere to a strict pre-registration procedure in order to prevent the potential biases and other problems.

Once there is a reliable basis of such studies in a couple of years, the comparison approach can develop its full potential—just as intended by Cohen years ago. For the time being, however, it might be wiser to interpret the size of an effect by looking at its unstandardized version and its real-world meaningfulness see Cohen, ; Kirk, ; Baguley, An alternative way to interpret the size of an effect—besides comparing it with effects from the past—is to apply conventional benchmarks for small, medium, and large effects.

Cohen provided his now well-known conventions very hesitantly as he was aware that global benchmarks might not be applicable to all fields of behavioral sciences and there is the risk of overuse. Our analysis of the distributions of effects within psychological sub-disciplines revealed that Cohen was much more right than he may have thought: Effects differ considerably, partially in such a way that their confidence intervals do not even overlap.

Whether publication bias has an influence on the size of these differences is unclear; more pre-registered studies are needed to reliably compare their effects between sub-disciplines. Nonetheless, these differences clearly speak against the use of general benchmarks.

Instead, benchmarks should, if at all, be derived for homogeneous categories of psychological sub-disciplines. Again, the hope is that the future will bring many more pre-registered studies in all sub-disciplines to accomplish this task. As Kelley and Preacher , p. What also speaks against the use of general benchmarks is the difference between effects from within-subject versus between-subjects designs. Due to the omission of between-subjects variance, within-subject designs reveal considerably larger effects see also the review of Rubio-Aparicio et al.

We cannot rule out that there might be a self-selection effect in researchers pre-registering their studies. Pre-registered studies are more common in more highly ranked journals, which might provide a biased selection of well-established and mostly experimental research paradigms we indeed found that the share of experimental designs is much larger with pre-registered studies. This might cause published effects to even be the larger ones. In contrast, one might suspect that researchers pre-register a study when they expect their studied effects to be small, in order to ensure publication in any case.

This might cause published effects to be the smaller ones. As said, we need more pre-registered studies in the future to say something definite about the representativeness of pre-registered studies.

With regard to the different kinds of pre-registration, we also found that there is a difference between studies that were explicitly registered reports and studies that were not.

That is, we cannot rule out that published pre-registered studies that are not registered reports are still affected by publication bias at least to a certain degree.

Any categorization of psychological sub-disciplines is vulnerable. We decided to use the SSCI since it is a very prominent index and provides a rather fine-grained categorization of sub-disciplines. In any case, we showed that differences in the effect sizes between the sub-disciplines are considerable. As explained, from each study, we analyzed the first main effect that clearly referred to the key research question of an article. For articles reporting a series of several studies this procedure might cause a certain bias if the first effect reported happened to be particularly small or particularly large.

To our knowledge, however, there is no evidence that this should be the case although this might be a worthwhile research question on its own. As we have argued throughout this article, biases in analyzing, reporting, and publishing empirical data i. Having said this, we definitely recommend addressing the question of how pre-registered or newer studies might differ from conventional or older studies in future research. We can now draw conclusions regarding the two main focuses of effect sizes: answering research questions and calculating statistical power.

We have shown that neither the comparison approach nor the conventions approach can be applied to interpret the meaningfulness of an effect without running into severe problems. Comparisons are hard to make when there is no reliable empirical basis of real population effects; and global conventions are useless when differences between sub-disciplines and between study designs are so dramatic. One pragmatic solution for the time being is something that Cohen himself had suggested: express effects in an unstandardized form and interpret their practical meaning in terms of psychological phenomena see also Baguley, —thereby accepting the problem that unstandardized effects are hard to compare across different scales and instruments.

We also expressed our hope for the future that many more pre-registered studies will be published, providing a more reliable picture of the effects in the population. We will then be able to really exploit the comparison approach.

Moreover, separately for sub-disciplines and for between-subjects versus within-subject studies, new benchmarks could then be derived. Our finding that effects in psychological research are probably much smaller than it appears from past publications has an advantageous and a disadvantageous implication.

On the downside, smaller effect sizes mean that the under-powering of studies in psychology is even more dramatic than recently discussed e. Thus, our findings once more underline the necessity of power calculations in psychological research in order to produce reliable knowledge. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Google Scholar. Appelbaum, M.

Journal article reporting standards for quantitative research in psychology: the APA publications and communications board task force report. Baguley, T. Standardized or simple effect size: what should be reported? Bakker, M. The rules of the game called psychological science.

Brandt, M. The replication recipe: what makes for a convincing replication? Cohen, J. The statistical power of abnormal social psychological research: a review. Statistical Power Analysis for the Behavioral Sciences. Things i have learned so far. A power primer. Cooper, H. Expected effect sizes: estimates for statistical power analysis in social psychology.

Cumming, G. The new statistics: why and how. New York, NY: Routledge. Duval, S. Trim and fill: a simple funnel-plot—based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, — Ellis, P. Cambridge: Cambridge University Press. Fanelli, D. Negative results are disappearing from most disciplines and countries. Published on December 22, by Pritha Bhandari. Revised on February 18, Effect size tells you how meaningful the relationship between variables or the difference between groups is.

It indicates the practical significance of a research outcome. A large effect size means that a research finding has practical significance, while a small effect size indicates limited practical applications. Table of contents Why does effect size matter? How do you calculate effect size? How do you know if an effect size is small or large? When should you calculate effect size? Frequently asked questions about effect size. While statistical significance shows that an effect exists in a study, practical significance shows that the effect is large enough to be meaningful in the real world.

Statistical significance is denoted by p- values , whereas practical significance is represented by effect sizes. Increasing the sample size always makes it more likely to find a statistically significant effect, no matter how small the effect truly is in the real world. In contrast, effect sizes are independent of the sample size. Only the data is used to calculate effect sizes. The APA guidelines require reporting of effect sizes and confidence intervals wherever possible.

However, a difference of only 0. Adding a measure of practical significance would show how promising this new intervention is relative to existing interventions. There are dozens of measures for effect sizes.

Similarly, the larger the sample size the more information we have and so our uncertainty reduces. Suppose that we want to estimate the proportion of adults who own a smartphone in the UK.

We could take a sample of people and ask them. The larger the sample size the more information we have and so our uncertainty reduces. We can also construct an interval around this point estimate to express our uncertainty in it, i. In other words, if we were to collect different samples from the population the true proportion would fall within this interval approximately 95 out of times.

Suppose we ask another people and find that, overall, out of the people own a smartphone. However, our confidence interval for the estimate has now narrowed considerably to Because we have more data and therefore more information, our estimate is more precise. As our sample size increases, the confidence in our estimate increases, our uncertainty decreases and we have greater precision. This is clearly demonstrated by the narrowing of the confidence intervals in the figure above.

If we took this to the limit and sampled our whole population of interest then we would obtain the true value that we are trying to estimate — the actual proportion of adults who own a smartphone in the UK and we would have no uncertainty in our estimate. Increasing our sample size can also give us greater power to detect differences.

Suppose in the example above that we were also interested in whether there is a difference in the proportion of men and women who own a smartphone. We can estimate the sample proportions for men and women separately and then calculate the difference. When we sampled people originally, suppose that these were made up of 50 men and 50 women, 25 and 34 of whom own a smartphone, respectively. The difference between these two proportions is known as the observed effect size.

Is this observed effect significant, given such a small sample from the population, or might the proportions for men and women be the same and the observed effect due merely to chance? We find that there is insufficient evidence to establish a difference between men and women and the result is not considered statistically significant.