relationship between z value and replication rate

[Yefeng0920]

Hey @DominikVogel @FBartos , I am addicted to your method. Just curious about whether there is a way to construct the relationship between z value and replication rate. I am meant to make a plot with x-axis as the z value and the y-axis as the replication rate estimate. My main point is that in the case of non-normal distributions, it is not that meaningful to calculate the average power or the so-called expected replication rate. Rather, we should visualize the relationship between z value and the replication rate estimate

[FBartos]

I think Ulrich Schimmack did some work along these lines, I will let him point him here.

[DominikVogel]

Sorry, I have no merits in developing the method or the package. All fame should go to @FBartos, Ulrich Schimmack, and others.

[Yefeng0920]

@FBartos Thanks for bringing Ulrich Schimmack and would like to look at his solution. Another relevant question I have is that the replication rate derived from the z-curve seems to be an observational level estimate that does not account for sampling error variance. Because z = mu_observation / sd, while mu_observation is often overestimated. And the power calculated using this way is a kind of post-hoc power or observed power.

[FBartos]

I'm not sure I followed your question sampling error question correctly. The common problem with post-hoc power is a) selection on statistical significance and b) the high uncertainty of power of a single study.
Z-curve handles a) by modeling only statistically significant z-scores and accounting for selection on significance via truncated likelihood and b) by aggregating across many test statistics. Both z-curve papers explain this in more detail.

Brunner, J., & Schimmack, U. (2020). Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology, 4.
Bartoš, F., & Schimmack, U. (2022). Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology, 6.