Hostname: page-component-6bb9c88b65-vpjdr Total loading time: 0 Render date: 2025-07-26T03:53:46.022Z Has data issue: false hasContentIssue false
  • English
  • Français

Laboratory experiments can pre-design to address power and selection issues

Published online by Cambridge University Press:  17 January 2025

Weili Ding Open the ORCID record for Weili Ding [Opens in a new window]
Weili Ding*
Affiliation:
Queen’s University, Kingston, Canada
*
e-mail: dingw@queensu.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

In this paper, motivated by aspects of preregistration plans we discuss issues that we believe have important implications for how experiments are designed. To make possible valid inferences about the effects of a treatment in question, we first illustrate how economic theories can help allocate subjects across treatments in a manner that boosts statistical power. Using data from two laboratory experiments where subject behavior deviated sharply from theory, we show that the ex-post subject allocation to maximize statistical power is closer to these ex-ante calculations relative to traditional designs that balances the number of subjects across treatments. Finally, we call for increased attention to (i) the appropriate levels of the type I and type II errors for power calculations, and (ii) how experimenters consider balance in part by properly handling over-subscription to sessions.

Information

Type
Original Paper
Information
Journal of the Economic Science Association , Volume 6 , Issue 2: Special Issue: Statistical Issues for Experimental Economists , December 2020 , pp. 125 - 138
Copyright
Copyright © Economic Science Association 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

I would like to thank one anonymous reviewer, the guest editor John Ham and Steven Lehrer for many helpful comments and suggestions that have substantially improved the manuscript. Steven Lehrer also generously provided the experimental data analyzed in the study. I wish to thank SSHRC for research support. I am responsible for all errors.

References

Abreu, D., Gul, F. (2000). Bargaining and reputation. Econometrica, 68(1), 85–117. 10.1111/1468-0262.00094Google Scholar
Baron, D. P., Ferejohn, J. A. (1989). Bargaining in legislatures. The American Political Science Review, 83(4), 1181–1206. 10.2307/1961664Google Scholar
Bush, S. A. (2015). Sample size determination for logistic regression: a simulation study. Communications in Statistics Simulation and Computation, 44(2), 360–373.Google Scholar
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T. et al., (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. 10.1126/science.aaf0918Google ScholarPubMed
Casari, M., Ham, J. C., Kagel, J. H. (2007). Selection bias, demographic effects, and ability effects in common value auction experiments. American Economic Review, 97(4), 1278–1304. 10.1257/aer.97.4.1278Google Scholar
Charness, G., Gneezy, U., Kuhn, M. A. (2012). Experimental methods: Between-subject and within subject design. Journal of Economic Behavior & Organization, 81(1), 1–8. 10.1016/j.jebo.2011.08.009Google Scholar
Cochran, W. G. (1977). Sampling Techniques, 3New York: Wiley.Google Scholar
Coffman, L. C., Niederle, M. (2015). Pre-analysis plans have limited upside, especially where replications are feasible. Journal of Economic Perspectives, 29(3), 81–98. 10.1257/jep.29.3.81Google Scholar
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2Hillsdale: Lawrence Erlbaum Associates.Google Scholar
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019). The dozen things experimental economists should do (more of). Southern Economic Journal, 86(2), 371–432.Google Scholar
Ding, W., Lehrer, S. F. (2011). Experimental estimates of the impacts of class size on test scores: robustness and heterogeneity. Education Economics, 19(3), 229–252. 10.1080/09645292.2011.589142Google Scholar
Ding, W., Lehrer, S. F. (2010). Estimating treatment effects from contaminated multiperiod education experiments: the dynamic impacts of class size reductions. Review of Economics and Statistics, 92(1), 31–42. 10.1162/rest.2009.11453Google Scholar
Duflo, E., Glennerster, R., Kremer, M., & Schultz, T., Strauss, J. (2007). Using randomization in development economics research: A toolkit Handbook of Development Economics, Amsterdam: Elsevier 3895–3962.Google Scholar
Embrey, M., Fréchette, G. R., Lehrer, S. F. (2015). Bargaining and reputation: An experiment on bargaining in the presence of behavioural types. Review of Economic Studies, 82(2), 608–631. 10.1093/restud/rdu029Google Scholar
Engle, R. F., & Griliches, Z., Intriligator, M. D. (1984). Wald, likelihood ratio, and Lagrange multiplier statistics in econometrics Handbook of Econometrics, Amsterdam: North Holland 776–828.Google Scholar
Fisher, R. A. (1925). Statistical Methods for Research Workers, 1Oliver and Boyd: Edinburgh.Google Scholar
Ford, I., Norrie, J., Ahmadi, S. (1995). Model inconsistency, illustrated by the Cox proportional hazards model. Statistics in Medicine, 14(8), 735–746. 10.1002/sim.4780140804Google ScholarPubMed
Fréchette, G. R., Kagel, J. H., Lehrer, S. F. (2003). Bargaining in legislatures: an experimental investigation of open versus closed amendment rules. American Political Science Review, 97(2), 221–232. 10.1017/S0003055403000637Google Scholar
Ham, J. C., Kagel, J. H., Lehrer, S. F. (2005). Randomization, endogeneity and laboratory experiments: The role of cash balances in private value auctions. Journal of Econometrics, 125(1–2), 175–205. 10.1016/j.jeconom.2004.04.008Google Scholar
Ham, J. C., LaLonde, R. (1996). The effect of sample selection and initial conditions in duration models: Evidence from experimental data on training. Econometrica, 64(1), 175–205. 10.2307/2171928Google Scholar
Heonig, J. M., Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician, 55(1), 19–24. 10.1198/000313001300339897Google Scholar
Hernandez, A. V., Steyerberg, E. W., Habbema, D. F. (2004). Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. Journal of Clinical Epidemiology, 57(5), 454–460. 10.1016/j.jclinepi.2003.09.014Google ScholarPubMed
Kagel, J. H., Harstad, R. M., Levin, D. (1987). Information impact and allocation rules in auctions with affiliated private values: A laboratory study. Econometrica, 55(4), 1275–1304. 10.2307/1913557Google Scholar
List, J. A., Sadoff, S., Wagner, M. (2011). So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design. Experimental Economics, 14(4), 439–457. 10.1007/s10683-011-9275-7Google Scholar
List, J. A., Shaikh, A. M., Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4), 773–793. 10.1007/s10683-018-09597-5Google Scholar
Maniadis, Z., Tufano, F., List, J. A. (2017). To replicate or not to replicate? Exploring reproducibility in economics through the lens of a model and a pilot study. The Economic Journal, 127(605), F209–F235. 10.1111/ecoj.12527Google Scholar
Manski, C. (2019). Treatment choice with trial data: Statistical decision theory should supplant hypothesis testing. The American Statistician, 73(s1), 296–304. 10.1080/00031305.2018.1513377Google Scholar
Manski, C., Tetenov, A. (2016). Sufficient trial size to inform clinical practice. Proceedings of the National Academy of Sciences, 113(38), 10518–10523. 10.1073/pnas.1612174113Google ScholarPubMed
Nikiforakis, N., Slonim, R. (2015). Editors preface: Statistics, replications and null results. Journal of the Economic Science Association, 1(2), 127–131. 10.1007/s40881-015-0018-yGoogle Scholar
Palmer, M. W. (1993). Potential biases in site and species selection for ecological monitoring. Environmental Monitoring and Assessment, 26, 277–282. 10.1007/BF00547504Google ScholarPubMed
Robinson, L. D., Jewell, N. P. (1991). Some surprising results about covariate adjustment in logistic regression models. International Statistical Review, 59(2), 227–240. 10.2307/1403444Google Scholar
Rodrik, D. (2015). Economics Rules: The Rights and Wrongs of The Dismal Science, New York: W.W. Norton.Google Scholar
Roth, A. E. (1986). Laboratory experimentation in economics. Economics and Philosophy, 2, 245–273. 10.1017/S1478061500002656Google Scholar
Shieh, G. (2000). On power and sample size calculations for likelihood ratio tests in generalized linear models. Biometrics, 56(4), 1192–1196. 10.1111/j.0006-341X.2000.01192.xGoogle ScholarPubMed
Slonim, R., Wang, C., Garbarino, E., Merrett, D. (2013). Opting-in: Participation bias in economic experiments. Journal of Economic Behavior and Organization, 90(1), 43–70. 10.1016/j.jebo.2013.03.013Google Scholar