Hostname: page-component-54dcc4c588-xh45t Total loading time: 0 Render date: 2025-10-06T16:11:19.928Z Has data issue: false hasContentIssue false
  • English
  • Français

Making replication mainstream

Published online by Cambridge University Press:  25 October 2017

Rolf A. Zwaan ,
Alexander Etz ,
Richard E. Lucas  and
M. Brent Donnellan
Rolf A. Zwaan
Affiliation:
Department of Psychology, Education, and Child Sciences, Erasmus University Rotterdam, 3000 DR Rotterdam, The Netherlands. zwaan@essb.eur.nlhttps://www.eur.nl/essb/people/rolf-zwaan
Alexander Etz
Affiliation:
Department of Cognitive Sciences, University of California, Irvine, CA 92697-5100. etz.alexander@gmail.comhttps://alexanderetz.com/
Richard E. Lucas
Affiliation:
Department of Psychology, Michigan State University, East Lansing, MI 48824. lucasri@msu.eduhttps://www.msu.edu/user/lucasri/
M. Brent Donnellan
Affiliation:
Department of Psychology, Texas A&M University, College Station, TX 77843. donnel59@msu.eduhttps://psychology.msu.edu/people/faculty/donnel59
Get access Rights & Permissions [Opens in a new window]

Abstract

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results – most notably in psychology, but in many other disciplines as well – cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy, and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

Keywords

psychological researchreplicationreproducibilityresearch programs

Information

Type
Target Article
Information
Behavioral and Brain Sciences , Volume 41 , 2018 , e120
Copyright
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

1.

Current address: Department of Psychology, Psychology Building, 316 Physics Road, Room 249A, Michigan State University, East Lansing, MI 48824.

References

Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., Buswell, K., Carlson, C., Carlson, M., Chu, S., Cislak, A., Colarusso, M., Colloff, M. F., Dellapaolera, K. S., Delvenne, J.-F., Di Domenico, A., Drummond, A., Echterhoff, G., Edlund, J. E., Eggleston, C. M., Fairfield, B., Franco, G., Gabbert, F., Gamblin, B. W., Garry, M., Gentry, R., Gilbert, E. A., Greenberg, D. L., Halberstadt, J., Hall, L., Hancock, P. J. B., Hirsch, D., Holt, G., Jackson, J. C., Jong, J., Kehn, A., Koch, C., Kopietz, R., Körner, U., Kunar, M. A., Lai, C. K., Langton, S. R. H., Leite, F. P., Mammarella, N., Marsh, J. E., McConnaughy, K. A., McCoy, S., McIntyre, A. H., Meissner, C. A., Michael, R. B., Mitchell, A. A., Mugayar-Baldocchi, M., Musselman, R., Ng, C., Nichols, A. L., Nunez, N. L., Palmer, M. A., Pappagianopoulos, J. E., Petro, M. S., Poirier, C. R., Portch, E., Rainsford, M., Rancourt, A., Romig, C., Rubínová, E., Sanson, M., Satchell, L., Sauer, J. D., Schweitzer, K., Shaheed, J., Skelton, F., Sullivan, G. A., Susa, K. J., Swanner, J. K., Thompson, W. B., Todaro, R., Ulatowska, J., Valentine, T., Verkoijen, P. P. J. L., Vranka, M., Wade, K. A., Was, C. A., Weatherford, D., Wiseman, K., Zaksaite, T., Zuj, D. V. & Zwaan, R. A. (2014) Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science 9:556–78.CrossRefGoogle ScholarPubMed
Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., Della Penna, N., Estel, V., Fedor, A., Fitneva, S. A., Frank, M. C., Grange, J. A., Hartshorne, J. K., Hasselman, F., Henninger, F., van der Hulst, M., Jonas, K. J., Lai, C. K., Levitan, C. A., Miller, J. K., Moore, K. S., Meixner, J. M., Munafò, M. R., Neijenhuijs, K. I., Nilsonne, G., Nosek, B. A., Plessow, F., Prenoveau, J. M., Ricker, A. A., Schmidt, K., Spies, J. R., Steiger, S., Strohminger, N., Sullivan, G. B., van Aert, R. C. M., van Assen, M. A. L. M., Vanpaemel, W., Vianello, M., Voracek, M. & Zuni, K. (2016) Response to comment on “estimating the reproducibility of psychological science.” Science 351(6277):1037.CrossRefGoogle ScholarPubMed
Baker, M. (2016) Is there a reproducibility crisis? Nature 533:452–54.CrossRefGoogle Scholar
Barsalou, L. W. (2016) Situated conceptualization offers a theoretical account of social priming. Current Opinion in Psychology 12:6–11.CrossRefGoogle Scholar
Baumeister, R. F. (2016) Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology 66:153–58. Available at: http://doi.org/10.1016/j.jesp.2016.02.003.CrossRefGoogle Scholar
Bem, D. J. (2003) Writing the empirical journal article. In: The compleat academic: A career guide, 2nd edition, ed . Darley, J. M., Zanna, M. P. & Roediger, H. L. III, pp . 185–219. American Psychological Association.Google Scholar
Bem, D. J. (2011) Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100:407–25.10.1037/a0021524CrossRefGoogle ScholarPubMed
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. & van ‘t Veer, A. (2014) The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology 50:217–24.CrossRefGoogle Scholar
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. & Munafò, M. R. (2013) Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14(5):365–76.10.1038/nrn3475CrossRefGoogle ScholarPubMed
Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmedj, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M. & Wu, H. (2016) Evaluating replicability of laboratory experiments in economics. Science 351(6280):1433–36.CrossRefGoogle ScholarPubMed
Cesario, J. (2014) Priming, replication, and the hardest science. Perspectives on Psychological Science 9:40–48.CrossRefGoogle ScholarPubMed
Chambers, C. (2017) The 7 deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press.Google Scholar
Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R. A., Aykutoğlu, B., Bahník, Š., Bowen, J. D., Bredbow, C. A., Bromberg, C., Caprariello, P. A., Carcedo, R. J., Carson, K. J., Cobb, R. J., Collins, N. L., Corretti, C. A., DiDonato, T. E., Ellithorpe, C., Fenrnández-Rouco, N., Fuglestad, P. T., Goldberg, R. M., Golom, F. D., Gündoğdu-Aktürk, E., Hoplock, L. B., Houdek, P., Kane, H. S., Kim, J. S., Kraus, S., Leone, C. T., Li, N. P., Logan, J. M., Millman, R. D., Morry, M. M., Pink, J. C., Ritchey, T., Root Luna, L. M., Sinclair, H. C., Stinson, D. A., Sucharyna, T. A., Tidwell, N. D., Uysal, A., Vranka, M., Winczewski, L. A. & Yong, J. C. (2016) Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro & Hannon (2002). Perspectives on Psychological Science 11(5):750–64.CrossRefGoogle ScholarPubMed
Cohen, J. (1990) Things I have learned (so far). American Psychologist 45(12):1304.10.1037/0003-066X.45.12.1304CrossRefGoogle Scholar
Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., Mcllor, W. & Oxman, A. D. (1993) Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 269(21):2749–53.CrossRefGoogle ScholarPubMed
Coyne, J. C. (2016) Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology 4:28. Available at: http://doi.org/10.1186/s40359-016-0134-3.CrossRefGoogle Scholar
Crandall, C. S. & Sherman, J. W. (2016) On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology 66:93–99. Available at: http://doi.org/10.1016/j.jesp.2015.10.002.CrossRefGoogle Scholar
Dunlap, K. (1926) The experimental methods of psychology. In: Psychologies of 1925, ed . Murchison, C., pp . 331–53. Clark University Press.CrossRefGoogle Scholar
Ebersole, C. R., Axt, J. R. & Nosek, B. A. (2016b) Scientists’ reputations are based on getting it right, not being right. PLoS Biology 14(5):e1002460. Available at: https://doi.org/10.1371/journal.pbio.1002460.CrossRefGoogle Scholar
Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P., Berger, S. A., Birt, A. R., Capezza, N., Carlucci, M., Crocker, C., Ferretti, T. R., Kibbe, M. R., Knepp, M. M., Kurby, C. A., Melcher, J. M., Michael, S. W., Poirier, C. & Prenoveau, J. M. (2016) Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science 11:158–71.CrossRefGoogle ScholarPubMed
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J. & Nosek, B. A. (2014) An open investigation of the reproducibility of cancer biology research. Elife 3:e04333.CrossRefGoogle ScholarPubMed
Etz, A. (2015, August 30) The Bayesian Reproducibility Project . Weblog post . Retrieved 23 August 2017 from: https://web.archive.org/web/20160407113631/http://alexanderetz.com:80/2015/08/30/the-bayesian-reproducibility-project/.Google Scholar
Etz, A. & Vandekerckhove, J. (2016) A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE 11(2):e0149794. Available at: http://doi.org/10.1371/journal.pone.0149794.CrossRefGoogle ScholarPubMed
Etz, A. & Wagenmakers, E. J. (2017) JBS Haldane's contribution to the Bayes factor hypothesis test. Statistical Science 32(2):313–29.CrossRefGoogle Scholar
Fabrigar, L. R. & Wegener, D. T. (2016) Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66:68–80.CrossRefGoogle Scholar
Ferguson, C. J. & Brannick, M. T. (2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods 17(1):120–28. Available at: http://doi.org/10.1037/a0024445.CrossRefGoogle ScholarPubMed
Ferguson, C. J. & Heene, M. (2012) A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7(6):555–61.CrossRefGoogle Scholar
Fetterman, A. K. & Sassenberg, K. (2015) The reputational consequences of failed replications and wrongness admission among scientists. PLoS One 10(12):e0143723. Available at: https://doi.org/10.1371/journal.pone.0143723.CrossRefGoogle ScholarPubMed
Fiedler, K. & Schwarz, N. (2015) Questionable research practices revisited. Social Psychological and Personality Science 7:45–52. Available at: http://doi.org/10.1177/1948550615612150.CrossRefGoogle Scholar
Finkel, E. J., Eastwick, P. W. & Reis, H. T. (2015) Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology 108:275–97. Available at: http://doi.org/10.1037/pspi0000007.CrossRefGoogle ScholarPubMed
Franco, A., Malhotra, N. & Simonovits, G. (2014) Publication bias in the social sciences: Unlocking the file drawer. Science 345(6203):1502–505.CrossRefGoogle ScholarPubMed
Gelman, A. & Loken, E. (2014) The statistical crisis in science data-dependent analysis – a “garden of forking paths” – explains why many statistically significant comparisons don't hold up. American Scientist 102(6):460.CrossRefGoogle Scholar
Gilbert, D. T., King, G., Pettigrew, S, & Wilson, T. D. (2016) Comment on “estimating the reproducibility of psychological science.” Science 351(6277):1037. Available at: http://doi.org/10.1126/science.aad7243.CrossRefGoogle ScholarPubMed
Greenwald, A. G. (1975) Significance, nonsignificance, and interpretation of an ESP experiment. Journal of Experimental Social Psychology 11:180–91.10.1016/S0022-1031(75)80020-5CrossRefGoogle Scholar
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., Bruyneel, S., Calvillo, D. P., Campbell, W. K., Cannon, P. R., Carlucci, M., Carruth, N. P., Cheung, T., Crowell, A., De Ridder, D. T. D., Dewitte, S., Elson, M., Evans, J. R., Fay, B. A., Fennis, B. M, Finley, A., Francis, Z., Heise, E., Hoemann, H., Inzlicht, M., Koole, S. L., Koppel, L., Kroese, F., Lange, F., Lau, K., Lynch, B. P., Martijn, C., Merckelbach, H., Mills, N. V., Michirev, A., Miyake, A., Mosser, A. E., Muise, M., Muller, D., Muzi, M., Nalis, D., Nurwanti, R., Otgaar, H, Philipp, M. C., Primoceri, P., Rentzsch, K., Ringos, L., Schlinkert, C., Schmeichel, B. J., Schoch, S. F., Schrama, M., Schütz, A., Stamos, A., Tinghög, G., Ullrich, J., vanDellen, M., Wimbarti, S., Wolff, W., Yusainy, C., Zerhouni, O. & Zwienenberg, M. (2016) A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science 11(4):546–73.CrossRefGoogle ScholarPubMed
Hewitt, J. K. (2012) Editorial policy on candidate gene association and candidate gene-by-environment interaction studies of complex traits. Behavior Genetics 42(1):1–2.10.1007/s10519-011-9504-zCrossRefGoogle Scholar
Hofstee, W. K. B. (1984) Methodological decision rules as research policies: A betting reconstruction of empirical research. Acta Psychologica 56:93–109.Google Scholar
Hüffmeier, J., Mazei, J. & Schultze, T. (2016) Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology 66:81–92.CrossRefGoogle Scholar
Ioannidis, J. P. (2005) Why most published research findings are false. PLoS Medicine 2(8):e124.CrossRefGoogle ScholarPubMed
Jeffreys, H. (1961) Theory of probability. Oxford University Press.Google Scholar
John, L. K., Loewenstein, G. & Prelec, D. (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23(5):524–32.CrossRefGoogle ScholarPubMed
Kahneman, D. (2003) Experiences of collaborative research. American Psychologist 58:723.10.1037/0003-066X.58.9.723CrossRefGoogle ScholarPubMed
Kerr, N. L. (1998) HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2:196–217. Available at: http://doi.org/10.1207/s15327957pspr0203_4.CrossRefGoogle ScholarPubMed
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B. Jr., Bahník, S., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., Hicks, J. A., Hovermale, J. F., Hunt, S. J., Hunstinger, J. R., IJerzman, H., John, M.-S., Joy-Gaba, J. A., Kappes, H. B., Krueger, L. E., Kurtz, J., Levitan, C. A., Mallett, R. K., Morris, W. L., Nelson, A. J., Nier, J. A., Packard, G., Pilati, R., Rutchick, A. M., Schmidt, K., Skorinko, J. L., Smith, R., Steiner, T. G., Storbeck, J., Van Swol, L. M., Thompson, D., van't Veer, A. E., Vaughn, L. A., Vranka, M., Wichman, A. L., Woodzicka, J. A. & Nosek, B. A. (2014a) Investigating variation in replicability: A “Many Labs” replication project. Social Psychology 45(3):142–52. Available at: http://doi.org/10.1027/1864-9335/a000178.CrossRefGoogle Scholar
Kühberger, A., Fritz, A. & Scherndl, T. (2014) Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One 9(9):e105825.CrossRefGoogle ScholarPubMed
Kunert, R. (2016) Internal conceptual replications do not increase independent replication success. Psychonomic Bulletin and Review 23(5):1631–38.CrossRefGoogle Scholar
Lakatos, I. (1970) Falsification and the methodology of scientific research programmes. In: Criticism and the growth of knowledge, ed. Lakatos, I. & Musgrave, A. , pp . 91–196. Cambridge University Press.10.1017/CBO9781139171434.009CrossRefGoogle Scholar
LeBel, E. P., Berger, D., Campbell, L. & Loving, T. J. (2017) Falsifiability is not optional. Journal of Personality and Social Psychology 113:254–61.CrossRefGoogle Scholar
Levelt Committee Noort Committee & Drent Committee (2012, November 28) Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Retrieved 21 August 2017 from www.tilburguniversity.edu/upload/3ff904d7-547b-40ae85febea38e05a34a_Final%20report%20Flawed%20Science.pdf.Google Scholar
Lupia, A. & Elman, C. (2014) Openness in political science: Data access and research transparency. PS – Political Science and Politics 47:19–42.10.1017/S1049096513001716CrossRefGoogle Scholar
Ly, A., Etz, A., Marsman, M. & Wagenmakers, E. J. (2017) Replication Bayes factors from evidence updating. PsyArXiv preprints. Available at: https://psyarxiv.com/u8m2s/.Google Scholar
Ly, A., Verhagen, J. & Wagenmakers, E. J. (2016) Harold Jeffreys's default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology 72:19–32.10.1016/j.jmp.2015.06.004CrossRefGoogle Scholar
Lykken, D. T. (1968) Statistical significance in psychological research. Psychological Bulletin 70:151–59.CrossRefGoogle ScholarPubMed
Makel, M. C., Plucker, J. A. & Hegarty, B. (2012) Replications in psychology research: How often do they occur? Perspectives on Psychological Science 7(6):537–42.10.1177/1745691612460688CrossRefGoogle ScholarPubMed
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W. & Wagenmakers, E.-J. (2015) The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General 144(1):e1–15. Available at: http://doi.org/10.1037/xge0000038.Google ScholarPubMed
Maxwell, S. E., Lau, M. Y. & Howard, G. S. (2015) Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist 70:487–98. Available at: http://dx.doi.org/10.1037/a0039400.CrossRefGoogle ScholarPubMed
Meehl, P. E. (1990b) Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry 1:108–41. Available at: http://doi.org/10.1207/s15327965pli0102_1.CrossRefGoogle Scholar
Mellers, B., Hertwig, R. & Kahneman, D. (2001) Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science 12:269–75.10.1111/1467-9280.00350CrossRefGoogle ScholarPubMed
Morey, R. D. & Lakens, D. (2016) Why most of psychology is statistically unfalsifiable . Available at: https://github.com/richarddmorey/psychology_resolution/blob/master/paper/response.pdfGoogle Scholar
National Cancer Institute–National Human Genome Research Institute (NCI-NHGRI) Working Group on Replication in Association Studies (2007) Replicating genotype-phenotype associations. Nature 447:655–60.10.1038/447655aCrossRefGoogle Scholar
Nosek, B. A. & Errington, T. M. (2017) Making sense of replications. eLife 6:e23383.10.7554/eLife.23383CrossRefGoogle ScholarPubMed
Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349(6251):aac4716. Available at: http://doi.org/10.1126/science.aac4716.Google Scholar
Pashler, H. & Harris, C. (2012) Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7:531–36.CrossRefGoogle Scholar
Pashler, H. & Wagenmakers, E.-J. (2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science 7:528–30.10.1177/1745691612465253CrossRefGoogle Scholar
Popper, K. R. (1959/2002) The logic of scientific discovery, translation of Logik der Forschung . Routledge.Google Scholar
Rohrer, D., Pashler, H. & Harris, C. R. (2015) Do subtle reminders of money change people's political views? Journal of Experimental Psychology: General 144(4):e73.Google ScholarPubMed
Rosenthal, R. (1979) The “file drawer problem” and tolerance for null results. Psychological Bulletin 86(3):638–41.CrossRefGoogle Scholar
Rotello, C. M., Heit, E. & Dubé, C. (2015) When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin and Review 22:944–54.CrossRefGoogle ScholarPubMed
Rothstein, H. R. & Bushman, B. J. (2012) Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods 17:129–36.10.1037/a0027128CrossRefGoogle Scholar
Schmidt, F. L. & Oh, I.-S. (2016) The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology 4(1):32–37. Available at: http://dx.doi.org/10.1037/arc0000029.CrossRefGoogle Scholar
Schmidt, S. (2009) Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13:90–100.CrossRefGoogle Scholar
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., Srinivasan, M., Tannenbaum, D., Bivolaru, E., Dana, J., Davis-Stober, C. P., du Plessis, C., Gronau, Q. F., Hafenbrack, A. C., Liao, E. Y., Ly, A., Marsman, M., Murase, T., Qureshi, I., Schaerer, M., Thornley, N., Tworek, C. M., Wagenmakers, E.-J., Wong, L., Anderson, T., Bauman, C. W., Bedwell, W. L., Brescoll, V., Canavan, A., Chandler, J. J., Cheries, E., Cheryan, S., Cheung, F., Cimpian, A., Clark, M. A., Cordon, D., Cushman, F., Ditto, P. H., Donahue, T., Frick, S. E., Gamez-Djokic, M., Hofstein Grady, R., Graham, J., Gu, J., Hahn, A., Hanson, B. E., Hartwich, N. J., Hein, K. Inbar, Y., Jiang, L., Kellogg, T., Kennedy, D. M., Legate, N., Luoma, T. P., Maibuecher, H., Meindl, P., Miles, J., Mislin, A., Molden, D. C., Motyl, M., Newman, G., Ngo, H. H., Packham, H., Ramsay, P. S., Ray, J. L., Sackett, A. M., Sellier, A.-L., Sokolova, T., Sowden, W., Storage, D., Sun, X., Van Bavel, J. J., Washburn, A. N., Wei, C., Wetter, E., Wilson, C. T., Darrous, S.-C. & Uhlmann, E. L. (2016) The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology 66:55–67.10.1016/j.jesp.2015.10.001CrossRefGoogle Scholar
Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., Tamman, A. J. & Puhlmann, L. (2015) Romance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives? Journal of Experimental Psychology: General 144(6):e142–58.Google ScholarPubMed
Silberzahn, R., Uhlmann, E. L., Martin, D., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T. J., Hederos, K., Heene, M., Hofelich Mohr, A. J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D. M., Lei, R., Lindsay, T. A., Liverani, S., Madan, C. R., Molden, D., Molleman, E., Morey, R. D., Mulder, L. B., Nijstad, B. R., Pope, N. G., Pope, B., Prenoveau, J. M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F. D., Sherman, M. F., Sommer, S. A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E.-J., Witkowiak, M., Yoon, S. & Nosek, B. A. (2017) Many analysts, one dataset: Making transparent how variations in analytical choices affect results. PsyArXiv Preprint. Available at: https://psyarxiv.com/qkwst/.10.1177/2515245917747646CrossRefGoogle Scholar
Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:1359–66. Available at: http://doi.org/10.1177/0956797611417632.CrossRefGoogle ScholarPubMed
Simons, D. J., Shoda, Y. & Lindsay, D. S. (2017) Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science 12:1123–28. Available at: http://doi.org/10.1177/1745691617708630.CrossRefGoogle ScholarPubMed
Simonsohn, U. (2015) Small telescopes: Detectability and the evaluation of replication results. Psychological Science 26:559–69.10.1177/0956797614567341CrossRefGoogle ScholarPubMed
Simonsohn, U. (2016, March 3) [47] Evaluating replications: 40% full≠60% empty . Available at: https://web.archive.org/web/20170709184952/http://datacolada.org/47.Google Scholar
Smaldino, P. E. & McElreath, R. (2016) The natural selection of bad science. Royal Society Open Science 3:160384.10.1098/rsos.160384CrossRefGoogle ScholarPubMed
Smart, R. G. (1964) The importance of negative results in psychological research. Canadian Psychologist 5:225–32.CrossRefGoogle Scholar
Spellman, B. A. (2015) A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science 10:886–99.10.1177/1745691615609918CrossRefGoogle Scholar
Sterling, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54(285):30–34. Available at: http://doi.org/10.2307/2282137.Google Scholar
Sterling, T. D., Rosenbaum, W. L. & Weinkam, J. J. (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician 49:108–12.10.1080/00031305.1995.10476125CrossRefGoogle Scholar
Stroebe, W. & Strack, F. (2014) The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9:59–71. Available at: http://doi.org/10.1177/1745691613514450.CrossRefGoogle ScholarPubMed
van Aert, R. C. & van Assen, M. A. (2017) Bayesian evaluation of effect size after replicating an original study. PLoS One 12(4):e0175302.CrossRefGoogle ScholarPubMed
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J. & Reinero, D. A. (2016) Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America 113(23):6454–59.10.1073/pnas.1521897113CrossRefGoogle ScholarPubMed
Verhagen, A. J. & Wagenmakers, E.-J. (2014) Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General 143:1457–75.10.1037/a0036731CrossRefGoogle ScholarPubMed
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B. Jr., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., Bulnes, L. C., Caldwell, T. L., Calin-Jageman, R. J., Capaldi, C. A., Carfagno, N. S., Chasten, K. T., Cleeremans, A., Connell, L., DeCicco, J. M., Dijkstra, K, Fischer, A. H., Foroni, F., Hess, U., Holmes, K. J., Jones, J. L. H., Klein, O., Koch, C., Korb, S., Lewinski, P., Liao, J. D., Lund, S., Lupianez, J., Lynott, D., Nance, C. N., Oosterwijk, S., Ozdoğru, A. A., Pacheco-Unguetti, A. P., Pearson, B., Powis, C., Riding, S., Roberts, T.-A., Rumiati, R. I., Senden, M., Shea-Shumsky, N. B., Sobocko, K., Soto, J. A., Steiner, T. G., Talarico, J. M., van Allen, Z. M., Vandekerckhove, M., Wainwright, B., Wayand, J. F., Zeelenberg, R., Zetzer, E. E. & Zwaan, R. A. (2016a) Registered replication report: Strack, Martin & Stepper (1988). Perspectives on Psychological Science 11:917–28.10.1177/1745691616674458CrossRefGoogle Scholar
Wagenmakers, E.-J., Verhagen, A. J. & Ly, A. (2016b) How to quantify the evidence for the absence of a correlation. Behavior Research Methods 48:413–26.10.3758/s13428-015-0593-0CrossRefGoogle Scholar
Wrinch, D. & Jeffreys, H. (1921) XLII. On certain fundamental principles of scientific inquiry. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 42(249):369–90.CrossRefGoogle Scholar
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K. & Zeelenberg, R. (2017) Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin and Review. Available at: http://doi.org/10.3758/s13423-017-1348-y.Google Scholar