Hostname: page-component-5447f9dfdb-lxwkt Total loading time: 0 Render date: 2025-07-31T02:24:53.387Z Has data issue: false hasContentIssue false
  • English
  • Français

Eliciting Beliefs as Distributions in Online Surveys

Published online by Cambridge University Press:  04 February 2021

Lucas Leemann Open the ORCID record for Lucas Leemann [Opens in a new window] ,
Lukas F. Stoetzer  and
Richard Traunmüller
Lucas Leemann*
Affiliation:
Department of Political Science, University of Zürich, Zürich, Switzerland. Email: leemann@ipz.uzh.ch
Lukas F. Stoetzer
Affiliation:
Humboldt University of Berlin, Cluster of Excellence SCRIPTS, Berlin, Germany. Email: lukas.stoetzer@hu-berlin.de
Richard Traunmüller
Affiliation:
School of Social Sciences, University of Mannheim, Mannheim, Germany. Email: traunmueller@uni-mannheim.de
*
Corresponding author Lucas Leemann
Get access Rights & Permissions [Opens in a new window]

Abstract

Citizens’ beliefs about uncertain events are fundamental variables in many areas of political science. While beliefs are often conceptualized in the form of distributions, obtaining reliable measures in terms of full probability densities is a difficult task. In this letter, we ask if there is an effective way of eliciting beliefs as distributions in the context of online surveys. Relying on experimental evidence, we evaluate the performance of five different elicitation methods designed to capture citizens’ uncertain expectations. Our results suggest that an elicitation method originally proposed by Manski (2009) performs well. It measures average citizens’ subjective belief distributions reliably and is easily implemented in the context of regular (online) surveys. We expect that a wider use of this method will lead to considerable improvements in the study of citizens’ expectations and beliefs.

Keywords

prior elicitationonline survey researchcitizen beliefsmeasurement

Information

Type
Letter
Information
Political Analysis , Volume 29 , Issue 4 , October 2021 , pp. 541 - 553
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Edited by Daniel Hopkins

References

Alvarez, R. M., and Brehm, J.. 1997. “Are Americans Ambivalent Towards Racial Policies?” American Journal of Political Science 41:345–374.CrossRefGoogle Scholar
Bartels, L. M. 2002. “Beyond the Running Tally: Partisan Bias in Political Perceptions.” Political Behavior 24(2):117–150.CrossRefGoogle Scholar
Berinsky, A. J. 2017. “Rumors and Health Care Reform: Experiments in Political Misinformation.” British Journal of Political Science 47(2):241–262.CrossRefGoogle Scholar
Berinsky, A. J, Huber, G. A., and Lenz, G. S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon. com’s Mechanical Turk.” Political Analysis 20(3):351–368.CrossRefGoogle Scholar
Berinsky, A. J, Margolis, M. F., and Sances, M. W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-administered Surveys.” American Journal of Political Science 58(3):739–753.CrossRefGoogle Scholar
Bullock, J. G. 2009. “Partisan Bias and the Bayesian Ideal in the Study of Public Opinion.” The Journal of Politics 71(3):1109–1124.CrossRefGoogle Scholar
Coppock, A. 2018. “Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach.” Political Science Research and Methods 7:1–16.Google Scholar
Cosmides, L., and Tooby, J.. 1996. “Are Humans Good Intuitive Statisticians after all? Rethinking Some Conclusions from the Literature on Judgment under Uncertainty.” Cognition 58(1):1–73.CrossRefGoogle Scholar
Delavande, A., and Rohwedder, S.. 2008. “Eliciting Subjective Probabilities in Internet Surveys.” Public Opinions Quarterly 72(5):866–891.CrossRefGoogle ScholarPubMed
Garthwaite, P. H., Kadane, J. B., and O’Hagan, A.. 2005. “Statistical Methods for Eliciting Probability Distributions.” Journal of the American Statistical Association 100(470):680–701.CrossRefGoogle Scholar
Gerber, A., and Green, D.. 1999. “Misperceptions About Perceptual Bias.” Annual Review of Political Science 2(1):189–210.CrossRefGoogle Scholar
Gill, J., and Freeman, J. R.. 2013. “Dynamic Elicited Priors for Updating Covert Networks.” Network Science 1(1):68–94.CrossRefGoogle Scholar
Gill, J., and Walker, L. D.. 2005. “Elicited Priors for Bayesian Model Specifications in Political Science Research.” The Journal of Politics 67(3):841–872.CrossRefGoogle Scholar
Goldstein, D. G., and Rothschild, D.. 2014. “Lay Understanding of Probability Distributions.” Judgment & Decision Making 9(1):1–14.Google Scholar
Kuru, O., Pasek, J., and Traugott, M. W.. 2017. “Motivated Reasoning in the Perceived Credibility of Public Opinion Polls.” Public Opinion Quarterly 81(2):422–446.CrossRefGoogle Scholar
Kynn, M. 2008. “The ‘Heuristics and Biases’ Bias in Expert Elicitation.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 171(1):239–264.Google Scholar
Lebo, M. J., and Cassino, D.. 2007. “The Aggregated Consequences of Motivated Reasoning and the Dynamics of Partisan Presidential Approval.” Political Psychology 28(6):719–746.CrossRefGoogle Scholar
Leemann, L., Traunmueller, R., and Stoetzer, L. F.. 2020. “Replication Data for: Eliciting Beliefs as Distributions in Online Surveys.” https://doi.org/10.7910/DVN/GEC2LS, Harvard Dataverse, V1, UNF:6:AJOSTeuI2rt9XmpP7/2rlg== [fileUNF].Google Scholar
Leiter, D., Murr, A., Ramirez, E. R., and Stegmaier, M.. 2018. “Social Networks and Citizen Election Forecasting: The More Friends the Better.” International Journal of Forecasting 34(2):235–248.CrossRefGoogle Scholar
Madson, G. J., and Hillygus, D. S.. 2019. “All the Best Polls Agree with Me: Bias in Evaluations of Political Polling.” Political Behavior 5:1–18.Google Scholar
Manski, C. F. 2009. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.Google Scholar
Mason, W, and Suri, S.. 2012. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Behavior Research Methods 44(1):1–23.CrossRefGoogle ScholarPubMed
Morris, D. E., Oakley, J. E., and Crowe, J. A.. 2014. “A Web-Based Tool for Eliciting Probability Distributions from Experts.” Environmental Modelling & Software 52:1–4.CrossRefGoogle Scholar
Mullinix, K. J., Leeper, T. J., Druckman, J. N., and Freese, J.. 2015. “The Generalizability of Survey Experiments.” Journal of Experimental Political Science 2(2):109–138.CrossRefGoogle Scholar
Murr, A. 2011. “‘Wisdom of Crowds’? A Decentralised Election Forecasting Model that Uses Citizens’ Local Expectations.” Electoral Studies 30(4):771–783.CrossRefGoogle Scholar
O’Hagan, A. et al. 2006. Uncertain Judgements: Eliciting Experts’ Probabilities. Boca Raton, FL: John Wiley & Sons.CrossRefGoogle Scholar
Peterson, C., and Miller, A.. 1964. “Mode, Median, and Mean as Optimal Strategies.” Journal of Experimental Psychology 68(4):363.CrossRefGoogle ScholarPubMed
Savage, L. J. 1971. “Elicitation of Personal Probabilities and Expectations.” Journal of the American Statistical Association 66(336):783–801.CrossRefGoogle Scholar
Schlag, K. H., Tremewan, J., and Van der Weele, J. J.. 2015. “A Penny for Your Thoughts: A Survey of Methods for Eliciting Beliefs.” Experimental Economics 18(3):457–490.CrossRefGoogle Scholar
Spetzler, C. S., and Stael von Holstein, C.-A. S.. 1975. “Exceptional Paper—Probability Encoding in Decision Analysis.” Management Science 22(3):340–358.CrossRefGoogle Scholar
Thomas, K. A., and Clifford, S.. 2017. “Validity and Mechanical Turk: An Assessment of Exclusion Methods and Interactive Experiments.” Computers in Human Behavior 77:184–197.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D.. 1971. “Belief in the Law of Small Numbers.” Psychological Bulletin 76(2):105.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D.. 1973. “Availability: A Heuristic for Judging Frequency and Probability.” Cognitive Psychology 5(2):207–232.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D. l.. 1974. “Judgment under Uncertainty: Heuristics and Biases.” Science 185(4157):1124–1131.CrossRefGoogle ScholarPubMed
Wallsten, T. S., and Budescu, D. V.. 1983. “State of the Art—Encoding Subjective Probabilities: A Psychological and Psychometric Review.” Management Science 29(2):151–173.CrossRefGoogle Scholar
Zaller, J., and Feldman, S.. 1992. “A Simple Theory of the Survey Response: Answering Questions and Revealing Preferences.” American Journal of Political Science 36:579–616.CrossRefGoogle Scholar
Supplementary material: PDF

Leemann et al. supplementary material

Leemann et al. supplementary material

Download Leemann et al. supplementary material(PDF)
PDF 593.3 KB