Hostname: page-component-6bb9c88b65-kzqxb Total loading time: 0 Render date: 2025-07-23T19:00:36.180Z Has data issue: false hasContentIssue false
  • English
  • Français

Measuring the Quality of Answers in Political Q&As with Large Language Models

Published online by Cambridge University Press:  16 July 2025

R. Michael Alvarez Open the ORCID record for R. Michael Alvarez [Opens in a new window]  and
Jacob Morrier Open the ORCID record for Jacob Morrier [Opens in a new window]
R. Michael Alvarez
Affiliation:
Division of the Humanities and Social Sciences, https://ror.org/05dxps055California Institute of Technology, Pasadena, CA, USA.
Jacob Morrier*
Affiliation:
Division of the Humanities and Social Sciences, https://ror.org/05dxps055California Institute of Technology, Pasadena, CA, USA.
*
Corresponding author: Jacob Morrier; Email: jmorrier@caltech.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

This article proposes a new approach for measuring the quality of answers in political question-and-answer sessions. We assess the quality of an answer based on how easily and accurately it can be recognized among a random set of candidate answers given the question’s text. This measure reflects the answer’s relevance and depth of engagement with the question. Drawing a parallel with semantic search, we can implement this approach by training a language model on the corpus of observed questions and answers without additional human-labeled data. We showcase and validate our methodology within the context of the Question Period in the Canadian House of Commons. Our analysis reveals that while some answers only have a weak semantic connection to questions, suggesting some evasion or obfuscation, they are generally at least moderately relevant, far exceeding what we would expect from random replies. We also find meaningful correlations between the quality of answers and the party affiliation of the members of Parliament asking the questions.

Keywords

answer qualityCanadian politicslarge language modelsnatural language processing (NLP)political Q&AsQuestion Period (QP)semantic search

Information

Type
Article
Information
Political Analysis , First View , pp. 1 - 18
Copyright
© The Author(s), 2025. Published by Cambridge University Press on behalf of The Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Edited by: Margaret E. Roberts

References

Arslan, Y., Allix, K., Veiber, L., Lothritz, C., Bissyandé, T. F., Klein, J., and Goujon, A.. 2021. “A Comparison of Pre-Trained Language Models for Multi-Class Text Classification in the Financial Domain.” In Companion Proceedings of the Web Conference, vol. 2021, 260–268.Google Scholar
Bast, H., Buchhold, B., and Haussmann, E.. 2016. “Semantic Search on Text and Knowledge Bases.” Foundations and Trends $\circledR$ in Information Retrieval 10 (2–3): 119–271.Google Scholar
Bates, S. R., Kerr, P., Byrne, C., and Stanley, L.. 2012. “Questions to the Prime Minister: A Comparative Study of PMQs from Thatcher to Cameron.” Parliamentary Affairs 67 (2): 253–280.10.1093/pa/gss044CrossRefGoogle Scholar
Bestvater, S. E., and Monroe, B. L.. 2023. “Sentiment Is Not Stance: Target-Aware Opinion Classification for Political Text Analysis.” Political Analysis 31 (2): 235–256.10.1017/pan.2022.10CrossRefGoogle Scholar
Bosc, M., and Gagnon, A., eds. 2017. House of Commons Procedure and Practice. Third Edition. Éditions Yvon Blais, Montréal, Canada.Google Scholar
Bull, P. 2000. “Equivocation and the Rhetoric of Modernization: An Analysis of Televised Interviews with Tony Blair in the 1997 British General Election.” Journal of Language and Social Psychology 19 (2): 222–247.10.1177/0261927X00019002003CrossRefGoogle Scholar
Bull, P. 1998. “Equivocation Theory and News Interviews.” Journal of Language and Social Psychology 17 (1): 36–51.10.1177/0261927X980171002CrossRefGoogle Scholar
Bull, P. 1994. “On Identifying Questions, Replies, and Non-Replies in Political Interviews.” Journal of Language and Social Psychology 13 (2): 115–131.10.1177/0261927X94132002CrossRefGoogle Scholar
Bull, P. 2004. “The Analysis of Equivocation in Political Interviews.” In Doing Social Psychology Research, edited by Breakwell, G. M., 205–228. John Wiley & Sons, Honoken, United States.10.1002/9780470776278.ch9CrossRefGoogle Scholar
Bull, P., and Mayer, K.. 1993. “How Not to Answer Questions in Political Interviews.” Political Psychology 14 (4): 651–666.10.2307/3791379CrossRefGoogle Scholar
Bull, P., and Strawson, W.. 2019. “Can’t Answer? Won’t Answer? An Analysis of Equivocal Responses by Theresa May in Prime Minister’s Questions.” Parliamentary Affairs 73 (2): 429–449.Google Scholar
Bächtiger, A. 2014. “Debate and Deliberation in Legislatures.” In The Oxford Handbook of Legislative Studies, edited by S. Martin, T. Saalfeld, and K. W. Strøm. Oxford University Press.Google Scholar
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1.Google Scholar
Goet, N. D. 2019. “Measuring Polarization with Text Analysis: Evidence from the UK House of Commons, 1811–2015.” Political Analysis 27 (4): 518–539.10.1017/pan.2019.2CrossRefGoogle Scholar
Henderson, M., Al-Rfou, R., Strope, B., Sung, Y., Lukacs, L., Guo, R., Kumar, S., Miklos, B., and Kurzweil, R.. 2017. “Efficient Natural Language Response Suggestion for Smart Reply.” arXiv:1705.00652.Google Scholar
Jurafsky, D., and Martin, J. H.. 2024. Speech and Language Processing. Third Edition. https://web.stanford.edu/~jurafsky/slp3/ed3bookfeb3_2024.pdf.Google Scholar
Kernaghan, K. 1979. “Power, Parliament and Public Servants in Canada: Ministerial Responsibility Reexamined.” Canadian Public Policy/Analyse de Politiques 5 (3): 383–396.10.2307/3550225CrossRefGoogle Scholar
Kukec, M. 2022. “Ask me Something I Know: Cabinet Members in Question Time.” The Journal of Legislative Studies 30 (4): 1–23.Google Scholar
Lakoff, G. 2014. The ALL NEW Don’t Think of an Elephant!: Know your Values and Frame the Debate. Chelsea Green Publishing, White River Junction, United States.Google Scholar
Laurer, M., van Atteveldt, W., Casas, A., and Welbers, K.. 2024. “Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI.” Political Analysis 32 (1): 84–100.10.1017/pan.2023.20CrossRefGoogle Scholar
Maricut-Akbik, A. 2021. “Q&A in Legislative Oversight: A Framework for Analysis.” European Journal of Political Research 60 (3): 539–559.10.1111/1475-6765.12411CrossRefGoogle Scholar
Martin, L. 2011. Harperland: The Politics of Control. Updated Edition. Penguin Canada.Google Scholar
Morrier, J., and Alvarez, R. M.. 2025. “Replication Data for: ‘Measuring the Quality of Answers in Political Q&As with Large Language Models.” https://doi.org/10.7910/DVN/MXNQ7F.CrossRefGoogle Scholar
Peterson, A., and Spirling, A.. 2018. “Classification Accuracy as a Substantive Quantity of Interest: Measuring Polarization in Westminster Systems.” Political Analysis 26 (1): 120–128.10.1017/pan.2017.39CrossRefGoogle Scholar
Rasiah, P. 2010. “A Framework for the Systematic Analysis of Evasion in Parliamentary Discourse.” Journal of Pragmatics 42 (3): 664–680.10.1016/j.pragma.2009.07.010CrossRefGoogle Scholar
Reimers, N., and Gurevych, I.. 2019. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” arXiv:1908.10084.10.18653/v1/D19-1410CrossRefGoogle Scholar
Ruder, S., Peters, M. E., Swayamdipta, S., and Wolf, T.. 2019. “Transfer Learning in Natural Language Processing.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, 15–18.Google Scholar
Waddle, M., and Bull, P.. 2020. “‘You’re Important, Jeremy, but Not that Important’: Personalised Responses and Equivocation in Political Interviews.” Journal of Social and Political Psychology 8 (2): 560–581.10.5964/jspp.v8i2.1095CrossRefGoogle Scholar
Wang, Y. 2023. “Topic Classification for Political Texts with Pretrained Language Models.” Political Analysis 31 (4): 662–668.10.1017/pan.2023.3CrossRefGoogle Scholar
Whyte, T. 2018. “Quantitative Measurement of Parliamentary Accountability Using Text as Data: The Canadian House of Commons, 1945-2015.” PhD dissertation, University of Toronto.Google Scholar
Widmann, T., and Wich, M.. 2023. “Creating and Comparing Dictionary, Word Embedding, and Transformer-Based Models to Measure Discrete Emotions in German Political Text.” Political Analysis 31 (4): 626–641.10.1017/pan.2022.15CrossRefGoogle Scholar
Supplementary material: File

Alvarez and Morrier supplementary material

Alvarez and Morrier supplementary material
Download Alvarez and Morrier supplementary material(File)
File 3.3 MB