Public opinion is a driving force in democracy – governments must be responsive to the needs of their citizens. It determines who will govern and which policies will be more likely to succeed. Yet, given constant societal flux, pinning public opinion down with precision requires a nuanced understanding of context. Political and economic actors so often underestimate or misunderstand public opinion.
This book is for the user of public opinion data presented through the lens of the principal purveyor of such data, the pollster. By users, we mean decision-makers, or those who counsel decision-makers, many of whom see public opinion as an ephemeral and easily manipulable phenomenon.Footnote 1 Nevertheless, contrary to these perceptions, practice and scientific evidence show that public opinion is a stable, measurable, and, ultimately, predictable phenomenon.Footnote 2 Furthermore, when it is ignored, desired outcomes on the policy and political fronts are less likely to be achieved.Footnote 3
Under certain conditions, public opinion can indeed be seemingly unpredictable.Footnote 4 The key from an analytical perspective is to know the conditions under which public opinion is more susceptible to volatility. Analysts must be able to identify the characteristics of these unique circumstances.
The pollster’s work does not stop there; this is only the beginning. Substantial empirical evidence underscores that, in some cases, public opinion is only a secondary or tertiary factor in determining an outcome. Again, analysts need to be able to identify when this may be the case and respond accordingly. This book will examine public opinion conceptually and in practice, while laying out an analytical road map for pollsters on how best to analyze and engage public opinion.
A vast literature exists on theories of public opinion; on how to measure it and how to analyze it.Footnote 5 Still others exist on predicting public opinion and on how to influence it.Footnote 6 So, what does this book contribute?
Missing Elements in the Professional Literature on Public Opinion Analysis
Telling this story through the eyes of the pollster is easier said than done. Why? Well, simply, there is no synthetic treatment of best practices in interpreting public opinion. As we see it, the existing knowledge base on public opinion is disparate – spread across many disciplines, including political science, survey methods, political psychology, statistics, social psychology, cognitive psychology, behavioral economics, data science, and sociology. These literatures do not necessarily talk to each other.
Such a global understanding of the activity of the pollster is especially important now. Take the recent polling misses; they should give everyone pause. After the polls blithely promised a victory for Hillary Clinton over Donald Trump in the 2016 US presidential elections, their accuracy has justifiably come under question. The US 2016 election is by no means the exception. Take Argentina in 2019, or the United States in 2020, or Brazil in 2022, and so on. How can we put such performances into a broader context?
The reality is that the raw tool at the pollster’s disposal – the survey – was refined for an era when people had fewer communication options. Now, almost everyone has a digital footprint, and landlines are going the way of the dinosaurs. Access to people has never been so varied yet so difficult. Naturally, the industry is aware of these issues and adapting its methodologies. Context matters to make sense of this.
Despite the very real challenges the polling industry faces, we can say, with confidence, that polls still offer real, meaningful, and actionable insights. The key is understanding how to interpret the results given our reality. Remember that polling results represent public opinion at a given moment in time. It is, by definition, always retroactive. Yet, when interpreted correctly, public opinion can give us a glimpse of the future, helping us make better decisions today.
The pollster also must contend with a world that is ever more polarized. The public is increasingly tribal in nature, and suspicion of the “other” is high. The fractured nature of public opinion makes it very difficult for practitioners to gain an overarching understanding of it as an analytical variable or decision input. Once again, context matters.
A polling professional today needs to understand a broad range of disciplines as noted previously. Complicating this, even if properly motivated, any given professional cannot necessarily be expected to integrate this vast literature without the requisite time, motivation, and resources. As such, there is not a single book that melds these different fields into one synthetic practically oriented approach. That’s what this book aims to achieve.
A Pollster’s TaleFootnote 7
As a freshly minted PhD from the University of Chicago, I thought of myself as hot stuff. Trained in the science of polling, I could do no wrong. Or so I thought, until theory met practice.
My first poll and paying client was for a group supporting a candidate for the presidency of Jockey Clube Brasileiro – a social club and horse racing track located in the city of Rio de Janeiro. I know it sounds crazy – a poll for an election in a social club. But these people had lots of money and were passionate!
Like any well-trained quantitative social scientist, I fretted about my sample design; it needed to be representative of the member base of the club. Check! I worried about the questionnaire design, no biased questions. Check! I also thought about my analytical plan for data analysis; I wanted to tell a clear simple story. Check!
All good; all great. The poll went well; the data looked great. On the day of the client presentation (really a chat by phone), I was pumped. Pumped, that is, until the client asked me, “So, will I win? What do I need to do to win?”
I sat there dumbfounded, without a ready answer. In fact, at that moment, I realized I lacked the knowledge base needed to provide the proper context and meaning for the client. Sure, I was well trained in the science of polling but had no framework, nor the requisite gray hairs, to shed light on what it all meant.
Questions raced through my mind, “Will he win? What are his chances? And what can he do to change them?” I had no idea where to start.
To add insult to injury, the client came back to me the next day. He had gotten his hands on the other side’s poll. Yes, they had one as well! And it was at odds with our own.
The client, of course, asked a very reasonable question, “What’s up? Are they right or are we right?” He was asking me to assess the polls. Again, I sat there speechless, not only because I spoke halting Portuguese at the time, but because I had no reasonable response but my own logic – a bad place to be as a scientist and expert. I had no empirically based rules of thumb to anchor my analysis. No context.
I would only learn later in my career that, indeed, such analytical heuristics do exist. That there are multiple ways to assess the relative odds of a given candidate. But this knowledge, for me, would be years in the making. Users of public opinion data naturally face the same challenges as newly minted pollsters.
The Knowledge Gap
Most professionals working as pollsters have had similar trials by fire. Their training gave them the skills to tackle certain dimensions of public opinion but left them wholly unable to address others.
For instance, the political scientist may be well versed in theories of public opinion, elections, and election forecasting but less so in the science of polling, messaging, and cognitive psychology/behavioral economics. Meanwhile, students of international studies and conflict have been exposed to approaches to assess political risk but lack a broader toolbox of skills. A cognitive psychologist will have a strong base in information processing, behavioral economics – and hence messaging and engagement – but, again, not necessarily in the other requisite fields.
To truly excel in the field, the practitioner would ideally be versed enough in all disciplines to incorporate them into their day-to-day work. Or, at least, be knowledgeable enough that they will be aware of their own blind spots and weaknesses.
What the Book Is and Is Not
We want to be clear from the outset what this book is and what it is not. First and foremost, this book is not a treatment of the use of public opinion in assessing the impact of public policies at a tactical level. In other words, we are not attempting to answer questions around something as specific as whether a healthcare policy bill should include one feature over another. For that, the reader should turn to the vast literature commonly referred to as program evaluation, impact evaluation, or evidence-based policy assessment.
The focus of this book, instead, is to aid the user of public opinion data in developing a systematic analytical approach for understanding, predicting, and engaging public opinion. More fundamentally, we want to help the reader understand how public opinion can be employed as a decision-making input. By decision-making input, we mean a factor, or variable, to assess, predict, or influence an outcome.
First, this book will unpack the different ways that public opinion can be utilized to interpret and predict sociopolitical outcomes. By sociopolitical outcomes, we mean elections, referenda, political or policy agendas, specific legislation, crises, scandals, and other existential threats to government and societal stability. In this case, public opinion can be thought of as a single decision input when assessing sociopolitical outcomes. For instance, government approval ratings are critical predictors of electoral outcomes. However, using government approval ratings alone to predict an electoral outcome is definitively not the most robust method. It is only when combined with other inputs, such as whether the incumbent is running, that our ability to accurately pick the winner is significantly improved. We will learn that multiple input forecasts are preferable to single input ones.
Second, our book makes a simplifying assumption that decision-makers – both political and economic – are heavy users of public opinion data. Both theory and practice support this assumption. Political theory, in the form of the median voter model,Footnote 8 suggests that parties and politicians are vote maximizers. As such, politicians will generally approximate their policy offer to the preferences of the average voter.Footnote 9 Policy disputes are typically won by those who meet public opinion where it is. We see this in practice all the time, although there are notable exceptions to this rule.
Third, this is not a book on polling or survey methodology; however, methods and methodological robustness will be a theme throughout, especially as it relates to our ability to use public opinion as an assessment tool. We will delve into this in greater detail in Part II of this book. Analysts must be able to determine if the measurement of public opinion is flawed, even if they are not experts in the science of polling.Footnote 10
For more than eighty years, since Gallup correctly predicted that Roosevelt would beat Landon in 1936, the scientific survey sample, and its workhorse, the poll, has been the primary mechanism for understanding public opinion. Prior to surveys, decision-makers used a variety of methods to assess public opinion, including elections, crowd sizes, local newspaper editorial board opinions, riots, protests, and mass marches. Later developments in questionnaires and survey methods further reinforced the poll as the preeminent tool for gauging the mood of the people.Footnote 11 Over the years, the poll has largely become synonymous with public opinion.Footnote 12
But our data world is significantly changing, and, as a result, so have our approaches to assessing public opinion. Looking forward, alternative methods to the poll will only grow in volume and application. Even today, analysts are increasingly employing nonpolling data sources to assess public opinion. Alternative data sources include analyzing social media, satellites, drones, ground sensors, APIs (application programming interfaces), aggregate cell phone data, and prediction marks. Think of the protests in 2022 that roiled Iran in the wake of the death of Jina Amini at the hands of the country’s “morality police.” Observers were able to track the spread of the protests through videos posted online – just one example of many.
Fourth, the book is about the use of public opinion across all domains – both political and nonpolitical. Most of the examples, however, are of elections. The reason for this is twofold. First, elections can be easily validated – we know the “true value” when the election returns come in. And second, most of these data are publicly available. In contrast, we have managed thousands of nonelectoral engagements, but they are proprietary in nature. We attempt to sprinkle in such examples when possible.
To date, the concept of the pollster has referred to an individual or organization that measures, analyzes, and interprets public opinion. But does this designation still apply in a world where the poll is no longer the sole vehicle to gauge public opinion? We argue, “yes.” While technological advances have expanded the means and methods of tracking public sentiment, the importance of public opinion as a decision input has never been more important. No matter the method to measure public opinion, an expert – the pollster – must give meaning to it. We believe that the traditional concept of the pollster is as meaningful today as it always has been. In this book, we focus on public opinion as a decision input, not the poll, even though we present polling data to illustrate our case.
Organization of the Book
The pollster has three primary identities that we will explore in detail in this book. As a framing device, we will refer to the pollster as the “three-hatted pollster,” reflecting the pollster’s distinct identities as a data scientist, fortune teller, and spin doctor (Table 1.1). These relate to the pollster’s primary functions: to assess the polls and related inputs, to predict public opinion outcomes, and to utilize these inputs to shape compelling messages that will resonate with chosen audiences. The pollster wears multiple hats, but before we unpack the various dimensions of the three-hatted pollster, we will explore the concept of public opinion, how our understanding of it has evolved over the centuries, and whether public opinion is a reliable, stable input or something more elusive and volatile.
Table 1.1 The three-hatted pollster
| Part II | Part III | Part IV | |
|---|---|---|---|
| The Data Scientist (to Assess) | The Fortune Teller (to Predict) | The Spin Doctor (to Convince) | |
| Primary Client Type | Media and all other clients | Private sector and financial sector | Politicians, governments, and private sector |
| Analytical Focus | Maximizing accuracy | Forecasting outcomes | Developing the most convincing message |
| Typical Questions | Is my public opinion data biased? | Who will win the next presidential election? | What is the winning message? |
Part I: What Is Public Opinion?
Part I includes two chapters of review and discussion of the relevant scientific and academic literature on public opinion, opinion formation, its effect on outcomes, and popular myths about it.
In Chapter 2, we examine public opinion as a concept. What is it? How has it been defined over the years? How has it been conceptualized by philosophers and social scientists? We also discuss how decision-makers employ public opinion as a decision input and examine the concept of convergence. Specifically, to what extent, and under what conditions, are decision-makers aligned with public opinion?
This, of course, is a central underlying analytic theme of our approach – that decision-makers use public opinion as an input because there is a real link between public opinion and sociopolitical outcomes. Put differently, if aligned with public opinion, decision-makers (candidates, politicians, policymakers) are more likely to win elections, push their policies and agendas forward, and win the messaging battle; but, if not, they won’t achieve success.
In Chapter 3, we explore the formation of attitudes. For the practitioner, understanding how humans process information and come to their opinions is critical in influencing public opinion. This was a crucial piece of the puzzle that was poorly understood until recently. First, we look at the multi-attribute model – a commonly used model by social psychologists and economists. In particular, we detail how attitudes can predict public opinion. We call this the static process or model. Second, we examine how information is cognitively assessed and how emotions shape public opinion. This we call the dynamic model.
In Chapter 4, we ask the simple question: Is public opinion stable or not? Many decision-makers have seen it as an ephemeral, whimsical thing; some social scientists do as well.
We come down squarely on the aggregate stability side of this debate: that public opinion is a stable, predictable phenomenon under most conditions. We argue that, for public opinion to be a decision input, it must be stable and predictable. However, we also detail under what conditions public opinion is more likely to change or be volatile. Ultimately, any analyst of public opinion must be able to identify when a movement in public opinion is real and when it is merely noise.
The Three-Hatted Pollster
In Parts II–IV, we show practically how to employ public opinion as a decision input. As noted earlier, we use the concept of the “the three-hatted pollster” to organize the primary ways in which pollsters use public opinion: (1) to assess; (2) to predict; and (3) to convince.
Part II: The Data Scientist
In Part II, The Data Scientist, we include three chapters (Chapters 5–7) on how to measure public opinion through a systematic discussion of the science of polling and data scientific assessment tools. The intent here is not to train the next generation of pollsters or data scientists but to equip the analyst with tools to assess the quality of their data sources.
In Chapter 5, we discuss the types of errors in polls. Specifically, we define concepts such as bias, error, the law of large numbers, and the margin of error. We employ a simplified version of total survey error – a commonly used bias assessment framework.Footnote 13
Our framework includes sampling and non-sampling error as the two dimensions that make up the concept. Sampling error is measured by the margin of error (MOE). Non-sampling error includes coverage bias, nonresponse bias, measurement error, and estimation error. Special attention will be given to coverage bias and estimation error. We find that they account for most of the polling misses. Specifically, we examine the challenges of estimating who is a likely voter – a special case of estimation error.
In Chapter 6, we apply our total survey error framework to the problem of a single poll. In this case, we assess an election tracking poll during the 2016 US presidential election. In July of that year, results from the Ipsos/Reuters election tracker were at odds with the market. Why was this the case? To answer this, we take the analyst step-by-step through the assessment process.
In Chapter 7, we use our framework to assess an aggregate of polls. We look at why the polls got the 2015 Greek referendum wrong. Was the problem down to sampling error or non-sampling error? A likely voter problem or coverage bias? Here, we determine through a process of elimination the most likely culprit. We also discuss concepts like spread, aggregation, and average absolute error (AAS).
Part III: The Fortune Teller
In Part III, The Fortune Teller, we include three chapters on using public opinion as an input to predict sociopolitical outcomes. The focus will be on rules of thumb and benchmarks, not on sophisticated statistical methods.
In Chapter 8, we detail the varying types of cognitive bias associated with prediction and discuss how to minimize them. We draw upon the cognitive psychology and behavioral economics literature, steeped in prediction misattribution.
We examine the weakness of expert prediction using the 2016 US presidential election as an example. Here, we discuss three broad classes of cognitive bias: (1) single input learning styles; (2) confirmation bias; and (3) binary thinking. We also look at herding and availability bias. Finally, we discuss methods to reduce such biases, including triangulation.
In Chapter 9, we take a special look at election prediction. This comprises a full review, from the simplest heuristics to more sophisticated aggregated election and poll-based forecasting models. Again, the treatment here is not concerned with statistical techniques but simple analytical frameworks, easily accessible to all. This chapter will also discuss combinatorial forecasting methods that mix, or triangulate, multiple sources. Again, the 2016 US presidential election will be our case study.
In Chapter 10, we look at a broad class of public opinion outcomes, such as approval ratings, referenda, impeachments, mass social protest, regime stability in nondemocratic societies, and big-picture scenario construction. Specifically, our focus will be on approval ratings and context-based approaches. Approval ratings both predict outcomes, like elections, and are a critical guide to engaging public opinion. Given their importance, the analyst needs to understand what impacts them. To do so, we provide a road map for the pollster. In terms of context analysis, pollsters are also required to paint the big picture for decision-makers. This can be done in multiple ways. We detail examples using main problem questions as well as multi-item indexes that capture underlying values and beliefs.
Part IV: The Spin Doctor
Part IV includes two chapters on how to message, engage, and move public opinion. Again, it will be a discussion of conceptual models and their application in practice.
In Chapter 11, we present conceptual frameworks to help the pollster engage public opinion. Here, we detail two frameworks: (1) the structural perspective and (2) the packaging perspective. In the structural perspective we discuss concepts like segmentation, message themes, messenger credibility, priority linkage, proof points, framing, and messenger–message fit. We also show how the multi-attribute model and its rankings and ratings can be redeployed to aid with messaging. In the packaging perspective, we show how emotions give special meaning to public opinion through words. The focus will be on concepts such as cuing, priming, hot cognition, packaging, and sticky messages.
In Chapter 12, we take the framework presented in Chapter 11 and apply it to a concrete case study. Here, we use both the structural and packaging perspectives to assess the 2022 Brazilian presidential election. Critically, we show the optimal communications strategy for both the Lula and Bolsonaro campaigns and then determine who did better and why – all this based on public opinion.
Concluding Thoughts
In Chapter 13, we review the primary points made throughout the book – the key takeaways. We also discuss some odds and ends not discussed in the preceding chapters. These include the pollster in society and reexamining the concept of public opinion as a decision input. We spend some time assessing non-survey inputs as a gauge of public opinion. Here, we think pollsters should have a framework for looking forward. Our data world is changing, but basic scientific concepts like bias, error, validation, and reliability are not.
