Representation bias as divergence

Adding reference metadata allows us to investigate the perspectivist nature of big historical data, and critically assess the extent to which missingness affects representativeness because it skews the digitised collection in particular ways. We can compare how observed aspects of a dataset diverge from a hypothetical scenario that encapsulates the properties and values set by the researcher. We focus on the political composition of the collections, analysing the distribution of newspaper allegiances, such as Liberal and Conservative.

Concretely, for a given attribute like the political leaning of a newspaper, we assess whether the distribution of these labels in sample data resembles a hypothetical distribution whose shape is anchored in a specific interpretation of representativeness. Imagine newspapers are classified as either Liberal or Conservative. If we adhere to the principle of equal representation – a scenario which requires each political orientation to appear with equal probability – we compute the divergence between the distribution of categories in the sample to one in which each category appears with equal probability. If the digitised collection only contains a few liberal papers, let us say 10%, we compute the bias-as-divergence by comparing the sample distribution $\overrightarrow {p}$ ([0.1, 0,9]) with a uniform one $\overrightarrow {q}$ ([0.5 0.5]). $\overrightarrow {p}$ and $\overrightarrow {q}$ are vectors with the first number indicating the probability of a label being Liberal, the second number shows the probability for the Conservative label.

To compare these distributions, we calculate the Jensen–Shannon Divergence (JSD) between two vectors $\overrightarrow {p}$ and $\overrightarrow {q}$ that represent the probability distribution of a variable in the sample and the hypothetical scenario, respectively,

$$ \begin{align*} \begin{aligned} JSD(\vec{p}, \vec{q}) = \frac{1}{2} \left[ KL\left(\vec{p} \bigg\| \frac{1}{2}(\vec{p} + \vec{q})\right) + KL\left(\vec{q} \bigg\| \frac{1}{2}(\vec{p} + \vec{q})\right) \right]. \end{aligned} \end{align*} $$

Because we model bias as a comparison between two probability distributions, JSD is generally a better choice than other common metrics such as cosine similarity or Euclidean distance. Unlike cosine or Euclidean metrics, JSD accounts for both the support (the number of non-zero values) and the distribution of probability mass. In other words, it captures the amount of information and uncertainty in the distribution. Moreover, JSD is symmetric, always finite, and bounded between 0 and 1. This ensures that it is more stable, interpretable, and robust, especially when distributions can have zero entries. JSD arose from information theory and reflects subtle shifts in distribution shape or entropy, making it an adequate and sensitive measure of bias.

In this article, the hypothetical distribution $\overrightarrow {q}$ can take different forms. We distinguish between three models of representativeness: proportional, equal or weighted.

Proportional representation assesses whether the sample is a proportional reflection of the population. In technical terms, we compute the divergence between the distribution of (political) labels in the digitised press and the historical press directories.

Equal representation computes the extent to which a variable’s probability distribution diverges from a uniform one (where each value occurs with equal probability).Footnote ⁷ In practice, we only apply this method to numerically or historically significant categories (not all labels are equal).

Weighted representation: Both proportional and equal representation can be misleading. Proportionality seems intuitively fair, but it ignores biases built into the newspaper corpus itself. The 19th-century press had strong commercial incentives to reflect the worldview of the propertied classes, and only a small number of newspapers were aimed specifically at working-class audiences even though they constituted a large majority of the Victorian population. By contrast, adhering to equal representation may radically over-represent some minority perspectives (depending on the categories incorporated). Such representation may be only hypothetical, practically impossible to achieve. Even new digitisation does not provide a solution as some perspectives will be absent from the archive. The weighted representation is a compromise between these two approaches. It attaches value to a substantive presence of minority voices but does not seek to put them on an equal footing with the dominant categories of Victorian newspapers. We use this measure only for diagnostic purposes, to see how far the corpus varies from this hypothetical distribution, but we also suggest that it could be a more practical way to guide future digitisation and rebalance skewed collections.

More formally, for this model of representation, we reweight the count $c_i$ of label $l_i$ (e.g., Radical) from a fixed set of labels L=Radical, Conservative, etc., using N, the total number of observations, and p, a penaliser value:

$$ \begin{align*} \begin{aligned} c_{reweighted} = \frac{c_i}{N + (c_i * p)}. \end{aligned} \end{align*} $$

We then convert these reweighted scores to a probability distribution. The penaliser value p determines the extent to which we oversample minority (and downsample majority) categories.

To give a concrete example, Figure 3 shows how each model or definition of representativeness produces different hypothetical scenarios (captured by the outcome vectors $\overrightarrow {q}$ ).Footnote ⁸ The black bars capture the distribution of the political labels in the press directories, used to assess whether a sample is proportionally representative. In this scenario, a sample without any Radical newspapers could still appear as statistically unbiased, notwithstanding the fact that it fails to include important strands of political argument within society that were marginalised because they were under- (or un-)represented in print. The white bars visualise a scenario where each category is equally represented. Situated between these two, in grey, is the re-weighted distribution, which tends to follow the proportional definition, but puts more probability on the minority categories by reducing the preponderance of the dominant categories.

Figure 3. Comparing hypothetical distributions.

Note: The figure shows how definitions of representativeness correspond with hypothetical distributions of political labels (nan = no category recorded).

Representativeness is a complex notion, and reducing it to a single, definite number infuses it with a certain rigidity and precision that is problematic. By introducing multiple hypothetical distributions, we offer a more elaborate and flexible diagnostic framework for establishing representatives (and bias) in historical collections based on varying assumptions. We developed (and implemented) the reweighted model to enable a certain parametrisation: it enables the adjustment of probabilities to match expectations about (the distribution of) the data. The choice of the penaliser depends, therefore, on the assumptions and expectations of the researchers. For our experiments, we gauge how putting more probability on categories such as Radical (which are historically important, but less present) might alter the results.

We treat hypothetical distributions mainly as a diagnostic tool to understand how our estimate of bias varies depending on our assumption of what a representative sample ought to look like. But it can serve different functions in other scenarios. For example, in the case of digitisation, the hypothetical distribution might serve as an ideal baseline against which we measure how adding data shapes the corpus in anticipated ways. Although bias often appears as a scourge, there is no single or definitive cure: it cannot be solved, and there is no single metric to guide us. Instead, we propose a versatile diagnostic framework which adopts different conceptual and ethical perspectives depending on the context, and which draws on the notion of “representativeness” (Chasalow and Levy Reference Chasalow and Levy2021).

Content analysis: Discriminating words and partisan bias

Historical biases also percolate through the language of the press. News, as it is recorded in the headlines and articles, is a representation of events, and not a direct reflection of reality itself. It involves curation instead of covering every possible topic. In this sense, it carries signals of its producers’ belief systems through the selection and representation of events. We therefore examine how demographic variables – primarily political allegiances – are associated with differences in language use: to what extent can we associate groups of newspapers with distinct priorities and patterns in language use?

For this content analysis, we use the NewsWords Data “Contextualized Word Counts”. NewsWords contains word count data derived from all types of newspaper items: articles, advertisements, notices, etc. This bag-of-words representation involves losing important distinctions when reducing texts to word counts; however, as an initial step toward outlining patterns of language use, these lexical patterns can nonetheless be revealing. To discover discursive biases in newspaper content, we estimate which words are discriminative and so distinguish, for example, the Conservative from the Liberal press, or cheap from expensive newspapers. By contrasting lexical patterns, we aim to uncover how newspapers created distinct representations of the world, often signalling strongly partisan or ideological stances.

Methods for modelling discriminative words often build on multinomial language models (Grimmer, Roberts, and Stewart Reference Grimmer, Roberts and Stewart2022). Monroe, Colaresi, and Quinn (Reference Monroe, Colaresi and Quinn2008) proposed the Fightin’ Words (FW) algorithm, a probabilistic model for feature selection that identifies the content of partisan conflict. Gentzkow, Shapiro, and Taddy (Reference Gentzkow, Shapiro and Taddy2019) extended this approach harnessing the power of Distributed Multinomial Regression (distrom) developed by (Taddy Reference Taddy2013). More recently, Kelly, Manela, and Moreira (Reference Kelly, Manela and Moreira2021) proposed a two-step approach. Their n-gram model (n-gramDMR) first estimates the inclusion of a phrase, and only subsequently models its repetition.

For the purposes of this article, we experimented extensively with the three variations on the multinomial language model, but ultimately opted for an adaptation of the FW algorithm. The main reasons were scalability and stability: running a full model on the complete dataset was significantly beyond the capacity of our (rather powerful) virtual machines. Secondly, we noted that the distrom and HurdleDMR models proved less reliable at identifying stable political differences in language use over time.

Tailoring FW to our research questions required significant adaptations. FW proposes a probabilistic model that factors in the varying levels of uncertainty associated with the different frequencies by which words appear. To obtain better estimates of a word’s distinctiveness Monroe, Colaresi, and Quinn (Reference Monroe, Colaresi and Quinn2008) propose a regularisation method that shrinks coefficients to a common value (Grimmer, Roberts, and Stewart Reference Grimmer, Roberts and Stewart2022), only words that are truly distinctive can deviate from this value. Rare words shrink towards the average.

Before explaining the FW algorithm in more detail, we point out some of its limitations. The principal aim is to foreground historically stable, partisan words by contrasting the content of different political categories of newspapers. Such a discriminative approach unfortunately reduces the complexity of the political press and removes important nuances by grouping newspapers from different places, price-points and interests together under one label, namely, Liberal, Conservative, Independent and Neutral. Moreover, the method assumes that frequencies point to meaningful (in our context “political” or “ideological”) differences between groups. If one group mentions X often, while another largely ignores X, our algorithm tends to attribute higher partisanship to X. However, because we rely on simple unigram frequencies (single tokens), we ignore the context in which words appear. If both groups use X often but disagree on its meaning and apply it to different contexts, X will not rank highly as a partisan feature.

While data-driven methods are useful for capturing macroscopic trends, we acknowledge that our processing of historical data is fundamentally different from how past audiences consumed information and the quantitative differences we observe are not self-evidently historically meaningful without further analysis and contextualisation.

We apply the FW algorithm in a binary setting, either contrasting two categories (e.g., Conservative versus Liberal) or one versus the rest (e.g., Conservative versus not Conservative). The regularised estimate of the probability that word j appears in document belonging to category k is:

$$ \begin{align*} \begin{aligned} \hat{u}_{jk} = \frac{W^{*}_{jk} + \alpha_j}{n_k + \sum_{j=1}^{J} \alpha_j}. \end{aligned} \end{align*} $$

$\alpha _j$ is used to smooth the probabilities. After computing this estimate of $\hat {u}_{jk}$ , we calculate the standardised log odds ratio, which compares the log odds between groups. For example, in the one versus all scenario, Liberal papers versus everything else.

$$ \begin{align*} \begin{aligned} \text{log odd ratio}_{kj} = \log \left( \frac{\mu_{kl}}{1 - \mu_{kj}} \right) - \log \left( \frac{\mu_{-kl}}{1 - \mu_{-kj}} \right). \end{aligned} \end{align*} $$

We use the standardised log odds ( $\Theta _{kj}$ ) as the measure of separation, which is obtained by dividing the log odds ratio by the square root of its variance, but adapt the procedure in important ways to align with our understanding of partisanship, which we (intuitively) defined as words that are:

1. 1. discriminative across categories,

2. 2. prevalent within categories,

3. 3. and exhibit both properties (i.e., 1 and 2) persistently over time.

To match the FW algorithm with this understanding of historical partisanship, we computed FW over time by only contrasting publications that appear within the same year. This produces a $\Theta _{kjt}$ score for each time step t. In some experiments, we would block selection on more attributes, for example, only comparing publications within a specific year and place, to account for systematic regional variation. This effectively introduces one more dimension in the equation.

Secondly, for inference, we computed $\Theta _{kjt}$ on subsamples of the data. Effectively, we create a sample containing 10% of the observations, in both $kt$ and $-kj$ (i.e., Liberal papers in 1846 versus all other papers published in that year), and calculate the standardised log odds ratio $\Theta _{kjts}$ for this section. We repeat this procedure n times, producing n different estimates. This helps us prioritise words that are prevalent within a category (i.e., those that tend to reappear as distinctive tokens in multiple subsamples).

Lastly, we compute a final score $p_{jk}$ for each word j, and use this to rank the complete vocabulary from most to least partisan. To foreground stable lexical markers of separation, we calculated $p_{jk}$ as

$$ \begin{align*} \begin{aligned} p_{jk} = \frac{\mathrm{mean}(\Theta_{kjts})}{1 + \mathrm{var}(\Theta_{kjts})}. \end{aligned} \end{align*} $$

This approach confers a higher score to words that, on average, have high distinctiveness values (across years and samples) but also low variance. This method is fast and straightforward to apply to a corpus with a lexicon containing 245,750 distinct words (including variants due to OCR errors) and more than 321,888 observations.

Missingness: What has been digitised?

Before we dig deeper, we sketch the BNA collection’s contours in broad strokes. The solid line in Figures 2 and 4 (top) shows the number of digitised titles by year, foregrounding the considerable temporal bias in the corpus: more data comes from the latter half of the 19th century. However, as a proportion of all published newspapers, the digitised corpus falls almost continuously from c.1860.

Figure 4. What has been digitised?

Note: a) Top: Total number of titles reported in the Newspaper Press Directories (dashed line) and number of digitised newspaper titles in the BNA (solid line); b) Bottom Left: The number of titles of Provincial (solid) and Metropolitan (dashed) reported in the Press Directories; c) Bottom Right show the digitised press as a proportion of the total number of titles for Provincial (solid) and Metropolitan (dashed) newspapers.

Overall, we estimate that around 20% of once printed British newspaper titles had been digitised by 2021. But this estimate hides quite substantive variation over time. The digital sample does not follow the trends of the larger newspaper landscape. It fails to reflect the steep rise in titles across the 1860s and 1870s, when the growth of the digitised sample is outpaced by the actual expansion of the press. Moreover, the digitised sample declines more sharply after 1900 than the number of titles listed in the directories. Press consolidation was accelerating in this period, and probably explains the blip in both series around 1897 (Lee Reference Lee1976, 133, 176), but the digital sample may also be registering the effect of tighter copyright restrictions for 20th-century publications. Generally, measured as a proportion of the landscape (in relative terms), the sample becomes smaller over time. This pattern clearly emerges in Figure 4 (bottom left), which exhibits a steep decline in the percentage of digitised titles over time. This trend holds for both the metropolitan (London) and provincial (including Scotland and Wales) newspapers. In general, a smaller portion of the metropolitan press has been digitised.Footnote ⁹

Divergence: Political biases in the digitised Victorian press

We now move our attention to representation bias with respect to the political orientation of newspaper titles. The directories record approximately 120 unique political allegiances, reflecting in part the freedom newspapers had to self-identify politically (one title, the London Morning Advertiser, returned five different political identifiers in just a few years, each time combining the term “independent” with other political terms). In addition, party names were themselves mutable, and many newspapers adjusted their allegiance over time. The Liberal Party was a coalition of pro-reform groupings, from Whigs to radicals, offering newspaper editors’ many possible qualifying labels and nuances. Similarly, after the Liberal split over Irish Home Rule in 1886, many Conservatives adopted the term “Unionist” to signal their alliance with the anti-Home Rule “Liberal Unionists,” while others chose to call themselves “Constitutionalist.” In the analyses that follow, we work with composite allegiances that take account of these historical specificities.

In broad terms, newspapers can be classified into five political blocks: Conservative, Liberal, Radical, Independent and Neutral. Alone or in combination, these five account for more than 83% of the returns (with a further 15% recording no allegiance). To measure bias, we simplified newspapers’ political allegiances to seven categories: Conservative, Liberal, Radical, Independent, and Neutral, Other,Footnote ¹⁰ and nan (“no label”). These collapse some of the fine-grained distinctions into larger categories, while retaining the main political distinctions (see Figure 5).Footnote ¹¹

Figure 5. Distribution of Political Labels: proportional distribution of political allegiances in the digitised press directories. Left: before reclassification (top 20 labels). Right: after simplification of the labels.

Turning to the distribution of these labels over time, Figure 6 visualises the shifting politics of the newspaper landscape. In terms of their share of the 19th-century press, Liberal newspapers show a steady decline, from almost 40% in the 1840s to only 25% in 1920. Conservative newspapers decline sharply in the 1850s, stabilise at around 20% of titles until 1890 and then register a modest increase. The category that gained the most ground over this period is the Independent press, from a rarity in the mid-19th century to the largest category of newspapers by the 1920s. If we amalgamate Independent and Neutral titles as the non-partisan press, they represent a third of titles from 1860, and were the largest grouping by the 1890s. This conflicts with the usual emphasis on party politics in accounts of the Victorian press. (See Lee (Reference Lee1976), Koss (Reference Koss1981), Jones (Reference Jones2016), and Hampton (Reference Hampton2004); with Hobbs (Reference Hobbs2018) as an important exception.) That said, as the preceding discussion suggests, there were also significant differences between Independent and Neutral newspapers, with the latter more resolutely non-political as well as non-party (an aspect of the contemporary distinction that is too often missed, e.g., Lee (Reference Lee1976)).

Figure 6. Proportion of newspaper titles by press directory political label over time.

When comparing the digitised press to the wider newspaper landscape, it appears that overtly political newspapers tend to be over-represented (in proportional terms). For almost the complete period, Liberal newspapers are over-represented, comprising approximately 10% more of the sample than the population (Figure 7). Conservative titles are also over-represented, but less markedly. For the neutral and independent press, the historical pattern generally points in the opposite direction, even though the differences between the digitised sample and the newspaper landscape widen or shrink at different moments. The neutral press is under-represented for the third quarter of the 19th century, while for independent titles, under-representation is greatest in the fourth quarter. Although in these cases the differences remain smaller, generally around 5 percentage points.

Figure 7. A timeline showing the proportion of newspapers by political leaning-based press directories and BNA.

Note: Dashed lines show the proportional presence of each leaning in the sample (BNA), while the dotted line represents the population (press directories). The top-left figure shows trends for Liberal newspapers (grey); the top-right Conservative newspapers (black). The bottom-left figure shows Neutral newspapers (grey); the bottom-right Independent titles (black).

As noted, the proportional scenario in which we measure how closely the sample is a proportional reflection of the larger landscape, is only one model of representativeness. Using the JSD, we can assess how the digitised press diverges from the other two models (equal and reweighted). To demonstrate how interpretations of bias-as-divergence may vary, we applied all three approaches to two different categorisations of the British political landscape:

1. 1. Coarse-grained, which comprises the four principal categories (Liberal, Conservative, Independent and Neutral).

2. 2. Fine-grained, including the minority category of Radical titles as well as papers which did not report any affiliation (nan).

By diagnosing a corpus using different categories and bias metrics, we avoid reducing bias to a single score. Instead, we treat it as a way of investigating or interrogating big data, which requires us to look at a collection from multiple angles. If data are oligoptic, so too are our bias measures.

The definitions of representation yield slightly different interpretations of how bias in the data changes over the 19th century. Emphasising proportionality, the timelines suggest a drop in representativeness in the 1860s that is reversed from 1870, climbing to a higher level again in the early 20th century (Figure 8). In other words, it suggests that during this period of rapid press expansion, the BNA becomes less representative of the total titles indexed in the press directories.

Figure 8. JSD scores over time. The Y-axis measures the divergence between the observed distribution of political labels and a hypothetical distribution, i.e., the extent to which the sample achieves proportional, equal or “reweighted” representation.

Note: The left figure focuses on the coarse-grained taxonomy, while the right uses the fine-grained, original categories.

However, when prioritising equal or reweighted representation, the scores suggest an overall reduction of bias; attention is more equally distributed over the principal political categories at the end of our period. Especially after 1900, the corpus is closer to an equal than a proportional representativeness. However, looking at the fine-grained taxonomy challenges this interpretation, and underlines the importance of including a wider array of partisan and non-partisan voices, not just the dominant ones. There is still evidence of declining bias, but this is now modest. The BNA is far from representative once we include minority voices such as the Radical press.

We inspect changes in the bias scores systematically by computing the contributions of each coarse-grained label to the divergence, and repeating this for the different types of bias (Figure 9). Again, the narrative tends to depend on what constitutes representativeness. Also in this case, we need to underline the importance of tackling bias from multiple perspectives.

Figure 9. Contribution of the four principal political labels to bias scores for each measure of representativeness (coarse-grained classification).

All measures largely agree on the over-representation of the partisan press at the expense of Independent and Neutral titles (regardless of whether we base our calculation on coarse or fine-grained classifications). But simultaneously, these figures reveal crucial distinctions. For example, they clearly disagree in their timing: from a proportional perspective, the over-representation of the Liberal press grows with time, while it declines in the other figure. Moreover, these bias measures give different weight to the under-representation of the non-partisan press. Equal and reweighted representation measures clearly signal the radical under-representation of Independent titles in the middle of the 19th century, and Neutral titles later on. The preceding analysis demonstrates how the ES enables scholars to switch between different models of representation. Each of these postulate a shape to which a sample ought to adhere, and thereby offer different measures of the representativeness of the digitised collection. Again, the goal here is not reduce bias to a number, but to better understand its oligoptic view.

We also looked more closely at the representation of minority voices, especially interested in the oppositional categories, such as Radical, Labour and Socialist, newspapers aimed to represent working-class, plebeian readers. Proportionally speaking, there is no overall anti-radical or plebian bias, but other models of representation suggest an under-representation of radical titles. In times of rapid democratisation of the press, the BNA increasingly underrepresents radical voices. Titles without an explicit political allegiance in the press directories, and often with minimal contextual information in the BL catalogue, are also less likely to be digitised. A detailed analysis is available in Appendix B.

Partisanship: Content bias in the Victorian press

While we have been discussing the Liberal or Conservative press, it remains unclear to what extent these categories reflect different discursive repertoires (in terms of newspapers’ printed content). In this section, we investigate differences in language by political allegiance to assess the extent and nature of Victorian newspapers’ partisanship. This helps us to assess what, for example, a Conservative bias implies in terms of content, but can also help us to understand whether, as suggested from their directory entries, nominally non-party-aligned newspapers could still be partisan on specific topics.

We discuss the main themes that emerge from a contrastive analysis. We first computed the most distinctive words for each category, and subsequently grouped these discriminative features in a bottom-up fashion. Clustering words ignores some of the complexity and ambiguity of language, but it clarifies strong lexical patterns in the data. We refrained from assigning words to a category if we deemed them too ambiguous or multi-valent.Footnote ¹²

We grouped the FW into 14 categories (see Appendix C). Clustering by semantic theme makes it easier to compare patterns across types of newspapers. Table 2 shows the top five most distinctive words by category in the top 200 most distinctive words (some clusters have no occurrences in the top 200, others fewer than five). The partisan cluster (1) confirms that Liberal newspapers used variants of “tory” to describe their Conservative opponents, while Conservative newspapers retaliated by talking about “radicals” and “agitators” (see Figure 10).

Table 2. Overview of the most distinctive words by newspapers’ political allegiances documented in the press directories and thematic clusters

Figure 10. Contrastive plot of distinctive Liberal and Conservative partisan words (cluster 1).

A party’s official name was also a distinguishing feature of partisan newspapers’ vocabulary (Figure 10). Other clear patterns include the monopoly of religious words associated with nonconformity in the Liberal press and with the Church of England for Conservative papers (cluster 3); and the stark differences in partisan newspapers’ distinctive languages of social description (cluster 8). The distinctive social vocabulary of Conservative newspapers leans heavily on Burke’s Peerage, quite probably including its distinctive use of the term “Society” to refer to the social elite, whereas the distinctive language of Liberal newspapers focused on working-class occupations, and on demotic abstractions (“people” and “public,” see Figure 11). The strong association between party allegiance and religious denomination (“Church” or “chapel”) is expected and confirms the method’s validity, but the strong class polarisation in newspapers’ social language is less obvious in an age when Conservative leaders were seeking to cultivate the “tory working man” (Lawrence Reference Lawrence2017; Quinault Reference Quinault1979). It suggests that their efforts may not have been widely amplified in the provincial Conservative press.

Figure 11. Contrastive plot of distinctive Liberal and Conservative social group words (cluster 8).

One sees a similar pattern for cluster 4 (Economy), where Liberal newspapers’ vocabulary is considerably more quotidian than that of their Conservative rivals. The most striking patterns for Independent newspapers are probably the absence of partisan and political words, the mix of nonconformist and Anglican words in their religious lexicon (cluster 3), and the importance of economy, agriculture and military words (4, 10 and 11). Independent newspapers’ distinctive agriculture words are more vernacular than those associated with the conservative press (“dairy,” “wethers” and “cowhouse”), and military terms are strongly associated with the local Volunteer movement. Both may offer clues to the distinctive readership of Independent newspapers. The same can be said of the Neutral newspapers, which are also notable for their large-scale and distinctive use of sporting terms (cluster 14, see Figure 12).

Figure 12. Distribution of thematic word clusters by newspapers’ political leaning.

Only the Conservative newspapers’ use of (Anglican) religious words rivals this distinctive feature of the Neutral press. Figure 12 also underlines the importance of political process words for Liberal newspapers and the distinctive economic language of Independent newspapers.

We can also harness this method to track changing patterns of word use over time. With FW, we can investigate how the distinctiveness of a set of words changes over time. Thus, we can explore the extent to which “tory,” “liberal,” etc. are discriminative of Liberal newspapers, and how this tends to vary over the 19th century. In Figure 13, we computed how discriminative the partisan words were for Liberal and Conservative newspapers, respectively. For each year, we aggregated the scores for all partisan words (cluster 1). Focusing specifically on the use of these words by these two classes of party-aligned newspapers, we see two periods of high and broadly equal emphasis on partisan words: before 1850 and between 1885 and 1895. There are also two periods of more unequal usage: 1850–1885, when Conservative newspapers put more emphasis on partisan words, and 1895–1920 when Liberal newspapers did so (Figure 13). Both register a significant drop in traditional partisan language during World War I, but for Conservative newspapers, this appears as the continuation of a trend begun in the 1900s. This decline suggests that partisan newspapers may have been becoming less overtly political in this period despite considerable polarisation within society (Lawrence Reference Lawrence2009). This trend requires further investigation, but may point to the commercial pressures that would ultimately lead most local newspapers to disavow party political allegiances by the mid-20th century.

Figure 13. Mean FW scores by year for all partisan words in the Liberal (grey) and Conservative (black) press.

Note: Vertical bars denote General Election years, and the dashed lines indicate a coalition government.

The discursive impact of partisanship was not constant and can be measured in different ways. Bias in sample composition and newspaper content may interact, adding to the complexity of understanding the effects of bias and missingness for users of digitised corpora. When conducting the analysis of newspapers’ distinctive words, it was striking that OCR errors occurred much more frequently in lists generated for Liberal and Neutral newspapers than for Conservative or Independent ones. The same pattern existed when we compared cheap newspapers with more expensive ones, suggesting that paper and ink quality may help explain these differences (see Appendix D). Further work is needed to understand the significance of this pattern, but it is a salutary reminder that digitised newspaper collections offer a highly skewed view of the past.

Digitisation, divergence and the origins of bias

Up to this point, we have mainly scrutinised how the British digitised newspaper collection changed over time, where “time” has meant year of publication. While the digitised press allows us to study the past, we should also remember it carries its own history, having emerged through a long and complex digitisation process now spanning more than two decades. In this section, we take a closer look at digitisation as a process, and ask: how did (the BNA sample) biases change over time as digitisation progressed? We show that selection bias has fluctuated and argue that researchers therefore need to understand its extent and nature for the specific version of the corpus they are accessing.

In the following analysis, we sorted the newspaper titles by their position in the digitisation sequence: from the first digitised title to the most recent.Footnote ¹³ This step assigns a rank to each title, ranging from 1 (first digitised newspaper) to n (the last). After sorting all titles by their digitisation order (or rank), we iterate through the collection, and traverse the data from 1 to n with stride s (here 25 titles). We start by computing the bias on the first “digitisation batch,” i.e., newspapers with rank between 1 and s (or 25 in our case); in the second iteration, we expand this batch to the papers with rank between 1 and 2s (or 50), etc. We thus increment the batch size at each step with stride s and compute the bias by comparing the distribution of newspapers’ political allegiance labels in the digitisation batch to a target distribution (see Figure 14). The latter distribution remains static at each iteration. Put simply, we measure how bias has changed as a function of incremental corpus growth (as new titles get added). The X-axis on Figure 14 shows the highest rank of each digitisation batch (in other words, 200 means that the corpus at this step comprises the first 200 newspapers).

Figure 14. Political bias as a function of digitisation order.

Note: Left: Overall bias measure, lines relate to different interpretations of representativeness. Right: Proportional bias measure for the four principal categories of newspaper affiliation. Shading broadly corresponds with JISC (dark grey) and Gale (light grey). Stride size is equal to 25 titles. Bias was computed on the finer-grained classification.

Overall, the growth of the corpus has reduced political bias, and especially from a proportional perspective, the data has become more representative (less so according to the equality measure). However, this is not a linear process in all cases. The digitisation timeline suggests that at some point in the process, stagnation in bias reduction appears. For example, if digitisation is assessed using the model of equal representation – giving each group equal presence at each stage – scores show a starker decline during the earlier phase of digitisation, slowing after around 550 titles. A similar pattern occurs for the proportional timeline, albeit the stagnation occurs somewhat later in the process.

The right panel of Figure 14 partly explains these rather abstract patterns, by plotting the political labels’ contribution to overall divergence as a function of corpus growth. We used the proportional representation as the target distribution. The initial over-representation of Conservative and subsequently Liberal newspapers declines over time but persists. The opposite applies to the Neutral and Independent press.

For Figure 15, we computed the contribution of each political label to the divergence for each “digitisation batch.” This analysis is slightly different from the preceding one. Instead of incrementally adding data, we defined a digitisation batch as the set of newspapers that fall within a specific sliding time window w. For example, if we set s to 25 and w to 100, the first digitisation batch comprises all titles with rank 1 to 100, the second contains titles with rank 25 to 125, etc. We then compared the distribution of political labels in each batch to the proportions of those observed in the directories.

Figure 15. Partial Kullback–Leibler values for each batch of 200 newspapers, looping through the corpus in digitisation order, with a step size of 25 newspapers.

Note: The shading corresponds with JISC (dark grey) and Gale (light grey). No shading reflects the Findmypast period.

Visualising the creation of the BNA in this way underscores how digitisation priorities have changed over time. Earlier efforts in particular seem to have focused on the partisan press, alternating between over-representing first Conservative and then Liberal newspapers. Something drastic changes after approximately one third of the corpus has been digitised. We observe a steep increase in preference given to Neutral newspapers, which were largely ignored in earlier stages. This contributes to driving bias scores lower, as seen in Figure 15. It also corresponds with our previous intuitions about how organisational priorities shape the contours of research data. Here, we can clearly trace the consequences of a transition from digital collections being constructed as research resources for academic users to a focus on the needs of a broader public concerned primarily with local and family histories (Shaw Reference Shaw2005).

These decisions have important implications for the political and other biases encoded in the data; biases that by their nature have been largely undocumented. Therefore, at different stages in the digitisation process, the partisan profile, and hence the social perspective, of the newspapers we can access, changes. Newspaper data are not oligoptic in a static way, but dynamically: their view on the past will always be partial, but shifts with the characteristics of the newspapers selected for scanning.

It is critical to point out that many of these sub-corpora live their own lives. The BNA contains the results of multiple digitisation projects led first by JISC and later by Gale, all of which continue to survive as separate (structured text) datasets that are available to researchers. However, even though the data has changed significantly in its political and other biases, these variations are not flagged to users and each provider tends to emphasise the scale and comprehensiveness of their data (Tolfo et al. Reference Tolfo, Vane, Beelen, Hosseini, Lawrence, Beavan, McDonough, Bunout, Ehrmann and Clavert2021). As more researchers depend on these datasets – not simply to identify specific pieces of evidence from a very large collection but also to analyse macro trends, and train language models – the importance of providing clear guidance about their content and biases only increases.