1Over the last decade, since the publication of James Surowiecki’s bestseller The Wisdom of Crowds, [1] the concept of collective wisdom has acquired a unique place in political theory, particularly in the Anglo-American world. [2]
2The concept of collective wisdom has two possible uses in political theory. For some, it provides a way to defend the merits of democracy in opposition to regimes where the people has no say in political decision-making. This contributes to a line of argument in democratic theory that is called epistemic democracy: democracy is preferable to other political regimes because of its epistemic performance. [3]
3For others, the wisdom of crowds is not a way to legitimize democracy but to improve it by moving to a representative democracy, from a democracy of experts and political professionals to a direct democracy in which the entire population has a say and legislates through continuous referenda. For this latter group, the argument that not only do numbers ensure the quality of collective decision-making, but also that diversity increases it, is a powerful one. In their view, the crowd is often characterized by a diversity of components, and it is this very diversity that enables a complementarity of viewpoints, interests, and knowledge. Combined with epistemic democracy, this school of thought terms itself epistemic diversity. Numerous schools of political thought have recently based, and continue to base, themselves on these elements of political theory in order to implement their action [4] or analysis. [5]
4One of the remarkable features of the idea of collective wisdom is that it goes against a certain doxa inherited in particular from the late eighteenth century, according to which the masses were considered to be ruled by their passions, with the resulting preference for a representative government made up of highly educated members. Collective wisdom thus provides a key argument in favor of direct democracy.
5The present article in no way aims to defend or critique the concepts of collective wisdom, epistemic democracy, or epistemic diversity. Instead I am interested in the way these concepts are supported through the utilization of mathematical theorems. These theorems are the product of research by Lu Hong and Scott Page. In this article I will focus on their work which, initially limited to resolving mathematical problems with algorithms, has gradually spread into the field of political theory. Their work has in particular produced three results that are frequently invoked in political theory.
6First, the “diversity trumps ability theorem” states that a diverse group of people is better than a group of best-performing individuals, which supports the relevance of a vote by the people. To understand the impact of this theorem, the context must be explained: consider a group of individuals faced with a decision to make (a political decision, for example: should they vote in favor of a particular law, or which public policy is preferable to another?). The theorem considers the ability of the group to assess the suitability of options, as it would when assessing public policies (this option meets objective 1 but not objective 2, this other option meets both, and so on).
7The group of decision makers includes individuals with greater ability than others, i.e. with greater understanding of the issue, either due to their empirical knowledge or their theoretical knowledge concerning the issue. It is therefore possible to form a subgroup of best-performing individuals, in which the ability of any member is greater than the ability of any other individual in the main group.
8The diversity trumps ability theorem states a surprising result: the best-performing subgroup can be outperformed by the collective performance of a lower ability group with maximum diversity. This means that if individuals with the greatest possible diversity are selected from the initial group, they form a group that outperforms the best-performing group on the issue. Thus, diversity trumps ability.
9Secondly, the “diversity prediction theorem” states that the greater the diversity the better the prediction, which justifies diversity and mixed groups. Let us consider the case of a group that has to predict a value (or predict a characteristic of an object, whether this is the weight of an object or the value of a solution to a problem). The best-performing individuals are no longer distinguished from the rest of the group, but the idea of performance assessment is retained. We can thus define the gap between the decision taken and the optimal decision as an error, in the mathematical sense of the term.
10Broadly speaking, the result states that the error of the group is equal to the average error of each individual subtracted from its diversity. The authors conclude that diversity reduces the risk of error. The direct consequence of this is the third result.
11The “crowd beats averages law” states that collective prediction is better than that of the individual average: a mathematical justification for democracy being better than autocracy and oligarchy. It is a direct consequence of the diversity prediction theorem, which states that the group is at least as good as its parts. One consequence of this law is that a collective decision is necessarily better than an individual decision. This ultimately means that the greater the diversity of the individuals involved in the exercise, the better the collective performance.
12These three mathematical results support the idea of the epistemic and beneficial effect of diversity, which more generally supports democracy and collective wisdom. But to understand this work, its significance, and use, it is necessary to define what is meant by “diversity”.
13Hong and Page are mathematicians and economists whose work derives from research into artificial intelligence. [6] This undoubtedly explains one of the great original features of their work: the fact that it does not position itself in relation to rational choice theory. They have no need to consider notions of choice or preference since they pose the question of collective choice as a problem of optimization: the only premise is that a better option exists; it remains only to find it. Decision-making is thus absorbed into problem-solving. As a result, even in their bibliographies there are no references whatsoever that suggest the authors are interested in taking a position in the debate on the application of rational choice theory. Instead, they have created new models from scratch and in total freedom.
14It is also from the field of artificial intelligence that the concept of diversity is derived in the authors’ first article on this topic, published in 1998. A lengthy paragraph, whose opening lines are shown here, explains:
What do we mean by diversity? Do we then mean race, profession, gender, or ideology? We mean all of these and yet none of these. Zenisms aside, to us diversity means differences in problem solvers’ perspectives and heuristics–variations in how people encode and search for solutions to problems. [7]
16The authors specify from the outset that their concept of diversity is not the same as that of other economists, who use the term “to describe heterogeneous preferences”. In a 2004 article, they refine their definition by referring to diversity in the mappings that individuals have in their heads to solve a given problem. They therefore explicitly oppose the common understanding of diversity that refers instead to “demographic characteristics, cultural identities and ethnicity”. [8]
17The definition is then expanded in Page’s 2007 book, which increasingly refers to “difference” (this is in fact part of the title of the book) in the broad sense of the term. [9] By the time of their 2012 article, diversity is based simply on the fact that “people within [the collection], or their models, must differ”. [10]
18As their concept of diversity has expanded, the authors have also expanded the field of application for their results: from solving mathematical problems to economics, then to the implementation of public policies relating to democracy and collective wisdom. Their results have therefore gradually taken on more general value and have in particular been utilized by researchers in political theory.
19In this article, I will present the authors’ main works, outlining the necessary hypotheses, models, and resulting mathematical statements. I will then describe the way in which these theorems are used and combined with reasoning and arguments, either as evidence to support a theoretical position or conversely as a fundamental element of an argument. Finally, I will demonstrate three points: that this work as a whole is neither well-constructed mathematically nor even consistent in terms of its argument; that the authors, using a form of mathematical rhetoric, do not demonstrate what they claim to demonstrate; and that, in addition, the use of these mathematical results in political theory is entirely inappropriate.
Lu Hong and Scott Page’s Models
20The two researchers have published copiously on this topic, but I will focus on three articles, published over a period of twelve years, which demonstrate three different and complementary approaches to modeling diversity.
21The first of these, published in 2001 in the Journal of Economic Theory, [11] presents an algorithmic resolution to a particular type of problem to which the best solution is sought. [12] The second, published in 2004 in Proceedings of the National Academy of Sciences, explores and supplements the method in the previous article and further demonstrates that diversity may outperform ability. [13] The “diversity trumps ability theorem” was born from these two articles.
22Finally, the third article was published in 2012 in the book Collective Wisdom, edited by Hélène Landemore and Jon Elster. [14] Here, the authors present calculations that are more theoretical than algorithmic and deduce from them, in a different way from the two previous articles, that diversity is a positive factor in the context of collective decision-making through two results: the “diversity prediction theorem” and the “crowd beats averages law”. The mathematical appendix (which follows, p. X)[CAT1] demonstrates how this result is calculated.
23The fundamental concepts on which these results are based are those of the individual, the collective (or group of individuals), diversity among the individuals who make up a collective, error (measured as the distance between the best solution according to an individual and the absolute best solution), and finally ability (with a best-performing individual defined as an individual whose error is lower than the others). The upshot of these results is to demonstrate that a collective characteristic (diversity) is more relevant to the quality of collective decision-making than an individual characteristic (ability), something which has considerable significance for political theory with potential applications in regard to expert committees, referenda, mixed groups, and even methods of producing social innovation.
24Essentially, in the works I analyze below, Hong and Page show that diversity has two epistemic properties. First, a diverse group outperforms a group of best-performing individuals. Second, the more diverse the group, the better its performance.
Diversity as Beneficial Complementarity
25In the 2001 article, the model is quite simple: faced with a problem to solve, a certain number of people seek the best solution among the potential solutions. [15] Each agent begins his/her research from a specific starting point. The diversity of agents is linked to the multiplicity of starting points that allow each one to begin his/her search for the best solution, a division of labor that makes it possible for agents to work in parallel and to cover all potential solutions.
26For example, an agent initially decides that the first solution considered solves 80% of the problem as it makes it possible to resolve some points, but is not optimal for one reason or another. This may be the case for a public policy: the government wants to find money to fund a major project and the potential solution studied is increasing taxes.
27This same agent then considers another fairly similar potential solution and attributes it a score that (s)he compares to the score of the previous solution. [16] In our example, this might again be increasing taxes, but to a lesser extent, with the gap funded by a reduction in state expenditure. If the score of this second solution is higher than that of the first, the agent provisionally retains the second and eliminates the first, or if not, retains the first and eliminates the second. Then (s)he continues to compare the solution retained to a third solution and so on. Finally, (s)he retains the best solution among all those explored. To obtain the best solution from all those tested by all agents, the scores of the best potential solution found by each agent are compared and the best of the best is retained.
Diversity Trumps Ability
A More Precise Heuristic
28In the method outlined above, there is no specific definition of the way in which an agent passes from one potential solution to another. To overcome this weakness, the 2004 article presents the series of potential solutions to be evaluated by each agent in a more systematic fashion, with no room for uncertainty. [17] Let us suppose that we have 100 potential solutions at our disposal. [18] By numbering all the potential solutions from 0 to 99, we can put them in a circle: [19]
29Consider an agent A. His/her path from one solution to another–i.e. his/her heuristic–is predefined. For example, agent A will always move from solution i to solution j by adding 12 (so here j = i + 12). Thus, if agent A begins by testing solution no. 0, (s)he will then test solutions 12, 24, 36, etc. In the first stage, (s)he compares the performance of solution no. 0 to solution no. 12 and retains the best one. Then (s)he compares this solution with no. 36, retaining only the best, and so on. Similarly, agent B for example has a heuristic of the type (35; 24). (S)he thus initially tests potential solution no. 1, then continues with solution no. 36, then no. 60, then no. 95, etc.
Figure 1. The Circle of Solutions
Figure 1. The Circle of Solutions
30For this system to function in our circle, it is essential to consider the number of solutions modulo, i.e. in our example, we must always have a number of solutions between 0 and 99. [20] Thus, agent B, after testing solution 94, will test solution 94 + 25 = 119, i.e. solution 19 (since 119 – 100 = 19).
Collaborative Agents
31In this model, agents do not seek the best solution at the same time: it is not until A has retained solution i as the best provisional solution that B begins from i and proceeds with his/her heuristic, then concludes that j is the best, and so on until the last agent. Once this first round of agents is complete, A restarts his/her heuristic again from the solution retained by the last agent and the process begins again until all the agents retain as the best solution the one from which they began: the best solution will thus have been found.
32In addition, the authors add an important point in regard to their previous article: they define “best-performing agents”. These agents are characterized by the performance of their heuristic: the ability of an agent is here modeled as his/her capacity to find the best appropriate solutions. The best-performing agents are therefore those with the highest average scores.
33The authors thus show that a group of best-performing agents can be beaten by a group whose members have less ability but are more diverse. To do so, they model the opposition between a group of best-performing agents and a group of agents selected at random, with random selection supposed to ensure diversity. They then carry out a computation to show that the group with the greatest diversity (the one in which agents were chosen randomly) systematically outperforms the group of best-performing agents. This is what the authors call the “diversity trumps ability theorem”. Hong and Page have tested this model in a series of computational experiments, and their results show that diversity indeed has a positive effect on the performance of collective decision-making.
A Statistical Model of Diversity
34The 2012 article summarizes the most important results of Page’s 2007 book. [21] The method changes radically, leaving behind practical solution algorithms for theoretical statistical calculations, which are presented in detail in the Mathematical Appendix and upon which I offer a commentary here for the reader unfamiliar with mathematical formulae.
35The first result presented is called the “diversity prediction theorem”: a mathematical formula that essentially means that the error of a group equals the average error of group individuals minus the diversity of group individuals. The other result is the “crowd beats averages law”, which states, again broadly speaking, that the error of the group is always less than the average error of group individuals. The consequence of this law is that on average the crowd outperforms the individual. In addition, the consequence of this theorem is that the more diversity increases, the more error is reduced.
36The detailed calculations are presented in the Mathematical Appendix, but the following example will make it possible to understand the mechanism more simply. [22] To illustrate this, let us take the example of a game of Battleship in which five teammates seek to determine the coordinates, termed θ, of an enemy ship. [23] The five proposed answers are termed r1 to r5. The average of these five answers is noted. Let us imagine that the value to find for θ is (6;7). If the five proposed answers have the coordinates (1;5), (2;4), (3;3), (8;2), (4;6), then the coordinates of their average will be (3.6;4).
Figure 2. The Diversity of Prediction
Figure 2. The Diversity of Prediction
37In our example, the average squared error between the proposed answers (the ri) and the correct answer (θ) equals 22.6, which we will term E. [24] In addition, the square of the distance between the average of the proposed answers (c) and the correct answer (θ) is equal to 14.76 and will be termed M. Finally, the average squared distances of the proposed answers from the average of the answers (c) is here equal to 7.84, which we will term D. This final figure thus represents the diversity of proposed answers. [25]
38These three values can be calculated independently from one another and we note that the second value is equal to the first minus the last:
3914.76 = 22.6 – 7.84.
40This can be expressed more generally as the equation: E = M – D. This is the mathematical formula for the diversity prediction theorem: the error of the group is equal to the average error of each minus the diversity of the group. [26]
41From this theorem, the authors immediately deduce another result, named the “crowd beats averages law”, which states that collective error is less than the average individual error. [27] This is a direct corollary of the previous result, since from the equation E = M – D, knowing that D has a positive value, we get: E ≤ M. [28]
42This can be verified by returning to the example outlined above: here we have 14.76 ≤ 22.6, which means that the group is closer to the truth than the average of each respondent. The authors then explain that diversity makes it possible to reduce the prediction error of a group, since with diversity (D) appearing on the right side of the equation and preceded with a “minus” sign, it is possible to reduce the average error (M) to come to a lower collective error (E): the more diversity increases, the more collective error is reduced. Diversity is therefore a factor of precision.
Use of the Theorems to Support an Argument
43Hong and Page’s theorems have been revisited in many journals dedicated to political theory, management, economics, econometrics, education, as well as by NASA, the US Geological Survey, and the Lawrence Berkeley National Laboratory, among many other prestigious institutions. At the time of writing, their article “Problem Solving by Heterogeneous Agents” has been cited 329 times in Google Scholar, “Groups of Diverse Problem Solvers Can Outperform Groups of High-Ability Problem Solvers” 672 times, and Page’s book The Difference 2,332 times.
44Their mathematical work on epistemic diversity has thus acquired significant importance in a number of fields, including political theory in particular. These mathematical theorems are used in a variety of ways in the rhetoric of political theory, but all of them use diversity in the very broad sense of the term (ethnic, cultural, social, and ideological), a meaning initially rejected by Hong and Page.
Invoking Established Evidence
45For some authors, invoking these theorems is a way of providing evidence for the ideas they put forward, as if to reassure the reader with a scientific guarantee: the mathematical result is stated as an inarguable truth in order to establish its user’s argument. In such cases, reference to the mathematical work is very brief and in the majority of instances comes at the end of a paragraph in order to eliminate any possible questioning. Authors do not necessarily require detailed knowledge of Hong and Page’s work to be able to use it in this way.
46This is the case for Robert Putnam who, at the end of a paragraph in which he vaunts the merits of diversity and of immigration in particular, briefly concludes: [29]
Scott Page (2007) has powerfully summarized evidence that diversity (especially intellectual diversity) produces much better, faster problem-solving. [30]
48It is interesting to note that Putnam, despite his great intellectual renown and authority, believes it is beneficial to use Page to support his argument at the end of a sentence, despite the fact that he uses none of the results in the rest of his article. The “power” with which Putnam credits Page’s evidence makes him an essential author, even for an individual at the forefront of political science.
49Similarly, Adrian Vermeule uses this mathematical work as support at the beginning of a definition of diversity:
The standard theory, which emphasizes the judges’ competence, also overlooks that under a robust range of conditions “diversity trumps ability“ (Page 2007). [31]
51Comparable use is made for example by Richard Florida, well-known for his work on the so-called creative class, who has no hesitation in drawing on the work of Hong and Page in several articles, including explaining that “Page (2007) provides the basis for a general economic theory of human diversity and economic outcomes”. [32]
52It is also interesting to note that Florida states that Page “finds” results relating to diversity, while Florida himself “argues” in favor of diversity: this perfectly illustrates the weight of their work which, as it comprises mathematical results, enjoys scientific credibility. [33]
In-Depth Use in Order to Model Situations
53Other authors have attempted to see how these models can be applied to the core of the situations they study. They thus dedicate paragraphs, pages, or even entire articles to Hong and Page’s mathematical results: without them, their articles would be considerably weaker.
54To justify the decentralization of political bodies, for example, Gillian Hadfield and Barry Weingast invoke the mathematical work of Hong and Page at a very early stage, outlining the usefulness of their model as well as its underlying concepts. [34] The authors thus, from the beginning of their article, describe their own concept of idiosyncrasy, the essence of the entire publication, “not as a form of odd or unusual preferences but rather as a source of value-generating diversity in an economy (Hong and Page 2001)”. [35]
55This is also the case for Didier Caluwaerts and Juan E. Ugarriza, who base their experimental evidence of the merits of diversity on Hong and Page’s method of randomization and conclude from this that democracy outperforms ability. [36] And also Cedric Herring, who uses Hong and Page’s work as a theoretical framework for comparing experimental results, effectively showing that the issue of diversity cannot be considered without their results. [37]
56Others use this mathematical work as a pillar of epistemic democracy. This is the case for Josiah Ober, whose work largely focuses on proving that classical Athens is an example of collective wisdom and epistemic democracy. Here are the opening lines of his 2012 article: [38]
Lu Hong and Scott Page [...] offer a formal model of collective wisdom [...] The city-state of Athens [...] is a case study of a participatory epistemic democracy: an intensively studied historical example of a community whose remarkable success can, at least in part, be explained by Hong and Page’s two factors of sophistication and diversity. [...] While the Athenian case cannot, in and of itself, prove the general validity of Hong and Page’s model, it may offer some insight into how, in the real world, increased sophistication and sustained diversity of participants produce positive results over time.
58Although the aim of Ober’s article is to show that the Athenian example is a success to be emulated, the opening words offer mathematical models as an element of comparison (as a benchmark, we might say).
59In her article entitled “Why the Many are Smarter than the Few and Why it Matters”, [39] Hélène Landemore presents the foundations of epistemic democracy and proposes a kind of theorem of numbers (the crowd possesses wisdom, whether it is diverse or not) to generalize Hong and Page’s theorem of diversity. [40] Half of her article is dedicated to their work:
This paper aims to accomplish two things. One is to present in a condensed form, and defend, the theoretical connection that I argue exists between the phenomenon of “collective intelligence“ or “collective wisdom“ and the principle of democratic collective decision-making. On my view, the reason why the many can be expected to be smarter than the few is because of a plausible correlation between inclusive decision-making and the presence of an ingredient recently shown to be key to the emergence of collective intelligence, namely “cognitive diversity“ (Hong and Page 2001, 2004 and 2009; Page 2007). [41]
61She leans directly on their results in multiple lengthy passages, with the objective of generalizing them, moving from the “diversity trumps ability theorem” to what she calls a “numbers trumps ability theorem”. In particular, she applies their work to the case of the famous movie “Twelve Angry Men”. Hong and Page are at the heart of her article: so much so that, excluding the bibliography, Page’s name is cited no less than fifteen times.
62Similarly, in another article, Landemore cites their work as the cornerstone of her argument: [42]
According to Hong and Page’s Diversity Trumps Ability Theorem for example, under certain plausible conditions, a diverse sample of moderately competent individuals will outperform a group of the most competent individuals (Hong & Page, 2004). [...] That argument has been carried over from groups of problem-solvers in business and practical matters to democratically deliberating groups in politics (e.g. Anderson 2006, Landemore 2007). [43]
64In a 2006 article entitled “The Epistemology of Democracy”, Elizabeth Anderson examines the three mathematical models of epistemic democracy, including the Condorcet jury theorem and the diversity trumps ability theorem. [44] She thus explains:
The Diversity Trumps Ability Theorem helps solve some of the deficiencies of the Condorcet Jury Theorem. [...] [It] supports the claim that democracy, which allows everyone to have a hand in collective problem solving, is epistemically superior to technocracy, or rule by experts. [...] First, [it] explicitly represents the epistemic diversity of citizen inputs into democratic decision-making as an epistemic asset. Second, [it] models some of the epistemic functions of citizens’ associations and political parties. [45]
66Hong and Page’s theorem is thus posited to significantly improve that of Condorcet, which itself provides the basis for the whole school of epistemic democracy, since while Condorcet’s jury theorem is based on numbers, Hong and Page refine this by focusing on diversity. In addition, from the performance of this theorem Anderson draws the conclusion that democracy is better than autocracy. Hong and Page, alone, thus justify democracy as the best possible political system.
67In these articles, the use of mathematical theorems is central either as an argument to prove the defended thesis, as a starting point for reflection, or finally as an intermediary point of reasoning.
Discussion of Hong and Page’s Results
68Hong and Page’s results raise problems, however: diversity and ability are modeled in a surprising manner, at odds with the usual shape of these concepts. Worse, diversity is in reality completely absent from their algorithmic models–the “diversity trumps ability theorem” therefore has a very limited meaning. In addition, the “diversity prediction theorem”, obtained from their statistical model, is a result well-known to anyone familiar with probability and the way they interpret it is erroneous. Finally, even if these results were correct, they would be strictly inapplicable to political theory; at best, they may be used to model treasure hunt activities and collective measurement.
Modeling Choices that Depart from Common Ideas
Diversity and Ability
69The “diversity trumps ability theorem” states that a group of best-performing agents can make decisions that are less good than a diverse group. To form this latter group, the authors select agents at random. Unfortunately, this method in no way ensures group diversity.
70Let us imagine a population that is relatively homogeneous apart from a few individuals, as is the case in the diagram developed above: r1, r2, r3 and r5 form a fairly homogeneous group while r4 is relatively far away from the rest. If, like the authors, we randomly select two individuals from the five to form a subgroup, there is a 60% chance of forming a very homogeneous group. [46] In reality, to maximize the diversity of our subgroup, we must begin by taking r4 and then the one furthest away from it (r1 in this case). [47] Not only is there a general difference between what the authors call “diversity” and the common idea of this concept, but again the way in which the authors form their supposedly diverse subgroup in no way ensures its diversity (even in the sense used by the authors).
71But they also form the group of best-performing agents in a way that may provoke skepticism: they define the best-performing by the quality of their heuristic, modeled on the average score obtained by each agent. Let us imagine an agent A who is asked to evaluate five potential solutions with the scores 10%, 15%, 5%, 100%, and 8%. [48] The average of his/her scores is equal to 28%. Now let us imagine an agent B, also asked to evaluate five potential solutions with the scores 70%, 65%, 75%, 80%, and 85%. The average of his/her scores is equal to 75%. According to the authors’ model, agent B is therefore much better than agent A.
72But in reality, an agent’s performance in such a context is measured by his/her capacity to find the best possible solution. In fact, each agent, in the authors’ algorithm, retains only the best solution among those tested. The performance of an agent can therefore be measured not by the average of the scores tested but by the maximum score of all solutions tested. In our example, unlike agent B, agent A succeeded in finding the optimal solution: the one with the score of 100%. Agent A is therefore a true best-performing agent, but one lost by the authors when forming their group of best-performing agents. Thus, their “best-performing agents” are no such thing. The terms “diversity” and “ability” in the “diversity trumps ability theorem” turn out to be entirely misleading, as do the apparently satisfactory results of the computer simulations.
73If the challenger subgroup formed by the authors ultimately outperforms the “best-performing agent” group, it is partly because the “best-performing agent” group is poorly formed and partly because randomization improves algorithms, often considerably, something of which the authors appear to be unaware. This specific criticism of randomization has been raised by Abigail Thompson, [49] to which Page has responded with a very unusual defense of a theorem, arguing that his models “usually” work but not “always”. [50] He thus dodges the issue and does not respond to Thompson’s essential criticism.
74But my criticism seeks to be more systematic than Thompson’s, which focuses on only one of the authors’ articles and raises only specific mathematical points; here I propose the most comprehensive possible study of their mathematical failings and modeling choices and the relevance of these models for political theory applications. [51]
75What is more, even if the models of diversity and ability were well-constructed, one issue still remains entirely absent from the authors’ discussion: what happens if the best-performing agent group is more diverse than the public? Better still: why not seek to maximize the diversity of the best-performing agent group, whose evaluation might perhaps be better than that of an equally diverse lower ability group? In any event, Hong and Page weaken the concepts of ability and diversity, “forcing” their model to favor their argument.
A Heuristic that Denies Diversity
76Beyond the fact that the model of diversity is at odds with the commonly shared idea of diversity, meaning a necessarily weaker interpretation of the stated theorem, diversity is in fact absent from Hong and Page’s work. The authors use several agents to scan the entire circle of potential solutions and evaluate them one after the other, but in reality all the agents work in the same way, with heuristics that are numbered differently, but are identical in terms of conception. These are not different human heuristics but the same heuristic developed by several automatons. We cannot therefore speak of diversity in these models.
77Furthermore, even if agents are modeled as systematic processes, there is a simpler solution: using just one agent whose heuristic consists of beginning from solution no. 0 then increasing each time by p, p being a number that is not a factor of n (with n representing the number of available solutions). Thus, the unique agent will test solutions 0, p, 2p, 3p, and so on. As p is not a factor of n, it is certain that (s)he will test them all. [52] This demonstrates that Hong and Page’s model is equivalent to a model in which only one agent exists; speaking of diversity is thus incorrect.
78Thus, when Putnam, Florida, Herring, and Vermeule speak of diversity in the sense of diversity of origins, gender, sexual orientation, and social class, there is absolutely no reason for them to cite the work of Hong and Page, and it is equally difficult to understand how Ober can take up this work to support the performance, in collective decision-making, of diversity in the broad sense of the term.
Poor Mathematical Inference
79The other major result, the “diversity prediction theorem”, generated by the authors from statistical reasoning, is in fact well-known: as the König-Huygens formula, which dates. . . from the seventeenth century! [53] This result can be easily obtained and any beginner level student in probability or statistics is aware of it. [54]
80Written in the form E = M – D, it immediately suggests that diversity D reduces error E: as diversity D is preceded by a “minus” sign, it is quite natural to think that the more it increases, the more E reduces. Thus, the authors immediately ask: “How can we ensure diverse predictions?” [55]
81D would effectively reduce E if M and D were independent, i.e. if change in D did not mean variation in M. Yet M and D depend on the same variable (total responses) and it is easy to find situations in which the increase in diversity D is naturally accompanied by an increase in M, and thus error E. I propose a counterexample in the Mathematical Appendix that illustrates such a case and demonstrates that the result obtained by Hong and Page is erroneous. In effect, and contrary to the conclusion of the authors, diversity does not reduce collective error. This result, in addition to having been known for several centuries, is poorly interpreted to the point of becoming false.
The Irrelevance of these Models for Political Theory
82Even if the authors’ results were well-constructed and correct from a mathematical point of view, would they be broadly applicable?
Between Treasure Hunt and Collective Measurement
83Beyond the fact that modeling “agents” as algorithmic processes is fairly inapplicable to human beings, a problem arises: the potential solutions to be examined by the agents are of a finite number and even formed independently of the problem. [56] It is therefore strictly impossible to invent a new solution: innovation, however minor, is not considered in this model. Thus, when Hadfield and Weingast model diversity to mean the new ideas that people may generate, they are wrong to draw upon Hong and Page’s models.
84In addition, the fact of having a solution predefined independently of the problem, a truth external to the agents searching for it, is entirely contrary to the essence of decision-making, particularly of a collective nature. Decision-making is a matter of preference rather than truth: is it preferable to favor lower unemployment, commercial competitiveness, or the happiness of citizens? The question of purpose is absent from Hong and Page’s concerns. The application of the “diversity trumps ability theorem” to political theory is therefore beside the point.
85On the other hand, the algorithmic model, and thus the corresponding heuristic, is entirely appropriate for a treasure hunt: the agents are all equally capable of recognizing the treasure (the solution with the maximum score) and the treasure exists independently of the problem represented by the search for it. In addition, the statistical modeling would be perfect for modeling a collective measurement activity: e.g. the object to determine is the weight of a cow and passers-by are asked to provide an estimate. [57]
An Inarguably Shared Scoring System
86The most surprising element of their algorithmic approach is the fact that all the agents score the different potential solutions in the same way: the scoring system is never in question. Given the strong “jury” effect in competitions and exams, which means that giving the same answer to the same question posed by two different juries often produces different, and even very variable scores, this hypothesis would appear rather unrealistic. And in collective decision-making, particularly in regard to politics, the idea of everyone giving the same score to a potential solution is highly improbable.
87It is remarkable that the authors do not make more of this issue. This is because they make a clear epistemological choice: entirely removed from rational choice theory or any consideration of the concept of preferences, they pose the question of collective choice as a problem of optimization in which, reduced to an algorithmic or statistical analysis, collective decision-making no longer has anything to do with decisions. Therefore, neither these issues, nor the determination of objectives, nor the problem of the indeterminacy of the future come into play as it is simply a case of determining a path toward a solution that is predefined, recognized to be such, and certain.
88This is due to the fact that the authors were initially interested in mathematical-type problems, which are problems that can be evaluated in a binary fashion: true or false. It was from this perspective that they studied problems of an economic nature then crept into the field of political theory with collective decision-making and collective wisdom. [58] Hong and Page’s algorithmic articles simply attempt to resolve a problem of optimization by using a slight variation of the very well-known “divide and conquer” method, which consists of operating several similar algorithms in parallel across subfields rather than one single algorithm across the entire field of study.
Conclusion: Between Authority and Pertinence of a Mathematical Model
89I would recall that my objective is not to criticize the concept of collective wisdom or diversity, but rather the way in which authors in political theory have utilized the mathematical work of Hong and Page to argue in favor of the relevance of these concepts. Neither do I criticize the authors for eliding many of the elements demonstrated by experts on collective choice, such as cognitive bias, the complexities of communication between individuals within a group, including the mechanisms that can make a group persuade itself of an error, and the impact of leadership. It is normal, for a first model, to be forced to gloss over certain practical difficulties. But even from the most abstract and theoretical point of view possible, my examination makes it possible to conclude that there is nothing here for political theory: the two authors have generalized models limited to the field of mathematical problems, and I believe that this generalization is not only poorly constructed in terms of their modeling choices, but also erroneous from a mathematical point of view and inapplicable to political theory. And yet this work has had some success. This is undoubtedly because the implementation of mathematical models gives scientific assurance to a political idea that is often viewed as appealing. In conclusion, let us consider how thinkers in political theory might avoid this trap.
90First, by having the mathematical ability to evaluate the construction of these models–though I note that the authors have succeeded in passing the filter of very serious journals in which mathematicians frequently publish. In the absence of this ability, we must be wary of the transposition of objects and methods from one field to another: in particular, if authors had examined more carefully the concept of diversity as conceived in Hong and Page’s models, they would have understood quite quickly that they were not speaking of the same thing. This is precisely the case with Jon Elster, who did not fall into the trap, despite not undertaking the mathematical assessment necessary for a detailed analysis of their work. Elster simply recognizes its impracticability:
Scott Page has done much to bring formal rigor to the study of diversity. Although I have tried, I have not been able to link his models to actual processes of collective decision-making. [59]
92It would therefore seem that it is possible to be interested, or even attracted, by these mathematical models without misusing them. [60]
Mathematical Appendix
93Here I present a brief demonstration of the König-Huygens formula as rediscovered by Hong and Page, as well as a counterexample showing that their interpretation of this result is incorrect.
Demonstration of the König-Huygens Formula
94In comparison to the algorithmic models, i represents an agent rather than a response, with each agent i now proposing no more than a single response labeled ri. We can show all of these responses r1,..., ri,..., rn as a scatter plot, and score all of the responses.
95We then determine the squared error of individual number i as being the square of the distance between his/her response and the correct response: (ri – θ)2. We can then easily define the average squared error of all the n agents, termed SqE(r), as the average of the individual squared errors: SqE(r) = 
. Let us specify that the symbol 
means that we add up all of the elements (ri – θ)2, with i varying from 1 to n. We can then write in an expanded form:
96Furthermore, we can define collective prediction c as the average of individual predictions: 
We can then define the squared error between the collective prediction and the solution to be found as follows: SqE(c) = (c – θ)2.
97Finally, diversity prediction is defined as the variance in individual responses, i.e. the square of the average distance of individual responses from the average of individual responses, i.e. the square of the average distance of individual responses from the collective prediction: 
98We can then write:
99Thus:
100Which can be rewritten as:
101Which can be simplified to:
102Or in other words:
103SqE(c)= SqE(r) – PDiv(r)
104This is the result of Hong and Page’s so-called “diversity prediction theorem”, which is expressed thus: “the squared error of the collective prediction equals the average squared error minus the predictive diversity”. This result is a simple application to a particular case of the well-known König-Huygens formula.
Counterexample to the “Diversity Prediction Theorem“
105The following example, in comparison to the example developed within the body of the article, shows that increase in diversity does not necessarily mean a reduction in the squared error.
Figure 3. Diversity Does Not Guarantee Accuracy
Figure 3. Diversity Does Not Guarantee Accuracy
106Here the five respondents have proposed estimates that are more diverse than before, as the value of PDiv(r) is now equal to 24, compared to 7.84 in the example developed within the article. Through calculation we can establish the value of SqE(r) as 44.8, compared to 22.6 previously: this means that the individual errors are on average greater than before. As for the value of SqE(c), it is now equal to 20.8 compared to 14.76 before: the collective error has also increased.
107We thus perceive that collective error increases despite the increase in diversity, which disproves the authors’ conclusions. This arises from a simple matter: in the equation SqE(c) = SqE(r) – PDiv(r), increasing PDiv(r) naturally increases SqE(r) as these two terms are not independent from one another: when one increases, so does the other. This derives from the fact that both depend on the same variable (the total responses proposed by the agents). Ultimately, SqE(c) also increases.
Notes
-
[1]
James Surowiecki, The Wisdom of Crowds, New York, Doubleday, 2004.
-
[2]
To give a rough indication of its impact on contemporary research, this book has been cited in nearly 8,000 publications in Google Scholar.
-
[3]
The term epistemic indicates that a democracy is able to make better decisions because it better allows for the formation of collective knowledge. The other basis for democracy is the Condorcet jury theorem.
-
[4]
Participatory democracy in France, the Indignados movement, deliberative polls on education policy in Northern Ireland, various citizens’ movements around the world, and so on.
-
[5]
Cf. a number of commentaries on attempted Icelandic constitutional reform, as well as the creation of numerous think tanks such as Issues Deliberation Australia/America and Participedia.
-
[6]
Scott Page has a B.A. in Mathematics and a Ph.D. in Managerial Economics and teaches complex systems, political science, and economics at the University of Michigan. Lu Hong has a B.A. in Mathematics and a Ph.D. in Economics and teaches at Loyola University in Chicago.
-
[7]
Lu Hong and Scott Page, “Diversity and Optimality”, Research in Economics 98-08-077c, Santa Fe Institute (August 1998).
-
[8]
Lu Hong and Scott E. Page, “Groups of Diverse Problem Solvers Can Outperform Groups of High-Ability Problem Solvers”, Proceedings of the National Academy of Sciences in the United States of America (PNAS), 101(46), 2004, 16, 385. Online
-
[9]
Scott E. Page, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies, Princeton, Princeton University Press, 2007.
-
[10]
Lu Hong and Scott Page, “Some Microfoundations of Collective Wisdom”, in Hélène Landemore and Jon Elster (eds), Collective Wisdom: Principles and Mechanisms, Cambridge, Cambridge University Press, 2012, p. 57.
-
[11]
Lu Hong and Scott Page, “Problem Solving by Heterogeneous Agents”, Journal of Economic Theory, 97, 2001, 123-63.
-
[12]
An algorithm is a general method consisting of successive stages that must take place in a specific order in order to solve a problem. A recipe can be seen as a kind of algorithm, as can the so-called “hangman” method of long division taught in elementary school.
-
[13]
Hong and Page, “Groups of Diverse Problem Solvers”, 16, 385-16, 389.
-
[14]
Hong and Page, “Some Microfoundations of Collective Wisdom”, pp. 56-71.
-
[15]
Hong and Page, “Problem Solving by Heterogeneous Agents”.
-
[16]
Here the authors use the topological concept of neighborhood.
-
[17]
Hong and Page, “Groups of Diverse Problem Solvers”.
-
[18]
Here I present the model using the example of 100 solutions; the authors reason directly with n solutions.
-
[19]
Note that there are indeed 100 numbers from 0 to 99.
-
[20]
To formalize this mathematically, we can say that we reason in terms of the quotient group (Z/nZ, +), where n represents the number of potential solutions.
-
[21]
Hong and Page, “Some Microfoundations of Collective Wisdom”.
-
[22]
This applied example makes it possible to reason with randomly selected values in order to illustrate the authors’ proposed theory.
-
[23]
Coordinates are defined by two numbers: the x coordinate for the horizontal axis and the y coordinate for the vertical axis. Thus the point with the coordinates (6;7) is positioned 6 units to the right and 7 units up.
-
[24]
This means that, on the diagram, the distance between the answers ri and the correct answer (?) is equal to the square root of 22.6, i.e. 4.75. The same goes for the two other calculations.
-
[25]
If all the answers were identical, its value would be zero.
-
[26]
Strictly speaking: the squared error of the group is equal to the average squared error of the individuals minus the diversity of the group.Online
-
[27]
To be more precise: the squared collective prediction error is less than or equal to the average squared error of the individuals in the group.
-
[28]
D is a sum of squared numbers, so is itself positive.
-
[29]
Robert Putnam, “E Pluribus Unum: Diversity and Community in the Twenty-First Century: The 2006 Johan Skytte Prize Lecture”, Scandinavian Political Studies, 30(2), 2007, 137-74.
-
[30]
My emphasis.Online
-
[31]
Adrian Vermeule, “Collective Wisdom and Institutional Design”, in Landemore and Elster (eds), Collective Wisdom, pp. 338-67.Online
-
[32]
Charlotta Mellander and Richard Florida, “The Creative Class or Human Capital? Explaining Regional Development in Sweden”, Toronto, Martin Prosperity Institute, Joseph L. Rotman School of Management, University of Toronto, 2006, http://creativeclass.com/rfcgdb/articles/The_Creative_Class_or_Human_Capital.pdf, last accessed 19 March 2018.
-
[33]
Richard Florida, Charlotta Mellander, and Kevin Stolarick, “Inside the Black Box of Regional Development”, Journal of Economic Geography, 8(5), 2008, 615-49.Online
-
[34]
Gillian Hadfield and Barry Weingast, “What is Law? A Coordination Model of the Characteristics of Legal Order”, The Journal of Legal Analysis, 4, 2012, 471-514.
-
[35]
Hadfield and Weingast, “What is Law?”, 478.
-
[36]
Didier Caluwaerts and Juan E. Ugarriza, “Favorable Conditions to Epistemic Validity in Deliberative Experiments: A Methodological Assessment”, Journal of Public Deliberation, 8(1), 2012, https://www.publicdeliberation.net/jpd/vol8/iss1/art6, last accessed 19 March 2018.
-
[37]
Cedric Herring, “Does Diversity Pay? Race, Gender, and the Business Case for Diversity”, American Sociological Review, 74(2), 2009, 208-24.Online
-
[38]
Josiah Ober, “Epistemic Democracy in Classical Athens: Sophistication, Diversity and Innovation”, in Landemore and Elster (eds), Collective Wisdom, pp. 118-47.
-
[39]
Hélène Landemore, “Why the Many Are Smarter than the Few and Why It Matters”, Journal of Public Deliberation, 8(1), 2012, https://www.publicdeliberation.net/jpd/vol8/iss1/art7, last accessed 19 March 2018.
-
[40]
See also: Hélène Landemore, “Collective Wisdom: Old and New”, in Landemore and Elster (eds), Collective Wisdom, pp. 1-20; “Inclusive Constitution-Making: The Icelandic Experiment”, Journal of Political Philosophy, 23(2), 2015, 166-91; and Landemore and Elster (eds), Collective Wisdom.
-
[41]
Landemore, “Why the Many are Smarter than the Few”, 2.
-
[42]
Hélène Landemore and Hugo Mercier, “Reasoning Is for Arguing: Understanding the Successes and Failures of Deliberation”, Political Psychology, 33(2), 2012, 243-58.
-
[43]
This is the author’s doctoral thesis.
-
[44]
Elizabeth Anderson, “The Epistemology of Democracy”, Episteme, 3(1-2), 2006, 8-22.
-
[45]
Anderson, “The Epistemology of Democracy”, 12.
-
[46]
That is, not including r4. In effect, there is a four-fifths chance of not landing on the first individual and three-quarters for the second.
-
[47]
Which consists of maximizing the variance of the subpopulation.
-
[48]
This means that the first solution solves 10% of the problem, the second 15%, etc.
-
[49]
Abigail Thompson, “Does Diversity Trump Ability? An Example of the Misuse of Mathematics in Social Sciences”, Notices of the AMS, 61(9), 2014, 1024-30.
-
[50]
Josh Hedtke and Scott Page, “Mathematicians Refute Oft-Cited ‘Diversity Trumps Ability’ Study”, December 2014, https://www.thecollegefix.com/post/20375/, last accessed 19 March 2018.
-
[51]
Thompson focuses on the article published in 2004 in Proceedings of the National Academy of Sciences.
-
[52]
For example with n=100 as in our model, we might suggest p=31. The agent will test 0, 31, 62, 93, 24 (124-100), 55, etc. Eventually all the solutions will be tested.
-
[53]
Its standard expression in probability is:
, signifying that the variance of a random variable is equal to the difference between the expected value squared and the square of the expected value (V standing for variance and E for the expected value). -
[54]
See the mathematical appendix for the detailed calculation.
-
[55]
Hong and Page, “Some Microfoundations of Collective Wisdom”.
-
[56]
This is the other major point on which Thompson (“Does Diversity Trump Ability”) bases her criticism.
-
[57]
Yet Hong and Page’s work can be used in more puzzling ways. For example, when Caluwaerts and Ugarriza, and Landemore and Anderson, use the first of the two theorems to defend cognitive diversity, they use it wrongly as they put themselves in a situation akin to that of a group of blind people who, by pooling their senses, must determine that the object they are touching is an elephant. This kind of situation is not at all modeled by Hong and Page.
-
[58]
The opening lines of their 2001 article (“Problem Solving by Heterogeneous Agents”), refer to “problem solving activities in the economy”.
-
[59]
Jon Elster, Securities Against Misrule: Juries, Assemblies, Elections, Cambridge, Cambridge University Press, 2013, 280.
-
[60]
The author would like to thank Philippe Urfalino for his crucial support, advice, and patient review, as well as Thomas Boyer-Kassem, Charles Girard, and the anonymous readers of the journal whose valuable comments allowed him to improve this article.
