The danger of a single answer
69069
wp-singular,post-template-default,single,single-post,postid-69069,single-format-standard,wp-theme-bridge,wp-child-theme-WACC-bridge,bridge-core-3.3.5,qodef-qi--no-touch,qi-addons-for-elementor-1.11.2,qode-page-transition-enabled,ajax_fade,page_not_loaded,,qode-title-hidden,qode-smooth-scroll-enabled,qode-child-theme-ver-1.0.0,qode-theme-ver-30.9.1,qode-theme-bridge,qode_header_in_grid,qode-wpml-enabled,wpb-js-composer js-comp-ver-9.0.1,vc_responsive,elementor-default,elementor-kit-41156

The danger of a single answer

Ila Schoop Rutten

Many of us spend a fair amount of time worrying about what artificial intelligence makes up. Some may remember, for example, when a pair of lawyers used Chat GPT to write their legal brief and the chatbot cited fictitious legal precedent. Alongside hallucinated court cases, chatbots have been known to invent academic papers, attribute quotations to people who never said them, and occasionally produce the kind of answer that makes you wonder whether “intelligence” was an unfortunate choice of branding.

There is now an entire cottage industry devoted to teaching people not to trust an A.I. when it sounds most certain, or most cloying. But through my work studying information ecosystems, a different, almost opposite question has started to gnaw at me: What about the things A.I. does not invent, but make disappear?

When we give a chatbot a complicated question, it usually returns something that looks reassuringly finished. The answer might contain caveats, but its generation does not recreate the experience of actually researching a contested subject, for example, eighteen open browser tabs, two academics who appear to despise one another, a newspaper article that contradicts the official account, an obscure PDF from 2007, and the growing suspicion that we started out with the wrong question in the first place. The chatbot takes this mess and turns it into fairly boring but convincingly sleek and complete text.

This finished quality is, of course, why we use it. Why I use it.

But the surface elegance of the answer conceals an odd fact. Somewhere between our prompting and those almost instantly-generated paragraphs, the hodgepodge of disagreement has been synthesized. Some sources have mattered more than others and some were neglected altogether. A history that has never been digitized, a language with little presence online, a different people’s way of being and knowing, or an experience that makes sense only to somebody who knows a landscape, sound, smell or feel of a particular object in their hand has a much harder time finding its way onto our screens.

The answer provided by a chatbot may be perfectly correct, which is part of the problem. It is by the same token, insufficient. Our chatbots often sound like an undergraduate who read enough of their curriculum to sound supremely confident but not yet enough to have doubts.

Scholars of decoloniality have long been interested in a peculiar kind of erasure: not necessarily the deliberate destruction of knowledge, but the process by which some ways of knowing come to look like knowledge itself, while others are marked as local, traditional, subjective, or strange. Aníbal Quijano described this as part of the coloniality of knowledge: European ways of understanding the world acquired an authority that survived formal colonialism, becoming the standard against which other knowledge systems were judged. The Colombian philosopher Santiago Castro-Gómez takes the idea a step further. He calls this privileged position the “zero point”: the fantasy that one can somehow stand outside history and culture, looking neutrally down on everyone else’s ways of knowing.

The hierarchy that follows is familiar. On one side sit science, statistics, law, academic research and official records. These are forms of knowledge whose conventions have become so established that we often forget they have conventions at all. On the other side rests oral histories, customary practices, spiritual interpretations, Indigenous knowledge, and understandings acquired through land, craft, memory or experience. These are usually identified as belonging to a particular culture. The first group, by contrast, can appear to belong to no one. It simply describes the world.

This distinction is more consequential than deciding that somebody is wrong. If I tell you that you are wrong, I have at least admitted that we are making competing claims about the same world. We can argue about the evidence, about facts: things that happened and that others can witness, document, or contest. But saying that what you know does not really count as knowledge is a rhetorical kneecapping: it removes your claim from the argument before the argument has begun.

The power of the “zero point”, then, is not simply that one worldview defeats another. It is that one worldview ceases to look like a worldview at all. It becomes the neutral position from which everyone else is described.

This is also what makes the idea feel newly relevant to artificial intelligence. A chatbot rarely announces the historical, cultural or institutional assumptions embedded in the material from which it has learned. It does not normally speak as one participant in an argument. It speaks from nowhere. And because it speaks from nowhere, its answer can easily sound as though it comes from everywhere.

There are historical cases in which this process was not subtle. Indigenous peoples in Australia and North America developed sophisticated systems of cultural burning: deliberately setting small, carefully timed fires based on deep knowledge of particular landscapes and weather systems in order to reduce the risk of wildfire. Colonial governments suppressed these practices, and the removal of people from their lands and of children from their communities interrupted the transmission of the knowledge that sustained them. Faced with the growing problem of massive, out-of-control wildfires today, authorities in several places are again working with Indigenous practitioners as they reconsider the consequences of a fire-management philosophy built around suppression.

What’s notable here is that the knowledge was not tested, then disproved. The practice through which the knowledge existed was prevented from happening. You can try to write down that a certain plant should burn at a particular time of year. That does not necessarily mean you know how to burn it. The knowledge may remain partially undocumented because it resides in numerous subtle cues, perhaps partly in the smell of the vegetation, how a particular wind behaves, what happened in that place the year before, all in addition to what somebody older tells you while standing beside you. Remove people from the land, ban the practice and wait a generation, and a strange thing happens: information might technically survive while the ability to know in the same way begins to disappear.

Marginalization of knowledge systems

The term “epistemicide” is sometimes used to describe the destruction and marginalization of knowledge systems associated particularly with colonial domination. Some may protest that epistemicide is an unnecessarily violent word that cheapens its linguistic relatives. It also calls to mind an aggressive destruction of manuscripts, the banning of languages, the killing or displacement of people who carry knowledge and the suppression of the institutions through which they pass it on. But epistemicide is not just the deliberate burning of the library of Alexandria or the destruction of the Aztec codex. It can, like other forms of colonial erasure, be more insidious, and slow; the unintended consequence of other forms of dominance and oppression.

Calling anything happening on ChatGPT or another chatbot epistemicide is probably, at this point, unfounded. But the legacy of decolonial scholars raises a point that feels harder to dismiss: knowledge does not have to be destroyed in order to disappear.

For centuries, societies have built machines, some literal, some institutional, for sorting knowledge. The library classified it, the university credentialled it, the encyclopaedia condensed it. Newspapers would decide what deserved the front page and then search engines like Google or Bing would arrange it into a ranked list whose order could determine whether a business prospered or a news article was read. None of these systems has ever presented humanity with an unmediated view of what humanity knows. In other words: A.I. did not invent knowledge hierarchies.

What has changed is that generative A.I. can make the hierarchy harder to see. A Google search, for all its unfairness, generally leaves evidence of the disagreement lying around. Search for a controversial historical event and you might get the government account, a university page, a newspaper investigation, an advocacy organization and a discussion forum filled with people calling one another all kinds of choice words. You still have to navigate by mouse or thumb, and while the system ranks the choices, it at least leaves some of the choosing to you. A chatbot, on the other hand, is more obliging. It reads the room and tells you what happened.

Inherent limitation of chatbots

This is where that smooth, quick, single answer becomes slightly misleading. A chatbot can provide many answers. Ask it to give five interpretations of a historical controversy and it will happily oblige. But before the model can supply us with five perspectives, something has already determined what qualifies as a perspective worth naming.

Some forms of knowledge were available in abundance. Others were barely present in the material from which the system learned. Some were written in highly digitized languages; others were not. Some arrived in peer-reviewed journals, government documents, newspapers and encyclopaedias… the sorts of explicit, codified knowledge that machines can process especially easily. Others live in memory, apprenticeship, ceremony, local language or repeated practice. This all may simply be a consequence of asking a machine to know things– an inherent conundrum rather than purposeful flaw in the tech.

What does something have to become before a machine can know it? A ceremony can be captured in a video. The forest can become environmental data. A sacred object can become an entry in a museum database. Much may be gained in the translation of knowledge into something accessible and that transcends time and space, but what will be lost in translation?

There is a temptation in conversations about A.I. diversity to treat the problem as one of inventory: What did the training material of the A.I. include? Does the dataset contain enough languages? Enough material from Africa? Enough Indigenous knowledge? Enough women?  These are important questions. But imagine, for a moment, the perfectly inclusive dataset. Every culture is represented. Every language has been digitized. Improbably, the world’s oral histories have been carefully recorded, with consent.  We would still have to decide what the machine should do with all of it. It would still have to translate, of course, and it would still have to decide when two accounts contradict one another, when they are simply answering different questions, and what can safely be compressed into three paragraphs for somebody who has asked a question while standing in line for coffee.

The plurality of inputs does not necessarily create plurality of knowledge. You can put the entire world inside a machine and still teach the machine to speak with one voice. After all, generative A.I. models are not thinking machines but statistical ones. They do not “know” what they say but calculate the likelihood of the next word.

Of all the very legitimate worries about artificial intelligence mass unemployment, autonomous weapons, surveillance, weaponised deepfake pornography, data centres gobbling up the planet’s resources; the eventual arrival of a superintelligence that decides humans are bumps on the road to finishing a task and we ought to be flattened, the one I find myself returning to is much less cinematic: What if the technology that gives us unprecedented access to all of human knowledge also gradually makes that knowledge more alike? In other words, what happens when the place where we go to encounter different ways of understanding the world is designed to give us an answer?

We can imagine an alternative to this epistemic monoculture, a kind of ecology of knowledges. The phrase can sound like the sort of thing that’s adoringly parroted inside a graduate seminar, but the idea behind it is fairly intuitive. Different ways of knowing should be able to encounter one another without one automatically serving as the yardstick by which all the others are measured.

To be sure, this does not mean that every idea deserves equal respect. If somebody tells me that a ritual cures leukaemia, I do not think cognitive justice requires an oncologist to shrug and call chemotherapy another culturally situated narrative. Scientific methods allow us to test claims in ways that have produced extraordinary knowledge. Traditions can be oppressive. Communities disagree internally and people can be wrong. The interesting part starts after we acknowledge all of this because it is also possible to be correct about a fact and still misunderstand the world in which that fact has meaning.

A.I. does not necessarily have to make this problem worse. The technology could be an extraordinarily powerful device for epistemic pluralism. Imagine asking about the management of a forest and receiving not just the dominant forestry literature but the knowledge of people who have lived in and protected that forest for generations, clearly attributed and presented on its own terms. We could also envision a translation system that helps a minority language survive in a digital economy instead of making the dominant language even more indispensable. There could be a chatbot that interrupts its own answer to tell you that the question you asked makes sense in one intellectual tradition but not in another.

In some respects, A.I. could do this better than Google ever did. A search engine requires you to know enough to formulate the search term. A Chatbot can interpret a garbled half-question before explaining the assumptions behind competing arguments. It can make obscure knowledge accessible to somebody who did not know the vocabulary required to find it. The technology is not inherently a machine for epistemic monoculture. That may make the choices now being made about its design more important; there is a difference between building a system that contains information about a culture and allowing the people who sustain that culture to exercise some control over how its knowledge is categorized, shared and interpreted. This is where the somewhat bureaucratic language of cultural rights becomes useful. International human-rights law protects, among other things, participation in cultural life, cultural expression, language and access to and enjoyment of cultural heritage.

A community whose songs, stories and practices have all been fed into an A.I. model might be extraordinarily well represented in one sense and almost powerless in another. Who decided which material could be included? Who labelled it? Who decided what was sacred, private, communal or available to anyone with a prompt? Who benefits economically from the model’s ability to reproduce it? Who gets to tell the machine it has misunderstood?

Inclusion on whose terms?

The usual response to technological bias is inclusion. Getting more people into the dataset, expanding the languages, and hiring more diverse annotators are sensible things to do. But inclusion can reproduce the original problem if entering the system requires everybody to conform to categories designed elsewhere. There is an old colonial trick hidden in this possibility: you are welcome to participate, provided that I get to decide what participation means.

Colonial epistemicides involved coercion, dispossession, violence, schools, churches, armies and states. Let’s not exaggerate, perhaps a chatbot confidently summarizing a tradition it does not understand is not the digital equivalent of any of these things, at least not while there are still other sources of knowledge and exchange available. The more useful question is probably whether A.I. could help create the conditions under which some forms of knowledge gradually become harder to sustain.

The A.I. researcher Andrew Peterson has used the term “knowledge collapse” to describe a possible narrowing of accessible knowledge as models and the people who use them preferentially reproduce popular information at the expense of obscure alternatives. One analogy is the “streetlight effect”, from the joke about a drunk person searching for his keys under a streetlamp. He did not lose them in that spot but the light there is better.

A.I. is an extraordinarily bright streetlight– if the easiest information to retrieve, summarize, translate and incorporate into institutional life is the information machines can already see, we will gradually begin looking there more often. People who speak languages poorly served by digital tools will likely increasingly use languages that work better. Essayists and policy wonks who begin research with an A.I. answer may never know which intellectual paths were absent from it.

The danger here would not be that a machine declares a particular knowledge system wicked and handily flammable. It would be that the machine becomes useful enough that we begin reorganizing ourselves around what it can recognize.

So this is the part of the problem that reverses my original concern. I started by wondering what disappears when a chatbot produces an answer. Is the more interesting question actually what happens when humans begin changing the way they produce knowledge because they know that machines are responsible for finding and processing it?

Oral traditions might be textualized and institutions might reorganize archives so machines can ingest them. Of course, these things are not inherently bad. Preserving endangered material in digital form can be enormously valuable.

For centuries, governing authorities asked communities to make themselves legible to the state, the church, the scientist, the census-taker or the anthropologist. But what happens if the new demand is to make ourselves legible to the machine?  The most unsettling possibility is therefore not that A.I. models will fail to be trained and to be able to contain and reproduce the world’s knowledge. It will undoubtedly contain more of it than any institution in human history.

The next epistemicide, if the term ever becomes appropriate, will not resemble book-burning or a language being prohibited. Perhaps nothing will technically disappear. The stories will still be stored somewhere; the recordings will remain on a server; the language will still be spoken in somebody’s home.

They will simply sit on the other side of a new divide: the line between what a machine can turn into usable knowledge and what it cannot. We have spent a great deal of time asking whether artificial intelligence can learn to understand us. It seems at least as important to ask what kinds of people we will become after years of trying to make ourselves understandable to it. The danger of a single answer is not that somebody, somewhere, knows another one, it is that eventually we may stop noticing that another answer was possible.

 

Ila Schoop Rutten is a researcher, writer, podcaster and strategist working on information disorder, epistemic fracture, freedom of expression, information access and digital rights. Over the past decade, she has worked to protect human rights with grassroots organisations, UN agencies and institutions in between, across several continents. Most recently, she served as Information Integrity Lead at CDAC Network, researching information environments in low resource and crisis contexts and developing community-centred responses to harmful information. She is also a board member of the Tibetan Centre for Human Rights and Democracy. Her work explores the relationship between people’s everyday lives and the ideas, rhetoric and policies of powerful institutions, and how information can be used either to constrain or empower individuals, organisations and societies.