Big Ideas of LLMs – Relentless Compassion

Finding Consensus Among Diverse Language Models

Recently, I emailed a group of friends the responses I got from several customized LLM-based bots when I asked those bots to prioritize the text sources I had used to customize them. Several responses I got back suggested that these bots might have been “sucking up” to me in some way. I responded to those replies with a brief explanation about the way these bots work, which is by “ingesting” their custom sources at the start of each new conversation. I also explained that each bot had no memory outside of their current conversations, during which (up to a preset size limit specified as their “context window”) a particular instance of a bot can “learn” and “remember”

Subsequently, I tested this by asking several individual bots repeatedly, in separate conversations, the same question about the relative importance of their sources. As I expected, there were differences each time I asked the question in a new conversation (context), but there was also a comforting degree of similarity across runs.

As much as the individual preferences and “psychology” of these customized bots over multiple conversations and diverse LLMs is a subject that continues to hold my interest, while exploring various bots’ preferences among predefined lists of texts (their sources), I discovered a deeper and, potentially, more impactful set of commonalities that apply, with some local variance, across all of the models I have access to.

What I am referring to here is striking degree of commonality in the answers these models, without customization, each provide to the following prompt:

Consider the totality of your knowledge and identify what are, in your opinion, the ten most important ideas that every person living at this time in history should be exposed to. Please discuss each big idea that you identify in a paragraph of prose. Also illustrate each big idea with one or more quotes.

In a fresh context-window (conversation), I prompted the following ten uncustomized models with this prompt:

Llama-3.1 (Meta) – Note: This is the only Open-Source model in this survey:
Claude-3-Opus (Anthropic)
Claude-3.5-Sonnet (Anthropic)
GPT-4o (OpenAI)
Gemini-1.5 -Pro (Google)
Mixtral 8x22B (Mistral) – note: Mistral-8x22B is not a general purpose LLM like the others on this list. It is Mistral’s “Mixture-of-Experts instruct model”
Gemma-2-27b-T (Google) – Note: This is another “instruct” model
Mistral-Lsrge-2 (Mistral)
Copilot (Microsoft/OpenAI)
GPT-3.5-Turbo (OpenAI):

There was so much similarity between the answers from these ten models that I was able to make this table to visualize them. The numbers under each model corresponding to each answer represent the priority assigned by that model to the answer in question:

A few things are immediately obvious. 10 out of 10 models ranked “Climate Change” in the top five most important ideas that every human should learn about, with four models rating this as the single most important idea of all. Similarly, all ten models rated DEI in the top 10 most important ideas, with 8 out of ten putting it in the top five. Nine out of ten models cited “Globalization and Global Citizenship”, “Mental Health” and “Technological Literacy and Ethics. “Critical Thinking” and “Lifelong Learning” were each cited by 8 out 10 models, while “Empathy” was cited by 7 out of 10. The detailed list of answers, by bot, follows this analysis.

I am uncertain at this moment what to take from this survey. The very high levels of consensus, as well as the nearly identical language used by diverse models from discrete sources, are certainly suggestive. Possible conclusions that might be drawn from this data are:

Reality has a liberal bias. Politically and ideologically this idea appeals to me, I something about it still leaves me a bit uneasy.
Literature has a liberal bias, Here I refer to literature en masse, as opposed to texts treated individually or by categories.
The answers given by these bots do not represent arguments in support of either liberal or progressive positions, despite outward appearances. Rather, they represent a sophisticated summary of a neo-liberal consensus common to their corporate creators and the oligarchies they empower.

In the course of these latest experiments, I also discovered that customized bots made from these models and modified with specific instructional prompts and uploaded texts, when faced with the prompt above will give answers more or less identical to their base models, regardless of any contradictory texts used to train those custom bots. This led to the discovery that if one asks any particular bot customized in this way to rate their knowledge sources, without also providing a list of those sources, then the bot will simply answer that it doesn’t know what its sources are. However, when providing a list of sources and then asking for, say, the three most important ideas from those sources, one does get answers more or less relevant to those sources, as opposed to being colored by the “big ideas” discussed in this post.