Why should you think about the meaning of simulating algorithms? [0/5]
(and what does this imply about AI and human minds)
Posts in this series:
Why should you think about the meaning of simulating algorithms? (Motivating the question) [0/5]
What do you get if you simulate an algorithm? (Philosophy) [1/5]
Do (modern) artificial brains implement algorithms? (AI) [2/5]
Do biological brains implement algorithms? (Neuroscience) [3/5]
The Implications of Simulating Algorithms: Is “simulated addition” really an oxymoron? (Philosophy) [4/5]
When science fiction becomes science fact (Major book spoilers!) [5/5]
It’s 2024! Which means it’s the 30th anniversary of Greg Egan’s Permutation City (1994). I’m late to the party (my excuse is, I couldn’t read when it was first published), but I’ve finally got around to it.
Permutation City has no lack for distinguished recommenders and reviews. It’s full of philosophical insight and explores many themes, including consciousness, computation, and the nature of simulated minds. But there’s a special zing to reading this book in 2024, given the recent developments around artificial intelligence (AI), calls for GPU chip controls, certain geopolitical events etc.
(Have I mentioned that this book was written in 1994? As in, before Google was founded.)
There were 2 sentences in this book that I remember vividly, because it tied together a cloud of thoughts floating around in my head into a concrete, articulated line of reasoning about how to think about GPTs and AI reasoning. So I wanted to share this in case it resonates with anyone else.
And because I’m curious about something, here’s a before/after poll in case you want to help me track some questions about the limits of simulations. (I’ll ask the same question at the end to see if anyone’s changed their mind from reading these posts.)
(For other fun spoiler filled observations from Permutation City, see part 6.)
In Permutation City, a character muses how:
“A computer model which manipulated data about itself and its “surroundings” in essentially the same way as an organic brain would have to possess essentially the same mental states. “Simulated consciousness” was as oxymoronic as “simulated addition.” (Chapter 3, emphasis mine).
Given the current progress of AI, we are extremely on track to develop future models that we (as in, at least a few humans somewhere on earth) will allow to manipulate data about itself and its surroundings. I’ll have some future posts on which AI labs are asking (and answering) most of the intermediate questions required to get there.
But right now, let’s think about the latter bit. In 2024 and the era of deep learning, is “simulated addition” really an oxymoron?
What does it even mean to simulate any algorithm?
But first, why does this question even matter?
Why bother thinking about the outcome of simulating algorithms?
Because I was wondering about the in-principle limits of algorithms implemented through deep learning models. Like everyone else, I’ve been wondering whether AIs like ChatGPT, the Alpha/Mu-Go/Star/Fold/Zero family, the Claudes, CICEROs, PaLM-Es, Mixtrals, NPMPs, RT-Xs, etc., can do things like reasoning in a way that’s similar to humans or not. Even if the reasoning is “just” simulated human reasoning or planning.
Why? Many reasons.
One mundane reason is, if LLMs are going to reason like humans, then I can expect the LLMs to be susceptible to framing effects and potentially give me different inconsistent outputs when I give it questions or “input prompts” which are written or framed in one way vs. another, just like humans.
On the other hand, if GPTs aren’t reasoning like humans, then I’d like to:
know what they are doing differently
figure out whether it’s better to do whatever they’re doing under certain circumstances
copy them, if possible.
(I promise I’m a human person. I’m not sure why GPTs or RL agents or other artificial networks sometimes get criticised for “just mimicking” their training data. I like to think people have been doing it for as long as we’ve been alive to achieve goals like “don’t be a nuisance'', “blend in”, and “read the atmosphere”. And I observe that whenever I (or anyone) manages to do so successfully, they are generally socially rewarded as a good, non-disruptive member of whatever culture they happen to be in. Though of course there are times when we ask for the opposite.)
The other, slightly grander reason, is because there’s an ongoing debate about what sorts of specifically human-like mental and cognitive capacities AI might or might not be able to replicate or, more importantly, surpass. Usually, you can spot these debates when words like “consciousness”, “intelligence”, “abstracting”, or “reasoning” are thrown around.
Don’t get me wrong, human cognition is hardly the only type of cognition that’s around. (I check in on the cat cognition literature every so often, out of personal interest). But it is the type of cognition to understand, if you want to know why or how humans perceive and think.
Humans happen to be in control of a lot of things in the world (for now). At least some of that control can be attributed to human reasoning being useful at things like figuring out how the world works, and then bending it to our short-ish term preferences. (Which also results in occasional whoopsies like global warming, over-harvesting of crops or livestock, massive income/social inequality, and other tragedy of the commons type phenomena that are familiar to economists. We’re human. And while we’re known for a great many wonderful accomplishments — precise long-ish (80+ years) term foresight is not generally one of them).
For example, my cat does cat reasoning. In the video below, you can see her navigate a messy desk full of obstacles. It’s clear she’s not navigating it in an unreasoned, frantic, or random manner. How much reasoning or deliberation you think she’s displaying is a discussion for another day, but I think most would agree it’s not literally zero reasoning (see the Terminology section below on what I mean when I say “reasoning”). If you think she’s somehow memorised the layout of the desk, let me add that this is a desk that has constantly changing obstacles on it. It is not my desk (so the obstacles are unfamiliar to her), and this is the first day my cat saw this new desk.
And yet, despite whatever cat reasoning is present in cat brains, cats do not rule the world (…or do they?).
Can future stacks of multimodal GPTs and RL training algorithms, in principle, ever become as adept as humans at doing things like adding, planning, abstracting, and “intelligenting”, even if they don’t even implement “adding'“, “planning”, and “abstracting” the same way that humans do?
Can they do better than humans?
Because the answer determines who gets to shape the future.
Terminology notes
Reasoning
The human mind is large, multi-layered, and very lumpily shaped. So for the purposes of these few posts, when I say human reasoning, I mean the parts of human reasoning that is commonly known as intuitive reasoning, which includes both system 1 and system 0 (or perceptual reasoning).
System 0 reasoning is what you might know as “perception”. It’s the type of reasoning that your brain and mind does to generate perceptions or “hallucinations” of visual illusions. If you prefer to use the word “perception” rather than “intuitive reasoning”, that’s fine. I just find that if I say perception, it’s not super commonly known that perception involves your brain doing a whole bunch of unconscious inference for you to see and hear things. And inference is commonly used as a synonym for reasoning, isn’t it?
System 1 is the Kahneman type of subconscious and unconscious intuitive reasoning that human brains do. It’s the part that generates intuitions like “the phrase ‘green great dragons’ sounds grammatically weirder than ‘great green dragons’ ”, and the moral dumbfounding effect.
(Above, a human engaging in some sort of reasoning. You may notice some inconsistencies from typical adult human reasoning. You might even notice that you’ve seen similar looking reasoning failures in some LLMs before. You might wonder why this particular human would shake his head —commonly a gesture to indicate “no”— and then say “yes”, and remember that sometimes, LLMs will tell you it cannot do X and then proceed to do X anyways (example 1, example 2). Perhaps you then laughed or felt relieved that LLMs are still “unintelligent” and will never be as intelligent as “humans”. I hope it’ll be clear by the end of these posts why that is an incorrect assumption.)
This is in contrast to System 2, the typical deliberate, effortful, reasoning that might be going on in your mind’s voice. As in the type of reasoning that goes “Hmm if I need to be at Tokyo Narita Airport by 5pm on the Sunday of a Golden Week weekend in Japan, should I be on the road 1 hour earlier than normal or not?”.
I do not think that we currently have any LLMs (multimodal or not) that are strongly conscious (yet). This is despite the fact that it is so very tempting to see LLMs output streams of words that sound like the same words humans might speak in our mind’s voice, deliberately, system-2 style, “consciously”.
From what I can tell, the experimental results about what neural networks (language, vision, action, speech, audio, or multimodal etc.) can do generally best equates to the human brain’s perceptual learning systems.
For text-only LLMs with causal / unidirectional / one-way attention masking in particular, their abilities are probably best thought of as being similar to what a human brain does when acquiring and processing (i.e., “perceiving”) verbal, auditorily transmitted language.
Causal Attention can only incorporate information from prior tokens into the current token. It cannot see future tokens. Causal Attention is used when we need to preserve the temporal structure or a sequence, such as generating a next token based on prior tokens. (source)
Causal attention looks like trying to predict the next token ___ in the sentence “The best animal in the world is ___ because it is soft and furry.” without being allowed to see the words “because it is soft and furry” from the “future”.
The best analogy of this for humans is verbal language because verbal language must be transmitted syllable by syllable over time. Your ears cannot hear the sounds for a whole sentence spoken at the same instant, because people do not speak that way. Which makes it most similar to the processing enforced by a one-way attention masking in a transformer.
This is quite unlike using your eyes to read and process written symbolic language visually, where the text is displayed all at once, and can be referred to backwards if needed. This is also why I suspect vision is more similar to using a BERT-style, bi-directional, “fill in the blank”, type of attention masking, if anything.
Bidirectional Attention is used in encoder blocks in encoder-only models (BERT) or encoder-decoder models (BART). It allows the Attention mechanism to incorporate both prior and successive tokens, regardless of order. Bidirectional Attention is used when we want to capture context from the entire input, such as classification.(source)
We do have a literal blind spot in our eyes that our brain “fills in the blank” with the signals coming from the surrounding neurons on the retinal image, after all. A 2D image has no inherent concept of pixels that are “before” or “after” one another in time. (Video however, does). It also makes sense why a vision transformer would be designed to use bidirectional attention and while text-based “language” transformers generally use unidirectional attention. See section 13.1 “Twelve Ways Oral and Written Language Differ” for more differences. Also consider, why is it easier to reverse the words in a long-ish sentence when we see it laid visually, but harder if the same sentence was only verbally spoken out loud?
(Has anyone tried to reverse a random sentence —picked by a person who is not yourself of course— while listening to a podcast vs. the same sentence when reading the transcript?)
Here are a few more examples to show the difference between system 0, 1, and reasoning, but applied to auditory language.
There’s a paper which found a “Reversal Curse”, where LLMs trained on “A is B” fail to learn “B is A”. For example, if you prompt “Who is Tom Cruise’s mother?” GPT-4 correctly answers questions like these ~79% of the time, but if you ask “Who is Mary Lee Pfeiffer’s son?” GPT-4 answers correctly only ~33% of the time.
Here’s a question, you likely know the alphabet song forwards given that you are reading this post, but can you sing the ABC song backwards? (For extra brownie points, reverse also the melodic tune/notes.) I can’t do it in my head intuitively without first trying to visualize the alphabet in my head (or on paper).
Here’s another fun example, can you figure out what song this is? (I could not figure out what song this person was singing backwards, even though I definitely knew the original song played forwards.)
If there’s more than one person out there like me who similarly failed on one or more of the examples above, wouldn’t it be (literally) accurate to say then, “Human brains trained on A-B-C fail to learn C-B-A”?
And if you replace “B” with “is”, what’s the difference between knowing your CBAs when you know the ABCs, and knowing that Mary Lee Pfeiffer’s son is Tom Cruise when you know Tom Cruise’s mother is Mary Lee Pfeiffer?
It’s not that I can’t learn the ABCs backwards. It’s just that it doesn’t happen automatically even in humans. So why are we expecting it to happen automatically in LLMs? And it’s not that I can’t learn the alphabet backwards. I’d just have to write the CBAs multiple times, maybe put it in another catchy tune, and then I’ll be able to recite the alphabet backwards.
Or in other words, do exactly the same thing I did to learn the ABCs forwards - add that sequence into the training data for my brain.
This is why I think it’s important not to let the semantic content of words trip you up when trying to think about how LLMs might or might not be processing word-like tokens. It’s the same sort of caution neuroscientists need to take, when figuring out how the human brain processes sound frequencies vs. sound frequencies that also happen to be spoken words (here’s a lovely video that might help illustrate the difference for English speakers). Knowing that a brain can process sounds does not automatically imply a brain will also process sounds as speech just as easily. The average human one can, most times, but that’s only if you’re lucky enough not to suffer from impairments like degraded speech-in-noise perception.
(This isn’t to say that future LLMs will never figure out the semantic content from purely perceptual processing. After all, we know most human brains got there at some point, somehow.)
Unconscious
When I said unconscious in “unconscious intuitive reasoning”, I mean specifically that you are unconscious of the processing or execution of an algorithm, not necessarily that you are unconscious of the output. (If you prefer the word “implicit” for this, feel free to substitute that in.) This is the same way you are:
unconscious of how your brain generates the experience of magenta but you nonetheless can consciously see the colour.
unconscious of the particular grammatical structures of your native languages, yet you can “hear” when someone’s grammar sounds odd without being able to articulate consciously and exactly what went wrong.
For a non-English example just to make this point more obvious, here’s an example from Japanese.
I’m sure there are explicit rules of thumb that tell you how to pick the one of 2-5+ ways you can pronounce certain kanji characters. I imagine those rules are what you get if you acquire languages through explicit instruction in a language course or something.
But I am also certain that if you can ask most Japanese native speakers (who didn’t also specifically study the linguistics of their language); they won’t be able to explicitly tell you what the rules are for when you read a character one way and not the other.
We might have all learnt the explicit rules of our native languages and second languages in high school at some point, but the implicit and perceptual part of language and grammar acquisition is the sort of thing a human brain is demonstrably able to learn unconsciously. Think about how many current and previous (e.g., you) primary school children in any country can speak and “just use” the grammatical structures of their native languages before explicitly knowing what the grammar structures are. Have you seen a child use prepositions in English before knowing what a preposition is?
We have long known that human brains have at least 2 modes of learning. We just never figured out how to get computers to do the “harder” kind until now.
Every day, I thank the stars I get to just use my brain without having to know exactly how the thing works✨.