I'm not entirely sure if the mental model is somehow a layer deeper than the prediction that humans do. I used to believe it, and still use it as shorthand, but these days I'm not sure it's accurate.
The triangle example doesn't prove it because our predictive model could also say "hey, things don't just change color and shape like that, they need to move". It's similar to how an LLM can be more accurate with math when asked to step through and reason through it's logic - by stepping through the individual steps it can create a larger system.
The thing that made me question if a "mental model" is at the base of human cognition - people who do those memory competitions, the clear winning strategy is the memory palace, or imagining walking through a house where each room is another number - they have to build step by step memory, it's not like an SQL database where they can just SELECT from random. Another one was the insight from GTD that if you remember 7 things to do, you're always repeating those 7 things to yourself to keep them in active memory.
There's a strong argument that the mental model is derivative of a predictive model in the human brain, and we can just appear to have a mental model since we have an internal dialogue that runs so fast in the background that even we rarely recognize it. (anyone who has kept a steady meditation or similar practice should be familiar with it)
Well I can't say any of this for sure but I want to say upfront, I think llm's can in theory do a lot (not sure if most) of the computations a human can do, but it's important to realize, imo, it's not actually stepping through the steps in the way humans are. When we give complicated step by step prompts and so on, it only means it's creating new constraints for what the probability of the next token is (from what exists in the data/model). If the data/model doesn't contain the data needed to produce the desired result, or the data that it was trained does not have examples that can generalize (but not be specific to) the desired result then it can't produce it.
That's the difference between humans and llm's imo. We can generalize any "computation" we have to any other "desired output" we want, by thinking about it, while llm's aren't at least not now, so general that they use low level representations of all the 'objects' we can prompt about. Like humans can reason about the objects and things in our mind almost infinitely and recursively while also retaining all the physical realities and facts of those objects, while an llm is limited in this regard. Doesn't mean it can't in theory, there is some weird generalization going on as far as I can tell, but it feels like it's going to need a lot more data or something to do it.
The triangle example doesn't prove it because our predictive model could also say "hey, things don't just change color and shape like that, they need to move". It's similar to how an LLM can be more accurate with math when asked to step through and reason through it's logic - by stepping through the individual steps it can create a larger system.
The thing that made me question if a "mental model" is at the base of human cognition - people who do those memory competitions, the clear winning strategy is the memory palace, or imagining walking through a house where each room is another number - they have to build step by step memory, it's not like an SQL database where they can just SELECT from random. Another one was the insight from GTD that if you remember 7 things to do, you're always repeating those 7 things to yourself to keep them in active memory.
There's a strong argument that the mental model is derivative of a predictive model in the human brain, and we can just appear to have a mental model since we have an internal dialogue that runs so fast in the background that even we rarely recognize it. (anyone who has kept a steady meditation or similar practice should be familiar with it)