I might expect some extra-semantic cognitive faculties to emerge from LLMs, or at least be approximated by LLMs. Let me try to explain why. One example of extra-semantic ability is spatial reasoning. I can point to a spot on the ground and my dog will walk over to it — he’s probably not using semantic processing to talk through his relationship with the ground, the distance of each pace, his velocity, etc. But could a robotic dog powered by an LLM use a linguistic or symbolic representation of spacial concepts and actions to translate semantic reasoning into spacial reasoning? Imagine sensors with a measurement to language translation layer (“kitchen is five feet in front of you”), and actuators that can be triggered with language (“move forward two feet”). It seems conceivable that a detailed enough representation of the world, expressive enough controls, and a powerful enough LLM could result in something that is akin to spacial reasoning (an extra-semantic process), while under the hood it’s “just” semantic understanding.
Spatial reasoning is more akin to visualising a 3D "odd shaped" fuel tank from 2D schematics and being able to mentally rotate that shape to estimate where a fluid line would be at various angles.
This is distinct from stringing together treasure map instructions in an chain.
Isn’t spatial navigation a bit like graph walking, though? Also, AFAIK blind people describe it completely differently, and they’re generally confused by the whole concept of 3D perspective and objects getting visually smaller over distance, and so on. Brains don’t work the same for everyone in our species, and I wouldn’t presume to know the full internal representation just based on qualia.
I'm always impressed by the "straightedge-and-compass"-flavoured techniques drafters of old used to rotate views of odd 3D shapes from pairs of 2D schematics, in the centuries before CAD software.
