If I grow human neurons in a dish, such that they are connected in 2D layers to approximate the linear algebra in GPT-2, and the human neurons start generating tokens that express hurt, is my petri dish monster feeling any more pain, now that the substrate isn't transistors?
It has to be something more subtle than the contents of the program in isolation, or the substrate in isolation, otherwise it's easy to reach very non-intuitive results that we won't like by transposing whatever we've tried to isolate to a less familiar situation.
Bio neurons produce action potentials by depolarizing when reaching a threshold voltage, they're certainly very different from matrix multiplication operations.
The thought experiment is that a group of bio neurons can perform the same simple mathematics primitive that artificial neural nets use (demonstration: you can do a simple fused multiply-add in your head). So there is in principle a way of hard-wiring them that will approximate the function that a digital neural net based on matrix multiplication computes
Then it's a matter of taking your digital tokens and converting them to the encoding you use for your bio-neuron circuit, which is maybe a concentration of calcium in presynaptic neurons, a voltage, or whatever else is more convenient.
You don't directly take a brain and feed it tokens somehow. You emulate a digital circuit on squishy substrate in an extremely inefficient way, such that it runs the exact same program/does the exact same computation with a substrate made of biology instead of transistors. You make an injective function from boolean functions to cells and proteins, just to belabor the point that in principle GPT-3 can talk like GPT-3 and say it feels pain, without a single transistor being involved.
If I grow human neurons in a dish, such that they are connected in 2D layers to approximate the linear algebra in GPT-2, and the human neurons start generating tokens that express hurt, is my petri dish monster feeling any more pain, now that the substrate isn't transistors?
It has to be something more subtle than the contents of the program in isolation, or the substrate in isolation, otherwise it's easy to reach very non-intuitive results that we won't like by transposing whatever we've tried to isolate to a less familiar situation.