I think we're a long way from a good understanding of how consciousness works, but I also think a lot of people are going to subscribe to a sort of consciousness-of-the-gaps idea no matter how much progress is made in understanding the actual mechanisms. Even if we fully understood and and could reproduce it, there would be scores of people who would flat out refuse to see the evidence and would simply assert that the ineffable "experience" does not exist within beings for which they don't want to acknowledge it. The very concept of p-zombies illustrates this a priori refusal to admit any possible evidence whatsoever of consciousness. Another person could simply decide that I am in fact a p-zombie and lock themselves in a closed system of thought out of which there is no path to demonstrating that I "experience" anything at all.
I think if you want to put forth a hypothesis that there is some ghostly ineffable part of consciousness called "experience" that cannot ever be touched or measured by scientific means, then you have a self-defeating argument that cannot be supported. You might as well go full solipsism. There's nothing stopping you.
Consciousness is a genuine mystery at this point, but I think some people will still see it that way even if we solve it, and this is clearer to me every time I see people trash any kind of effort or progress made by science in understanding the brain, claiming that it is not in fact progress at all.
On the other hand, I think that many people are emotionally invested in believing that science must be capable of solving the hard problem of consciousness even though there is no reason to assume that science is.
It is perfectly possible that the hard problem of consciousness is in principle and forever beyond the reach of scientific investigation.
Just to toss off one more worthwhile idea to you since it seems like you're interested in this topic. p-zombies is not the most challenging scenario strong-AI deniers are likely to face. Brain cell replacement is.
With p-zombies you have two observers outside the system arguing about the system's inner life. With brain cell replacement, you have the subject directly and quite authoritatively experiencing the system in question and reporting back.
It seems many times more likely some of us will live to see this, but you just never know. Newtonian mechanics had it all locked up save for a few details and look what those details held.
Every brain science / cog-sci paper it seems has some alternative amputating conclusions to pronounce about consciouness.
They sort of have to do that because of the funding model they live under. Positive results only ! It's not the researcher's fault; I don't fault them. I just adopt a highly skeptical, wait and see, there's-probably-more-to-the-story attitude generally in science, that, and the more concrete counter-arguments I mentioned in my other comments make me a very highly dissident observer of this field.
"The very concept of p-zombies illustrates this a priori refusal to admit any possible evidence whatsoever of consciousness. Another person could simply decide that I am in fact a p-zombie and lock themselves in a closed system of thought out of which there is no path to demonstrating that I "experience" anything at all."
This is a good point and makes the problem interesting in an additional way. We (I) assume something like p-zombies exist in non-human consciousness, dogs and cats for example. It's like something to be a dog. How far down do we want to go ? Frogs? I'll bite; it's like something to be a frog:
But here's a counter to the p-zombies argument, OK?
The p-zombies argument is usually taken to mean there comes a point where what has been created is so indistinguishable from "real" people, ala Ex Machina, that arguing over it is a form of ideologically motivated perversion.
Let me turn that round and say that the p-zombie argument is (accidentally) making the following strong claim- it is impossible to build a machine which in every way acts human but has no experience.
That's a very very strong claim on this universe. I wouldn't take the bet, because someone's going to do it.
But if someone is going to do it, how can we tell when they have or they haven't? The Turing Test is outdated (as I see it) and anyway already passed for some judges ( re: ELIZA).
To me, this circles back to the original problem. We can't distinguish between the high probability that someone can eventually create an actual zombie and "real" experience-having artificial intelligence, and why is that ?
The issue is just another form of the basic problem- we don't have the conceptual framework to get our minds around what experience is.
Our basic assumptions may be off. Instead of quarks et.al. being the basic building blocks of matter and matter of brains and brains of consciouness, some people take experience to be the most basic building block of the universe.
This was my conclusion and I thought it would just brand me as an eccentric so I never pushed it, but now I see it's being kicked around by people with careers.
Another assumption is that experience/consciousness is comprehensible to the level of scientific causality/reality we're aiming at, (let's just shorthand it to "ultimate reality"), because there are separate, distinct things in the first place.
But what if separate things is not a fact about ultimate reality? What if they're more like a hardwired perceptual compulsion we can't escape? Then we might very well find truly insoluable mysteries on the foundational tier of our conceptual scaffolding, because none of the "things" we think about are real in the first place. Things which don't exist, don't have to "add up".
So this would mean our minds and ultimate reality are just not made for each other, even as that reality directly impinges on our personal daily lives in ways we can and do readily experience and talk about.
It seems like the most far fetched and deflating hypothesis possible, but consider we'd merely be joining the rest of the animal kingdom in this regards.
I think if you want to put forth a hypothesis that there is some ghostly ineffable part of consciousness called "experience" that cannot ever be touched or measured by scientific means, then you have a self-defeating argument that cannot be supported. You might as well go full solipsism. There's nothing stopping you.
Consciousness is a genuine mystery at this point, but I think some people will still see it that way even if we solve it, and this is clearer to me every time I see people trash any kind of effort or progress made by science in understanding the brain, claiming that it is not in fact progress at all.