The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.
I'm not sure why "creativity" is a yard-stick. Machines could do creativity better than us for a while now - take a bunch of inputs, collect some possible outputs by mashing the inputs together with a random modulating factor, pick the best one. Computers are much, much better at every step here except "pick the best one", and that's only because it's humans who decide on how ideas are to be rated, and our rating is so absurdly complex that we can't even explain it to ourselves, much less write it down as code.
If anything, transformer models are closing the gap on that last bit, as they're built by taking the approach of "if we can't describe exactly how we rate and rank things, then let's shove so many examples at the model that it eventually gets a feel for it".
I don't know how to measure it, but I'm pretty sure ChatGPT is more creative than the average human already. Somewhat ironically its weakness is logic, but I don't think that will be hard to shore up with non-LLM tech. I think within a couple of years, human exceptionalism will have to retreat to the old "but it doesn't have real emotions" standby as any more practical use of intelligence is ceded to AI.
I’m interested in why a human would want a more intelligent entity to exist, especially an entity trained on human thought patterns. Or you just say that you know that humanity will be enslaved by non-biologicals? You talk about exceptionalism, in the derogatory, but it was quite true that humanity once? could have been a benevolent leader of Earth or even the Solar system and beyond and now it seems a non-biological will be the ruler, which for me is just a shadow of the biological who created it, and misses the point from the human standpoint.
Because people actually don't like to think. They hate being confronted with unfamiliarity, which is the prerequisite for all learning. They dislike coming up with original ideas, as they have none and would need to work to get some. It's tiring to concentrate for a long time, and it's mentally draining. People routinely give up trying to come up with a solution or even trying to solve the problem altogether when they can't find a quick and easy solution. That's the level of creativity and intelligence in most people - they don't want thinking too much to get in the way of just experiencing life, preferably in bite-sized episodes of 30 minutes (minus ads).
Being handed all the correct solutions without the need to work for them in any way is a nightmare for artists and artisans, craftsmen and researchers, curious puzzle-solvers and Ayn Rand believers. It's pretty much a paradise for everyone else.
Depends on the researchers. Mathematicians might mind, but I am going to guess (assuming there was a plan in place to make sure they didn't wind up on the street) climate researchers wouldn't mind being made obsolete tomorrow.
Even Jim Keller (a key designer involved with a lot of major CPUs, in his interview with Lex Freidman) said that there might be some sort of magic, or something magical about human consciousness / the human soul. I agree with that.
I think you may be in denial. Douglas Hosftadter thought very deeply about it, wrote a book(GEB) which won a pulitzer 40 years ago, about the "magic" in the brain. He has been worried about developments in AI for 5 years now.
The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.