Rich people currently have little trouble controlling people who are much smarter and more capable than they are. Controlling resources and capital goes a long way and it isn't a given that AGI would transcend that dynamic.
If we can be confident of that, then most of the worst problems with AI are already solved.
Part of the problem is that "do what I said without question" will lead to disasters, but "figure out what I would approve of after seeing the result and be creative in your interpretation of my orders accordingly" has different ways it can go wrong.
(IMO, RLHF is the latter).
Both of those seem to be safer than "maximise my reward function", which is what people were worried about a decade ago, and with decent evidence given the limits of AI at the time.
> If we can be confident of that, then most of the worst problems with AI are already solved
which leaves unprecedented power in the hands of the most psychopathic[0] part of the population. so even if AI take off doesn't happen, we're still getting the boot on our necks.
> Roughly 4% to as high as 12% of CEOs exhibit psychopathic traits, according to some expert estimates, many times more than the 1% rate found in the general population and more in line with the 15% rate found in prisons.
On the plus side, this is still a small minority.
On the down side, these remind me a lot of Musk:
> CEO who worked with several pregnant women told people that he had impregnated his colleagues.
By way of Neuralink.
> CFO thought his CEO had a split personality, until he realized that he was simply playing different characters based on what he needed from his audience.
"Will my rocket explode?"-Musk is a lot more cautious and grounded than everything-else-Musk — including other aspects of work on SpaceX.
> Autocratic CEO fired a well-respected engineer “just to make a statement.” He fired anyone who challenged him, explaining there was no reason to second-guess him because he was always right and needed people to execute his vision rather than challenge it.
Basically all of Twitter, plus some other anecdotes from Starlink, SpaceX, Tesla.
And, this month, fighting with Asmongold about cheating in Path of Exile 2, before admitting to what he was accused of but trying to pretend it's fine rather than "cheating".
> CEO would show up to work and begin yelling at an employee (usually someone in sales) for no obvious reason.
The guy he called a pedo for daring to say a submarine wasn't useful for a cave rescue, the Brazilian judiciary, members of the British cabinet, …
But it looks to me like there's a decent number amongst the other nine who know what grenades are and don't want them to get thrown by the tenth.
The power dynamics here could be just about anything; I don't know how to begin to forecast the risk distribution, but I definitely agree that what you fear is plausible.
it's possible that the other 9 would keep the 10th under control, but if you look at the direction the US has taken, when two billionaires took over and declared inclusion verboten, the others rolled over and updated their policies to fall in line.
