I fully agree with the religious aspect of it (religion doesn't have to be theistic at all, but the particular beliefs in an after-life/resurrection/omniscient-omnipotent being do resemble the theistic, and indeed, monotheistic religions), but I think there's another aspect to the "AI threat" belief that is more than just "vestigial monotheism".
I think this is (in part) a power fantasy. The believers in the AI threat normally perceive themselves to be different from other people due to their (self-perceived or real) higher intelligence. They therefore describe that property (that they believe makes them special) as being particularly dangerous, and therefore powerful, because they want to believe that intelligence imbues its possessor with power and a lot of it. Sadly, superbugs, meteorites, H bombs, climate change, human-rights violations, inequality and other threats don't require or possess much intelligence, but an unstoppable AI is the purest manifestation of intelligence-as-power.
In the human world, intelligence is not very correlated with power, except very roughly. The most powerful people are generally more intelligent than average, but the correlation doesn't extend far beyond that. If anything, charm, confidence and courage seem to imbue their bearers with much more power than raw intelligence. Which is why I fear (only partly jokingly) a super-charming machine than a super-intelligent one. A super-charming machine is far more likely to bend humanity to its will -- and therefore pose a greater danger -- even if what it wants may be truly idiotic; a super-intelligent machine would be far more limited in its effects...
A television, or a tablet with facebook and youtube apps?
Its interesting to think of mass propaganda a la Bernays as an example of "super charming machine". In the old days the cogs and wheels in the marketing business are/were humans, just like in the old days "math calculator" was a job description but it got automated and accelerated into computers.
Populist political machine might be another example.
I think this is (in part) a power fantasy. The believers in the AI threat normally perceive themselves to be different from other people due to their (self-perceived or real) higher intelligence. They therefore describe that property (that they believe makes them special) as being particularly dangerous, and therefore powerful, because they want to believe that intelligence imbues its possessor with power and a lot of it. Sadly, superbugs, meteorites, H bombs, climate change, human-rights violations, inequality and other threats don't require or possess much intelligence, but an unstoppable AI is the purest manifestation of intelligence-as-power.
In the human world, intelligence is not very correlated with power, except very roughly. The most powerful people are generally more intelligent than average, but the correlation doesn't extend far beyond that. If anything, charm, confidence and courage seem to imbue their bearers with much more power than raw intelligence. Which is why I fear (only partly jokingly) a super-charming machine than a super-intelligent one. A super-charming machine is far more likely to bend humanity to its will -- and therefore pose a greater danger -- even if what it wants may be truly idiotic; a super-intelligent machine would be far more limited in its effects...