> The trouble is that the outputs of these models, by their very nature, look convincing.
Yes, this is pretty much what LLMs are designed to produce, and no more. This is why I say they are not HAL, just a better MegaHAL.
There was an Asimov short story called "Liar!" about a robot whose ability to read minds, combined with its First Law directive, always told people what they wanted to hear so as to avoid causing emotional harm to humans. (When confronted with the idea that by telling people falsehoods it was bringing harm to them, it simply stopped functioning.) LLMs can't read your mind, but they do choose their words based on a statistical model they have of what you might expect given what's been said before. Facts and logic be damned if they don't fit that model.
Yes, this is pretty much what LLMs are designed to produce, and no more. This is why I say they are not HAL, just a better MegaHAL.
There was an Asimov short story called "Liar!" about a robot whose ability to read minds, combined with its First Law directive, always told people what they wanted to hear so as to avoid causing emotional harm to humans. (When confronted with the idea that by telling people falsehoods it was bringing harm to them, it simply stopped functioning.) LLMs can't read your mind, but they do choose their words based on a statistical model they have of what you might expect given what's been said before. Facts and logic be damned if they don't fit that model.