First off that's theoretical. No damage of that scale has been done by an LLM yet. Second off nobody really believes this. That's why there's no age limit for LLM usage and there is for gun usage. Would you let your 10 year old kid play with a hand gun or chatGPT? Let's be real.
I agree with your pragmatic approach. LLMs are a "high magnitude" advancement but we can't really correlate that with "severe destruction" in a physical way - maybe in a theoretical or abstract way.
Kind of reminds me of the whole "dihydrogen monoxide kills so many people per year" parody
