Agreed. Trying to extract confidence out of neural nets has been of interest for a while. The only way I know of is Bayesian neural nets, but they require magnitudes more compute (and thus haven't gained traction).
And unfortunately seem to be difficult to train as well!
Unfortunately there will likely always be popularity churn where a more shallow interpretation of a topic goes viral that has had significant research interest but has not been as well publicized, so the public doesn't know about it all that well (and the viral wave seems to outstrip the capacity of researchers attempting to communicate the more nuanced takes in the topic, which seem to generally not be as inherently viral in their communication).
Hey! We have just published a review and benchmark of different uncertainty estimation techniques [1], it might be interesting to you if you want to get a general understanding of works and what doesn't in the specific case of LMs.
