[edit: I previously said I didn't understand Alex's point.] I now understand the point you were trying to make. A better way to put it - if a random number generator were used in a decision process, I'd favor making the algorithm and random seed explicit.

Any procedure you use will have assumptions. You can't escape this. The only question is whether we show or hide them. Can you give an argument in favor of hidden assumptions and non-explicit procedures?

> Can you give an argument in favor of hidden assumptions and non-explicit procedures?

So as counterintuitive as it sounds, I think there are actually a couple of good arguments that can be made here:

1) With significance testing, the burden of supplying the assumptions and determining meaning is largely on the reader. With bayesian, it's transferred to the author. While it might make sense to use Bayesian for things like the Cochrane report, it's not obvious to me that each person who designs a research study and collects/analyzes data should also be in the business of trying to say whether some phenomena is real when looking at all other studies.

Essentially each study now becomes a metastudy, with all of the practical and epistemological problems that entails. The fact that it's difficult to figure out what that even means should be a red flag. (And yes, I realize this is the Chewbacca defense.)

2) So TokenAdult actually turned me onto this book Measurement In Psychology, which is all about the epistemological problems with assuming that anything you can assign a number to is a measurement. That is, having the property of being meaningful when interpreted on a ratio scale. The exact argument is kind of esoteric, but the basic takeaway is that it's very easy to trick yourself into thinking that just because you can assign a number to something that it's a measurement, to the point where assigning numbers to things in the first place tends to lead to worse decision making than if you had just used a green/yellow/red system or whatever.

Regarding (1), with significance testing the burden of supplying assumptions is not placed on the reader. The assumptions are implicitly built into the NHST rather than explicitly built into the prior.

As for each study becoming a meta-study, that's silly. This is indeed the chewbacca defense. Rather, each empirical study provides Bayes factors which the reader can then use to update their posteriors.

Regarding (2), obviously not every number is a measurement. In Bayesian stats, numbers representing probabilities are quite explicitly opinions. They are meaningful on a ratio scale, and are even asymptotically known to be correct. But they aren't measurements.

(They are correct if your priors are absolutely continuous w.r.t. reality. If you hold a religious belief so strong that evidence can't change it ("100% certainty"), that's not an absolutely continuous prior.)