I think studying this stuff is always going to seem mysterious unless you account for the concept of fashion in science. Specifically what I mean is that two papers (or ideas or approaches) X and Y can have equal “objective scientific merit” but X is more popular than Y because of random initial conditions (e.g., a famous researcher happened upon X first and started mentioning it in their talks) that are self-reinforcing. The root cause of this phenomenon is that most/all researchers can’t justify what they work on from first principles; for both good and bad reasons, they ultimately rely on the wisdom of the crowd to make choices about what to study and cite. This naturally leads to big “flips” when a critical mass of people realize that Y is better than X, and then suddenly everyone switches en mass.