It is easy to check one reference. It is not easy to check all the reference, and read the whole article, and check each paragraph, and check that there is no flaw in the logic, and check that there is no hypothesis missed, and check that ...
My point is that when you peer-review, you usually choose your priorities (unless you don't do much work next to that). Sure, it's not good that the review of the references was not done perfectly. But it's just unrealistic to think that references are always reviewed perfectly and that if you spot one article with badly reviewed reference, it's the proof that the reviewers were part of an ideological conspiracy. It's a survivor fallacy: you see here a bad article where the reference is wrong, but you don't see all of the work of checking everything else. If you were presented with another bad article where the most obvious flaw was something else, you would have said "the reviewers are so bad, checking this particular element would have me taken 5 minutes" and you would not even have considered to check the references yourself for this particular example.
Honestly, checking the content of the reference is a low gain job with respect to the effort. Because citing a reference incorrectly is very risky (there is a strong probability the authors will notice, and it looks very very bad when they tell it then), it is pretty rare that checking the reference really lead to a correction, and it is not unusual that the reviewers assume that the authors are of good faith and missed something more subtle rather than got it so wrong. It does not mean it does not happen, and again, in an ideal case, the reference should be checked. But there are plenty of things that can be missed during a review, and checking everything is a real effort that is usually not worth it.
> The reference author isn't going through every work that cites theirs.
That seems stupid: they are building work around a specific subject. If they are going to read one article, it will be an article on a related subject. And what better relation than a relation that lead to a reference?
In other words: what are these authors doing if they are not reading the articles that talk to related subject? Can you even call those authors experts: they have read few article, they wrote one article, and since then, they have totally abandon their research and ignored all the new elements that bring a more precise and advanced light on the subject.
It's also a strange argument: in one hand, you say it's easy to read the article that need to be reviewed + all the article that are in its reference. In the other hand, you are explaining that it's too much work for an author to read one article.
And also, it's not only reading: 90% of research is about debating between researchers. That's the whole point of conferences and seminars. Either the reviewed article is totally ignored, in which case who care if the references are wrong, or it is not ignored, and the most it is not ignored, the most it is exposed to people for which an incorrect reference will be obvious.
> The peer-review is the last step before submitting the article to the whole community.
Exactly. And what is the point of submitting the article to the whole community: create the debate and the discussion, bringing new arguments that are then being disputed, refuted, ... until we get a new position that we trust.
The point of peer review is a basic first filter. The goal of this filter is excluding articles that, at 100%, are not worth the time to be discussed (or not yet worth the time). But this filter NEEDS to let some bad articles pass. The goal of this filter is to be 100% sure that the article is not worth before rejecting it. But as long as the rate of articles is manageable, then there is no reason to make this filter too stringent. It's just dangerous: rather have 100 bad articles passing peer-review than 1 good articles failing peer-review.
> So you shouldn't look into questionable claims because sometimes false claims sound perfectly believable?
It is not what I'm saying. What I'm saying is that you should not bias yourself by your preconception: be critical of EVERYTHING, it does not matter if it sounds correct or sounds incorrect. Sounding correct or sounding incorrect is a really really bad way of knowing if something is correct or incorrect. It even creates unconscious biases where we end up thinking something false should be treated as if it was proven.
My point is that when you peer-review, you usually choose your priorities (unless you don't do much work next to that). Sure, it's not good that the review of the references was not done perfectly. But it's just unrealistic to think that references are always reviewed perfectly and that if you spot one article with badly reviewed reference, it's the proof that the reviewers were part of an ideological conspiracy. It's a survivor fallacy: you see here a bad article where the reference is wrong, but you don't see all of the work of checking everything else. If you were presented with another bad article where the most obvious flaw was something else, you would have said "the reviewers are so bad, checking this particular element would have me taken 5 minutes" and you would not even have considered to check the references yourself for this particular example.
Honestly, checking the content of the reference is a low gain job with respect to the effort. Because citing a reference incorrectly is very risky (there is a strong probability the authors will notice, and it looks very very bad when they tell it then), it is pretty rare that checking the reference really lead to a correction, and it is not unusual that the reviewers assume that the authors are of good faith and missed something more subtle rather than got it so wrong. It does not mean it does not happen, and again, in an ideal case, the reference should be checked. But there are plenty of things that can be missed during a review, and checking everything is a real effort that is usually not worth it.
> The reference author isn't going through every work that cites theirs.
That seems stupid: they are building work around a specific subject. If they are going to read one article, it will be an article on a related subject. And what better relation than a relation that lead to a reference?
In other words: what are these authors doing if they are not reading the articles that talk to related subject? Can you even call those authors experts: they have read few article, they wrote one article, and since then, they have totally abandon their research and ignored all the new elements that bring a more precise and advanced light on the subject.
It's also a strange argument: in one hand, you say it's easy to read the article that need to be reviewed + all the article that are in its reference. In the other hand, you are explaining that it's too much work for an author to read one article.
And also, it's not only reading: 90% of research is about debating between researchers. That's the whole point of conferences and seminars. Either the reviewed article is totally ignored, in which case who care if the references are wrong, or it is not ignored, and the most it is not ignored, the most it is exposed to people for which an incorrect reference will be obvious.
> The peer-review is the last step before submitting the article to the whole community.
Exactly. And what is the point of submitting the article to the whole community: create the debate and the discussion, bringing new arguments that are then being disputed, refuted, ... until we get a new position that we trust.
The point of peer review is a basic first filter. The goal of this filter is excluding articles that, at 100%, are not worth the time to be discussed (or not yet worth the time). But this filter NEEDS to let some bad articles pass. The goal of this filter is to be 100% sure that the article is not worth before rejecting it. But as long as the rate of articles is manageable, then there is no reason to make this filter too stringent. It's just dangerous: rather have 100 bad articles passing peer-review than 1 good articles failing peer-review.
> So you shouldn't look into questionable claims because sometimes false claims sound perfectly believable?
It is not what I'm saying. What I'm saying is that you should not bias yourself by your preconception: be critical of EVERYTHING, it does not matter if it sounds correct or sounds incorrect. Sounding correct or sounding incorrect is a really really bad way of knowing if something is correct or incorrect. It even creates unconscious biases where we end up thinking something false should be treated as if it was proven.