Horrible article telling bright eyed prospective grad students to fellate the prestige obsessed academic politics machine rather than actually try to advance science.
Remember that Einstein told journal editors to piss off [1] when they tried to get his papers peer reviewed.
I happen to be an (associate) editor of an academic journal.
What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication.
We need some process to gate-keep.
a) The venue of publication is a good signal, whether the time to read a paper is well-spent. b) The PhD students learning the craft need objective feedback. The supervising professior/university often has the incentive to "just submit" -- even if they know that a publication does not meet the quality standard.
Before peer-review, somebody also needed to make a decision on what to publish. This typically fell to a single individual. The editor or some well-known member of the community who could recommend a paper for publication. On old journal issues they even mention the "recommender".
So the question is not whether peer-review is bad, the question is which alternative gate-keeping process would be better. Otherwise we will drown in crap publications (even more) and the PhD students don't get a honest feedback signal upon which they can improve their craft.
As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that
> We need some process to gate-keep.
Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore.
Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community.
Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1].
Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed
> The venue of publication is a good signal, whether the time to read a paper is well-spent
Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities.
The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.
The Adam optimizer was published at ICLR, a top conference in machine learning. So, basically, analogous to a peer-reviewed journal for the purposes of this discussion. ML (and some other subfields of computer science) have the particularity that the really comptetitive gatekeeping happens even more in conferences than in journals.
That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review.
If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance.
PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.
> The Adam optimizer was published at ICLR, a top conference in machine learning.
Fair, must have misremembered that one.
> If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact.
I disagree on this one. I did my PhD at an institution that ranks in the top 10 in the most well known university rankings, and I distinctly remember that one time when I was submitting a manuscript to a prominent journal in my field, got some reviews back which weren't positive yet were quite valid criticisms, and my professor told me not to worry because the editor is his buddy and my manuscript will get published for sure.
When that sort of "scratch my back and I'll scratch yours" culture exists in journals I don't see how peer review can be an equalizer. It just means everyone who publishes at a journal, including the less well esteemed ones, can claim they went through the rigor of peer review. Of course, we all know peer review is just a vibe check and is actually not that rigorous at all, and besides no one cares unless you published in a prestigious journal anyway. The less revered journals exist to collect $5k in open access fees for the publisher in return for hosting a pdf at the marginal cost of maybe a cent a year.
It seems to me that something like eLife's model is the best solution to this [0]: you still have a minimal amount of curation, but generally if a paper is written well enough and within the field it won't be desk rejected. Then, it gets published on the site and sent off for peer review. Peer reviewers assess how sound the paper is and pass a judgement which readers can view, as well as provide some recommendations to the authors make it stronger. The authors can then either revise the paper, or do nothing at all. In either case, papers don't languish in reviewer hell and the larger scientific community gets to see it.
Exactly, everybody can publish on arXiv.
And there are enough semi-predatory journals/conferences which basically accept everybody. Especially since the LLM can rewrite any paragraph nicely.
So the role of journals and conferences is not to prevent "the word getting out". It is to provide value by a curated list of on-topic and high-quality publications.
So no need to wade through tons of crap.
Especially for PhD students who might take more time to detect crap as such.
In my experience, the publications at the good venues get a lot more eye-balls and by consequence citations. So there seem to be a lot of people who like this role as a "filter" for what to focus on.
> Someone putting a crap article on arxiv does not hurt anyone else
> ... only reason we need to gatekeep today is to feed the academic prestige and politics machine
This to me says you have may not have experienced some parts of the (long-term) research process. It suggests that you have infinite resources to filter out noise, which is probably not the case. It suggests you're willing to spend a lot more time figuring out why something doesn't quite make sense, rather than get to the heart of the problem, while this is fine in many cases it sucks when you're hot on the trail of something interesting, and you're slammed by a million twisty paths full half-baked hot-takes.
We need to filter ("gate-keep" is pretty inflammatory term) information and processes so that we don't have 12 different screw types with 12 different electric screwdrivers, instead of "just" 6 (sigh). We need to come to consensus and that means some things go in and something are left out. We need many mechanisms to filter.
> tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.
All of these things feel like filters, when does a filter become a gate: colleagues - i.e. not everybody, but some selected few, who and how?; conference - filter (well, gate!); scientific community - != your baker's community; decisions directed by you, not on my own (i.e. a pointer to my paper) - filter.
Conference does not imply gatekeeping. There are many conferences out there which accept almost all submissions.
As for the "scientific community" being a filter, there is a difference between "elite" researchers being the ultimate arbiters of scientific truth via their positions in the editorial teams of journals versus everyone being allowed to publish on open platforms like arxiv and bad ideas/quackery being filtered out naturally.
Because the former is what makes or breaks a scientist's career, grad students and postdocs hyper optimize to publish at prestigious venues, as opposed to optimizing for doing science. These two are aligned only sometimes.
Per Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." [1]
I think it's straightforward to make an argument that many of today's conferences are as bad as journals, accepting submissions is only one way conferences filter. IRL they are prohibitively expensive to enter, let alone attend (but again, see "Zoom"), and therefor eliminate all but the elite, they are run by commercial entities in all cases with more than ~200 people, they are more or less required venues for networking and therefor selling yourself for a tiny chance at academic permanence, they give plenaries to elites (filtering to one voice), they have special symposia by invite only, with other submissions dumped to inaccessible parallel sessions (which one will you choose to see?), the submissions you make are published in much more ephemeral ways, and tend to be more difficult to discover in the long term, making the event important but the research not as much (at least in my experience), etc.
Yes, there are many scam conferences that don't do any peer review. They are a huge problem. They waste researchers time and money and exist only to extract dollars from people, not to advance science.
> What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication.
Publication is antiquated. HN doesn't need reviewers to boost the best content or to provide commentary on how to improve a paper or fact check its contents. Join the 21st century.
We are on the cusp of being able to plow through vast quantities of literature and data in an instant using multimodal machine learning models. Journal articles are written for people. The landscape is changing. We are headed toward a future where scientist upload data and thoughts en masse into the cloud to be consumed by interpreter models that in turn feed back into the scientific machine. Data quality and attribution (scientist performance rating) is automatically allocated by models.
The post made some points and you’ve said it was wrong to make them. Who am I supposed to believe? I would like to understand more about your position, but you’ve chosen not to actually address any of their points.
Sorry, don’t mistake my tone for condescension. I just wanted to explain how this looks to an outside observer.
The post is telling young researchers to ignore the main reason they went into research - scientific discovery (what the author calls "Science 1") - and focus on playing the academia prestige pissing contest game ("Science 2") in a cynical career optimization move.
Academic politics always existed of course. We should not be under any illusions that the greats of the past just wrote their manuscripts in isolation. However it was not an industrialized machine like it is now, and incentives are misaligned in academia to such a degree that the machination of academic politics killed the reason why academia exists in the first place (scientific discovery).
To advance in the academic cursus honorum, one goes to a presitgious undergrad to go to a prestigious grad school, so you can get a prestigious postdoc grant, so you can get a tenured position at a prestigious institution, so you can get a fat and prestigious government grant, which you use to hire bright young students who want to do the same. Note that scientific advancement does not play a role in this cycle, it's actually safer to pursue incremental and irrelevant improvements which you get published through thanks to the connections you made throughout your prestige optimization career.
As a result, academia has produced no notable scientific advances in a long time. It has instead evolved into an organism which selects individuals that excel at funneling money into itself under the guise of doing science while not necessarily doing science (though it incidentally occasionally may).
The kind of behaviour the author is promoting is telling individual prospective grad students you're small, the academic politics machine is so big, yes we all know it's a farce but you just need to suck it up and play the game. In doing so, the prospective grad students will strengthen the machine that is actually killing the very thing they want to cherish and promote, in the hope that they receive a few scraps in return.
(Writing all this as a PhD and former journal reviewer)
Hot take, the elephant in the room is that the firehose of easy discoveries has run dry in most physical sciences. Accordingly, the academic community is probably an order of magnitude oversized, but there will be little incentive to scale itself down. As a result you see an academic machine idling.
Invention is a slow process filled with periods where progress feels like it has stalled. Then, we have a breakthrough, sometimes major, other times incremental. Seeing it in hindsight is easy, living through it is hard as we measure our experiences in seconds.
There's a pragmatic issue involved: A PhD student faces a high risk of getting knocked off of their track and leaving without a degree due to circumstances entirely outside of their control. Delays are super costly, both financially and by adding to the risk. This is a hugely asymmetric burden, and overshadows all other considerations. For this reason, the top three priorities of the PhD student are:
I would extra: avoid any activities which does not add any value to your thesis and obviously weight it against potential conflicts. Having PhD year earlier is better value, than year later with "better experience" (depends on situation, but having written thesis makes things little easier)
Indeed and if you feel that theres more work to be done along the lines of your thesis, funded advisors will often let you stay on for a while as a post doc. This can also help if you end up finishing without a job offer.
Remember that Einstein told journal editors to piss off [1] when they tried to get his papers peer reviewed.
[1] https://theconversation.com/hate-the-peer-review-process-ein...