Can AI Agents Solve the Publication Crisis in Academia?
Below is an illuminating exchange that I had before my jog this morning about the publication crisis in academia. The exchange was triggered by my recent essay titled “Academic Harassment”, posted here. The incoming email read as follows:
“Dear Avi,
I’m a Professor of Mathematics at Chalmers University of Technology.
I’ve come across your posts on Medium in connection to another question (which have only a slight intersection with the one below). What I want to discuss is your recent post about a publication.
It seems to me that most science fields have recently encountered the same problem: The academia has scaled rapidly and so has the system of publications, but we kept the same system which appears to be unfit to deal with the current scale.
Let me explain the situation in my field: There are now about 50 000–100 000 identifiable mathematicians, who publish about 100 000 papers a year. It is clear that no one can read even a significant fraction of the new publications. And I want to stress — this does not mean that most of the papers are not worth being published and read, but this means that no one is physically able to read all the worthy papers being published. Which means we really don’t have an idea if some results are repeated or have big overlap. Another problem is that all the papers have to be refereed on the unpaid time of other researchers, which does take a lot of time if one reads the paper carefully — so many editors suggest that a referee only asses if the result is interesting and further make a gut estimate if it is plausible that it is correct. This implies that now we have a lot of papers published which are probably not correct (the amount of “Erratum” every year is staggering and this is likely only the tip of an iceberg).
I did discuss with my colleagues what is the role of publication nowadays and many confess that they do not browse journals anymore but read only papers which they were referred to by colleagues they know personally (sometimes in conferences).
It seems that a non-refereed publishing on an open source would have the same effect as the refereed publishing when it comes to dissemination of the research results.
Yet it would be unthinkable for any individual to abandon the current publishing system as it now stands as a basis for evaluation of scholarly activity necessary for hiring, promotion, and funding.
This locks us in quite unhealthy situation: We have a system initially designed for dissemination of research, which is not only inefficient for its original purpose but also is used for a purpose for which it was never designed (something like using a hummer to open a car — you can smash the window and get the door open but it is not at all efficient).
I know quite a few colleagues who says that even if the system is not perfect it has been working so far, but I think there are reason to doubt that it was working ever — I remember the discussions from 20 years ago which were warning against it and they showed all to be true and give the negative consequences for the system of academic careers which were predicted with some more:
1. The assessment of a researcher is delegated to an editor (who at the best think that his/her role is an administrative one, and at the worst is biased) and an unknown referee, who is knowingly biased to his/her area of interest (and at the worst to some subgroup within the field)
2. The young researchers get additional disadvantage, as now they depend on the supervisor not only on pointing them to a research area and assistance with funding but also to point them to a more prestigious journal to publish (and often to lend them their name as a collaborator)
3. The system became so rigged that I have just heard a graduate student questioning a senior person in publishing a result on the ground that “that result would be a good publication for a student, and he denied someone the chance”. That is research became a strategic point in making an academic career rather than pursuit of knowledge. That is — one not just do research because it is interesting but also think about it “strategically” — not making a paper with a purpose to deliver knowledge but making a strategic decision about subdividing the results in “publishable” bunches. And this “strategical” consideration can make a significant impact on a career.
4. There are a number of cases where the bounds were misused — for example if one talk in CV about a paper as “submitted” to a good journal where it would have no chance to be accepted and it is counted as a “merit”, or one sends a bad paper in a sequence to a list of journals hoping that it would be accepted by mistake (which they occasionally do).
The phenomenon you have described — with biased editors is just a consequence of the problem described above: If there is a position of real power which lacks control and can be abused — it will be abused sooner or later.
I think we need to raise this question from the vague feeling that something is not quite working in academia at its cutting edge: We need to divide the system for managing knowledge (reports, papers, conference presentations, books) from the system of evaluation for the career purpose. Otherwise, in situations where secular power promotes notions that are no different than religious beliefs — the academy will collapse (we actually can see it in connection with the other question you were discussing on Medium).
Unfortunately, I do not have a suggestion either of a system which would work, not how to get the change in practice. There are other system of evaluation: For example, in France all applications for academic positions and promotions are centralized, which means instead of every university considering every applicant on its own — which is in practice a multiple of universities per applicant of assessments to be made — they consider every applicant just once and let the departments use this assessment to choose their new hires. A system which was in place in the 14th century was an open competition for a position where participants had a day to solve problems in their field suggested to them by competitors (just to widen the scope of possibilities). If one values today’s networks within the research community, then one probably can return to the system of recommendation letters…
I think we, as members of academia, do not have the luxury option of just digging ourselves into our research and hoping that someone else will solve this problem.
Best regards,
My response was as follows:
“Dear Maria,
You bring up excellent insights into a multifaceted problem. I agree with everything you said. If I have the power to make a change, I will do it. But as you know, a broken system takes a lot of time to repair because there are always people who gain their power from it.
One solution that came to my mind while reading your comments is that perhaps we can use AI for help on this. AI agents can digest the large number of papers and potentially review or organize them for us. Perhaps we should develop an AI agent that is trained to do that and see how well it does in successfully assessing papers that are verified to be good or bad.
Avi”
ABOUT THE AUTHOR
Avi Loeb is the head of the Galileo Project, founding director of Harvard University’s — Black Hole Initiative, director of the Institute for Theory and Computation at the Harvard-Smithsonian Center for Astrophysics, and the former chair of the astronomy department at Harvard University (2011–2020). He is a former member of the President’s Council of Advisors on Science and Technology and a former chair of the Board on Physics and Astronomy of the National Academies. He is the bestselling author of “Extraterrestrial: The First Sign of Intelligent Life Beyond Earth” and a co-author of the textbook “Life in the Cosmos”, both published in 2021. The paperback edition of his new book, titled “Interstellar”, was published in August 2024.
Professional website:
https://lweb.cfa.harvard.edu/~loeb/
Social media:
https://avi-loeb.medium.com/
https://www.youtube.com/@ProfessorAviLoeb
https://open.spotify.com/show/1zhndXkvSY2b8FdjspFpCd
https://x.com/ProfAviLoeb
