The exponential growth in new scientific and technological knowledge creates a fertile foundation for major advances. However, a new Nature paper discovered a steady drop from 1945 through 2010 in the fraction of disruptive advances within science and technology, with progress slowing down in many major fields. The study examined 45 million scientific papers and 3.9 million patents and found that investigators and inventors have recently made fewer breakthroughs and innovations relative to the growing volume of science and technology research. The authors found a steady drop in the fraction of disruptive discoveries, suggesting that scientists today are more likely to make incremental progress than advance in quantum leaps. “Eureka!” moments that change everything known within a field, are getting rarer than those associated with incremental progress.
The new analysis differentiates between routine work and true breakthroughs by tallying citations not only to the research being pursued but also to the previous studies it cites. Past work is cited far more often if the finding is routine rather than groundbreaking. The new measure, so-called the CD index, characterizes how papers and patents change networks of citations and calibrates the transition from consolidating to disrupting the body of existing knowledge. The study finds that papers and patents are increasingly less likely to break with the past in ways that push science and technology in new directions. This pattern holds universally across fields and is robust across multiple citation metrics.
In particular, the average CD index declined by more than 90% between 1945 and 2010 for research papers and by more than 78% from 1980 to 2010 for patents. Disruptiveness declined in all of the analyzed research fields and patent types, even when factoring in potential differences in citation practices.
The trend is also apparent in the language being used. In the 1950s, research papers were more likely to use words associated with creative work such as ‘produce’ or ‘determine’, whereas papers in the 2010s were more likely to refer to incremental progress using terms like ‘improve’ or ‘enhance’.
The new study views the most common “discoveries” at present as representing routine science, and true leaps in knowledge as sometimes missing altogether from the research lexicon. It is therefore not surprising that the Nobel prizes in physics over the past decade include the discovery of the Higgs boson, the observation of gravitational waves, the confirmation of black holes and the detection of quantum entanglement, all of which validate old theoretical predictions from the past century but do not break unexpected new grounds.
One explanation for this slowdown is a dearth in ‘low-hanging fruit’. Another explanation suggests that scientists and inventors require ever more training to reach the frontiers of their fields, leaving less time to push those frontiers forward.
Yet, another possible cause for this trend involves the increase in the number of scientists. This enhances societal pressures toward groupthink and the specialization in niches with narrow fields-of-view. Large research teams have become more common and are more likely to produce incremental rather than disruptive science.
In the context of fundamental physics, there might be more to this trend as a result of a lack in anomalous experimental data that drives innovation. The cancellation of the superconducting super collider in 1993 by the US Congress, led to a long hiatus of decades with no new experimental data in high-energy physics, creating a new culture within the mainstream of theoretical physics that is divorced from experimental tests. Could this be another reason for the decline in disruptions within physics?
In a podcast interview yesterday, I discussed the huge pushback that I received to a simple suggestion in a published paper that the first reported interstellar object, `Oumuamua, was artificial in origin based on its observed anomalies. My proposal was not triggered by pure thought but represented a disruptive interpretation of astronomical data about multiple anomalies that this object exhibited, which separated it from familiar asteroids or comets in the solar system. The data implied that some interstellar objects may have a different origin than solar system rocks. The potential disruptive nature of my proposal was challenged by experts on space rocks. They insisted that `Oumuamua must be of natural origin but concluded that even in that case — it should be an object of a type never seen before, such as a hydrogen iceberg, a nitrogen iceberg or a rarefied dust bunny. Subsequently, I discovered with my student, Amir Siraj, two interstellar meteors that predated `Oumuamua and were tougher in material strength than all known meteors from the solar system. If these interstellar meteors were hydrogen or nitrogen icebergs or a rarefied dust cloud, they would have evaporated much higher up in the Earth’s atmosphere.
Disruptive interpretations cause anxiety among experts, because they question the completeness of their knowledge and threaten their academic stature and authority. This is in contrast to the great benefit offered by imagined realities, which could always flatter our ego because they are not disruptive. By tailoring imagined realities to our liking, we can always show off as knowledgable and avoid experimental tests that may prove us wrong.
Consider, for example, the notion of the multiverse. It was characterized by the pioneer of cosmic inflation, Alan Guth, as a realm where “everything that can happen will happen an infinite number of times.” This represents the ultimate paradise where no hypothesis can be ruled out because everything is possible. Similarly, string theory does not make a unique prediction for the value of the cosmological constant. As of now, the theory allows for 10 to the power of 500 or perhaps even 10 to the power of 272,000 possible values for the cosmological constant. These big numbers are indistinguishable from infinity for all practical purposes. For theoretically-inclined physicists, the benefit of a theory that allows everything is that it cannot be ruled out by data that proves them wrong.
But when there is no experimental option for ruling out a theory, there is no opportunity for learning something new about reality. This eliminates the potential for a disruption in our body of scientific knowledge because we tend to imagine what we already know.
ABOUT THE AUTHOR
Avi Loeb is the head of the Galileo Project, founding director of Harvard University’s — Black Hole Initiative, director of the Institute for Theory and Computation at the Harvard-Smithsonian Center for Astrophysics, and the former chair of the astronomy department at Harvard University (2011–2020). He chairs the advisory board for the Breakthrough Starshot project, and is a former member of the President’s Council of Advisors on Science and Technology and a former chair of the Board on Physics and Astronomy of the National Academies. He is the bestselling author of “Extraterrestrial: The First Sign of Intelligent Life Beyond Earth” and a co-author of the textbook “Life in the Cosmos”, both published in 2021. His new book, titled “Interstellar”, is scheduled for publication in August 2023.
