The Curious Case of the Declining Disruption’s Disappearance

The study is based on the CD index, a citation-based measure of how much the successors of a work still rely on its predecessors, running from −1 (consolidating) to +1 (disruptive). Key detail: a paper with zero references automatically gets CD = +1 as soon as it is cited once.

Due to a (then unreported) bug in the seaborn plotting software, the published histograms silently dropped the largest data points. A peak of papers with the maximum disruption score of CD = +1 was hidden from view, while these papers were kept in the analysis.

When removing the hidden outliers with CD = +1, the reported decline in disruption largely disappears across all databases used in the Nature study. For Web of Science, it reduces by 93%.

Why so many papers with a CD index of exactly 1? In the open-source SciSciNet database, 97% of them make zero references. Of 100 randomly sampled source documents, 93 do contain references. They are dataset artefacts.

Why didn't the robustness checks catch this? The linear regression did not control for the discontinuity of the CD index at zero references. Simply adding a dummy for zero references improves the adjusted R² from 0.15 to 0.95, and the adjusted CD index becomes essentially flat.

The Monte Carlo simulations randomly rewired citations while preserving each paper's number of references. Zero-reference papers are thus preserved, and the rewired networks inherit the peak at CD = +1. Their decline mirrors the observed decline in disruption.

Instead of a decline in the disruptiveness of scientific and technological knowledge, the data mainly show an increasing metadata quality over time: relatively fewer papers with zero references due to database errors.

Our Matters Arising flags a factual plotting mistake in Park et al. (2023) and presents the corrected figure. We then identify that the outliers that were hidden from view in the published histogram are the major drivers of the reported decline in disruption. Moreover, we show that they largely correspond to dataset artefacts mainly pertaining to older publications. Our Matters Arising also shows why the main robustness checks in Park et al. (2023) did not catch the effect of this particular data quality issue.

Our Matters Arising was not intended to:

• discuss the very concept of disruption in science;
• discuss the validity of the CD index;
• defend any particular regression analysis;
• analyse the meaning of the remaining signal.

Following best-practice guidelines recommended by Leibel & Bornmann (2024), we even saw increasing curves in disruption. This analysis can easily be reproduced from our shared data. In our interpretation, the remaining trend is therefore not stable.

About this figure

This figure displays that justifiable pre-processing choices can lead to a slight increase in disruption across all research areas, an observation that persists even after controlling for potential confounders with a multivariate regression analysis. More specifically, we consider the SciSciNet database and analyse the recomputed CD₅ indices, with a forward citation window of five years, where we focus on citations between journal articles. We subset the database to include only papers with at least ten references and ten forward citations (n = 5,621,075). a, The temporal evolution of the CD index reveals a slight increase in disruption across all research areas. b, Even after controlling for potential confounders, such as the log-transformed number of references, the slight increase in disruption persists and is statistically significant.

Leibel, C., & Bornmann, L. (2024). “What do we know about the disruption index in scientometrics? An overview of the literature.” Scientometrics, 129(1), 601–639.

The first-order analysis (showing the peak in the CD index distributions and the consequences of its removal on the reported decline in disruption) was demonstrated across all databases used by Park et al., including Web of Science. For the second-order analysis on the origin of the hidden outliers, we needed access to the number of cited references. Therefore, this analysis was conducted for the open SciSciNet database, which is a successor of the Microsoft Academic Graph, a database used by Park et al. (2023) themselves.

Additionally, in our supplementary information (Fig. S17), we also reconstructed the robustness checks for Web of Science, showing that they did not detect the consequences of the hidden outliers (see the reply).

As mentioned in our acknowledgements, in response to our initial submission, Park et al. highlighted the different robustness checks in their manuscript. When we showed that both the regression adjustment and the Monte Carlo simulations did not control for the effect of the hidden outliers, they claimed that document-type settings in SciSciNet affected the results. When we showed them it didn't, they simply ignored this (see the children's books question). Importantly, every analysis that could directly test the artefact explanation for Web of Science is absent from their reply (see the reply). This goes as far as not sharing a critical robustness check figure for Web of Science, despite indicating during peer review it was available upon request. We therefore had to recreate the figure. Every iteration took many months. During the 32 months, Brandolini's law really took its toll on us.

The mistake was discoverable at all because Nature requires open replication data and code, and because the open-source seaborn community documents its bugs in public. Shoutout to Michael Waskom for maintaining seaborn and quickly fixing the bug! From there, however, the process ran into closed doors. Web of Science, the largest data source in the study, is proprietary, so no public metadata existed to trace where the outliers came from. We turned to SciSciNet, an open database built on the same source the original authors also used, to verify our findings independently. Interestingly, a crucial piece of information was a figure made by the authors of the original paper during peer review that was “available upon request”. We requested it several times, but never got it.

Three concrete changes could follow from this case. First, timing: our critique of a factual, uncontested (!) mistake took 32 months to publish, against at most 17 for the original cover story. During that window the article gathered over a thousand citations, an editorial built on its conclusion, and appearances in a US congressional hearing, all without any notice on the paper. Corrections should be treated as a priority for the journal, and readers should be able to see that a documented critique is pending.

Second, journals need explicit norms for replies to critiques. A reply that abandons the evaluation methods of the original paper is not a defence of that paper, but a different paper. Every claim in a reply should be demonstrated on the data at hand: either show the criticism fails on your own data, or show the alleged flaws in the critic's data actually bias the result.

Third, “available upon request” should mean something. Materials cited in peer review to satisfy referees must actually be shareable, and refusing a request should carry consequences. Corrections, finally, need to propagate. The literature and policy documents that cite a contested finding should learn of the correction.

One central tension in the exchange is that Web of Science is a closed-source database. Park et al. publicly shared raw CD-index values, which allowed us to identify that the published histogram contained a plotting mistake hiding a peak at CD = +1, and that removing those outliers largely negates the decline in disruption. However, in our exchange, only Park et al. had access to Web of Science metadata such as the number of cited references. Yet their reply omits the analyses that would directly test whether zero-reference records and the CD₅ = +1 peak drive the trend that was initially reported:

• The original raw CD₅ curves after excluding zero-reference papers.
• The reference-count distribution among Web of Science papers with CD₅ = +1.
• The relative frequency over time of papers with zero references and CD₅ = +1.
• A source-document audit sampled from the Web of Science zero-reference category.
• The observed and rewired raw CD₅ trajectories for the original Web of Science network plotted next to each other, which Park et al. described in peer review as “unreported analyses, available upon request”.

In the publicly available peer-review files of their original paper, Park et al. write:

“Consequently, if the decline in the CD5 index is driven primarily by changes in these structural factors, then we should see a decline over time in the CD5 index values obtained from the rewired networks that is comparable to the decline in the observed networks (because the number of citations made, the number of citations received, and the ages of the citing and cited works are identical in the observed and rewired networks). Consistent with the intuition sketched out above, we do indeed find (in unreported analyses, available upon request) that the values of the CD5 index in the rewired networks declines over time, suggesting that some component of the decline is attributable to structural changes in the citation networks.”

We requested those raw curves repeatedly. Park et al. refused to share them and to this day have not provided their version. We therefore reconstructed the figure ourselves. That reconstruction process was described in detail during peer review and is documented in our data availability statement, reporting summary and the caption of Supplementary Fig. S17. We find it remarkable that the figure “available upon request” is still not available, as well as the audacity to frame it as a transparency concern from our side.

About this figure

This figure displays that the inclusion of the zero-reference dummy variable in the regression adjustment reduces the decline for the Web of Science data source by 80% (with 24,659,076 papers, of which 22,479,429 have a defined CD₅ index) and that randomly rewired citation networks exhibit a similar temporal decline of CD₅ across all Web of Science research areas (for a 10% random subsample of papers). To recreate the figure, we used all the relevant data publicly deposited by the authors across different versions of their original manuscript and during our correspondence. More precisely, we used aggregate values obtained either as source data from the Nature website (curves labeled “original,” respectively) or by digitizing figures from (Funk, 2024; Park et al., 2021) using WebPlotDigitizer. We then plotted these data points either directly (panel a) or through straightforward arithmetic combination, that is by multiplying the ratio (average rewired CD index) / (average observed CD index) with the average observed CD index (panels b–e). This ensures identical preprocessing steps.

Funk, R. J. In support of our claim that heterogeneity in the data is driving H et al.’s findings, when we run H. . . [Image attached] [Post]. X (2024). Posted February 26, 2024. Retrieved August 30, 2024, from https://x.com/russellfunk/status/1761950261671526618.

Park, M., Leahey, E. & Funk, R. J. The decline of disruptive science and technology. arXiv preprint arXiv:2106.11184v4 (2021).

Every children's book listed by Park et al. is classified as ‘Book’ in SciSciNet. We remove books in the robustness check that restricts the sample to journal and conference contributions. The result remains the same: adding the zero-reference indicator reduces the adjusted decline by 67%, from −0.15 to −0.05. These robustness checks have been available to Park et al. for more than two years.

Restricting only to citations between journal articles, and excluding Humanities, gives the same answer. Removing CD₅ = +1 reduces the decline by 97%, and removing zero-reference records reduces it by 85%. The books therefore do not affect the result. Both robustness checks can be found in our supplementary information.

Presenting children's books as if they invalidate our results for SciSciNet is simply misinformation, expressed in condescending language that is not worthy of a journal like Nature.

One final embarrassing detail: Captain Underpants and the Preposterous Plight of the Purple Potty People does not receive a citation within five years after its publication, so its CD₅ index remains undefined. We did not compute it.

About this figure

This figure replicates the observation that papers with CD₅ = 1 drive the decline in disruptive science for the SciSciNet data source, irrespective of document-type settings. Results are shown for two specifications of the SciSciNet data source: the left panel includes all document types (a; n = 39,888,199 papers) and the right panel shows the raw journal data (b; n = 30,869,742 papers; CD₅ indices are computed using only citations between journal publications, excluding contributions from art, history, and philosophy). Shaded bands correspond to 95% confidence intervals.