6 Dec (NATURE) – The US National Institutes of Health (NIH) is the largest funder of biomedical research in the world. Between 2002 and 2011, it issued around 460,000 research grants totaling almost US$200 billion. The NIH has unquestionably propelled numerous medical advances and scientific breakthroughs, and its funding makes much of today’s scientific research possible.
However, concern is growing in the scientific community that funding systems based on peer review, such as those currently used by the NIH, encourage conformity if not mediocrity, and that such systems may ignore truly innovative thinkers. One tantalizing question is whether biomedical researchers who do the most influential scientific work get funded by the NIH.
The influence of scientific work is difficult to measure, and one might have to wait a long time to understand it. One proxy measurement is the number of citations that scientific publications receive. Using citation metrics to appraise scientists and their work has many pitfalls, and ranking people on the basis of modest differences in metrics is precarious. However, one uncontestable fact is that highly cited papers (and thus their authors) have had a major influence, for whatever reason, on the evolution of scientific debate and on the practice of science.
To explore the link between highly cited research and NIH funding, we evaluated scientists who have published papers since 2001 — as first, last or single authors — that have so far received 1,000 citations or more. We found that three out of five authors of these influential papers do not currently have NIH funding as principal investigators. Conversely, we found that a large majority of the current members of NIH study sections — the people who recommend which grants to fund — do have NIH funding for their work irrespective of their citation impact, which is typically modest.
There are probably many reasons why highly cited scientists do not have current funding. They might have changed careers or moved to industry, for instance. Perhaps they are receiving some funding as co-investigators, or are still young and have just started their own lab. But the NIH’s mandate is to fund “the best science, by the best scientists” — regardless of age or employment sector. We think our findings suggest that this aim is not being met.
To ensure that we captured people who were eligible for NIH funding, we focused on scientists in the life and health sciences whose affiliation address was in the United States. We aimed to assess whether these scientists currently receive NIH funding as principal investigators using information from the NIH RePORTER website. (For detailed methods, see Supplementary Information.)
Of the more than 20 million papers published worldwide between 2001 and 2012 and catalogued by the Scopus database, 1,380 had received 1,000 citations or more as of April 2012. Of those 1,380 papers, 700 were catalogued in the life or health sciences and had an author affiliation in the United States. These 700 papers had a total of 1,172 discrete single, first or last authors.
We stratified eligible authors of extremely highly cited papers according to whether or not they were current members of NIH study sections. This was because the NIH policy is to invite principal investigators of funded projects to become members of study sections.
We also wanted to look closely at members of NIH study sections because these scientists are arguably the most influential group in the grant-funding process. Studying their track records and impact could reveal problems or discrepancies in that process.
We discovered that serving on a study section is not necessarily tied to impact in the scientific literature. (see ‘Is funding tied to impact?). When we cross-checked the NIH study-section rosters against the list of 1,172 authors of highly cited papers, we found only 72 US-based authors who between them had published 84 eligible articles with 1,000 or more citations each and who were current members of an NIH study section. These 72 authors comprised 0.8% of the 72 (n = 64, 88.9%) currently received NIH funding.
We then randomly selected 200 eligible life- and health-science papers with 1,000 or more citations (analyzing all 700 would have required intensive effort and yielded no extra information in terms of statistical efficiency). We excluded those in which the single, first or last author was a member of an NIH study section, and those in which the single or both the first and last author were not located in the United States on the basis of their affiliations at the time of publication. This generated a group of 158 articles with 262 eligible US authors who did not participate in NIH study sections. Only the minority (n = 104, 39.7%) of these 262 authors received current NIH funding.
The rate of NIH funding among highly cited researchers is not much better and may be worse than that of biomedical scientists in general. Annual data for the years 2001–11 suggest that 24–37% of biomedical scientists who applied for grants were funded as principal investigators (these rates even exclude some types of grants; see go.nature.com/gohji3). The acceptance rates for individual grants are substantially lower, but if one allows for several grants submitted and for several years of submissions, a sizeable portion of general applicants — if not the majority — probably end up being awarded at least one grant.
Among authors of extremely highly cited papers, study-section members and non-members showed no significant difference in their total number of highly cited papers, despite the fact that members of study sections were significantly more likely than non-members to have current NIH funding. This was true both for authors with multiple highly cited papers (13/13 versus 13/19, p = 0.024) and for those with a single eligible highly cited paper (51/59 versus 91/243, p < 0.0001) and overall in a stratified analysis (p < 0.0001).
One can also examine the similarity between different grants by using a similarity (‘match’) score provided by NIH RePORTER. To calculate the match score, a fingerprint is created of each grant that contains its key terms, weighted by how frequently they appear in the grant. That fingerprint can then be compared against the fingerprints of other grants.
We found that the grants of study-section members were more similar to other currently funded NIH grants than were non-members’ grants (median score 421.9 versus 387.6, p = 0.039). This could suggest that study-section members fund work that is more similar to their own, or that they are chosen to serve as study-section members because of similarities between their own and funded grants.
If NIH study-section members are well-funded but not substantially cited, this could suggest a double problem: not only do the most highly cited authors not get funded, but worse, those who influence the funding process are not among those who drive the scientific literature. We thus examined a random sample of 100 NIH study-section members. Not surprisingly, 83% were currently funded by the NIH. The citation impact of the 100 NIH study-section members was usually good or very good, but not exceptional: the most highly cited paper they had ever published as single, first or last author had received a median of 136 (90–229) citations and most were already mid- or late-career researchers (80% were associate or full professors). Only 1 of the 100 had ever published a paper with 1,000 or more citations as single, first or last author (see Appendix 1 of Supplementary Information for additional citation metrics).
This overall picture (see ‘Is funding tied to impact?’) might, in part, be explained by the NIH policy to try to recruit reviewers who are successful in securing grants. Even so, it is worrying that the majority of highly cited investigators do not have current NIH funding as principal investigators.
This does not mean that these investigators are not funded at all. Moreover, it is impossible to tell whether these investigators have chosen not to seek funding from the NIH (for example, if they did not like the agency’s funding process or had left science) or if they have sought funding and been repeatedly rejected.
To investigate a little, a Nature editor contacted a sample of highly cited investigators who are not currently listed as principal investigators on NIH grants. She found that there were many reasons why these scientists were not currently funded by the NIH. In some cases, they had applied for NIH grants but had been unsuccessful. In others, the researchers were graduate students at the time of the high-impact publication, so had just started their own labs (the average age for being awarded a first NIH grant as primary investigator is 44 years). Some had moved into industry or changed fields entirely (becoming a venture capitalist, for instance). One had retired; another was receiving some NIH funding as a co-investigator but was not listed as a principal investigator, so did not appear in our search. Future samples might also have to account for an increasing number of highly collaborative papers for which the first and last authors are not principal investigators, but this was not a prominent issue in this sample.
Although it might seem as if these explanations suggest that highly cited researchers are not lacking support, we maintain our concerns. The mission of the NIH is to support the best scientists, regardless of whether they are young, old or in industry. If they have left academia — or moved out of research entirely — it could suggest that these authors of high-impact work did not want to continue the struggle for federal funding. Such innovative thinkers should not have so much trouble obtaining funding as principal investigators. One cannot assume that investigators who have authored highly cited papers will continue to do equally influential work in the future. However, a record of excellence may be the best predictor of future quality, and it would seem appropriate to give these scientists the opportunity of funding for their projects.
We feel that by allowing grant holders to serve as grant reviewers, a conflict of interest becomes inescapable. Exceptional creative ideas may have difficulty surviving in such a networked system. Scientists who think creatively may be discouraged by the funding process and outcomes, or might not have time to contribute as reviewers to a process that is arduous and not perfectly meritocratic. Not surprisingly, although NIH funding has been instrumental in maintaining and expanding the biomedical research endeavor, there are many examples of major scientific discoveries, including Nobel prizes that emerged from unfunded work.
More alternative funding modes should be tested in pilot schemes and in experimental controlled studies of optimizing funding processes. For example, the American Cancer Society uses impartial laymen known as stakeholders in their grant reviews to limit bias, which may reduce the influence of strongly opinionated group members. Using non-experts or experts from different scientific fields in the study sections could also help to reduce the impact of a vocal minority.
Serious consideration should be given to increasing funding for investigators of outstanding ability who have already proved that they can accomplish something major; for example, as judged by extreme citation impact. Such investigators could be funded by processes analogous to those of the Howard Hughes Medical Institute or the NIH MERIT awards, without having to submit grant proposals or, as is currently done, by calling for proposals that present only broad goals.
Funding all scientists who are key authors of unrefuted papers that have 1,000 or more citations would be a negligible amount in the big picture of the NIH budget, simply because there are very few such people. This could foster further important discoveries that would otherwise remain unfunded in the current system.
Joshua M. Nicholson is in the Department of Biological Sciences, Virginia Tech, Blacksburg, Virginia, USA. John P. A. Ioannidis is at the Stanford Prevention Research Center, Stanford, California, USA. e-mail: firstname.lastname@example.org