Data Regarding the A2/No A2 Policy

Nov 13 2014 Published by under Uncategorized

Few NIH policy issues have elicited as much discussion and passion as the elimination of the option of submitting an "A2" application. This policy had its roots in the NIH "Enhancing Peer Review" process. During this process, NIH staff presented the data below, demonstrating that the mixture of A0, A1, and A2 applications within the pool of funded R01 grants had been shifting dramatically in the post-doubling period with the fraction of R01s funded at the A0 stage dropping from near 60% to below 30%.

Screen Shot 2014-11-12 at 8.53.00 PM

These data demonstrated increasing inefficiency in the NIH funding system as applications that were quite likely to be funded eventually were being queued up with A1 and, particularly, A2 applications receiving better scores and getting funded in preference to A0 applications that "had more chances" to come in for reconsideration. These data, together with other analyses and discussion, led the Enhancing Peer Review committee to recommend that the NIH "consider all applications be being new." This proposal was not well received by the scientific community including FASEB. Instead, NIH announced that only a single amended application would be allowed and proposals not funded after two attempts would have to be "substantially re-designed."

After implementation of this policy, many in the scientific community raised concerns about its consequences, in part through a petition with more than 2000 signatures. Nonetheless, NIH Deputy Director of Extramural Research Sally Rockey and NIH Principal Deputy Director Larry Tabak posted a progress report showing that the policy "was working" including the figure below:

Screen Shot 2014-11-11 at 4.10.42 PM

The key point from this figure was not that the percentage of A2 applications in the funded pool was dropping (this was inevitable given the policy) but rather that the percentage of A0 applications was increasing while the percentage of A1 applications was essentially static.

Much to everyone's surprise, in April, NIH announced a reversal of this policy, allowing unsuccessful A1 applications to be resubmitted without the need for any change as new, A0 applications.

What might have been the basis for this change? I extended the data analysis through fiscal year 2013 (when the new policy was announced) as shown below:

Type1-2000-2013 graph-2

 

These data show that, in the three years after the initial progress report (shown with darker lines), the trends had changed with the percentage of A1 applications in the funded pool surpassing the percentage of A0 applications. The percentage of A0 applications in the funded pool was only slightly over 40% (even with no A2s), well below the 60% historical value.

These data are just for new (Type 1) grants and not for competing renewal (Type 2) grants. For completeness, I present the results for both Type 1 and Type 2 grants, going back to 1991 (when data are first available in RePORTER) to 2014 below:

1991-2014 Type 1-2 graph

This graph reveals that the distributions were relatively stable before and during the NIH budget doubling. Then the percentage of new (Type 1) A0 awards fell dramatically with the competing renewal (Type 2) A0 drop lagging by a year or two. The curves for Type 1 and Type 2 awards have been essentially coincident since the implementation of the "No-A2" policy and its elimination. It will be interesting to see how these curves evolve although the new policy with new, new applications and recycled new applications will introduce some lack of clarity about interpretation.

6 responses so far

  • Established PI says:

    Datahound, I have been staring at this for a while to figure out what the message is. Putting aside the larger issue of declining overall support, it seems that study sections have settled back into the queuing system once the last A2s were finally shaken out of the system. Once again, there is the mind set that a grant has to wait its turn and go through a second round of review to give time for the previous A1s to get funded. Old habits die hard.

    I fear that the new policy establishing a hybrid system is going to make the review process even more capricious. In contrast to the NSF and private funding agencies, the NIH has stood virtually alone in having an entrenched system of rewards for applicants who bow and scrape before the study section and address their criticisms. Never mind that study sections are increasingly populated with ad hocs and standing members do not have to attend each meeting; there are still built-in rewards for those who follow the sage guidance outlined in the summary statement. We are supposed to use the taxpayers' money to fund the best science, not reward people for learning how to play study section games.

    With the new policy enabling applicants to choose whether to submit as an A0 or A1, my guess is that everyone who got a decent score (or any score) on their A0 is going to resubmit as an A1, whereas anyone who got triaged is going to submit again as an A0 and hope that no one remembers their previous application. My prediction is that A1s will still be favored and the % of A0s funded will go down even further. I look forward to further posts from you to see if this actually comes to pass.

    • datahound says:

      I do not think there is a strong message here. I started assembling these data and then tried to figure out why NIH abandoned its strong position with regard to the "No A2" policy which they had held even in the face of strong concerns from the scientific community.

      I think you are probably right about applicant behavior. In the short run, the percentage of applications funded as "A0s" will almost certainly go up as many such applications will be submitted including unfunded but scored A1s and other relatively strong applications. Over the long run, it seems difficult to guess what the balance of "A0s" and "A1s" will settle out to be. Of course, it is equally difficult to know that this balance even means (in terms of new versus older scientific ideas).

  • drugmonkey says:

    Per usual it is an outrage to *ever* pass along the NIH claims that changes in A0 funding after they killed the A2 meant the time-to-funding goal was served. Until and unless they show data on the thinly recycled A0s which are really A2, A3, etc this should be viewed with intense skepticism and *never* mentioned without this caveat. It is pretty damn close to a lie, otherwise.

    As far as the meaning of the data shown here it underlines a point I make constantly. If the NIH refuses to fix problems (biases) of review *at the point of review* then they are doomed to have their bandaid fixes fail. Just like what happened with the ESI applications, the review process fights back, doubling down on the original problem. I would suggest that the A2asA0 policy was done to protect their claims to have solved the time-to-funding problem. Or at least to have obscured matters such that nobody can easily show that little has changed.

  • Established PI says:

    DM - I completely agree that a major problem is the review process itself. However, my understanding of the move to allow A2 as A0 was a response to the reality that grants at the 15th or 12th percentile were failing to get funded, leaving smaller labs with no resources to either continue an existing project or come up with an entirely new one from scratch. The problem with the new policy is that it is a halfway measure that will sow further confusion at the study sections. I am sure glad I am not a junior PI and have some reserves to fall back on in case my just-submitted renewal doesn't get funded on the first round.

  • drugmonkey says:

    The unevenness in how A2asA0 will be treated is just....maddening. The NIH rules insisting no mention of the prior version be made are just.....maddening. Everything about this is insane.

  • […] recent posts, I presented data regarding the trends in the percentage of A0 applications among funded R01 grants, showed how the percentage of A0 applications varied between the NIH-wide population and a select […]

Leave a Reply