Percentages of A0 Applications in the Funded Pool by Institute and by Institution

Nov 25 2014 Published by under Uncategorized

In recent posts, I presented data regarding the trends in the percentage of A0 applications among funded R01 grants, showed how the percentage of A0 applications varied between the NIH-wide population and a select group (members of a section of the National Academy of Sciences), and examined the variations in R01 success rates between different NIH Institutes and Centers. Here, I bring these threads together to look at how the percentage of A0 applications among R01 grants varies across NIH Institutes and Centers and then extend the analysis to see how these percentages varies across different universities and other institutions.

Curves showing the percentages of A0 applications among funded R01 grants for six NIH institutes are shown below:

6 IC percent A0 curves

 

The top two curves are from NIGMS and NEI, two institutes that have had relatively high R01 success rates. The curves lie above the NIH-wide curve. The other curves are for four large institutes with lower success rates.

The average percentages of A0 applications among funded R01 grants over the period from FY2001 to FY2013 for these institutes are compared with the average R01 success rates over this period are shown below:

Institute Average % A0s Ave Success Rate
NIGMS 0.556 0.290
NEI 0.533 0.325
NCI 0.380 0.205
NIAID 0.399 0.232
NHLBI 0.457 0.235
NICHD 0.389 0.175

These parameters are highly correlated with a correlation coefficient of 0.90. The slope of the line fit to these data is 0.54 ± 0.09. Thus, s a rule of thumb, the average success rate is approximately one half of the average percentage of A0 applications among funded R01 grants over this period.

These data are extended to all ICs with R01 success rates for individual fiscal years plotted versus he percentages of A0 applications among funded R01s for those years below:

Correlation A0 percent-SR

 

As could be anticipated, there is more scatter in these data, given the factors that influence both parameters as well as the changes in policy over this period. Nonetheless, a correlation is observed with a correlation coefficient of approximately 0.4 and a best fit line with a similar slope.

The observation of this correlation suggested that the percentage of A0 applications among funded R01 grants could serve as a publicly available parameter to examine the experiences of average investigators at different extramural institutions. The average percentages of A0 applications among funded R01 grants for the period over the period from FY2001 to FY2014 were calculated for the 100 institutions that received the most NIH funding in FY2013. This distribution of these averages are shown below:

100 Institution histogram-2

The distribution includes a number of institutions with relatively high percentages of A0 applications. These are labelled above. Examination reveals that these institutions consist largely of basic science-focused institutions with substantial hard money support for salaries including the prominent schools of Arts and Sciences or Engineering. This observation is supported by the fact that Princeton University (which is not in the top 100 institutions in terms of overall NIH funding) has a high percentage of A0 applications (62.2%).

Curves for individual institutions are compared below. These include the institution with higher percentage (Rockefeller), an institution with one of the highest percentage but not an outlier (UCSF), and application in the center of the distribution (University of Pennsylvania), and an institution at the bottom edge of the distribution (Wayne State University).

Percent A0 Institutions-2

Based on the correlation with success rates, these data suggest that the average success rate for investigators at Rockefeller is approximately 50% higher than that for the NIH-wide average.

16 responses so far

  • Comradde PhysioProffe says:

    Two thoughts.

    It would be interesting, but perhaps unknowable, to separately track real A0s and virtual A2s.

    Am I remembering correctly that at least one IC was using a looser payline for A0s for a while?

    • datahound says:

      Unfortunately, tracking virtual A2s compared with A0s is likely impossible, even inside NIH with access to the applications (although it is likely that NIH is trying to figure out how to do this).

      I believe NHLBI did this for a while (shortly after the "Enhancing Peer Review" report).

  • Established PI says:

    Any chance you can provide a list showing which institutions correspond to the rest of the bars?

    • datahound says:

      Here you go...

      ORGANIZATION Percent of A0 apps

      ROCKEFELLER UNIVERSITY 0.666
      CALIFORNIA INSTITUTE OF TECHNOLOGY 0.652
      MASSACHUSETTS INSTITUTE OF TECHNOLOGY 0.617
      HARVARD UNIVERSITY 0.591
      UNIVERSITY OF CALIFORNIA BERKELEY 0.565
      CORNELL UNIVERSITY 0.560
      HARVARD UNIVERSITY (MEDICAL SCHOOL) 0.545
      FRED HUTCHINSON CAN RES CTR 0.528
      DANA-FARBER CANCER INST 0.524
      RUTGERS THE ST UNIV OF NJ NEW BRUNSWICK 0.524
      STANFORD UNIVERSITY 0.520
      NORTHWESTERN UNIVERSITY 0.503
      CHILDREN'S HOSPITAL CORPORATION 0.501
      UNIVERSITY OF CALIFORNIA SAN FRANCISCO 0.488
      JACKSON LABORATORY 0.487
      UNIVERSITY OF WASHINGTON 0.484
      YALE UNIVERSITY 0.484
      UNIV OF NORTH CAROLINA CHAPEL HILL 0.476
      DUKE UNIVERSITY 0.476
      UT SOUTHWESTERN MEDICAL CENTER 0.475
      WEILL MEDICAL COLL OF CORNELL UNIV 0.474
      BRIGHAM AND WOMEN'S HOSPITAL 0.473
      MAYO CLINIC ROCHESTER 0.473
      SLOAN-KETTERING INST CAN RES 0.473
      UNIVERSITY OF UTAH 0.472
      CINCINNATI CHILDRENS HOSP MED CTR 0.472
      UNIVERSITY OF ILLINOIS URBANA-CHAMPAIGN 0.472
      HARVARD UNIVERSITY (SCH OF PUBLIC HLTH) 0.470
      CLEVELAND CLINIC LERNER COM-CWRU 0.470
      COLUMBIA UNIVERSITY HEALTH SCIENCES 0.467
      BAYLOR COLLEGE OF MEDICINE 0.465
      BROWN UNIVERSITY 0.465
      SCRIPPS RESEARCH INSTITUTE 0.463
      OREGON HEALTH & SCIENCE UNIVERSITY 0.463
      RESEARCH TRIANGLE INSTITUTE 0.463
      UNIVERSITY OF CALIFORNIA IRVINE 0.462
      STATE UNIVERSITY NEW YORK STONY BROOK 0.460
      UNIVERSITY OF CALIFORNIA SAN DIEGO 0.459
      UNIVERSITY OF CHICAGO 0.457
      JOHNS HOPKINS UNIVERSITY 0.455
      WASHINGTON UNIVERSITY 0.452
      UNIVERSITY OF VIRGINIA 0.452
      ST. JUDE CHILDREN'S RESEARCH HOSPITAL 0.451
      UNIV OF MASSACHUSETTS MED SCH WORCESTER 0.450
      MASSACHUSETTS GENERAL HOSP 0.446
      UNIVERSITY OF WISCONSIN-MADISON 0.444
      VANDERBILT UNIVERSITY MED CTR 0.443
      MICHIGAN STATE UNIVERSITY 0.443
      UNIVERSITY OF PENNSYLVANIA 0.442
      UNIVERSITY OF CALIFORNIA LOS ANGELES 0.441
      SANFORD-BURNHAM MEDICAL RESEARCH INSTIT 0.439
      UNIVERSITY OF SOUTHERN CALIFORNIA 0.438
      DARTMOUTH COLLEGE 0.437
      UNIVERSITY OF MICHIGAN 0.436
      MOUNT SINAI SCHOOL OF MEDICINE 0.434
      UNIVERSITY OF ROCHESTER 0.434
      UNIVERSITY OF TEXAS, AUSTIN 0.431
      UNIVERSITY OF CALIFORNIA DAVIS 0.429
      CASE WESTERN RESERVE UNIVERSITY 0.429
      UNIVERSITY OF CINCINNATI 0.425
      ALBERT EINSTEIN COLLEGE OF MEDICINE 0.424
      WAKE FOREST UNIVERSITY HEALTH SCIENCES 0.423
      PENNSYLVANIA STATE UNIVERSITY 0.420
      PENNSYLVANIA STATE UNIVERSITY-UNIV PARK 0.420
      NORTHWESTERN UNIVERSITY AT CHICAGO 0.419
      UNIVERSITY OF VERMONT & ST AGRIC COLLEGE 0.418
      UNIVERSITY OF PITTSBURGH AT PITTSBURGH 0.416
      UNIVERSITY OF IOWA 0.416
      TULANE UNIVERSITY OF LOUISIANA 0.415
      UNIVERSITY OF ARIZONA 0.412
      EMORY UNIVERSITY 0.411
      UNIVERSITY OF FLORIDA 0.411
      TUFTS UNIVERSITY BOSTON 0.411
      UNIVERSITY OF MINNESOTA 0.408
      UNIVERSITY OF ILLINOIS 0.404
      UNIVERSITY OF TEXAS MEDICAL BR GALVESTON 0.402
      UNIVERSITY OF MARYLAND BALTIMORE 0.397
      NEW YORK UNIVERSITY SCHOOL OF MEDICINE 0.395
      UNIVERSITY OF TEXAS HLTH SCI CTR HOUSTON 0.395
      UNIVERSITY OF MIAMI SCHOOL OF MEDICINE 0.394
      BOSTON UNIVERSITY MEDICAL CAMPUS 0.393
      UNIVERSITY OF CONNECTICUT SCH OF MED/DNT 0.393
      UNIVERSITY OF ALABAMA AT BIRMINGHAM 0.388
      INDIANA UNIV-PURDUE UNIV AT INDIANAPOLIS 0.388
      BETH ISRAEL DEACONESS MEDICAL CENTER 0.386
      MEDICAL COLLEGE OF WISCONSIN 0.385
      UT MD ANDERSON CANCER CTR 0.383
      UNIVERSITY OF COLORADO DENVER 0.378
      CHILDREN'S HOSP OF PHILADELPHIA 0.376
      STATE UNIVERSITY OF NEW YORK AT BUFFALO 0.367
      UNIVERSITY OF KANSAS 0.364
      UNIV OF TX HSC, SA 0.360
      TEMPLE UNIV OF THE COMMONWEALTH 0.360
      UNIVERSITY OF KENTUCKY 0.359
      UNIVERSITY OF NEBRASKA MEDICAL CENTER 0.346
      VIRGINIA COMMONWEALTH UNIVERSITY 0.343
      OHIO STATE UNIVERSITY 0.330
      MEDICAL UNIVERSITY OF SOUTH CAROLINA 0.329
      WAYNE STATE UNIVERSITY 0.320

  • drugmonkey says:

    I think you are identifying an aspect of reputational bias here that has consequences far beyond the appearance.

    • datahound says:

      I agree that these data are quite interesting and consequential and the role on reputational bias is potentially important. Indeed, the relative role of bias versus application characteristics is a key issue. When I was doing the analysis, I was struck by the frequency of schools of arts and sciences and engineering in the top-ranked institutions. One hypothesis (which can be tested) is that many investigators at such institutions function with single R01 grants (since their salaries are covered (at least 75%) by their institutions and students can be partially supported by TA positions). With single R01s and substantial other resources, the fraction of Type 2 (competing renewal) applications would be expected to be higher than for other institutions with different "business models" with the associated higher success rates.

  • Established PI says:

    Thanks for the additional info. I have been staring at this for a while, trying to make sense of the trends. Even looking just at the subset of institutions with large clinical faculties, I am surprised by the lack of correlation between overall NIH funding and A0 success rate. Are some school just throwing more darts at the dartboard, succeeding because of the sheer number of grants submitted (and resubmitted) rather than average quality?

  • datahound says:

    There are several factors to consider. First, the variations in the percentage of A0 applications in the funded R01 pool are relatively small across most of the distribution so that considerable scatter might be expected. For example, if you look at the curves for UCSF (near the top of the distribution with %A0 = 0.488 and University of Pennsylvania in the middle with %A0 = 0.442, they are not very different with the largest differences during the doubling period. Second, other factors contribute to overall NIH funding (large centers or contracts, etc.) so that some institutions have substantial funding even with relatively modest R01 funding portfolios. For example, George Washington University is often in the top 100 in overall NIH funding due to 2 large (>$10M awards) but has only a small number of R01s with a %A0 0f < 0.25. Finally, I think investigators at many institutions are "throwing darts" given the randomness of peer review with funding levels where they are. I certainly know of investigators and organized units within institutions where submission of a proposal for essentially every NIH grant cycle is the expectation.

    • drugmonkey says:

      You seem to be entering into the common belief that submitting apps every round means lower quality (and therefore success rates) with this comment datahound. Any evidence for this belief that you know of or are we in truthy-land again?

      • Datahound says:

        Unfortunately, I do not have real data, just personal experience and anecdotes. On the personal experience front, I know of investigators who apply relatively infrequently, but work extremely hard on their proposals and almost always receive outstanding scores (top 5 %ile). I also know of investigators who are pushed to submit essentially every cycle and rarely receive fundable scores. I did not mean to imply that application quality correlates strongly with application rate.

        It would be interesting to have data from within NIH but I do not know if anyone with access to such data has looked at this issue.

        • drugmonkey says:

          The people who "always get 5%ile scores" of my acquaintance have quite a number of attributes other than "work really hard" on their applications. Having reviewed many proposals across a wide swath of investigator types in my limited experience I am highly confident that lots of less than perfect grants get low percentile scores on then basis of PI attributes other than their grant crafting. So again, it is not well justified to suggest, as you do here, that exquisite grant preparation makes the difference. It is a traditional theme, true, but it is wrong. Not only that but it is *perniciousky* and *offensively* wrong from the perspective of those who are limited by their "other attributes" but not their effort or writing chops.

          That is why I ask if you have data. Or anything other than the usual bias associated with thinking the successful within the system are uniquely deserving in an objective analysis. Because my experience reviewing the grants of the wonderfully successful and the struggling newb says this not true.

          • Datahound says:

            I worded my response poorly. I did not mean to imply that "working really hard" on their applications was the source of their success. The folks that I was referring to are established and highly productive scientists who also work hard in crafting their proposals. I have also seen established and productive scientists who do not work as hard to craft their proposals, some of whom are doing okay in the present environment and some of whom are struggling.

            In my opinion, success within the system is a mix of "being deserving" by being a productive and creative scientist, biases that work in the favor of some investigators due to institution, pedigree, etc., and (particularly these days) random factors that can be the difference between a very good score and a fundable score.

    • drugmonkey says:

      Point being, if PIs are differentially dis/advantaged by reputational bias, fewer or more apps correlating with success rate is not a causal relationship. Both are driven by the bias. OR, if you credit that populations of PIs are distinguished by objective grant quality traits that generalize across all of their apps there is still a third variable driver of the application rate / success relationship. Neither of those scenarios tells an individual that changing his or her submission rate will change the success rate.

  • Chemstructbio says:

    are there any stats on the number of ESI apps funded in a given year per institute?

  • […] my previous post, I presented (among other things) the distribution of the percentage of A0 applications within the […]

Leave a Reply