K99-R00 Publication Analysis-Part 1

(by datahound) Oct 19 2014

The NIH K99-R00 program is an important program related to the transition from postdoc to faculty positions. This program also presents an unusual opportunity for evaluation since cohorts of scientists at similar career stages compete for initial K99 awards and then can transition to R00 awards and then to R01s and other awards. I have previously posted analysis including the transitions to R00 and R01 grants, gender disparity in R01 transition probabilities, differences between NIH institutes and centers, and gender differences between R0o institutions.

I am now starting to analyze the publication patterns of K99-R00 awardees. For this study, I examined the initial 2007 K99 cohort of 182 investigators, of whom 170 transitioned to R00 awards. I examined the publications of these investigators through the Advanced Search function of PubMed. In many cases, this appeared to produce a relatively comprehensive list of publications based on comparisons with websites and other sources. In other cases, there results appeared problematic due to issues of name ambiguity or a significant number of publications that do not appear in PubMed. Publication lists through the present were generated for 135 investigators.

The total number of publications for each investigator is shown below:

Total Pub Distributions

The number of publications ranges from less than 10 to nearly 100. In some cases for investigators with a relatively small number of publications, technical issues may have resulted in undercounting publications while in a few other cases, the investigators appear to have left academia sometime after receiving the R00 award. Of course, publication numbers have considerable limitations and no attempt has been made at this point to examine individual publications in terms of the citations or other measures.

These publications can be broken down roughly into those leading up to the K99 award and those that occurred after receiving this award. While doing this relatively precisely would require going though individual publications, I used the number of publications in 2007 or before as a surrogate:


The publications after 2007 (2008-2014) are shown below:

Post 2007 pubs

These correspond to publications produced during the K99 award, during the R00 award, subsequent publications, as well as some publications of results generated prior to the K99 award that were somewhat slow to be published.

The correlation between the number of publications 2007 and before and the number of post-2007 publication is shown below:

Pre-Post Correlation

Not surprisingly, these are relatively strongly correlated with a correlation coefficient of approximately 0.6. Of course, this reflects differences in the publication patterns between fields and other factors in addition to some more calibrated measure of investigator productivity.

One additional factor that I have examined involves the meme that a publication in Science, Nature, or Cell is highly correlated with receiving a K99 award. Examination of the publication lists reveals that approximately 20% of the K99 awardees have a publication in Science, Nature, Cell or New England Journal of Medicine prior to or in 2007. In addition, approximately 40% have a publication in other relatively high profile journals such as PNAS, other Nature or Cell journals, and the Journal of Clinical Investigation.

With this list of nearly 4000 publications along with the other data that I have assembled on this cohort of investigators, much more analysis is possible and I welcome thoughts about what might be interesting.

5 responses so far

Perspectives on the NPR NIH Stories

(by datahound) Sep 26 2014

Recently, NPR (through the work of Richard Harris and colleagues) aired a series of 7 stories about biomedical research and NIH funding with 5 stories on Morning Edition (Tuesday, Wednesday, Monday, Tuesday2, Wednesday2) and 2 on All Things Considered (TuesdayTuesday2).

The first set of stories on Tuesday, September 9th, focused on the "boom and bust" funding environment beginning with the budget "doubling" followed by the past decade with its associated loss of buying power and on profiles of a couple of scientists who had moved on to non-scientific careers. These were followed by stories about over-building of research space, non-reducibility of animal studies ascribed to hyper-competitiveness, the mismatch between the number of trainees and the number of academic jobs, alternative models for setting research agendas with the National Breast Cancer Coalition as an example, and concluded with a discussion with former NIH Director and current NCI Director Harold Varmus about some potential adjustments to the system.

There has been an active set of discussions about these stories and related topics over at Drugmonkey (here, here, here, and here).

I learned that at least one story about the NIH was in the work when Richard Harris emailed me to initiate a discussion about these issues back in April. This was just prior to the panel discussion at the Experimental Biology meeting that I had been planning with the ASBMB Public Affairs Advisory Committee on related topics. I sent Richard our white paper on Building a More Sustainable Biomedical Enterprise as well as my recent ASBMB Today column about the impact of the sequester on the number of R funded investigators. Over the course of our discussions, I helped Richard and his colleagues about the use of NIH Reporter, both to confirm statistics but, more importantly, to compile a list of investigators who has recently lost funding to identify potential subjects for stories about the impact of the sequester and the disequilibrium of the biomedical research enterprise.

Two points.  First, this highlights a key challenge of journalism. Stories that focus on statistics (e.g. 1000 investigators lost R funding due to the sequester) tend to be rather sterile and not compelling in the public (as opposed to the scientific) sphere. Thus, he was seeking specific people to approach to find some who would go on the record about their experiences and the impact of the funding situation on their career situations. Of course, each specific example has its own idiosyncrasies and it is very difficult to find a few "typical" cases that approximately capture the full reality of what is going on. For example, the scientists who had left academic positions to start a business to produce liquor or to run a grocery struck some (including me) as odd examples given that they were more familiar with those leaving academia (and research) to move into communications or other "more traditional" science career alternatives.

In any event, I feel it is important to recognize the journalistic challenge of finding real human examples to make a story three-dimensional and compelling to the public. We should be appreciative of reporters who make the effort and of individuals who are willing to share their own stories so publicly.

Second, I was struck by the differences between reporting and advocacy. The story about how animal model studies relevant to ALS research turned out to be not very robust does not paint a flattering picture of some aspects of the biomedical research enterprise. In a short piece, it is difficult to explore all of the factors contributed (or might have contributed) to such outcomes so that the piece might come across as unfair. Nonetheless, in my opinion, it is very important to understand how the public perceives these issues (again, as discussed at Drugmonkey here and here) and having them aired in public, while uncomfortable, certainly has an upside.

My bottom line is that the scientific community needs to capitalize on the public awareness that comes from such press coverage. We need to learn from the stories and the public reactions to them, work to address the issues that we can tackle, and focus energy into productive channels for improving the scientific enterprise and the public understanding of it, to the best of our ability.

7 responses so far

Federal RePORTER-A New Tool of Science Data Wonks

(by datahound) Sep 26 2014

Recently, Drugmonkey put up a post with the understated title Federal RePORTER!!!!!!!!!!!!! He noted to a new project from the Star Metrics program with a version of the NIH RePORTER website that now has data from NSF, USDA, and EPA, in addition NIH. Needless to say, I could not resist having a look.

One question that occurred to me right away is how many NIH funded investigators also have NSF funding. A quick download from Federal RePORTER and I had an answer (given my previous work on NIH data).

For FY2013:

25361 investigators had R-mechanism funding from NIH

11440 awards were listed on Federal RePORTER from NSF corresponding to 10260 unique investigators (with some uncertainty due to potential name overlap)

196 individuals were on both lists.

This strikes me as a surprisingly low number, corresponding to a few investigators per institution. However, I grew up in chemistry departments which is likely an area where funding from both NIH and NSF is most common.

Suggestions about other questions are welcome although the data available from Federal RePORTER is still limited (e.g. only back to 2004).

7 responses so far

Gender Differences in R00 Institutions

(by datahound) Aug 14 2014

Following my post noting the occurrence of differences between men and women K99 awardees in their likelihood of receiving an R01 grant NIH, through Sally Rockey's blog, noted that application rates may play a role:

"Of the 2007 cohort of K99 PIs, 86 percent of the men had applied for R01s by 2013, but only 69 percent of the women had applied."

This point has been taken up over at DrugMonkey.

Although such differences in application rates between genders are common in NIH statistics, I was surprised that the rates were this different since the K99 cohort from a single year is, presumably, relatively uniform in terms of career stage, accomplishment (having successfully competed through the same program), and so on.

In considering factors that could contribute to this difference, I thought of the nature of the institutions at which these individuals get their R00 awards (if they do transition). As one (certainly imperfect) measure of institutional characteristics, I used the FY2013 institutional ranking of NIH funding. For the FY2007 cohort, of the 108 men with R00 awards, the median ranking for their R00 institution is 37 and the mean is 81. In contrast, for the 62 women, the median is 57 and the mean is 113. For the FY2008 K99 cohort, the median for men is 44 and the mean is 71. For women, the median is 45 and the mean is 103.

For the purposes of further analysis, I divided institutions into 5 groups (NIH funding ranking 1-25, 6-50, 51-75, 76-100, and >100. The distributions for men and women for the two cohorts are shown below:

2007-2008-Rank plot

The distributions are relatively similar for the institutions near the top of NIH funding rankings. However, there are differences in the remainder of the distribution, most strikingly for institutions with NIH funding ranking >100. For men, 20-21% of the R00 awardees were at such institutions whereas 31-36% of the women were. This reveals that a larger percentage of women over men with K99 awards are beginning their independent careers at institutions that are relatively less research intensive, by opportunity or choice.

How does this relate to the likelihood of receiving an R01 award? The results for the FY2007 cohort are shown below:

2007 Funding Groups New Plot-2

For the investigators at institutions with rank 1-25, the percentages of investigators who have achieved R01 funding is comparable for men and women. However, this is not true for the other sets when a higher percentage of men than women have received R01 funding. For example, more than 20% of all women in this cohort are at institutions with NIH funding rankings >100 and have not received R01 funding compared with 7% of all men.

The corresponding plot for the FY2008 K99 cohort is shown below:

2008 Funding Group New Plot-2

Again, more than 20% of all women are at institutions with NIH funding rankings >100 and have not received R01 funding. In addition, for this cohort, the fraction of women at institutions with NIH funding rankings from 1-25 who have received R01 funding is substantially lower than that for men at the same set of institutions.

These data provide insights into some factors that may contribute to the disparities in R01 funding for women and men in the K99-R00 program. Of course, as one parses the program into smaller groups, the statistical power decreases. Nonetheless, these analyses should provide guidance to allow a better understanding of the role of different factors in NIH funding outcomes.

7 responses so far

A Pilot Study of Continued Funding after Holding a Single R21 Award

(by datahound) Aug 08 2014

In a recent post, I highlighted the growth in the number of applications and, to a lesser extent, awards for R21s. At the end of the post, I noted that many individuals who held R21 awards in FY2013 had no other R-mechanism funding and noted that one could track outcomes for these individuals over time going back to an earlier year.

As a first step, I have examined a sample of approximately 800 investigators who received an R21 award in FY2009 and held no other NIH awards (including both R and all other mechanisms). I then examined the funding for these investigators in FY2013. Of the sample of 801 investigators, 576 investigators (72%) had no funding in FY2013.

FY2009 was a year in which NIH received additional funds through ARRA. Of the R21 awards in the sample, 367 were supported by ARRA and 434 were not. Of the ARRA-supported investigators, 277 (75%) had no support in FY2013. Of the non-ARRA-supported investigators, 299 (69%) had no support in FY2013. Of the investigators who were funded, it appears that slightly more than half have R01 funding.

This study is a preliminary study with a sample from a single year, but it provides a general sense of the outcomes after having a single R21 awards. Not that this sample includes investigators at a variety of career stages.

9 responses so far

Non-R01 Individual Investigator Mechanisms: The Growth of R21 Applications

(by datahound) Aug 04 2014

In a previous post about R01s, I noted that the fraction of the NIH budget going to R01s decreased over the period from FY2003 to FY2013. This fact is of concern, of course, for a variety of reasons. But first, it is important to understand the observation as completely as we can. One set of factors that has contributed is the growth in the number of R21 awards, driven in large part by a huge increase in the number of R21 applications and the introduction of additional mechanisms.

For the R21 mechanism, the numbers of applications and awards NIH-wide for the period from FY2003 to FY2013 are shown below:

R21 Apps Awards plot

These data reveal that the number of applications increased more than 2.6-fold from 5283 in FY2003 to almost 14000 in FY2012. NIH responded to this "proposal pressure" by increasing the number of awards from 1255 in FY2003 to almost 2000 in FY2012. Nonetheless, the success rate for R21s has remained between 12.9 and 14.9 % over the past five years, generally 2 or more percentile points below the R01 success rate, even for new (as opposed to competing renewal) applications. Thus, the competition for R21 awards is more severe that it is for R01 awards, a fact which is not, in my experience, widely appreciated by the all in the community.

When I was Director of NIGMS, we decided to stop accepting unsolicited R21 applications. This decision was made for two reasons. First, we found the peer review process for R21s quite frustrating. NIGMS was trying to use the R21 mechanism to support "high risk-high potential reward" research, that is, new ideas for which a modest investment could provide a proof of principle that could be used to drive future inquiry. However, despite efforts by CSR and NIGMS staff to orient reviewers, we frequently received scores and summary statements that did not align e.g. 'This is a potentially important and impactful project, but there is no preliminary data' and a bad score or 'This is a solid proposal supported by much preliminary data' with a good score. Because of this, we often struggled to develop sensible paylists. NIH made this problem worse by using the R21 mechanism for many other purposes other than the "high risk-high potential reward" goal. This confused reviewers, applicants, and even NIH staff.

Second, we had misgivings about whether the duration of the R21 and the size of the award would, in general, support substantial research compared to taking the same funds and supporting a smaller number of R01-sized grants. This led to the EUREKA R01 awards, used by NIGMS and a few other institutes.

I do not know of any studies that bear of the success of R21 awards in promoting scientific discovery or in keeping investigators "in the game". One interesting observation is that, in FY2013, more than 2300 (62%) of the R21 awards were held by investigators who had no other R21 or R01 awards (competing or non-competing) in the same year. Looking back to earlier year, one could track subsequent results for such investigators if that would be of interest.

12 responses so far

K99-R00 Evaluation: IC distribution

(by datahound) Jul 22 2014

The K99-R00 program is an NIH-wide program but, as is typical at NIH, each institute and center has considerable flexibility about the details about how the program is administrated. For example, for the two cohorts of K99 awardees that I have been examining, the number of K99 awardees ranges from 1 for the National Institute (then National Center) for Minority Health and Health Disparities (NIMHD, MD) to 54 for the National Cancer Institute (NCI, CA). The number was not simply proportional to budget size. For example, the number of K99 awards from the National Institute of Allergy and Infectious Diseases (NIAID, AI) was 13, smaller than five other institutes despite the fact that NIAID has the second largest budget at NIH. Moreover, as one would expect by chance, the fraction of women and men among K99 awardees varies from IC to IC. This may be relevant to understanding the disparity between women and men in the probability of transitioning from an R00 award to an R01 or similar award.

These data are summarized graphically below:

IC distribution


The size of each circle is proportional to the number of R00 awardees from each IC.

These data may be relevant to understanding the gender disparity. For example, both NIAID and the National Institute for Neurological Diseases and Stroke (NINDS, NS) have percentages of men among R00 awardees that are slightly higher than the NIH average but all of the women from these institutes who have received R00 awards have gone on to obtain R01 or equivalent (DP2) funding through the present.

Understanding the origins of the gender disparity between women and men going from a K99 award to R01 or equivalent funding is important for determining what policy adjustments should be considered.

10 responses so far

K99-R00 Evaluation: A Striking Gender Disparity

(by datahound) Jul 21 2014

In a recent post, I examined the NIH K99-R00 (Pathway to Independence) program. Looking at two cohorts of individuals who received K99 awards in FY2007 and FY2008, this analysis revealed that 91% of these 360 awardees transitioned from the K99 phase to the R00 phase, indicating that they had found suitable academic positions. Further, of those with such positions, 51% had obtained R01 funding through the present and some others had received other types of grants.

Over the process of organizing these data, I noticed that the fraction of women receiving K99 awards appeared to be lower than that for men. In order to examine this point more systematically, I assigned gender to these 360 awardees based on name and web searches. Overall, 142 of the 360 (or 39%) of the K99 awardees in these two cohorts are women. This is consistent with the value reported by NIH for this program.

Of the 218 men with K99 awards, 201 (or 92%) went on to activate the R00 portion. Of the 142 women, 127 (or 89%) went on to these R00 phase. These differences in these percentages are not statistically different.

Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).

These results are summarized graphically below:

Men-Women-K99-R00-R01 percent

To investigate this further, I looked at the two cohorts separately. For the FY2007 cohort, 70 of the 108 men (65%) with R00 awards have received R01 grants whereas only 31 of the 62 women (50%) have (P value = 0.07). For the FY2008 cohort, 44 of the 93 men (47%) with R00 awards have received R01s whereas only 22 of the 65 women (34%) have (P value = 0.10). The lack of statistical significance is due to the smaller sample sizes for the cohorts separately rather than any difference in the trends for the separate cohorts, which are quite similar.

What could account for these disparities? Of course, it is unknown at this point if the differences reflect differences in R01 success rate as this would depend on the number of applications submitted by each applicant. I do not have access to data regarding the numbers of applications submitted by these grantees (although NIH does and could do this analysis).

One possibility is that the disparities is due to differences in the timing of funding for men versus women, that is, women might become R01 funded at more similar rates, but a greater disparity is observed at this point because of the lack of time for follow-up after the K99 award. To examine this point, I looked at the percentage of first R01 grants awarded each year to men and women as shown below:


The curves for men and women for each cohort are quite similar. This does not support the hypothesis that the timing of R01 funding is primarily responsible for the observed gender disparity.

Another factor that I considered is the impact of other R01-like awards, primarily the NIH Director's New Innovator (DP2) awards, on the analysis. Overall, nine K99-R00 awardees have received DP2 awards. Of these, six are men and three are women. Interestingly, three of these six men have subsequently received additional R01 awards whereas none of the three women have. In addition, one male K99 awardee received a Pioneer (DP1) award. The inclusion of these additional grants does not significantly affect the overall conclusion about the occurrence of a gender disparity.

Confirmation of and further analysis of these disparities is certainly warranted. Indeed, the K99-R00 program may be highly appropriate for such analyses as investigators at similar career stages competed for initial support and then obtained relatively comparable resources to launch their independent careers.

22 responses so far

My first pass evaluation of the K99-R00 program

(by datahound) Jul 17 2014

In 2006, NIH initiated a new grant program The Pathway to Independence award program. This transition award was a direct response to a National Academy of Sciences report "Bridges to Independence" that highlighted the long time between degree completion and the initiation of an independent research career. This program using a combination of the K99 and R00 mechanisms (and are often referred to a "Kangaroo" awards).

The first awards were made in FY2007. In 2011, Drugmonkey posted a qualitative evaluation of the K99-R00 program and noted that some K99-R00 recipients had gone on to receive R01 funding. However, it was too early to perform a more quantitative analysis. However, now sufficient time has passed that quantitative data are more interpretable.

I analyzed the grant trajectories of the 182 K99 recipients awarded in FY2007.

Here are the key results:

170 (or 93%) transitioned to the R00 phase, indicating that they had obtained an appropriate faculty position.

Of these 170 individuals, 101 (or 59%) have obtained at least 1 R01 grant through the present. In addition, 4 had obtained New Innovators (DP2s), 1 had obtained a Pioneer (DP1) and 1 had moved to the NIH intramural program.

The timing for obtaining R01 grants is shown below:

2007 plot

The plot shows that R00 awardees received R01 support as early as FY2009. The number of R00 awardees receiving their first R01 peaked in FY2012 and has been falling. The line to FY2014 is dotted as this fiscal year is not yet complete. The cumulative percentages of R00 awardees with R01 support are as follows:

FY2009-4%; FY2010-14%; FY2011-27%; FY2012-43%, FY2013-53%; FY2014-59%


For comparison, I also analyzed the 178 K99 awardees from FY2008.

158 (or 89%) transitioned to the R00 phase.

Of these 158 individuals, 66 (or 42%) have obtained a least 1 R01 grant through the present. In addition, 6 had obtained New Innovators (DP2s) and 1 had moved to the NIH intramural program.

The timing for obtaining R01 grants is shown below:

2007 2008 plot

The FY2008 cohort is obtaining R01 grants with a similar but slightly longer time delay compared to the FY2007 cohort, due at least in part to the impact of the ARRA funds on the FY2007 cohort. For the FY2008 cohort, the determination of the overall fraction obtaining R01 funding and the final trend will depend on the results through the end of FY2014 and, perhaps, beyond.

The cumulative percentages are:

FY2009-0.6%; FY2010-1.2%; FY2011-7%; FY2012-16%, FY2013-37%; FY2014-42%

From these data, there can be no doubt that the the K99-R00 program has been effective in transitioning individuals from postdocs to faculty positions. It is clear that individuals with these awards compete well for such positions through some combination of the skills and qualities that allowed them to successfully compete for the K99 award and the interests of the hiring institutions in having faculty with some extant funding and a track record of obtaining such funding. The role of this program in influencing hiring has been discussed in another Drugmonkey post.

The results in obtaining R01s is also quite impressive to me. Of course, it is not clear what percentage of these individuals would have obtained positions without the "kangaroo hop". One comparison could be to look at F32 recipients who went on to faculty positions during the similar period, but this will take some work. Again, the factors that have led to this success include scientific skills, grant writing skills, relatively good contacts with NIH staff, funds for preliminary data, and others.

It is less clear how much impact this program has had on the timing of the transition to independence if this is defined by obtaining independent R01 funding. There is still a substantial lag between obtaining a faculty position and obtaining R01 funding, comparable to what Drugmonkey posted with my data from NIGMS (although these results were from times where success rates were somewhat higher). Nonetheless, overall it is hard not to see the K99-R0o program as a substantial success.


19 responses so far

R01 Success Rates: New and Renewal Applications

(by datahound) Jul 08 2014

In a recent post, I looked at the distributions of A0, A1, and A2 awards for new (Type 1) versus renewal (Type 2) R01 grants. I noted differences between trends for new and renewal awards over time and commenters Ryan and DrugMonkey noted possible reasons for these differences. As DrugMonkey said "Remember the structural issues here with Type 2s. There is a fixed number of possible Type 2s and an unlimited number of Type 1s that can come in for consideration. As each Type 2 flails out at the A1 (or previously A2) stage, it is gone forever from the pool of potential Type 2s."

NIH does release data regarding the number of applications for new versus renewal R01s (and many other mechanisms. The data for new R01 applications and awards for FY2003 to FY2013 are shown below:


These data show that the number of applications for new (Type 1) R01s grew with an initial burst immediately after the end of the doubling and then another stimulated by the American Recovery and Reinvestment Act (ARRA) implementation. The number of applications in FY2013 was 24% larger than that for FY2003. The number of awards declined relatively steadily over this period of time with a 26% drop from FY2003 to FY2013.

The situation is very different for renewal (Type 2) applications as shown below:


The number of applications for renewal (Type 2) applications grew from FY2003 to FY2006 and then declined relatively steadily through FY2013. There has been a 32% drop in the number of applications for Type 2 applications from the peak to FY2013. The number of renewal awards fell by 47% from FY2003 to FY2013.

The success rates for new and renewal R01 applications are compared below:

R01 Success Rates-2

The success rate curves parallel one another relatively closely. The success rate for renewal applications was 48.3% in FY2003 and it fell to 31.4% in FY2013, a drop of 35%. The success rate for new applications was 24.1% in FY2003 and fell to 14.3%, a drop of 40%. As DrugMonkey noted, the pool of projects eligible to be the subject of renewal applications limited.

9 responses so far

Older posts »