K99-R00 Publication Analysis-Part 4-Numbers of Authors

(by datahound) Oct 23 2014

In my first K99-R00 publication analysis post, I presented the distribution of the number of publications for the FY2007 K99-R00 cohort. In this post, I examine the distributions of the numbers of authors per paper.

For 134 K99 investigators for which I have been able to identify publications relatively unambiguously through PubMed and who transitioned to a R00 award, a histogram of the number of authors per publication is shown below:

Publication histogram

 

The median number of authors is 5 and the mean is 6.5. Of course, average number of authors per publication varies from investigator, reflecting different circumstances, areas of science, and other factors. The distribution is shown below:

Average Authors per Paper Histogram

How does the average number of authors per publication relate to the number of publications?

No Authors vs No Publications

 

As this plot shows, these two parameters are correlated with a correlation coefficient of 0.47. Investigators with a larger number of publications tend to have more authors per publication.

One way to correct for the influence of an increased number of authors on the number of publications is to weight each publication by 1/(number of authors) (as was suggested by a comment on Twitter). In this scenario, a paper with two authors would be worth 1/2 while a paper with 10 authors would be worth 1/10.

This weighted sum of publication is plotted versus the total number of publications below:

1 over n versus No Pubs

This tightens the distribution substantially and the correlation coefficient increases to 0.83. One interpretation of this is that differences in authorship practices do influence the number of publication for a substantial part of the distribution. The investigators on the tail of the distribution with a large number of publications still lie on the edge of the distribution with this correction. In general, these investigators have been successful in obtaining substantial resources subsequent to their R00 awards, leading to an increased number of publications.

4 responses so far

K99-R00 Publication Analysis-Part 3-IC Investments

(by datahound) Oct 22 2014

After my previous post, Drugmonkey commented on the number of K99 awards expected from each IC based on its share of the overall NIH budget. I examined the number of new K99 awards for each IC for fiscal years 2007-2013. There have been only relatively modest variation from year to year for each IC. Moreover, the number of K99 awards for each IC is approximately proportional to its share of the overall NIH budget with a few notable exceptions. This is shown below:

K99 Obs vs Expected

 

This plot shows the expected number of awards based on a total of 190 awards across NIH versus the average number of awards per year for each IC. The number of awards for most ICs is relatively close to that expected. The biggest outlier by far is NIAID where 30 awards per year are expected, but the actual average number of awards is approximately 6. For NCI, the numbers of awards per year expected is 33 and the average is 27. In the other direction, the number of awards expected is 20 but the average number of awards is 29.

15 responses so far

K99-R00 Publication Analysis-Part 2-IC Distributions

(by datahound) Oct 21 2014

In response to comments on my recent post, I have examined the IC distribution of both numbers of publications and "high profile" publications prior to receiving the K99 award as a function of the funding institute or center. Recall that my analysis included only those K99 awardees who went on to receive an R00 award and for whom the investigator's name was unambiguous enough to allow relatively reliable retrieval of publications from PubMed. For several investigators, name changes had occurred over the period examined and these were accounted for where possible through web searches. The analysis reflects 135 K99 awardees out of the total of 182 K99 awardees for FY2007.

The numbers of publications by each investigator (black dots) organized by the funding IC along with the median for each IC (red bars) are shown below:

Pubs by IC

No dramatic trends are observed although, given that the number of K99 awardees per IC ranges from 1 to 20 with typical numbers less than 10, the sample sizes are too small to support any robust conclusions.

The numbers of K99 awardees with very high profile (Cell, Nature, Science, or NEJM) or high profile publications for each IC are tabulated below:

IC Total C, N, S, NEJM Other high profile pub
NCI 13 4 9
NIAID 7 3 3
NHLBI 20 2 8
NIGMS 11 3 4
NIDDK 9 2 3
NINDS 9 3 4
NIMH 7 2 1
NICHD 5 0 1
NCRR 4 2 0
NIA 6 3 2
NIDA 4 1 1
NIEHS 5 0 0
NEI 4 0 0
NIAMS 5 1 3
NHGRI 3 1 2
NIAAA 3 1 0
NIDCD 6 1 2
NIDCR 4 0 0
NLM 2 0 0
NIBIB 1 0 1
NINR 4 0 1
NCCAM 1 0 0
FIC 2 0 1

Here, more striking trends are apparent with all 13 analyzed awardees from NCI having a high profile publication in or prior to 2007. Similar results are observed for other large ICs (e.g. NIAID, 6/7; NHLBI, 10/20; NIGMS, 7/11). This supports the notion that a record of one or more high profile publication was very important for receiving a K99 award from some ICs or in some fields.

7 responses so far

K99-R00 Publication Analysis-Part 1

(by datahound) Oct 19 2014

The NIH K99-R00 program is an important program related to the transition from postdoc to faculty positions. This program also presents an unusual opportunity for evaluation since cohorts of scientists at similar career stages compete for initial K99 awards and then can transition to R00 awards and then to R01s and other awards. I have previously posted analysis including the transitions to R00 and R01 grants, gender disparity in R01 transition probabilities, differences between NIH institutes and centers, and gender differences between R0o institutions.

I am now starting to analyze the publication patterns of K99-R00 awardees. For this study, I examined the initial 2007 K99 cohort of 182 investigators, of whom 170 transitioned to R00 awards. I examined the publications of these investigators through the Advanced Search function of PubMed. In many cases, this appeared to produce a relatively comprehensive list of publications based on comparisons with websites and other sources. In other cases, there results appeared problematic due to issues of name ambiguity or a significant number of publications that do not appear in PubMed. Publication lists through the present were generated for 135 investigators.

The total number of publications for each investigator is shown below:

Total Pub Distributions

The number of publications ranges from less than 10 to nearly 100. In some cases for investigators with a relatively small number of publications, technical issues may have resulted in undercounting publications while in a few other cases, the investigators appear to have left academia sometime after receiving the R00 award. Of course, publication numbers have considerable limitations and no attempt has been made at this point to examine individual publications in terms of the citations or other measures.

These publications can be broken down roughly into those leading up to the K99 award and those that occurred after receiving this award. While doing this relatively precisely would require going though individual publications, I used the number of publications in 2007 or before as a surrogate:

Pubs-Pre2007

The publications after 2007 (2008-2014) are shown below:

Post 2007 pubs

These correspond to publications produced during the K99 award, during the R00 award, subsequent publications, as well as some publications of results generated prior to the K99 award that were somewhat slow to be published.

The correlation between the number of publications 2007 and before and the number of post-2007 publication is shown below:

Pre-Post Correlation

Not surprisingly, these are relatively strongly correlated with a correlation coefficient of approximately 0.6. Of course, this reflects differences in the publication patterns between fields and other factors in addition to some more calibrated measure of investigator productivity.

One additional factor that I have examined involves the meme that a publication in Science, Nature, or Cell is highly correlated with receiving a K99 award. Examination of the publication lists reveals that approximately 20% of the K99 awardees have a publication in Science, Nature, Cell or New England Journal of Medicine prior to or in 2007. In addition, approximately 40% have a publication in other relatively high profile journals such as PNAS, other Nature or Cell journals, and the Journal of Clinical Investigation.

With this list of nearly 4000 publications along with the other data that I have assembled on this cohort of investigators, much more analysis is possible and I welcome thoughts about what might be interesting.

21 responses so far

Perspectives on the NPR NIH Stories

(by datahound) Sep 26 2014

Recently, NPR (through the work of Richard Harris and colleagues) aired a series of 7 stories about biomedical research and NIH funding with 5 stories on Morning Edition (Tuesday, Wednesday, Monday, Tuesday2, Wednesday2) and 2 on All Things Considered (TuesdayTuesday2).

The first set of stories on Tuesday, September 9th, focused on the "boom and bust" funding environment beginning with the budget "doubling" followed by the past decade with its associated loss of buying power and on profiles of a couple of scientists who had moved on to non-scientific careers. These were followed by stories about over-building of research space, non-reducibility of animal studies ascribed to hyper-competitiveness, the mismatch between the number of trainees and the number of academic jobs, alternative models for setting research agendas with the National Breast Cancer Coalition as an example, and concluded with a discussion with former NIH Director and current NCI Director Harold Varmus about some potential adjustments to the system.

There has been an active set of discussions about these stories and related topics over at Drugmonkey (here, here, here, and here).

I learned that at least one story about the NIH was in the work when Richard Harris emailed me to initiate a discussion about these issues back in April. This was just prior to the panel discussion at the Experimental Biology meeting that I had been planning with the ASBMB Public Affairs Advisory Committee on related topics. I sent Richard our white paper on Building a More Sustainable Biomedical Enterprise as well as my recent ASBMB Today column about the impact of the sequester on the number of R funded investigators. Over the course of our discussions, I helped Richard and his colleagues about the use of NIH Reporter, both to confirm statistics but, more importantly, to compile a list of investigators who has recently lost funding to identify potential subjects for stories about the impact of the sequester and the disequilibrium of the biomedical research enterprise.

Two points.  First, this highlights a key challenge of journalism. Stories that focus on statistics (e.g. 1000 investigators lost R funding due to the sequester) tend to be rather sterile and not compelling in the public (as opposed to the scientific) sphere. Thus, he was seeking specific people to approach to find some who would go on the record about their experiences and the impact of the funding situation on their career situations. Of course, each specific example has its own idiosyncrasies and it is very difficult to find a few "typical" cases that approximately capture the full reality of what is going on. For example, the scientists who had left academic positions to start a business to produce liquor or to run a grocery struck some (including me) as odd examples given that they were more familiar with those leaving academia (and research) to move into communications or other "more traditional" science career alternatives.

In any event, I feel it is important to recognize the journalistic challenge of finding real human examples to make a story three-dimensional and compelling to the public. We should be appreciative of reporters who make the effort and of individuals who are willing to share their own stories so publicly.

Second, I was struck by the differences between reporting and advocacy. The story about how animal model studies relevant to ALS research turned out to be not very robust does not paint a flattering picture of some aspects of the biomedical research enterprise. In a short piece, it is difficult to explore all of the factors contributed (or might have contributed) to such outcomes so that the piece might come across as unfair. Nonetheless, in my opinion, it is very important to understand how the public perceives these issues (again, as discussed at Drugmonkey here and here) and having them aired in public, while uncomfortable, certainly has an upside.

My bottom line is that the scientific community needs to capitalize on the public awareness that comes from such press coverage. We need to learn from the stories and the public reactions to them, work to address the issues that we can tackle, and focus energy into productive channels for improving the scientific enterprise and the public understanding of it, to the best of our ability.

7 responses so far

Federal RePORTER-A New Tool of Science Data Wonks

(by datahound) Sep 26 2014

Recently, Drugmonkey put up a post with the understated title Federal RePORTER!!!!!!!!!!!!! He noted to a new project from the Star Metrics program with a version of the NIH RePORTER website that now has data from NSF, USDA, and EPA, in addition NIH. Needless to say, I could not resist having a look.

One question that occurred to me right away is how many NIH funded investigators also have NSF funding. A quick download from Federal RePORTER and I had an answer (given my previous work on NIH data).

For FY2013:

25361 investigators had R-mechanism funding from NIH

11440 awards were listed on Federal RePORTER from NSF corresponding to 10260 unique investigators (with some uncertainty due to potential name overlap)

196 individuals were on both lists.

This strikes me as a surprisingly low number, corresponding to a few investigators per institution. However, I grew up in chemistry departments which is likely an area where funding from both NIH and NSF is most common.

Suggestions about other questions are welcome although the data available from Federal RePORTER is still limited (e.g. only back to 2004).

7 responses so far

Gender Differences in R00 Institutions

(by datahound) Aug 14 2014

Following my post noting the occurrence of differences between men and women K99 awardees in their likelihood of receiving an R01 grant NIH, through Sally Rockey's blog, noted that application rates may play a role:

"Of the 2007 cohort of K99 PIs, 86 percent of the men had applied for R01s by 2013, but only 69 percent of the women had applied."

This point has been taken up over at DrugMonkey.

Although such differences in application rates between genders are common in NIH statistics, I was surprised that the rates were this different since the K99 cohort from a single year is, presumably, relatively uniform in terms of career stage, accomplishment (having successfully competed through the same program), and so on.

In considering factors that could contribute to this difference, I thought of the nature of the institutions at which these individuals get their R00 awards (if they do transition). As one (certainly imperfect) measure of institutional characteristics, I used the FY2013 institutional ranking of NIH funding. For the FY2007 cohort, of the 108 men with R00 awards, the median ranking for their R00 institution is 37 and the mean is 81. In contrast, for the 62 women, the median is 57 and the mean is 113. For the FY2008 K99 cohort, the median for men is 44 and the mean is 71. For women, the median is 45 and the mean is 103.

For the purposes of further analysis, I divided institutions into 5 groups (NIH funding ranking 1-25, 6-50, 51-75, 76-100, and >100. The distributions for men and women for the two cohorts are shown below:

2007-2008-Rank plot

The distributions are relatively similar for the institutions near the top of NIH funding rankings. However, there are differences in the remainder of the distribution, most strikingly for institutions with NIH funding ranking >100. For men, 20-21% of the R00 awardees were at such institutions whereas 31-36% of the women were. This reveals that a larger percentage of women over men with K99 awards are beginning their independent careers at institutions that are relatively less research intensive, by opportunity or choice.

How does this relate to the likelihood of receiving an R01 award? The results for the FY2007 cohort are shown below:

2007 Funding Groups New Plot-2

For the investigators at institutions with rank 1-25, the percentages of investigators who have achieved R01 funding is comparable for men and women. However, this is not true for the other sets when a higher percentage of men than women have received R01 funding. For example, more than 20% of all women in this cohort are at institutions with NIH funding rankings >100 and have not received R01 funding compared with 7% of all men.

The corresponding plot for the FY2008 K99 cohort is shown below:

2008 Funding Group New Plot-2

Again, more than 20% of all women are at institutions with NIH funding rankings >100 and have not received R01 funding. In addition, for this cohort, the fraction of women at institutions with NIH funding rankings from 1-25 who have received R01 funding is substantially lower than that for men at the same set of institutions.

These data provide insights into some factors that may contribute to the disparities in R01 funding for women and men in the K99-R00 program. Of course, as one parses the program into smaller groups, the statistical power decreases. Nonetheless, these analyses should provide guidance to allow a better understanding of the role of different factors in NIH funding outcomes.

7 responses so far

A Pilot Study of Continued Funding after Holding a Single R21 Award

(by datahound) Aug 08 2014

In a recent post, I highlighted the growth in the number of applications and, to a lesser extent, awards for R21s. At the end of the post, I noted that many individuals who held R21 awards in FY2013 had no other R-mechanism funding and noted that one could track outcomes for these individuals over time going back to an earlier year.

As a first step, I have examined a sample of approximately 800 investigators who received an R21 award in FY2009 and held no other NIH awards (including both R and all other mechanisms). I then examined the funding for these investigators in FY2013. Of the sample of 801 investigators, 576 investigators (72%) had no funding in FY2013.

FY2009 was a year in which NIH received additional funds through ARRA. Of the R21 awards in the sample, 367 were supported by ARRA and 434 were not. Of the ARRA-supported investigators, 277 (75%) had no support in FY2013. Of the non-ARRA-supported investigators, 299 (69%) had no support in FY2013. Of the investigators who were funded, it appears that slightly more than half have R01 funding.

This study is a preliminary study with a sample from a single year, but it provides a general sense of the outcomes after having a single R21 awards. Not that this sample includes investigators at a variety of career stages.

9 responses so far

Non-R01 Individual Investigator Mechanisms: The Growth of R21 Applications

(by datahound) Aug 04 2014

In a previous post about R01s, I noted that the fraction of the NIH budget going to R01s decreased over the period from FY2003 to FY2013. This fact is of concern, of course, for a variety of reasons. But first, it is important to understand the observation as completely as we can. One set of factors that has contributed is the growth in the number of R21 awards, driven in large part by a huge increase in the number of R21 applications and the introduction of additional mechanisms.

For the R21 mechanism, the numbers of applications and awards NIH-wide for the period from FY2003 to FY2013 are shown below:

R21 Apps Awards plot

These data reveal that the number of applications increased more than 2.6-fold from 5283 in FY2003 to almost 14000 in FY2012. NIH responded to this "proposal pressure" by increasing the number of awards from 1255 in FY2003 to almost 2000 in FY2012. Nonetheless, the success rate for R21s has remained between 12.9 and 14.9 % over the past five years, generally 2 or more percentile points below the R01 success rate, even for new (as opposed to competing renewal) applications. Thus, the competition for R21 awards is more severe that it is for R01 awards, a fact which is not, in my experience, widely appreciated by the all in the community.

When I was Director of NIGMS, we decided to stop accepting unsolicited R21 applications. This decision was made for two reasons. First, we found the peer review process for R21s quite frustrating. NIGMS was trying to use the R21 mechanism to support "high risk-high potential reward" research, that is, new ideas for which a modest investment could provide a proof of principle that could be used to drive future inquiry. However, despite efforts by CSR and NIGMS staff to orient reviewers, we frequently received scores and summary statements that did not align e.g. 'This is a potentially important and impactful project, but there is no preliminary data' and a bad score or 'This is a solid proposal supported by much preliminary data' with a good score. Because of this, we often struggled to develop sensible paylists. NIH made this problem worse by using the R21 mechanism for many other purposes other than the "high risk-high potential reward" goal. This confused reviewers, applicants, and even NIH staff.

Second, we had misgivings about whether the duration of the R21 and the size of the award would, in general, support substantial research compared to taking the same funds and supporting a smaller number of R01-sized grants. This led to the EUREKA R01 awards, used by NIGMS and a few other institutes.

I do not know of any studies that bear of the success of R21 awards in promoting scientific discovery or in keeping investigators "in the game". One interesting observation is that, in FY2013, more than 2300 (62%) of the R21 awards were held by investigators who had no other R21 or R01 awards (competing or non-competing) in the same year. Looking back to earlier year, one could track subsequent results for such investigators if that would be of interest.

12 responses so far

K99-R00 Evaluation: IC distribution

(by datahound) Jul 22 2014

The K99-R00 program is an NIH-wide program but, as is typical at NIH, each institute and center has considerable flexibility about the details about how the program is administrated. For example, for the two cohorts of K99 awardees that I have been examining, the number of K99 awardees ranges from 1 for the National Institute (then National Center) for Minority Health and Health Disparities (NIMHD, MD) to 54 for the National Cancer Institute (NCI, CA). The number was not simply proportional to budget size. For example, the number of K99 awards from the National Institute of Allergy and Infectious Diseases (NIAID, AI) was 13, smaller than five other institutes despite the fact that NIAID has the second largest budget at NIH. Moreover, as one would expect by chance, the fraction of women and men among K99 awardees varies from IC to IC. This may be relevant to understanding the disparity between women and men in the probability of transitioning from an R00 award to an R01 or similar award.

These data are summarized graphically below:

IC distribution

 

The size of each circle is proportional to the number of R00 awardees from each IC.

These data may be relevant to understanding the gender disparity. For example, both NIAID and the National Institute for Neurological Diseases and Stroke (NINDS, NS) have percentages of men among R00 awardees that are slightly higher than the NIH average but all of the women from these institutes who have received R00 awards have gone on to obtain R01 or equivalent (DP2) funding through the present.

Understanding the origins of the gender disparity between women and men going from a K99 award to R01 or equivalent funding is important for determining what policy adjustments should be considered.

10 responses so far

Older posts »