Rock Talk Age Data: Effective Indirect Cost Rates 1998-2014

(by datahound) Mar 26 2015

A recent post on Rock Talk presented data on the amount of funding as a function of PI age group. These data were not presented in a terribly informative way but a file was available for downloading and Michael Hendricks normalized the data by the number of PIs in age group to reveal more interesting trends, discussed at Drugmonkey.

The downloadable data includes a breakdown of Direct and Total Costs. I have been looking for such data over a longer period than the last couple of year and thought I would take a look. Below is a plot of the Effective Indirect Cost Rate ((Total Costs-Direct Costs)/Direct Costs) for the overall data set.

Overall indirect graph

The Effective Rate drops from 44.2% to a low of 37.2% in 2012 before rising slightly over the past two years. These values are all somewhat lower than I anticipated based on my previous analysis on R01s.

To try to gain some insight, I looked at these data as a function of PI age group.

Indirect cost graph

The differences between the age groups quite substantial and surprising. For the lowest three PI age groups, the Effective Rate is relatively constant around 47%, consistent with my previous R01 indirect cost analysis. For the older PI age groups, the Effective Rate falls steadily from 1998 to 2012, reaching rates as low as 27.5% for PIs 61-65 in 2012.

I certainly do not understand what underlies these trends, but differences in mechanisms could certainly be involved. It may be that mechanisms as as U01s for larger efforts could be important. As always, it would be best to see data broken down by mechanism to facilitate accurate interpretation.

Any other thoughts on these data are most welcome.

16 responses so far

NIH Institute and Center Strategic Plans

(by datahound) Mar 17 2015

The National Institute of General Medical Sciences (NIGMS) just released its new strategic plan. I was involved in writing the first NIGMS Strategic Plan released in 2008. I must admit that I was skeptical of the value to writing a strategic plan when we started down the path toward this plan. However, I found both the process and the final product useful. It allowed the institute to formalize our commitment to research and training activities, the balance of the use of particular mechanisms, and so on. Even though many aspects of the plan seemed obvious in terms of the culture of NIGMS, I found myself looking back to the plan and referring others to important sections over the next years.

The new NIGMS plan takes a similar approach. One section of interest is:

Objective 1-2: Promote the ability of investigators to pursue new research directions, novel scientific insights and innovative ideas.

This objective includes the following implementation strategies:

1. Align funding decisions with the need to create a broad and diverse research portfolio that maximizes the scientific return on taxpayers’ investments.

2. Support investigators working in underexplored areas of science that are relevant to the NIGMS mission.

3. Conduct regular analyses of NIGMS’ scientific investments to assess their efficacy, distribution and impact, and use this information to help guide programmatic and funding decisions.

4. Pilot and assess alternative mechanisms of funding that emphasize individual investigators, rather than individual projects, to better meet NIGMS goals and objectives.

5. Increase the Institute’s ability to conduct in-depth portfolio analyses and evaluation activities.

Importantly, this includes commitments to analysis and analysis capabilities (strategies 3, 5) as well as a strategy related to the new Maximizing Investigators' Research Award (MIRA) mechanism.

While I am still going though the plan, plans for other NIH Institutes and Centers are available (although some links on this site are broken).

Do you find these plans of any use? How could they be made more useful?

15 responses so far

The NIH Office of Extramural Research (OER)

(by datahound) Mar 02 2015

In many discussions including a recent one at Drugmonkey, issues around the role of the NIH Deputy Director for Extramural Research (DDER) (a position currently held by Sally Rockey) have arisen. DDER is a big job with many responsibilities. Below is the Organizational Chart of OER.

OER_main

 

As you can see, OER is responsible for including extramural policies and programs, data analysis and sharing (e.g. NIH RePORTER), laboratory animal welfare, and administrative operations.

OER has a limited role in communications, specifically related to extramural research. This role expanded with the founding of Rock Talk blog. As Sally Rockey has credited the NIGMS Feedback Loop blog which I started when I was at NIGMS as a model for this blog, I take some pride and responsibility for the existence of this blog. With that said, I share many of the concerns that have been raised about NIH's reactions to comments of this blog.

8 responses so far

Number of New and Competing Renewal Awards from 1995-2014

(by datahound) Feb 26 2015

In my recent post, I noted in passing that the number of Type 2 (Competing Renewal) awards (R01s and R37s) fell from 2653 in FY1995 to 1532 in FY2014. This led to both comments on the post and a post on this topic from Drugmonkey. Since I was also struck by this observation, I was already working on additional analysis.

Below is a plot of the number of New (Type 1) and Competing Renewal (Type 2) awards (just R01s this time for simplicity) as a function of time from FY1995 to FY2014.

Type 1-2 Award number graph

The first striking observation is the dramatic increase in New (Type 1) awards from FY1997 to FY2000 (at the beginning of the NIH "budget doubling" with no corresponding increase in Type 2 awards. This lack of increase in Type 2 awards is almost certainly due to the lack of an increase in applications although I have found no readily available data from these years. Note, further, that success rates for Type 2 applications were likely around 50% (or perhaps above) during this period (see below for data for later years).

From NIH RePORT Funding Facts, data are available for the number of applications and awards for Type 1 and Type 2 R01 grants from FY2001 to the present. Note that these data different slightly from those above and do not appear to include awards made associated with the Recovery Act. These data are plotted below.

Type 1-2 Apps Award Plot-2

This plot shows the further increase in Type 1 Applications over this period. As shown here and in the first figure, the number of Type 1 Awards has been relatively flat (after the 75% increase just prior to FY2000. The number of Type 2 applications increased gradually from FY2001 to FY2006 (by 35%), slowly fell from FY2006 to FY2010 (by 15%), and then fell somewhat more dramatically (by 25%) from FY2010 to FY2014. The number of Type 2 awards decreased by 11% from FY2001 to FY2006, by 4% from FY2006 to FY2010, and then dramatically (by 31%) from FY2010 to FY2014.

These trends are reflected in success rates for Type 1 and Type 2 R01 grants over this period shown below:

Success rate plot-2

 

The success rates fell dramatically shortly after the end of the "budget doubling" and then stabilized to some extent from FY2007 to FY2014.

Taken together, these data reveal that there has been a sharp drop in the number of Competing Renewal Awards, particularly over the past 4 years. This have been driven in large part by a drop in the number of Type 2 applications. This, in turn, may be due to the "No A2" policy or to changes in application behavior around and after the Recovery Act.

2 responses so far

Year Distribution of Competing Renewal (Type 2) Grants from 1995-2014

(by datahound) Feb 25 2015

R01 grants may be renewed, typically every 4-5 years. These are called "competing renewals" or "Type 2" grants. In the context of discussions of the "Emeritus Award" discussion, I examined the distribution of R01 grants that had been renewed over a long period of time. Here, I look at the distribution of Type 2 grants over the period from 1995 to 2014.

Data for Type 2 R01 grants (as well as the corresponding but much smaller number of R37 MERIT awards) for each year were compiled from NIH RePORTER. Note that these grants only include grants that were competitively renewed in a given year and not non-competing continuations (Type 5 awards). The distributions of the 2653 Type 2 awards from FY1995 and 1532 Type 2 awards from FY2014 over the Support Year are shown below:

1995-2014 plot

The distribution for FY1995 shows peaks at Year 4, Year 6 corresponding to renewals of initial (Type 1) grants of 3 and 5 years, respectively. There are additional peaks at Year 9 and Year 14. These correspond primarily to grants that initially made for a period of 3 years and then were renewed for 5 year periods.

The distribution for FY2014 is similar but shows some important differences. First, the overall amplitude is smaller as there were 58% as many Type 2 R01 grants awarded in FY2014 compared to FY1995. Second, the FY2014 shows peaks at Year 6 and Year 11. These correspond to grants awarded initially for a period of 5 years and then renewed for an additional 5 year period. Third, the FY2014 distribution shows a longer tail extended to Year 40 and beyond, corresponding to long-running grants.

The distributions are shown in normalized and integrated form below:

1995-2014 frac plot

These curves show more clearly the tail extending longer grant durations. For FY1995, 20% of the grants are in Years 13 or beyond whereas for FY2014, 20% of the grants are in Years 18 and beyond.

The distribution of Type 2 grants can be seen more clearly by looking are the distribution summed over all of the years from FY1995 to FY2014 as shown below.

Overall Type 2 distribution

This shows peaks from Years 5-6 and Years 10-11 and then the tail extending from Year 15 and beyond.

The structure of this tail can be seen more clearly by replotting the data on a log (base 10) scale as shown below:

Log plot-2-60

The portion of this graph corresponding to the extended tail is very nearly linear with a slight downward curvature. This indicates the tail that is approximately exponential. Fitting this curve reveals an exponent of approximately -0.064/year. This corresponds to a half-life 0f 4.7 years. In order words, the chance that a long-standing grant is renewed every 4-5 years is approximately 50%. The NIH reported success rate for all competing renewal Grants averaged 38% from FY2001 to 2014. Thus, it appears that the likelihood that a longstanding R01 is competitively renewed is slightly, but not dramatically, higher than that for R01 grants overall. The slight downward curvature of the log plot likely reflects that fact that there were a smaller number of longstanding grants in the earlier years in this analysis.

An alternative approach to examining trends the duration of these grants involves looking at the fraction of A0 applications among the funded Type 2 grants. I have previously examined this parameter in other contexts. A plot of the fraction of A0 applications among funded Type 2 grants as a function of the Year of the grant is shown below:

Fraction A0 plot

 

This fraction dips slightly for grant years from 4-7 (corresponding to the first renewal) and then reaches a relatively stable level extending out to 40 years. This suggests there is not a major increase in the likelihood of application success as the year of the grant increases.

Overall, these observations are consistent with the notion that R01 grants reach longstanding status through the perseverance of the principal investigators. Over time, these investigators continue to execute research programs that are sufficiently productive that they compete for renewal with a success rate close to 50% (at least historically). There does not appear to be a substantial advantage for longstanding grant applications above the general advantage for Type 2 versus Type 1 applications based on these publicly available data.

15 responses so far

"Age and the Trying Out of New Ideas"-Initial Impressions

(by datahound) Feb 18 2015

Alerted by a post on Nature News and Comment, I read with interest a newly posted paper from Mikko Packalen and Jay Bhattacharya from the National Bureau of Economic Analysis entitled "Age and the Trying Out of New Ideas."

The abstract of this working paper states:

Older scientists are often seen as less open to new ideas than younger scientists. We put this assertion to an empirical test. Using a measure of new ideas derived from the text of nearly all biomedical scientific articles published since 1946, we compare the tendency of younger and older researchers to try out new ideas in their work. We find that papers published in biomedicine by younger researchers are more likely to build on new ideas. Collaboration with a more experienced researcher matters as well. Papers with a young first author and a more experienced last author are more likely to try out newer ideas than papers published by other team configurations. Given the crucial role that the trying out of new ideas plays in the advancement of science, our results buttress the importance of funding scientific work by young researchers. (Emphasis added)

Needless to say, I was intrigued. After a quick read, I looked deeper into the methodology, particularly with regard to the highlighted terms above.

The study is based on the use of MEDLINE (accessed through PubMed). More precisely, they used “Author-ity” MEDLINE, a previously constructed version of MEDLINE with the names of authors disambiguated as much as possible. This database was used for two purposes. First, new ideas were identified by searching titles and abstracts for two- or three-word strings and associating these with the year when they first appears. Strings that subsequently occurred with high frequency were deemed to be important new ideas. The Nature commentary includes a list of the ten most frequent concepts for each decade and inspection reveals these to be sensible. Second, the "age" of each investigator was estimated by determining the year in which the first publication by this investigator appeared. Thus, this is "career age" rather than chronological age. This is a sensible approach which has both advantages and disadvantages. Most importantly, it is workable from the available data. I know from some of my recent analyses, estimating chronological ages of investigators can be quite difficult. In addition, this automatically at least partially corrects for increasing training periods over time. A disadvantage is that an early publication can "age" an investigator compared to peers.

With these two parameters, the authors were set to do some analysis. Figure 1 in the paper is shown below:

Screen Shot 2015-02-18 at 8.28.34 AM

Panel A shows the fraction of publications trying out new ideas versus the career age of the first author. Clearly, there is a broad peak in which the first authors are within 3-12 years of his/her first publication. This, of course, primarily reflects the accomplishments of graduate students and postdocs!

Panel B shows the comparison for All Authors. This shows a more featureless downward trend. This probably reflects the contributions of graduate students and postdocs but with more senior members of the research teams in the mix.

Panel C shows the distribution with Key (both first and last) Authors. This shows a peak in the career age range of 10-15 years. Given the first author distribution from Panel A, this suggests that the last author distribution has a peak around 20 years. This is confirmed by the data presented in Figure 3 which shows the data in two-dimensional format with Career Age of Last Author versus Career Age of First Author.Screen Shot 2015-02-18 at 8.37.45 AM

 

The maximum for this post occurs with first authors with career ages between 1 and 10 years (i.e. graduate students and postdocs) and last authors with career ages between 8 and 25 years (early to mid-career faculty).

Does this mean that early to mid-career investigators are the most productive in trying out new ideas? Yes and no. As a population, they certain do appear to be. However, these data have not been normalized (as far as I can tell) to the distribution of the ages of investigators. This distribution (which has been changing over time as we were recently reminder by Drugmonkey) shows peaks (both for R01 grantees and medical school faculty) in the range of 45-55 years old over the period covered by this analysis.  If one assumes an age of the beginning on independent careers of 36 (over this period not just at present), the data are consistent with the number of faculty at each career stage being an important factor.

Overall, the paper clearly supports the roles of graduate students and postdocs in being first authors of many (most) papers that appear to break new ground. This is, as George Carlin would say, "near-fetched." The results regarding last authors will require a more careful reading of the paper and, perhaps, more analysis by the authors. However, this is clearly an important data set and approach to provide empirical evidence that bears on these important issues.

6 responses so far

Longevity and Transitions in in R01s in Years 40+...Part 2

(by datahound) Feb 10 2015

I realized that my previous analysis was missing a key bit of information, namely how many long-standing R01s from previous years failed to make it to the present. I examined R01s in years 40+ from FY2010. There were 47 grants awarded to 47 distinct PIs. Of these grants, 24 do not appear as active, funded awards at present. Thus, approximately 50% of the R01s in years 40+ in FY2010 are still funded at present and 50% are not.

Of the PIs corresponding to the 24 year 40+ R01 grants that are no longer funded, 9 still have other NIH funding at present. In most cases, these are other long-standing (but less than year 40) grants while in a few cases they appear to be new projects.

One response so far

Should grants be limited to a single renewal?

(by datahound) Feb 10 2015

In the context of the discussion of the "Emeritus Award" from NIH, Neuro-conservative commented:

I am curious what you (and others here) would think about limiting grants to a single renewal, or any other limitations on duration? I previously thought it reasonable that renewal of ongoing solid work should be slightly favored within the system. But I think that Prof. Rosenbaum has inadvertently persuaded me otherwise.

Thoughts?

13 responses so far

Longevity and Transitions for R01s in Years 40+

(by datahound) Feb 07 2015

In the context of the potential "Emeritus Award" discussion, two of the points on interest were (1) an understanding of the situations of the senior investigators to whom such an award mechanism would be presumably targeted and (2) the fact that mechanisms already exist for transitioning labs to more junior faculty if that is desired. To get a look at one aspect of this, I examined active R01 grants in years 40 or larger. Of course, this is an atypical slice of this pie as many investigators, even if they have been continuously funded for decades, have not done so on individual grants that have been renewed.

I identified 62 active R01 grants in years 40-58. These were held by 59 investigators (three investigators each had two R01s on the list). Seven of the grants included co-PIs. The ages or year of degree could be identified for most investigators through internet searches. For 13 of the grants, it appeared that the grants had been transferred from another PI at some stage of its existence. In two cases, this appeared to be due to the death of the original PI. In seven cases, the point of transition could be identified and the original PI could be identified and all appear to be still alive. In these cases, the ages of the original PIs at the time of transition were estimated to range from 56 to 86 with a median of 74 while the ages of the PIs to which the grant was transferred were estimated to range from 42 to 65 with a median of 50. In the remaining four cases, the point of transition could not be identified, but the current PIs did not appear to be old enough to be the original PIs.

Overall, the ages of the current PIs for these grants are estimated to range from 49 to 93 with a median of 74. The ages at which the original PIs were awarded these grants were estimated to range from 24 to 40 with a median of 32.

21 responses so far

Request for Information: Potential Emeritus Award for Senior Researchers

(by datahound) Feb 04 2015

Even though I have heard discussion of the concept over the years, I must admit I was a bit stunned to see the Request for Information (RFI) from NIH regarding a "Potential Emeritus Award for Senior Researchers". The introduction for this RFI reads (in part):

An important issue for NIH is the long term succession planning for the research we support.  Over the years, NIH has been persistent and creative in efforts to support early career investigators through policy changes and new programs.  But we must also consider the needs of our senior investigators and how NIH can assist with the continuation of their well-established research programs, should they wish to transition to new positions.  While many senior investigators may be happy pursuing their research questions in the laboratory, others may be looking to move into other roles, such as full time teaching and mentoring.  Our senior investigators have invested their careers to establish the intellectual and technical infrastructure needed to pursue their research questions, and even if they wish to pursue new roles, they may not wish to dismantle their long-standing programs.

I find many aspects of this request surprising. These include:

(1) This problem already has a solution. An investigator can (with approval from the relevant IC) name a new Principal Investigator for a grant. Assuming the PI is qualified and NIH approves, this is an effective transition strategy that has been used many times.

(2) For most research programs, is "succession planning" something that NIH staff are worried about? Given that many investigators train numerous younger scientists over the course of their careers and that the system is currently flooded with accomplished younger scientists, the solution to this problem without any mechanism seems to be at hand.

(3) Even proposing such a mechanism seems quite inappropriate and tone deaf at this juncture when so many younger scientists are struggling to establish and maintain their careers.

Let me add two more personal observations. First, as someone who has changed roles several times over the course of my career, I know it can be done without any formal mechanism. Changing career directions can be a bit scary but I have been blessed with some tremendous opportunities and am glad that I have followed combinations of my heart, my brain, and my family to pursue them. I have developed many new skills and have had the privilege of going through the tenure process four times. In my experience, you just have to try to do the right thing, for yourself, your family, and your communities.

Second, when I was at NIH, I discovered that a senior and very accomplished faculty member had not tried to renew his R01. I emailed him and asked what was up. He said 'I have sources for some other funding and it is time to give someone else a turn'. I have considerable admiration for many senior scientists who have accomplished much over the course of their careers, but there does come a time when it is time to give someone else a turn.

NIH's requests information about:

  • Community interest in an emeritus award that allows a senior investigator to transition out of a role or position that relies on funding from NIH research grants
  • Ideas for how one would utilize an emeritus award (e.g., to facilitate laboratory closure; to promote partnership between a senior and junior investigator; to provide opportunities for acquiring skills needed for transitioning to a new role)
  • Suggestions for the specific characteristics for an emeritus award (e.g., number of years of support; definition of a junior faculty partner)
  • Ways in which NIH could incentivize the use of an emeritus award, from the perspectives of both senior investigators and institutions
  • Impediments to the participation in such an award program, from the perspectives of both senior investigators and institutions
  • Any additional comments you would like to offer to NIH on this topic

I hope you all will take advantage of the opportunity and share your thoughts. I certainly plan to.

42 responses so far

Older posts »