NIH "High Risk" Programs-Gender Issues-Part 1-Pioneer Program

Oct 12 2015 Published by under Uncategorized

With the recent announcement of the "high-risk" research awards from the NIH, a discussion on Twitter began around the relatively low number of women awardees for these awards. I will return to this issue later but first I want to provide some background. For this post, I will focus on the first in this suite of what is now four programs.

The NIH Director's Pioneer Award (DP1) was initiated as part of the original NIH Roadmap for Medical Research. The award was intended to be an experiment driven by a frequent concern raised to Elias Zerhouni, then the NIH Director, that there were a number of highly innovative researchers in fields relevant to the NIH mission who were not applying to NIH because the relatively opacity and complexity of the NIH application process.

There were a large number of nominees (the program initially involved nominations rather than applications) and most of the nominations were submitted shortly before the deadline catching the NIH slightly off guard (more of this later). The Pioneer application involves a 5 page essay (rather than the more standard R01-type application of 25 pages at the time) and 22 of the most highly rated applicants are interviewed in person in Bethesda. This program was initiated just as I was starting my position as Director of NIGMS and I was not involved in the program in the first year.

When the first Pioneer awards were announced in September of 2004, I was surprised and disappointed by the outcome. There were nine recipients, several of whom were relatively well established within the NIH community including, for example, Steve McKnight (who was already well recognized within NIH as an innovative and productive scientist although he has gone on to make some controversial statements about the scientific community) and Homme Hellinga (who was recognized as a rising star at the time although much of his research has turned out to be, at best, irreproducible). I was hoping to go back to my office to google the awardees because I had not heard of them or did not know much about them. In addition, all nine of the awardees were male and this, appropriately, raised concerns within the scientific community both outside and inside NIH.

After the next meeting of the Institute and Center directors, I was sharing my views with Raynard Kington, then Deputy Director of NIH. He listened carefully and told me that Dr. Zerhouni needed to hear such concerns and I dutifully went back to my office and composed a long email. A couple of days later, I walked into a meeting at which both Drs. Kington and Zerhouni were present. They called me over and asked if I/NIGMS would like to take over running the Pioneer program. I was delighted if a bit daunted by this opportunity and asked some of my key colleagues including former acting-NIGMS Director Judith Greenberg if she would be willing to help with this effort.

We had a bit of time to review the processes that were used the first year and made a number of small changes including removing a "leadership potential" criterion that was used the first year since it seemed to peripheral to the goals of the program and had the potential to introduce biases of various sorts, allowing self-nominations and later applications, recruiting a more diverse pool of reviewers (more on this later), reaching out more aggressively through many outlets about the Pioneer program, reminding applicants and reviewers at all stages that "pioneering" researchers are quite diverse in all dimensions including gender, race and ethnicity, field, and career stage.

We again received a large number of applications and the process worked fairly smoothly. The end result was 13 awardees in a wide range of fields and career stages including 7 men and 6 women. As one would expect given access to $500K per year for 5 years as well as a competitive selection process, these investigators have done quite well, some exceptionally so.

The process continued for several more years with relatively similar results. After a total of five years were complete (so that we would have a reasonable data set), we initiated a process evaluation. This was completed and released in 2010. This is quite a thorough report and I encourage interested readers to have a look in its entirety.

With regard to gender distributions of Pioneer applicants, interviewees, and awardees, the key findings were:

The percentage of female applicants ranged from 22% to 27% with a mean of 25%. This number increased the year after we took over the program, a reassuring results after the results of the first year.

The percentage of female interviewees was 27% and the percentage of female awardees was 29%. The differences between these percentages and the applicant pool were not significantly significant.

The percentage of female awardees at 29% was higher than the percentage of female R01 awardees over the same period (23%).

One striking and distress result from the first year was the percentage of women among the reviewers. These results are shown below:

Pioneer Evaluators

While it is important to keep in mind that gender makeup of a review groups often does not eliminate or even reduce unconscious gender bias (example), the results from the first year of the Pioneer program were quite worrisome. The NIH staff running the program did not anticipate the number of nominees (1331) and had to scramble to recruit enough reviewers on short notice. With that constraint, the result was 59 men and 4 women including only 1 woman on the interview committee.

As an aside, the first years of the Pioneer program were run before Grants.gov existed. A special system had to be built and this allowed collection of data about exactly when applications were submitted. The results for the first year that NIGMS ran the program are shown below:

Pioneer timing

This shows the number of nominations/applications as a function of the data from the opening of the submission site (3/1) to the closing date (4/1). This reveals that many applicants submitted within the last few days before the due date. In addition, the eventual awardees (shown with red bars) tended to submit late in the day including a few minutes before the deadline. I would never have some much faith in a website.

I will discuss some of the other programs in subsequent posts. For now, I welcome thoughts about this analysis of the Pioneer program including gender balance issues. I have submitted a FOIA request for information about the applicant pools for these programs for the current year so that I hope to have data to do some analysis beyond looking at the awardees.

24 responses so far

  • Pinko Punko says:

    I think Hellinga came across very poorly in that episode and makes me feel icky. I think that your word "much" might be problematic in assessing his research. How much of his output relates to the two-ish papers at issue?

    • datahound says:

      I meant much of his research that formed the basis for his receiving the award. The two papers involved generating an enzyme by designing an active site into a binding protein. The results could not be reproduced by other labs and the activity appeared to be due to contaminating E. coli enzyme. In addition, site-directed mutants showed reduced by still present activity. This is hard to reconcile with the contamination issue without considerable selective use of data. Duke completed an investigator but the results were never made fully public to my knowledge.

      In addition, some of his work on binding proteins could not be reproduced. One of his former postdocs published a paper in PNAS shortly after she presented at the NIH symposium that called into question this work. See http://www.pnas.org/content/106/44/18491.abstract.

      He has not received any NIH funding since the Pioneer award and most of his subsequent papers are will other senior authors from Duke.

      I did not mean to imply that all of his work was problematic but he had several episodes where high profile results could not be confirmed and it is often hard to reconcile his conclusions with the results of actual experiments.

      • Pinko Punko says:

        I was aware of the main episode you mention, and I did not realize that was the basis for his Pioneer award. What I was not aware of was other episodes of poorly reproduced science. I think his behavior, and Duke's behavior were completely inappropriate in terms of how he very obviously sacrificed his student. I'm not fan of that guy at all. Just making sure that you were being fair as possible (and as you usually are).

  • Spiny Norman says:

    There also remains the never-resolved question of how Hellinga's purportedly designed enzyme — now known to be inactive — could possibly have genetically complemented an E. coli strain deleted for the native enzyme (triose phosphate isomerase).

  • drugmonkey says:

    Wait. I thought HH was busted. No?

  • datahound says:

    While the Hellinga saga is an important one, any thoughts about the Pioneer award and gender balance issues?

  • SS says:

    On a related note,in the very recent news release for the 4D Nucleome project (also part of the NIH Common Fund), only one of the 29 grants appears to have a female PI, at least as the lead (contact) PI.
    http://commonfund.nih.gov/4Dnucleome/FundedResearch

  • eeke says:

    I am a little confused about the nomination process - is it now all self nominations, or does this award still primarily solicit non-self nominations? If the latter, that will definitely skew the applicant pool toward men. My other question is how does the gender distribution for these awards compare to all other R01s (excluding 2004 - a blatant exercise in boy's clubbing)? I thought that women made up about 30% of the R01 awardees anyway?

    One other comment, I'm not sure why receipt of applications early vs. on the deadline would matter for anything. Why would the NIH expect anyone to submit their applications early - is this encouraged? Who does this? Moreover, it's the institution that submits, and not the applicant, right? For all you know, these applicants may have turned their materials into the research office early in the day, and it wasn't submitted by a signing official until the end of the day, which could be the case if the deadlines are shared with standard R01 deadlines.

  • Zuska says:

    I don't find it surprising that most people would submit apps close to deadline - that's human nature, to procrastinate and/or tinker w/the app till the end. It's striking that all the awardees appear to have come from those last minute, last days submissions though. Does fussing with the app till the bitter end really make it better?!?

    Why would NIH be surprised that people would submit last minute - is this an abnormal pattern? Even if the # of applicants was unanticipated & NIH had to scramble for reviewers, that does not explain the gender imbalance in the reviewer pool the first year. It just shows that when people aren't consciously trying to create a diverse group, they don't - they round up the usual suspects. Who generally happen to be dudes. If your usual suspects were a more gender balanced group to begin with - if you had a go-to group of women as well as men - then when you did a last minute scramble, you'd collect women reviewers as well as men. But you can't find women & build relationships last minute. Last minute work yields last minute results, which are shitty, in terms of diversity.

    Diversity is a muscle. You don't go to the gym once late Saturday evening & expect to be physically fit come Monday morning.

    The process you describe for later years of the program, wherein you consciously built a diverse reviewer pool and did upfront education with them about what a successful Pioneer awardee looked like, in terms of diversity of a variety of characteristics, yielded results that one would expect - higher percentage of women applicants, which was higher than other NIH program, and correspondingly percentage of women awardees.

    • datahound says:

      Thanks. I agree with you on all points. I was stunned when others at NIH would comment on the number of submissions a week before the deadline. I kept saying that this observation had very little information content since most people would submit in the last two days. Why would you not?

      I also agree with you about diversity. The 4 women, 59 men result for 2004 is a striking example of what happens if a group of scientists work to recruit "top" folks without any thought about other issues.

  • DJMH says:

    Agree w Zuska, It looks as though you worked to develop the pool, encourage diverse applicants, and it paid off in terms of eventual awardees.

    • datahound says:

      Yes, this is not "rocket science." Essentially everything that we did was well precedented in other sectors. It was especially gratifying when the final selection of awardees occurred with no discussion of gender since the process was reasonably balanced throughout.

      • Zuska says:

        It is impossible to over-emphasize how very important it is that you are discussing this process & the outcomes in public. Thank you.

        • datahound says:

          You are most welcome. As always, it is important to base such discussions on facts and data including experiences from other fields and social science studies. It is some much better to fix the process rather that jiggle the outcomes after a poorly conceived process.

          Thank you for your comments.

        • Spiny Norman says:

          Agreed with Zuzka. What you're doing here is hugely important & appreciated.

  • Ginther redux says:

    Apart from gender imbalance, the absence of underrepresented racial and ethnic minorities among the awardees is also very stark, even more so than other years, which is saying something, considering those other years might mean one or at the most two such investigators. It's appalling given NIH's purported concerns subsequent to the Ginther study. I really do not understand how the program staff find this to be acceptable.

    It's also true that behavioral and social sciences are essentially nil in funding rates in this program (and for the most part the other HRHR programs). This was mentioned in the evaluation of the Pioneer program but they haven't seemed to do anything to rectify it.

    I would like to see analyses for both awardees and reviewers by race and ethnicity and by disciplinary domain.

  • […] at NIH so that the percentages in the High Risk programs do tend to be low. However, as I noted in my previous post, it is difficult to interpret these percentages without knowledge of the pool of individuals who […]

  • […] have previously discussed the history of the NIH Director's Pioneer Award program. The daughter program of the Pioneer […]

Leave a Reply