Becoming Reviewer 1
One of my altruistic goals during the pandemic was to try and keep the scientific enterprise running so I served on many study sections for both the National Institutes of Health and several foundations. As many people were overwhelmed and taking care of kids, funding officials were struggling to find reviewers, so I rarely said "No" or at least tried to negotiate a load that I could handle. I learned many things and stretched from project grants (R01 or R21-type) to program grants and fellowships with the additional selfish goal of figuring out what makes different types of grants successful. As I finally became comfortable with different review processes, I think the most important thing I learned was to become an effective Reviewer 1.
Every grant usually has three reviewers (if you need a primer on NIH review there are a few posts in my old blog and on the NIH website). Reviewer 1 is the first to speak and has the responsibility to outline the goals and significance of the grant providing score-driving strengths and weaknesses. They are theoretically expected to have the longest time allotment during the review….until someone else goes “Reviewer 1 did an excellent job summarizing the application, I really don’t have much to add” and then talks for 10 minutes.
While every reviewer must review the grant thoroughly, Reviewer 1 can hold the most power, setting the tone of the discussion for the entire study section. They can make or break a grant. Summarizing the application clearly without getting lost in minutiae, predicting and anticipating possible blows from other reviewers, and arguing my case, took a while to learn. Not sure I have mastered it yet, but I have definitely improved from the very first time when I was terrified of making a fool out of myself.
First up, the summary.
A full review load of 7-8 grants is not a trifling matter. It can take a long time to get through and in the end they all kind of turn into word soup. You may get 2-4 where you are Reviewer 1 and usually, 1-2 will survive triage. I find that going through the grant again the night before the study section and preparing a written summary is best. The review officers usually discourage reading the points in the critique verbatim and trying to make a summary on the fly is not the best way to collect your thoughts. I tried.
Most people on the panel will not have read the grant and will have time to glance at the specific aims and at the figures, so they rely on Reviewer 1 to orient them on the work. In general, I write a short (1 page or so) narrative of the grant picking and choosing from the summary and subsections of the NIH critique templates so that I touch on significance, innovation, investigators, approach, etc. It is particularly important to be clear on the significance and major goals of the project establishing why this would be impactful (or not) if successful. You then follow with concerns about the experimental design, investigator expertise, and/or environment. If I loved it and another reviewer hated it, I make sure to also go over their concerns carefully and if I am still convinced I am right, I add something to the effect “I know Reviewer 2 had issues with the design of experiment X, but that didn’t concern me too much because…”
The best Reviewer 1s I have seen are crystal clear and to the point on why they loved a proposal or hated it or think it still needs some work. They don’t drone on and on and keep the attention of the group. In a fully virtual environment, this is more critical than ever, but it’s the same after lunch or at 4pm after a full day of reviewing in a room.
Then, the discussion.
When you’re done with your summary you pass the baton to Reviewer 2. If you all pretty much agree andeveryone’s impact scores are close, it’s easy. You listen, possibly answer some questions from the panel, and then restate scores to move on with the vote that will seal the fate of the grant. The final scores of the three reviewers set the mythical “range” that the entire herd of reviewers will remain inside of unless someone really objects and declares themselves “outside the range”.
If scores are spread out, let’s say a 2 and a 6, there may or may not be a lively discussion. This is where you have the power to decide someone’s fate, especially if you’re the reviewer that has to sway everyone towards a 2. You will have had access to everyone’s critiques and you will know what the other reviewers are going to say.
Sometimes, you were super excited about the idea, but you knew the grant had minor problems, and the reviewer with a lower score found major technical issues. You knew you were going to come down and may state so at the beginning of your summary, but you may still want to reach a consensus that is “encouraging” to the investigator, like a 4. (Note: a 4 is a good score…I know people think it’s not, but it’s a good score!) You need to prepare to counter, especially if you think the other reviewer is being nitpicky and you need them to improve their score. More often than not the discussion is cordial and very productive. This is also where allies come into play, in the form of the other reviewer who may be agreed with you and can back you up to shift the panel.
Sometimes scores realign just by virtue of panel calibration which is the process by which in the course of the first few grants the entire panel decides what constitutes a 2, or a 4, or a 6. I will write a whole post about panel calibration because it’s fascinating to watch and not always good for the applicants. There is a strong human component in how a study section functions and most people want to follow the calibration, even if they will complain endlessly about score compression or score inflation afterward. Anyways, it happens that someone comes down 1-2 points just by virtue of calibrating their scores to the group.
More rarely, the dueling reviewers remain entrenched in their positions and you may need to utter those scary words you were raised not to say, especially as a woman, “I disagree.” The panel is now confused, the range is very wide, and the “sheepdogs” are fighting. Some will ask questions to decide where to vote, some will vote firmly in the middle, and some will choose to side with whatever “dog” is most trustworthy to them. I never actually know how things go in those situations. I imagine it’s very rare for one of those grants to get funded.
In the end, the chair summarizes the discussion, and all three reviewers check that the main points will be reported fairly. Then Reviewer 1 is usually responsible for making sure that the additional criteria are met, budget, human subjects, vertebrate animals, etc.
And then you’re done. Unless of course you’ve changed your scores substantially and need to reflect that in the critique so that the investigator can make sense of the summary statement. 😀
Every grant usually has three reviewers (if you need a primer on NIH review there are a few posts in my old blog and on the NIH website). Reviewer 1 is the first to speak and has the responsibility to outline the goals and significance of the grant providing score-driving strengths and weaknesses. They are theoretically expected to have the longest time allotment during the review….until someone else goes “Reviewer 1 did an excellent job summarizing the application, I really don’t have much to add” and then talks for 10 minutes.
While every reviewer must review the grant thoroughly, Reviewer 1 can hold the most power, setting the tone of the discussion for the entire study section. They can make or break a grant. Summarizing the application clearly without getting lost in minutiae, predicting and anticipating possible blows from other reviewers, and arguing my case, took a while to learn. Not sure I have mastered it yet, but I have definitely improved from the very first time when I was terrified of making a fool out of myself.
First up, the summary.
A full review load of 7-8 grants is not a trifling matter. It can take a long time to get through and in the end they all kind of turn into word soup. You may get 2-4 where you are Reviewer 1 and usually, 1-2 will survive triage. I find that going through the grant again the night before the study section and preparing a written summary is best. The review officers usually discourage reading the points in the critique verbatim and trying to make a summary on the fly is not the best way to collect your thoughts. I tried.
Most people on the panel will not have read the grant and will have time to glance at the specific aims and at the figures, so they rely on Reviewer 1 to orient them on the work. In general, I write a short (1 page or so) narrative of the grant picking and choosing from the summary and subsections of the NIH critique templates so that I touch on significance, innovation, investigators, approach, etc. It is particularly important to be clear on the significance and major goals of the project establishing why this would be impactful (or not) if successful. You then follow with concerns about the experimental design, investigator expertise, and/or environment. If I loved it and another reviewer hated it, I make sure to also go over their concerns carefully and if I am still convinced I am right, I add something to the effect “I know Reviewer 2 had issues with the design of experiment X, but that didn’t concern me too much because…”
The best Reviewer 1s I have seen are crystal clear and to the point on why they loved a proposal or hated it or think it still needs some work. They don’t drone on and on and keep the attention of the group. In a fully virtual environment, this is more critical than ever, but it’s the same after lunch or at 4pm after a full day of reviewing in a room.
Then, the discussion.
If scores are spread out, let’s say a 2 and a 6, there may or may not be a lively discussion. This is where you have the power to decide someone’s fate, especially if you’re the reviewer that has to sway everyone towards a 2. You will have had access to everyone’s critiques and you will know what the other reviewers are going to say.
Sometimes, you were super excited about the idea, but you knew the grant had minor problems, and the reviewer with a lower score found major technical issues. You knew you were going to come down and may state so at the beginning of your summary, but you may still want to reach a consensus that is “encouraging” to the investigator, like a 4. (Note: a 4 is a good score…I know people think it’s not, but it’s a good score!) You need to prepare to counter, especially if you think the other reviewer is being nitpicky and you need them to improve their score. More often than not the discussion is cordial and very productive. This is also where allies come into play, in the form of the other reviewer who may be agreed with you and can back you up to shift the panel.
Sometimes scores realign just by virtue of panel calibration which is the process by which in the course of the first few grants the entire panel decides what constitutes a 2, or a 4, or a 6. I will write a whole post about panel calibration because it’s fascinating to watch and not always good for the applicants. There is a strong human component in how a study section functions and most people want to follow the calibration, even if they will complain endlessly about score compression or score inflation afterward. Anyways, it happens that someone comes down 1-2 points just by virtue of calibrating their scores to the group.
More rarely, the dueling reviewers remain entrenched in their positions and you may need to utter those scary words you were raised not to say, especially as a woman, “I disagree.” The panel is now confused, the range is very wide, and the “sheepdogs” are fighting. Some will ask questions to decide where to vote, some will vote firmly in the middle, and some will choose to side with whatever “dog” is most trustworthy to them. I never actually know how things go in those situations. I imagine it’s very rare for one of those grants to get funded.
In the end, the chair summarizes the discussion, and all three reviewers check that the main points will be reported fairly. Then Reviewer 1 is usually responsible for making sure that the additional criteria are met, budget, human subjects, vertebrate animals, etc.
And then you’re done. Unless of course you’ve changed your scores substantially and need to reflect that in the critique so that the investigator can make sense of the summary statement. 😀
Do you have additional tips for Reviewer 1? Add them in the comments.
Image credit: Great Pyrenees Guarding the Flock - Wikimedia Commons
Comments
Post a Comment