Comments on NSF’s Merit Review Criteria

Today, I submitted these comments to the consultation on the National Science Foundation’s Task Force on Merit Review (see). They are provided solely in my personal capacity.

What do you see as the strengths and weaknesses of each criterion?

The current Broader Impacts merit review criterion includes the consideration, “Will the results be disseminated broadly to enhance scientific and technological understanding?” NSF should retain and strengthen this consideration of the merit review.

Broad dissemination must be a high priority for NSF-funded research. As a federal agency operating with taxpayer funding, NSF has a responsibility to maximize return on investment by removing barriers to access for the scientific community, as well as to ensure access for taxpayers. In addition, broad and equitable dissemination of scientific information advances the goals of the Universal Declaration of Human Rights, which affirms that “everyone has the right freely to … share in scientific advancement and its benefits,” and the International Covenant on Economic, Social and Cultural Rights, in which governments agree to take the steps “necessary for the conservation, the development and the diffusion of science;” the United States is a signatory to both documents.

Under the current criteria, which are more than a decade old, proposals may be inadequately reviewed with regard to the dissemination consideration. During that time, the landscape of scholarly publishing and information has undergone significant changes. In particular, the exceptional opportunities created by the Internet behoove NSF to ensure that its funded research takes full advantage of the new technology to maximize cost-effective dissemination.

At present, broad dissemination of results is promoted through two avenues at NSF: the merit review criteria, which investigators must address in their proposals, and the Award and Administration Guide (AAG), which governs projects after an award is issued. This dual approach is beneficial and should be maintained and strengthened.

NSF establishes minimum requirements for dissemination in Chapter VI.D.4 of the AAG. In particular:

  • “Investigators are expected to promptly prepare and submit for publication … all significant findings from work conducted under NSF grants;” and
  • “Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants.”

This policy is underpinned by Sec. 7011 of the America COMPETES Act of 2007, which requires NSF to enforce the standards by making any researcher who fails to comply ineligible for future funding.

Additionally, Sec. 7010 of the 2007 COMPETES Act requires NSF to make project reports freely available to the public online, along with citations to any publications resulting from NSF funding. This provision of the law is implemented by Chapter II.E.3 of the AAG, which requires grantees to submit a report describing the project outcomes, written specifically for the public, to be made freely available via

In addition to the baseline standards of the AAG, investigators are required to address the dissemination consideration of the Broader Impacts merit review criterion. This dual approach is beneficial because it encourages creative approaches to dissemination beyond the minimum, as appropriate to the proposed activity. Moreover, by including dissemination in the merit review criteria rather than only in post-award requirements, NSF ensures that investigators plan for dissemination beginning from the proposal stage, a valuable way to keep dissemination in mind throughout the project’s life cycle.

Unfortunately, both aspects of this approach currently are insufficient. Both the dissemination requirements of the AAG and the dissemination consideration of the Broader Impacts criterion should be updated and strengthened.

While NSF is to be recognized for its leadership in recently requiring investigators to develop data management plans, in other regards the AAG is outdated and should be reformed. Most importantly, NSF has not implemented a requirement that its funded investigators provide public access to resulting peer-reviewed manuscripts, rather than merely the citations to those publications. Among federal science agencies, the National Institutes of Health (NIH) is the current leader in this area, having adopted a mandatory public access policy as required by Sec. 218 of the Consolidated Appropriations Act, 2008. Dozens of other public and private research foundations worldwide have successfully adopted similar policies. NSF should incorporate a similar mandatory policy in the AAG. Furthermore, NSF should improve on the NIH policy by reducing the “embargo” period when manuscripts can be withheld from public access from the current maximum of twelve months to a six-month maximum, as several other research funders worldwide have done.

However, as this comment is directed to the Task Force on Merit Review, I will focus on the dissemination consideration of the Broader Impacts criterion, rather than the requirements of the AAG. To be clear, even if the dissemination requirements of the AAG are strengthened, the dissemination consideration of the merit review criteria also should be retained and strengthened.

The first regard in which the dissemination consideration of the merit review criteria is inadequate is its construction. Currently, broad dissemination is a consideration only of the Broader Impacts criterion, not the Intellectual Merit criterion. However, dissemination is properly understood as fundamental to both the intellectual merits of the proposed activity as well as its broader impacts.

The thrust of the current consideration is that maximizing the social value of the funded project requires communicating the project’s results to relevant audiences outside the research community, such as industry and policymakers, and to the public to enhance the public understanding of science. This is useful strategy to increase the broader impact of NSF funding and should be maintained. In particular, it supports the requirements of Sec. 526 of the America COMPETES Reauthorization Act of 2010, which directs NSF’s Broader Impacts criterion to advance the goal of “increased public scientific literacy.”

However, even within the academic community, the broad dissemination of research results cannot be taken for granted. Access barriers imposed by the high and rising cost of serials and monographs can significantly hamper the circulation of knowledge. These barriers can be particularly imposing to researchers and students at smaller institutions and in developing countries. Additionally, reticence to share data or materials with other researchers, or delays in doing so, also hinder the progress of science.

Thankfully, online technologies enable innovative approaches to the broad dissemination of research information which previously was only shared in small circles. These approaches already have begun to bear fruit. For instance, the NIH-supported Alzheimer’s Disease Neuroimaging Initiative was recently highlighted by the New York Times for its innovative approach to data sharing which is already being emulated. Given the promise of openness, the Board should ensure that the merit review criteria promote the broadest possible dissemination of results.

What changes, if any, would you like to see made to the merit review criteria?

If the Board retains the current criteria, it should add a consideration for broad dissemination in the Intellectual Merit criterion, in addition to the current consideration in the Broader Impacts criterion. Such a consideration might read, “Will the proposed activity ensure the broadest possible access to its results within its own field or across different fields?”

Alternatively, the Board might adopt a single consideration that addresses broad dissemination both within and beyond the research community.

What role should the institution play to ensure that the intellectual merit and broader impacts in NSF proposals can be realized?

NSF should provide additional guidance to proposers on how best to address the dissemination consideration. The Task Force will be aware of Sec. 526 of the COMPETES Reauthorization Act, which directs NSF to better inform proposers as well as staff and reviewers about the Broader Impacts criterion’s requirements, among other changes. The process of implementing these statutory provisions offers a timely opportunity for NSF to suggest “proven strategies and models” for cost-effective broad dissemination.

Currently, NSF provides a list of examples of representative activities for the dissemination consideration. These examples should be updated to better take advantage of proven strategies. Most critically, NSF should encourage investigators to provide open access to their publications by publishing them in an open access journal or by depositing them in an open access repository immediately upon publication, under an open copyright license. NSF could also encourage investigators to consider the impact of their publishers’ policies and pricing on the broad dissemination of their research. Further, NSF should encourage investigators to post their data online for free public access, and to freely distribute online any software created resulting from NSF funding under a free software license, depositing both in appropriate repositories as applicable.

Beyond mere suggestions of activities, NSF should refer investigators to resources on how to accomplish these activities, such as existing guides prepared by the library community and others. NSF should also encourage institutions to play a more proactive role in supporting investigators in maximizing the dissemination of their research, including collaboratively sharing resources and strategies between institutions.

Additionally, NSF should ensure that these resources and guidance are available to its reviewers and staff, and that they duly take into account these activities in reviewing proposals’ commitment to broad dissemination of research.

What impact, if any, has NSF’s two review criteria had on how you think about developing your research projects?

Any other comments?

Thank you for the opportunity to comment on the merit review criteria. Please do not hesitate to contact me with any questions.

This entry was posted in Academia, Copyright, Creative Commons, FOSS, Open access, Publishing, Science. Bookmark the permalink.

Leave a Reply