What do you see as the strengths and weaknesses of each criterion?

The current Broader Impacts merit review criterion includes the consideration, “Will the results be disseminated broadly to enhance scientific and technological understanding?” NSF should retain and strengthen this consideration of the merit review.

Broad dissemination must be a high priority for NSF-funded research. As a federal agency operating with taxpayer funding, NSF has a responsibility to maximize return on investment by removing barriers to access for the scientific community, as well as to ensure access for taxpayers. In addition, broad and equitable dissemination of scientific information advances the goals of the Universal Declaration of Human Rights, which affirms that “everyone has the right freely to … share in scientific advancement and its benefits,” and the International Covenant on Economic, Social and Cultural Rights, in which governments agree to take the steps “necessary for the conservation, the development and the diffusion of science;” the United States is a signatory to both documents.

Under the current criteria, which are more than a decade old, proposals may be inadequately reviewed with regard to the dissemination consideration. During that time, the landscape of scholarly publishing and information has undergone significant changes. In particular, the exceptional opportunities created by the Internet behoove NSF to ensure that its funded research takes full advantage of the new technology to maximize cost-effective dissemination.

At present, broad dissemination of results is promoted through two avenues at NSF: the merit review criteria, which investigators must address in their proposals, and the Award and Administration Guide (AAG), which governs projects after an award is issued. This dual approach is beneficial and should be maintained and strengthened.

NSF establishes minimum requirements for dissemination in Chapter VI.D.4 of the AAG. In particular:

• “Investigators are expected to promptly prepare and submit for publication … all significant findings from work conducted under NSF grants;” and
• “Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants.”

This policy is underpinned by Sec. 7011 of the America COMPETES Act of 2007, which requires NSF to enforce the standards by making any researcher who fails to comply ineligible for future funding.

Additionally, Sec. 7010 of the 2007 COMPETES Act requires NSF to make project reports freely available to the public online, along with citations to any publications resulting from NSF funding. This provision of the law is implemented by Chapter II.E.3 of the AAG, which requires grantees to submit a report describing the project outcomes, written specifically for the public, to be made freely available via Research.gov.

In addition to the baseline standards of the AAG, investigators are required to address the dissemination consideration of the Broader Impacts merit review criterion. This dual approach is beneficial because it encourages creative approaches to dissemination beyond the minimum, as appropriate to the proposed activity. Moreover, by including dissemination in the merit review criteria rather than only in post-award requirements, NSF ensures that investigators plan for dissemination beginning from the proposal stage, a valuable way to keep dissemination in mind throughout the project’s life cycle.

Unfortunately, both aspects of this approach currently are insufficient. Both the dissemination requirements of the AAG and the dissemination consideration of the Broader Impacts criterion should be updated and strengthened.

While NSF is to be recognized for its leadership in recently requiring investigators to develop data management plans, in other regards the AAG is outdated and should be reformed. Most importantly, NSF has not implemented a requirement that its funded investigators provide public access to resulting peer-reviewed manuscripts, rather than merely the citations to those publications. Among federal science agencies, the National Institutes of Health (NIH) is the current leader in this area, having adopted a mandatory public access policy as required by Sec. 218 of the Consolidated Appropriations Act, 2008. Dozens of other public and private research foundations worldwide have successfully adopted similar policies. NSF should incorporate a similar mandatory policy in the AAG. Furthermore, NSF should improve on the NIH policy by reducing the “embargo” period when manuscripts can be withheld from public access from the current maximum of twelve months to a six-month maximum, as several other research funders worldwide have done.

However, as this comment is directed to the Task Force on Merit Review, I will focus on the dissemination consideration of the Broader Impacts criterion, rather than the requirements of the AAG. To be clear, even if the dissemination requirements of the AAG are strengthened, the dissemination consideration of the merit review criteria also should be retained and strengthened.

The first regard in which the dissemination consideration of the merit review criteria is inadequate is its construction. Currently, broad dissemination is a consideration only of the Broader Impacts criterion, not the Intellectual Merit criterion. However, dissemination is properly understood as fundamental to both the intellectual merits of the proposed activity as well as its broader impacts.

The thrust of the current consideration is that maximizing the social value of the funded project requires communicating the project’s results to relevant audiences outside the research community, such as industry and policymakers, and to the public to enhance the public understanding of science. This is useful strategy to increase the broader impact of NSF funding and should be maintained. In particular, it supports the requirements of Sec. 526 of the America COMPETES Reauthorization Act of 2010, which directs NSF’s Broader Impacts criterion to advance the goal of “increased public scientific literacy.”

However, even within the academic community, the broad dissemination of research results cannot be taken for granted. Access barriers imposed by the high and rising cost of serials and monographs can significantly hamper the circulation of knowledge. These barriers can be particularly imposing to researchers and students at smaller institutions and in developing countries. Additionally, reticence to share data or materials with other researchers, or delays in doing so, also hinder the progress of science.

Thankfully, online technologies enable innovative approaches to the broad dissemination of research information which previously was only shared in small circles. These approaches already have begun to bear fruit. For instance, the NIH-supported Alzheimer’s Disease Neuroimaging Initiative was recently highlighted by the New York Times for its innovative approach to data sharing which is already being emulated. Given the promise of openness, the Board should ensure that the merit review criteria promote the broadest possible dissemination of results.

What changes, if any, would you like to see made to the merit review criteria?

If the Board retains the current criteria, it should add a consideration for broad dissemination in the Intellectual Merit criterion, in addition to the current consideration in the Broader Impacts criterion. Such a consideration might read, “Will the proposed activity ensure the broadest possible access to its results within its own field or across different fields?”

Alternatively, the Board might adopt a single consideration that addresses broad dissemination both within and beyond the research community.

What role should the institution play to ensure that the intellectual merit and broader impacts in NSF proposals can be realized?

NSF should provide additional guidance to proposers on how best to address the dissemination consideration. The Task Force will be aware of Sec. 526 of the COMPETES Reauthorization Act, which directs NSF to better inform proposers as well as staff and reviewers about the Broader Impacts criterion’s requirements, among other changes. The process of implementing these statutory provisions offers a timely opportunity for NSF to suggest “proven strategies and models” for cost-effective broad dissemination.

Currently, NSF provides a list of examples of representative activities for the dissemination consideration. These examples should be updated to better take advantage of proven strategies. Most critically, NSF should encourage investigators to provide open access to their publications by publishing them in an open access journal or by depositing them in an open access repository immediately upon publication, under an open copyright license. NSF could also encourage investigators to consider the impact of their publishers’ policies and pricing on the broad dissemination of their research. Further, NSF should encourage investigators to post their data online for free public access, and to freely distribute online any software created resulting from NSF funding under a free software license, depositing both in appropriate repositories as applicable.

Beyond mere suggestions of activities, NSF should refer investigators to resources on how to accomplish these activities, such as existing guides prepared by the library community and others. NSF should also encourage institutions to play a more proactive role in supporting investigators in maximizing the dissemination of their research, including collaboratively sharing resources and strategies between institutions.

Additionally, NSF should ensure that these resources and guidance are available to its reviewers and staff, and that they duly take into account these activities in reviewing proposals’ commitment to broad dissemination of research.

What impact, if any, has NSF’s two review criteria had on how you think about developing your research projects?

Any other comments?

Thank you for the opportunity to comment on the merit review criteria. Please do not hesitate to contact me with any questions.

I haven’t used OJS as an author, editor, referee, publisher, nor as a sysadmin, but I have used it as a reader. It’s generally very usable, although there are a few areas, mostly related to current awareness, where some simple tweaks to the defaults would make things easier.

• Make subscribing easier and more obvious. When you visit an OJS using a standard theme, there’s no big button that says “subscribe” or a similar term. Instead, there are two options that lead down that path: “Register” and “For Readers”.

  The “For Readers” page, by default, directs readers to register to receive the table of contents of new issues via email. The “Register” page, by default, requires you to create a username and password, fill out a captcha, and give your full name in addition to your email address. It also asks, optionally, for your gender, mailing address, and other information. That’s a lot of effort just to get an email when new issues are released.

  There’s a reason: OJS uses the same page to register authors and reviewers — situations where more information than just an email address is required. There may be a bit of wishful thinking here, too: the hope of converting readers into authors and reviewers. But let’s cross that bridge when we come to it. For now, let’s just get people on the mailing list. This is a tenant of Web design (and publicity generally): convert one visit into future contacts. That’s why barackobama.com started with a splash page which asked for one thing: your email address.

  A journal should make a prominent pitch for visitors to subscribe before they navigate away from the page and forget about the journal. More subscribers leads to more readers, which leads to more authors and referees and commentary.

  The OJS default themes should include a sidebar section that says “Enter your email to receive free announcements when a new issue is released”. It should ask for only an email address. (The confirmation email can ask subscribers to register a username, if desired.)

• RSS feeds by default. OJS includes a plugin to produce RSS feeds, but it doesn’t appear to be on by default; many OJS journals don’t offer RSS feeds. See above comments about the importance of turning visitors into subscribers.
• OpenID support. With 2,000 OJS journals floating around, it seems a bit silly to have to create an account at each one, doesn’t it? OpenID would give users a single login not only across other OJS journals, but any site supporting OpenID. Good news, though: OpenID support is in the OJS roadmap.

As a closing comment, I’ll point out that OJS is under active development and progressing quickly. I was going to make a comment on how feed display is ugly, based on Open Medicine as an example, but I thought I ought to check the versions first, to make sure the problem hadn’t been resolved. Sure enough, Open Medicine is using OJS 2.1.1; the release notes of more recent versions mention improvements to feed handling, so this may have been fixed already.

My new column for Free Software Magazine is titled “Free software vs. software-as-a-service: Is the GPL too weak for the Web?”. It looks at the rise of the Web application — including Facebook, Flickr, Google Apps, and the like — and how to preserve user freedom in this new environment. You can read the article or download it as a PDF.

Update: Slides are here. Thanks to a great audience!

I would expect libraries to be receptive to the message of eliminating DRM — DRM is diametrically opposed to libraries’ missions of access and preservation, and is an awful abuse of copyright law against the public interest. But it’s hard to argue that libraries should forgo access altogether to DRM-crippled content if they can’t negotiate out of it or acquire the content from another source. (Exhibit A: The Boston library’s response.) With that said, there’s plenty of room for library pushback against DRM, which could further signal vendors that there’s no market for it. We’ll see how this turns out.

The EPrints package is even included in the GNU Project. I am very disappointed in this move; they should know better. Supporting Apple’s anti-competitive business practices that limit users’ freedoms is not in the public interest.

You do realize that OS X doesn’t even come with Firefox pre-loaded, right?

And you do realize that other manufacturers offer laptops pre-loaded with Firefox and Linux, right?

So why in the world should the prize for promoting open source be closed?

Update: Mozilla is also giving away a Macbook for their Extend Firefox contest. For the avoidance of doubt: “Apple is not a sponsor of the Extend Firefox competition.”

I’ll post the presentation here when I have a chance.