National Academies Report Turns Up the Pressure for Large Facility Reform at NSF

Citing a host of concerns about how NSF ranks large research facility projects for funding, a National Academies panel has recommended that NSF “create and implement” a 10- to 20-year road map for such projects that would serve as a guide for both policymakers and scientists. In addition to defining and ranking projects, the report, released in mid-January, recommends the creation of independent oversight committees to monitor both construction and operations of the foundation’s scientific facilities.

“This is an opportunity for the National Science Foundation to further improve its work in the development of major research facilities by incorporating a more objective and transparent approach to screening and planning into its process,” said Princeton University physicist William Brinkman, who chaired the National Academies committee. Brinkman said the committee began with the premise that NSF had to be much more open to the scientific community and others about how it made facilities selections, “and that’s when we came up with the idea of road mapping.” For the report and a list of projects under discussion, see http://www7.nationalacademies.org/NSF-Priorities .

The report recommends that the road map should “take into consideration the need for continued funding of existing projects” and then “provide a set of well-defined potential new project starts for the near term,” defined as the next 10 years. For funding purposes, each project should be ranked against others based on a specific set of criteria developed by the National Academies panel. Proposals for facilities 10 to 20 years in the future “will necessarily be less well defined and ranked qualitatively to yield a vision of the future rather than a precise funding agenda,” the report says.

The report is detailed and specific to the point of bordering on micromanaging, according to one of the committee members. But it comes after several years of increasing criticism of NSF’s handling of the selection, funding, and development of large facilities. Indeed, six US senators requested the report in a June 2002 letter they sent to National Academy of Sciences President Bruce Alberts. The senators wrote that, “despite several efforts, questions remain as to whether NSF has a satisfactory process for prioritizing multiple competing large-scale research facility proposals. As a result, funding requests by the foundation for large facility projects appear to be ad hoc and subjective.”

The letter was written shortly after Christine Boesz, NSF inspector general, testified that the foundation had failed to address half of the recommendations she made in 2001 to improve management and oversight of the large facilities projects. According to the senators’ letter, Boesz’s testimony indicated that poor project management had resulted in “significant cost overruns not contemplated in [the projects’] original budgets.”

Although large facility projects represent less than 4% of NSF’s $5.5 billion budget, the report says they are important because of “their potential to shape the course of future research, the economic benefits they bring to particular regions, and the prominence of the facilities in an increasing number of research fields.” The report notes that the “ability of new projects to be considered for approval… has stalled in the face of a backlog of approved but unfunded projects.” The report also says that “the rationale and criteria used to select projects and set priorities among projects … have not been clearly and publicly articulated.” It further notes that there has been a dearth of funding for scientific disciplines to “conduct idea-generating and project-ranking activities” and little money for “conceptual development, planning, and engineering” to determine if a project is ready for funding.

Brinkman said that the creation of the road map recommended in the report would “blunt the criticism from Congress. It would allow policymakers to say, Here’s what the NSF wants to do, and here is where it fits in the big picture.”

Three-stage review

The report recommends that the road map be developed by moving large facilities proposals through three stages of review. In the first stage, researchers within a field or interdisciplinary area should determine which proposals have the greatest scientific potential and merit. The technological readiness of the proposed projects, the project scientists’ credentials, and the proposal team’s management capabilities should also be considered.

In the second stage, a proposal that survives the first stage should be reviewed by the leadership of the relevant NSF directorate to see how well it meets the foundation’s strategic goals. Such factors as impact on related fields, US workforce development, and interagency or international collaboration should be considered.

Finally, the National Science Board and NSF’s director should review recommended proposals based on “national criteria.” This final review should ask “which projects maintain US leadership in key scientific and engineering fields or enable the greatest numbers of researchers, educators, and students.”

The report notes that ranking projects across scientific disciplines “is inherently not an exact science. Nevertheless, these criteria … provide a framework for a discussion of why one project was accorded a higher priority than another and a mechanism for the discussion to be as objective as possible in ranking projects across fields.”

The report also recommends the following:

Brinkman said that OSTP Director John Marburger consulted closely with the committee in developing the report. The committee also looked at the US Department of Energy’s process for developing its 20-year road map for large science projects (see Physics Today, January 2004, page 23 ).

An NSF spokesman said the foundation was reviewing the report and was not ready to comment. Brinkman said NSF Director Rita Colwell and other NSF officials had been briefed by the committee and “they rightfully have to sit and look at this. There are some tough things they have to do.”