Enabling the Nation’s future through
discovery, learning and innovation.
STREALIZING THE PROMISE OF THE 21 CENTURY DEPENDS
IN LARGE MEASURE ON TODAY’S INVESTMENTS
IN SCIENCE, ENGINEERING AND
MATHEMATICS RESEARCH AND EDUCATION.
NSF INVESTMENTS – IN PEOPLE, IN THEIR IDEAS, AND IN
THE TOOLS THEY USE – WILL CATALYZE THE STRONG
PROGRESS IN SCIENCE AND ENGINEERING NEEDED TO
SECURE THE NATION’S FUTURE.
NSF is a Source of Human Capital
FOR PEOPLE WHO WIN NOBEL PRIZES-
AND WHO BECOME TEACHERS,
AND CEOS, AND POLITICIANS.
WE FOSTER AND ENCOURAGE CREATIVE THINKING; WE
SELECT FOR EXCELLENCE; AND WE FUND THE BEST AND
THE BRIGHTEST BASED ON MERIT. WE ALL BENEFIT-AND
THE AMERICAN PEOPLE MOST OF ALL.
Rita Colwell, Director, NSF
Message from the Director
I am pleased to present the National Science Foundation’s Annual Program Performance Report for fiscal year 2000, as required by the Government Performance and Results Act of 1993 (GPRA). Here you will learn about NSF’s progress in meeting a broad range of challenging
goals that aim to improve our ability to invest in the nation’s future.
thThe year 2000 marked NSF’s 50 Anniversary. During this half century, knowledge about the
world has exploded and the pace of scientific discovery and technological innovation has accelerated unabatedly. By spurring innovation, advances in science and engineering knowledge lay the foundation for new jobs and fuel economic growth, raising U.S. living standards and improving the quality of life.
Through its public investments in fundamental research and education, the Foundation has played an important role in keeping the U.S. at the forefront of these developments. They have brought us advances in biomedicine, new modes of communication, and led to improvements in public safety, agriculture, and industry. They have enabled the U.S. to sustain a half-century of world leadership in science, engineering and technology.
Just consider the following achievements highlighted in the report:
; New evidence of life in extreme environments.
; The Boomerang project, which brought new insights into the “geometry” of the universe.
; New nanowires: self assembling structures that hold the potential to increase the
capacity of computer memory chips by a factor of 200.
; Substantial increases in student achievement – and a narrowing of the gap between
minority and majority students – through investments in educational systemic reform.
It is also noteworthy that of the 11 Nobel laureates announced in 2000, six have been supported by NSF.
NSF’s mission is to strengthen the nation’s capabilities across the entire spectrum of research and education in the sciences, engineering, and mathematics. Each year the Foundation invests in the creative people, the innovative ideas, and the cutting-edge technologies that will have the highest returns in advancing discovery at the frontiers of knowledge. It works to promote science, engineering, and mathematics learning – from pre-school through post-doctoral – to
prepare the next generation of scientific talent and foster a U.S. workforce that is second to none in the world.
Because the results of fundamental research are often realized only years later as they are transformed into the products and social benefits that improve our lives, NSF faces unique
challenges in measuring and evaluating performance. NSF has developed an evaluation process that reflects these challenges. Each year, NSF uses the GPRA review process as an opportunity to improve and refine the way it measures and rates its success.
In conducting its FY 2000 performance review, NSF raised the bar on performance. We increased the stringency of evaluation criteria, set goals that were significantly beyond past performance, added new categories of achievement, and submitted NSF activities to unprecedented levels of internal and external review. In addition to Foundation staff, about 400 external evaluators participated in the performance assessment, and generated 64 reports covering 78 of the NSF’s 200 programs.
Using these high standards, NSF met two-thirds of its 28 revised goals. The agency met six of its eight goals relating directly to the results of research and education, and outside evaluators noted progress toward realizing the other two. Among the management and investment process goals that were not met, NSF achieved substantial improvements in many areas. Examples are increased capacity to process proposals electronically, and an increase in the percent of awards involving new investigators – a key measure of the “openness of NSF’s system.”
In other areas, more stringent ratings produced results that were actually lower than in FY 1999, despite the fact that outside evaluators cited progress in those areas. One goal not met in FY 2000 – NSF’s successful implementation of the new merit review criteria – is the result of the
establishment of two new merit review criteria in FY 1998. For this performance report, the baseline for the agency’s evaluation (a period of three years) includes one fiscal year of merit
review that did not incorporate the new criteria.
NSF’s FY 2000 Report describes in detail the criteria NSF uses to assure the credibility of the data used to verify and validate progress in accomplishing its goals. This year, NSF submitted its data and methods to independent, external review by PricewaterhouseCoopers LLP.
It is our aim at NSF to adhere to the highest standards of management efficiency and integrity, and to produce outcomes of substantial benefit to the nation. I am therefore pleased to report that the data measuring NSF’s performance that are contained in this report are complete and reliable.
Rita R. Colwell
This report, made pursuant to the Government Performance and Results Act (1993), covers activities of the National Science Foundation during Fiscal Year 2000. It is substantially more comprehensive than its predecessor, and records an unprecedented level of effort and achievement.
In conducting the FY 2000 assessment, NSF undertook the most rigorous and challenging performance review in its history. Several goals and indicators were added or revised, and evaluation criteria were made much more stringent.
Previously used ratings such as ―partially successful‖ or ―minimally effective‖ were eliminated;
all outcomes were judged either ―successful‖ or ―not successful.‖ As a result, some of NSF‘s performance scores were actually lower than in FY 1999, despite the fact that outside evaluators cited progress in those particular areas since 1999.
In addition to Foundation staff, about 400 external evaluators participated in the performance assessment, and generated 64 reports covering 78 of the NSF‘s 200 programs. NSF engaged an independent outside examiner, PricewaterhouseCoopers LLP, to verify data compilations.
The Foundation aimed extremely high. For example, NSF set itself the goal of processing 70 percent of proposals within six months of submission. That would have been unmatched in the Foundation‘s history, and would far exceed the norm at comparable granting agencies. In fact,
during FY 2000 well over half of all NSF proposals were fully processed within six months and 71 percent were fully processed within seven months. By most measures, that would be considered extraordinary; yet it was not successful by NSF‘s high standards.
Similarly, NSF attempted to develop the technological capability that would permit the electronic review and processing of tens of thousands of competitive proposals each year - making it possible to do so without generating any paperwork within the Foundation. No other research and education funding organization in government has attempted such a feat. NSF did encounter significant technological challenges in trying to realize this goal, but robust progress was made and the agency will initiate pilot projects to demonstrate its electronic review process capability during FY 2001.
Other goals in which NSF was rated ―not successful‖ simply may not be achievable within a short time, such as ―improved achievement in mathematics and science skills needed by all
Americans‖. Performance assessment activities in FY 2000 clearly demonstrated that a goal of reaching all Americans was unrealistic. Consequently, NSF will revise its performance indicators in future years to focus on related aspects more directly within the agency's responsibility and control.
However, even with new and revised goals and more exacting definitions of success, the Foundation met two-thirds of its 28 goals, which are divided into three broad areas: outcome goals, management goals, and investment process goals.
Outcome goals concern the practical, concrete results of NSF grants and programs, as opposed to the procedures and methods whereby the Foundation carries out its work.
NSF achieved 75 percent (6 out of 8) of its goals in the Outcomes category, notably including:
; production of ―discoveries at and across the frontiers of science and engineering‖;
; rapid and widespread connections between those discoveries and society as a whole;
; ensuring that more than 80 percent of schools participating in education-improvement
projects called ―systemic initiatives‖ make substantial progress;
; providing intensive professional development programs for at least 65,000 teachers in
; prompt compilation and electronic dissemination of essential national data sets; and
; development of ways to determine and assure the quality of survey materials.
Two ambitious goals were rated as ―not fully successful‖ (that is, not successful) by reviewers.
One was the objective of ensuring that NSF grants contributed to improved scores in ―mathematics and science skills needed by all Americans,‖ as judged by independent external evaluators.
The other was the attempt to produce a ―diverse, globally oriented workforce of scientists and
engineers.‖ Both outcomes are extremely difficult to achieve, and neither is completely within the Foundation‘s responsibility and control. In NSF's pursuit of both goals, however, evaluators noted progress since FY 1999.
Management performance goals concern the effectiveness and efficiency of the way NSF handles its workload. For FY 2000, the Foundation identified 6 specific management goals, and achieved 5 of them (83 percent).
Goals that were met included:
; ensuring that at least 60 percent of full proposals are submitted electronically through the
computer-based ―FastLane‖ system;
; increasing the total number of science and engineering hires from under-represented
groups, as judged against the FY 1997 baseline (NSF achieved a 120 percent increase in
female hires and a 27 percent increase in minority hires);
; providing FastLane orientation for all NSF staff and insisting on practice in key modules
for at least 80 percent of program and support staff;
; completing work on all ―Y2K‖ information-technology problems as planned, on schedule
and within budget; and
; ensuring that at least 85 percent of all project reports are submitted electronically through
a new, computer-based Project Reporting System.
Only one management performance goal was not met. NSF had set itself the objective of having the technological capability to move competitive proposals submitted electronically through the entire review and award/decline process without generating any paperwork.
While this may have been extremely ambitious, significant progress was made. By the end of FY 2000, the only significant impediment to attaining a full electronic review capability was the development of a secure ―electronic signature‖. This issue will continue to be addressed in FY
2001 when the agency will pilot ten all-electronic review projects.
Investment process goals involve the specific procedures whereby NSF makes grants, funds and manages capital projects, and serves its customers in general.
For FY 2000, NSF identified 15 such goals. One was found to be inapplicable. Of the remaining 14, the Foundation clearly achieved 7, or 50 percent. Those included:
; allocating at least 90 percent of funds to projects reviewed by external peer groups and
selected through merit-based competition;
; identifying possible reasons for any customer dissatisfaction with NSF merit review and
; improving NSF‘s overall American Customer Satisfaction Index;
; devising systems that require Principal Investigators to integrate educational components
into their research proposals, and verifying the outcome;
; developing methods of requesting and tracking reviewer answers to NSF merit review
criterion - ―what are the broader impacts of the proposed activity‖;
; find ways to increase the number of women and under-represented minorities in the pool
of applicants for grants; and
; keep costs of construction and upgrades on facilities within 110 percent of estimates.
NSF did not meet 7 of its Investment Process Goals, even though efforts during FY 2000 often produced remarkable, measurable progress toward achieving those goals.
For example, NSF attempted to ensure that 95 percent of program announcements and solicitations would be available at least three months prior to proposal deadlines or target dates. In fact, 89 percent of announcements and solicitations met that standard - up from 75 percent in the preceding year - and 97 percent of program announcements and solicitations were available within 5 days of the three-month goal.
Another important area involves the goal of making at least 30 percent of competitive research grants to new investigators. In determining its performance on this goal, NSF counted only awards made to new Principal Investigators. During FY 2000, 28 percent of awards were made to new Principal Investigators – up from 27 percent in FY 1999. However, more than 33 percent
of FY 2000 awards were made to teams of Investigators where at least one Investigator was new – up from 31 percent in FY 1999. So although the goal was not achieved in its strictest interpretation, the results reported clearly demonstrate an increasing "openness in the system" that the agency is committed to maintain.
Finally, NSF has committed to ensure that external merit reviewers take both NSF criteria fully
into account when evaluating proposals. The two generic criteria are, simply put, scientific importance to the individual field, and broader significance to science and society as a whole. Although evaluators noted considerable improvement in this area, they rated NSF‘s overall performance as ―not fully successful", that is, not successful.
That outcome was not unexpected since the new merit review criteria were implemented only in early FY 1998. In assessing the agency's FY 2000 performance, external evaluators examined proposals considered for funding in FY 1997, FY 1998, FY 1999 and in a few cases, FY 2000. Therefore, only about two thirds of proposals examined actually could have been measured against the new merit review criteria. However, evaluators did note that for those proposals subject to the new review criteria (i.e. those submitted for funding consideration in FY 1998, FY 1999 and FY 2000), NSF staff had been generally successful in employing both criteria in making funding decisions. They also noted that NSF needs to increase its outreach efforts to the research and education community to ensure that proposers and reviewers alike adequately address both criteria in proposals and in the proposal review process.
Realization of this goal is increasingly likely over the next few years as NSF effectively communicates to proposers and reviewers the importance of addressing both criteria in proposals and reviews, and as evaluators examine a full complement of proposals subject to these criteria.
In four other goal areas, NSF did not successfully realize the high standards it set. In one, NSF had attempted to process 70 percent of proposals within six months of receipt. Although only 54 percent of proposals were processed within that time frame, 71 percent of proposals were processed within seven months. This represents an accomplishment, considering