Supercomputing Challenge

By Gladys Collins,2014-06-17 23:36
14 views 0
Supercomputing Challenge

    Supercomputing Challenge: Program Evaluation Design

    Introduction and Background

     The Supercomputing Challenge (referred to as the ―Challenge‖) is a not-for-profit

    entity with partnerships and relationships with the New Mexico Public Education

    Department, the New Mexico State Legislature, Los Alamos and Sandia National

    Laboratories, all state universities, all state community colleges, the Santa Fe Institute-

    Project GUTS, the New Mexico Computer Application Center, the New Mexico

    Information Technology and Software Association, numerous New Mexico computer

    associations, private businesses, professional education organizations, and scientifically

    connected communities. The purpose of the Supercomputer Challenge is to promote

    computational thinking in science and engineering so that the next generation of high

    school graduates is better prepared to compete in an information-based economy. With

    accountability being the watchword in public education for the last decade, at no other

    time has it been as important to provide evidence of effective programs and their impact

    on student learning.

    Although the positive impact of hands-on, reality-based education has been

    recognized in educational research for several years, the application of this information

    has been slow to find its way into routine instructional practice. Continued evaluation of

    innovative programs that put students in touch with the practical world of computing and

    provide relevant experiences that support their learning are essential in moving forward

    in educational reform for the 21

    st century. In the twenty years of the Challenge, various pre-event surveys have been administered and occasional post-event reports have been

    submitted, all of which have been useful to the management team for planning and

    development but perhaps not indicative of the programs full impact on participating

    teachers and students. Since 1995, when an internal evaluation was conducted, no

    comprehensive evaluation of the program and its various components has been completed.

    The Challenge provides a solid opportunity for examining the impact of a reality-based

    program in computing on New Mexico’s teacher and student participants.

The Supercomputing Challenge 1

    Evaluation Plan 2009-2010

    The mission of the twenty year-old Supercomputing Challenge is to teach teams of New Mexico middle and high school students how to use powerful computing techniques

    to analyze, model and solve real world problems. Through a yearlong program of events,

    teams of five or more students with a teacher sponsor compete for team awards,

    individual awards, independent awards, and scholarships. Program personnel are

    sensitive to the demographics and needs of all students in New Mexico, seeking a broad

    representation of the diversity of our state. Program staff is interested in knowing:

    (1) Who are the participants in the Supercomputing Challenge? Does the Challenge

    reach students across NM equally? Are minority students participating in the same

    numbers as others?

    (2) What are the perceptions of the participants? Which event, process, or incentive has

    contributed to their participation and learning? Are the programming, events,

    training, mentoring and incentives making a difference in student achievement? (3) What is the impact of the Challenge on student learning? What is the impact of the

    Challenge on teaching practices? Is achievement in math and science improving? Is

    the Challenge encouraging NM students to seek careers in computing? Are

    participants taking advantage of opportunities for STEM study in higher education?

    The Synergy Team has been contracted to assist the Board of Directors and management team by proposing an evaluation design that will supplement the developing

    strategic planning document and processes underway. The following evaluation design is

    based on the Synergy Team’s understanding of the Challenge’s complex system of

    formal and informal networks that comprise the program. Various entities have been

    solicited for funding, event volunteers, scholarships, prizes, and other services or awards

    to support the Challenge, resulting in many individuals being involved across various

    components and role groups. Thorough evaluation requires examination of all people,

    tasks and organizational needs that are necessary to achieve the program’s goals.

    Through a series of initial interviews with the Board of Directors’ chair and two of

    the management team members, the participants, events, relationships and expected

    outcomes of the Challenge were identified, resulting in the organizational chart in Figure

    1. Identified role groups include the Board of Directors, management team, Consult,

    The Supercomputing Challenge 2

    Evaluation Plan 2009-2010

teachers, students, volunteers, and partners. (See attachments for lists of current

    organization members by group.) With an understanding of who becomes Challenge

    participants and the relationships necessary in conducting the yearlong Supercomputing

    Challenge, the calendar of events illustrates the complexity of the program. Beginning

    with the July institutes and concluding with the April awards, nine ―events‖ constitute a

    year’s cycle for the Challenge. Figure 2 tracks the various events leading to the

    participating students receiving of scholarships and awards. Clarification of program

    goals, an understanding of the organizational framework, and tracking of the program

    events provide the basis of the evaluation plan, data sources, and timeline.

    Insert Figure 1 here.

    Insert Figure 2 here.

The Supercomputing Challenge 3

    Evaluation Plan 2009-2010

    Evaluation Design

    The Vision of the Supercomputing Challenge is to promote computational thinking in science and engineering so that the next generation of high school graduates is better

    prepared to compete in an information-based economy.

    The Mission of the Supercomputing Challenge is to teach teams of middle and high

    school students how to use powerful computers to analyze, model and solve real world


    Outcomes (goals and objectives) for Students

    Students participating in the Supercomputing Challenge will: (1) gain skills in computer programming, modeling, simulation, data collection and analysis; (2) devise computational

    solutions to scientific problems; (3) learn about challenges facing the environment; (4) understand properties of the physical universe; (5) look at trends in the social sciences that

    have implication for their own lives; and (6) develop 21

    st century career skills.

    Students participating in the Supercomputing Challenge will be representative of the cultural,

    ethnic and socio-economic diversity of the state and will demonstrate continuing interest in

    STEM initiatives.

    Evaluation Data Source Demographics Pre-survey (registration)





    Levels of participation Pre- and post-survey

     New participants Completion results

     Repeat participants

     Individual competitors

    Scholarships awarded Awards/scholarship data

     Money awarded


     Actual scholarship use

    Cash awards Awards data

    Senior follow-up Follow-up survey

     College graduation rates

     College majors

    Out-of-school activities Post-survey The Supercomputing Challenge 4

    Evaluation Plan 2009-2010

     Dual credit courses Interviews: Students

     AP courses

     MESA, science fairs, etc.

    Participation of Challenge alumnae Volunteer survey

Impact of the Challenge experience on Interviews: Students, teachers, volunteers

    participants End-of-activity feedback

Outcomes (goals and objectives) for Teachers

    Teachers participating in the Supercomputing Challenge will: (1) understand the complexity of science and computational tools and methods; (2) advance their knowledge, understanding

    and skills in STEM domains; (3) be able to prepare and support students participating in the Supercomputing Challenge; and (4) participate in professional development that provides

    opportunities for collaboration among peers and enhances student learning.

Evaluation Data Source

    Demographics Pre-survey


     Grade level




Levels of participation Pre-survey

     Summer institutes


     Repeat participants

    Impact of the Challenge experience on Pre- and post-survey teaching End-of-activity feedback

     Completion of graduate program Interviews: Teachers

     Methods in use/classroom practice

     Evidence of impact on students

    Non-completers Follow-up phone survey

Outcomes (goals and objectives) for Volunteers

    The Supercomputing Challenge 5

    Evaluation Plan 2009-2010

    Volunteers who participate in the Supercomputing Challenge will: (1) become actively involved in educational initiatives; (2) collaborate with scientists and educators across the

    state; (3) mentor students and teachers; (4) provide guidance, evaluation and feedback to

    local Challenge teams; (5) lend their expertise to the educational experiences of students; and

    (6) help identify and disseminate best practices in STEM education. [Note: These goals are not explicit but implied in the Supercomputing Challenge materials

    and documentation.]

Evaluation Data Source

    Demographics Volunteer data base

     Professional position Volunteer survey




    Levels of participation Volunteer data base

     Volunteer role Volunteer survey

     Repeat participants

     Challenge alumnae

    Perception of the Challenge’s impact on Volunteer survey student participants End-of-activity feedback

     Interviews: Volunteers

    Impact of the Challenge experience on End-of-activity feedback volunteer participants Interviews: Volunteers

Evaluation of Program Implementation

    Program implementation goals include: (1) the maintenance of effective governance,

    research, planning and management structures; (2) the annual recruitment of teacher and student participants who reflect the diversity of the state; (3) the development of human resources (scientists, mentors, and volunteers); (4) the procurement of financial resources; and (5) sound fiscal management of program funding.

Evaluation Data Source

    Oversight Board of Directors data base

     Governance Board minutes

     Research Consult minutes

     Planning Weekly correspondence with teachers/sponsors

     Management Annual reports to funding sources, partners, etc.

     (as required)

    Annual Program Evaluation Report The Supercomputing Challenge 6

    Evaluation Plan 2009-2010

    Participant recruitment Participant data base

     Teachers Pre- and post-survey demographics


    Human resources Volunteer data base

     Volunteer survey

    Partners data base

    Financial resources and management Annual Budget Report

     Funding sources



The Supercomputing Challenge 7

    Evaluation Plan 2009-2010

    Definition of Data Sources

    Student Pre- and Post-Survey

    Student surveys will include demographic information, questions regarding previous

    and current participation in STEM activities, and items that assess student growth and

    learning. Pre-surveys will be required of all students during the registration process,

    and post-surveys will be completed at the conclusion of the Challenge (possibly as

    part of the Judging and Awards events).

    Teacher Pre- and Post-Survey

    Teacher surveys will include demographic information, program participation data,

    and items that assess the program’s impact on teaching and personal professional

    development. Teachers will complete the pre-survey at the Summer Teacher Institute

    or with their team’s registration. Post-surveys will be completed at the conclusion of

    the Challenge (possibly as part of the Judging and Awards events).

    Volunteer Survey

    Volunteers will complete a short survey about their involvement and experience in

    the Supercomputing Challenge at the conclusion of their individual activities.

    End-of Activity Feedback

    Feedback from all participants will be collected at the conclusion of each program

    event throughout the year: Summer Teacher Institute, Round-ups, Challenge Kickoff,

    Site Visits, Interim Reports and Evaluation, and Final Submission and Awards.

    Feedback forms should be very brief and provide basic assessment of the quality and

    value of each event.


    Individual interviews of samples of students, teachers, and volunteers will add rich

    description and anecdotal information as well as validation to other data sources.

    Interviews may be conducted throughout the year-long series of events.

    Student Product

     Proposals, interim reports, final reports, judges’ feedback, evaluator observation.

    Awards/Scholarship Data and Completion Results

    Complete information from the final project submission, judging and awards will be

    maintained in a detailed data base. Information will include participant names,

    schools/districts, project titles, awards, scholarships, and prizes for each.

    Follow-up Surveys Student Participants

    Selected student participants from the previous five years will be surveyed to follow-

    up on their educational progress and involvement in STEM fields. This survey

    should be conducted annually with a new participant sample each year.

    Follow-up Phone Survey Non-completers

    Teachers/sponsors for those teams that do not complete the Challenge will be

    interviewed by phone to determine reasons for non-completion and possible barriers

    inherent in the program or at local schools.

     Note: Protocols for each of these data sources should be created by the Program Evaluator(s). Participant surveys will be expanded but should include some items consistent with previous Supercomputing Challenge (Survey Monkey) surveys in order to extract

    longitudinal data.

    The Supercomputing Challenge 8

    Evaluation Plan 2009-2010

    Evaluation Timeline

On-going data collection

     Maintenance of all data bases

    Documentation of Board meetings, Consult meetings,

    team correspondence, and program events

     Monitoring of revenues and expenses

July: Round-ups

     End-of- activity feedback (all participants)

     Interviews with students, teachers, and volunteers

July: Summer Teacher Institute

     Pre-survey of teachers

     End-of-activity feedback

July-October: Registration

     Pre-survey of teachers (not in attendance at STI)

     Pre-survey of students

October: Challenge Kickoff

     End-of-activity feedback (all participants)

     Interviews with students, teachers, and volunteers

    Student product proposals & feedback

October-January: Site Visits

     End-of-activity feedback (all participants)

     Interviews with students, teachers, and volunteers

October-February: Interim Reports and Project Evaluation

     End-of-activity feedback (all participants)

     Interviews with students, teachers, and volunteers

     Student product interim reports & feedback from judges and mentors

April: Final Submission, Judging, and Awards

     Student product Final report & feedback from the judges

     End-of-activity feedback (all participants)

     Interviews with students, teachers, and volunteers

     Post-survey of students and teachers

     Volunteer survey

May: Follow-up

     Compilation of program data bases

     Phone survey of non-completers

     Follow-up survey of former student participants

June: Submission and review of Final Evaluation Report

    The Supercomputing Challenge 9

    Evaluation Plan 2009-2010


     The proposed evaluation design is comprehensive and multifaceted which is

    necessary for a program as complex as the Supercomputing Challenge. Accountability

    for student learning and evidence of program effectiveness cannot be limited to simplistic

    measures, particularly when the potential for program effects are so far reaching.

    Continuous and substantive evaluation will provide hard data that justifies the levels of

    involvement--whether in terms of volunteer hours, student and teacher participation, or

    financial contributionsthat have been invested in the Supercomputing Challenge.

    Evidence that a program is working well should lead to fiscal resources and continued

    commitment that will sustain the program far into the future.

The Supercomputing Challenge 10

    Evaluation Plan 2009-2010

Report this document

For any questions or suggestions please email