Module 5 - Evaluation
Overview | Quality | Eval Questions | Instruments | R & E | Proposal Steps | Resources
Quality
Characteristics of high-quality evaluations for ITEST Proposals
What makes a good evaluation
There is no single recipe for designing a “good” evaluation. The American Evaluation Association (AEA) provides many resources to its more than 7,000 members, and the frameworks used vary widely. In this section we provide a few different ways to think about the evaluation for your proposed project to help you to consider key evaluation issues.
What ITEST is looking for in an evaluation
The ITEST solicitation specifies that the following components must be included in the evaluation section of the proposal:
-
Identification of the mechanisms for providing independent oversight and review of these activities (e.g., an independent, third-party evaluator or an external advisory board).
-
Articulation of evaluation questions relevant to the project's scope of work.
-
Delineation of the activities and data that will be used to gather evidence that informs the evaluation questions and identification of the project staff who will be responsible for gathering this evidence.
-
Description of how the project plans to use the evaluation evidence, including how feedback will be shared, with whom (e.g., project leadership, external advisors), and for what purpose (e.g., to inform ongoing project management, to supplement research findings and contribute to the generation of knowledge).
-
Inclusion of project evaluation activities in the project timeline.
The evaluation section should describe how the project plans to use the evaluation evidence, including how feedback will be shared, with whom (e.g., project leadership, external advisors), and for what purpose (e.g., to inform ongoing project management, to supplement research findings and contribute to the generation of knowledge).
Take home message: The project evaluation should evaluate all aspects of the work throughout its life cycle.
A proposal must describe appropriate mechanisms to assess success through project-specific external review and feedback processes. These might include an external review panel or advisory board proposed by the project or a third-party evaluator. The external critical review should be sufficiently independent and rigorous to influence the project's activities and improve the quality of its findings.
Take home message: Give some thought to how to structure the evaluation and who should be involved, whether as part of an external review panel or a third-party evaluator. You need to consider both independence and rigor, and design the evaluation so that it can help improve the project’s activities and quality of its findings.
In the video below, former ITEST Program Officer David Haury from NSF address whether there is a preference between projects selecting an evaluator or an advisory board.
The feedback processes should enable the project team to answer two overarching questions relating to the goals and objectives of both the research and development aspects of the project:
- Is the project making sufficient progress toward meeting the goals and objectives?
- What are the impacts of the project with respect to its intended outcomes?
Take home message: Align your evaluation questions with the two overarching questions above.
Successful proposals will:
- describe the expertise of the evaluator(s);
- explain how that expertise relates to the goals and objectives of the proposal; and
- specify how the PI will report and use results of the project's external, critical review process
Take home message: Include these elements in the evaluation section of your proposal.
Look at the elements required by the ITEST solicitation to be included in successful proposals (above). Write 1-2 sentences indicating how your project will meet each requirement. If you do not yet have the needed information, write the information you need and how you will get it. This text should provide justification for using either an external evaluator or an external review board.