Module 4 - Research
Overview | Big Picture | RQS | Design | R&E | Proposal Steps | Resources
Big Picture
Setting up your research plan
The research plan is a critical component of your project description because the answers to your research questions will contribute to the project’s Intellectual Merit, one of the key factors that reviewers look for in strong proposals. The research questions should answer a bigger “So what?” question and provide evidence that contributes to the field, builds on and improves the work you are doing, and leads to the broader impact of your project.
Your ITEST project will be developing and implementing a fabulous idea that has the potential to have a positive impact on young people’s connections to STEM careers. Answers to your research questions will help to identify what works, what needs to be modified, and what the potential impact is on the people you are hoping to affect, whether youth, teachers, or communities.
In order to do this well, your research design must be coherent: it must align with your logic model and your evaluation design. This section provides some ideas and resources to help you design research that will provide the most useful results for you and for the work.
Determine who is going to do the research.f you, as PI, are a researcher, you can lead the research. If you prefer to focus on implementation, you can ask the evaluator to take on a research role or include a researcher on your team. Be sure that you have someone with the methodological expertise needed to assure a high-quality research design.
Figuring out the research design goes hand in hand with identifying the research questions you want to answer. Different types of research questions are suited to different types of designs, and the type of question also depends on the stage of research in your proposal. Do you want to describe what happens during the intervention? Understand how participants are experiencing the intervention? Examine if participants change attitudes? Improve skills? Each of these types of questions lends itself to different types of designs. Identifying the appropriate research questions and research design is an iterative process that happens together with the development of your proposal idea. The previous module (Module 3) provided information about developing your key idea; this module focuses on the research you will engage in to develop that idea. For ITEST projects in particular, you have to answer four questions.
ITEST Pillars
How will your project incorporate the ITEST Pillars? All ITEST proposals must address the three pillars: (1) Innovative Use of Technologies in Learning and Teaching, (2) Partnerships for Career and Workforce Preparation, and (3) Strategies for Equity in STEM Education.
Research Questions and Outcomes
How will your research address the goal of increasing knowledge of, and interest in, STEM and ICT careers? Your project’s research will be grounded in well-crafted and measurable research questions that measure both cognitive and social or affective outcomes.
Research Stage
What is the stage of your research? NSF provides a helpful frame for thinking about how research answers questions at different stages of the development of an intervention. Different stages of research address different types of questions and use different research designs.
Project Type
Which type of ITEST project is most appropriate? The solicitation includes three project types (ETD, DTI, and SEI; see below for further information).
Designing Innovations that Meet ITEST Pillars
The Program Description section of the ITEST solicitation outlines the three key elements (pillars) of the ITEST program: 1) Innovative Use of Technologies in Learning and Teaching, 2) Partnerships for Career and Workforce Preparation, and 3) Strategies for Equity in STEM Education. Excerpts from the solicitation about each of these elements are included below. Refer to the solicitation for further details about the requirements for each of these components.
Research Stage
NSF and the U.S. Department of Education published the Common Guidelines for Education Research and Development in 2013 to help education researchers think about different stages of research and the different designs that can best contribute to answering questions at each stage. For example, a project that is at the exploratory stages should not use a quasi-experimental research design.
Foundational research
Provides the fundamental knowledge that may contribute to improved learning and other relevant education outcomes. Studies of this type seek to test, develop, or refine theories of teaching or learning and may develop innovations in methodologies and/or technologies that will influence and inform research and development in different contexts.
Early stage or exploratory research
Examines relationships among important constructs in education and learning to establish logical connections that may form the basis for future interventions or strategies to improve education outcomes. These connections are usually correlational rather than causal.
Design and development research
Develops solutions to achieve a goal related to education or learning, such as improving student engagement or mastery of a set of skills. Research projects of this type draw on existing theory and evidence to design and iteratively develop interventions or strategies, including testing individual components to provide feedback in the development process. These projects may include pilot tests of fully developed interventions to determine whether they achieve their intended outcomes under various conditions. Results from these studies could lead to additional work to better understand the foundational theory behind the results or could indicate that the intervention or strategy is sufficiently promising to warrant more advanced testing.
Efficacy, Effectiveness and Scale-up research
Contributes to evidence of impact, generating reliable estimates of the ability of a fully developed intervention or strategy to achieve its intended outcomes. The three types of Impact Research share many similarities of approach, including designs that eliminate or reduce bias arising from self-selection into treatment and control conditions, clearly specified outcome measures, adequate statistical power to detect effects, and data on implementation of the intervention or strategy and the counterfactual condition.
ITEST Project Types
You’ve already heard quite a bit about the three different types of ITEST projects, which NSF refers to as “funding strands.” In Module 1, the three types were introduced as components of the ITEST Solicitation. The title of your project (as it appears on the cover letter of your application) must start with the project type. While reference to your project type appears in several other sections of your proposal, the type of your project is the foundational element of the Research and Development Design section of your Project Description.
We highly recommend you download and carefully read the sections of the Common Guidelines for Educational Research and Development (“Common Guidelines”) that pertain to your project type.
Below, we outline important information about each project type, and how each relates to the Common Guidelines.
EXPLORING THEORY AND DESIGN PRINCIPLES (ETD)
Key excerpts from the ITEST Program Solicitation:
- ETD projects describe and explore extant conditions and factors in the field intended to increase students’ (and educators’) STEM knowledge and motivation, participation, persistence, confidence, and resilience in STEM and ICT fields.
- This project type is appropriate for projects in the conceptualization and planning phase with the goal of developing proof of concept.
- ETD projects can be up to three years in duration, with a maximum award size of $500,000. Key excerpts from the Common Guidelines for Education Research and Development:
Key excerpts from the Common Guidelines for Early-Stage or Exploratory: The ETD research plan should include the following:
- a set of hypotheses or research questions that are derived from the theoretical and empirical rationale for the study;
-
a detailed research design that is appropriate for the hypotheses or research questions;
-
a justification for the proposed research context and sample
-
a description of the data sources if secondary analyses are proposed;
-
a detailed description of data collection procedures and instruments, as well as evidence of and strategies for ensuring reliability and validity;
-
if applicable, a plan to study the opportunities for interventions (i.e., programs, policies, practices or technologies) to address education and learning challenges; and
-
a detailed description of data analysis procedures and the reporting plan.
DEVELOPING AND TESTING INNOVATIONS (DTI)
Key excerpts from the ITEST Program Solicitation:
- DTI proposals involve designing a theoretically driven innovation, pilot-testing or implementing the innovation, and analyzing its outcomes.
- DTI studies focus on direct engagement with students and educators and assessment of student outcomes relevant to ITEST’s primary goals of increasing student knowledge of and interest in STEM and ICT careers as well as the development of STEM/ICT knowledge and skills required for pursuit of those careers.
- DTI projects can be up to four years in duration, with a maximum award size of $1,300,000.
- For DTI projects, use a research design appropriate for Design and Development Research as described in the Common Guidelines.
Key excerpts from the Common Guidelines for Education Research and Development:
- The DTI research plan should describe the following:
- method for developing the intervention to the point where it can be used by the intended end-users (iterative development process);
- (2) method for collecting evidence on the feasibility that end-users can implement the intervention in an authentic education or learning setting (evidence of feasibility of implementation); and
- (3) method for obtaining pilot data on the promise of the intervention for achieving the expected outcomes (pilot study).
SCALING, EXPANDING, AND ITERATING INNOVATIONS (SEI)
-
Key excerpts from the ITEST Program Solicitation:
- SEI studies are designed to build on and expand an existing innovation that has evidence of success (including DTI projects or similar innovations previously developed within or outside of the ITEST portfolio.)
- SEI projects can range up to five years in duration, with a maximum award size of $3,500,000. Key excerpts from the Common Guidelines for Education Research and Development
Key excerpts from the Common Guidelines:
The research plan should identify and justify;
- the study design used to estimate causal impact of the intervention on the outcomes of interest;
- the key outcomes of interest for the impact study and the minimum size impact of the intervention that would have policy or practical relevance;
- the study setting(s) and target population(s);
- the sample, including the power it provides for detecting an impact;
- the data collection plan, including information about procedures and measures, including evidence on and strategies for ensuring reliability and validity, and plans for collecting data on program implementation, comparison group practices, and study context; and
- the analysis and reporting plan.
Important Note: The solicitation states that the first sentence of your Project Summary must state the project type of your proposal. See the Proposal Preparation Instructions section of the solicitation for further information.
Activity 4.01
Confirm your ITEST proposal type and the research “Stage of Development”