WEBVTT 1 00:00:01.022 --> 00:00:02.772 Sarah MacGillivray: Hello, everyone. 2 00:00:04.852 --> 00:00:11.492 Sarah MacGillivray: Welcome to today's webinar. I'm just gonna speak slowly as people roll into the main room. 3 00:00:18.392 --> 00:00:27.911 Sarah MacGillivray: All right. So thank you for joining us today. Today's webinar is the 2,023 Nsf, I test solicitation overview. 4 00:00:28.172 --> 00:00:39.582 Sarah MacGillivray: we are. I am a member of the stellar center, and we are joined today by a number of Nsf program officers who, I believe you see there. 5 00:00:39.642 --> 00:00:53.222 Sarah MacGillivray: Oh, you might just be looking at. But anyway, we we're joined today by a great group of program officers, and we are going to get started in just a moment. I do want to point out that 6 00:00:53.432 --> 00:00:56.161 Sarah MacGillivray: for technical 7 00:00:56.322 --> 00:01:15.401 Sarah MacGillivray: to to orient us. Technically, we are using the Q. A. For any questions at all. But I believe that the chat is open for people to introduce themselves. Say Hello! We just ask that. You make sure that you If you have a question that you want to have answered that you put it in the separate Q&A pod. 8 00:01:15.432 --> 00:01:30.251 Sarah MacGillivray: So with that, I'm going to just do a quick, interjectory thing before we get rolling with the main event. so hello again, everyone, and thank you for joining us for this webinar today, which is the 2,023 9 00:01:30.502 --> 00:01:33.372 Sarah MacGillivray: and Sf, I test solicitation overview. 10 00:01:33.502 --> 00:01:39.311 Sarah MacGillivray: We're joined today by our our esteemed colleagues, Nsf. Program officers, Alicia Santiago 11 00:01:39.382 --> 00:01:45.141 Sarah MacGillivray: Leila Lions and Bob Russell, and we have additional program officers who will be responding online. 12 00:01:45.312 --> 00:02:01.631 Sarah MacGillivray: So this webinar is being hosted by Stellar, the resource center for the I test program on behalf of the National Science Foundation. So before we kick off with the presentations, and to give a little more time for people to join I'd like to just give a brief introduction to the stellar center and what it is that we do. 13 00:02:03.912 --> 00:02:18.131 Sarah MacGillivray: Okay, so Stellar stands for the stem learning and research center and is the resource center for the Nsf I test program, and our mission is to build the capacity in and magnify the results of, I test projects in order to deepen the impact of and 14 00:02:18.242 --> 00:02:21.502 Sarah MacGillivray: broaden participation in the I test program. 15 00:02:23.152 --> 00:02:33.022 Sarah MacGillivray: The main areas of our work are in facilitating project success through technical support by informing and influencing the field by disseminating it, test project findings 16 00:02:33.372 --> 00:02:39.961 Sarah MacGillivray: and in deepening the impact and reach of the I test program by broadening participation in the I test portfolio. 17 00:02:39.992 --> 00:02:50.162 Sarah MacGillivray: With that I'll mention that stellar supports both. I test grantees and those who are interested in submitting a proposal. Oh, I guess I'm having technical difficulties. Hold on. 18 00:02:51.192 --> 00:02:52.312 Sarah MacGillivray: There we go. 19 00:02:53.162 --> 00:03:19.551 Sarah MacGillivray: And you see my slides now. Yes, okay, well, thankfully. This is just the beginning and not crucial. So I will just keep rolling. So we invite you to check out the resources on the stellar website, which include a rich library of curriculum materials, research and video and evaluation instruments. I test publications and videos. we have current and past, I test project profiles. 20 00:03:19.712 --> 00:03:43.902 Sarah MacGillivray: which provide project details staff and organizational information as well as related resources submitted by that project. our, our test proposal development resource page is designed to help those interested in developing a submission, and it includes a new page specifically designed to support those interested in sei proposals. So we have a lot of rich rich resources on the stellar website. 21 00:03:44.432 --> 00:03:59.361 Sarah MacGillivray: So this is my final slide. I'll close just by mentioning a couple of upcoming activities. we have, I test solicitation. Q, a. Sessions scheduled bi weekly on Fridays. There is one tomorrow at one pm, so if you have a 22 00:03:59.542 --> 00:04:06.291 Sarah MacGillivray: question that you would like to get a quick answer on that doesn't get addressed today. You can always join us tomorrow at one Pm. 23 00:04:06.531 --> 00:04:13.301 Sarah MacGillivray: And then we will have these by weekly. I believe the schedule is June ninth, 20, third, seventh, and 24 00:04:13.492 --> 00:04:15.981 Sarah MacGillivray: oh, July seventh, and July 20. First. 25 00:04:16.042 --> 00:04:40.212 Sarah MacGillivray: The other thing I will mention is that for those that are interested in submitting the Sei type of proposal. which is the scale up proposal. The concept worksheets that are that we can link to on our website are due to the Nsf email by Friday, June ninth. So if anyone attended those earlier webinars or had heard about the 26 00:04:40.212 --> 00:04:48.221 Sarah MacGillivray: worksheet previously, Just make sure that you have that date on your calendar. The other thing I'm excited to announce is that we are going to be hosting 27 00:04:48.222 --> 00:05:13.351 Sarah MacGillivray: some regional workshops in support of Nsf. F. Score jurisdictions. And so we have the date there for the first one that we're having in August of Maine, and we invite anyone from Maine, New Hampshire, Vermont, or Rhode Island who would be interested in attending to just go ahead and to email us. And we'll share that information with you. We would also be interested in talking to anybody else from another app score jurisdiction about arranging your own event. 28 00:05:13.402 --> 00:05:21.561 Sarah MacGillivray: so with that, I believe. I'm going to turn things over now to Alicia Santiago. 29 00:05:24.062 --> 00:05:28.602 Alicia Santiago: Thanks so much. Sarah and Lola. Hello! 30 00:05:28.762 --> 00:05:35.961 Alicia Santiago: Everyone in Benitos. so welcome to the ingredients of a competitive. I test proposal. Webinar 31 00:05:35.992 --> 00:05:59.251 Alicia Santiago: my name is Alicia Santiago, and I'm here with my colleagues, Leila Lions and Bob Russell. we'll quickly talk to you today about, of course, the access program just a brief overview and some very important components that you need to include in your proposals. And just as a quick reminder, the deadline for submitted proposals this year is August eleventh at 5 Pm. 32 00:05:59.252 --> 00:06:06.611 Alicia Santiago: Local time. Of this submitting institution. And what you see there on the screen is the agenda for today. 33 00:06:06.612 --> 00:06:27.181 Alicia Santiago: so we'll I'll go through a brief overview of the at this program. we'll also talk about types of projects. Funding levels require pillars, research and evaluation, a specific review of criteria and briefly, talk about at score funding. And I test emerging working workforce priorities. 34 00:06:28.542 --> 00:06:50.391 Alicia Santiago: so I test, which stands for innovative technology experiences for students and and teachers is an applied research and development program the goal of I test is to a thought to advance the equitable inclusive integration of technology in in the learning and teaching of stem from free prepay through high school. 35 00:06:50.572 --> 00:07:07.422 Alicia Santiago: And the program aims to help all students develop a strong foundation in stem subjects. ensuring that they are all well prepared for both present and future employment opportunities that rely heavily on on technology and and computing skills. 36 00:07:09.312 --> 00:07:27.852 Alicia Santiago: so I just is assigned to address societal needs and emerging areas of stem and related careers. these emerging areas may include quantum computing, artificial intelligence, computational thinking, cyber security, environmental science 37 00:07:27.852 --> 00:07:44.392 Alicia Santiago: and stem entrepreneurship. Among others. I test encourages proposals that incorporate, well planned approaches to effectively integrate these emerging fields into innovative teaching and learning methods. 38 00:07:45.842 --> 00:07:56.961 Alicia Santiago: So now let's talk about a few really important things, which is some common misconceptions about I test The first one is that? 39 00:07:56.972 --> 00:08:02.952 Alicia Santiago: I just only funds proposal situated in formal education? The short answer is, no. 40 00:08:02.952 --> 00:08:31.231 Alicia Santiago: I test funds, projects in both formal and or informal settings. I to support proposals, aim to enhance stem learning experiences in in traditional school environment and and also in non traditional settings such as community organizations, museums or libraries, for example. And I just really recognizes that learning occurs in in various contexts and and aims to provide a diverse learning opportunities 41 00:08:31.302 --> 00:08:51.292 Alicia Santiago: for students beyond the traditional classroom. So keep that in mind. the second misconception is that I just proposals are primarily submitted by people from K through 12 schools. Now, while Kate and 12 schools, personal, can submit proposals, most principal investigators 42 00:08:51.342 --> 00:09:19.651 Alicia Santiago: for it, just projects are actually university and faculty stem educational leaders and researchers from various organizations. So Pis are encouraged to collaborate with K through 12 schools in formal learning organizations or related end to things to to develop their project ideas. And and these collaborations are really important because they foster interdisciplinary approaches and promotes brother in button on stem education. 43 00:09:20.082 --> 00:09:32.621 Alicia Santiago: The third misconception, is university faculty members and students cannot be paid because it is a program targeting just a Pre K through top students. 44 00:09:32.712 --> 00:09:54.102 Alicia Santiago: no. So, so contrary to this misconception, I just allows for the payment of university faculty members, undergraduate students, grad students, and assistants that are involving the projects, and they can receive stipend similar to other research programs. And this acknowledges really the time and effort in invested in the project. 45 00:09:55.222 --> 00:10:01.852 Alicia Santiago: The next misconception is colleges or universities cannot charge indirect costs. 46 00:10:01.862 --> 00:10:16.681 Alicia Santiago: Because I test, is a program targeting 3 K through 12 students. no. Actually, colleges and universities participating in it as can charge in direct costs which are common in research programs. 47 00:10:16.912 --> 00:10:35.412 Alicia Santiago: The indirect cost rates remain consistent with other research funding and are not restricted due to the to the problems focus on free K through 12 students. these enables institutions to provide you the necessary support and and resources to implement high quality. I test projects effectively. 48 00:10:36.602 --> 00:10:39.572 Alicia Santiago: Now, on the 49 00:10:39.832 --> 00:11:07.611 Alicia Santiago: yeah, that's fine. Thank you. So the next misconception here is my university will not reward me for my friends on supporting free K. 12 education. So we'd really recommend that you check with your being your chair and provost about this. we know that many universities have actually acknowledged and and appreciated faculty contributions to K to job stem education within the tenure and and promotion process. So please check on that. 50 00:11:08.232 --> 00:11:19.921 Alicia Santiago: The other misconception is that I just just focuses on outreach. Well, I just goes beyond. you know me or outreach update it is, it is a research on development program. So that's 51 00:11:19.952 --> 00:11:28.752 Alicia Santiago: that's really really important that aims to support innovative technology experiences for Pre K through 12 students and teachers. 52 00:11:28.752 --> 00:11:54.511 Alicia Santiago: So competitive proposals should incorporate research components that contribute to national efforts in in promoting increased participation stem education and careers. And and by incorporating these research components. I, this projects contribute to advancing a knowledge and understanding of effective stem educational practices. So this is very important. This research based approach, you know, allows for the development of evidence based strategies that can 53 00:11:54.692 --> 00:12:08.592 Alicia Santiago: have brother implications for improving stem education and nationwide the last misconceptions I want to talk about is that I test is about fun after school experiences. Well. 54 00:12:08.602 --> 00:12:35.792 Alicia Santiago: I this is not limited just to fun. After school experiences. it actually supports a wide range of activities and experiences that align with the goals of of increasing participation in in stem education and potential careers. These experiences can take place in schools, museum, science centers, committee, many locations, universities. industry sites. Actually, the world is your oyster when when it comes to locations for for learning experiences. 55 00:12:35.792 --> 00:13:05.442 Alicia Santiago: and these experiences can occur in regular school hours or they could be after school summer weekends as long as they are appropriate for achieving the projects objective. So so the flexibility in terms of of location and timing accommodates different learning environments and schedules, and and it allows projects to adapt to the needs and and availability of participants providing more equitable access to to stem learning experiences and and maximizing their impact. 56 00:13:06.022 --> 00:13:14.431 Alicia Santiago: so on the next slide. Oh, okay, so I'm gonna 57 00:13:14.612 --> 00:13:21.472 Alicia Santiago: now. I'm going to have my colleague, Leila. Tell you a little bit more about project types. Thank you, Leila. 58 00:13:21.672 --> 00:13:24.122 Leilah B. Lyons: Thank you, Lisa. So 59 00:13:24.142 --> 00:13:38.582 Leilah B. Lyons: One thing to note is that we kept the solicitation the same as last year. So if you applied last year. These categories should not be a surprise to you. But I'll walk you through the different types of projects that we're welcoming this year. 60 00:13:38.582 --> 00:13:57.212 Leilah B. Lyons: the first 2 types conference or workshop projects and synthesis studies are the only ones that have a a slightly different submission process. We are requesting that you contact a program officer prior to submission. just to, you know, to discuss the ideas with us before you go to the trouble of of constructing a proposal. 61 00:13:57.212 --> 00:14:04.091 But the way those are structured are conference or workshop projects last up to a year and up to $100,000, 62 00:14:04.122 --> 00:14:29.111 Leilah B. Lyons: and the idea behind these is that you know, we want you to be exploring topics that will help further the work that we're trying to do within. I test. So we would want you to. in in the context of your proposal, establish the timeliness and the value to the community. The I just cleaned you the identified issue. You know why this is something that is worthy of our support, how it might be able to be put 63 00:14:29.112 --> 00:14:47.301 Leilah B. Lyons: parlayed into a a larger area of work or research. we would want you to talk about who you're going to invite to these conferences or workshops what their expertise is, or if you haven't fully selected your slate of participants, what kind of selection, criteria or room bricks for that you're going to use 64 00:14:47.302 --> 00:14:59.881 Leilah B. Lyons: to choose who gets to attend. we want you to include a conceptual framework So you know, you have some sort of a theoretical basis that is outlining this area of of study or interest. 65 00:14:59.882 --> 00:15:20.821 Leilah B. Lyons: we'd like you to include a draft agenda. So we have a sense of what you're planning on doing during those couple of days that you'll have people working together. And we'd also like you to describe the expected outcomes and the dissemination plan for those outcomes, because ultimately we do want these projects to inform the field and help shape the direction of the kinds of research that Other folks in the field are investigating as well. 66 00:15:21.792 --> 00:15:43.212 Leilah B. Lyons: synthesis studies last up to 2 years and up to $400,000. And the idea behind these is that? you know, we're trying to provide an opportunity for folks to really, you know, build a a critical mass of interest, attention, and evidence around a particular question, issue, or topic of critical importance to one of the I test pillars which we'll discuss shortly. 67 00:15:43.252 --> 00:15:54.712 Leilah B. Lyons: in these proposals we need you to present the current state of knowledge. So what do we know already about this topic or area? And what don't we know? Why is this worthy of a synthetic approach. 68 00:15:54.872 --> 00:16:05.582 Leilah B. Lyons: we want you to explain or justify the methods that you're going to use. this. Most likely it's going to include literature reviews, but it might include other things, too, like qualitative metasyntheses or meta analyses 69 00:16:05.592 --> 00:16:18.011 Leilah B. Lyons: And then we want, of course, the output of these kinds of awards to generate products that are going to be useful by researchers and practitioners. So don't forget that there are multiple audiences for these synthesis studies. 70 00:16:18.392 --> 00:16:31.822 Leilah B. Lyons: The next category exploring theory and design principles. It's probably what we see. we see a lot of those we see a lot of Dtis, So the exporting theory. Design principles are up to 3 years and up to $500,000, 71 00:16:31.822 --> 00:16:50.442 Leilah B. Lyons: and for these, you know, they intended to be for earlier, more exploratory work. We want you to investigate something that you know, is the beginning of a solid project idea. We want you to explore the problem domain itself. You know what kinds of factors need to be investigated to increase your understanding of the problem domain and how to approach it. 72 00:16:50.442 --> 00:17:06.362 Leilah B. Lyons: these proposals should be building on and advancing theory. you know about how people are learning or engaging with with the topic at hand. and they the idea for the output of these proposals is that they should produce design principles or frameworks for further innovations. 73 00:17:06.362 --> 00:17:21.422 Leilah B. Lyons: So the way we've structured these different projects is, they sort of add on to each other in terms of the maturity of the ideas that you're investigating as a researcher. So the next category up would be the Dti, the designing and testing innovation. They can last up to 4 years, and up to a million $300,000. 74 00:17:21.442 --> 00:17:38.442 Leilah B. Lyons: and for these, you know, we expect you to be a little farther along the way of understanding what the problem domain and the solution that you're presenting for the intervention you're presenting. We will want these projects to support the design and testing of a innovation that you have come up with. And we want you to analyze the outcomes of those tests. 75 00:17:38.452 --> 00:17:59.162 Leilah B. Lyons: and the idea is that these proposals should be attending to the design principles, and how they influence how the learners are engaging in the knowledge that you're presenting the stem knowledge the career fields related to the stem knowledge in presenting and and other pathways through their growth as a professional 76 00:17:59.482 --> 00:18:17.742 Leilah B. Lyons: the final category. And and you heard our our colleagues at Stellar mentioned this before. Are these scaling, expanding and iterating innovations, proposals. We're putting a special emphasis on these this year trying to help people craft good proposals. And so just a quick reminder that. we did put together a 77 00:18:17.802 --> 00:18:38.371 Leilah B. Lyons: seminar that is posted on the stellar website that discusses what makes for a good sei proposal. And we also constructed the worksheets that were mentioned. we do. June ninth for review so you can get feedback on your potential sei proposal early enough to be able to act on that feedback. 78 00:18:38.372 --> 00:18:58.612 Leilah B. Lyons: These proposals are the longest and the most amount of money. They last up to 5 years and 3.5 million dollars, and they're attended to broaden an innovation at a significant scale so that can include extending an innovation to new populations, new regions, new ages or new contexts, and the hope is that the research should be transferable and generalizable. 79 00:18:58.672 --> 00:19:12.632 Leilah B. Lyons: we want you to be assessing the cognitive and possibly the social, emotional outcomes stem ICT knowledge as well as career any of those things, or are fair game, most likely with a large scaling project. You'd be targeting more than just one, though. 80 00:19:15.032 --> 00:19:19.521 Leilah B. Lyons: Right? So now we are going to talk about killers, I think. 81 00:19:19.562 --> 00:19:22.182 Leilah B. Lyons: No, I'm talking about pillars. That's right. 82 00:19:22.242 --> 00:19:43.112 Leilah B. Lyons: So the pillars are the you know, kind of foundational DNA behind. I test the kinds of things that that we really want you to be pursuing as researchers through this pro program. so the first one is integrating technology and learning. so I test projects. have to 83 00:19:43.112 --> 00:20:00.712 Leilah B. Lyons: engage learners and the use of technologies that are going to support their foundational preparation in stem as well as information communication technologies. And so when you're thinking about addressing this pillar and your proposal. We want you to make sure that you're giving consideration to the design features of the technology foremost. 84 00:20:00.712 --> 00:20:14.472 Leilah B. Lyons: So why is this technology, the one that you've chosen, you know, it can't just be because it's cool, although that can help sometimes with motivational things. but it really needs to serve a purpose in the learning activities that you're trying to support with it. 85 00:20:14.602 --> 00:20:24.231 Leilah B. Lyons: we also want you to give us some evidence. from the literature that supports. Why this technology choice is appropriate for the population setting in content that you're trying to study. 86 00:20:24.592 --> 00:20:50.292 Leilah B. Lyons: we want you to think about how technology can be used in how specific stem disciplinary concepts will be taught. You know, what? What role does the technology play in this instruction? Right? So you know, you might see this happen with, say, data, science proposals and the kinds of data science tools that are used to visualize and analyze data. And how does that support certain stem understandings? That could be something you would investigate. 87 00:20:50.512 --> 00:20:57.091 Leilah B. Lyons: we want to hear about how you're going to support the proposed use of this technology. So you know, I just 88 00:20:57.172 --> 00:21:15.191 Leilah B. Lyons: projects don't take place in a lab or a vacuum. They take place in real learning settings. So how are you going to make sure that it works? how are you going to integrate? It's the existing learning environment. How are you going to support the instructors or teachers or facilitators who may not be well versed in that use of technology? 89 00:21:15.282 --> 00:21:30.831 Leilah B. Lyons: and how are you going to engage their act their acumen about how learners engage with things? yeah. Design process? how are you going to support their learning about technology via professional development? These are all questions that that you might want to address 90 00:21:31.282 --> 00:21:43.361 Leilah B. Lyons: And then finally, you know what's going to be studied, you know, all kinds of different perspectives can be brought on technology which aspects of the technology where it's deployment. Our novel. That's what we're most interested in. 91 00:21:43.422 --> 00:21:57.621 Leilah B. Lyons: We don't want to see people reinventing. We also want people to be exploring new wheels. So you know, what outcomes are you going to be attending to with your research? and the outcomes for which populations and and why is that filling a whole in the literature? Those are important things to establish in your proposal 92 00:21:58.762 --> 00:22:20.251 Leilah B. Lyons: the second pillar is pertaining to partnerships for career and workforce preparation. So for this pillar, the core aspect of it is that we're asking investigators to work with community stakeholders to identify and define opportunities to support students, awareness and preparation for stem and ICT careers. 93 00:22:20.312 --> 00:22:27.202 Leilah B. Lyons: So those opportunities might be things like entrepreneurship, apprenticeships, externships, internships, or just mentoring. 94 00:22:27.272 --> 00:22:38.972 Leilah B. Lyons: and the kinds of stakeholders that you might engage in this work might include neighborhood and community groups, nonprofit or philanthropic organizations, businesses, libraries, museums, educational institutions, or other agencies. 95 00:22:39.762 --> 00:22:48.372 Leilah B. Lyons: And what we want you to be doing in your proposal. Is you know thinking about the ways in which these partnerships can strengthen 96 00:22:48.842 --> 00:23:10.371 Leilah B. Lyons: existing collaborations and develop new connections between partners. We want to make sure that the voices of these partners are heard. And you want to give some thought to what role would be appropriate for your partners to play in this proposal. You know what kind of project leadership access? Do you want to give them? Do you want them to be engaged in the research itself? The conceptualization of the partnership 97 00:23:10.372 --> 00:23:22.371 Leilah B. Lyons: to what extent are they involved in the decision making processes? to what extent might they be employed in making sense of the data and disseminating it outwards. Those are all things that you should be thinking about. 98 00:23:22.412 --> 00:23:49.902 Leilah B. Lyons: we want you to specify how the partnerships are directly engaging the target audiences. And we certainly want you to discuss the collaborative theory of action and how it's being implemented. How do you think this partnership is going to function. And why? How is it going to benefit the target audiences and stakeholders? an important component of this theory of action is how you're going to establish trust among partners, so that you can negotiate any differences that emerge between policies and culture and things like that. 99 00:23:50.622 --> 00:23:53.462 Leilah B. Lyons: Oops The third pillar 100 00:23:53.782 --> 00:24:20.602 Leilah B. Lyons: pertains to strategies for equity and stem education. So the I just program wants to advance. Nsf vision of stem inclusivity by leveraging the diversity of intellectual and cultural perspectives to meet the goal of fully inclusive and fully diverse stem workforce. So then, while it's a couple of different things, operationalizing is really a key one, and and this is one where we often see proposals be fairly weak. 101 00:24:20.602 --> 00:24:47.981 Leilah B. Lyons: we want you to tell us which specific audiences are being supported by this work, and what inequities do they uniquely face that? The proposed strategies are uniquely going to address? you know, generic descriptions of underrepresented populations, often not really sufficient, because there's so many different ways under representation can emerge from historical factors, and then that in turn affects what kinds of interventions might be more or less effective for them. 102 00:24:48.082 --> 00:25:00.612 Leilah B. Lyons: conceptualization is also very important. How are your strategies for equity and stem exploring or building, depending on what they're doing, Ed or Dti on theories, methodologies, or design principles, you know. 103 00:25:00.942 --> 00:25:13.642 Leilah B. Lyons: we. We don't expect that. you know, this kind of outreach happens in a vacuum. There's a literature on this, and how to engage different populations. We'd like you to build on that and build on the the prior literature on that. 104 00:25:14.042 --> 00:25:39.542 Leilah B. Lyons: and then we need you to connect what you're doing for strategies, for equity to the first 2 pillars. So with respect to technology. You know your technology choice. Again, why is that technology right? The right choice for engaging this particular target audience? Given their unique assets and histories. And then, with respect to pillar, 2 are the voices, knowledge, and experiences of communities who will be most impacted at the center of the strategic partnerships. 105 00:25:42.192 --> 00:25:58.791 Leilah B. Lyons: All right solicitation, specific review criteria. So this is something we require you to talk about in the proposal narrative. and we want you to address these 4 things, and we will be evaluating proposals according to these criteria. So it is important. You make it as clear for your reviewers as possible how you're addressing them. 106 00:25:58.932 --> 00:26:13.031 Leilah B. Lyons: So. you know, in your proposal you want to make sure that you state to what extent the proposal includes explicit and adequate strategies for recruiting and selecting participants describing approaches to address diversity, access, equity, and inclusion. 107 00:26:13.052 --> 00:26:25.302 Leilah B. Lyons: you want to describe research informed instructional approaches to build on strengths and challenges of the target audience. And you want to explain how innovations with technology are developmentally and age appropriate for these audiences. 108 00:26:27.212 --> 00:26:30.202 Leilah B. Lyons: And with that I'm going to turn it over to my colleague, Bob. 109 00:26:36.192 --> 00:26:41.542 Robert L Russell: Oh, okay, hello, everybody, Bob, here and 110 00:26:41.632 --> 00:26:42.652 Robert L Russell: So 111 00:26:43.062 --> 00:26:50.722 Robert L Russell: as a background to discussing research and evaluation, I should know that these questions relate 112 00:26:50.732 --> 00:27:10.001 Robert L Russell: relate to the merit review criteria And and these 2 specific questions relate to the merit review criteria. And just to remind you as to what those are. There are 2 broadmer view criteria that in a Sf, as you'd address in your proposal, and and especially those 2 areas, are explicitly 113 00:27:10.062 --> 00:27:21.352 Robert L Russell: noted in your project summary, and and one of those is intellectual merit, and that is more or less the research value of your proposal. What are we going to learn about learning environments? 114 00:27:21.442 --> 00:27:30.921 Robert L Russell: How participants learn within that environments, how technologies work to contribute to learning? You know, in other words, the the not not just the what, but the why 115 00:27:30.952 --> 00:27:32.731 Robert L Russell: and and then 116 00:27:33.762 --> 00:27:49.341 Robert L Russell: okay. And then the the other is, does the plan incorporate a mechanism to access to assess success, and that that is primarily the evaluation. she's trying to find out. Did people do what they said they were going to do? And was the 117 00:27:49.612 --> 00:27:59.051 Robert L Russell: was the program or the intervention effective? Did it work? And that's part of what evaluation is designed to do. 118 00:27:59.262 --> 00:28:09.732 Robert L Russell: Okay, so what are the differences in it. And you know there are, there is not an entirely clean line between what research is and what evaluation is. 119 00:28:10.112 --> 00:28:23.862 Robert L Russell: But with research, you want to know, how do the moving parts of a of a project work? What is the context? what are the specifics of the experience or the intervention that are supposed to 120 00:28:24.522 --> 00:28:30.452 Robert L Russell: result in some impacts. and and what are those impacts? 121 00:28:30.572 --> 00:28:40.631 Robert L Russell: And so that's that's really, you know the how you develop your research questions. And so for this, in your proposal, you don't want to present a a strong 122 00:28:41.122 --> 00:29:03.731 Robert L Russell: but succinct research background out of which you would. You know, it's like, here we here's what we know now, based on. You know the literature you site and the discussion you provide, and that would provide the foundation for asking the research questions. So then, the next part of that is developing valid and reliable measures of of learning processes and outcomes and other elements that are important for the approach that you're taking. 123 00:29:03.902 --> 00:29:09.352 Robert L Russell: So that's why you need well-founded research questions grounded in 124 00:29:09.602 --> 00:29:16.382 Robert L Russell: as much theory and and and review of existing research, you know, as would be relevant for your proposal 125 00:29:17.142 --> 00:29:42.021 Robert L Russell: and and itself is is about innovation and contributing new knowledge to the field. So if if the area that you're thinking about for your project is something that's very well researched in the stabulus, you may want to think about a different angle for your project where you you could not just re it, or you know, reconfirm what we already know, but explore some new areas that would really contribute to the field. 126 00:29:42.362 --> 00:29:45.042 Robert L Russell: you know, and then that's to get to the final point. 127 00:29:45.152 --> 00:29:56.011 Robert L Russell: So more or less already said, the point of the research and your proposal is to advance new knowledge and to add to the evidence base on these important questions about 128 00:29:56.202 --> 00:30:04.631 Robert L Russell: how do students learn in the and learning environment that you're presenting, and and what is contributing, what factors are contributing to that. 129 00:30:04.642 --> 00:30:17.512 Robert L Russell: and and then moving over to evaluation. You know, as the project is being designed, developed, and implemented? Is it moving along the plan that you outlined in your proposal? 130 00:30:18.422 --> 00:30:42.981 Robert L Russell: And are you making, you know? Often the the iterative is used. And that means that you know, along the way there are probably multiple points in time where you're getting some evidence from evaluation that tells you. Well, maybe you should try this approach instead. Or maybe you should tweak this part of the design, or you know that sort of thing in order to improve your improve the experience so that it's even more effective, and so that it will. 131 00:30:43.252 --> 00:30:53.132 Robert L Russell: so that it will result in some significant findings that will contribute new knowledge. and so obviously related to. So one thing is, are you? 132 00:30:53.262 --> 00:31:03.052 Robert L Russell: Are you making progress? you know? Are you following the your your development plan? And are you, you know, hitting the marks that you proposed 133 00:31:03.132 --> 00:31:05.072 Robert L Russell: in that development plan. 134 00:31:05.182 --> 00:31:15.792 Robert L Russell: you know, and and that usual. Another way of translating, that is, are you? Are you in the developing and implementing what you stated you were going to implement in the proposal? And then 135 00:31:15.982 --> 00:31:26.762 Robert L Russell: the next question is, to what extent are the products and processes effective, and that you may do, you may assess that at several points in time. So here you're trying to find out. Is it really working. 136 00:31:26.832 --> 00:31:33.181 Robert L Russell: And and what is the nature of experience? And is it really working? so okay, next slide 137 00:31:34.192 --> 00:32:04.112 Robert L Russell: so kinds of evaluation? Well, there's formative evaluation which can begin with, you know, some front end work, or before you even design the project you may want to consult well, in the I test program. It would be to your great advantage to consult with perspective participants in the community before you design your project so that you can identify strengths, interest and needs of of the prospective target audience, and then you can use that to help fine tune your design. 138 00:32:04.252 --> 00:32:05.102 Robert L Russell: Oh. 139 00:32:05.762 --> 00:32:28.281 Robert L Russell: and so that might occur as a process throughout the development phase of your project. And then some of the evaluation you know at the end, you'll wanna you want to demonstrate? Is your project, effective or not. And so you would want to pick usually an an external evaluator, or if for evaluation team is included in in many projects. 140 00:32:29.022 --> 00:32:30.322 Robert L Russell: And 141 00:32:30.472 --> 00:32:33.842 Robert L Russell: just to qualify that a little bit 142 00:32:34.192 --> 00:32:47.912 Robert L Russell: by external. That means that they're not. They're not part of the design and development process other than providing evaluation information along the way, and and then the eventual summative evaluation. So some 143 00:32:48.002 --> 00:32:58.482 Robert L Russell: someone at someone or a different department at any university can serve as the external evaluation person or team. okay, going over to 144 00:32:58.512 --> 00:33:05.551 Robert L Russell: project evaluation personnel. Many projects have an Advisory board. In some cases 145 00:33:05.692 --> 00:33:14.442 Robert L Russell: the Advisory Board can serve as the evaluator But in any event, if you have an Adv Advisory Board, you should have you know you should. 146 00:33:14.722 --> 00:33:24.961 Robert L Russell: They should bring to the proposal or to the project. A relevant range of expertise and advisory boards can often be useful when you know when you're 147 00:33:25.342 --> 00:33:47.302 Robert L Russell: let's say you're doing an 8 and exploring theory and and design project. Well, they might be very useful to suggest ideas along the way, because what you're trying to do is, you're trying to refine the approach that you'd like to take when you present a more well developed project, you know, in A, in a design designing and testing implementation proposal. 148 00:33:47.402 --> 00:34:16.991 Robert L Russell: so an advisory board can have that function. But let's say you already have. you already have a well developed project. Let's say you're doing a a scale up type proposal. Well, you you don't really need to. You don't necessarily need to test out the intervention that is the curriculum, or whatever the experience is. Again, through a full evaluation process, you might be able to simply provide the Advisory Board with information along the way, you know, to help 149 00:34:17.542 --> 00:34:20.592 Robert L Russell: gain feedback on how you're doing. 150 00:34:21.472 --> 00:34:48.021 Robert L Russell: So in some cases you may not need an external evaluator, but in most cases you'll want to have an external evaluator who will, who will provide you not only with some formative but summative evaluations. And and as the point bullet here says, these are strongly developed. These are strongly recommended for design and development type studies and at the bottom you'll know a couple of couple of resources that are available from still are, and then 151 00:34:48.232 --> 00:34:59.541 Robert L Russell: the still our resource center and then the other one is for the informal science program. But there are also some useful evaluation resources that you can find on that website as well. Okay, next slide. 152 00:35:02.672 --> 00:35:04.872 Robert L Russell: So yeah, here we go. 153 00:35:05.082 --> 00:35:18.462 Robert L Russell: So here are the key elements of the research plan, you know, and, as I already have stated, you know, an overview of relevant research that establishes. you know, the importance and validity of the research are doing. And and 154 00:35:18.962 --> 00:35:27.101 Robert L Russell: you know, which leads to research questions that are linked with your research background. And then 155 00:35:27.192 --> 00:35:55.001 Robert L Russell: that's one that's one aspect of your project. The other aspect is the actual learning, experience or learning environment. And so the rationale should also flow out a bit flow from your research background and and theoretical approach that really provides the rationale. For why, you know, you're you're presenting, you're providing or developing a learning environment that will that will support in your view the impacts 156 00:35:55.032 --> 00:35:56.772 Robert L Russell: and the outcomes, and 157 00:35:56.872 --> 00:36:12.631 Robert L Russell: and also according to the rationale you present, you know. So that's your project. And so you want. You'll want to have research questions, you know, that are clear and not just, general and vague and not grounded in theory. 158 00:36:12.652 --> 00:36:17.522 Robert L Russell: and it's it's also to your advantage to explain. So you have research questions. And then 159 00:36:17.672 --> 00:36:24.481 Robert L Russell: the the research design which you know, that would that would include the various assessment tools. 160 00:36:24.572 --> 00:36:28.271 Robert L Russell: They can be qualitative, they can be quantitative. 161 00:36:28.552 --> 00:36:44.772 Robert L Russell: that will inform the research questions. And and so you should explain you, you should explain in relation to research questions, what data are you collecting? And what what tools are you using to collect that data? And how will that data be analyzed so that 162 00:36:45.192 --> 00:36:46.942 Robert L Russell: so that reviewers can 163 00:36:47.182 --> 00:36:56.291 Robert L Russell: can assess whether you have. You know you've selected or you've defined careful research questions in the outset. And if the research tools 164 00:36:56.482 --> 00:36:58.631 Robert L Russell: or evaluation tools. If that's 165 00:36:58.692 --> 00:37:08.901 Robert L Russell: the other aspect. If if the approaches that you're taking will provide useful data, you know, relevant data to address the research questions you've defined. 166 00:37:09.392 --> 00:37:24.592 Robert L Russell: And then, of course, you want a description of the research team's role in expertise as you would for the overall project. You'd want to define the roles of the various key play key staff members and and partners in the project. Okay, next slide 167 00:37:25.322 --> 00:37:37.641 Robert L Russell: line of stuff is encouraging applicants from Epsilon, from Eps core states, and you can see the listing of the eligible States. These are states that in most cases they'll have a a lower 168 00:37:37.752 --> 00:37:57.312 Robert L Russell: lower number of people living in the State than you know, like California or New York. But you know these are the States that that don't. The typically don't have as many people. But but the the main thing is that these States don't receive a Federal research funding at the same rate as some of the other States do so. Nsf. Is really encouraging eligible 169 00:37:57.642 --> 00:38:20.862 Robert L Russell: pis from eligible states, you know, for applying. And and there is a listing of some of the kinds of projects that upscore likes to fund that. They help develop research capabilities, you know, in in your, in your location. they establish a sustainable initiatives that help support stem education, and that they address 170 00:38:20.882 --> 00:38:22.072 Robert L Russell: diverse 171 00:38:22.592 --> 00:38:32.102 Robert L Russell: participants as well as institutions and organizations. So Hispanic serving institutions tribal colleges and 172 00:38:32.322 --> 00:38:39.262 Robert L Russell: black historically, black colleges and universities are strongly encouraged to to apply to. I test. 173 00:38:41.712 --> 00:38:43.862 Robert L Russell: So anyway, 174 00:38:44.472 --> 00:38:52.082 Robert L Russell: further, it's it's supposed to support the sustainable engagement of project participants, you know, and part of it is by 175 00:38:52.132 --> 00:38:58.741 Robert L Russell: communicating, sharing outreach, and so forth, to disseminate the results of the project and to exchange ideas 176 00:38:59.072 --> 00:39:05.331 Robert L Russell: and the practical aspect. Also very important. The impact research 177 00:39:05.352 --> 00:39:19.212 Robert L Russell: should have an in should have a reach, or in these projects, they're encouraged to have a reach that just beyond research and education itself, but also economic development that goes outside university or organizational settings. 178 00:39:19.552 --> 00:39:24.922 Robert L Russell: So you can see below, there are a couple of links that will lead you to more details about the upscore program. 179 00:39:25.322 --> 00:39:27.252 Robert L Russell: next slide. 180 00:39:29.452 --> 00:39:47.082 Robert L Russell: So these are some of the, you know, since I test focus on technology. These are some of the areas that are really they're very current. And if you can develop a project that will engage participants in any of these areas. You can. I don't have to read the list to you there, but there. 181 00:39:47.642 --> 00:40:05.941 Robert L Russell: there they are. And and you can see most of these areas. You know, most of these areas appear in the news like especially artificial intelligence at the moment, but appear in the news all the time, which which is just. I don't know if that's a great metric of their relevance. But these these 4 areas are of great interest to 182 00:40:06.022 --> 00:40:11.182 Robert L Russell: to Nsf. And and and because they are of great value to the nation. 183 00:40:11.492 --> 00:40:13.651 Robert L Russell: last slide. 184 00:40:14.512 --> 00:40:25.552 Robert L Russell: So let's say you have a project idea, and and maybe and you would probably be to your advantage to get, you know, some further feedback. The way you can do that is to send a a brief 185 00:40:25.582 --> 00:40:35.411 Robert L Russell: one or 2 page summary of the research that you would like to conduct. You know the the research, and also what is it you're going to do, you know? So 186 00:40:35.512 --> 00:40:49.481 Robert L Russell: what are? What need are you addressing? What is little capsule summary of what is the project? And and then what is the research that you're going to do to investigate that. And so 187 00:40:49.692 --> 00:40:55.502 Robert L Russell: I I kind of just said in different words the last sentence of of what the synopsis should include. 188 00:40:55.572 --> 00:41:08.352 Robert L Russell: And so you can write up your one pager, and you can email that to the I to the address listed at the top. There, Dr. L. I test at Nsf. Gov. And we monitor that 189 00:41:08.452 --> 00:41:21.212 Robert L Russell: email every day, and then your inquiry would be forwarded to a to a program officer who would contact you and offer you feedback, or try to set up an appointment when you could have a discussion on the phone 190 00:41:21.402 --> 00:41:48.032 Robert L Russell: or a or a Zoom Meeting. So that's a really a great way to get feedback, and in that case we don't. Necessarily we don't critic your idea. What we try to do is we try to try to advise you on what elements of your proposal to make sure you fully develop whatever aspect of your proposal that we highlight, you know, as when we're discussing your summary, and also provides you, you know, with a opportunity to ask any questions you may have. 191 00:41:48.952 --> 00:41:50.571 Robert L Russell: So I think. Now 192 00:41:50.632 --> 00:42:11.611 Robert L Russell: well, here's the full proposal, deadline, August eleventh. And there's the solicitation that that gives you all of the specifics you need. in order to all all the specifics you need, except there is also an an additional Nsf guide called the Ppg. Maybe Stellar could put that in the in the chat and that's that's a 193 00:42:12.372 --> 00:42:23.801 Robert L Russell: 500 page plus kind of manual that tells you everything you need to know about proposal. So you don't need to read it through word by word, but it tells you, for example, what are, what are the allowable costs? How many pages can you 194 00:42:24.032 --> 00:42:32.641 Robert L Russell: can you have in your proposal what it? What font size can you use? And and so on, and so forth. So that's always good to consult that as well as the solicitation.