Project Profile

The recent development of artificial intelligence (AI) tools such as ChatGPT and Bard present new opportunities and challenges for learners in the elementary and middle grades. Despite their potential for facilitating and supporting scientific reading and writing, there are pressing concerns about how AI tools can produce harmful, biased, and false content, which then gets reproduced by users who lack sufficient critical evaluation skills or do not understand the importance of evaluating AI-generated content. Thus, there is an urgent need to develop instructional materials and strategies to support students? development of these important critical thinking skills. The purpose of this time-sensitive project is to develop a web-based platform, called Compose With AI, aimed at guiding students to evaluate AI-generated content and use factual information to compose common types of science-focused writing (e.g., composing arguments, claims or solutions related to science topics). The Compose with AI platform will: 1) guide students to gather and critically evaluate content produced by AI, 2) guide students on beneficial and ethical uses of content produced by AI, and 3) scaffold students? use of AI-generated content as a model and resource for composing science-focused texts. Project researchers will use gathered data to determine what critical evaluation approaches and strategies inhibit and enhance students? abilities to use and critically evaluate content generated with AI, including when using the Compose With AI platform.

The project team will follow a design-based implementation research approach to evaluate the usefulness and usability of the Compose With AI. Data to address the address the research questions will be collected as participants engage in a guided task where they use Compose With AI to evaluate AI-generated content and generate a response to a prompt. A verbal reporting methodology will be used to understand students? cognitive processes as they think aloud during the task (concurrently) and as they reflect on their work after the task (retrospectively). Verbal protocol analysis will focus on which approaches and strategies students selected from the list presented in the platform, how they used the strategies, which strategies were successful, and which uses of AI were most useful for integrating AI-generated content into their responses. Teacher participants will also provide feedback on the platform design and the student think-aloud tasks. All data will be combined and analyzed qualitatively to answer the research questions. Findings will be used to refine and publicly share the Compose With AI platform for classroom use. This proposal was received in response to the Dear Colleague Letter (DCL): Rapidly Accelerating Research on Artificial Intelligence in K-12 Education in Formal and Informal Settings (NSF 23-097) and funded by the Innovative Technology Experiences for Students and Teachers (ITEST) program, which supports projects that build understandings of practices, program elements, contexts and processes contributing to increasing students' knowledge and interest in science, technology, engineering, and mathematics (STEM) and information and communication technology (ICT) careers.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.