Our Generative AI in Education Experiments: Lessons Learned

Navigating the Evolution of Generative AI

Like our education partners, we have been closely monitoring the evolution of generative AI and exploring its implications for education. When OpenAI released ChatGPT to the public in late 2022, we recognized that a new era of accelerated change was upon us. We watched with interest as school districts around the country debated whether to ban generative AI or to embrace its possibilities with enthusiasm. We marveled at the new generative AI tools being released every day, and we explored how various large language model (LLM) tools might create efficiencies for teachers and unlock learning opportunities for students.

As we make sense of the impact this technology will have on teaching and learning, we are reminded of Neil Postman’s claim that “a new technology does not add or subtract something. It changes everything” (1992, Technopoly: The Surrender of Culture to Technology). Generative AI tools follow a long line of technologies that have changed education, some more dramatically than others. Recent discussions about generative AI have recalled a worry from just a few decades ago, when people thought that the introduction of graphing calculators would mean students no longer needed to learn math. That proclamation never came true. Instead, change happened in the form of teaching math more conceptually and encouraging learners to leverage calculators as tools. Similarly, we see value in embracing the potential of generative AI as tools that can change how we empower learning.

As schools adapt and respond to the use of generative AI in education, we understand that it can feel overwhelming when trying to simultaneously:

• Learn how to incorporate these tools in a variety of use cases

• Stay up to date as the technologies evolve

• Know which AI tool works best for each teaching and learning situation

• Enact responsible policies that acknowledge the numerous ethical and privacy concerns

• Redesign lessons, assessments, and learning environments in response to AI

We’re here to help by sharing lessons learned from our recent experiments with generative AI to create teacher efficiencies and student supports.

Committed to Being a Responsible Partner

Our commitment to putting educators first and helping students thrive means that we aim to provide safe, trustworthy, unbiased, and effective guidance to our partners about the use of generative AI. As generative AI began to dominate educators’ conversations and the media, we listened first to teachers and school leaders to understand their enthusiasm and their worry. We heard excitement about AI’s potential for:

• Writing lesson plans

• Generating visuals

• Brainstorming and organizing ideas

• Creating sample assignment responses to discuss with students

• Adapting texts to support differentiation

• Editing and improving writing

Along with these and other possibilities for saving time and supporting learning, we heard concerns about:

• Reliability of the information AI generates

• Students cheating on assignments

• Increased educational inequity

• Data privacy and copyright

• Professional development needs

As we explored a variety of AI tools ourselves, we kept our partners’ needs in mind. We also created an internal shared hub for posting resources and insights from the podcasts, webinars, and articles that we found most helpful so we could advise our partners on generative AI accordingly.

Our Lessons Learned about Generative AI in Education

From all of our testing, listening, and learning about generative AI, we offer these lessons to guide partners as they consider how best to navigate the changing educational landscape now that these tools are becoming more widely available.

1. Keep humans in the loop for the best results.

Our experiments have shown us that worries about AI tools replacing people are premature, especially when it comes to teaching, writing, and demonstrating learning. It remains vital to keep humans in the loop to design the prompts that get the best results and to verify the information that AI generates. After all, we know that generative AI tools sometimes hallucinate—that is, they can generate grammatically correct and sometimes plausible content that is false or that misrepresents information and relationships between ideas. People must also be involved to ensure a human voice and perspective in what is created.

2. Encourage a culture of experimentation and sharing.

As these technologies evolve and the implications for education are not yet fully realized, we recommend creating space to experiment with generative AI and share together in schools, where leaders, teachers, students, and other stakeholders can learn together and talk about possibilities and concerns with perspectives informed by experience. We found such space to discuss what we learn with colleagues along the way has accelerated our understanding and helped us navigate challenges with generative AI.

3. Try different tools.

Our investigations taught us that AI tools all function differently and continue to change with each update. We saw benefits to learning from the patterns we discovered as we explored different generative AI. Therefore, we are not recommending any individual tool, and we suggest exploring the full range of tools to learn their strengths and weaknesses.

4. Rely on trusted sources for information.

With so many people, podcasts, platforms, and publications claiming to offer insights about generative AI, it can be hard to know where to turn. We found it helpful to read and listen to sources informed by a combination of research and use in education settings. We have learned a great deal from Dr. Ethan Mollick as he generously shares how he recommends using generative AI with students, including providing detailed prompts, and as he encourages users to embrace the “weirdness” of this technology. We also turn to Common Sense Education as they offer guidance about how to handle generative AI in schools.

5. Center critical thinking and reflection.

Our experiments have benefited from approaching generative AI with criticality, questioning why we got the results we did with different prompts and tools and analyzing outputs as we revised our prompts. We also appreciated having time to reflect on what we were learning and to integrate and iterate on our findings. With that in mind, we recommend that educators and students be given built-in time for questioning and reflecting as they review sources and experiment with the tools.

Our Experiments with Generative AI

Thus far, we have focused our experimental efforts on exploring how generative AI can:

• Save teachers time when grading student work

• Translate text for multilingual learners

• Scaffold content learning and help differentiate texts for readers of varying abilities

• Facilitate language practice opportunities for students

• Promote critical thinking as students learn to analyze AI’s output

Below we share details about the teacher and student use cases we investigated.

Teacher Use Cases for Generative AI

Generative AI as a Grading Assistant

Grading open-response items and essays is one of the most time-consuming tasks for teachers, so we spent a great deal of energy experimenting with ways to save teachers time on this type of grading. We investigated grading with a few different generative AI tools and discovered that you have to become a tool’s teacher before it can grade a response as a human would.

We started with a prompt that provided the AI tool with a rubric and asked it to grade a class set of student responses with that rubric. Next, we analyzed the grades given by the tool and determined how each grade differed from what a human scorer would award. We used this analysis to increase the specificity of our rubric and to chat with the AI to give it feedback. We iterated on this process until the AI tool was performing at the level of a human scorer. This training process took two to three hours per assignment, which was a significant time investment. Still, we believe this initial effort would be worthwhile for an educator scoring the same assignment across multiple classes and years. Once the AI tool was fully trained on an assignment, it tended to perform well when given a new set of responses for that assignment. However, we also discovered that in a span of just a few months, each AI tool performed inconsistently, likely due to updates to the model. Thus, it’s important to spot-check the scores given by an AI tool over time even if the assignment and rubric have not changed.

Generative AI as a Translator

The number of U.S. public school students identified as English language learners continues to rise, and they speak a range of languages at home. Our partners are always looking for ways to support their multilingual learners effectively. Thus, we tested generative AI tools’ ability to translate course materials and teacher feedback, and to provide real-time translation during instruction. We experimented with Spanish, Arabic, and Greek because we have access to native speakers of these languages who analyzed the generated translations.

Our experiments indicated generative AI does well at most translations, no matter which tool we used. We observed that generative AI produced the highest-fidelity translations for written instructional materials rather than for transcribed spoken language or informal written feedback. We hypothesize that we got these results because we used published materials that have been edited for clarity, grammar, and syntax, and generally do not contain idioms. Nonetheless, the translations of transcribed spoken language and informal written feedback were precise enough to be useful. Therefore, we see the potential in using generative AI to support engagement and increase student outcomes for multilingual learners.

Generative AI as a Scaffold for Text Readability

Generative AI produced mixed results when we used it to scaffold content. Most of our experiments focused on adjusting the reading level of learning materials. We started by asking the tool to identify the Lexile and Flesch-Kincaid level of a text. None of the tools we used were able to accurately and consistently identify the readability level based on either system of measure. Next, we prompted AI tools to revise text based on Lexile or Flesch-Kincaid level targets. The tools also struggled with this task.

When we changed our prompts to tell the AI tools to rewrite text for students at a certain grade level, the performance of most generative AI improved. However, we noticed issues with imprecise or misleading language, and AI-generated text often replaced important vocabulary words with nonacademic terms. After several iterations, we found that text could be better adjusted to grade level with a prompt that told generative AI how we wanted it revised, as in: Revise this text to simplify grammar, shorten sentences, and avoid complex clauses. Leave the vocabulary intact. Results improved further when we could specify a particular target reading level by grade. For example, to scaffold a high school text for a learner who independently reads at a sixth-grade level, the following prompt tended to produce good results: Revise this text for a sixth-grade student by simplifying the grammar, shortening sentences, and avoiding complex clauses. Leave the vocabulary intact.

Student Use Cases for Generative AI

Practicing Language with Generative AI

Edmentum’s courses include activities and questions that require students to discuss their learning. For students who are completing these courses independently, options for practicing discussion skills may be limited. Students can use generative AI to practice responding to discussion prompts and questions, starting discussions about topics in their coursework, and presenting information clearly. Students can also practice asking questions, summarizing points of agreement and disagreement, and making connections between new ideas.  

Our experimentation showed us that to help students get the most out of such language practice using generative AI, teachers can provide students with some starter prompts and stems. This approach helps students know how to interact with AI when they do not understand something, are finished with the discussion, or would like feedback on their part of the discussion.

How to craft an opening for the discussion

"In my ___ class, I'm learning about ____. I think/know/feel ____. What do you think?"

How to ask for a simpler response

"Can you simplify what you said there? I don't understand a lot of the words you used."

How to end the discussion and request feedback

"Thanks for having a discussion with me. I am finished now. However, I would like some feedback on how I did in this discussion. Can you give me two examples of what I did well in this discussion and one example of something I could do better in my next discussion?"

We found that at times during the discussion, generative AI may present incorrect information or introduce topics that have not been explored in class, just as a peer might say something incorrect or get off topic in a classroom discussion. Such quirks of AI allow students to practice identifying incorrect information and directing the conversation back to the intended topic.

Analyzing for Generative AI Hallucinations

Throughout our experiments, we often came across AI hallucinations. In some cases, these hallucinations may be considered a drawback to using generative AI. However, they can also be useful for supporting students’ critical thinking. We recommend teaching students to always question AI-generated content and to perform an analysis for quality. Such instruction might start by having students ask AI a question about something of particular interest to them. Then, students can be instructed to analyze the response following a series of steps, such as these:

1. Read for understanding. Do you understand the main ideas presented in the response? Does the response make sense to you? If not, why not?

2. Identify errors. The errors might involve grammar, spelling, facts, or logic. To identify errors, you can:

• Ask the AI tool for its sources of information, then look up those sources. Do they actually exist? Do they give the same information that the AI tool generated?

• Use a variety of tools to help you identify errors. Tools might include grammar checkers, plagiarism checkers, and fact-checking websites.

• Ask another generative AI tool to look for errors in the response.

• Research the facts stated in the response. Do other sources give contradictory information?

• Classify the errors. Once you have identified the errors, you can classify them into different categories. For example, you could classify grammatical errors as subject-verb agreement errors, pronoun errors, or verb tense errors.
  
  
  

3. Analyze the errors. Once you have classified the errors, you can look for patterns that might lead to greater understanding of how AI works

4. Evaluate for credibility. Is the tool providing you with trustworthy information? Is the information backed by evidence, objective, and up to date?

Having students analyze AI-generated responses in this way can help them understand the limitations of this technology while they build analytic skills.

Moving Forward

Given that generative AI is now widely available for educators and learners, we believe it necessary to continue to explore its possibilities and to keep an eye toward how these tools can be used to help our partners achieve equitable education outcomes. Generative AI has the potential to support teachers trying to meet the needs of diverse learners and to help students develop flexibility and criticality when it comes to using digital technologies. It’s important to keep educators informed so that they may guide whether and how students utilize generative AI for education as the technology continues to evolve.

We hope our education partners will engage in their own experiments with generative AI, exploring use cases that address their localized needs and sharing their reflections and findings with us as we all learn together about the power and possibilities of these tools. Edmentum will continue to monitor the ever-changing AI landscape and discover new ways to leverage generative AI in education. Reach out to your Customer Success Manager to stay updated on AI best practices.