A William & Mary computer science professor studied how well students who use generative artificial intelligence compare to students who rely on human tutors.
Her initial results show that AI tools helped students improve their overall scores in class, but that students struggled to use the technology when they needed in-depth help.
Janice Zhang became interested last year in the impact of generative AI on student work as ChatGPT and similar tools became widely available in the spring. She and her research assistants designed CodeTutor, a tool powered by AI. During the semester, the research team divided an entry-level computer programming class into groups — one with access to CodeTutor and one without. Both groups had access to a human teaching assistant.
The team found a “significant” improvement in final scores among students who used the AI tool versus those who did not. They plan to conduct further studies this fall.
But, Zhang noted, the study showed the students found AI less helpful with time and turned to the human tutor. One theory is that the AI tool was helpful with more simple tasks in earlier lessons, and was not as helpful with tasks that required more critical thinking.
Zhang said as she and her team examined the prompts that students used to ask for help, they recognized that how the questions were asked often impacted the quality of the response.
“The way that you ask prompts is very important,” she said, “otherwise it’s just garbage in, garbage out.”
Some people in the education community have been hesitant to allow AI into the classroom. But Zhang believes AI is an unavoidable tool that will transform teaching and learning, and said the focus should be on teaching students how to use it well and ethically. Students should be taught to think how they structure prompts and be aware of bias and the risk of inaccurate information. Zhang said she tells her students that AI tools can be good for brainstorming, but warns them that they are responsible for what they write.
Zhang is working with some K-12 school divisions, including on the Peninsula and in Northern Virginia, to create AI literacy camps for upper elementary and middle school students about how to use the tool for math modeling.
The research team is also looking at ways to incorporate features into the CodeTutor tool that can help decrease teacher workload.
Nour Habib, nour.habib@virginiamedia.com
