How to Assess Learning in the Age of AI

As a physics teacher in New Zealand, I have recently used the level 3 NCEA physics standard 3.2: Demonstrate an understanding of the application of physics to a selected context as a way to assess my students’ learning of physics concepts involved in electronic circuits. Generally, this standard requires students to research a particular physics context by applying the ideas previously taught for the external assessments in the Level 3 physics course. This is a challenging task that involves not only understanding the physics concepts, but also applying them to a real-world situation, synthesising information from various sources, and communicating it effectively.

However, I have also noticed from conversations with my colleagues, that some students are tempted to use artificial intelligence (AI) tools such as large language models like Chat GPT to generate their responses and paste them into their assignments. This is a serious issue that undermines the integrity and validity of the assessment, as well as the learning process. How can I ensure that my students are actually learning and not just copying from AI?

In this blog post, I will share some of the strategies that I have used to mitigate this problem and to promote the ethical and effective use of AI in the classroom. I will also explain how I use student and teacher conversation as a way to assess learning and foster deeper understanding and higher-order thinking skills.

Why AI is not a substitute for learning

AI is a powerful tool that can help students with various aspects of their learning, such as getting started with an assignment, providing feedback on drafts, explaining complex physics in ways that they can understand, and helping consider different perspectives. However, AI is not a substitute for learning, and students should not rely on it to do their work for them.

There are several reasons why AI is not a substitute for learning:

• AI does not always produce accurate or reliable information. Sometimes, AI can generate false or misleading information, or even hallucinate facts that do not exist. For example, AI might claim that the speed of light is variable, or that gravity is a force that pushes objects away from each other. These are obviously wrong, and students need to be able to verify and evaluate the information that they get from AI.
• AI does not always produce relevant or appropriate information. Sometimes, AI can generate information that is off-topic, out of context, or not suitable for the purpose or audience of the assignment. For example, AI might include irrelevant details, use informal language, or express personal opinions that are not supported by evidence. These are not acceptable, and students need to be able to select and adapt the information that they get from AI.
• AI does not always produce original or creative information. Sometimes, AI can generate information that is plagiarised, copied, or recycled from existing sources. This is not ethical, and students need to be able to acknowledge and cite the sources that they use, as well as to reword the information in their own words.
• AI does not always produce meaningful or coherent information. Sometimes, AI can generate information that is nonsensical, illogical, or inconsistent. This is not effective, and students need to be able to organise and structure the information that they get from AI, as well as to connect it to the physics concepts and the selected context.

In summary, AI is not a substitute for learning, because it cannot guarantee the quality, relevance, originality, or coherence of the information that it produces. Students still need to use their own critical thinking, analytical, and communication skills to complete their assignments.

How to use AI ethically and effectively in the classroom

AI is not a substitute for learning, but it can be a useful tool to support learning if used ethically and effectively. Here are some of the ways that I encourage my students to use AI in the classroom:

• Using AI as a starting point, not an endpoint. AI can help students generate some ideas, keywords, or sentences that they can use as a starting point for their assignments. However, students should not stop there, they should use their own research, analysis, and synthesis skills to develop and refine their responses.

• Using AI for feedback, not grading AI can help students check their spelling, grammar, or style, as well as identify some of the strengths and weaknesses of their responses. However, students should not rely on AI to grade their work, and they should use their own judgement, reflection, and revision skills to improve their responses.

• Using AI as a learning, not cheating. AI can help students explain some of the complex physics concepts, formulas, or diagrams that they encounter in their assignments. However, students should not use AI to cheat or to avoid learning, and they should use their own understanding, application, and evaluation skills to demonstrate their learning.

• Using AI responsibly, not recklessly. AI can help students to access a wide range of information sources that they can use for their assignments. However, students should not use AI irresponsibly or recklessly, and they should use their own ethical, academic, and digital citizenship skills to acknowledge and cite the sources that they use, as well as to reword the information in their own words.

In summary, AI can be a useful tool to support learning, if used ethically and effectively. Students should use AI as a starting point, a feedback tool, a learning tool, and a responsible tool, but not as a substitute for learning.

How to use student and teacher conversation to assess learning

AI can be a useful tool to support learning, but it cannot replace the role of the teacher in assessing learning. AI detection software is often marketed as a reliable tool to check for plagiarism in academic writing. However, these tools are not as effective as they claim to be. AI detection software works by comparing the submitted text to a large database of sources, and flagging any similarities that exceed a certain threshold. However, this method has several limitations. First, AI detection software can only detect plagiarism from sources that are included in its database. This means that any original or obscure sources that are not indexed by the software will be missed. Second, AI detection software can be easily fooled by paraphrasing, rewording, or editing the text. Some tools claim to detect AI-generated content, such as ChatGPT and Google Bard, but their accuracy is not very high. Third, AI detection software does not account for the context, purpose, and citation style of the text. It may flag legitimate quotations, references, or common phrases as plagiarism, or ignore intentional plagiarism that is disguised as a citation. Therefore, AI detection software is not a substitute for human judgment and critical thinking of a teacher. 

One of the ways that I assess my students’ learning is by using student and teacher conversation. This is a form of oral assessment that involves having a dialogue with the students about their assignments and asking them questions that probe their understanding and thinking skills.

Student and teacher conversation can help me to assess learning in several ways:

• It can help me to verify the authenticity and originality of the student’s responses. By asking the students to explain how they came up with their responses, what sources they used, and how they reworded the information in their own words, I can check if they actually did the work themselves, or if they copied from AI or other sources.
• It can help me to evaluate the accuracy and reliability of the student’s responses. By asking the students to justify their responses, provide evidence, and cite their sources, I can check if they used correct and relevant information, or if they used false or misleading information from AI or other sources.
• It can help me to measure the depth and breadth of the student’s understanding. By asking the students to apply their responses to different contexts, scenarios, or problems, I can check if they understood the physics concepts and how they relate to the real world, or if they just memorised some facts or formulas from AI or other sources.
• It can help me to gauge the level and complexity of the students’ thinking skills. By asking the students to analyse, synthesise, compare, contrast, critique, or evaluate their responses, I can check if they used higher-order thinking skills, or if they just used lower-order thinking skills from AI or other sources.

Additionally, students also had to provide their primary sources, summarise the information they obtained from the sources, the keywords they extracted from the information and how they reworded the source information in their own words in order to mitigate the possibility of students just copying and pasting information into their assignments. It also gave me, the teacher, the ability to further probe their understanding of the fundamental physics concepts underpinning the assessment.

In summary, student and teacher conversation can help me to assess learning in a more authentic, valid, and comprehensive way than AI can. It can also help me to provide feedback, guidance, and encouragement to the students, and to foster a positive and trusting relationship with them.

Conclusion

AI is a powerful tool that can help students with various aspects of their learning, but it is not a substitute for learning. Students should not use AI to copy responses and paste them into their assignments, as this undermines the integrity and validity of the assessment, as well as the learning process. Instead, students should use AI ethically and effectively, as a starting point, a feedback tool, a learning tool, and a responsible tool. Learning is not something that can be done by AI alone. Learning is a human endeavour that requires curiosity, creativity, and collaboration. Students should use AI as a tool to enhance their learning, not as a shortcut to avoid it. The most important tool for learning and teaching is the relationship between the student and the teacher. Through conversations, teachers can assess the students’ learning, provide feedback, and challenge them to think deeper and higher.  From the student's perspective, they can demonstrate their learning, ask questions, and express their opinions. Through this respectful dialogue, both teachers and students can learn from each other, and grow together. By doing so, we can ensure that our students are learning.