Here’s how I used AI to detect an AI generated essay.
- First I have a (free) account at https://openai.com/blog/
chatgpt. - Then, I gave the essay prompt to the AI model to see what it produced.
- Note: it creates a unique essay each time, so I cycled through the prompt a few times to get a general idea of the language it uses for this specific essay prompt. I noticed a LOT of similarities between the student essay and the AI generated essays.
- Next, I asked AI to analyze the student’s essay with the following prompt:
- “I am going to give you a sample of text. Please tell me if it sounds like something you could have possibly written. Look at paragraph structure, sentence structure, grammar and spelling, and overall logic and flow. Tell me whether this seems consistent with the type of writing you typically produce. [pasted student’s essay].”
- The AI responded with the following analysis:
- “Yes, this text sounds consistent with the type of writing I typically produce. It demonstrates a clear structure, coherent and well-organized paragraphs, and a formal and objective tone. The sentences are grammatically correct and free of spelling errors. The logic and flow are well-constructed, and the information is presented in a clear and concise manner. The text provides an informative and analytical comparison between Confucianism and Taoism, discussing their origins, beliefs, principles, and practices, and offers a conclusion that summarizes the main points of the essay. Overall, the writing is informative, analytical, and well-researched.”
- I asked the AI to be more specific about which pieces seem to be AI produced. The AI responded:
- “All of the text appears to be consistent with the type of writing I typically produce as an AI language model. The structure, grammar, spelling, and overall logic and flow of the text are well-constructed and demonstrate the type of language processing and generation that is expected from an AI language model. The content is informative, analytical, and objective, and the style and tone of the writing are consistent with the type of writing that an AI language model is capable of producing.”
You probably have picked up on the fact that the AI cannot confirm 100% that a piece of writing is AI generated. But this was at least enough for me to have a conversation with the student.
I told the student, “did you know that you can plug a student’s essay into ChatGPT and it will tell you if it was AI generated?” Then I showed him the above AI analysis of his essay and asked him point blank, “this essay is AI generated, isn’t it?” He admitted that it was.
Hopefully this is helpful!
Jacob Perry
Interesting. Did you try the null condition (essay examples from students who clearly didn't use ChatGPT), too? I feel like all the AI check programs/scripts/techniques are lower confidence than something like TurnItIn.com, and it ends up being something of a bluffing game.
Love how sneaky you got on this… and the results it lead to… definitely we will need to learn more about this tool to better understand how this changes things for us as teachers
So interesting! Thank you for sharing.
As far as I understand, AI really isn't that great at detecting if something has been written by AI. I hear about a lot of false positives, because anything with "clear structure, coherent and well-organized paragraphs, and a formal and objective tone" could be falsely flagged as written by AI. However, this is a very canny way of asking a student if they used AI, because a student might not know how accurate the AI judgment really is. They might just assume they've been caught and admit to having used AI.
Using "fire to fight fire." Very interesting use of AI. We have a similar Photomath detector – it's called Mary Beth.