AI & Plagiarism
As you’re certainly aware, many educators are concerned about students using OpenAI’s artificial intelligence conversational chatbot ChatGPT for plagiarism or to cheat on their work. So, what should we do about it?
AI Detection Tools
The first question that many asked was “Can we detect when a student has used ChatGPT or other AI tools?” Well, a number of different tools claim that they can. Let’s see if they’re right.
1. GPTZero
GPTZero was launched by 22-year-old Princeton senior, Edward Tian, on January 2nd. Ironically, he created GPTZero using a tool – GitHub Co-Pilot – that is powered by GPT-3, the same powerful machine-learning model that powers ChatGPT. GPTZero crashed due to demand within days, before his hosting service gave him extra resources to keep it running.
The tool looks at the complexity of the text and the variation of the sentences to determine how likely it is that the text you put into it was AI-generated.
Let me repeat an important phrase there: how likely it is to have been AI-generated. Not if, but how likely.
You can use this tool to see if it may have been written by AI, but it can’t guarantee it. And, it actually doesn’t tell you one way or another. Instead, it gives you a burstiness score and perplexity score – two measures that Tian created – and then uses both to tell you how likely it was that it was an AI-generated sample.
As you can probably guess after hearing that, it’s not foolproof and it’s not 100% accurate. I tested it on two sets of text:
📕 An excerpt from my book – The result? ❌ It said it had some parts “that may have been written by AI.” Nope. I wrote my book myself, I promise.
🤖 A silly story that I had ChatGPT write – The result? 🤷♂️ It once again said that it had some parts “that may have been written by AI.” May have? I think we’re going to need better than that. As you can see though, it does have a lower average perplexity score and burstiness score than the excerpt of my book did.
2. OpenAI’s Text Classifier
A few weeks later, the company behind ChatGPT – OpenAI – released their own tool meant to detect AI-generated text. In the press release announcing this free tool, which is called AI Text Classifier, Open AI said that it is “not fully reliable” and “should not be used as a primary decision-making tool.”
There are a few additional disclaimers on the site telling you:
- it requires a minimum of 1,000 characters – that’s 3 and a half tweets!
- you can easily edit AI-generated text and trick this tool,
- it struggles with children’s writing
- it has difficulty with non-English text
🤨 If you’re not feeling skeptical after reading those 4 bullets, you may want to re-read them.
As with the previous tool, I tested it on two sets of text:
- 📕 An excerpt from my book – The result? ✅ “very unlikely AI-generated.“
- 🤖 A silly story that I had ChatGPT write – The result? ❌ It said it was “unclear if it was computer generated.”
3. Fictitious.ai
Another tool, Fictitious.ai, offers a freemium model for AI Detection. In the free version, you can integrate Fictitious.ai into the Canvas LMS, but you have to schedule a consultation to get started. The paid versions add additional LMSs and a few other features.
Their site proclaims that their machine learning model can “detect generated content with 99.9% accuracy.” The screenshots shown here indicate that it measures the “integrity” of each submission on a percentage scale, with higher numbers indicating a higher likelihood of AI-generated text. It also breaks this down on the sentence, paragraph, and essay levels.
I should point out – one of the human-written paragraphs in their example showed up as 20.3% AI-generated. Meaning that even their tool thought there was a 1/5th chance that something they had written was written by AI. So, again, not 100% accurate and not foolproof.
(I was unable to test this tool myself)
4. AI Writing Check
Quill.org and CommonLit teamed up to provide the free AI Writing Check service at AIWritingCheck.org. They report that “based on testing with 15,000 essays, their tool is accurate 80-90% of the time.”
😑 So, use it on 100 students’ essays and be prepared to be wrong on 10-20 of them.
As with the previous tool, I tested it on two sets of text:
- 📕 An excerpt from my book – The result? ✅ It said it was written by a human.
- 🤖 A silly story that I had ChatGPT write – The result? ❌ It predicted that it had been written by a human.
By the way, these two nonprofits – Quill and CommonLit – developed a 4-page PDF toolkit about addressing AI Plagiarism in the Classroom. It’s worth checking out.
5. Writer.com
Finally, Writer.com offers an AI Content Detector, 
which is actually geared towards making sure web pages don’t have – or appear to have – too much AI-generated content, as some search engines are less likely to show those results. However, we could potentially use it for identifying AI-generated “student work” as well. Like the other tools we’ve discussed, this free tool shows a percentage of how likely it is that text was AI- or human-generated. With this one, the higher % indicates a higher likelihood that a human wrote it. So you’re hoping for 100.
As with the previous tools, I tested it on two sets of text:
- 📕 An excerpt from my book – The result? ✅ It said it was 100% human-generated.
- 🤖 A silly story that I had ChatGPT write – The result? ❌ It also said it was 100% human-generated. Oops.
I forget – is 50% accuracy a passing score?
Good news and Bad news
The good news, there are tools that detect AI-generated text.
The bad news, none of them – not even the one that OpenAI themselves made – are 100% accurate.
You can’t downplay that. If you falsely accuse a student of cheating on an assignment, whether by looking at a peer’s paper, copying text from Wikipedia, or using AI, you are destroying the trust, rapport, and respect that you have with that student. This cannot be taken lightly. So, my advice would be to only use these tools when you suspect AI was used. And that means knowing your students’ typical writing style, quality, and patterns and being able to notice departures from those. And then, if that happens, maybe you consider using one of these tools for extra evidence. But remember: you won’t be sure. So tread very carefully.
A shout-out, by the way, to Matt Miller and Karly Moura at “Ditch That Textbook” – I knew of a few of these AI-detection tools by staying plugged into Edtech news, but the rest I found through their awesome AI in education page.
[ Image(s) Source: https://www.npr.org/2023/01/09/1147549845/gptzero-ai-chatgpt-edward-tian-plagiarism, https://www.theverge.com/2023/1/31/23579942/chatgpt-ai-text-detection-openai-classifier, https://www.fictitious.ai/features, https://aiwritingcheck.org/, https://ditchthattextbook.com/ai/, https://s3.amazonaws.com/quill-image-uploads/uploads/files/A_Toolkit_for_Addressing_AI_Plagiarism_in_the_Classroom_2747.pdf, https://writer.com/ai-content-detector/, https://na.linkedin.com/posts/canvaslms_how-are-you-incorporating-ai-into-your-courses-activity-7026225972429406209-TPP2]
Find more updates like this one by tuning into the EdTech News Brief on . . .
