A Florida teenager was wrongfully accused of academic dishonesty by Honorlock's AI proctoring system, which used Amazon Rekognition to flag her for looking away from the screen during an online exam, resulting in a zero grade and academic misconduct warning despite the ambiguous nature of the evidence.
In February, a 17-year-old Black female student at Broward College was flagged by Honorlock's AI proctoring system during a biology exam for 'frequently looking down and away from the screen before answering questions.' Honorlock, founded in Boca Raton, Florida, administered nine million exams in 2021 and uses Amazon's Rekognition facial detection technology to monitor test-takers. The system flagged the student as suspicious within the first minute of her exam, generating at least one red flag for highly suspicious behavior. A 50-second video clip showed the student looking down for up to 10 seconds after reading questions, but the footage was ambiguous about what she was actually doing. Despite the student's explanation that she looks down to think and fiddles with her hands to jog her memory, Professor Jonelle Orridge and the dean found her 'responsible' for 'noncompliance with directions,' resulting in a zero on the exam and a warning on her academic record. The incident highlights broader issues with AI proctoring systems that have faced criticism for invasiveness, glitches, false allegations, and failure to work equally well for all demographic groups. Amazon's Rekognition has been specifically criticized for bias, with research showing it works least well on darker-skinned females.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed