Racist AI behaviour is not a new problem

St George's Hospital Medical School developed a computer program in the early 1980s to automate the initial screening of medical school applications. The program was created by Dr Franglen to reduce workload and eliminate inconsistencies in the admissions process. It was trained on historical admissions data and by 1979 achieved a 90-95% correlation with human selection panel decisions. From 1982, all initial selection was done by computer, with the system generating scores based on UCCA application forms to determine which candidates should be interviewed. The system deduced race from surnames and place of birth since UCCA forms contained no explicit racial information. In December 1986, Dr A Burke and Dr J Collier, both senior lecturers at St George's, complained to the Commission for Racial Equality that the program unfairly discriminated against women and people with non-European sounding names. The Commission found the school guilty of practicing racial and sexual discrimination, determining that as many as 60 applicants each year among 2000 may have been refused interviews purely because of their sex or racial origin. The Commission noted that the program was not introducing new bias but merely reflecting discrimination already present in the system. Three previously unsuccessful applicants were subsequently offered places at the school.