Is AI marking accurate? What the trials show

Studies of AI marking accuracy measure agreement with human markers, and they measure it in different ways, so their figures cannot be compared directly:

• Exact agreement: the AI and the human give the same mark.
• Agreement within one mark: the marks are the same or one apart.
• Kappa: agreement after allowing for chance, where 1 is perfect. Quadratic weighted kappa also counts bigger disagreements for more.
• Comparative judgement agreement: how often the AI picks the same one of two pieces of writing as a teacher, which is not the same as giving the same mark.

Agreement on its own does not prove a marking system is sound. Ofqual’s 2026 working paper makes that point: a system can match human marks on average and still fail particular groups of students.

Human marking is not perfectly consistent either, which is the benchmark AI has to be judged against. Ofqual’s 2018 analysis of 2017 exams found the probability of a candidate getting the grade a senior examiner would give ranged from 0.96 for a maths qualification to 0.52 for an English language and literature qualification, although the grade given was the senior examiner’s or one next to it more than 95% of the time in every qualification.

In a 2024 US study of essay scoring, two trained human raters gave the same score 51% of the time and were within one point 92% of the time. The fair question for AI is whether it agrees with a teacher as often as another teacher would.

The evidence splits by task, and several of the figures below are a company’s own, so each entry says who ran the study:

• Short answers: in a 2024 study by researchers at Oxford and Carousel Learning, using short answers in science and history from pupils aged 5 to 16 on Carousel, GPT-4 reached a kappa of 0.70 against human markers, close to the 0.75 between the human markers themselves.
• GCSE mocks: a June 2026 preprint by Medly AI, using its own data from 32,534 double-marked GCSE mock answers, found the best models matched examiners more closely than examiners matched each other (quadratic weighted kappa of 0.74 against 0.69 in science, and 0.86 against 0.84 in maths). It has not been peer reviewed.
• Writing judged by comparison: in No More Marking’s March 2025 trial with 5,251 Year 7 pupils in 44 schools, the AI agreed with 81% of teachers’ decisions, against 87% between humans in an earlier assessment. This is the company’s own trial.
• University essays: a May 2026 Cambridge-led study of 761 psychology essays from three universities found AI put essays in the same degree band as humans between 35% and 63% of the time, depending on the university.
• Physics: a 2026 preprint from Durham University found AI marking of university physics essays harsher and more variable than human marking, while its marks for structured physics questions agreed well with humans.

The university essay studies above point to these weaknesses, which are worth checking for in any tool:

• It pulls marks towards the middle, marking the best essays down and the weakest up.
• It rewards length and vocabulary more than human markers do.
• It can be harsher or more lenient than teachers, depending on the model and the task.
• It is weaker on open, higher-mark questions than on short, structured ones.
• Several models can agree with each other and still disagree with the human marker.

• Short, points-based answers and maths working: close to teacher level in recent studies, so spot-check a sample.
• Extended responses against a levels-of-response mark scheme: a useful first mark with feedback, as long as a teacher checks it, especially at the top and bottom of the range.
• Classwork, homework and mock exams: a reasonable use, with a teacher reviewing the marks.
• Exams and non-exam assessment that count towards a qualification: AI cannot be the sole marker under Ofqual’s rules.

A tool’s own accuracy figures say little about your subject and your mark scheme. A small test with work you have already marked tells you more:

1. Take 20 to 30 answers you have already marked, across the full range of marks.
2. Have the tool mark them without seeing your marks.
3. Count exact matches and matches within one mark, question by question.
4. If you can, have a colleague mark the same set, so you have a human benchmark to compare with.
5. Look separately at the strongest and weakest answers, where AI drifts most.
6. Read the feedback: it should refer to the mark scheme and to what the student wrote.
7. Mark a few answers twice to see whether the mark changes.

Dolly marks every answer against the assignment’s own mark scheme, written with the question at levels of response or points. Each mark is checked against the mark scheme’s structure before it is saved and comes with a confidence score, so you know which answers to check first. Nothing reaches students until you have reviewed the marks, changed any you disagree with and published them.

Dolly is built for classwork, homework and practice, not for exams or non-exam assessment that counts towards a qualification. It has not yet published an agreement study of its own. Because Dolly marks work set in Dolly, test it on your own class: set one assignment, mark a sample of the answers yourself before you look at Dolly’s marks, then compare them using the steps above.