高考风向预测 · 外刊精读 15:Style over Substance|AI批改大学论文,为什么＂会写＂不等于＂写得好＂?

Cambridge researchers tested whether leading AI systems could reliably grade authentic university essays. Image source: University of Cambridge

Universities around the world are beginning to explore whether artificial intelligence could reduce the large amount of time teachers spend marking student work. Modern AI systems can already summarize essays, identify errors and produce detailed feedback. But grading an essay requires something more difficult: deciding how good the thinking actually is.

Researchers led by the University of Cambridge recently tested whether leading generative AI systems could do this reliably. The study used 761 real undergraduate psychology essays written by 125 students at three British universities. The essays had already been formally graded by human examiners. Researchers then gave the same work to several advanced AI systems and compared the results.

At first, the results appeared promising. Depending on the university, AI placed essays in the same broad degree category as human markers between about 35 and 63 percent of the time.

However, a closer look revealed an important pattern. AI was most accurate for essays near the middle of the grading scale. When human examiners gave exceptionally high marks, AI often marked the essays lower. When humans gave very low marks, AI tended to raise them. In other words, the AI pulled scores toward the middle. Researchers describe this as a central tendency bias.

This matters because the largest mistakes appeared precisely where grading decisions can be most important: near the boundaries between excellent and ordinary work, or between passing and failing.

The researchers also found that AI paid unusual attention to surface features of writing. Longer essays, a wider vocabulary and more complicated sentences were often associated with higher AI scores. Human examiners, by contrast, placed greater weight on qualities such as the strength of an argument, the use of evidence and the quality of critical judgment. This created a risk that AI might reward style over substance.

The problem was not simply that AI behaved randomly. In fact, when researchers asked the same AI system to grade the same essay again, it often produced a similar result. Different AI models also tended to agree with one another more closely than they agreed with human markers. Consistency, therefore, was not the same as accuracy.

The study also showed why results from one university cannot automatically be applied elsewhere. AI matched human grading most often on Cambridge essays and least often at Manchester Metropolitan University. The researchers suggest that differences in assignment types and the range of student marks may partly explain this gap. This means an AI system that appears reliable in one educational setting may perform quite differently in another.

The researchers do not argue that AI has no role in assessment. It may be useful as a "second pair of eyes," helping teachers identify possible errors, check consistency or flag essays where the AI and human marker strongly disagree. But they argue that current systems should not make final grading decisions.

Assessment, they point out, is not simply about producing a number. A grade represents a judgment about reasoning, evidence and academic quality. It also affects students' opportunities and their trust in the educational system.

For now, AI may be able to assist the examiner. It is much less clear that it can replace one.

这一篇最重要的逻辑，其实只有一句：

文章里最关键的概念之一：AI 喜欢把成绩往 middle 拉。

所以 style over substance 就是：过度奖励"看起来写得高级"，而不是"思想真的好"。Cambridge 报告明确指出，AI 评分对这些语言表面特征过度敏感，而真人评分者更重视论证和学术判断。

这是非常典型的研究设计题。如果只用 Cambridge essays，研究结果很好，你可能马上说：AI grading works。但是加入另外两所大学以后发现：准确率差很多。

不要只记数字。它真正说明：AI performance varied substantially across institutions。所以不能在学校 A 测得不错，就直接推出学校 B 也能用。这叫 generalizability——研究结果能不能推广到其他情境。

研究把 AI 评分和 human examiner marks 比较。这里的 agreement 意思只是：两者给出了同一个等级。它不等于哲学意义上的"真人一定绝对正确"。这项研究采用真人正式评分作为现实中的 reference standard，来测试 AI 能否符合现有大学评分实践。所以更准确的表达是：AI matched human-assigned bands 35%—63% of the time. 而不是：AI was objectively correct 35%—63% of the time.

听起来：几个 AI 都给差不多的分，不是说明很可靠吗？不一定。如果所有系统都共享某种类似偏差，它们就可能 agree with one another while disagreeing with expert judgment。这就是为什么 reliability 和 validity 不是一回事——一个尺子每次都多量 5 厘米，它非常 reliable，但不 valid。

因为他们认为 AI 仍然可能适合：error detection、consistency checks、flagging unusual disagreements。也就是说：AI 可以当 a second pair of eyes，而不是 the final judge。例如：老师给 75，AI 给 52。这个巨大差异可以提醒：Maybe this essay deserves another human review. 这是 supporting human judgment，而不是 replacing human judgment。