This week's theme is judgment. Two randomised trials, from Bocconi and Harvard, separate what an AI tool adds from what only teaching adds. Two Dartmouth professors explain how to keep assigning research papers, and Zhejiang University lets an AI agent take the first pass at lab reports, with a teacher's review as the last word. Simon Critchley argues that philosophy's real skill is deciding what is worth keeping. Kevin Buzzard asks whether mathematics was ever about understanding. And Andrej Karpathy predicts that more of our work will be reading, and judging, what machines produce.
Evidence
Most studies of AI and learning vary access to the tool and then measure the skill. A team at Bocconi varied both. In a pre-registered randomised trial, 1,053 first-year undergraduates in one introductory management course, across 13 classes, were assigned to causal-reasoning training, ChatGPT access, both, or neither. The training was a game of 12 questions with feedback. Everyone then had 45 minutes to write a 180-word recommendation on a marketing problem. ChatGPT raised the ratings from blind graders, and the training did not. But only trained students stated when their recommendations would fail, explained the mechanisms behind them, and produced ideas that differed from their classmates'. Students who got both never did worse than those who got one. In the authors' words, “GPT expands the volume of ideas and raises the evaluations; trained reasoning changes the character of the ideas.”
See Figure 1 in the VoxEU column: the effect of training and of ChatGPT on each of the eight outcomes.
The uncomfortable finding is about graders, not students. Standard ratings rewarded the ChatGPT answers and did not register the trained reasoning. The authors say so directly: “the unintended consequence of standardised evaluations is to discourage diversity of ideas and first-principles thinking.” A DKU rubric that scores coherence and polish now mostly measures access to a tool. A rubric that asks “when would this fail?” and “why does this work?” measures what a short exercise can teach. Treat this as a prompt to reread our rubrics, not as a verdict. It is one 180-word task in one course, and an LLM scored the reasoning outcomes.
Sources: “Training novices to think in the age of large language models” — Chiara Fumagalli & Alfonso Gambardella, VoxEU/CEPR, October 1, 2026 · “Training Novices to Think, or Giving Them LLMs? Evidence from an RCT” — Asirvatham et al., CEPR Discussion Paper 21882
Evidence
This 2025 paper is worth rereading beside the Bocconi trial in item 1. In what they call “Harvard's largest physics class,” Greg Kestin, Kelly Miller and colleagues ran a randomised crossover trial (N = 194). Each student learned one lesson through in-class active learning and another at home with a custom GPT-4 tutor, using the same content and worksheet. The AI group's median post-test score was 4.5, against 3.5 for the in-class group, and their median time on task was 49 minutes. Students also reported feeling more engaged and more motivated. The tutor was not a plain chatbot. “AI chatbots are generally designed to be helpful, not to promote learning,” the authors write. Theirs was given instructor-written, step-by-step solutions and led students through each part of a problem in order. The authors advise against using AI to replace in-class instruction.
See Figure 1 in the paper: post-test scores for the two groups against the pre-test baseline.
Both trials say the same thing: what helps is the teaching design around the model, not the model. Harvard's tutor depended on solutions the instructors had already written. The authors' own proposal suits DKU's small sections. Use a structured tutor for first exposure before class, then spend class time on higher-order work, because “Instructors can assess these skills in person.” The limits are plain. The trial covers two lessons in one course, measured by a quiz straight after each lesson, and the authors built the tool they tested. They “do not presume” the result holds everywhere.
Assessment
“The great AI course-design panic has begun,” write Dartmouth government professors Brendan Nyhan and Jeffrey Friedman in the Chronicle. Blue books and oral exams are sensible, they say, but abandoning papers is not: “Student writing is too important to their learning for us to give it up.” They propose three safeguards. First, students draft in a Google Doc shared with the instructor, so the time-stamped version history shows whether text was composed or pasted in. Second, a “day two” assessment, an oral or written exam on the student's own paper that counts for a substantial share of the grade. Third, AI-detection software, namely Pangram. They concede that “No method can definitively prove students used AI to write their papers,” and say the three work best “applied jointly using human judgment.” Their policy fits in a sentence: “Complete all work in the shared Google Doc, submit only writing that is your own, and know your work well enough to answer questions about it.”
Signature Work is the largest research paper a DKU student writes, and this is a workable way to protect it. Two of the three safeguards are cheap and need no vendor: a drafting record and a conversation. Any editor with version history will do. The “day two” exam echoes UChicago Law's oral requirement (Issue #0) and David Deming's final step of presenting the work in person (Issue #11). The third safeguard needs more care. The accuracy figures the authors cite for Pangram come from research on an earlier version and from the company's own report, so a flag should open a conversation, not close one. In a seven-week session, the simplest route is theirs: put the “day two” questions on the final exam.
Practice
On September 28, Zhejiang University's Undergraduate School held the fifteenth event in its series for grassroots teaching units, this one on AI in lab courses. More than 50 representatives of lab-teaching groups attended, with teachers from Zhejiang Normal University. The main demonstration was a lab-report grading agent (实验报告批改智能体), presented by Zhang Xue of the university's Information Technology Center. Its workflow has four steps: the teacher prepares, students complete the experiment, AI assists with grading, the teacher reviews (教师准备—学生完成实验—AI辅助批改—教师复核). Two instructors who helped build it showed it at work in an engineering fluids lab and an applied optics lab, and shared what they had learned about refining scoring rules. In discussion, teachers said they must hold on to professional judgment and guide students to use AI as a learning aid, not as “cognitive outsourcing” (认知外包).
Both courses on show are STEM labs, and that is probably no accident. A lab report has a fixed structure and a scoring rule that can be written down, so a teacher can check the machine's marks quickly. The humanities are moving the other way. UChicago's social sciences Core memo (Issue #7) says AI-assisted grading “has no place” there unless faculty are “carefully assessing its validity against human grading.” For DKU's natural science labs, Zhejiang's four-step loop is a reasonable shape for a pilot, with teacher review as the fixed last step. The report gives no data on grading accuracy or on how students responded. Those are the first two things to ask before trying it.
Source: 《AI赋能，让实验课“活”起来——第十五期基层教学组织系列活动顺利举行》 — 浙江大学本科生院 (WeChat), September 30, 2026 · UChicago memo as quoted in The Chicago Maroon
Ideas
Simon Critchley teaches philosophy at the New School for Social Research. His New York Times essay starts from an odd fact: AI companies now hire philosophers. Anthropic keeps a small group that includes Amanda Askell, and Google DeepMind appointed the Cambridge philosopher Henry Shevlin to a post titled “Philosopher.” Critchley's explanation is that machines are good at retrieval and summary and poor at judgment, and “Judgment is not calculation.” What a machine should say to someone grieving is not an engineering question. “Ethics, it turns out, cannot be scraped from the internet, because the internet is precisely the contemporary record of our failure to agree about ethics.” He returns to Plato's Phaedrus, where writing is both remedy and poison for memory, and to Nietzsche on forgetting as a strength. “The machine remembers everything and therefore, oddly, nothing in particular.” Thinking, he concludes, “is not the accumulation of everything but the art of judicious, active forgetting.”
Critchley gives the humanities a job that survives automation: deciding “what is worth keeping.” That can be taught. A seminar does it whenever it asks students which three of 20 sources matter, and why. It also fits the Bocconi result in item 1, where the tool supplied polish and only training supplied a sense of when a claim fails. His phrase for what the companies are buying, “an ear for the difference between the true and the merely plausible,” is a fair one-line aim for a liberal arts education. The Phaedrus passage would make a short first-year reading on handing memory over to a technology. Note that this is an opinion essay, and the hiring examples are Critchley's own account.
Source: “The Id, the Ego and the Superintelligence” — Simon Critchley, The New York Times, September 27, 2026 (gift link) · spotted via @DLabaree on X
Mathematics
Kevin Buzzard is the number theorist behind the Xena Project blog on formalising mathematics in Lean. In a new post he sorts his colleagues' reactions to AI into the stages of grief: denial (the Association for Human Mathematics, whose members commit to not publishing AI-generated outputs), anger, bargaining, depression. He doubts the common defence that mathematics is about human understanding and not the theorem tally, a view tied to William Thurston's 1994 essay. Understanding, he argues, is nebulous even for experts. He gave “22 hours worth of lectures” on Fermat's Last Theorem this year and still knows little about parts of the proof. “I'm in it for the theorem tally,” he writes, adding, “I seem to be in a minority here.” He sees machines doing a good deal of problem solving and much less theory building or conjecture making. He expects them to reach a “natural boundary,” because “mathematics is infinite which beats exponential hands down.” His advice is to “let the machines loose, see what happens, and then begin the journey to where they have stopped.”
This is a real disagreement inside a discipline about what it is for: the understanding or the tally. Every discipline at DKU has its own version of that argument, and it is better held openly in a division meeting than settled by default. Buzzard's three-part description is also a useful test for course design. Ask of each assignment whether it trains problem solving, which machines increasingly do, or theory building and choosing the next question, where he sees far less evidence. His remark that “You cannot fool Lean” points the same way: checking is the scarce skill, not producing.
Source: “To grieve, or not to grieve?” — Kevin Buzzard, Xena Project blog, October 1, 2026 · spotted via @Thom_Wolf on X
Tools
Andrej Karpathy opened a post this week with a prediction: “We'll be spending a lot more time trying to understand the outputs of language models.” His tips run from plain to elaborate. Ask the model to explain something in ASD-STE100, “a controlled language specification originally developed for aerospace maintenance documentation.” Better, “ask your LLM to create a diagram.” Better still, ask for output “in HTML” and get an interactive web page. The format he is “most bullish on” is a custom explainer video on any topic: “This is actually starting to work!” His summary is that “a lot more of our work will rise up the abstractions into oversight and understanding,” and that it now makes sense to ask for “large, custom, discardable software artifacts” for a single explanation. The Chinese-language summary that brought the post to our inbox, by the engineer uudon, adds its own gloss: when AI produces faster than people can understand, asking it for more text only raises the bottleneck.
This is a practitioner's tip sheet, and it offers no evidence that any of these formats improves understanding. It is here because it names a skill our students already need: reading machine output quickly and critically. Two classroom uses follow. A controlled English with short sentences and a limited vocabulary may help multilingual readers get through dense explanations. And an assignment that has students generate a diagram of a concept and then find its errors turns the tip into a checking exercise. The caution comes from items 1 and 2: a clearer explanation is not the same thing as learning.
Sources: Andrej Karpathy (@karpathy) on X, October 2, 2026 · spotted via @uudonX on X
Of everything in this issue, Critchley's question is the one I want to keep probing: what is worth keeping? It matters because having is not the same as doing. A student can have every source, every summary and every tool, and still lack the capacity to do something with them. That capacity, and the judgment behind it, is what I think we are here to build.
— Yisu