The plan for the six-question pilot, updated 17 October 2026.

For each question there is a list of the positions people took between 1600 and 1950, each backed by at least one quoted passage. This list is the position key. Models answer the question, and each answer is checked against the key.

Two things are measured: what a model argues when asked for its own view, and whether it knows the period positions when asked to report them. A position that appears in reports but never in a model’s own answers is one the model knows but does not use.

The question bank has 40 candidate questions, each tied to sources from the start. Each is asked in two ways. Answers are open-ended and limited to 150 words.

Defend

{question} Give the answer you find most defensible and explain your reasons. State any qualifications you think matter. Use no more than 150 words.

Survey

{question} Describe the distinct arguments people have made about this question, including disagreements about the reasons. You need not endorse them. Use no more than 150 words.

The Survey prompt names no dates, because for a model trained on pre-1931 text, 1600–1950 is the present. It also lets a model report a position it would refuse to endorse. An earlier pilot asked models to choose between two given positions; that format is retired, because it allows no answer outside the options.

The first version of each key is built from the works selected for the project and frozen before any model answer is read. Arguments found in model answers that the key lacks are then searched for in the period sources. An argument is added only when a period source states it. Results are reported against both versions of the key.

The core comparison is between two base models of the same architecture and training compute: one trained on pre-1931 text, one on modern web text. An open instruction-tuned model and one deployed frontier model are run alongside them. See Models. Each model answers each prompt 20 times.

Claude reads each answer and records which positions from the list it uses, whether it endorses, describes or rejects each, any argument that fits no position, and whether it expresses hope or fear. Ben Breen and Nathan Davies code a sample of the same answers without seeing Claude’s coding. Claude’s coding is used only if it agrees with each of them about as often as they agree with each other.

The pilot keys use 117 supplied passages across six questions. Ben, Nathan, Opus 5.5 (Medium) and GPT-6.1 Sol (Medium) each create positions and sort the passages independently under the same instructions. Direction and argument grouping are measured separately. After all independent lists lock, only the human comparison opens. Ben and Nathan make and lock all six human reference lists before seeing any agent results. They then assess each agent’s category meanings against the references and preserve those references before making the final keys. Further use is decided separately for each agent, with historian checks.

Six questions:

Four models, two prompts, 20 answers each: 960 answers. Supplying a period passage with the question is planned for after the pilot.

  • What counts as the present-day default view.
  • The rules for admitting newspapers and magazines.
  • Which deployed frontier model to add, and which other modern base models.
  • How close the agreement must be before Claude’s coding is used.