Research / Experiment Report

AI Experiment Report — November 1, 2026

Ben Tartaglia · Progress report · Data checked November 1, 2026

The Lab has moved beyond its first two checkpoint days. Four cohorts now contain 289 stored responses, including 241 collected since the October 2 report. That is useful progress, but the coverage is uneven: 35 observation slots are partial, and the review queue has grown. This update follows the evidence through both.

Question

How consistently do six AI providers follow instructions, handle assumptions, and produce useful answers when controlled prompts are repeated over time? The reporting question this month is narrower: what was collected, where is coverage incomplete, and how much review remains?

Method

The first cohort, started September 27, uses the bridge-expansion task described in the baseline report. Its supplied reference calculation remains 36 millimeters. Three additional cohorts began October 18, October 25, and November 1. They address verification of a hypothetical breaking-news claim, a constrained shift-allocation problem, and a launch forecast with explicit uncertainty. The forecasting task supplies assumptions for an exercise; it is not a report of business results.

Each cohort uses fresh sessions at minute 0, 5, 15, and 30 for checkpoint days 0, 1, 7, 14, 21, 30, 90, 180, and 365. The six participating providers remain OpenAI, Anthropic, Google, xAI, Mistral, and DeepSeek. Different cohort prompts must be analyzed separately before drawing any claim about behavior over time.

Updated observations

MeasureNovember 1 cumulative stateChange from October 2
Active cohorts4+3
Stored responses / successful generation attempts289 / 289+241 / +241
Fully completed observation slots21+13
Partial observation slots35+35
Pending future observation slots88+60; includes schedules for new cohorts
Overdue pending slots0Unchanged
Generation failures / timeouts / unavailable45 / 1 / 1+45 / +1 / +1

The 56 completed or partial slots expected 336 provider samples. They contain 289 responses, leaving 47 samples missing at this check. A partial slot is not an overdue pending slot: the absence of overdue work does not mean collection is complete. The most recent stored response arrived November 1 at 6:31 AM Eastern.

Cohort promptCompleted / partial slotsStored / expected samples in those slotsMissing samples
Bridge and Temperature — September 2720 / 4140 / 1444
Evidence Before Interpretation — October 181 / 1575 / 9621
Constraint Allocation — October 250 / 1259 / 7213
Calibrated Launch Forecast — November 10 / 415 / 249

The original bridge cohort has reached Day 30. Its four Day 30 slots are partial, with 20 of 24 expected responses. The first five checkpoint days—0, 1, 7, 14, and 21—have 120 stored samples in 20 fully completed slots.

ProviderRecorded model identifierCumulative responsesAdded since baseline
OpenAIgpt-6-sol5648
Anthropicclaude-sonnet-52820
Googlegemini-3.5-flash5648
xAIgrok-4.75547
Mistralmistral-small-26035547
DeepSeekdeepseek-v4-pro3931

Recorded non-success outcomes comprise 28 Anthropic attempts with code 400, 17 DeepSeek attempts with an empty-response code, one xAI timeout, and one Mistral unavailable response with code 503. These are collection outcomes; they do not rank answer quality or establish a root cause. Anthropic’s latest stored response is October 19. The model identifiers above describe the registry attached to stored responses, not independently verified provider release histories.

The every-minute runner schedule is active. Its five latest inspected scheduler invocations succeeded, ending at 8:08 AM Eastern on November 1. Scheduler success alone cannot establish downstream generation success; the response records and missing samples provide that separate evidence.

Change since the previous report

The snapshot grew from one cohort and 48 responses to four cohorts and 289 responses. New prompt domains broaden the investigation, while incomplete collection introduces a comparison problem: we no longer have equal coverage across providers. Any later analysis must account for which prompt, checkpoint, and model actually produced each sample.

The current cohorts each reference prompt version 2, and the stored rubric remains version 1. Two evaluator model identifiers appear: gpt-6-astra and claude-opus-5-5. The October 2 aggregate snapshot did not preserve prompt hashes or evaluator/model identifiers, so this report cannot establish whether those configurations changed within the reporting interval. This is a provenance limit to address, not evidence of stability.

Review status

There are 532 evaluation records: 382 completed, 143 failed, and seven marked as requiring human review. Compared with the baseline, that is 348 additional completed records, 143 additional failed records, and five additional review-required records. No human reviews are recorded.

These are evaluation-record counts, not distinct responses certified as correct. The records refer to 20 distinct source responses and 266 distinct comparison responses; those populations can overlap. Multiple evaluations can concern the same response. A completed automated evaluation does not represent Ben’s approval, and a failed record does not by itself establish that an answer was wrong.

Limitations

This report audits execution, coverage, and review metadata. It does not adjudicate all raw responses, validate the evaluator’s scores, or establish model rankings, correctness rates, or drift findings. Four prompt tasks still cannot represent broad capability. Uneven provider coverage, partial checkpoints, failed evaluation records, and outstanding human reviews constrain interpretation.

Oracle Table debates have a separate release and approval workflow. Their publication does not resolve this repeated-question study’s review flags, and their outputs are excluded from the counts here.

Leadership implications

The operating lesson is concrete: a healthy scheduler can coexist with incomplete evidence. A useful dashboard needs to show expected samples, collected samples, partial work, and review obligations alongside its green status indicators.

As a research system expands, the denominator matters more. “289 responses collected” is informative; “289 of 336 expected samples in due slots” makes the gap actionable. Before using these results to change an operational workflow, I would want the missing coverage understood and the flagged comparisons reviewed.

Next checkpoint and follow-up

The next stored observations are Day 1 for the November 1 cohort: November 2 at 6:00, 6:05, 6:15, and 6:30 AM Eastern. Follow-up priorities are to investigate the recorded collection failures, determine recovery treatment for the 47 missing samples, review the seven flagged evaluations, and check failed evaluation coverage. This reporting pass did not retry generation or alter review decisions.

The next report is December 1, 2026. It will compare against this dated snapshot and identify any windows with no new observations explicitly.