The Science Prediction Universe study, led by UniPat AI and submitted to arXiv on Nov. 11, tested 13 agent configurations spanning 12 models on 100 public tasks and 16 private ones. On public tasks, the average Spearman rank correlation between scored configurations reached 0.52, versus 0.33 against laboratory measurements, and inter-agent agreement exceeded agreement with measurements on 81 of the 100 tasks. Pooling rankings improved scores by an average of 0.08 points, with a 95% confidence interval of 0.05 to 0.11, improving results on 73 public tasks. The authors note that averaging can preserve shared errors, illustrated by an oncology task where eleven configurations agreed at 0.9 while averaging just 0.27 against measurements. The study is limited by uneven coverage (Claude Opus 5.5 and Claude Fable 5.1 were scored on only 44 and 30 tasks, largely due to safety refusals), a small private set of 16 tasks and the absence of a human baseline.
Source: Read the original article
