Jakub Pachocki, OpenAI’s chief scientist, proposes using log-perplexity as a common measure of model progress and as a trigger for safety monitoring. OpenAI published its internal pretraining scores: approximately 0.99 for GPT-3.5, 0.91 for GPT-4, below 0.8 for newer models and 0.76 for a mathematics model that likes me. Pachocki identifies the roughly 0.68 to 0.70 range as where transformative capabilities could emerge and recommends not crossing it until adequate monitoring frameworks are in place. The proposal builds on Anthropic’s framework for deliberately slowing frontier AI development, but its practical impact will depend on adoption of a shared dataset, transparency from other laboratories and a precise definition of adequate monitoring.
Source: Read the original article


