AI just exposed a major weakness in anonymous peer reviews

Share

Anthropic’s Claude Fable 5.2 correctly identified the writer of 56% of 398 private peer-review reports and meta-reviews in a Nov. 19 preprint by Atrey Desai and colleagues. The study covered reports written between 2015 and 2026 for 26 double-blind machine learning and natural language processing conferences and workshops, with six volunteers supplying unsigned texts and the model seeing one report at a time without web access or a list of possible authors. Accuracy reached 68.4% on 266 standard reviews and 31.1% on 132 shorter meta-reviews drawn from just two contributors, with an average accuracy of 49.1% when each reviewer was weighted equally. On a separate set of 1,559 public reviews from 47 open-review journals, Claude named the reviewer in only 21.4% of the 1,530 cases it answered, and 39.3% on the 789-review subset labeled human or mixed by the Pangram 4.1 detector. Performance was far higher on reviews whose writers cited their own work (80.2% versus 17.1%), indicating that stylistic and topical cues remain informative despite anonymization. The researchers note that an editor or author would still need independent evidence to establish who wrote an unsigned review, and plan to add results from a much larger set of anonymous peer reviews in an update to the preprint.

Source: Read the original article

Disclaimer: this content is for information purposes only and is not financial advice. Cryptocurrencies are highly volatile: you may lose all of your capital. Always do your own research. Legal notice
Telemac
Telemachttp://cryptoinfo.ch
Passionné de nouvelles technologies, j’explore l’univers de la blockchain et des cryptomonnaies pour partager l’actualité et les innovations du secteur.

Read More

Items