What We Learned from the Peer Review Assistant Trial

‍Ann Michael‍

Chief Transformation Officer, AIP Publishing

‍

Back in April, I shared how the Purpose-Led Publishing coalition - AIP Publishing, the American Physical Society, and IOP Publishing - was piloting the Peer Review Assistant with Hum and GroundedAI. That pilot has since evolved into Alchemist Review.

The goal was simple: explore how AI might support, not replace, editors and reviewers by taking on some of the routine, time-consuming parts of peer review.

Now that the trial is complete, here’s what we learned.

‍

What Worked

Across all three publishers, editors agreed that tools like the Peer Review Assistant can make a real difference, especially when they focus on repetitive administrative work.

Citation checking, metadata validation, and ethics compliance stood out as areas where automation added genuine value. These tasks aren’t glamorous but reducing the time editors spend on this work allows them more space to focus on the science, which is exactly the point.

‍

What Still Needs Work

Not every feature landed perfectly. The manuscript digest didn’t always capture a paper’s nuance, and there were some latency issues - the full report sometimes took longer than ideal to generate. Addressing the latency issue is a top priority to allow a more effective fit into editorial workflows.

These challenges underscored an important truth: innovation in peer review isn’t just about speed or output - it’s about knowing where technology adds value, and where human expertise must lead.

‍

What’s Next

With the pilot now wrapped up, each member of the Purpose-Led Publishing coalition is taking forward the learnings in ways that best suit their needs. IOP Publishing will explore how Alchemist Review can support its in-house editorial teams, while AIP Publishing is considering broader applications across its editorial workflows. Meanwhile, APS is drawing on the valuable insights gained during the pilot to strengthen its internal approach to leveraging AI tools responsibly to help improve processes.

Tools like the Peer Review Assistant, now Alchemist Review, show what’s possible when publishers, editors, and technologists share a vision: a peer review process that’s efficient, rigorous, and deeply human.

We’re excited to keep learning.

‍