“The bioRxiv pilot is part of a broader trend of using LLMs to help researchers — and the public — keep afloat in the tsunami of scientific literature. The physics-focused preprint server arXiv uses AI to generate audio summaries of some papers, and publishers and funders are starting to roll out features that allow users to ‘talk to a paper’ through a chatbot….
Before rolling out the service, Sever and his colleagues evaluated several dozen summaries produced by the tool. Most were pretty good, he says, and some were even better than the abstracts scientists had written. Others included clear falsehoods. “We know there are going to be errors in these things,” Sever says….
If the pilot becomes a fully fledged service, bioRxiv might look at routinely involving authors in proofreading and approving the content, Sever says. For now, to minimize the consequences of errors, the pilot is not being rolled out to medRxiv, a sister medical-research preprint server run by Cold Spring Harbor Laboratory Press, the London-based publisher BMJ and Yale University in New Haven, Connecticut. MedRxiv studies typically have clinical relevance, and errors could guide patient behaviour. By limiting the pilot to bioRxiv, says Sever, “the consequences of being wrong are more that somebody might feel misled or misunderstand a fairly arcane study in cell biology”. …”