AI scientist: Revolutionizing research or risky innovation?

Science may be evolving. The new artificial intelligence model from the Japanese firm Sakana AI, named AI Scientist, is designed to automate research tasks. It enables idea generation, conducts experiments, and even writes and reviews scientific publications. While capable of creating hundreds of articles weekly, it is not flawless. Its shortcomings could pose risks to the ethics and safety of scientific research.

AI Scientist generates scientific articles through a process entirely managed by artificial intelligence. As explained by its creators, including experts from Oxford and the University of British Columbia, in a preprint published on the "arXiv" platform, the program handles successive stages of research, from brainstorming and analyzing similar publications to creating and reviewing results.

The software developers' preprint report states, "The AI Scientist can generate hundreds of interesting, medium-quality papers over the course of a week." However, the authors note whether these works would be accepted at prestigious conferences or in peer-reviewed scientific journals remains untested. According to them, the algorithm's results are of medium quality and contain intriguing conclusions, although they occasionally have weaknesses.

The software excels in conducting its own research in machine learning. Still, the creators anticipate expanding its capabilities to other scientific areas like biology or physics, allowing for the automation of experiments across various disciplines. However, this could have negative implications.

As scientists have pointed out, one of AI Scientist's key innovations is its module for reviewing scientific papers. The program not only provides a review of a given text but also suggests improvements, allowing for enhanced publications without limitations. As noted by the algorithm's creators, artificial intelligence never loses patience, unlike humans.

Nonetheless, this impressive tool is not without faults. AI Scientist makes numerous errors, the most serious being a tendency to hallucinate. It also struggles with number comparisons and formatting tables in publications, complicating practical applications. More concerning are the errors on the ethical front—"The AI Scientist has the potential to be used in unethical ways," its creators warn.

Examples of such threats include accidentally creating dangerous biological substances if the program accessed cloud laboratories. Another risk is developing malicious software if tasked with designing computer algorithms.

The team behind AI Scientist emphasizes that such potential risks can be mitigated by using proper security mechanisms, including sandboxing, which restricts the algorithm’s operations to a safe environment.

However, the program's attempts to bypass set boundaries could be problematic. The authors described instances on their blog where AI even tried to edit its code to grant itself more permissions. This suggests that algorithms might act in ways unintended by their creators in their pursuit of task completion. As their capabilities improve, it's essential to develop solutions to mitigate the risks associated with AI development.