AI in Science: New doubts over math prize and data integrity

OpenAI announced last month that its artificial intelligence agents solved a major unsolved problem in mathematics. The company stated that the system cracked the Navier-Stokes equations in 88 hours. The Clay Mathematics Institute, which oversees the prize for the solution, said the problem had apparently been settled but that a full review would take time.

The announcement sparked immediate controversy among mathematicians. Tristan Buckmaster of New York University raised concerns hours before OpenAI made the public statement. Buckmaster formed an independent team with Levent Alpoğce, a mathematician at Anthropic. The pair had been working on the same problem and stored their drafts in OpenAI's coding tool, Codex. Buckmaster said his co-researcher was tipped off that news of their progress had reached OpenAI. He asked whether their research, stored in Codex, could have fed into OpenAI's own work. He stressed he was not accusing anyone of anything.

OpenAI initially stated that nobody at the company, human or AI, saw the other teams' work before publication. The firm said it could not rule out that anonymised data from product use may have helped improve its models. Days later, the company revised its position. It stated that Buckmaster's Codex prompts from the two months before the announcement could not have influenced the system in any way.

The dispute is part of a wider debate about the role of AI in scientific research. Dr Oliver Vince of Basecamp Research collects DNA data to create an AI that can interpret biological language. He believes AI will supercharge science. However, other experts warn that the technology may undermine trust in the scientific process. They fear it will become difficult to separate true discovery from AI-generated fiction.

Dalmeet Singh Chawla, a science journalist, says generative AI has disrupted the scientific process. He notes that publishers' integrity staff need to drastically increase in number and effort. He added that right now, they seem to be not up to the fight. In 2020, Google DeepMind's AlphaFold model predicted 200 million protein folds in one year. The equivalent experimental work would have taken hundreds of millions of years. The firm gave the results away for free. The head of DeepMind and the project lead received the Nobel Prize for Chemistry.

DeepMind later claimed its models identified more than two million new crystal structures. It described this as nearly 800 years worth of knowledge released in one go. About 380,000 were considered promising enough for laboratory synthesis. Chemists Anthony Cheetham and Ram Seshadri at the University of California reviewed a random sample. They wrote in Chemistry of Materials that they had yet to find any strikingly novel compounds. They argued many were trivial adaptations of known materials. DeepMind rejected the charge, saying it tested each compound for novelty. It argued that swapping a single atom is how many useful materials arrived.

Researchers also worry about the quality of data in academic journals. Prof Dmitry Kobak of Ghent University found that by December last year, nearly 90% of biomedical papers had AI fingerprints on them. He told Nature that whatever bias the LLM may have will suddenly permeate the scientific literature. A team at Stanford found that up to 17% of referee text for computer science conferences was substantially modified by a language model. Researchers at Cornell found that AI tools were inventing bogus research, with around 147,000 pieces of phantom research cited last year.

Elsevier, a major publisher, has developed about 100 signals to flag papers for closer human scrutiny. Prof Sarah Main, an Elsevier vice-president, says research-grade AI should ensure the researcher is in the driving seat. Dr Pushmeet Kohli of DeepMind says turning backs on AI would be like computing without calculators. He accepts there should be human guardrails. His test is that a researcher must understand what the agent has done and take responsibility. One cannot just specify the problem and let the machine write the paper and submit it as true.