By Karen Rouse
CITP faculty member Arvind Narayanan told his co-organizers that he would consider the reproducibility workshop they were planning to be a success if 30 people attended. By the time the July 28 online event went live, more than 1,700 people from 500 institutions and 30 countries had registered for The Reproducibility Crisis in ML-based Science workshop. To the nearly 600 watching via Zoom and YouTube combined, Narayanan, a professor of computer science, declared, “this is a surreal moment for me.”
Two weeks later, Wired magazine profiled the work of Narayanan and lead workshop organizer Sayash Kapoor, a CITP Graduate Student, in an August 10 feature story: Sloppy Use of Machine Learning Is Causing a ‘Reproducibility Crisis’ in Science. Clearly, the pair struck a chord with researchers who, like them, have been wrestling with reproducibility failures in machine learning research and other quantitative sciences.
Their paper, Leakage and the Reproducibility Crisis in ML-based Science, was the impetus for the workshop. In it, Kapoor and Narayanan discuss 20 papers from a variety of fields – including genomics, medicine, software engineering, radiology, satellite imaging, nutrition research, and autism diagnostics – that revealed erroneous or faulty conclusions in at least 329 other papers. Kapoor and Narayanan also propose solutions, including designing a rubric researchers can use for applying machine learning and minimize data leakage.
The workshop, hosted by Princeton’s Center for Statistics and Machine Learning, featured presentations from 10 speakers and addressed various factors in what has been dubbed a “reproducibility crisis” — from data leakage to the pressure researchers feel to churn out papers that produce positive results. In addition to Kapoor and Narayanan, the organizers included researchers Priyanka Nanayakkara, a graduate student at Northwestern University; Kenny Peng, an incoming graduate student at Cornell University, and Hien Pham, a Princeton undergraduate.
To access speaker and panel slides and the annotated reading list, please visit The Reproducibility Crisis in ML-based Science workshop website. You may watch the full workshop recording via this video link.