Suppose we want to score contributions of information to a platform and thus promote effort amongst participants, but have no easy way to verify the quality of information. Responses may be subjective, or simply too costly to verify; e.g., what emotion do you feel watching video content, is a restaurant good for a family, what grade does a student deserve for an essay, should a paper be accepted to a conference, is a social media story real or fake? Peer prediction (Miller, Resnick and Zeckhauser 2005) seeks to promote effort by scoring contributions based on how predictive they are of contributions by others. But the challenge has been to score contributions without promoting unintended “group think” equilibria or coordinated misreports. In this talk, the correlated agreement mechanism will be described, which uses reports on multiple questions to provide a remarkably robust method of peer prediction. Its properties will be demonstrated on statistical models derived from reports on places in a city and peer-evaluation on a MOOC platform. Time permitting, it will also be explained how it can be coupled with methods from machine learning to handle the heterogeneity of participants.

Joint work with Arpit Agarwal (U Penn), Rafael Frongillo (CU Boulder), Matthew Leifer (Harvard), Debmalya Mandal (Harvard), Galen Pickard (Google), Nisarg Shah (Harvard), and Victor Shnayder (Harvard).

Bio:

David C. Parkes is George F. Colony Professor of Computer Science, Harvard College Professor, and Area Dean for Computer Science at Harvard University, where he founded the EconCS research group and leads research with a focus on market design, artificial intelligence, and machine learning. He received his Ph.D. in Computer and Information Science from U. Penn., and an M. Eng. (First class) in Eng. and Comp. Sci. from U. Oxford. He teaches in Applied Mathematics and Computer Science, including courses on machine learning, artificial intelligence, econ/cs and optimization.

In addition to his research and teaching, Parkes is co-director of the Harvard University Data Science Initiative and a faculty lead for the Paulson School of Engineering’s new Allston campus. A Fellow of the Association for the Advancement of Artificial Intelligence (AAAI) and recipient of the 2017 ACM/SIGAI Autonomous Agents Research Award, Parkes was awarded the NSF Career Award, the Alfred P. Sloan Fellowship, the Thouron Scholarship, and the Roslyn Abramson Award for Teaching. He was a member of the inaugural panel of the Stanford 100 Year Study on Artificial Intelligence (AI100) and co-organized the 2016 OSTP/AAAI/CCC Workshop on AI for Social Good. He served as chair of the ACM Special Interest Group on electronic commerce, is an editor for Games and Economic Behavior and the Journal of Artificial Intelligence Research. Parkes also serves on several international scientific advisory boards.