IDEAL Presents a Special Program on Data Economics

The Institute for Data, Econometrics, Algorithms, and Learning (IDEAL) aims to accelerate transformative advances in the theoretical foundations of data science through research and education programs on machine learning and optimization; high-dimensional data analysis and inference; and emerging topics including reliability, interpretability, privacy, and fairness.

IDEAL investigators hold backgrounds in computer science, economics, electrical engineering, law, mathematics, operations research, and statistics. To foster interdisciplinary, inter-institute collaborative research, IDEAL runs thematically focused “special programs.” The Fall 2022 IDEAL Special Program on Data Economics, which aimed to define novel research directions with a particular focus on eliciting information, concluded in early December. 

The special program was organized by Northwestern Engineering’s Jason Hartline, professor of computer science, and Zhaoran Wang, assistant professor of industrial engineering and management sciences and (by courtesy) computer science, in collaboration with Ian Kash (University of Illinois at Chicago), Nicolas Lambert (University of Southern California), Grant Schoenebeck (University of Michigan), Bo Waggoner (University of Colorado Boulder), and Haifeng Xu (University of Chicago).

“I very much liked this novel and exciting frontier of ‘data economics,’ which I currently view as an expansion of the celebrated field of information economics. This expansion looks to be more data-driven and more learning-centric, and thus naturally connects to AI models and computation,” Xu said. “I am very excited to be part of the organizing community and hope that, after 10 years, people will view this special quarter as an important event that kicks out an influential new field!"

The special program featured a weekly virtual institute, two graduate courses, and three workshops.

Virtual Institute

Modeled after the Fall 2021 Virtual Institute on Prophet Inequalities, the virtual institute used a meeting platform provided by Virtual Chair and powered by Gather.town to simulate an in-person experience. Controlling virtual avatars, participants joined weekly meetings, which included a technical seminar, research updates, discussion within working groups, and unscheduled “open door” time at virtual desks for spontaneous conversation with fellow participants.

The IDEAL virtual institute prioritized collaboration among the 15 participating faculty, postdoctoral researchers, and students across multiple institutions.

"I thought this special quarter was fantastic, the talks inspiring, and the workshop on site at Northwestern a real delight,” said Lambert. “The idea of having a virtual space where a group of people with common interests try to come once a week around a given time from everywhere in the country is just great. We can see each other and have casual discussions like we would if we were office neighbors, and the short talks in the middle set a pace and generate inspiration. And this doesn't happen when we organize conference calls, because we only set up these calls with a specific goal in mind, whereas a lot of research ideas and new papers come from casual chats."


The virtual institute commenced on September 30 with a discussion on open problems in the field, which informed subsequent research discussions that continued throughout the special program.

"I think this special quarter is great. I have learned a lot of new things. For example, I listened to several great talks in the workshop and learned multiple new research topics. In the open question session, I learned new techniques from other researchers, which can potentially solve the question I am interested in,” said Yuqing Kong, assistant professor, Center on Frontiers of Computing Studies, Peking University. “It has been difficult for me to talk with other institutes' researchers these days. The virtual institute provides a great platform where I can communicate with researchers who have the similar research interests but know many things that I do not know. Thus, I would like to thank all organizers of the special quarter."



Graduate Courses

In addition, two graduate courses were offered during the fall special program to build community, expand research networks, and identify important new directions for research.

Hartline led a course on data economics (COMP_SCI 497: Data Economics) at Northwestern and Xu taught a corresponding session at the University of Chicago. During the advanced topics seminar, students read and presented papers and conducted preliminary research on topics including valuing data, eliciting data, incentivizing data collection and sharing, adaptive data analysis, and game theory with data.


“This kind of seminar can give momentum to emerging topics of research like data economics by getting students up to speed on the background literature and starting them on research projects,” Hartline said. “Running this kind of seminar across several universities has additional benefits. When coteaching faculty collaborate on the reading list — ours was based on suggestions from the organizers of the special quarter and crowdsourced on social media — the class is able to consider a more thorough survey of the research landscape. When students from different universities participate in research discussions and joint projects, they build their network of colleagues and collaborators outside their own university. Deliberate cultivation of these research connections is often overlooked in traditional PhD student training.”

Annie Liang, assistant professor of economics at Northwestern’s Weinberg College of Arts and Sciences and (by courtesy) of computer science at Northwestern Engineering, instructed a course on the economics of information (ECON 414: Economics of Information). The course examined information imperfections and asymmetries in markets and organizations and the theory and application of mechanism design to markets and contracts. Students explored topics including modeling information, search, the value of information, games with incomplete information, adverse selection and moral hazard.

Workshops

The Special Program on Data Economics aimed to define novel research directions — in particular on eliciting information — and to examine the economic and social impact of the data revolution.

Elicitation Mechanisms in Practice Workshop

Organized by Hartline, Kash, and Schoenebeck, the Elicitation Mechanisms in Practice Workshop explored incentives for information procurement, which are integral to a wide range of applications including peer grading, peer review, prediction markets, crowd-sourcing, and conferring scientific credit.

“We learned from experts who have been building and studying practical elicitation mechanisms with the aim of identifying opportunities where theoretical studies could have significant impact,” Hartline said. “Two main application areas — peer grading of homework and peer review of scholarly work — are especially exciting opportunities where researchers can both develop the theory of elicitation and deploy it to make their classes and conferences better.”

Workshop speakers included:

• Raul Castro Fernandez (University of Chicago) – “Protecting Data Markets from Strategic Buyers”
• Kevin Leyton-Brown (University of British Columbia) – “Better Peer Review via AI”
• Schoenebeck – “Peer Prediction, Measurement Integrity, and the Pairing Mechanism”

• Nihar Shah (Carnegie Mellon University) – “Peer Review, Incentives, and Open Problems”

“The variety of research styles and perspectives, yet with tightly interconnected topics, made for a fantastic workshop,” Waggoner said. “And the virtual experience was flawless.”

“I particularly enjoyed the lively discussion of market design for data marketplaces based on a talk by Raul Castro Fernandez,” Kash said. “Designing a sustainable marketplace for data is a surprisingly challenging problem, with a lot of concrete challenges highlighted by Raul Castro Fernandez.”

Elicitation and Evaluation Workshop

The Elicitation and Evaluation Workshop — organized by Waggoner and Wang — invited theorists from fields including machine learning and economics-and-computation to discuss methods of evaluating information and predictions and to explore how each area approaches the problem of choosing or designing an evaluation method, such as a loss function or scoring rule.

Speakers presented on topics including a population mean based aggregation mechanism, estimating ground-truth quality rankings, peer loss functions, indirect elicitation, and upper and lower bounds on elicitation complexity.

Workshop speakers included:

• Raf Frongillo (University of Colorado Boulder) – “Elicitation in Economics and Machine Learning”
• Yang Liu (University of California- Santa Cruz) – “Learning from Noisy Labels without Knowing Noise Rates”
• Mallesh Pai (Rice University and Centre for Economic Policy Research) – “The Wisdom of the Crowd and Higher-Order Beliefs”
• Weijie Su (University of Pennsylvania- Wharton) – “A Truthful Owner-Assisted Scoring Mechanism Video”

“This was a great way for me to work with and learn about the research of people working on similar questions as me, across field boundaries,” Pai said. “The workshops were a great way to get deep into the latest research in a subarea, and the weekly virtual institute helped me start thinking about new questions.”

Challenges in Data Economics Workshop

In concert with continuing advances in artificial intelligence (AI) and computation, the data revolution of the past decade is changing the way organizations and markets operate, profoundly impacting economics and social behavior.

The final workshop of the IDEAL Special Program on Data Economics focused on the challenges of data economics. The event was organized by Lambert, Liang, and Xu.

 Guest speakers presented on topics including the stochastic modeling of user behavior with inconsistent and conflicting preferences, modeling of intermediated digital markets, regulating data usage and its impacts on innovation in digital markets, and differential privacy and privacy elasticity. 



The workshop presenters were:

• Alessandro Bonatti (MIT Sloan School of Management) – “Data, Competition, and Digital Platforms”
• Rachel Cummings (Columbia University) – “The Privacy Elasticity of Behavior: Conceptualization and Application”
• Jon Kleinberg (Cornell University) – “The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimization”
• Erik Madsen (New York University) – “Insider Imitation”

“It was fantastic to see this workshop bringing together leading researchers in computer science, economics and operations research to identify foundational challenges in data economics, and discussing approaches towards tackling them,” said IDEAL site director Aravindan Vijayaraghavan, associate professor of computer science and (by courtesy) industrial engineering and management sciences at Northwestern Engineering. “This workshop is an excellent example of the interdisciplinary discussion that we have tried to foster in the IDEAL workshops and special programs.”