Exploring Overreliance on WEIRD Samples in AI Research: Calls for Inclusivity and Transparency
A study reveals alarming disparity in data collection practices and highlights the need for diverse perspectives in AI research
At this week’s ACM FAccT conference, Nokia Bell Labs researchers will present a study that has exposed a potential blind spot in the samples studied in the same conference over the years —the overreliance on WEIRD samples. Nokia Bell Labs, committed to a responsible development of AI systems, sheds light on this critical issue and calls for more inclusivity and transparency in AI research.
WEIRD, an acronym for Western, Educated, Industrialized, Rich, and Democratic populations, represents a subset of individuals that may not accurately represent the diverse global population. The study, which analyzed 128 papers published between 2018 and 2022 at FAccT, revealed an alarming trend. 84% of these papers solely relied on Western participants, with a predominant focus on the United States (63%). Inclusive perspectives were only introduced when researchers made an extra effort to collect data from local participants through interviews or surveys.
To address this issue, the researchers propose a fundamental shift in the collection of data for AI research. By actively seeking participation from under-represented populations, researchers can obtain a more comprehensive and inclusive worldview. This approach will not only lead to a more accurate understanding of AI's impact on society but also pave the way for fairer and more accountable AI systems.
The study emphasizes the critical role that scientific communities play in championing this cause. Researchers, practitioners, and policymakers must unite to encourage transparent reporting of data biases, enabling a more nuanced understanding of research limitations. By doing so, the AI research community can foster an inclusive environment that values diversity and ensures that research findings resonate with a global audience.
In a compelling comparison with CHI, the leading conference on human factors in computing, the study revealed significant disparities. FAccT papers drew less from Educated, Industrialized, Rich, and Democratic samples but relied heavily on Western participants.
The researchers urge the AI research community to reflect upon these findings and take proactive steps towards inclusivity and transparency. Collecting data from diverse populations will enable a more accurate understanding of the challenges and opportunities that arise from AI systems. By transparently reporting biases, researchers can build public trust and confidence in AI technologies. "It is time to embrace inclusivity and champion diversity in AI research for a fairer and more accountable future," states the Nokia Bell Labs scientists. "We have an opportunity and responsibility to ensure that the technologies we develop reflect the needs and values of a global society."