Loading Events

Assessing Boundaries of Generative AI in Multilingual and Multicultural Foundation Models

Overview

This talk delves into the intricacies of large language models by examining their semantics and reasoning capabilities in multilinguality and cultural understanding. We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we examine the brittleness of foundation models in the dimensions of semantics and multilinguality. Our investigations encompass both open-source and proprietary models, shedding light on their behaviors in classic Natural Language Processing tasks, reasoning, and cultural contexts. Notably, (1) Most models respond inconsistently to paraphrased instructions. (2) Exposure bias pervades, evident in both standard Natural Language Processing tasks and cultural understanding. (3) For questions rooted in factual, scientific, or common-sense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, many models intriguingly demonstrate inconsistent performance on such queries. (4) Models trained multilingually still lack ``balanced multilingual'' capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for in-depth investigations for multilingual and multicultural evaluations.

Learning Outcome

Course Outline

Who Should Attend

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Speakers

A*STAR Fellow, Program Head<br>Group Leader and Senior Principal Scientist, I2R (Institute for Infocomm Research)<br>Principal Investigator, CFAR (Centre for Frontier AI Research) Dr Nancy F. Chen

Dr Nancy F. Chen

A*STAR Fellow, Program Head
Group Leader and Senior Principal Scientist, I2R (Institute for Infocomm Research)
Principal Investigator, CFAR (Centre for Frontier AI Research)

Dr Nancy F. Chen is an A*STAR fellow, programme head, group leader, senior principal scientist at I2R (Institute for Infocomm Research) and principal investigator at CFAR (Centre for Frontier AI Research). Her expertise lies in generative AI in speech, language, and multimodal conversational technology. Dr. Chen consistently garners best paper awards for her AI research across diverse applications, exemplified by accolades at IEEE ICASSP 2011 (forensics), APSIPA 2016 (education), SIGDIAL 2021 (social media), MICCAI 2021 (neuroscience), and EMNLP 2023 (healthcare). Multilingual technology from her team has led to commercial spin-offs and is deployed at Singapore’s Ministry of Education to support home-based learning. Dr. Chen has supervised 100+ students/staff. She has won awards from USA National Institute of Health, IEEE, Microsoft, P&G, UNESCO, L’Oréal. She serves as IEEE SPS Distinguished Lecturer (2023-2024), Programme Chair of ICLR 2023, Board Member of ISCA (2021-2025), and is honoured as Singapore 100 Women in Tech (2021). Prior to A*STAR, she worked at MIT Lincoln Lab while pursuing a PhD at MIT and Harvard.

Details

Status


This event is now closed.

Share This Event