Simultaneous EEG and pupillometry as indicators of listening effort for enhanced speech in adverse conditions

Understanding speech in noisy conditions requires additional cognitive processing or listening effort compared to quite environments. AdaptDRC is a near-end-listening enhancement algorithm developed to improve intelligibility (Schepker et al., 2013, Interspeech 3577) by altering the frequency content and dynamic range of a speech signal, dependent on the environmental noise. As more advanced speech enhancement technologies make their way into public use, quantifying their effects on listening effort has become an important area of research.

There are many methods for capturing changes in listening effort, each with their own theoretical bases and physiological underpinnings. Ongoing debate considers their respective methodological advantages and limitations. By measuring subjective ratings, pupil size, and EEG data concurrently, we may investigate how they relate to each other.

Native English speakers with normal hearing aged 18-30 listened to English OLSA sentences presented in cafeteria noise at -5, 0, +5 dB SNR, in which the speech is unprocessed or processed with AdaptDRC. Participants rated the subjective effort on an adapted version of the adaptive categorical listening effort scale (Krueger et al., 2017, JASA 141:4680). EEG data were collected using a 24-channel EEG system. Pupil size was recorded using the Eyelink 1000 plus. Further, participants each completed a short cognitive task battery including measures of working memory and inhibition.

In this poster we present preliminary data showing the relative sensitivity of subjective ratings, pupil size and EEG alpha power, and how they relate to each other for measuring changes in listening effort for speech in noise. All methods are anticipated to indicate changes in effort, as manipulated by SNR and speech enhancement, but the shapes of the responses may vary. We also examine the relationship between the identified markers of effort and individual differences in cognitive abilities.