Data on neurophysiological and psychological responses to audio immersive experience in stereo and 3D audio formats

Abstract

Objectives

This dataset presents demographic, psychological, auditory and neurophysiological information of 31 volunteers, who participated in an experiment measuring the auditory immersive experience in two audio formats: stereophonic downmix and three-dimensional audio. This dataset could help understand the objectiveness (based on the nervous system response) behind the subjectiveness of immersion brought about by the audio format (based on the listener evaluation). The final objective of this dataset is to study the psychological and neurophysiological responses of immersive attributes in auditory events in future studies.

Data description

Participants were randomly assigned to an audio format and attended two listening sessions, where soundscapes of a concert hall and urban park were randomly played. Electroencephalography, electrodermal activity, and blood volume pressure data were collected during auditory stimulation, and psychological responses to the auditory immersive experience were collected at the end of the session.

Clinical trial number

Not applicable.

Objective

The concept of immersion has been a compelling topic in the field of extended reality without reaching a consensus among researchers. Immersive audio has been used interchangeably with three-dimensional (3D) audio, assuming that a high-fidelity spatial audio system causes an immersive experience. However, this assumption lacks evidence [1].

Nervous system responses could help researchers objectively understand the attributes of the immersive experience [2]. In electroencephalography (EEG) data, research on the sense of presence has been associated with neural oscillations within clinical bands such as theta and beta [3, 4]. For blood volume pressure (BVP), heart rate variability, and its frequency composition have been related to involvement [5, 6]. For electrodermal activity (EDA), descriptive features in skin conductance response and levels can be extracted to monitor behavior in emotional activation [5, 7].

Therefore, understanding auditory immersive experiences through behavioral questionnaires and neurophysiological responses motivated the data collection, enabling the extraction of neural features and the study of their association with immersive attributes. The audio recordings used are also included. Firstly, participants voluntarily registered for the experiment. Secondly, participants were selected based on exclusion criteria on auditory and neurological health. Thirdly, participants answered the Immersive Tendencies Questionnaire (ITQ) [8] to be assigned to stereophonic downmix (stereo) or 3D audio group. Lastly, participants attended two listening sessions, where the soundscape of a concert hall and an urban park were randomly played. During each listening session, EEG, BVP and EDA recordings of resting state and auditory stimulation were collected while participants were seated in the middle of a critical listening room with their eyes closed. After each soundscape, participants answered an immersive experience questionnaire.

Previous work using this dataset showed statistical significance between the mean heart rate variability and the immersive attribute of involvement [4], indicating that the volume of data may be suitable for further signal and statistical analyses.

Data description

Data corresponds to 31 subjects (stereo: 10 males, five females, mean ITQ = 49.2. 3D: 10 males, six females, mean ITQ = 48.3) aged 18–36 years, all university students. Participants were excluded if they: (1) were outside the age range to minimize the likelihood of presenting hearing loss [9], (2) reported auditory and neurological problems, and (3) reported never having been in a concert hall or an urban park. After registration, participants were randomly and blindly assigned to an audio format group (stereo or 3D) based on ITQ scores. Table 1 provides an overview of data files and data sets available in the repository.

DataSpecs excel file (Data file 1) contains six worksheets: (1) Demographic data: ID, audio format group, ITQ score, Sex, Age, Highest education level, Country of origin, and Handedness; (2) Biofiles: ID, audio format group, listening condition, biosignal, sample rate, number of events and total length in seconds; (3) ITQ: ID, audio format group, and responses to items of ITQ; (4) Responses: ID, listening condition, audio format group, responses to the Self-Assessment Manikin test [10], igroup Presence Questionnaire [11, 12], and system performance questionnaire designed by the authors based on the Wheel of Sound recommendation [13, 14]; (5) Items: questionnaires, item codes if available by the authors of the assessments, the question or statement of the item, type of scale, and minimum and maximum values; and (6) EEGchannels: electrode positions in Theta/Phi coordinates.

LoudspeakerInformation file (Data file 2) explains the alignment and positions of loudspeakers for stereo, PCMA-3D, and ESMA-3D audio playback. PCMA-3D and ESMA-3D are horizontally spaced and vertically coincident microphone techniques dedicated to music and sound field recordings, respectively [15].

RawData folder (Data set 1) has individual subfolders of participants with EEG, EDA, and BVP recordings in three conditions: (1) resting state (Bl), (2) concert hall (Music), and (3) urban park (Park) soundscapes. Neurophysiological data were collected with OpenViBE v3.3.1 and the following devices: (1) EEG: 24-channel electrode cap following the 10/20 international electrode system connected to a Smarting mbrain train amplifier at 500 Hz sample rate; (2) BVP and (3) EDA signals: Empatica E4 wristband connected via Lab Streaming Layer code [16] with 64 Hz and 4 Hz sample rates, respectively. Condition 8 code in GDF files indicates the start of audio playback and can be ignored for Bl files. Code 33031 refers to one second before the end of the experiment and is not indicated in Music and Park files since the recording was stopped a few seconds after the sound finished to avoid audio interruption.

LatencyAdjustment folder (Data set 2) mirrors Raw Data organization. Codes latencies were adjusted due to latencies caused by sound traveling through the air and digital delay. This process is explained in Data file 2.

AudioFiles folder (Data set 3) contains two subfolders: (1) Music: 2-minute-long WAV audio files of concert hall recordings [17]; and (2) Park: 2-minute-long WAV audio files of urban park recordings [18]. All files were generated at 48k Hz sample rate and 24-bit depth. These files are organized for the three loudspeaker configurations detailed in Data file 2. Max 8 software was used for audio playback.

Table 1 Overview of data files and data sets

Limitations

• The data of three participants in specific soundscapes is missing because they did not attend the last session: Music – ID 20 and 40, and Park - ID 25.

• In some neurophysiological recordings, signal discontinuities and noisy segments may be present due to momentary device disconnection or subject movements, respectively, modifying the actual length of the recording as indicated in Data file 1.

• Stereophonic versions were created from the downmix process of PCMA-3D and ESMA-3D recordings. As a result, future findings may be applicable only to these specific playback conditions.