Challenge has ended. Full results for this task can be found here

Description

The goal of acoustic scene classification is to classify a test recording into one of predefined classes that characterizes the environment in which it was recorded — for example "park", "home", "office".

Audio dataset

TUT Acoustic scenes 2016 dataset will be used for the task. The dataset consists of recordings from various acoustic scenes, all having distinct recording locations. For each recording location, 3-5 minute long audio recording was captured. The original recordings were then split into 30-second segments for the challenge.

Acoustic scenes for the task (15):

• Bus - traveling by bus in the city (vehicle)
• Cafe / Restaurant - small cafe/restaurant (indoor)
• Car - driving or traveling as a passenger, in the city (vehicle)
• City center (outdoor)
• Forest path (outdoor)
• Grocery store - medium size grocery store (indoor)
• Home (indoor)
• Lakeside beach (outdoor)
• Library (indoor)
• Metro station (indoor)
• Office - multiple persons, typical work day (indoor)
• Residential area (outdoor)
• Train (traveling, vehicle)
• Tram (traveling, vehicle)
• Urban park (outdoor)

Detailed description of acoustic scenes included in the dataset can be found here.

The dataset was collected in Finland by Tampere University of Technology between 06/2015 - 01/2016. The data collection has received funding from the European Research Council.

Recording and annotation procedure

For all acoustic scenes, the recordings were captured each in a different location: different streets, different parks, different homes. Recordings were made using a Soundman OKM II Klassik/studio A3, electret binaural microphone and a Roland Edirol R-09 wave recorder using 44.1 kHz sampling rate and 24 bit resolution. The microphones are specifically made to look like headphones, being worn in the ears. As an effect of this, the recorded audio is very similar to the sound that reaches the human auditory system of the person wearing the equipment.

Postprocessing of the recorded data involves aspects related to privacy of recorded individuals, and possible errors in the recording process. For audio material recorded in private places, written consent was obtained from all people involved. Material recorded in public places does not require such consent, but was screened for content, and privacy infringing segments were eliminated. Microphone failure and audio distortions were also annotated and segments containing such errors were also eliminated.

After eliminating the problematic segments, the remaining audio material was cut into segments of 30 seconds length.

Challenge setup

TUT Acoustic scenes 2016 dataset consist of two subsets: development dataset and evaluation dataset. The partitioning of the data into the subsets was done based on the location of the original recordings. All segments obtained from the same original recording were included into a single subset - either development dataset or evaluation dataset. For each acoustic scene, 78 segments (39 minutes of audio) were included in the development dataset and 26 segments (13 minutes of audio) were kept for evaluation. Development set contains in total 9h 45mins of audio, and evaluation set 3h 15mins.

Participants are not allowed to use external data for system development. Manipulation of provided data is allowed.

Download

** Development dataset **

TUT Acoustic scenes 2016, development dataset (7.5 GB)

** Evaluation dataset **

TUT Acoustic scenes 2016, evaluation dataset (2.5 GB)

In publications using the datasets, cite as:

@inproceedings{Mesaros2016_EUSIPCO,
    author = "Mesaros, Annamaria and Heittola, Toni and Virtanen, Tuomas",
    title = "{TUT} Database for Acoustic Scene Classification and Sound Event Detection",
    abstract = "We introduce TUT Acoustic Scenes 2016 database for environmental sound research, consisting ofbinaural recordings from 15 different acoustic environments. A subset of this database, called TUT Sound Events 2016, contains annotations for individual sound events, specifically created for sound event detection. TUT Sound Events 2016 consists of residential area and home environments, and is manually annotated to mark onset, offset and label of sound events. In this paper we present the recording and annotation procedure, the database content, a recommended cross-validation setup and performance of supervised acoustic scene classification system and event detection baseline system using mel frequency cepstral coefficients and Gaussian mixture models. The database is publicly released to provide support for algorithm development and common ground for comparison of different techniques.",
    year = "2016",
    address = "Budapest, Hungary",
    booktitle = "24th European Signal Processing Conference 2016 (EUSIPCO 2016)"
}

Cross-validation with development dataset

A cross-validation setup is provided for the development dataset in order to make results reported with this dataset uniform. The setup consists of four folds distributing the 78 available segments based on location. The folds are provided with the dataset in the directory evaluation setup.

If not using the provided cross-validation setup, pay attention to the segments extracted from same original recordings. Make sure that all files recorded in same location are placed on the same side of the evaluation.

Submission

Detailed information for the challenge submission can found from submission page.

One should submit single text-file (in CSV format) containing classification result for each audio file in the evaluation set. Result items can be in any order. Format:

[filename (string)][tab][scene label (string)]

Task rules

• Only the provided development dataset can be used to train the submitted system.
• The development dataset can be augmented only by mixing data sampled from a pdf; use of real recordings is forbidden.
• The evaluation dataset cannot be used to train the submitted system; the use of statistics about the evaluation dataset in the decision making is also forbidden.
• Technical report with sufficient description of the system has to be submitted along with the system outputs.

More information on submission process.

Evaluation

The scoring of acoustic scene classification will be based on classification accuracy: the number of correctly classified segments among the total number of segments. Each segment is considered an independent test sample.

Code for evaluation is available with the baseline system:

• Python implementation from src.evaluation import DCASE2016_SceneClassification_Metrics.
• Matlab implementation, use class src/evaluation/DCASE2016_SceneClassification_Metrics.m.

Results

Complete results and technical reports can be found at Task 1 result page

Baseline system

The baseline system for the task is provided. The system is meant to implement basic approach for acoustic scene classification, and provide some comparison point for the participants while developing their systems. The baseline systems for task 1 and task 3 share the code base, and implements quite similar approach for both tasks. The baseline system will download the needed datasets and produces the results below when ran with the default parameters.

The baseline system is based on MFCC acoustic features and GMM classifier. The acoustic features include MFCC static coefficients (0th coefficient included), delta coefficients and acceleration coefficients. The system learns one acoustic model per acoustic scene class, and does the classification with maximum likelihood classification scheme.

The baseline system provides also reference implementation of evaluation metric. Baseline systems are provided for both Python and Matlab. Python implementation is regarded as the main implementation.

Participants are allowed to build their system on top of the given baseline systems. The systems have all needed functionality for dataset handling, storing / accessing features and models, and evaluating the results, making the adaptation for one's needs rather easy. The baseline systems are also good starting point for entry level researchers.

In publications using the baseline, cite as:

@inproceedings{Mesaros2016_EUSIPCO,
    author = "Mesaros, Annamaria and Heittola, Toni and Virtanen, Tuomas",
    title = "{TUT} Database for Acoustic Scene Classification and Sound Event Detection",
    abstract = "We introduce TUT Acoustic Scenes 2016 database for environmental sound research, consisting ofbinaural recordings from 15 different acoustic environments. A subset of this database, called TUT Sound Events 2016, contains annotations for individual sound events, specifically created for sound event detection. TUT Sound Events 2016 consists of residential area and home environments, and is manually annotated to mark onset, offset and label of sound events. In this paper we present the recording and annotation procedure, the database content, a recommended cross-validation setup and performance of supervised acoustic scene classification system and event detection baseline system using mel frequency cepstral coefficients and Gaussian mixture models. The database is publicly released to provide support for algorithm development and common ground for comparison of different techniques.",
    year = "2016",
    address = "Budapest, Hungary",
    booktitle = "24th European Signal Processing Conference 2016 (EUSIPCO 2016)"
}

Python implementation

Matlab implementation

Results for TUT Acoustic scenes 2016, development set

Evaluation setup

• 4-fold cross-validation, average classification accuracy over folds
• 15 acoustic scene classes
• Classification unit: one file (30 seconds of audio).

System parameters

• Frame size: 40 ms (with 50% hop size)
• Number of Gaussians per acoustic scene class model: 16
• Feature vector: 20 MFCC static coefficients (including 0th) + 20 delta MFCC coefficients + 20 acceleration MFCC coefficients = 60 values
• Trained and tested on full audio
• Python implementation

Citation

If you are using the dataset or baseline code please cite the following paper:

@inproceedings{Mesaros2016_EUSIPCO,
    author = "Mesaros, Annamaria and Heittola, Toni and Virtanen, Tuomas",
    title = "{TUT} Database for Acoustic Scene Classification and Sound Event Detection",
    abstract = "We introduce TUT Acoustic Scenes 2016 database for environmental sound research, consisting ofbinaural recordings from 15 different acoustic environments. A subset of this database, called TUT Sound Events 2016, contains annotations for individual sound events, specifically created for sound event detection. TUT Sound Events 2016 consists of residential area and home environments, and is manually annotated to mark onset, offset and label of sound events. In this paper we present the recording and annotation procedure, the database content, a recommended cross-validation setup and performance of supervised acoustic scene classification system and event detection baseline system using mel frequency cepstral coefficients and Gaussian mixture models. The database is publicly released to provide support for algorithm development and common ground for comparison of different techniques.",
    year = "2016",
    address = "Budapest, Hungary",
    booktitle = "24th European Signal Processing Conference 2016 (EUSIPCO 2016)"
}

When citing challenge task and results please cite the following paper:

@article{Mesaros2018_TASLP,
    author = "Mesaros, A. and Heittola, T. and Benetos, E. and Foster, P. and Lagrange, M. and Virtanen, T. and Plumbley, M. D.",
    journal = "IEEE/ACM Transactions on Audio, Speech, and Language Processing",
    title = "Detection and Classification of Acoustic Scenes and Events: Outcome of the {DCASE} 2016 Challenge",
    year = "2018",
    volume = "26",
    number = "2",
    pages = "379--393",
    abstract = "Public evaluation campaigns and datasets promote active development in target research areas, allowing direct comparison of algorithms. The second edition of the challenge on detection and classification of acoustic scenes and events (DCASE 2016) has offered such an opportunity for development of the state-of-the-art methods, and succeeded in drawing together a large number of participants from academic and industrial backgrounds. In this paper, we report on the tasks and outcomes of the DCASE 2016 challenge. The challenge comprised four tasks: acoustic scene classification, sound event detection in synthetic audio, sound event detection in real-life audio, and domestic audio tagging. We present each task in detail and analyze the submitted systems in terms of design and performance. We observe the emergence of deep learning as the most popular classification method, replacing the traditional approaches based on Gaussian mixture models and support vector machines. By contrast, feature representations have not changed substantially throughout the years, as mel frequency-based representations predominate in all tasks. The datasets created for and used in DCASE 2016 are publicly available and are a valuable resource for further research.",
    keywords = "Acoustics;Event detection;Hidden Markov models;Speech;Speech processing;Tagging;Acoustic scene classification;audio datasets;pattern recognition;sound event detection",
    doi = "10.1109/TASLP.2017.2778423",
    issn = "2329-9290",
    month = "Feb"
}

Follow up paper about assessing the human and machine performance for acoustic scene classification task:

@inproceedings{Mesaros2017_WASPAA,
    author = "Mesaros, Annamaria and Heittola, Toni and Virtanen, Tuomas",
    title = "Assessment of Human and Machine Performance in Acoustic Scene Classification: {DCASE} 2016 Case Study",
    year = "2017",
    pages = "319--323",
    booktitle = "2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)",
    publisher = "IEEE Computer Society",
    language = "English",
    abstract = "Human and machine performance in acoustic scene classification is examined through a parallel experiment using TUT Acoustic Scenes 2016 dataset. The machine learning perspective is presented based on the systems submitted for the 2016 challenge on Detection and Classification of Acoustic Scenes and Events. The human performance, assessed through a listening experiment, was found to be significantly lower than machine performance. Test subjects exhibited different behavior throughout the experiment, leading to significant differences in performance between groups of subjects. An expert listener trained for the task obtained similar accuracy to the average of submitted systems, comparable also to previous studies of human abilities in recognizing everyday acoustic scenes.",
    doi = "10.1109/WASPAA.2017.8170047"
}