Challenge has ended. Full results for this task can be found here

Description

The goal of acoustic scene classification is to classify a test recording into one of the provided predefined classes that characterizes the environment in which it was recorded — for example "park", "home", "office".

Audio dataset

TUT Acoustic Scenes 2017 dataset will be used as development data for the task. The dataset consists of recordings from various acoustic scenes, all having distinct recording locations. For each recording location, 3-5 minute long audio recording was captured. The original recordings were then split into segments with a length of 10 seconds. These audio segments are provided in individual files.

Acoustic scenes for the task (15):

• Bus - traveling by bus in the city (vehicle)
• Cafe / Restaurant - small cafe/restaurant (indoor)
• Car - driving or traveling as a passenger, in the city (vehicle)
• City center (outdoor)
• Forest path (outdoor)
• Grocery store - medium size grocery store (indoor)
• Home (indoor)
• Lakeside beach (outdoor)
• Library (indoor)
• Metro station (indoor)
• Office - multiple persons, typical work day (indoor)
• Residential area (outdoor)
• Train (traveling, vehicle)
• Tram (traveling, vehicle)
• Urban park (outdoor)

Detailed description of acoustic scenes included in the dataset can be found DCASE2016 Task1 page.

The dataset was collected in Finland by Tampere University of Technology between 06/2015 - 01/2017. The data collection has received funding from the European Research Council.

Recording and annotation procedure

For all acoustic scenes, the recordings were captured each in a different location: different streets, different parks, different homes. Recordings were made using a Soundman OKM II Klassik/studio A3, electret binaural microphone and a Roland Edirol R-09 wave recorder using 44.1 kHz sampling rate and 24 bit resolution. The microphones are specifically made to look like headphones, being worn in the ears. As an effect of this, the recorded audio is very similar to the sound that reaches the human auditory system of the person wearing the equipment.

Postprocessing of the recorded data involves aspects related to privacy of recorded individuals. For audio material recorded in private places, written consent was obtained from all people involved. Material recorded in public places does not require such consent, but was screened for content, and privacy infringing segments were eliminated. Microphone failure and audio distortions were annotated, and the annotations are provided with the data. Based on experiments in DCASE 2016, eliminating the error regions in training does not influence the final classification accuracy. The evaluation set does not contain any such audio errors.

Download

In case you are using the provided baseline system, there is no need to download the dataset as the system will automatically download needed dataset for you.

** Development dataset **

TUT Acoustic scenes 2017, development dataset (10.7 GB)

** Evaluation dataset **

TUT Acoustic scenes 2017, evaluation dataset (3.6 GB)

Task setup

TUT Acoustic Scenes 2017 dataset consist of two subsets: development dataset and evaluation dataset. The development dataset consists of the complete TUT Acoustic Scenes 2016 dataset (both development and evaluation data of the 2016 challenge). The partitioning of the data into subsets was done based on the location of the original recordings, so the evaluation dataset contains recordings of similar audio scenes but from different geographical locations. All segments obtained from the same original recording were included into a single subset - either development dataset or evaluation dataset. For each acoustic scene, there are 312 segments (52 minutes of audio) in the development dataset.

A detailed description of the data recording and annotation procedure is available in:

@inproceedings{Mesaros2016_EUSIPCO,
    author = "Mesaros, Annamaria and Heittola, Toni and Virtanen, Tuomas",
    title = "{TUT} Database for Acoustic Scene Classification and Sound Event Detection",
    abstract = "We introduce TUT Acoustic Scenes 2016 database for environmental sound research, consisting ofbinaural recordings from 15 different acoustic environments. A subset of this database, called TUT Sound Events 2016, contains annotations for individual sound events, specifically created for sound event detection. TUT Sound Events 2016 consists of residential area and home environments, and is manually annotated to mark onset, offset and label of sound events. In this paper we present the recording and annotation procedure, the database content, a recommended cross-validation setup and performance of supervised acoustic scene classification system and event detection baseline system using mel frequency cepstral coefficients and Gaussian mixture models. The database is publicly released to provide support for algorithm development and common ground for comparison of different techniques.",
    year = "2016",
    address = "Budapest, Hungary",
    booktitle = "24th European Signal Processing Conference 2016 (EUSIPCO 2016)"
}

Development dataset

A cross-validation setup is provided for the development dataset in order to make results reported with this dataset uniform. The setup consists of four folds distributing the available segments based on location. The folds are provided with the dataset in the directory evaluation setup.

Fold 1 of the provided setup reproduces the DCASE 2016 challenge setup, by using the 2016 development set as training subset and the 2016 evaluation set as test subset.

Important: If you are not using the provided cross-validation setup, pay attention to the segments extracted from same original recordings. Make sure that for each given fold, ALL segments from same location must be either in the training subset OR in the test subset.

Evaluation dataset

Evaluation dataset without ground truth will be released one month before the submission deadline. Full ground truth meta data for it will be published after the DCASE 2017 challenge and workshop are concluded.

Submission

Detailed information for the challenge submission can found on the submission page.

System output should be presented as a single text-file (in CSV format) containing classification result for each audio file in the evaluation set. Result items can be in any order. Format:

[filename (string)][tab][scene label (string)]

Multiple system outputs can be submitted (maximum 4 per participant). If submitting multiple systems, the individual text-files should be packaged into a zip file for submission. Please carefully mark the connection between the submitted files and the corresponding system or system parameters (for example by naming the text file appropriately).

Task rules

These are the general rules valid for all tasks. The same rules and additional information on technical report and submission requirements can be found here. Task specific rules are highlighted with green.

• Participants are not allowed to use external data for system development. Data from another task is considered external data.
• Manipulation of provided training and development data is allowed.

  The development dataset can be augmented without use of external data (e.g. by mixing data sampled from a pdf or using techniques such as pitch shifting or time stretching).

• Participants are not allowed to make subjective judgments of the evaluation data, nor to annotate it. The evaluation dataset cannot be used to train the submitted system; the use of statistics about the evaluation data in the decision making is also forbidden.

Evaluation

The scoring of acoustic scene classification will be based on classification accuracy: the number of correctly classified segments among the total number of segments. Each segment is considered an independent test sample.

The evaluation is done automatically in the baseline system. Evaluation is done using sed_eval toolbox.

Results

Complete results and technical reports can be found here.

Baseline system

The baseline system for the task is provided. The system is meant to implement a basic approach for acoustic scene classification, and provide some comparison point for the participants while developing their systems. The baseline systems for all tasks share the code base, implementing quite similar approach for all tasks. The baseline system will download the needed datasets and produces the results below when ran with the default parameters.

The baseline system is based on a multilayer perceptron architecture using log mel-band energies as features. A 5-frame context is used, resulting in a feature vector length of 200. Using these features, a neural network containing two dense layers of 50 hidden units per layer and 20% dropout is trained for 200 epochs. Classification decision is based on the network output layer which is of softmax type. A detailed description is available in the baseline system documentation. The baseline system includes evaluation of results using accuracy as metric.

The baseline system is implemented using Python (version 2.7 and 3.6). Participants are allowed to build their system on top of the given baseline system. The system has all needed functionality for dataset handling, storing / accessing features and models, and evaluating the results, making the adaptation for one's needs rather easy. The baseline system is also a good starting point for entry level researchers.

Python implementation

Results for TUT Acoustic scenes 2017, development dataset

Evaluation setup

• 4-fold cross-validation, average classification accuracy over folds
• 15 acoustic scene classes
• Classification unit: one file (10 seconds of audio).
• Python 2.7.13 used

System parameters

• Frame size: 40 ms (with 50% hop size)
• Feature vector: 40 log mel-band energies in 5 consecutive frames = 200 values
• MLP: 2 layers x 50 hidden units, 20% dropout, 200 epochs (using early stopping criteria, monitoring started after 100 epoch, 10 epoch patience), learning rate 0.001, softmax output layer
• Trained and tested on full audio

Citation

If you are using the dataset or baseline code, or want to refer challenge task please cite the following paper:

@inproceedings{DCASE2017challenge,
    Author = "Mesaros, A. and Heittola, T. and Diment, A. and Elizalde, B. and Shah, A. and Vincent, E. and Raj, B. and Virtanen, T.",
    title = "{DCASE} 2017 Challenge Setup: Tasks, Datasets and Baseline System",
    booktitle = "Proceedings of the Detection and Classification of Acoustic Scenes and Events 2017 Workshop (DCASE2017)",
    year = "2017",
    month = "November",
    pages = "85--92",
    abstract = "DCASE 2017 Challenge consists of four tasks: acoustic scene classification, detection of rare sound events, sound event detection in real-life audio, and large-scale weakly supervised sound event detection for smart cars. This paper presents the setup of these tasks: task definition, dataset, experimental setup, and baseline system results on the development dataset. The baseline systems for all tasks rely on the same implementation using multilayer perceptron and log mel-energies, but differ in the structure of the output layer and the decision making process, as well as the evaluation of system output using task specific metrics.",
    keywords = "Sound scene analysis, Acoustic scene classification, Sound event detection, Audio tagging, Rare sound events"
}

When citing challenge results please cite the following paper:

@inproceedings{Mesaros2018_IWAENC,
    author = "Mesaros, Annamaria and Heittola, Toni and Virtanen, Tuomas",
    title = "Acoustic Scene Classification: An Overview of {DCASE} 2017 Challenge Entries",
    booktitle = "2018 16th International Workshop on Acoustic Signal Enhancement (IWAENC)",
    month = "September",
    year = "2018",
    pages = "411--415",
    keywords = "acoustic scene classification, audio classification, DCASE challenge",
    abstract = "We present an overview of the challenge entries for the Acoustic Scene Classification task of DCASE 2017 Challenge. Being the most popular task of the challenge, acoustic scene classification entries provide a wide variety of approaches for comparison, with a wide performance gap from top to bottom. Analysis of the submissions confirms once more the popularity of deep-learning approaches and mel-frequency representations. Statistical analysis indicates that the top ranked system performed significantly better than the others, and that combinations of top systems are capable of reaching close to perfect performance on the given data.",
    doi = "10.1109/IWAENC.2018.8521242"
}