The submission deadline is June 10th 2019 23:59 Anywhere on Earth (AoE)
Introduction
Challenge submission consists in submission package (one zip-package) containing system outputs, system meta information, and technical report (pdf file). The technical report can be the same as your DCASE2019 Workshop submission, but please use the template provided for each.
Submission process shortly:
- Participants run their system with evaluation dataset, and produce the system output in specified format. Participants are allowed to submit 4 different system outputs per task or subtask (except in Task2, where participants are allowed to submit 3).
- Participants create a meta information file to go along the system output to describe the system used to produce this particular output. Meta information file has a predefined format to help the automatic handling of the challenge submissions. Information provided in the meta file will be later used to produce challenge results.
- Participants describe their system in a technical report in sufficient detail. There is a template provided for the technical report.
- Participants prepare the submission package (zip-file). The submission package contains system outputs, maximum 4 per task, systems meta information and the technical report.
- Participants submit the submission package and the technical report to DCASE2019 Challenge.
Please read carefully the requirements for the files included in the submission package!
Submission system
The submission system is now available:
- Create user account and login
- Go to "All Conferences" tab in the system and type DCASE to filter the list
- Select "2019 Challenge on Detection and Classification of Acoustic Scenes and Events"
- Create a new submission
The challenge deadline is 10 June 2019 (AOE).
The technical report in the submission package must contain at least title, authors, and abstract. An updated camera-ready version of the technical report can be submitted separately until 17 June 2019 (AOE).
Note: the submission system does not any send confirmation email. You can check that your submission has been taken into account in your author console. A confirmation email will be sent to all participants once the submissions are closed.
Submission package
Participants are instructed to pack their system output(s), system meta information, and technical report into one zip-package. Example package:
Please prepare your submission zip-file as the provided example. Follow the same file structure and fill meta information with similar structure as the one in *.meta.yaml -files. The zip-file should contain system outputs for all tasks/subtasks, maximum 4 submissions per task/subtask, separate meta information for each system, and technical report(s) covering all submitted systems.
If you submit similar systems for multiple tasks, you can describe everything in one technical report. If your approaches for different tasks are significantly different, prepare one technical report for each and include it in the corresponding task folder.
More detailed instructions for constructing the package can be found in the following sections. Technical report template is available here.
Submission label
Submission label is used to index all your submissions (systems per tasks). To avoid overlapping labels among all submitted systems, use following way to form your label:
[Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number][subtask letter (optional)]_[index number of your submission (1-4)]
For example the baseline systems would have the following labels:
Heittola_TAU_task1a_1Fonseca_UPF_task2_1Adavanne_TAU_task3_1Serizel_ULO_task4_1Cartwright_NYU_task5_1
Package structure
Make sure your zip-package follows provided file naming convention and directory structure:
Zip-package root
│
└───task1 Task1 submissions
│ │ Heittola_TAU_task1.technical_report.pdf Technical report covering all subtasks
│ │ Heittola_TAU_task1a.technical_report.pdf (optional) Technical report for subtask A system only
│ │ Heittola_TAU_task1b.technical_report.pdf (optional) Technical report for subtask B system only
│ │
│ └───Heittola_TAU_task1a_1 Subtask A System 1 submission files
│ │ Heittola_TAU_task1a_1.meta.yaml Subtask A System 1 meta information
│ │ Heittola_TAU_task1a_1.output.csv Subtask A System 1 output
│ :
│ └───Heittola_TAU_task1a_4 Subtask A System 4 submission files
│ │ Heittola_TAU_task1a_2.meta.yaml Subtask A System 4 meta information
│ │ Heittola_TAU_task1a_2.output.csv Subtask A System 4 output
│ │
│ └───Heittola_TAU_task1b_1 Subtask B System 1 submission files
│ │ Heittola_TAU_task1b_1.meta.yaml Subtask B System 1 meta information
│ │ Heittola_TAU_task1b_1.output.csv Subtask B System 1 output
│ :
│ └───Heittola_TAU_task1b_4 Subtask B System 4 submission files
│ │ Heittola_TAU_task1b_1.meta.yaml Subtask B System 4 meta information
│ │ Heittola_TAU_task1b_1.output.csv Subtask B System 4 output
│ │
│ └───Heittola_TAU_task1c_4 Subtask C System 1 submission files
│ │ Heittola_TAU_task1c_1.meta.yaml Subtask C System 1 meta information
│ │ Heittola_TAU_task1c_1.output.csv Subtask C System 1 output
│ :
│ └───Heittola_TAU_task1c_4 Subtask C System 4 submission files
│ Heittola_TAU_task1c_4.meta.yaml Subtask B System 4 meta information
│ Heittola_TAU_task1c_4.output.csv Subtask B System 4 output
│
└───task2 Task2 submissions
│ │ Fonseca_UPF_task2.technical_report.pdf Technical report
│ │
│ └───Fonseca_UPF_task2_1 System 1 submission files
│ │ Fonseca_UPF_task2_1.meta.yaml System 1 meta information
│ │ Fonseca_UPF_task2_1.output.csv System 1 output
│ :
│ │
│ └───Fonseca_UPF_task2_3 System 3 submission files
│ Fonseca_UPF_task2_3.meta.yaml System 3 meta information
│ Fonseca_UPF_task2_3.output.csv System 3 output
│
└───task3 Task3 submissions
│ │ Adavanne_TAU_task3.technical_report.pdf Technical report
│ │
│ └───Adavanne_TAU_task3_1 System 1 submission files
│ │ Adavanne_TAU_task3_1.meta.yaml System 1 meta information
│ │ Adavanne_TAU_task3_1 System 1 output files both development and evaluation (500 files in total)
│ :
│ │
│ └───Adavanne_TAU_task3_4 System 4 submission files
│ Adavanne_TAU_task3_2.meta.yaml System 4 meta information
│ Adavanne_TAU_task3_2 System 4 output files both development and evaluation (500 files in total)
│
└───task4 Task4 submissions
│ │ Serizel_ULO_task4.technical_report.pdf Technical report
│ │
│ └───Serizel_ULO_task4_1 System 1 submission files
│ │ Serizel_ULO_task4_1.meta.yaml System 1 meta information
│ │ Serizel_ULO_task4_1.output.csv System 1 output
│ :
│ │
│ └───Serizel_ULO_task4_4 System 4 submission files
│ Serizel_ULO_task4_4.meta.yaml System 4 meta information
│ Serizel_ULO_task4_4.output.csv System 4 output
│
└───task5 Task5 submissions
│ Cartwright_NYU_task5.technical_report.pdf Technical report
│
└───Cartwright_NYU_task5_1 System 1 submission files
│ Cartwright_NYU_task5_1.meta.yaml System 1 meta information
│ Cartwright_NYU_task5_1.output.csv System 1 output
:
│
└───Cartwright_NYU_task5_4 System 4 submission files
Cartwright_NYU_task5_2.meta.yaml System 4 meta information
Cartwright_NYU_task5_2.output.csv System 4 output
System outputs
Participants must submit the results for the provided evaluation datasets.
-
Follow the system output format specified in the task description.
-
Tasks are independent. You can participate to a single task or multiple tasks.
-
Multiple submissions for the same task are allowed (maximum 4 per task, except in Task2 where only 3 are allowed). Use a running index in the submission label, and give more detailed names for the submitted systems in the system meta information files. Please mark carefully the connection between the submitted systems and system parameters description in the technical report (for example by referring to the systems by using the submission label or system name given in the system meta information file).
-
Submitted system outputs will be published online on the DCASE2019 website later to allow future evaluations.
Technical report
All participants are expected to submit a technical report about the submitted system, to help the DCASE community better understand how the algorithm works.
Technical reports are not peer-reviewed. The technical reports will be published on the challenge website together with all other information about the submitted system. For the technical report it is not necessary to follow closely the scientific publication structure (for example there is no need for extensive literature review). The report should however contain sufficient description of the system.
Please report the system performance using the provided cross-validation setup or development set, according to the task. For participants taking part in multiple tasks, one technical report covering all tasks is sufficient, if the systems have only small differences. Describe the task specific parameters in the report.
Participants can also submit the same report as a scientific paper to DCASE 2019 Workshop. In this case, the paper must respect the structure of a scientific publication, and be prepared according to the provided Workshop paper instructions and template. Please note that the template is slightly different, and you will have to create a separate submission to the DCASE2019 Workshop track in the submission system. Please refer to the workshop webpage for more details. DCASE2019 Workshop papers will be peer-reviewed.
Template
Reports are in format 4+1 pages. Papers are maximum 5 pages, including all text, figures, and references, with the 5th page containing only references. The templates for technical report are available here:
Meta information
In order to allow meta analysis of submitted systems, participants should provide rough meta information presented in a structured and correctly formatted YAML-file.
See example meta files below for each baseline system. These examples are also available in the example submission package. Meta file structure is mostly the same for all tasks, only the metrics collected in results->development_dataset-section differ per challenge task.
Example meta information file for Task 1 baseline system task1/Heittola_TAU_task1a_1/Heittola_TAU_task1a_1.meta.yaml:
# Submission information
submission:
# Submission label
# Label is used to index submissions, to avoid overlapping codes among submissions
# use following way to form your label:
# [Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number]_[index number of your submission (1-4)]
label: Heittola_TAU_task1a_1
# Submission name
# This name will be used in the results tables when space permits
name: DCASE2019 baseline system
# Submission name abbreviated
# This abbreviated name will be used in the results table when space is tight, maximum 10 characters
abbreviation: Baseline
# Submission authors in order, mark one of the authors as corresponding author.
authors:
# First author
- lastname: Heittola
firstname: Toni
email: toni.heittola@tuni.fi # Contact email address
corresponding: true # Mark true for one of the authors
# Affiliation information for the author
affiliation:
abbreviation: TAU
institute: Tampere University
department: Computing Sciences
location: Tampere, Finland
# Second author
- lastname: Mesaros
firstname: Annamaria
email: annamaria.mesaros@tuni.fi # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: TAU
institute: Tampere University
department: Computing Sciences
location: Tampere, Finland
# System information
system:
# System description, meta data provided here will be used to do
# meta analysis of the submitted system. Use general level tags, if possible use the tags provided in comments.
# If information field is not applicable to the system, use "!!null".
description:
# Audio input
input_channels: mono # e.g. one or multiple [mono, binaural, left, right, mixed, ...]
input_sampling_rate: 48kHz #
# Acoustic representation
acoustic_features: log-mel energies # e.g one or multiple [MFCC, log-mel energies, spectrogram, CQT, ...]
# Data augmentation methods
data_augmentation: !!null # [time stretching, block mixing, pitch shifting, ...]
# Machine learning
# In case using ensemble methods, please specify all methods used (comma separated list).
machine_learning_method: CNN # e.g one or multiple [GMM, HMM, SVM, kNN, MLP, CNN, RNN, CRNN, NMF, random forest, ensemble, ...]
# Ensemble method subsystem count
# In case ensemble method is not used, mark !!null.
ensemble_method_subsystem_count: !!null # [2, 3, 4, 5, ... ]
# Decision making methods
decision_making: !!null # [majority vote, ...]
# External data usage method
external_data_usage: !!null # [directly, embeddings, pre-trained model, ...]
# System complexity, meta data provided here will be used to evaluate
# submitted systems from the computational load perspective.
complexity:
# Total amount of parameters used in the acoustic model. For neural networks, this
# information is usually given before training process in the network summary.
# For other than neural networks, if parameter count information is not directly available,
# try estimating the count as accurately as possible.
# In case of ensemble approaches, add up parameters for all subsystems.
# Use numerical value.
total_parameters: 116118
# List of external datasets used in the submission. Development dataset is used here as example, list only external datasets
external_datasets:
# Dataset name
- name: TAU Urban Acoustic Scenes 2019, Development dataset
# Dataset access url
url: https://doi.org/10.5281/zenodo.2589280
# Total audio length in minutes
total_audio_length: 2400 # minutes
# URL to the source code of the system [optional]
source_code: https://github.com/toni-heittola/dcase2019_task1_baseline
# System results
results:
# Full results are not mandatory, but for through analysis of the challenge submissions recommended.
# If you cannot provide all results, also incomplete results can be reported.
development_dataset:
# System result for development dataset with provided the cross-validation setup.
# Overall accuracy (mean of class-wise accuracies)
overall:
accuracy: 62.5
# Class-wise accuracies
class_wise:
airport:
accuracy: 48.4
bus:
accuracy: 62.3
metro:
accuracy: 65.1
metro_station:
accuracy: 54.5
park:
accuracy: 83.1
public_square:
accuracy: 40.7
shopping_mall:
accuracy: 59.4
street_pedestrian:
accuracy: 60.9
street_traffic:
accuracy: 86.7
tram:
accuracy: 64.0
public_leaderboard:
# Team name used in public leaderboard (https://www.kaggle.com/c/dcase2019-task1a-leaderboard)
team_name: DCASE2019 Task1A baseline
# System score from public leaderboard (https://www.kaggle.com/c/dcase2019-task1a-leaderboard)
overall:
accuracy: 64.3
Example meta information file for Task 2 baseline system task2/Fonseca_UPF_task2_1/Fonseca_UPF_task2_1.meta.yaml:
# Submission information
submission:
# Submission label
# Label is used to index submissions, to avoid overlapping codes among submissions
# use following way to form your label:
# [Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number]_[index number of your submission (1-4)]
label: Fonseca_UPF_task2_1
# Submission name
# This name will be used in the results tables when space permits
name: DCASE2018 baseline system
# Submission name abbreviated
# This abbreviated name will be used in the results table when space is tight, maximum 10 characters
abbreviation: Baseline
# Kaggle Team Name as in Leaderboard (this is important to match the DCASE submission to the Kaggle team)
kaggle_team_name: my_team_name_as_in_leaderboard
# Kaggle submission ID
# Two submissions are eligible for the final private leaderboard, and one additional submission for Judges Award, hence three in total
# There must be one different *.yaml file per submission, hence three yaml files in total (if you submit to the Judges Award)
kaggle_submission_ID: 1234567
# Submission authors in order, mark one of the authors as corresponding author.
authors:
# First author
- lastname: Fonseca
firstname: Eduardo
email: eduardo.fonseca@upf.edu # Contact email address
corresponding: true # Mark true for one of the authors
# Affiliation information for the author
affiliation:
abbreviation: UPF
institute: Universitat Pompeu Fabra, Barcelona
department: Music Technology Group
location: Barcelona, Spain
# Second author
- lastname: Font
firstname: Frederic
email: frederic.font@upf.edu # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: UPF
institute: Universitat Pompeu Fabra, Barcelona
department: Music Technology Group
location: Barcelona, Spain
# Third author
- lastname: Plakal
firstname: Manoj
email: plakal@google.com # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: GOOGLE
institute: Google Research
department: Machine Perception Team
location: New York, USA
# Fourth author
- lastname: Ellis
firstname: Daniel P. W.
email: dpwe@google.com # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: GOOGLE
institute: Google Research
department: Machine Perception Team
location: New York, USA
# System information
system:
# System description, meta data provided here will be used to do
# meta analysis of the submitted system. Use general level tags, if possible use the tags provided in comments.
# If information field is not applicable to the system, use "!!null".
description:
# Audio input
input_channels: mono # e.g. one or multiple [mono, binaural, left, right, mixed, ...]
input_sampling_rate: 44.1kHz #
# Acoustic representation
acoustic_features: log-mel energies # e.g one or multiple [MFCC, log-mel energies, spectrogram, CQT, ...]
# Data augmentation methods
data_augmentation: !!null # [time stretching, block mixing, pitch shifting, ...]
# Machine learning
# In case using ensemble methods, please specify all methods used (comma separated list).
machine_learning_method: CNN # e.g one or multiple [GMM, HMM, SVM, kNN, MLP, CNN, RNN, CRNN, NMF, random forest, ensemble, ...]
# Ensemble method subsystem count
# In case ensemble method is not used, mark !!null.
ensemble_method_subsystem_count: !!null # [2, 3, 4, 5, ... ]
# Decision making methods
decision_making: arithmetic mean # [majority vote, arithmetic mean, geometric mean,...]
# Approach used to exploit noisy subset of the train set
# In case the noisy set is not used, mark !!null.
noisy_subset: using provided labels # e.g one or multiple [using provided labels, automatic re-labeling, unsupervised, manual re-labeling, ...]
# System complexity, meta data provided here will be used to evaluate
# submitted systems from the computational load perspective.
complexity:
# Total amount of parameters used in the acoustic model. For neural networks, this
# information is usually given before training process in the network summary.
# For other than neural networks, if parameter count information is not directly available,
# try estimating the count as accurately as possible.
# In case of ensemble approaches, add up parameters for all subsystems.
total_parameters: 3.3M
# Approximate training time followed by the hardware used
trainining_time: 8h (1 x Tesla V-100)
# URL to the source code of the system [optional]
source_code: https://github.com/DCASE-REPO/dcase2019_task2_baseline
# System results
results:
public_leaderboard:
# System score from public leaderboard (https://www.kaggle.com/c/freesound-audio-tagging-2019/leaderboard)
overall:
lwlrap: 0.537
Example meta information file for Task 3 baseline system task3/Adavanne_TAU_task3_1/Adavanne_TAU_task3_1.meta.yaml:
# Submission information
submission:
# Submission label
# Label is used to index submissions, to avoid overlapping codes among submissions
# use following way to form your label:
# [Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number]_[index number of your submission (1-4)]
label: Adavanne_TAU_task3_1
# Submission name
# This name will be used in the results tables when space permits
name: DCASE2019 Ambisonic example
# Submission name abbreviated
# This abbreviated name will be used in the results table when space is tight, maximum 10 characters
abbreviation: Ambisonic
# Submission authors in order, mark one of the authors as corresponding author.
authors:
# First author
- lastname: Adavanne
firstname: Sharath
email: sharath.adavanne@tuni.fi # Contact email address
corresponding: true # Mark true for one of the authors
# Affiliation information for the author
affiliation:
abbreviation: TAU
institute: Tampere University
department: Audio Research Group
location: Tampere, Finland
# Second author
- lastname: Politis
firstname: Archontis
email: archontis.politis@tuni.fi # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: TAU
institute: Tampere University
department: Audio Research Group
location: Tampere, Finland
# Third author
- lastname: Virtanen
firstname: Tuomas
email: tuomas.virtanen@tuni.fi # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: TAU
institute: Tampere University
department: Audio Research Group
location: Tampere, Finland
# System information
system:
# System description, meta data provided here will be used to do
# meta analysis of the submitted system. Use general level tags, if possible use the tags provided in comments.
# If information field is not applicable to the system, use "!!null".
description:
# Audio input
input_format: Ambisonic # e.g. Ambisonic or Microphone Array or both
input_sampling_rate: 48kHz #
# Acoustic representation
acoustic_features: phase and magnitude spectrogram # e.g one or multiple [phase and magnitude spectrogram, GCC, TDOA ...]
# Data augmentation methods
data_augmentation: !!null # [time stretching, block mixing, pitch shifting, ...]
# Machine learning
# In case using ensemble methods, please specify all methods used (comma separated list).
machine_learning_method: CRNN # e.g one or multiple [GMM, HMM, SVM, kNN, MLP, CNN, RNN, CRNN, NMF, random forest, ensemble, ...]
# System complexity, meta data provided here will be used to evaluate
# submitted systems from the computational load perspective.
complexity:
# Total amount of parameters used in the acoustic model. For neural networks, this
# information is usually given before training process in the network summary.
# For other than neural networks, if parameter count information is not directly available,
# try estimating the count as accurately as possible.
# In case of ensemble approaches, add up parameters for all subsystems.
total_parameters: 116118
# URL to the source code of the system [optional]
source_code: https://github.com/sharathadavanne/seld-dcase2019
# System results
results:
development_dataset:
# System result for development dataset with the provided cross-validation setup.
# Overall score
overall:
error_rate: 0.34
f_score: 79.9
doa_error: 28.5
frame_recall: 85.4
Example meta information file for Task 4 baseline system task4/Serizel_ULO_task4_1/Serizel_ULO_task4_1.meta.yaml:
# Submission information
submission:
# Submission label
# Label is used to index submissions, to avoid overlapping codes among submissions
# use following way to form your label:
# [Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number]_[index number of your submission (1-4)]
label: Serizel_UL_task4_1
# Submission name
# This name will be used in the results tables when space permits
name: DCASE2019 baseline system
# Submission name abbreviated
# This abbreviated name will be used in the results table when space is tight, maximum 10 characters
abbreviation: Baseline
# Submission authors in order, mark one of the authors as corresponding author.
authors:
# First author
- lastname: Serizel
firstname: Romain
email: romain.serizel@loria.fr # Contact email address
corresponding: true # Mark true for one of the authors
# Affiliation information for the author
affiliation:
abbreviation: ULO
institute: University of Lorraine, Loria
department: Department of Natural Language Processing & Knowledge Discovery
location: Nancy, France
# Second author
- lastname: Turpault
firstname: Nicolas
email: nicolas.turpault@inria.fr # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: INR
institute: Inria Nancy Grand-Est
department: Department of Natural Language Processing & Knowledge Discovery
location: Nancy, France
- lastname: Salamon
firstname: Justin
email: justin.salamon@nyu.edu # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: ADO
institute: Adobe Research
location: Adobe Research, United States
- lastname: Parag Shah
firstname: Ankit
email: aps1@andrew.cmu.edu # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: CMU
institute: Carnegie Mellon University
department: School of Computer Science
location: Pittsburgh, United States
- lastname: Eghbal-Zadeh
firstname: Hamid
email: hamid.eghbal-zadeh@jku.at # Contact email address
# Affiliation information for the author
affiliation:
abbreviation: JKP
institute: Johannes Kepler University
department: Department of Computational Perception
location: Linz, Austria
# System information
system:
# System description, meta data provided here will be used to do
# meta analysis of the submitted system. Use general level tags, if possible use the tags provided in comments.
# If information field is not applicable to the system, use "!!null".
description:
# Audio input
input_channels: mono # e.g. one or multiple [mono, binaural, left, right, mixed, ...]
input_sampling_rate: 44.1kHz #
# Acoustic representation
acoustic_features: log-mel energies # e.g one or multiple [MFCC, log-mel energies, spectrogram, CQT, ...]
# Data augmentation methods
data_augmentation: !!null # [time stretching, block mixing, pitch shifting, ...]
# Machine learning
# In case using ensemble methods, please specify all methods used (comma separated list).
machine_learning_method: CRNN # e.g one or multiple [GMM, HMM, SVM, kNN, MLP, CNN, RNN, CRNN, NMF, random forest, ensemble, ...]
# Ensemble method subsystem count
# In case ensemble method is not used, mark !!null.
ensemble_method_subsystem_count: !!null # [2, 3, 4, 5, ... ]
# Decision making methods
decision_making: !!null # [majority vote, ...]
# Semi-supervised method used to exploit both labelled and unlabelled data
machine_learning_semi_supervised: mean-teacher student # e.g one or multiple [pseudo-labelling, mean-teacher student...]
# Segmentation method
segmentation_method: !!null # E.g. [RBM, attention layers...]
# Post-processing, followed by the time span (in ms) in case of smoothing
post-processing: median filtering (93ms) # [median filtering, time aggregation...]
# System complexity, meta data provided here will be used to evaluate
# submitted systems from the computational load perspective.
complexity:
# Total amount of parameters used in the acoustic model. For neural networks, this
# information is usually given before training process in the network summary.
# For other than neural networks, if parameter count information is not directly available,
# try estimating the count as accurately as possible.
# In case of ensemble approaches, add up parameters for all subsystems.
total_parameters: 126090
# Approximate training time followed by the hardware used
trainining_time: 3h (1 GTX 1080 Ti)
# The training subsets used to train the model. Followed the amount of data (number of clips) used per subset.
subsets: # [weak (xx), unlabel_in_domain (xx), synthetic (xx)]
# URL to the source code of the system [optional, highly recommended]
source_code: https://github.com/DCASE-REPO/dcase2018_baseline/tree/master/task4/
# System results
results:
# Full results are not mandatory, but for through analysis of the challenge submissions recommended.
# If you cannot provide all results, also incomplete results can be reported.
development_dataset:
# System result for development dataset with provided the cross-validation setup.
overall:
F-score: 23.7
# Class-wise accuracies
class_wise:
Alarm_bell_ringing:
F-score: 20.0
Blender:
F-score: 25.1
Cat:
F-score: 7.2
Dishes:
F-score: 3.2
Dog:
F-score: 14.1
Electric_shaver_toothbrush:
F-score: 42.7
Frying:
F-score: 54.1
Running_water:
F-score: 11.6
Speech:
F-score: 21.5
Vacuum_cleaner:
F-score: 60.8
Example meta information file for Task 5 baseline system task5/Cartwright_NYU_task5_1/Cartwright_NYU_task5_1.meta.yaml:
# Submission information
submission:
# Submission label
# Label is used to index submissions, to avoid overlapping codes among submissions
# use following way to form your label:
# [Last name of corresponding author]_[Abbreviation of institution of the corresponding author]_task[task number]_[index number of your submission (1-4)]
label: Cartwright_NYU_task5_1
# Submission name
# This name will be used in the results tables when space permits
name: DCASE2019 baseline system
# Submission name abbreviated
# This abbreviated name will be used in the results table when space is tight, maximum 10 characters
abbreviation: Baseline
# Submission authors in order, mark one of the authors as corresponding author.
authors:
# First author
- firstname: Mark
lastname: Cartwright
email: mark.cartwright@nyu.edu # Contact email address
corresponding: true # Mark true for one of the authors
# Affiliation information for the author
affiliation:
institution: New York University
department: Music and Audio Research Laboratory, Department of Computer Science and Engineering, Center for Urban Science and Progress
location: New York, New York, USA
# Second author
- firstname: Jason
lastname: Cramer
# Affiliation information for the author
affiliation:
institution: New York University
department: Music and Audio Research Laboratory, Department of Electrical and Computer Engineering
location: New York, New York, USA
# Third author
- firstname: Ana Elisa
lastname: Mendez Mendez
# Affiliation information for the author
affiliation:
institution: New York University
department: Music and Audio Research Laboratory, Department of Music and Performing Arts Professions
location: New York, New York, USA
# Fourth author
- firstname: Ho-Hsiang
lastname: Wu
# Affiliation information for the author
affiliation:
institution: New York University
department: Music and Audio Research Laboratory, Department of Music and Performing Arts Professions
location: New York, New York, USA
# Fifth author
- firstname: Vincent
lastname: Lostanlen
# Affiliation information for the author
affiliation:
institution: Cornell University
department: Cornell Lab of Ornithology
location: Ithaca, New York, USA
# Sixth author
- firstname: Juan P.
lastname: Bello
# Affiliation information for the author
affiliation:
institution: New York University
department: Music and Audio Research Laboratory, Department of Computer Science and Engineering, Center for Urban Science and Progress
location: New York, New York, USA
# Seventh author
- firstname: Justin
lastname: Salamon
# Affiliation information for the author
affiliation:
institution: Adobe Research
department: Machine Perception Team
location: San Francisco, CA, USA
# System information
system:
# System description, meta data provided here will be used to do
# meta analysis of the submitted system. Use general level tags, if possible use the tags provided in comments.
# If information field is not applicable to the system, use "!!null".
description:
# Audio input
input_channels: mono # e.g. one or multiple [mono, binaural, left, right, mixed, ...]
input_sampling_rate: 44.1kHz #
# Acoustic representation
acoustic_features: vggish # e.g one or multiple [MFCC, log-mel energies, spectrogram, CQT, deep embedding (e.g. vggish), ...]
# Data augmentation methods
data_augmentation: !!null # [time stretching, block mixing, pitch shifting, ...]
# Machine learning method
# In case using ensemble methods, please specify all methods used (comma separated list).
# You do not need to repeat model types used multiple times in the ensemble.
machine_learning_method: logistic regression # e.g one or multiple [GMM, HMM, kNN, MLP, CNN, RNN, CRNN, NMF, random forest, ensemble, ...]
# Ensemble method subsystem count
# In case ensemble method is not used, mark !!null.
ensemble_method_subsystem_count: !!null # [2, 3, 4, 5, ... ]
# Specify if any of the additional metadata was used for training
used_annotator_id: false
used_proximity: false
used_sensor_id: false
# Method for aggregating predictions over time, if relevant
aggregation_method: !!null # [mean, ...]
# External data approach
external_data: !!null # [pre-trained model, audio data, ...]
# Comma separated list of external data sources used
external_data_sources: !!null # [pre-trained model, audio data, ...]
# Annotation level targeted by model
# That is, if the model only predicts fine level annotations
# should be specified as "fine". A model that is specifically
# trained to predict both fine and coarse annotations should
# be specified as "both".
target_level: fine # [fine, coarse, both]
# Method for determining targets for training from annotations
target_method: minority vote # [minority vote, majority vote, ...]
# Re-labeling of the train set
re_labeling: !!null # [automatic, manual, ...]
# NOTE: These should only be provided if providing detections, rather than
# probabilities of class presence.
#
# Type of method used to determine detection thresholds
detection_threshold_method: !!null # [automatic, manual, fixed, !!null]
# Specify if the method for determining threshold was done over all classes
# or per class
detection_threshold_level: !!null # [global, classwise, !!null]
# System complexity, meta data provided here will be used to evaluate
# submitted systems from the computational load perspective.
complexity:
# Total amount of learned parameters used in the acoustic model.
# For neural networks, this information is usually given before training process
# in the network summary. For other than neural networks, if parameter count
# information is not directly available,try estimating the count as accurately
# as possible. In case of ensemble approaches, add up parameters for all subsystems.
total_parameters: 2967
# URL to the source code of the system
source_code: https://github.com/sonyc-project/urban-sound-tagging-baseline