This task focuses on sound event detection in a few-shot learning setting for animal (mammal and bird) vocalisations. Participants will be expected to create a method that can extract information from five exemplar vocalisations (shots) of mammals or birds and detect and classify sounds in field recordings.

Challenge has ended. Full results for this task can be found in the Results page.

Description

Few-shot learning is a highly promising paradigm for sound event detection. It is also an extremely good fit to the needs of users in bioacoustics, in which increasingly large acoustic datasets commonly need to be labelled for events of an identified category (e.g. species or call-type), even though this category might not be known in other datasets or have any yet-known label. While satisfying user needs, this will also benchmark few-shot learning for the wider domain of sound event detection (SED).

Few-shot learning describes tasks in which an algorithm must make predictions given only a few instances of each class, contrary to standard supervised learning paradigm. The main objective is to find reliable algorithms that are capable of dealing with data sparsity, class imbalance and noisy/busy environments. Few-shot learning is usually studied using N-way-K-shot classification, where N denotes the number of classes and K the number of examples for each class.

Some reasons why few-shot learning has been of increasing interest:

• Scarcity of supervised data can lead to unreliable generalisations of machine learning models.
• Explicitly labeling a huge dataset can be costly both in time and resources.
• Fixed ontologies or class labels used in SED and other DCASE tasks are often a poor fit to a given user’s goal.

Development Set

The development set is pre-split into training and validation sets. The training set consists of five sub-folders deriving from a different source each. Along with the audio files multi-class annotations are provided for each. The validation set consists of two sub-folders deriving from a different source each, with a single-class (class of interest) annotation file provided for each audio file.

Training Set

The training set contains five different sub-folders (BV, HV, JD, MT,WMW). Statistics are given overall and specific for each sub-folder.

Overall

BV

The BirdVox-DCASE-10h (BV for short) contains five audio files from four different autonomous recording units, each lasting two hours. These autonomous recording units are all located in Tompkins County, New York, United States. Furthermore, they follow the same hardware specification: the Recording and Observing Bird Identification Node (ROBIN) developed by the Cornell Lab of Ornithology. Andrew Farnsworth, an expert ornithologist, has annotated these recordings for the presence of flight calls from migratory passerines, namely: American sparrows, cardinals, thrushes, and warblers. In total, the annotator found 2,662 from 11 different species. We estimate these flight calls to have a duration of 150 milliseconds and a fundamental frequency between 2 kHz and 10 kHz.

HT

Spotted hyenas are a highly social species that live in "fission-fusion" groups where group members range alone or in smaller subgroups that split and merge over time. Hyenas use a variety of types of vocalizations to coordinate with one another over both short and long distances. Spotted hyena vocalization data were recorded on custom-developed audio tags designed by Mark Johnson and integrated into combined GPS / acoustic collars (Followit Sweden AB) by Frants Jensen and Mark Johnson. Collars were deployed on female hyenas of the Talek West hyena clan at the MSU-Mara Hyena Project (directed by Kay Holekamp) in the Masai Mara, Kenya as part of a multi-species study on communication and collective behavior. Field work was carried out by Kay Holekamp, Andrew Gersick, Frants Jensen, Ariana Strandburg-Peshkin, and Benson Pion; labeling was done by Kenna Lehmann and colleagues.

JD

Jackdaws are corvid songbirds which usually breed, forage and sleep in large groups, but form a pair bond with the same partner for life. They produce thousands of vocalisations per day, but many aspects of their vocal behaviour remained unexplored due to the difficulty in recording and assigning vocalisations to specific individuals, especially in natural settings. In a multi-year field study (Max-Planck-Institute for Ornithology, Seewiesen, Germany), wild jackdaws were equipped with small backpacks containing miniature voice recorders (Edic Mini Tiny A31, TS-Market Ltd., Russia) to investigate the vocal behaviour of individuals interacting normally with their group, and behaving freely in their natural environment. The jackdaw training dataset contains a 10-minute on-bird sound recording of one male jackdaw during the breeding season 2015. Field work was conducted by Lisa Gill, Magdalena Pelayo van Buuren and Magdalena Maier. Sound files were annotated by Lisa Gill, based on a previously established video-validation in a captive setting.

MT

Meerkats are a highly social mongoose species that live in stable social groups and use a variety of distinct vocalizations to communicate and coordinate with one another. Meerkat vocalization data were recorded at the Kalahari Meerkat Project (Kuruman River Reserve, South Africa; directed by Marta Manser and Tim Clutton-Brock), as part of a multi-species study on communication and collective behavior. Data in the training set were recorded on small audio devices (TS Market, Edic Mini Tiny+ A77, 8 kHz) integrated into combined GPS/audio collars which were deployed on multiple members of meerkat groups to monitor their movements and vocalizations simultaneously. Recordings were carried out during daytime hours while meerkats were primarily foraging (digging in the ground for small prey items). Field work was carried out by Ariana Strandburg-Peshkin, Baptiste Averly, Vlad Demartsev, Gabriella Gall, Rebecca Schaefer and Marta Manser. Audio recordings were labeled by Baptiste Averly, Vlad Demartsev, Ariana Strandburg-Peshkin, and colleagues.

WMW

WMW consist on a selection of recordings from the Western Mediterranean Wetlands Bird dataset. The recordings are taken from the Xeno-Canto portal. The present selection consists in 161 audio recordings of different lengths that have at least 10 positive events. These have been annotated for 26 different classes of 20 species of birds.

Training annotation format

Annotation files have the same name as their corresponding audiofiles with extension *.csv. For the training set multi-class annotations are provided, with positive (POS), negative (NEG) and unknown (UNK) values for each class. UNK indicates uncertainty about a class and participants can choose to ignore it. Example of an annotation file for audio.wav:

Audiofilename,Starttime,Endtime,CLASS_1,CLASS_2,...,CLASS_N
audio.wav,1.1,2.2,POS,NEG,...,NEG
.
.
.
audio.wav,99.9,100.0,UNK,UNK,...,NEG

Validation Set

The validation set comprises of six sub-folders (HB, PB, ME, RD, PB24, and PW). Specific information about the source of the recordings are not provided for the participants for the duration of the challenge, as to make information available for the validation set as similar to the evaluation set (once that is also published). More information about both will be made available after the end of the challenge.

There is no overlap between the training set and validation set classes.

Overall

HB

PB

ME

RD

PB24

PW

Validation annotation format

Annotation files have the same name as their corresponding audiofiles with extension *.csv. For the validation set single-class (class of interest) annotations are provided, with positive (POS), unkwown (UNK) values. UNK indicates uncertainty about a class and participants can choose to ignore it. Each audio file should be treated separately of the rest, as there is possible overlap between the classes of the evaluation set across different audio files.

Participants must treat the task as a 5-shot setting and only use the first five POS annotations for the class of interest for each file, when trying to predict the rest.

Example of an annotation file for audio_val.wav:

Audiofilename,Starttime,Endtime,Q
audio_val.wav,1.1,2.2,POS
.
.
.
audio_val.wav,99.9,100.0,UNK

Download

DCASE2024 Task 5 Development dataset (4.8 GB)

Evaluation Set

The Evaluation set is identical to the one utilized in the 2023 challenge and consists of 66 audio files acquired from different bioacoustic sources organized by 8 subsets (CHE, CHE23, CT, CW, DC, MGE, MS, QU) At this time only the first 5 annotations are provided for each file, with events marked as positive (POS) for the class of interest. The annotation files follow the same format as for the validation set. This dataset is to be used for evaluation purposes during the task.

DCASE 2024 Task 5: Few-shot Bioacoustic Event Detection Evaluation Set (3.2 GB)
version 1.0

Task setup

This few-shot task will run as a 5-shot task. Hence, five annotated calls from each recording in the evaluation set will be provided to the participants. Each recording of the evaluation set will have a single class of interest which the participants will then need to detect through the recording. Each recording can have multiple types of calls or species present in it, as well as background noise, however only the label of interest needs to be detected.

During the development period the participants are required to treat the validation set in the same way as the evaluation set by using the first five positive (POS) events for their models. Participants should keep in mind that our evaluation metric ignores anything before the end time of the fifth positive event, hence using randomly selected events from the validation set may lead to incorrect performance values.

Task rules

• Use of external data (e.g. audio files, annotations) is allowed only after approval from the task coordinators (contact: burooj.ghani@naturalis.nl or i.dealmeidanolasco@qmul.ac.uk ). Typically these external datasets should be public, open datasets.
• Use of pre-trained models is allowed only after approval from the task coordinators (contact: burooj.ghani@naturalis.nl or i.dealmeidanolasco@qmul.ac.uk).
• List of external datasets and models that are allowed (note that this list is not exhaustive, please ask the coordinators for an approval for other models/datasets):

• The development dataset (i.e. training and validation) can be augmented without the use of external data.
• Participants are not allowed to use VGG Sound dataset, DCASE2018 Bird Audio Detection task dataset and AudioSet Strong
• Participants are not allowed to make subjective judgments of the evaluation data, nor to annotate it.
• Participants are not allowed to use extra annotations for the provided data.
• Participants are only allowed to use the first five positive (POS) annotations from each validation set annotation file and use the rest for evaluation of their method.
• Participants must treat each file in the validation set independently of the others (e.g. for prototypical networks do not save prototypes between audio files). This is due to the fact that the classes of the validation set are hidden and there is possible overlap between them inside the validation set.
• Ensemble models are not allowed.

Submission

Official challenge submission consists of:

• System output file (*.csv)
• Metadata file (*.yaml)
• Technical report explaining in sufficient detail the method (*.pdf)

System output should be presented as a single text-file (in CSV format, with a header row as shown in the example output below).

For each system, meta information should be provided in a separate file, containing the task-specific information. This meta information enables fast processing of the submissions and analysis of submitted systems. Participants are advised to fill the meta information carefully while making sure all information is correctly provided.

We allow up to 4 system output submissions per participant/team. For each system, metadata should be provided in a separate file, containing the task specific information. All files should be packaged into a zip file for submission. Please make a clear connection between the system name in the submitted metadata (the *.yaml file), submitted system output (the *.csv file), and the technical report. The detailed information regarding the challenge information can be found in the Submission page. Finally, for supporting reproducible research, we kindly ask from each participant/team to consider making available the code of their method (e.g. in GitHub) and pre-trained models, after the challenge is over.

Please note: automated procedures will be used for the evaluation of the submitted results. Therefore, the column names should be exactly as indicated in the example output below. Events in each file should be in order of start time.

Example output:

Audiofilename,Starttime,Endtime
BUK5_20161101_002104a.wav,356.0134694,356.1384127
BUK5_20161101_002104a.wav,356.18839,356.488254
BUK5_20161101_002104a.wav,356.5882086,356.7131519

Metadata file

Example meta information file for task 5 baseline system task5/Morfi_QMUL_task5_1/Morfi_QMUL_task5_1.meta.yaml:

# Submission information
submission:
  # Submission label
  # Label is used to index submissions, to avoid overlapping codes among submissions
  # use the following way to form your label:
  # [Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number]_[index number of your submission (1-4)]
  label: Morfi_QMUL_task5_1

  # Submission name
  # This name will be used in the results tables when space permits
  name: Cross-correlation baseline

  # Submission name abbreviated
  # This abbreviated name will be used in the results table when space is tight, maximum 10 characters
  abbreviation: xcorr_base

  # Submission authors in order, mark one of the authors as corresponding author.
  authors:
    # First author
    - lastname: Morfi
      firstname: Veronica
      email: g.v.morfi@qmul.ac.uk                     # Contact email address
      corresponding: true                             # Mark true for one of the authors

      # Affiliation information for the author
      affiliation:
        abbreviation: QMUL
        institute: Queen Mary University of London
        department: Centre for Digital Music
        location: London, UK

    # Second author
    - lastname: Stowell
      firstname: Dan
      email: dan.stowell@qmul.ac.uk                  # Contact email address

      # Affiliation information for the author
      affiliation:
        abbreviation: QMUL
        institute: Queen Mary University of London
        department: Centre for Digital Music
        location: London, UK

        #...


# System information
system:
  # SED system description, meta data provided here will be used to do
  # meta analysis of the submitted system. Use general level tags, if possible use the tags provided in comments.
  # If information field is not applicable to the system, use "!!null".
  description:

    # Audio input
    input_sampling_rate: any               # In kHz

    # Acoustic representation
    acoustic_features: spectrogram   # e.g one or multiple [MFCC, log-mel energies, spectrogram, CQT, PCEN, ...]

    # Data augmentation methods
    data_augmentation: !!null             # [time stretching, block mixing, pitch shifting, ...]

    # Embeddings
    # e.g. VGGish, OpenL3, ...
    embeddings: !!null

    # Machine learning
    # In case using ensemble methods, please specify all methods used (comma separated list).
    machine_learning_method: template matching         # e.g one or multiple [GMM, HMM, SVM, kNN, MLP, CNN, RNN, CRNN, NMF, random forest, ensemble, transformer, ...]
    # the system adaptation for "few shot" scenario.
    # For example, if machine_learning_method is "CNN", the few_shot_method might use one of [fine tuning, prototypical, MAML] in addition to the standard CNN architecture.
    few_shot_method: template matching         # e.g [fine tuning, prototypical, MAML, nearest neighbours...]

    # External data usage method
    # e.g. directly, embeddings, pre-trained model, ...
    external_data_usage: !!null

    # Ensemble method subsystem count
    # In case ensemble method is not used, mark !!null.
    ensemble_method_subsystem_count: !!null # [2, 3, 4, 5, ... ]

    # Decision making methods (for ensemble)
    decision_making: !!null                 # [majority vote, ...]

    # Post-processing, followed by the time span (in ms) in case of smoothing
    post-processing: peak picking, threshold				# [median filtering, time aggregation...]

  # System complexity, meta data provided here will be used to evaluate
  # submitted systems from the computational load perspective.
  complexity:

    # Total amount of parameters used in the acoustic model. For neural networks, this
    # information is usually given before training process in the network summary.
    # For other than neural networks, if parameter count information is not directly available,
    # try estimating the count as accurately as possible.
    # In case of ensemble approaches, add up parameters for all subsystems.
    total_parameters: !!null    # note that for simple template matching, the "parameters"==the pixel count of the templates, plus 1 for each param such as thresholding. 
    # Approximate training time followed by the hardware used
    trainining_time: !!null
    # Model size in MB
    model_size: !!null


  # URL to the source code of the system [optional, highly recommended]
  source_code:   

  # List of external datasets used in the submission.
  # A previous DCASE development dataset is used here only as example! List only external datasets
  external_datasets:
    # Dataset name
    - name: !!null
      # Dataset access url
      url: !!null
      # Total audio length in minutes
      total_audio_length: !!null            # minutes

# System results 
results:
  # Full results are not mandatory, but for through analysis of the challenge submissions recommended.
  # If you cannot provide all result details, also incomplete results can be reported.
  validation_set:
    overall:
      F-score: 2.01 # percentile

    # Per-dataset
    dataset_wise:
      HV:
        F-score: 1.22 #percentile
      PB:
        F-score: 5.84 #percentile

Evaluation Metric

We implemented an event-based F-measure, macro-averaged evaluation metric. We use IoU followed by bipartite graph matching. The evaluation metric ignores the part of the file that contains the first five positive (POS) events and measure are estimated after the end time of the fifth positive event for each file. Furthermore, real-world datasets contain a small number of ambiguous or unknown labels marked as UNK in the annotation files provided. This evaluation metrics treats these separately during evaluation, so as not to penalise algorithms that can perform better than a human annotator. Final ranking of methods will be based on the overall F-measure for the whole of the evaluation set.

Results

Complete results and technical reports can be found in the results page

Complete results and technical reports can be found at results page

Baseline system

The following baseline is provided:

• Deep learning prototypical network (based on a 3-top ranked systems from task5 at the DCASE challenge 2022)

Baseline Results

Citation

If you are using/ referencing the baseline system, please cite the following paper:

@misc{liang2024mind,
    author = "Liang, Jinhua and Nolasco, Ines and Ghani, Burooj and Phan, Huy and Benetos, Emmanouil and Stowell, Dan",
    title = "{Mind the Domain Gap: a Systematic Analysis on Bioacoustic Sound Event Detection}",
    year = "2024",
    journal = "arXiv preprint arXiv:2403.18638",
    url = "https://arxiv.org/abs/2403.18638",
    eprint = "2403.18638",
    archiveprefix = "arXiv",
    primaryclass = "cs.SD"
}