Automatic creation of textual content descriptions for general audio signals.

Description

Automated audio captioning (AAC) is the task of general audio content description using free text. It is an inter-modal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. AAC methods can model concepts (e.g. "muffled sound"), physical properties of objects and environment (e.g. "the sound of a big car", "people talking in a small and empty room"), and high level knowledge ("a clock rings three times"). This modeling can be used in various applications, ranging from automatic content description to intelligent and content-oriented machine-to-machine interaction.

The task of AAC is a continuation of the AAC task from DCASE2021. As in DCASE2021, this year the task of AAC will allow the usage of any external data and/or pre-trained models. For example, participants are allowed to use other datasets for AAC or even datasets for sound event detection/tagging, acoustic scene classification, or datasets from any other task that might be deemed fit. Additionally, participants can use pre-trained models, such as (but not limited to) Word2Vec, BERT, and YAMNet, wherever they want in their model. Please see below for some recommendations for datasets and pre-tuned models.

This year, the task of Automated Audio Captioning is partially supported by:

• the European Union's Horizon 2020 research and innovation programme under grant agreement No 957337, project MARVEL.
• the ANR under Grant No ANR-18-CE23-0020, project LEAUDS.

Audio dataset

The AAC task in DCASE2022 will be using Clotho v2 dataset for the evaluation of the submissions. Though, participants can use any other dataset, from any other task, for the development of their methods. In this section, we will describe Clotho v2 dataset.

Clotho dataset

Clotho v2 is an extension of the original Clotho dataset (i.e. v1) and consists of audio samples of 15 to 30 seconds duration, each audio sample having five captions of eight to 20 words length. There is a total of 6972 (4981 from version 1 and 1991 from v2) audio samples in Clotho, with 34 860 captions (i.e. 6972 audio samples * 5 captions per each sample). Clotho v2 is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. The new data in Clotho v2 will not affect the splits used in order to assess the performance of methods using the previous version of Clotho (i.e. the evaluation and testing splits of Clotho V1). All audio samples are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing.

Clotho v2 has a total of around 4500 words and is divided in four splits: development, validation, evaluation, and testing. Audio samples are publicly available for all four splits, but captions are publicly available only for the development, validation, and evaluation splits. There are no overlapping audio samples between the four different splits and there is no word that appears in the evaluation, validation, or testing splits, and not appearing in the development split. Also, there is no word that appears in the development split and not appearing at least in one of the other three splits. All words appear proportionally between splits (the word distribution is kept similar across splits), i.e. 55% in the development, 15% in the and validation, 15% in the evaluation, and 15% in the testing split.

Words that could not be divided using the above scheme of 55-15-15-15 (e.g. words that appear only two times in the all four splits combined), appear at least one time in the development split and at least one time to one of the other three splits. This splitting process is similar to the one used for the previous version of Clotho. More detailed info about the splitting process can be found on the paper presenting Clotho, freely available online here and cited as:

@inproceedings{Drossos_2020_icassp,
    author = "Drossos, Konstantinos and Lipping, Samuel and Virtanen, Tuomas",
    title = "Clotho: an Audio Captioning Dataset",
    booktitle = "Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)",
    year = "2020",
    pages = "736-740",
    abstract = "Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online (https://zenodo.org/record/3490684)."
}

The data collection of Clotho received funding from the European Research Council, grant agreement 637422 EVERYSOUND.

Audio samples in Clotho

Audio samples have durations ranging from 10s to 30s, no spelling errors in the first sentence of the description on Freesound, good quality (44.1kHz and 16-bit), and no tags on Freesound indicating sound effects, music, or speech. Before extraction, all files were normalized and the preceding and trailing silences were trimmed.

The content of audio samples in Clotho greatly varies, ranging from ambiance in a forest (e.g. water flowing over some rocks), animal sounds (e.g. goats bleating), and crowd yelling or murmuring, to machines and engines operating (e.g. inside a factory) or revving (e.g. cars, motorbikes), and devices functioning (e.g. container with contents moving, doors opening/closing). For a thorough description how the audio samples are selected and filtered, you can check the paper that presents Clotho dataset.

In the following figure is the distribution of the duration of audio files in Clotho v2.

Captions in Clotho

The captions in the Clotho dataset range from 8 to 20 words in length, and were gathered by employing the crowdsourcing platform Amazon Mechanical Turk and a three-step framework. The three steps are:

1. audio description,
2. description editing, and
3. description scoring.

In step 1, five initial captions were gathered for each audio clip from distinct annotators. In step 2, these initial captions were edited to fix grammatical errors. Grammatically correct captions were instead rephrased, in order to acquire diverse captions for the same audio clip. In step 3, the initial and edited captions were scored based on accuracy, i.e. how well the caption describes the audio clip, and fluency, i.e. the English fluency in the caption itself. The initial and edited captions were scored by three distinct annotators. The scores were then summed together and the captions were sorted by the total accuracy score first, total fluency score second. The top five captions, after sorting, were selected as the final captions of the audio clip. More information about the caption scoring (e.g. scoring values, scoring threshold, etc.) is at the corresponding paper of the three-step framework.

We then manually sanitized the final captions of the dataset by removing apostrophes, making compound words consistent, removing phrases describing the content of speech, and replacing named entities. We used in-house annotators to replace transcribed speech in the captions. If the resulting caption were under 8 words, we attempt to find captions in the lower-scored captions (i.e. that were not selected in step 3) that still have decent scores for accuracy and fluency. If there were no such captions or these captions could not be rephrased to 8 words or less, the audio file was removed from the dataset entirely. The same in-house annotators were used to also replace unique words that only appeared in the captions of one audio clip. Since audio clips are not shared between splits, if there are words that appear only in the captions of one audio clip, then these words will appear only in one split.

A thorough description of the three-step framework can be found at the corresponding paper, freely available online here and cited as:

@inproceedings{Drossos_2019_dcase,
    Author = "Lipping, Samuel and Drossos, Konstantinos and Virtanen, Tuoams",
    title = "Crowdsourcing a Dataset of Audio Captions",
    year = "2019",
    booktitle = "Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop ({DCASE})",
    month = "Nov.",
    abstract = "Audio captioning is a novel field of multi-modal translation and it is the task of creating a textual description of the content of an audio signal (e.g. “people talking in a big room”). The creation of a dataset for this task requires a considerable amount of work, rendering the crowdsourcing a very attractive option. In this paper we present a three steps based framework for crowdsourcing an audio captioning dataset, based on concepts and practises followed for the creation of widely used image captioning and machine translations datasets. During the first step initial captions are gathered. A grammatically corrected and/or rephrased version of each initial caption is obtained in second step. Finally, the initial and edited captions are rated, keeping the top ones for the produced dataset. We objectively evaluate the impact of our framework during the process of creating an audio captioning dataset, in terms of diversity and amount of typographical errors in the obtained captions. The obtained results show that the resulting dataset has less typographical errors than the initial captions, and on average each sound in the produced dataset has captions with a Jaccard similarity of 0.24, roughly equivalent to two ten-word captions having in common four words with the same root, indicating that the captions are dissimilar while they still contain some of the same information.",
    url = "https://arxiv.org/abs/1907.09238"
}

Development, validation, and evaluation datasets of Clotho

Clotho v2 is divided into a development split of 3839 audio clips with 19195 captions, a validation split of 1045 audio clips with 5225 captions, an evaluation split of 1045 audio clips with 5225 captions, and a testing split of 1043 audio clips with 5215 captions. These splits are created by first constructing the sets of unique words of the captions of each audio clip. These sets of words are combined to form the bag of words of the whole dataset, from which we can derive the frequency of a given word. With the unique words of audio files as classes, we use multi-label stratification. More information on the splits of Clotho can be found in the corresponding paper.

The name of the splits for Clotho differ from the DCASE terminology. To avoid confusion for participants, the correspondence of splits between Clotho and DCASE challenge is:

Clotho development and validation splits are meant for optimizing audio captioning methods. The performance of the audio captioning methods can then be assessed (e.g. for reporting results in a conference or journal paper) using Clotho evaluation split. Clotho testing split is meant only for usage in scientific challenges, e.g. DCASE challenge. For the rest of this text, the DCASE challenge terminology will be used. For differentiating between Clotho development and evaluation, the terms development-training, development-validation, and development-testing will be used, wherever necessary. Development-training refers to Clotho development split, development-validation refers to Clotho validation split, and development-testing refers to Clotho evaluation split.

Clotho download

DCASE Development split of Clotho can be found at the online Zenodo repository. Make sure that you download Clotho v2.1, as there were some minor fixes in the dataset (fixing of file naming and some corrupted files).

Clotho v2.1 audio captioning dataset (6.5 GB)
version 2.1

Development-training data are:

• clotho_audio_development.7z: The development-training audio clips.
• clotho_captions_development.csv: The captions of the development-training audio clips.
• clotho_metadata_development.csv: The meta-data of the development-training audio clips.

Development-validation data are:

• clotho_audio_validation.7z: The development-validation audio clips.
• clotho_captions_validation.csv: The captions of the development-validation audio clips.
• clotho_metadata_validation.csv: The meta-data of the development-validation audio clips.

Development-testing data are:

• clotho_audio_evaluation.7z: The development-testing audio clips.
• clotho_captions_evaluation.csv: The captions of the development-testing audio clips.
• clotho_metadata_evaluation.csv: The meta-data of the development-testing audio clips.

DCASE Evaluation split of Clotho (i.e. Clotho-testing) can be found at the online Zenodo repository.

Task 6 Evaluation dataset (1.2 GB)
version 1

Evaluation data are:

• clotho_audio_test.7z: The DCASE evaluation (Clotho-testing) audio clips.
• clotho_metadata_test.csv: The meta-data of the DCASE evaluation (Clotho-testing) audio clips, containing only authors and licence.

NOTE: Participants are strongly prohibited to use any additional information for the DCASE evaluation (testing) of their method, apart the provided audio files from the DCASE Evaluation split.

In this year's edition, we are introducing supplementary data for evaluation. The objective is to provide contrastive analysis and performance measures on out-of-domain audio and captions. This supplementary subset Clotho-analysis can be obtained on Zenodo:

Task 6a Analysis dataset (12.7 GB)
version 1

Note that results on this new subset will not affect submission rankings.

Other suggested resources

Participants are allowed to use any available resource (e.g. pre-trained models, other datasets) for developing their AAC methods. In order to help participants discover and use other resources, we have put-up a GitHub repository with AAC resources.

Participants are encouraged to browse the information at the GitHub repository with the other suggested AAC resources and use any resource that they deem proper. Additionally, participants are allowed to use any other resource, no matter if it is listed or not at the information of the GitHub repository.

Task setup

Participants are free to use any dataset and pre-trained model, in order to develop their AAC method(s). The assessment of the methods will be performed using the withheld split of Clotho, which is the same as last year, offering direct comparison with the results of the AAC task at DCASE2020 and DCASE2021.

Task rules

Participants are allowed to:

• Use external data (e.g. audio files, text, annotations).
• Use pre-trained models (e.g. text models like Word2Vec, audio tagging models, sound event detection models).
• Augment the development dataset (i.e. development-training and development-testing) with or without the use of external data.
• Use all the available metadata provided, but they must explicitly state it and indicate if they use the available metadata. This will not affect the rating of their method.

Participants are NOT allowed to:

• Make subjective judgments of the DCASE evaluation (testing) data, nor to annotate it.
• Use additional information of the DCASE evaluation (testing) for their method, apart from the provided audio files from the DCASE Evaluation split.

Submission

All participants should submit:

• the output of their audio captioning with the Clotho-testing split (*.csv file),
• the output of their audio captioning with the Clotho-analysis split (*.csv file),
• metadata for their submission (*.yaml file), and
• a technical report for their submission (*.pdf file).

We allow up to 4 system output submissions per participant/team. For each system, metadata should be provided in a separate file, containing the task specific information. All files should be packaged into a zip file for submission. Please make a clear connection between the system name in the submitted metadata (the .yaml file), submitted system output (the .csv file), and the technical report (the .pdf file)! For indicating the connection of your files, you can consider using the following naming convetion:

<author>_<institute>_task6a_submission_<submission_index>_<testing_output or analysis_output or metadata or report>.<csv or yaml or pdf>

For example:

Gontier_INR_task6a_submission_1_testing_output.csv
Gontier_INR_task6a_submission_1_analysis_output.csv
Gontier_INR_task6a_submission_1_metadata.yaml
Gontier_INR_task6a_submission_1_report.pdf

The field <submission_index> is to differentiate your submissions in case that you have multiple submissions.

System output file

The system output file should be a *.csv file, and should have the following two columns:

1. file_name, which will contain the file name of the audio file.
2. caption_predicted, which will contain the output of the audio captioning method for the file with file name as specified in the file_name column.

For example, if a file has a file name test_0001.wav and the predicted caption of the audio captioning method for the file test_0001.wav is hello world, then the CSV file should have the entry:

    file_name      caption_predicted
        .               .
        .               .
        .               .
   test_0001.wav    hello world

Please note: automated procedures will be used for the evaluation of the submitted results. Therefore, the column names should be exactly as indicated above.

Two output files are expected for each submitted system: one for the DCASE evaluation subset (Clotho-testing), and one for the Clotho-analysis subset.

Metadata file

For each system, metadata should be provided in a separate file. The file format should be as indicated below.

# Submission information for task 6, subtask A
submission:
  # Submission label
  # Label is used to index submissions.
  # Generate your label following way to avoid
  # overlapping codes among submissions:
  # [Last name of corresponding author]_[Abbreviation of institute of the corresponding author]_task[task number]_[index number of your submission (1-4)]
  label: gontier_inr_task6a_1
  #
  # Submission name
  # This name will be used in the results tables when space permits
  name: DCASE2022 baseline system
  #
  # Submission name abbreviated
  # This abbreviated name will be used in the results table when space is tight.
  # Use maximum 10 characters.
abbreviation: Baseline

  # Authors of the submitted system. Mark authors in
  # the order you want them to appear in submission lists.
  # One of the authors has to be marked as corresponding author,
  # this will be listed next to the submission in the results tables.
  authors:
    # First author
    - lastname: Gontier
      firstname: Felix
      email: felix.gontier@inria.fr               # Contact email address
      corresponding: true                         # Mark true for one of the authors

      # Affiliation information for the author
      affiliation:
        abbreviation: INRIA
        institute: INRIA
        department: Multispeech            # Optional
        location: Nancy, France

    # Second author...

# System information
system:
  # System description, meta data provided here will be used to do
  # meta analysis of the submitted system.
  # Use general level tags, when possible use the tags provided in comments.
  # If information field is not applicable to the system, use "!!null".
  description:

    # Audio input / sampling rate
    # e.g. 16kHz, 22.05kHz, 44.1kHz, 48.0kHz
    input_sampling_rate: 16kHz

    # Acoustic representation
    # Here you should indicate what can or audio representation
    # you used. If your system used hand-crafted features (e.g.
    # mel band energies), then you can do:
    #
    # `acoustic_features: mel energies`
    #
    # Else, if you used some pre-trained audio feature extractor, 
    # you can indicate the name of the system, for example:
    #
    # `acoustic_features: audioset`
    acoustic_features: VGGish

    # Word embeddings
    # Here you can indicate how you treated word embeddings.
    # If your method learned its own word embeddings (i.e. you
    # did not used any pre-trained word embeddings) then you can
    # do:
    #
    # `word_embeddings: learned`
    #  
    # Else, specify the pre-trained word embeddings that you used
    # (e.g. Word2Vec, BERT, etc).
    word_embeddings: BART

    # Data augmentation methods
    # e.g. mixup, time stretching, block mixing, pitch shifting, ...
    data_augmentation: !!null

    # Method scheme
    # Here you should indicate the scheme of the method that you
    # used. For example:
    machine_learning_method: encoder-decoder

    # Learning scheme
    # Here you should indicate the learning scheme. 
    # For example, you could specify either
    # supervised, self-supervised, or even 
    # reinforcement learning. 
    learning_scheme: supervised

    # Ensemble
    # Here you should indicate if you used ensemble
    # of systems or not.
    ensemble: No

    # Audio modelling
    # Here you should indicate the type of system used for
    # audio modelling. For example, if you used some stacked CNNs, then
    # you could do:
    #
    # audio_modelling: cnn
    #
    # If you used some pre-trained system for audio modelling,
    # then you should indicate the system used (e.g. COALA, COLA,
    # transfomer).
    audio_modelling: transformer

    # Word modelling
    # Similarly, here you should indicate the type of system used
    # for word modelling. For example, if you used some RNNs,
    # then you could do: 
    #
    # word_modelling: rnn
    #
    # If you used some pre-trained system for word modelling,
    # then you should indicate the system used (e.g. transfomer).
    word_modelling: transformer

    # Loss function
    # Here you should indicate the loss fuction that you employed.
    loss_function: crossentropy

    # Optimizer
    # Here you should indicate the name of the optimizer that you
    # used. 
    optimizer: adamw

    # Learning rate
    # Here you should indicate the learning rate of the optimizer
    # that you used.
    leasrning_rate: 1e-5

    # Gradient clipping
    # Here you should indicate if you used any gradient clipping. 
    # You do this by indicating the value used for clipping. Use
    # 0 for no clipping.
    gradient_clipping: 0

    # Gradient norm
    # Here you should indicate the norm of the gradient that you
    # used for gradient clipping. This field is used only when 
    # gradient clipping has been employed.
    gradient_norm: !!null

    # Metric monitored
    # Here you should report the monitored metric
    # for optimizing your method. For example, did you
    # monitored the loss on the validation data (i.e. validation
    # loss)? Or you monitored the SPIDEr metric? Maybe the training
    # loss?
    metric_monitored: validation_loss

  # System complexity, meta data provided here will be used to evaluate
  # submitted systems from the computational load perspective.
  complexity:
    # Total amount of parameters used in the acoustic model.
    # For neural networks, this information is usually given before training process
    # in the network summary.
    # For other than neural networks, if parameter count information is not directly
    # available, try estimating the count as accurately as possible.
    # In case of ensemble approaches, add up parameters for all subsystems.
    # In case embeddings are used, add up parameter count of the embedding
    # extraction networks and classification network
    # Use numerical value (do not use comma for thousands-separator).
    total_parameters: 140000000

  # List of external datasets used in the submission.
  # Development dataset is used here only as example, list only external datasets
  external_datasets:
    # Dataset name
    - name: Clotho

      # Dataset access url
      url: https://doi.org/10.5281/zenodo.3490683

      # Has audio:
      has_audio: Yes

      # Has images
      has_images: No

      # Has video
      has_video: No

      # Has captions
      has_captions: Yes

      # Number of captions per audio
      nb_captions_per_audio: 5

      # Total amount of examples used
      total_audio_length: 24430

      # Used for (e.g. audio_modelling, word_modelling, audio_and_word_modelling)
      used_for: audio_and_word_modelling

  # URL to the source code of the system [optional]
      source_code: https://github.com/audio-captioning/dcase-2021-baseline

# System results
results:
  development_evaluation:
    # System results for development evaluation split.
    # Full results are not mandatory, however, they are highly recommended
    # as they are needed for through analysis of the challenge submissions.
    # If you are unable to provide all results, also incomplete
    # results can be reported.
    bleu1: 0.555
    bleu2: 0.358
    bleu3: 0.239
    bleu4: 0.156
    rougel: 0.364
    meteor: 0.164
    cider: 0.358
    spice: 0.109
    spider: 0.233

Open and reproducible research

Finally, for supporting open and reproducible research, we kindly ask from each participant/team to consider making available the code of their method (e.g. in GitHub) and pre-trained models, after the challenge is over.

Evaluation

The submitted systems will be evaluated according to their performance on the withheld evaluation split. For the evaluation, the captions will not have any punctuation and all letters will be small case. Therefore, the participants are advised to optimized their methods using captions which do not have punctuation and all letters are small case.

All of the following metrics will be reported for every submitted method:

1. BLEU₁
2. BLEU₂
3. BLEU₃
4. BLEU₄
5. ROUGE_L
6. METEOR
7. CIDEr
8. SPICE
9. SPIDEr

Ranking of the methods will be performed according to the SPIDEr metric, which is a combination of CIDEr and SPICE. Specifically, the evaluation will be performed based on the average of CIDEr and SPICE, referred to as SPIDEr and shown to have the combined benefits of CIDEr and SPICE. More information is available on the corresponding paper, available online here.

For a brief introduction and more pointers on the above-mentioned metrics, you can refer to the original paper of audio captioning:

@inproceedings{Drossos_2017_waspaa,
    Author = "Drossos, Konstantinos and Adavanne, Sharath and Virtanen, Tuomas",
    title = "Automated Audio Captioning with Recurrent Neural Networks",
    booktitle = "IEEE Workshop on Applications of Signal Processing to Audio and Acoustics ({WASPAA})",
    address = "New Paltz, New York, U.S.A.",
    year = "2017",
    month = "Oct.",
    abstract = "We present the first approach to automated audio captioning. We employ an encoder-decoder scheme with an alignment model in between. The input to the encoder is a sequence of log mel-band energies calculated from an audio file, while the output is a sequence of words, i.e. a caption. The encoder is a multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a multi-layered GRU with a classification layer connected to the last GRU of the decoder. The classification layer and the alignment model are fully connected layers with shared weights between timesteps. The proposed method is evaluated using data drawn from a commercial sound effects library, ProSound Effects. The resulting captions were rated through metrics utilized in machine translation and image captioning fields. Results from metrics show that the proposed method can predict words appearing in the original caption, but not always correctly ordered.",
    url = "https://arxiv.org/abs/1706.10006"
}

In this edition of the task, we are also introducing more recent model-based metrics for captioning. Note that these additional metrics are contrastive, and will not affect rankings of the submissions.

Results

Complete results and technical reports can be found in the results page

Baseline system

To provide a starting point and some initial results for the challenge, there is a baseline system for the task of automated audio captioning. The baseline system is freely available online, is a sequence-to-sequence model, and is implemented using PyTorch.

The baseline system consists of three parts:

1. the caption evaluation part,
2. the dataset pre-processing/feature extraction part, and
3. the deep neural network (DNN) method part

You can find the baseline system of automated audio captioning task on GitHub.

Caption evaluation

Caption evaluation is performed using a version of the caption evaluation tools used for the MS COCO challenge. This version of the code has been updated in order to be compliant with Python 3.6 and above and with the needs of the automated audio captioning task. The code can for the evaluation is included in the baseline system, but also can be found online.

Dataset pre-processing/feature extraction

Clotho examples are available as WAV and CSV files. For use in the development of an audio captioning method, features have to be extracted from the audio clips (i.e. WAV files) and the captions in the CSV files have to be pre-processed (e.g. punctuation removal). The extracted features and processed words then have to be matched as input-output pairs.

In the baseline system there is code that implements the above. This code is also available as stand-alone, in the following repository:

Deep neural network (DNN) method

Lastly, the baseline is a sequence-to-sequence system consisting of an encoder and a decoder, and is implemented with the Pytorch HuggingFace framework. Specifically, it is a standard sequence-to-sequence transformer with 6 encoder and 6 decoder layers. The encoder takes as an input embeddings from a pre-trained VGGish model with dimension 128, transformed by an affine layer to embeddings of dimension 768, and outputs a sequence of embeddings of the same length as the input sequence. Each transformer layer of the encoder outputs 768 features for each timestep of the input representation.

The decoder employs encoder outputs to generate the caption in an autoregressive manner. To do so, previously generated words are tokenized and transformed into embeddings as inputs to the decoder. In the baseline model, the tokenizer is pre-trained with a byte-pair-encoding process, i.e. each token corresponds to a sub-word from the model vocabulary instead of a full English word. This tokenizer has a vocabulary size of 50265 tokens. Each token in the past sequence is then associated to a feature vector through an embedding map, and input to the decoder. Each layer of the decoder attends on the previously generated tokens with self-attention, as well as on the entire encoder output sequence with cross-attention. The embedding dimension of each decoder layer is 768. Lastly, a classifier head consisting of one linear layer with softmax activation outputs a probability for each token of the vocabulary.

At evaluation, generation is performed through beam search, although greedy decoding is also provided in the code.

Hyper-parameters

Feature extraction and caption processing: The baseline system uses the following hyper-parameters for the feature extraction:

• Pre-trained VGGish embeddings of dimension 128
• Windows of 1s without overlap

Captions are pre-processed according to the following:

• Removal of all punctuation
• All letters to small case
• Tokenization
• Pre-pending of start-of-sentence token (i.e. <s>)
• Appending of end-of-sentence token (i.e. </s>)

Neural network and optimization hyper-parameters: The deep neural network used in the baseline system has the following hyper-parameters:

• One affine layer with no activation function, which transforms 128-dimensional VGGish features into 768-dimensional embeddings.
• Six transformer encoder layers with 768 output embeddings, each composed of:
  • A multi-head self-attention layer with 12 heads, followed by a residual connection to the layer input, and layer normalization.
  • Two affine layers of dimension 3072 and 768 respectively, followed by a residual connection to the output of self-attention, and layer normalization.
• Six transformer decoder layers with 768 output embeddings, each composed of:
  • A multi-head self-attention layer with 12 heads, followed by a residual connection to the layer input, and layer normalization.
  • A multi-head cross-attention layer attending on the encoder outputs with 12 heads, followed by a residual connection to the self-attention output, and layer normalization.
  • Two affine layers of dimension 3072 and 768 respectively, followed by a residual connection to the output of cross-attention, and layer normalization.
• An affine layer with 50265 output features and softmax activation, which predicts tokens from the output of the last decoder layer.

Parameter optimization is performed using AdamW on the cross-entropy loss for 20 epochs, with a batch size of 4 examples and gradient accumulation of 2 steps (equivalent to a batch size of 8). The learning rate is 10^-5. The loss on the validation set is evaluated every 1000 iterations, and the checkpoint corresponding to the lowest validation loss is retained for inference. Training takes about 2 hours on a single GPU (GTX 1080ti).

All input audio features and captions in a batch are padded to a fixed length for convenient batching. Specifically, the input audio features are padded with zero vectors, and tokenized captions are padded with a special <pad> token. For both audio and token input sequences, an attention mask is provided to the model in order to ignore padded elements.

Results for the development dataset

The results of the baseline system for the development dataset of Clotho v2.1 are:

The pre-trained weights for the DNN of the baseline system yielding the above results are freely available on Zenodo:

Baseline system pre-trained weights (16.9 MB)
version 1

Citations

If you participating in this task, you might want to check the following papers. If you find a paper that had to be cited here but it is not (e.g. a paper for some of the suggested resources that is missing), please contact us and report it to us.

• The initial publication on audio captioning:

@inproceedings{Drossos_2017_waspaa,
    Author = "Drossos, Konstantinos and Adavanne, Sharath and Virtanen, Tuomas",
    title = "Automated Audio Captioning with Recurrent Neural Networks",
    booktitle = "IEEE Workshop on Applications of Signal Processing to Audio and Acoustics ({WASPAA})",
    address = "New Paltz, New York, U.S.A.",
    year = "2017",
    month = "Oct.",
    abstract = "We present the first approach to automated audio captioning. We employ an encoder-decoder scheme with an alignment model in between. The input to the encoder is a sequence of log mel-band energies calculated from an audio file, while the output is a sequence of words, i.e. a caption. The encoder is a multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a multi-layered GRU with a classification layer connected to the last GRU of the decoder. The classification layer and the alignment model are fully connected layers with shared weights between timesteps. The proposed method is evaluated using data drawn from a commercial sound effects library, ProSound Effects. The resulting captions were rated through metrics utilized in machine translation and image captioning fields. Results from metrics show that the proposed method can predict words appearing in the original caption, but not always correctly ordered.",
    url = "https://arxiv.org/abs/1706.10006"
}

• The three-step framework, employed for collecting the annotations of Clotho (if you use the three-step framework, consider citing the paper):

@inproceedings{Drossos_2019_dcase,
    Author = "Lipping, Samuel and Drossos, Konstantinos and Virtanen, Tuoams",
    title = "Crowdsourcing a Dataset of Audio Captions",
    year = "2019",
    booktitle = "Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop ({DCASE})",
    month = "Nov.",
    abstract = "Audio captioning is a novel field of multi-modal translation and it is the task of creating a textual description of the content of an audio signal (e.g. “people talking in a big room”). The creation of a dataset for this task requires a considerable amount of work, rendering the crowdsourcing a very attractive option. In this paper we present a three steps based framework for crowdsourcing an audio captioning dataset, based on concepts and practises followed for the creation of widely used image captioning and machine translations datasets. During the first step initial captions are gathered. A grammatically corrected and/or rephrased version of each initial caption is obtained in second step. Finally, the initial and edited captions are rated, keeping the top ones for the produced dataset. We objectively evaluate the impact of our framework during the process of creating an audio captioning dataset, in terms of diversity and amount of typographical errors in the obtained captions. The obtained results show that the resulting dataset has less typographical errors than the initial captions, and on average each sound in the produced dataset has captions with a Jaccard similarity of 0.24, roughly equivalent to two ten-word captions having in common four words with the same root, indicating that the captions are dissimilar while they still contain some of the same information.",
    url = "https://arxiv.org/abs/1907.09238"
}

• The Clotho dataset (if you use Clotho consider citing the Clotho paper):

@inproceedings{Drossos_2020_icassp,
    author = "Drossos, Konstantinos and Lipping, Samuel and Virtanen, Tuomas",
    title = "Clotho: an Audio Captioning Dataset",
    booktitle = "Proc. IEEE Int. Conf. Acoustic., Speech and Signal Process. (ICASSP)",
    year = "2020",
    pages = "736-740",
    abstract = "Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online (https://zenodo.org/record/3490684)."
}