Challenge has ended. Full results for this task can be found in the Results page.

Summary of task

Before Submission

• Participants build foley sound synthesis models that generate various sounds, for 7 sound classes we defined.
  • class index --> [Model] --> 100 sounds (4-second / mono / 16-bit / 22,050 Hz).
  • class:{"DogBark": 0, "Footstep": 1, "GunShot": 2, "Keyboard": 3, "MovingMotorVehicle": 4, "Rain": 5, "Sneeze_Cough": 6}
• On Track A, participants can use external data/models. On Track B, no external resources are allowed. One can submit for either or both.
• An entry consists of two items.
  • A yaml: In a pre-defined format that includes URL for a Colab notebook (which also follows a pre-defined format)
  • A pdf: Technical report

After Submission

• Submissions will be evaluated first by Frechet Audio Distance (FAD), then by subjective test.
• Participants are required to contribute to the subjective tests.
  • One evaluator per entry / About 2-3 hours of listening test / During the week of May 22-27.
• FAD decides four finalists. Subjective test decides final ranking among them (FAD score is ignored)

Description

Foley sound, in general, refers to sound effects that are created to convey (and sometimes enhance) the sounds produced by events occurring in a narrative (e.g. radio or film). Foley sounds are commonly added to multimedia to enhance the perceptual audio experience. This sound synthesis challenge requires the generation of original audio clips that represent a category of sound, such as footsteps.

Why is it an important task?

First, time-consuming post-production is inevitable to obtain a perfectly matched sound effect. By generating sound that belongs to a target sound category, Foley sound synthesis can make the workflow much more time and cost-effective. With the rise of virtual environments such as the metaverse, we expect a growing need for the automated generation of more and more complex and creative sound environments. Second, it can be utilized for dataset synthesis or augmentation for a wide variety of DCASE tasks including sound event detection (SED). SED has drawn great attention and synthesized datasets have been used already, e.g., URBAN-SED dataset. A high-quality Foley sound synthesis model could lead to development of better SED models.

Task setup

This task consists of generating sounds from two conditions. The first condition is the class id number, and the second condition is the number of the sounds to be generated. The data type of both conditions is an integer. The length of each generated audio should be 4 seconds.

The challenge has two subproblems - development of models with external resources (track A) and without external resources (track B). Participants are expected to submit a system for one of the two problems, and each problem is evaluated independently.

Audio dataset

Development Set

The development set is composed of audio samples from 3 datasets: UrbanSound8K, FSD50K, and BBC Sound Effects. Participants are not allowed to use these 3 datasets as external resources. The development set consists of 4,850 labeled sound excerpts, which are divided into 7 classes. All audio was converted to mono 16-bit 22,050 Hz audio. In addition, for consistency of the challenge, it was zero-padded or segmented to fit a length of 4 seconds. The 7 class IDs and the corresponding classes of audio are as follows.

Evaluation Set

The evaluation dataset has the same classes as the development dataset. The sample rate, the number of channels, and the bit depth of audio correspond to those of audio in the development dataset. The length of each audio is 4 seconds, and there are 100 audio samples per category. This dataset will be released after the end of the challenge.

Download

List of external data resources allowed (Track A):

For Track A, You can use external resources (except FSD50k and Urbansound8k), whether they are public or not. If you want to check the list of the audio dataset, you can refer to LAION-AI audio dataset

List of external data resources allowed (Track B):

Task rules

Track A

• Participants are allowed to use external resources, including public/private datasets and pre-trained models, except that no sounds may be used (for model training, etc.) from FSD50k or Urbansound8k (because those databases were used to create the eval set).
• Participants are not allowed to submit systems that duplicate any pre-existing sounds (on the web or in private databases). They also may not submit systems that simply apply alterations or audio effects to pre-existing sounds. The sounds must be unique and must be generated by the code supplied in the colab.
• When submitting a technical report, participants are required to specify all the external resources they used.
• Participants are allowed to submit a rule-based system that doesn't have trainable parameters (as long as only sounds from the development set are used).
• Participants are allowed to submit an ensemble system having multiple systems for each sound class.

Track B

• Participants are NOT allowed to use external resources.
• Participants are allowed to submit a rule-based system that doesn't have trainable parameters (as long as only sounds from the development set are used).
• Participants are allowed to submit an ensemble system having multiple systems for each sound class.

Submission

A submission would consist of the following items.

• Metadata file (*.yaml) with the pre-defined format
  • The URL for Colab notebook is included here.
  • Colab template
• Technical report (*.pdf)

The audio files will be generated by the organizers using the Colab notebook. On audio files: the duration, sample rate, bit depth, and the number of channels must follow those of the development set (4 seconds, 16 bit, 22,050 Hz, mono). The number of audio files generated per class should be 100. Participants need to declare in the metafile for which track the submission is. The technical report should contain a description, including the implementation details of the submission.

The confidential statement for the submitted Colab code and model checkpoint is included in the submission package.

Evaluation

Submissions will be evaluated by Frechet Audio Distance (FAD), followed by a subjective test.

Evaluation Metric

FAD, based on Frechet inception distance (FID) widely used metric for generative models, is a reference-free evaluation metric. Because FAD is calculated from sets of hidden representations of generated and real samples, it can be used even when the ground truth reference audio does not exist. FAD is calculated in the following sequence. First, audio representations are extracted from both evaluation datasets and generated samples. We use VGGish, a classification model that was trained on AudioSet. Second, each set of representations is fitted to a multivariate Gaussian distribution. Finally, the FAD score of the generated samples is calculated by the Frechet distance of these two distributions as follows:

\(F ({\mathcal N}_{r}, {\mathcal N}_{g}) = \| \mu_{r} - \mu_{g} \|^{2} + \mathrm{tr} ( \Sigma_{r} + \Sigma_{g} - 2 \sqrt{\Sigma_{r} \Sigma_{g}} )\)

where \({\mathcal N}_{r} (\mu_{r},\Sigma_{r})\) and \({\mathcal N}_{g} (\mu_{g},\Sigma_{g})\) are multivariate Gaussians of the VGGish embeddings, from the generated samples and evaluation set, respectively.

Subjective Test

Complementary to quantitative evaluation, we will evaluate the performance through a subjective test. Four submissions with the highest FAD scores will be evaluated. Generated audio excerpts for the subjective test are randomly sampled from the submitted data. Each challenge submission is evaluated by the organizers and one author of each of the other challenge submissions. The test will follow an incomplete strategy where the samples generated using an algorithm submitted by one team will not be evaluated by anyone of the team. The listening test will be separate for each category. Evaluators will listen to generated audio samples from challenge submissions, with real samples and audio samples generated by the baseline model. Audio fidelity, category suitability, and diversity will be evaluated. The weighted average of the three ratings are based on a ratio of audio quality : category fit : diversity that is 2:2:1. One evaluator from each entry must be available to spend 2-3 hours doing listening tests during the week of May 22-27.

Ranking

The final ranking is determined by the FAD score and the subjective tests. First, four submissions with the highest FAD scores are selected. They are then ranked, finally, by the subjective tests regardless of their FAD scores.

Results

Track A

Track B

In the case that multiple systems were submitted by one team, only the system with the highest FAD score per team was perceptually evaluated.

Complete results and technical reports can be found at results page.

Baseline system

The task organizers will provide a baseline system consisting of VQ-VAE, PixelSNAIL, and HiFi-GAN. We chose the baseline system to be a cascaded model, rather than an end-to-end one, due to its simplicity, explainability, and efficiency.

Repository

Detailed information can be found in the GitHub repository. The code of the baseline repository is mostly from this repository. If you use this code, please cite the task description paper, which will be published in May, and the following paper:

@article{Liu_MLP2022_01,
    author = "Liu, Xubo and Iqbal, Turab and Zhao, Jinzheng and Huang, Qiushi and Plumbley, Mark D. and Wang, Wenwu",
    title = "Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning",
    journal = "2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP)",
    year = "2021",
    pages = "1--6"
}

Parameters

• Audio features:
  • Log mel-band energies (80 bands), 22,050 Hz sampling frequency, frame length 1024 points with 25% hop size
• Neural network:
  • Input shape: 80 * 344 (4 seconds)
  • VQ-VAE encoder
    • Strided convolutional layer #1: 4 × 4 conv, ReLU, 4 × 4 conv, ReLU, 3 × 3 conv
    • Residual block #1: ReLU, 3 × 3 conv, ReLU, 1 × 1 conv
    • Strided convolutional layer #2: 2 × 2 conv, ReLU, 2 × 2 conv, ReLU, 3 × 3 conv
    • Residual block #2: ReLU, 3 × 3 conv, ReLU, 1 × 1 conv
    • Strided convolutional layer #3: 6 × 6 conv, ReLU, 6 × 6 conv, ReLU, 3 × 3 conv
    • Residual block #3: ReLU, 3 × 3 conv, ReLU, 1 × 1 conv
    • Strided convolutional layer #4: 8 × 8 conv, ReLU, 8 × 8 conv, ReLU, 3 × 3 conv
    • Residual block #4: ReLU, 3 × 3 conv, ReLU, 1 × 1 conv
  • VQ-VAE decoder
    • 3 × 3 conv
    • Residual block #1: ReLU, 3 × 3 conv, ReLU, 1 × 1 conv
    • Residual block #2: ReLU, 3 × 3 conv, ReLU, 1 × 1 conv
    • ReLU
    • Transposed convolutional block: 4 × 4 transposed conv, ReLU, 4 × 4 transposed conv
  • PixelSNAIL
    • Parameters inherited from the original github repo
  • HiFi-GAN
    • Parameters inherited from this github repo

FAD results for the development dataset

We evaluated the FAD score on the development dataset. In each category, we used 100 gound truth (natural) and 100 synthesized sounds. Participants can utilize this repo to compute the FAD score between the sounds generated by their system and those of the evaluation set.

Citation

If you are participating in this task or using the baseline code, please cite the following task description paper.

@article{Choi_arXiv2023_01,
    author = "Choi, Keunwoo and Im, Jaekwon and Heller, Laurie and McFee, Brian and Imoto, Keisuke and Okamoto, Yuki and Lagrange, Mathieu and Takamichi, Shinosuke",
    title = "Foley Sound Synthesis at the DCASE 2023 Challenge",
    journal = "In arXiv e-prints: 2304.12521",
    year = "2023"
}