Sound Event Detection System For DCASE 2021 Challenge

Bajzik, Jakub
University of Zilina, Department of Mechatronics and Electronics, Žilina 010 26, Slovak Republic

Abstract

This paper presents the systems proposal for the DCASE 2021 challenge Task 4 (Sound event detection and separation in domestic environments). The aim is to provide the event time localization timestamps in addition to event class probabilities. In this paper, the two systems are proposed. System 1 is a convolutional neural network trained for sound event classification using only weakly labeled and unlabeled data. The strong labels are obtained using the class activation mapping technique. System 1 does not reach the baseline performance. System 2 is the convolutional neural network and recurrent neural network which uses the class activation mapping technique as a part of the attention mechanism to increase the baseline performance. The second model was trained using weakly labeled, strongly labeled, and unlabeled data. Both architectures are based on the Mean Teacher baseline system 2021.

System characteristics

Optimizing Temporal Resolution Of Convolutional Recurrent Neural Networks For Sound Event Detection

Boes, Wim and Van Hamme, Hugo
ESAT, KU Leuven, Leuven, Belgium

Abstract

In this technical report, the systems we submitted for subtask 4 of the DCASE 2021 challenge, regarding sound event detection, are described in detail. These models are closely related to the baseline provided for this problem, as they are essentially convolutional recurrent neural networks trained in a mean teacher setting to deal with the heterogeneous annotation of the supplied data. However, the time resolution of the predictions was adapted to deal with the fact that these systems are evaluated using two intersection-based metrics involving different needs in terms of temporal localization. This was done by optimizing the pooling operations. For the first of the defined evaluation scenarios, imposing relatively strict requirements on the temporal localization accuracy, our best model achieved a PSDS score of 0.3609 on the validation data. This is only marginally better than the performance obtained by the baseline system (0.342): The amount of pooling in the baseline network already turned out to be optimal, and thus, no substantial changes were made, explaining this result. For the second evaluation scenario, imposing relatively lax restrictions on the localization accuracy, our best-performing system achieved a PSDS score of 0.7312 on the validation data. This is significantly better than the performance obtained by the baseline model (0.527), which can effectively be attributed to the changes that were applied to the pooling operations of the network.

System characteristics

Convolution-Augmented Conformer For Sound Event Detection

Chen, YuHang
Royal Flush, 18, Tongshun Street, Wuchang Street, Yuhang District. HangZhou 310000, CHINA

Abstract

In this technical report, we describe our submission system for DCASE2021 Task4: sound event detection and separation in domestic environments. Our model employs conformer blocks, which combine the self-attention and depth-wise convolution networks, to efficiently capture the global and local context information of an audio feature sequence. In addition to this novel architecture, we further improve the performance by utilizing a mean teacher semi-supervised learning technique, data augmentation for each sound event class. We demonstrate that the proposed method achieves the PSDS-1 and PSDS-2 score of 34%, 55.7% on the validation set, outperforming that of the baseline score.

System characteristics

Multi-Resolution Mean Teacher For DCASE 2021 Task 4

de Benito-Gorron, Diego and Segovia, Sergio and Ramos, Daniel and T. Toledano, Doroteo
AUDIAS Research Group, Universidad Autónoma de Madrid, Calle Francisco Tomás y Valiente, 11, 28049 Madrid, Spain

Abstract

This technical report describes our participation in DCASE 2021 Task 4: Sound event detection and separation in domestic environments. Aiming to take advantage of the different lengths and spectral characteristics of each target category, we follow the multiresolution feature extraction approach that we proposed for last year’s edition. It is found that each one of the proposed Polyphonic Sound Detection Score (PSDS) scenarios benefits from either a higher temporal resolution or a higher frequency resolution. Furthermore, combining several time-frequency resolutions via model fusion is able to improve the PSDS results in both scenarios.

System characteristics

The Smallrice Submission To The Dcase2021 Task 4 Challenge: A Lightweight Approach For Semi-Supervised Sound Event Detection With Unsupervised Data Augmentation

Dinkel, Heinrich and Cai, Xinyu and Yan, Zhiyong and Wang, Yongqing and Zhang, Junbo and Wang, Yujun
Xiaomi Corporation, Beijing, China

Abstract

This paper describes our submission to the DCASE 2021 challenge. Different from the baseline and most other approaches, our work focuses on training a lightweight and well-performing model which can be used in real-world applications. Compared to the baseline, our model only contains 600k (15 %) parameters, resulting in a size of 2.7 Mb on disk, making it viable for applications on low-resource devices such as mobile phones. Our model is trained using unsupervised data augmentation as its consistency criterion, which we show can achieve competitive performance to the more common mean teacher paradigm. Our submitted results on the validation set result in a single model peak performance of 36.91 PSDS-1 and 57.17 PSDS2, outperforming the baseline by an absolute of 2.7 and 5.0 points respectively. Notably our approach achieves an Event-F1 score on the development set of 39.29 without post-processing. The best submitted ensemble system using a 4-way fusion achieves a PSDS-1 of 38.23 and PSDS-2 of 62.29 on the validation dataset.

System characteristics

Self-Trained Audio Tagging And Sound Event Detection In Domestic Environments

Ebbers, Janek Haeb-Umbach, Reinhold
Paderborn University, Department of Communications Engineering, Paderborn, Germany

Abstract

In this report we present our system for the Detection and Classification of Acoustic Scenes and Events (DCASE) 2021 Challenge Task 4: Sound Event Detection and Separation in Domestic Environments. Our presented solution is an advancement of our system used in the previous edition of the task.We use our previously proposed forward-backward convolutional recurrent neural network (FBCRNN) for tagging and pseudo labeling and tag-conditioned sound event detection (SED) models which are trained using the strong pseudo labels provided by the FBCRNN. Our advancement over our previous model is threefold. Firstly, we introduce a strong label loss in the objective of the FBCRNN to take advantage of the strongly labeled synthetic data during training, which leads to both better tagging and detection performance. Secondly, we perform multiple iterations of self-training for both the FBCRNN and tag-conditioned SED models. Thirdly, while we used only tag-conditioned CNNs as our SED model in the last edition we here explore sophisticated SED model architectures, namely, tag-conditioned bidirectional CRNNs and tag-conditioned bidirectional convolutional transformer neural networks (CTNNs) and combine them. With scenario and class dependent tuning of median filter lengths for post-processing, our final SED model, consisting of 6 submodels (2 of each architecture), is able to achieve validation poly-phonic sound event detection scores (PSDS) of 0.454 for scenario 1 and 0.758 for scenario 2 as well as a collar-based F1-score of 0.602 outperforming the baselines and our model from the last edition by far. Source code will be made publicly available at https://github.com/fgnt/pb_sed.

System characteristics

Improved Pseudo-Labeling Method For Semi-Supervised Sound Event Detection

Gong, Yaguang and Li, Changlong and Wang, Xintian and Ma, Lu and Yang, Song and Wu, Zhongqin Wu
TAL Education Group, China

Abstract

This report illustrates a framework for the DCASE2021 task4 - Sound Event Detection. The proposed framework is built on the pseudo-labeling method widely applied for semi-supervised learning(SSL) tasks. The proposed method synthesizes weak pseudo-labels for the large amount of unlabeled data by utilizing the model’s predictions on weakly augmented spectrograms. Weak pseudo-labels are then used as supervision for strongly augmented spectrograms of the same sample. Along to this main contribution, this work introduces data augmentation techniques including random frequency masking and time shifting, training techniques such as class-specific weighted loss, and model ensemble techniques. Experimental results demonstrate that the proposed method achieves PSDS of 0.407/0.653(scenario1/scenario2) on the validation set, which presents superior performance against the baseline score 0.342/0.527.

System characteristics

Task Aware Sound Event Detection Based On Semi-Supervised CRNNWith Skip Connections: DCASE 2021 Challenge, Task 4

Hafsati, Mohammed and Bentounes, Kamil
¹Beijing Kuaishou Technology Co., Ltd, China ²The State Key Laboratory of Automotive Satefy and Energy, Tsinghua University, Beijing, China

Abstract

Sound Event Detection (SED) is the task of classifying different sounds occurring in a recorded environment and their onset and offset times. This assignment is the primary goal of the fourth task of the DCASE challenge using some strongly labeled, partially labeled, and unlabeled datasets. In this paper, we describe our submitted approach for this challenge. Our neural network is based on sequential convolutional neural networks with skipping some layers and a recurrent neural network. To overcome the challenge of using unlabeled data, we used semi-supervised learning, and to improve the performance further, we propose to use data augmentation techniques. With our model, we can slightly outperform the baseline with fewer filters and therefore fewer parameters. Moreover, similar amount of parameters as the baseline, we significantly outperform it.

System characteristics

Self-Training With Noisy Student Model And Semi-Supervised Loss Function For DCASE 2021 Challenge Task 4

Kim, Nam Kyun ¹ and Kim, Hong Kook ^1,2
1School of Electrical Engineering and Computer Science, 123 Cheomdangwagi-ro, Gwangju 61005, Republic of Korea ² AI Graduate School Gwangju Institute of Science and Technology, 123 Cheomdangwagi-ro, Gwangju 61005, Republic of Korea

Abstract

This report proposes a polyphonic sound event detection (SED) method for the DCASE 2021 Challenge Task 4. The proposed SED model consists of two stages: a mean-teacher model for providing target labels regarding weakly labeled or unlabeled data and a self-training-based noisy student model for predicting strong labels for sound events. The mean-teacher model, which is based on the residual convolutional recurrent neural network (RCRNN) for the teacher and student model, is first trained using all the training data from a weakly labeled dataset, an unlabeled dataset, and a strongly labeled synthetic dataset. Then, the trained mean-teacher model predicts the strong label to each of the weakly labeled and unlabeled datasets, which is brought to the noisy student model in the second stage of the proposed SED model. Here, the structure of the noisy student model is identical to the RCRNN-based student model of the mean-teacher model in the first stage. Then, it is self-trained by adding feature noises, such as time-frequency shift, mixup, SpecAugment, and dropout-based model noise. In addition, a semi-supervised loss function is applied to train the noisy student model, which acts as label noise injection. The performance of the proposed SED model is evaluated on the validation set of the DCASE 2021 Challenge Task 4, and then, several ensemble models that combine five-fold validation models with different hyperparameters of the semi-supervised loss function are finally selected as our final models.

System characteristics

Sound Event Detection Based On Self-Supervised Learning Of Wav2vec 2.0

Koo, Hyejin¹ and Park, Hyung-Min¹ and Park, Jonghyeon² and Oh, Myungwoo^2
1Dept. of Electronic Engineering, Sogang University, Seoul 04107, South Korea, ²NAVER Corp. Gyeonggi-do 13561, South Korea

Abstract

In this report, we present our system for DCASE2021 Task4: Sound Event Detection (SED) and Separation in Domain Environments. This task evaluates how to capture information of SED with a relatively small amount of labeled data in addition to lots of unlabeled data. We apply wav2vec 2.0 on the SED for the first time. Even though wav2vec 2.0 pre-training using the DCASE2021 Taksk4 dataset spends long time to train audio representations, the presented model achieved higher intersection F1 and PSDS2. The baseline’s mean-teacher model and dataset was used to compare wav2vec 2.0 and log-mel features. Under the same conditions, we present how wav2vec 2.0 features work on the SED task.

System characteristics

Adaptive Focal Loss With Data Augmentation For Semi-Supervised Sound Event Detection

Liang, Yunhao and Tang, Tiantian and Long, Yanhua
Shanghai Normal University, Shanghai, China

Abstract

In this technical report, we describe our submission system for DCASE2021 Task4: sound event detection and separation in domestic environments. In our submissions, two different deep models are investigated. The first one is a mean-teacher model with convolutional recurrent neural network (CRNN). The second one is a joint framework with adaptive focal loss based on the Guided Learning architecture. To improve the performance of system, we propose to use various methods such as the specaugment data augmentation method, adaptive focal loss, event specific post-processing. To combine sound separation with sound event detection, we train models using the outputs of the sound separation baseline system. We demonstrate that the proposed method achieves the event-based macro F1 score of 44.4%, 0.428 in PSDS1 and 0.736 in PSDS2 on the validation set.

System characteristics

Combined Sound Event Detection And Sound Event Separation Networks For DCASE 2021 Task 4

Liu, Gang and Liu, Zhuang Zhuang and Fang, Jun Yan and Liu, Yi Liu and Zhou, Ming Kun
Beijing University of Posts and Telecommunications, Beijing,China

Abstract

Audio tagging aims to assign one or more labels to the audio clip. In this paper, we proposed our solutions applied to our submission for DCASE2021 Task4. The target of the systems is to provide not only the event class but also the event time localization given that multiple events can be present in an audio recording. We present a convolutional recurrent neural network (CRNN) with two recurrent neural network (RNN) classifiers sharing the same preprocessing convolutional neural network (CNN). Both recurrent networks perform audio tagging. One is processing the input audio signal in forward direction and the other in backward direction. We also use a spatial attention layer which called Fcanet to improve our system. We also make an independent system to achieve sound event speration.

System characteristics

Integrating Advantages Of Recurrent And Transformer Structures For Sound Event Detection In Multiple Scenarios

Lu, Rui ¹ and Hu, Wenzheng ² and Duan Zhiyao ¹ and Liu, Ji ^1
1Beijing Kuaishou Technology Co., Ltd, China ²The State Key Laboratory of Automotive Satefy and Energy, Tsinghua University, Beijing, China

Abstract

In this technical report, we detail our submitted systems for task4 of DCASE2021: Sound Event Detection and Separation in Domestic Environments. Our systems exploit both recurrent structure and transformer structure to model the complicated dynamics in real life domestic audio data. In addition to prevalent tricks such as semi-supervised mean-teacher learning, data augmentation and ensemble, we find that different models exhibit differently under the two scenarios, which emphasize different system properties. By integrating advantages of both the recurrent and transformer structures, our proposed systems achieve an overall poly-phonic sound event detection scores (PSDS-scores) of 1.171 (PSDS-scenario1 + PSDS-scenario2) on the hold-out test set of the development dataset, outperforming the baseline system by 34.8%.

System characteristics

Convolutional Network With Conformer For Semi-Supervised Sound Event Detection

Na, Tong and Zhang, Qinyi
Beijing University of Posts and Telecommunications, Beijing,China

Abstract

In this technical report, we describe our system submission for DCASE 2021 Task 4. Our model employs a convolutional network in conjunction with conformer blocks and utilizes the Mean-Teacher semi-supervised learning technique for further improvement.

System characteristics

Heavily Augmented Sound Event Detection utilizing Weak Predictions

Nam, Hyeonuk and Ko, Byeong-Yun and Lee, Gyeong-Tae and Kim, Seong-Hu and Jung, Won-Ho and Choi, Sang-Min and Park, Yong-Hwa
Korea Advanced Institute of Science and Technology, Department of Mechanical Engineering, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, South Korea

Abstract

The performance of Sound Event Detection (SED) systems are greatly limited by the difficulty in generating large strongly labeled dataset. In this work, we used two main approaches to overcome the lack of strongly labeled data. First, we applied heavy data augmentation on input features. Data augmentation methods used include not only conventional methods used in speech/audio domains but also our proposed method named FilterAugment. Second, we propose two methods to utilize weak predictions to enhance weakly supervised SED performance. As a result, we obtained the best PSDS1 of 0.4336 and best PSDS2 of 0.8161 on the DESED real validation dataset.

System characteristics

Domain-Adapted Sound Event Detection System With Auxiliary Foreground-Background Classifier

Olvera, Michel¹ and Vincent, Emmanuel¹ and Gasso, Gilles^2
1Université de Lorraine, Inria, Loria, F-54000 Nancy, France, ²LITIS EA 4108, Université & INSA Rouen Normandie, 76800 Saint-Étienne du Rouvray, France

Abstract

In this technical report, we propose a sound event detection system for the DCASE 2021 task 4 challenge, which consists of a foreground-background classification branch that is jointly trained with the baseline architecture. Furthermore, to account for the mismatch between synthetic annotated data and real unlabeled data used for training, we also propose a frame-level domain adaptation scheme to improve detection performance over real soundscapes. We show that these improvements to the baseline method help in the generalization of the sound event detection task.

System characteristics

Sound Event Detection with Cross-Referencing Self-Training

Park Sangwook¹ and Choi, Woohyun² and Elhilali, Mounya^1
1Department of Electrical and Computer Engineering, Johns Hopkins University, United States, ²LG electronics

Abstract

This report describes a sound event detection method submitted to the DCASE2021 challenge, task 4. In this approach, we design a residual convolutional recurrent neural network and train this network with a cross-referencing self-training approach that leverages an extensive unlabeled data in combination with labeled data. This approach takes advantage of semi-supervised training using pseudo-labels from a balanced student-teacher model, and outperforms DCASE2021 challenge baseline in terms of Poly-phonic Sound event Detection Score. Additionally, the proposed network has more accurate predictions in class-wise collar-based-F1, compared to the baseline.

System characteristics

Sound Event Detection Using Metric Learning And Focal Loss For DCASE 2021 Task 4

Tian, Gangyi¹ and Huang, Yuxin^1,2 and Ye, Zhirong^1,2 and Ma, Shuo^1,2 and Wang, Xiangdong¹ and Liu, Hong¹ and Qian, Yueliang¹ and Tao, Rui³ and Yan, Long³ and Ouchi, Kazushige³ and Ebbers, Janek⁴ Haeb-Umbach, Reinhold^4
1Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, ²University of Chinese Academy of Sciences, Beijing, China, ³Toshiba China R&D Center, Beijing, China,Beijing University of Posts and Telecommunications, Beijing,China, ⁴Paderborn University, Germany

Abstract

In this technical report, we describe our system submission for DCASE 2021 Task 4. Our model employs a convolutional network in conjunction with conformer blocks and utilizes the Mean-Teacher semi-supervised learning technique for further improvement.

System characteristics

Training Sound Event Detection On A Heterogeneous Dataset

Turpault, Nicolas and Serizel, Romain
Université de Lorraine, CNRS, Inria, Loria, France

Abstract

Training a sound event detection algorithm on a heterogeneous dataset including both recorded and synthetic soundscapes that can have various labeling granularity is a non-trivial task that can lead to systems requiring several technical choices. These technical choices are often passed from one system to another without being questioned. We propose to perform a detailed analysis of DCASE 2020 task 4 sound event detection baseline with regards to several aspects such as the type of data used for training, the parameters of the mean-teacher or the transformations applied while generating the synthetic soundscapes. Some of the parameters that are usually used as default to replicate other approaches are shown to be sub-optimal.

System characteristics

Improving Sound Event Detection In Domestic Environments Using Sound Separation

Turpault, Nicolas¹ and Wisdom, Scott² and Erdogan, Hakan² and Herhey, John R.² and Serizel, Romain¹ and Fonseca, Eduardo³ and Seetharaman, Prem⁴ and Salomon, Justin^5
1Universite de Lorraine, CNRS, Inria, Loria, Nancy, France, ²Google Research, AI Perception, Cambridge, United States, ³Music Technology Group, Universitat Pompeu Fabra, Barcelona, ⁴Interactive Audio Lab, Northwestern University, Evanston, United States⁵Adobe Research, San Francisco, United States

Abstract

Performing sound event detection on real-world recordings often implies dealing with overlapping target sound events and non-target sounds, also referred to as interference or noise. Until now these problems were mainly tackled at the classifier level. We propose to use sound separation as a pre-processing stage for sound event detection. In this paper we start from a sound separation model trained on the Free Universal Sound Separation dataset and the DCASE 2020 task 4 sound event detection baseline. We explore different methods of combining separated sound sources and the original mixture within the sound event detection. Furthermore, we investigate the impact of adapting the universal sound separation model to the sound event detection data in terms of both separation and sound event detection performance.

System characteristics

CHT+NSYSU Sound Event Detection System With Multiscale Channel Attention And Multiple Consistency Training For DCASE 2021 Task 4

Wang, Yih-Wen ¹ and Chen, Chia-Ping ¹ and Lu, Chung-Li ² and Chan, Bo-Cheng ^1
1National Sun Yat-Sen University, Taiwan ² Chunghwa Telecom Laboratories, Taiwan

Abstract

In this technical report, we describe our submission system for DCASE 2021 Task4: sound event detection and separation in domestic environments. The proposed system is based on mean-teacher framework of semi-supervised learning and neural networks of CRNN and CNN-Transformer. We employ consistency training of interpolation (ICT), shift (SCT), and clip-level (CCT) to enhance the generalization and representation. A multiscale CNN block is applied to extract various features to mitigate the influence of the event length diversity for the network. An efficient channel attention network (ECA-Net) and exponential softmax pooling enable the model to obtain definite sound event predictions. To further improve the performance, we use data augmentation including mixup, time shift, and time-frequency masks. Our ensemble system achieves the PSDS-scenario1 of 40.72% and PSDS-scenario2 of 80.80% on the validation set, significantly outperforming that of the baseline score of 34.2% and 52.7%, respectively.