A ResNet-Based Clip Text-to-Audio Retrieval System for DCASE Challenge 2022 Task

Yongquan Lai, Jinsong Pan, Buxian Chen
Ping An Property & Casualty Insurance Company of China, Ltd., China

Abstract

Language-based audio retrieval aim to use language to retrieval audios in a given dataset. This technical report presents an text-to-audio retrieval system submitted to Task 6b of the DCASE 2022 challenge. The proposed system is based on AudioCLIP, which incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet and clothe V2 datasets and introduces a pre-training method to perform bimodal querying. the original AudioCLIP acquired poor retrieval performance on the clothe V2 dataset in a zero-shot inference fashion. So we used AudioCLIP’s model as a weight initializer, and finetuned audio encoder and text encoder using symmetric cross entropy loss over similarity measure among the mini-batch (audio, text) pairs. Through pre-training and data augmentation methods, our model achieved R1 score of 0.35 and mAP10 score of 0.51 on Clotho V2 evaluation set.

System characteristics

Take It Easy: Relaxing Contrastive Ranking Loss with CIDEr

Theodore Lamort de Gail, Dawid Kicinski
Samsung R&D Institute Poland, Warsaw, Poland

Abstract

This report presents our approach and results for task 6B of the DCASE2022 challenge concerning natural-language-based audio retrieval. To match the audio-text pairs, we learn cross-modal embeddings. The audio samples are encoded by an ensemble of four frozen expert models with transformer heads for time aggregation. Captions are encoded using a pre-trained language model. The model is trained with a modified contrastive ranking loss, enhanced with a heuristic caption similarity prior based on the CIDEr metric. We train the system on the AudioCaps and Clotho audio captioning datasets. Furthermore, we use an NLP classifier to gather additional useful audio-caption pairs from Freesound. We achieve 0.48 R10 and 0.23 mAP10 on the Clotho evaluation split (vs. 0.19 and 0.07 respectively for the challenge baseline).

System characteristics

Language-Based Audio Retrieval with Pre-trained Models

Xinhao Mei, Xubo Liu, Haohe Liu, Jianyuan Sun, Mark D. Plumbley, Wenwu Wang
Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey, UK

Abstract

This technical report presents a language-based audio retrieval system that we submitted to Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge 2022 Task 6b. Language-based audio retrieval is a cross-modal task aiming at retrieving a matched audio clip from a pool of candidates given a language query such as a sentence. Cross-modal retrieval tasks are often solved by using deep learning models where the features from different modalities are extracted and then mapped to a joint embedding space. These models usually require a large amount of training data to obtain reasonable performance. However, the audio captioning dataset employed in this audio retrieval task is limited in size. In this work, we propose to use large-scale pre-trained models as both audio and text encoders to mitigate the data scarcity problem and learn the acoustic semantic embeddings. Results on the Clotho dataset show that our proposed system significantly improves the scores of all the evaluation metrics as compared to the baseline system

System characteristics

CAU Submission to DCASE 2022 Task6B: Language-Based Audio Retrieval using Transfer Learning

Jiwon Park, Chaewon Hwang, Il-Youp Kwak, Changwon Lim
Chung-Ang University, Department of Applied Statistics, Seoul, South Korea

Abstract

This report proposes a language-based audio retrieval model for the 2022 DCASE audio retrieval challenge. In this challenge, to make use of the learned feature from AudioSet data, we utilized CNN10 network pre-trained on AudioSet data. With the transfer learning, our proposed model took 10-layers CNN and adding GRU after CNN Module. We used pre-trained Word2Vec as text encoder[1]. Experiments show that the proposed model achieved mAP score of 0.091 and showed better performance compared to baseline mAP score of 0.067.

System characteristics

IRIT-UPS DCASE 2022 Language-Based Audio Retrieval System

Thomas Pellegrini
Computing Sciences, University Toulouse III, Toulouse, France

Abstract

This technical report is a short description of the IRIT-UPS systems used in the DCASE 2022 task 6b dedicated to audio captioning. Four submissions were made: i) a baseline one using pretrained representations for both the audio signal (scene embeddings extracted with PaSST), and for the caption queries (using a large pretrained sentence transformer called all-MPNet), ii) the same baseline system but adding information from AudioSet tags in the audio encoder part, iii) the same as ii) but pretrained on an external dataset (AudioCaps), iv) an ensemble of two systems iii).

System characteristics

Aligning Audio and Text Embeddings for the Language-Based Audio Retrieval Task of the DCASE Challenge 2022

Benno Weck^1,2, Miguel Pérez Fernández^1,2, Holger Kirchhoff¹, Xavier Serra^2
1Huawei Technologies, Munich Research Center, Germany, ²Universitat Pompeu Fabra, Music Technology Group, Spain

Abstract

Our challenge submission shows how large-scale pretrained deep learning models can serve as a strong basis for a cross-modal (text-to-audio) retrieval system. Our system uses embeddings extracted by these models in a general alignment framework to connect matching pairs of audio and text. It processes audio and text separately through different pretrained models, each returning an embedding. Shallow neural networks map the embeddings to a common dimensionality. The cross-modal alignment of the individual embeddings is optimised using a contrastive loss. We employ the RoBERTa foundation model as the text embedding extractor. A pretrained PANNs model extracts the audio embeddings. The embedding extractor model weights remain frozen. To improve the generalisation of our model, we investigate how pretraining with audio and associated noisy text collected from the online platform Freesound improves the performance of our method. We find that a two-stage training process consisting of pretraining with noisy data and fine-tuning with the challenge datasets gives the best results for our approach. Our system showcases a simple yet effective method which is superior to the challenge baseline.

System characteristics

Text-to-Audio Retrieval via Large-Scale Contrastive Training

Yusong Wu^1,2, Tianyu Zhang^1,2, Ke Chen^3
1University of Montreal, Quebec, Canada, ²Mila, Quebec, Canada, ²University of California San Diego, San Diego, United States

Abstract

Although there is an abundance of data available on the internet, audio data is still limited in terms of dataset size and label precision. Scaling the size of audio datasets would therefore be one of the most valuable ways to develop models for better audio understanding. In this report, we propose a pipeline to better learn the audio understanding mechanism by combining audio data with more abundantly available natural language descriptions. We collected a mixed dataset consisting of over 2 million data pairs and trained a contrastive model based on Contrastive Language–Image Pre-training (CLIP) in order to discover correspondence between audio and text. As an audio encoder, we use HTS-AT as a transformer-based model and PANN and a CNN-based model, and as a text encoder, we employ the frozen pre-trained CLIP text encoder. The resulting models are submitted to Task 6B of the DCASE 2022 challenge and achieve a mAP@10 score of at least 0.214.

System characteristics

Language-Based Audio Retrieval with Pretrained CNN and Graph Attention

Feiyang Xiao¹, Jian Guan^1∗, Haiyan Lan¹, Qiaoxi Zhu², and Wenwu Wang^3
1Group of Intelligent Signal Processing, College of Computer Science and Technology, Harbin Engineering University, Harbin, China, ²Centre for Audio, Acoustic and Vibration, University of Technology Sydney, Ultimo, Australia, ³Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

Abstract

This technical report describes our submission for Task 6B of the DCASE2022 Challenge (language-based audio retrieval). Our audio retrieval system has an audio encoder composed of a pretrained CNN module (i.e., pretrained audio neural network, PANNs) and a novel graph attention module. Its text encoder is the pretrained word2vec model, the same as the baseline system of Task 6B. Experiments show that our audio retrieval system can achieve the mAP10 metric (used for ranking) of 13% on the development-testing dataset of Task 6B.

System characteristics

The SJTU System for DCASE2022 Challenge Task 6: Audio Captioning with Audio-Text Retrieval Pre-training

Xuenan Xu, Zeyu Xie, Mengyue Wu, Kai Yu
MoE Key Lab of Artificial Intelligence X-LANCE Lab, Department of Computer Science and Engineering AI Institute, Shanghai Jiao Tong University, Shanghai, China

Abstract

This technical report describes the system submitted to the Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 challenge Task 6. There are two involving subtasks: text-to-audio retrieval and automated audio captioning. The text-to-audio retrieval system adopts a bi-encoder architecture using pre-trained audio and text encoders. The system is first pre-trained on AudioCaps and then fine-tuned on the challenge dataset Clotho. For the audio captioning system, we first train a retrieval model on all public captioning data and then take the audio encoder as the feature extractor. Then a standard sequence-to-sequence model is trained on Clotho based on the pre-trained feature extractor. The captioning model is first trained by word-level cross entropy loss and then finetuned using self-critical sequence training. Our system achieves a SPIDEr of 32.5 on captioning and an mAP of 29.9 on text-to-audio retrieval.

Awards: Judges’ award

System characteristics