Ensemble learning for audio captioning with graph audio feature representation

Feiyang Xiao¹, Jian Guan¹, Haiyan Lan¹, Qiaoxi Zhu², Wenwu Wang^3
1Group of Intelligent Signal Processing, College of Computer Science and Technology, Harbin Engineering University, Harbin, China, ²Centre for Audio, Acoustic and Vibration, University of Technology Sydney, Ultimo, Australia, ³Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

Abstract

This technical report describes our submission for Task 6A of the DCASE2022 Challenge (automated audio captioning). Our system is built based on the ensemble learning strategy. It integrates the advantages of different audio captioning methods, including the graph attention-based audio feature representation method. Experiments show that our ensemble system can achieve the SPIDEr score (used for ranking) of 30.2 on the evaluation split of the Clotho v2 dataset.

System characteristics

Exploring audio captioning with keyword-guided text generation

Dawid Kicinski, Teodor Lamort de Gail, Pawel Bujnowski
Samsung R&D Institute Poland, Artificial Intelligence, Warsaw, Poland

Abstract

This technical report describes our submission to the DCASE 2022 challenge, Task 6 A: automated audio captioning. In our system, we explore the use of pre-trained language models for the audio captioning task. The proposed system is an encoder-decoder architecture consisting of a pre-trained PANN encoder and a GPT2 decoder. Audio embeddings are encoded to language model prompts using a simple mapping network. We further develop our system by employing strategies of guiding the decoder with textual information. We prompt the decoder with keywords extracted from semantically similar audios and use them to choose the best matching caption by their occurrence.

System characteristics

Efficient audio captioning transformer with patchout and text guidance

Thodoris Kouzelis^1,2, Grigoris Bastas^1,2, Athanasios Katsamanis², Alexandros Potamianos^1
1Institute for Language and Speech Processing, Athena Research Center, Athens, Greece, ²School of ECE, National Technical University of Athens, Athens, Greece

Abstract

This technical report describes an Automated Audio Captioning model for the Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Challenge, Task 6. We propose a full Transformer architecture that utilizes Patchout as proposed in [1], significantly reducing the computational complexity and avoiding overfitting. The caption generation is partly conditioned on textual AudioSet tags extracted by a pre-trained classification model which is fine-tuned to maximize the semantic similarity between AudioSet labels and ground truth captions. To mitigate the data scarcity problem of Automated Audio Captioning (AAC) we pre-train our model on an enlarged dataset. Moreover, we propose a method to apply Mixup augmentation for AAC. Our best model achieves the SPIDEr score of 0.296.

Awards: Judges’ award

System characteristics

IRIT-UPS DCASE 2022 task6a system: stochastic decoding methods for audio captioning

Etienne Labbé, Thomas Pellegrini, Julien Pinquier
IRIT (UMR 5505), Université Paul Sabatier, CNRS, Toulouse, France

Abstract

This document presents a summary of our models used in the Automated Audio Captioning task (6a) for the DCASE2022 challenge. Four submissions were made using different decoding methods : beam search, top k sampling, nucleus sampling and typical decoding.

System characteristics

Automated audio captioning with keywords guidance

Xinhao Mei, Xubo Liu, Haohe Liu, Jianyuan Sun, Mark D. Plumbley, Wenwu Wang
Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey, UK

Abstract

This technical report describes an automated audio captioning system we submitted to Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge 2022 Task 6a. The proposed system is built on an encoder-decoder architecture we submitted to DCASE 2021 Challenge Task 6 last year, where the encoder is a pre-trained 10-layer convolutional neural network and the decoder is a Transformer network. In this new submission, we investigate the use of keywords estimated from input audio clips to guide the caption generation process. The results show that keywords guidance can improve the system performance especially when the pre-trained encoder is frozen, and can also reduce the variance of the results when the model is trained with different seeds. The overall system consists of a pre-trained keywords estimation model and a CNN-Transformer audio captioning model. The captioning model is first trained via the cross-entropy loss and then fine-tuned with reinforcement learning to optimize the evaluation metric CIDEr. The proposed system significantly improves the scores of all the evaluation metrics as compared to the baseline system.

System characteristics

Audio captioning using pre-trained model and data augmentation

Tianyang Huang¹, Chaofan Pan¹, Wenyao Chen¹, Chenyang Zhu³, Shengchen Li², Xi Shao^1
1College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China, ²School of Advanced Technology, Xi’an Jiaotong-liverpool University, Suzhou, China, ³School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China

Abstract

This technical report describes an automatic audio captioning system for task 6, Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Challenge. Based on an encoder-decoder architecture, the system is composed of convolutional neural network (CNN) encoder and transformer decoder. Instead of using pre-trained models only in audio modal, we try to introduce pretrained models in text modal as well. In addition, we consider using a data argumentation method with and without noise to improve the data quality and thus improve the generalization and robustness of the model. The experimental results show that our system can achieve the SPIDEr of 0.257 (official baseline: 0.233) on the Clotho evaluation set.

System characteristics

CP-JKU's submission to task 6a of the DCASE2022 challenge: a BART encoder-decoder for automatic audio captioning trained via the reinforce algorithm and transfer learning

Paul Primus¹, Gerhard Widmer^1,2
1Institute of Computational Perception (CP-JKU), ²LIT Artificial Intelligence Lab, Johannes Kepler University, Austria

Abstract

This technical report details the CP-JKU submission to the automatic audio captioning task of the 2022’s DCASE challenge (task 6a). The objective of the task was to train a sequence-to-sequence model that automatically generates textual descriptions for given audio recordings. The approach described in this report enhances the BART-based encoder-decoder model used as the challenge’s baseline system in three directions: firstly, the VGGish embedding model was replaced with a custom CNN10-like model that we pretrained on AudioSet. Secondly, the BART encoder-decoder model was pre-trained on AudioCaps, which led to faster convergence. And finally, the best model was further fine-tuned by optimizing the non-differentiable CIDEr metric using the REINFORCE algorithm. Our best model achieves a SPIDEr score of .29 (single-model performance), which is an improvement of 6.6 pp. over the challenge’s baseline score.

System characteristics

The SJTU System for DCASE2022 Challenge Task 6: Audio Captioning with Audio-Text Retrieval Pre-training

Xuenan Xu, Zeyu Xie, Mengyue Wu, Kai Yu
MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Department of Computer Science and Engineering AI Institute, Shanghai Jiao Tong University, Shanghai, China

Abstract

This technical report describes the system submitted to the Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 challenge Task 6. There are two involving subtasks: text-to-audio retrieval and automated audio captioning. The text-to-audio retrieval system adopts a bi-encoder architecture using pre-trained audio and text encoders. The system is first pre-trained on AudioCaps and then fine-tuned on the challenge dataset Clotho. For the audio captioning system, we first train a retrieval model on all public captioning data and then take the audio encoder as the feature extractor. Then a standard sequence-to-sequence model is trained on Clotho based on the pre-trained feature extractor. The captioning model is first trained by word-level cross entropy loss and then finetuned using self-critical sequence training. Our system achieves a SPIDEr of 32.5 on captioning and an mAP of 29.9 on text-to-audio retrieval.

System characteristics

Automated audio captioning with multi-task learning

Zhongjie Ye¹, Yuexian Zou¹, Fan Cui², Yujun Wang^2
1ADSPLAB, School of ECE, Peking University, Shenzhen, China, ²Xiaomi Corporation, Beijing, China

Abstract

This technical report describes an automated audio captioning (AAC) model for the Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Task 6A Challenge. Our model consists of a convolution neural network (CNN) encoder and a single layer (LSTM) decoder with a temporal attention module. In order to enhance the representation in the domain dataset, we use the ResNet38 pretrained on the AudioSet dataset as our audio encoder and finetune it with keywords of nouns and verbs as labels which are extracted from the captions. For training the whole caption model, we first train the model with the standard cross entropy loss, and fine-tune it with reinforcement learning to directly optimize the CIDEr score. Experimental results show that our single model can achieve a SPIDEr score of 31.7 on the evaluation spilt.

System characteristics