Description

Anomalous sound detection (ASD) is the task of identifying whether the sound emitted from a target machine is normal or anomalous. Automatic detection of mechanical failure is an essential technology in the fourth industrial revolution, which involves artificial-intelligence-based factory automation. Prompt detection of machine anomalies by observing sounds is useful for monitoring the condition of machines. Figure 1 shows an overview of the detection system.

This task is the follow-up from DCASE 2020 Task 2 to DCASE 2023 Task 2. The task this year is to develop an ASD system that meets the following five requirements.

1. Train a model using only normal sound (unsupervised learning scenario)
   Because anomalies rarely occur and are highly diverse in real-world factories, it can be difficult to collect exhaustive patterns of anomalous sounds. Therefore, the system must detect unknown types of anomalous sounds that are not provided in the training data. This is the same requirement as in the previous tasks.

2. Detect anomalies regardless of domain shifts (domain generalization task)
   In real-world cases, the operational states of a machine or the environmental noise can change to cause domain shifts. Domain-generalization techniques can be useful for handling domain shifts that occur frequently or are hard-to-notice. In this task, the system is required to use domain-generalization techniques for handling these domain shifts. This requirement is the same as in DCASE 2022 Task 2 and DCASE 2023 Task 2.

3. Train a model for a completely new machine type
   For a completely new machine type, hyperparameters of the trained model cannot be tuned. Therefore, the system should have the ability to train models without additional hyperparameter tuning. This requirement is the same as in DCASE 2023 Task 2.

4. Train a model using a limited number of machines from its machine type
   While sounds from multiple machines of the same machine type can be used to enhance the detection performance, it is often the case that only a limited number of machines are available for a machine type. In such a case, the system should be able to train models using a few machines from a machine type. This requirement is the same as in DCASE 2023 Task 2.

5. Train a model both with or without attribute information
   While additional attribute information can help enhance the detection performance, we cannot always obtain such information. Therefore, the system must work well both when attribute information is available and when it is not.

The last requirement is newly introduced in DCASE 2024 Task2.

First-shot problem under attribute-available and unavailable conditions: Focus of task

The focus of this year's task is the same as last year, the first-shot problem, with one modification. We explain the importance of the first-shot problem and how they are reflected on the requirements.

First, in real-world applications, tuning hyperparameters using test data is often infeasible because we can face completely new machine types or the amount of test data can be insufficient for tuning hyperparameters. This problem motivated the organizers to set the third requirement described above and prepare completely different set of machine types between the development dataset and evaluation dataset.

Second, there can be a limited number of machines for a machine type. Until DCASE 2023 task 2, multiple sections from multiple different machines were provided for each machine type. Although this feature has led to the development of outlier exposure approaches that use sound clips from different machines as anomalies, in many practical cases, the number of machines for a machine type can be limited. This is because the customers may not have multiple machines or they may first plan to install the system for a few machines. Considering these cases, the organizers set the forth requirement and prepared only one section for each machine type since DCASE 2023 task 2.

In addition to the above two features of the first-shot problem, we introduced another requirement following real-world applications. When recording machine sounds, we cannot always obtain information on the machine conditions or types of noise along with them. In such cases, some effective methods such as attribute classification methods cannot be used. To reflect such situations, we introduced the fifth requirement and hid the additional attribute information for some machine types.

In summary, the main features in the task this year are that: (1) The set of machine types in the development dataset and evaluation dataset are completely different, (2) Each machine type contains only one section, and (3) additional attribute information is not given for some machine types.

Schedule

Based on the DCASE Challenge 2024 schedule, the task important days will be as follows.

• Task open: 1st of April 2024
• Additional training dataset release: 15th of May 2024
• Evaluation dataset release: 1st of June 2024
• External resource list lock: 1st of June 2024
• Challenge deadline: 15th of June 2024
• Challenge results: 30th of June 2024

External resources on the "List of external datasets and models allowed" can be used (cf. external data resource section). List of external datasets and models allowed will be updated upon request. Any external resource which are freely accessed before 15th of May 2024 can be added. Please send a request email to the task organizers. The list will be locked after the release date of evaluation dataset (1st of June 2024). To avoid developing new external resources using machine information in the evaluation dataset, we will release the additional training dataset after 15th of May 2024. Note that the additional training dataset contains matching training data of machines used in the evaluation dataset (cf. dataset section).

Audio datasets

Dataset overview

Three datasets (development dataset, additional training dataset, and evaluation dataset) are provided for this task.

The development dataset consists of normal/anomalous operating sounds of seven types of machines. Each recording is a single-channel 10-sec length audio clip that includes both the sounds of the target machine and environmental sounds. The following seven types of machines are used:

• Fan
• Gearbox
• Bearing
• Slide rail
• Toy car
• Toy train
• Valve

The additional training and evaluation datasets also consist of normal/anomalous operating sounds of machines, but the sets of machine types are completely different from the development dataset.

The datasets consist of sounds from seven types of real/toy machines. Each recording is single-channel audio, including a machine's operating sound and environmental noise. The duration of recordings varies from 6 to 18 sec, depending on the machine type. The detailed information on the additional machine types will be provided after 15th of May.

Figure 2 shows an overview of the datasets for development, additional training, and evaluation. Each dataset consists of several types of machines, and each type of machine consists of one “section”.

Definition

We first define the key terms in this task: "machine type," "section," "source domain," "target domain," and "attributes."

• "Machine type" indicates the type of machine, which in the development dataset is one of seven: fan, gearbox, bearing, slide rail, valve, ToyCar, and ToyTrain.
• A section is defined as a subset of the dataset for calculating performance metrics.
• The source domain is the domain under which most of the training data and some of the test data were recorded, and the target domain is a different set of domains under which some of the training data and some of the test data were recorded. There are differences between the source and target domains in terms of operating speed, machine load, viscosity, heating temperature, type of environmental noise, signal-to-noise ratio, etc.
• Attributes are parameters that define states of machines or types of noise.

Development, additional training, and evaluation datasets

Our entire dataset consists of three datasets:

1. Development dataset: This dataset consists of seven machine types. For each machine type, one section is provided, and the section is a complete set of training and test data. For each section, this dataset provides (i) 990 clips of normal sounds in the source domain for training, (ii) ten clips of normal sounds in the target domain for training, and (iii) 100 clips each of normal and anomalous sounds for the test. The source/target domain of each sample is provided. Additionally, the attributes of each sample in the training and test data are provided in the file names and attribute csv files.

2. Additional training dataset: This dataset also consists of several machine types, but the set of machine types are completely different from the development dataset. This dataset also provides one section for each machine type. Each section consists of (i) 990 clips of normal sounds in the source domain for training and (ii) ten clips of normal sounds in a target domain for training. The domain of each sample and attributes are provide. Participants may also use this dataset for training. The additional training dataset will be open on May 15th.

3. Evaluation dataset: This dataset provides test clips for the sections in the additional training dataset. Each section has 200 test clips, none of which have a condition label (i.e., normal or anomaly) or information about the domain to which it belongs (i.e., source or target). Attributes are not provided. The additional training dataset will be open on June 1st.

File names and attribute csv files

File names and attribute csv files provide reference labels for each clip. The given reference labels for each training/test clip include machine type, section index, normal/anomaly information, and attributes regarding the condition other than normal/anomaly. The machine type is given by the directory name. The section index is given by their respective file names. For the datasets other than the evaluation dataset, the normal/anomaly information and the attribute information are given by their respective file names. Note that for machine types that has its attribute information hidden, the attribute information in each file names are only labeled as "noAttributes". Attribute csv files are for easy access to attributes that cause domain shifts. In these files, the file names, name of parameters that cause domain shifts (domain shift parameter, dp), and the value or type of these parameters (domain shift value, dv) are listed. Each row takes the following format:

[filename (string)], [d1p (string)], [d1v (int | float | string)], [d2p], [d2v]...

For machine types that have their attribute information hidden, all columns except the filename column are left blank for each row.

Recording procedure

Normal/anomalous operating sounds of machines and its related equipment are recorded. Anomalous sounds were collected by deliberately damaging target machines. For simplifying the task, we use only the first channel of multi-channel recordings; all recordings are regarded as single-channel recordings of a fixed microphone. We mixed a target machine sound with environmental noise, and only noisy recordings are provided as training/test data. The environmental noise samples were recorded in several real factory environments. We will publish papers on the dataset to explain the details of the recording procedure by the submission deadline.

Short description of each section in the development dataset

Short descriptions of each section in the development dataset and the attribute format in the file names of their training data are as follows:

External data resources

Based on the past DCASE's external data resource policy, we allow the use of external datasets and trained models under the following conditions:

1. Any test data in both development and evaluation datasets shall not be used for training.
2. Any data in ToyADMOS, ToyADMOS2, MIMII Dataset, MIMII DUE Dataset, MIMII DG Dataset, the dataset of DCASE 2020 Challenge Task 2 , the dataset of DCASE 2021 Challenge Task 2 , the dataset of DCASE 2022 Challenge Task 2, and the dataset of DCASE 2023 Challenge Task 2 shall not be used.
3. Datasets, pre-trained models, and pre-trained parameters on the "List of external data resources allowed" can be used. The list will be updated upon request. Datasets, pre-trained models, and pre-trained parameters, which are freely accessible by any other research group before 15th of May 2024, can be added to the list.
4. To add sources of external datasets, pre-trained models, or pre-trained parameters to the list, send a request to the organizers by the evaluation set publishing date. To give an equal opportunity to use them for all competitors, we will update the "list of external data resources allowed" on the web page accordingly.
5. Once the evaluation set is published, no further external sources will be added. The list will be locked after 1st of June 2024.

List of external data resources allowed:

Download

Task 2 Development dataset (2.2 GB)

Task setup and rules

Participants are required to submit both an anomaly score and a normal/anomaly decision result for each test clip. The anomaly score for each test clip will be used to calculate the area under the receiver operating characteristic (ROC) curve (AUC) and partial-AUC (pAUC) scores, which are used to calculate an official score and a final ranking. The normal/anomaly decision result for each test clip is used to calculate the precision, recall, and F1 scores, which will also be published when the challenge results are open. The method of evaluation is described in the Evaluation section.

The anomaly score takes a large value when the input signal seems to be anomalous, and vice versa. To calculate the anomaly score, participants need to train an anomaly score calculator \(\mathcal{A}\) with parameter \(\theta\). The input of \(\mathcal{A}\) is a machine's operating sound \(x \in \mathbb{R}^L\) and its machine information including machine type, section index, and other attribute information, and \(\mathcal{A}\) outputs one anomaly score for the whole audio clip \(x\) as \(\mathcal{A}_\theta (x) \in \mathbb{R}\). Then, \(x\) is determined to be anomalous when the anomaly score exceeds a pre-defined threshold value. Thus, \(\mathcal{A}\) needs to be trained so that \(\mathcal{A}_\theta(x)\) will be a large value both when the whole audio clip \(x\) is anomalous and when a part of \(x\) is anomalous, such as with collision anomalous sounds.

Figure 3 shows the overview of this task, where the example is a procedure for calculating the anomaly scores of the test clips of (fan, section 00, target domain). First, the participants train an anomaly score calculator \(\mathcal{A}\) using training data both in the source and target domains and optional external data resources. Then, by using \(\mathcal{A}\), participants calculate anomaly scores of all the test clips of (fan, section 00, target domain). By repeating this procedure, participants calculate the anomaly score of all the test clips of all the machine types, sections, and domains.

Arbitral numbers of an anomaly score calculator \(\mathcal{A}\) can be used to calculate the anomaly scores of test clips. The simplest strategy is to use a single \(\mathcal{A}\) to calculate the anomaly scores for a single section (e.g., section 00). In this case, \(\mathcal{A}\) is specialized to a single section, so users of such a system are required to train \(\mathcal{A}\) for each machine type, each product, and each condition. A more challenging strategy is to use a single \(\mathcal{A}\) to calculate the anomaly scores of all the test clips of all the machine types and sections. The advantage of this strategy is that participants can use all the training clips provided; however, they need to consider the generalization of the model. Another typical scenario that can be inspired by real-world applications is where you train a general model only with the source-domain data. The task organizers do not impose this constraint but would appreciate participants’ efforts to impose constraints on themselves based on various real-world applications.

All training data with arbitrary splitting can be used to train an anomaly score calculator. For example, to train \(\mathcal{A}\) to calculate the anomaly score of (valve, section 00, source domain), participants can opt to use training data only in (valve, section 00, source domain), training data in both the source domain and target domains, training data of all sections of valves, all provided training data, and/or other strategies. Of course, normal/anomalous clips in test data cannot be used for training; however, simulating anomalous samples using the listed external data resources is allowed.

Changing the model (model/architecture/hyperparameters) between machine types within a single submission is allowed. However, we expect participants to develop a simple ASD system, (i.e. keep the model and hyperparameters fixed and only change the training data to adapt to each machine type).

Submission

The official challenge submission consists of:

• System output for the evaluation data
• Meta information files

System output should be presented as a text-file that corresponds to each machine type, section index. Its file name should be:

• Anomaly score file: anomaly_score_<machine_type>_section_<section_index>.csv
• Detection result file: decision_result_<machine_type>_section_<section_index>.csv

The anomaly score file (in CSV format, without header row) contains the anomaly score for each audio file in the test data of the evaluation dataset. Result items can be in any order. All rows must be in the following format:

[filename (string)],[anomaly score (real value)]

Anomaly scores in the second column can take a negative value. For example, typical auto-encoder-based anomaly score calculators use the squared reconstruction error, which takes a non-negative value, while statistical model-based methods (such as GMM) use the negative log-likelihood as the anomaly score, which can take both positive and negative values.

The decision result file (in CSV format, without header row) contains the normal/anomaly decision result for each audio file in the test data of the evaluation dataset. Result items can be in any order. All rows must be the following format:

[filename (string)],[decision result (0: normal, 1: anomaly)]

We allow up to four system output submissions per participant/team. For each system, meta information should be provided in a separate file that contains the task-specific information. All files should be packaged into a zip file for submission. Detailed information on the submission process can be found on the Submission page.

Evaluation

Metrics

This task is evaluated with the AUC and the pAUC. The pAUC is an AUC calculated from a portion of the ROC curve over the pre-specified range of interest.

Because the anomaly detector is expected to work with the same threshold regardless of the domain, data from both domains in a section are used to calculate the AUC and pAUC. Also, in order to evaluate the detection performance for each domain, the AUC is calculated for each domain. The AUC for each machine type, section, and domain (source/target) and the pAUC for each machine type and section are defined as

where \(m\) represents the index of a machine type, \(n\) represents the index of a section, \(d = \{ {\rm source}, {\rm target} \}\) represents a domain, \(\lfloor \cdot \rfloor\) is the flooring function, and \(\mathcal{H} (x)\) returns 1 when \(x\) > 0 and 0 otherwise. Here, \(\{x_{i}^{−}\}_{i=1}^{N^{-}_{d}}\) is normal test clips in the domain \(d\) in the section \(n\) in the machine type \(m\) and \(\{x_{j}^{+}\}_{j=1}^{N^{+}_{n}}\) is anomalous test clips in the section \(n\) in the machine type \(m\), respectively, and they have been sorted so that their anomaly scores are in descending order. Here, \(N^{-}_{d}\) is the number of normal test clips in the domain \(d\) in the section \(n\) in the machine type \(m\), \(N^{-}_{n}\) are the number of normal test clips in the section \(n\) in the machine type \(m\), and \(N^{+}_{n}\) is the number of anomalous test clips in the section \(n\) in the machine type \(m\), respectively.

In our metric, the pAUC is calculated as the AUC over a low false-positive-rate (FPR) range \([0, p]\). The reason for the additional use of the pAUC is based on practical requirements. If an ASD system frequently gives false alarms, we cannot trust it. Therefore, it is especially important to increase the true-positive-rate under low FPR conditions. In this task, we will use \(p=0.1\).

The official score \(\Omega\) for each submitted system is given by the harmonic mean of the AUC and pAUC scores over all the machine types, sections, and domains as follows:

where \(h\left\{\cdot\right\}\) represents the harmonic mean (over all machine types, sections, and domains), \(\mathcal{M}\) represents the set of the machine types, and \(\mathcal{S}(m)\) represents the set of the sections for the machine type \(m\).

As the equations above show, a threshold value does not need to be determined to calculate AUC, pAUC, or the official score because the threshold value is the anomaly scores of normal test clips. However, in real applications, the threshold value must be determined, and a decision must be made as to whether it is normal or anomalous. Therefore, participants are also required to submit the normal/anomaly decision results. The organizers will publish the AUC, pAUC, and official scores as well as the precision, recall, and F1-scores calculated for the normal/anomaly decision results.

Note: The submitted normal/anomaly decision results will not be used for the final ranking because the task organizers do not want to encourage participants to use a forbidden approach (i.e., threshold tuning based on the distribution in the evaluation dataset). Do not use other test clips to determine anomalies for each test clip.

Ranking

The final ranking will be decided by sorting based on the official score \(\Omega\).

Baseline system

The task organizers provide a baseline system that give a reasonable performance in the dataset of Task 2. It is the same system as DCASE 2023 Challange Task 2 that has two different operating modes of simple autoencoder and selective Mahalanobis. They are good starting points, especially for entry-level researchers who want to get familiar with the ASD task.

Simple Autoencoder mode

The anomaly score is calculated as the reconstruction error of the observed sound. To obtain small anomaly scores for normal sounds, the AE is trained to minimize the reconstruction error of the normal training data. This method is based on the assumption that the AE cannot reconstruct sounds that are not used in training, that is unknown anomalous sounds.

In the baseline system, we first calculate the log-mel-spectrogram of the input \(X = \{X_t\}_{t = 1}^T\) where \(X_t \in \mathbb{R}^F\), and \(F\) and \(T\) are the number of mel-filters and time-frames, respectively. Then, the acoustic feature at \(t\) is obtained by concatenating consecutive frames of the log-mel-spectrogram as \(\psi_t = (X_t, \cdots, X_{t + P - 1}) \in \mathbb{R}^D\), where \(D = P \times F\), and \(P\) is the number of frames of the context window. The anomaly score is calculated as:

where \(r_{\theta}\) is the vector reconstructed by the autoencoder, and \(\| \cdot \|_2\) is \(\ell_2\) norm.

To determine the anomaly detection threshold, we assume that \(A_{\theta}\) follows a gamma distribution. The parameters of the gamma distribution are estimated from the histogram of \(A_{\theta}\), and the anomaly detection threshold is determined as the 90th percentile of the gamma distribution. If \(A_{\theta}\) for each test clip is greater than this threshold, the clip is judged to be abnormal; if it is smaller, it is judged to be normal.

Selective Mahalanobis mode

The anomaly score is calculated as the reconstruction error of the observed sound in the Mahalanobis metric with the covariance matrixes calculated after the last epoch of the training phase.

Parameters

The basic architecture and parameters are the same for both modes, simple autoencoder mode and selective Mahalanobis mode.

Acoustic features

• The frame size for STFT is 64 ms (50 % hop size)
• Log-mel energies for 128 (\(= F\)) bands
• 5 (\(= P\)) consecutive frames are concatenated.
• 640 (\(= D = P \times F\)) dimensions are input to the autoencoder.

Network Architecture

• Input shape: 640
• Architecture:
  • Dense layer #1
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #2
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #3
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #4
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Bottleneck layer
    • Dense layer (units: 8)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #5
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #6
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #7
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #8
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Output layer
    • Dense layer (units: 640)
• Learning (epochs: 100, batch size: 256, data shuffling between epochs)
  • Optimizer: Adam (learning rate: 0.001)

Repository

Results with the development dataset

We evaluated the AUC and pAUC on the development dataset with three random seeds on a V100. The following table shows the results with average and standard deviations.

Citation

If you are participating in this task or using the challenge dataset and the baseline code, please cite a task description paper, which will be available soon.