Challenge has ended. Full results for this task can be found in the Results page.

Description

Anomalous sound detection (ASD) is the task of identifying whether the sound emitted from a target machine is normal or anomalous. Automatic detection of mechanical failure is an essential technology in the fourth industrial revolution, which involves artificial intelligence (AI)-based factory automation. Prompt detection of machine anomalies by observing sounds is useful for monitoring the condition of machines. Figure 1 shows an overview of the detection system.

This task is the follow-up to DCASE 2020 Task 2 and DCASE 2021 Task 2. The task this year is to detect anomalous sounds under three main conditions:

1. Only normal sound clips are provided as training data (i.e., unsupervised learning scenario). In real-world factories, anomalies rarely occur and are highly diverse. Therefore, exhaustive patterns of anomalous sounds are impossible to create or collect and unknown anomalous sounds that were not observed in the given training data must be detected. This condition is the same as in DCASE 2020 Task 2 and DCASE 2021 Task 2.

2. Factors other than anomalies change the acoustic characteristics between training and test data (i.e., domain shift). In real-world cases, operational conditions of machines or environmental noise often differ between the training and testing phases. For example, the operation speed of a conveyor can change due to seasonal demand, or environmental noise can fluctuate depending on the states of surrounding machines. This condition is the same as in DCASE 2021 Task 2.

3. In test data, samples unaffected by domain shifts (source domain data) and those affected by domain shifts (target domain data) are mixed, and the source/target domain of each sample is not specified. Therefore, the model must detect anomalies with the same threshold value regardless of the domain (i.e., domain generalization).

Domain generalization: Focus of task

Domain shifts are differences in acoustic characteristics between the training and test data, such as differences in operational speed, machine load, and environmental noise. Because these shifts are caused by factors other than anomalies, models trained with the source domain data can easily cause false positives. One solution is to track these shifts and use domain adaptation techniques to adapt the model using the target domain data. However, in real-world cases, the use of domain adaptation techniques can be costly or even impractical, so domain generalization techniques are preferred. Domain generalization techniques mainly use the source domain data to learn common features across different domains so that the model can generalize to both the source and target domain in the test data. In the training data, there are little to no samples of the target domain data. Domain generalization techniques are mainly preferred over domain adaptation techniques in the following four scenarios:

1. Domain shifts caused by differences in machine’s physical parameters
Characteristics of a machine sound can change due to changes in the machine’s physical parameter. Although these shifts can be tracked, if the parameter changes within a short period of time, it can be too costly to adapt the model every time the value changes.

2. Domain shifts caused by differences in environmental conditions
Because characteristics of background noise can be affected by various factors, it is difficult to track these shifts. Therefore, a model that is unaffected by these shifts is desirable.

3. Domain shifts caused by maintenance
Characteristics of a machine sound can change after maintenance or parts replacement. Though these shifts can be tracked, adapting the model every time can be costly.

4. Domain shifts caused by differences in recording devices
In real-world scenarios, many microphones are installed at different locations, and these microphones may be from different manufacturers. Although these shifts can be tracked, adapting the model for each location or microphone can be too costly.

Each section of the dataset this year was made to reflect one of these scenarios.

Schedule

Based on the DCASE Challenge 2022 schedule, the task important days will be as follows.

• Task open: 15th of March 2022
• Additional training dataset release: 15th of April 2022
• Evaluation dataset release: 1st of June 2022
• External resource list lock: 1st of June 2022
• Challenge deadline: 15th of June 2022
• Challenge results: 1st of July 2022

External resources on the "List of external datasets and models allowed" can be used (cf. external data resource section). List of external datasets and models allowed will be updated upon request. Any external resource which are freely accessed before 15th of April 2022 can be added. Please send a request email to the task organizers. The list will be locked after the release date of evaluation dataset (1st of June 2022). To avoid developing new external resources using machine information in the evaluation dataset, we will release the additional training dataset after 15th of April 2022. Note that the additional training dataset contains matching training data of machines used in the evaluation dataset (cf. dataset section).

Audio datasets

Dataset overview

The data used for this task consists of the normal/anomalous operating sounds of seven types of machines. Each recording is a single-channel 10-sec length audio clip that includes both the sounds of the target machine and environmental sounds. The following seven types of machines are used in this task:

• Fan
• Gearbox
• Bearing
• Slide rail
• Toy car
• Toy train
• Valve

Figure 2 shows an overview of the datasets for development, additional training, and evaluation. Each dataset consists of the seven types of machines, and each type of machine consists of three “sections” (Sections 00, 01, and 02 for the development dataset and Sections 03, 04, and 05 for the additional training dataset and the evaluation dataset).

Definition

We first define key terms in this task: "machine type," "section," "source domain," "target domain," and "attributes.".

• "Machine type" indicates the kind of machine, which in this task is one of seven: fan, gearbox, bearing, slide rail, valve, ToyCar, and ToyTrain.
• A section is defined as a subset of the dataset for calculating performance metrics. Each section is dedicated to a specific type of domain shift.
• The source domain is the domain under which most of the training data and part of the test data were recorded, and the target domain is a different set of domains under which a few of the training data and part of the test data were recorded. There are differences between the source and target domains in terms of operating speed, machine load, viscosity, heating temperature, type of environmental noise, SNR, etc.
• Attributes are parameters that define states of machines or types of noise.

Development, additional training, and evaluation datasets

Our entire dataset consists of three datasets:

1. Development dataset: This dataset consists of three sections for each machine type (Sections 00, 01, and 02), and each section is a complete set of training and test data. For each section, this dataset provides (i) 990 clips of normal sounds in the source domain for training, (ii) ten clips of normal sounds in the target domain for training, and (iii) 100 clips each of normal and anomalous sounds for the test. The source/target domain of each sample is provided. Additionally, the attributes of each sample in the training and test data are provided in the file names and attribute csv files.

2. Additional training dataset: This dataset provides the other three sections for each machine type (Sections 03, 04, and 05). Each section consists of (i) 990 clips of normal sounds in the source domain for training and (ii) ten clips of normal sounds in a target domain for training. The domain of each sample and attributes are provide. Participants may also use this dataset for training. The additional training dataset will be open on April 15th.

3. Evaluation dataset: This dataset provides test clips for the three sections identical to the additional training dataset (Sections 03, 04, and 05). Each section has 200 test clips, none of which have a condition label (i.e., normal or anomaly) or information about the domain to which it belongs (i.e., source or target). Attributes are not provided. Note that the sections of the evaluation dataset (Sections 03, 04, and 05) are the same as that of the additional training dataset but different from the development dataset (Sections 00, 01, and 02).

As described above, the main modification from the 2021 version is that the task is a domain generalization task and the source/target domain of test data in the evaluation dataset is not provided. Most of the training data (990 clips for each section) are from a domain called the source domain, and only a few normal clips (ten clips for each section) are provided for another domain called the target domain. Because the domain of test data in the evaluation dataset is not specified, the participants are encouraged to develop a system that can detect anomalies with the same threshold value for all domains.

File names and attribute csv files

File names and attribute csv files provide reference labels for each clip. The given reference labels for each training/test clip include machine type, section index, normal/anomaly information, and attributes regarding the condition other than normal/anomaly. The machine type is given by the directory name. The section index is given by their respective file names. For the datasets other than the evaluation dataset, the normal/anomaly information and the attributes are given by their respective file names. Attribute csv files are for easy access to attributes that cause domain shifts. In these files, the file names, name of parameters that cause domain shifts (domain shift parameter, dp), and the value or type of these parameters (domain shift value, dv) are listed. Each row takes the following format:

[filename (string)], [d1p (string)], [d1v (int | float | string)], [d2p], [d2v]...

Recording procedure

Normal/anomalous operating sounds of machines and its related equipment are recorded. Anomalous sounds were collected by deliberately damaging target machines. For simplifying the task, we use only the first channel of multi-channel recordings; all recordings are regarded as single-channel recordings of a fixed microphone. We mixed a target machine sound with environmental noise, and only noisy recordings are provided as training/test data. The environmental noise samples were recorded in several real factory environments. We will publish papers on the dataset to explain the details of the recording procedure by the submission deadline.

Short description of each section in the development dataset

Short descriptions of each section in the development dataset and the attribute format in the file names of their training data are as follows:

Short description of each section in the additional training dataset

Short descriptions of each section in the additional training dataset and the attribute format in the file names of their training data are as follows:

External data resources

Based on the past DCASE's external data resource policy, we allow the use of external datasets and trained models under the following conditions:

1. Any test data in both development and evaluation datasets shall not be used for training.
2. Any data in ToyADMOS, ToyADMOS2, MIMII Dataset, MIMII DUE Dataset, the datasets of DCASE 2020 Challenge Task 2 , and the dataset of DCASE 2021 Challenge Task 2 shall not be used.
3. Datasets, pre-trained models, and pre-trained parameters on the "List of external data resources allowed" can be used. The list will be updated upon request. Datasets, pre-trained models, and pre-trained parameters, which are freely accessible by any other research group before 15th of April 2022, can be added to the list.
4. To add sources of external datasets, pre-trained models, or pre-trained parameters to the list, send a request to the organizers by the evaluation set publishing date. To give an equal opportunity to use them for all competitors, we will update the "list of external data resources allowed" on the web page accordingly.
5. Once the evaluation set is published, no further external sources will be added. The list will be locked after 1st of June 2022.

List of external data resources allowed:

Download

Development dataset (6.2 GB)
version 1.0

Additional training dataset (5.2 GB)
version 1.0

Evaluation dataset (1.0 GB)
version 1.0

Task setup and rules

Participants are required to submit both an anomaly score and a normal/anomaly decision result for each test clip. The anomaly score for each test clip will be used to calculate the area under the receiver operating characteristic (ROC) curve (AUC) and partial-AUC (pAUC) scores, which are used to calculate an official score and a final ranking. The normal/anomaly decision result for each test clip is used to calculate the precision, recall, and F1 scores, which will also be published when the challenge results are open. The method of evaluation is described in the Evaluation section.

The anomaly score takes a large value when the input signal seems to be anomalous, and vice versa. To calculate the anomaly score, participants need to train an anomaly score calculator \(\mathcal{A}\) with parameter \(\theta\). The input of \(\mathcal{A}\) is a machine's operating sound \(x \in \mathbb{R}^L\) and its machine information including machine type, section index, and other attribute information, and \(\mathcal{A}\) outputs one anomaly score for the whole audio clip \(x\) as \(\mathcal{A}_\theta (x) \in \mathbb{R}\). Then, \(x\) is determined to be anomalous when the anomaly score exceeds a pre-defined threshold value. Thus, \(\mathcal{A}\) needs to be trained so that \(\mathcal{A}_\theta(x)\) will be a large value both when the whole audio clip \(x\) is anomalous and when a part of \(x\) is anomalous, such as with collision anomalous sounds.

Figure 3 shows the overview of this task, where the example is a procedure for calculating the anomaly scores of the test clips of (fan, section 00, target domain). First, the participants train an anomaly score calculator \(\mathcal{A}\) using training data both in the source and target domains and optional external data resources. Then, by using \(\mathcal{A}\), participants calculate anomaly scores of all the test clips of (fan, section 00, target domain). By repeating this procedure, participants calculate the anomaly score of all the test clips of all the machine types, sections, and domains.

Arbitral numbers of an anomaly score calculator \(\mathcal{A}\) can be used to calculate the anomaly scores of test clips. The simplest strategy is to use a single \(\mathcal{A}\) to calculate the anomaly scores for a single section (e.g., section 00). In this case, \(\mathcal{A}\) is specialized to a single section, so users of such a system are required to train \(\mathcal{A}\) for each machine type, each product, and each condition. A more challenging strategy is to use a single \(\mathcal{A}\) to calculate the anomaly scores of all the test clips of all the machine types and sections. The advantage of this strategy is that participants can use all the training clips provided; however, they need to consider the generalization of the model. Another typical scenario that can be inspired by real-world applications is where you train a general model only with the source-domain data. The task organizers do not impose this constraint but would appreciate participants’ efforts to impose constraints on themselves based on various real-world applications.

All training data with arbitrary splitting can be used to train an anomaly score calculator. For example, to train \(\mathcal{A}\) to calculate the anomaly score of (valve, section 03, source domain), participants can opt to use training data only in (valve, section 03, source domain), training data in both the source domain and target domains, training data of all sections of valves, all provided training data, and/or other strategies. Of course, normal/anomalous clips in test data cannot be used for training; however, simulating anomalous samples using the listed external data resources is allowed.

Changing the model (model/architecture/hyperparameters) between machine types within a single submission is allowed. However, we expect participants to develop a simple ASD system, (i.e. keep the model and hyperparameters fixed and only change the training data to adapt to each machine type).

Submission

The official challenge submission consists of:

• System output for the evaluation data
• Meta information files

System output should be presented as a text-file that corresponds to each machine type, section index. Its file name should be:

• Anomaly score file: anomaly_score_<machine_type>_section_<section_index>.csv
• Detection result file: decision_result_<machine_type>_section_<section_index>.csv

The anomaly score file (in CSV format, without header row) contains the anomaly score for each audio file in the test data of the evaluation dataset. Result items can be in any order. All rows must be in the following format:

[filename (string)],[anomaly score (real value)]

Anomaly scores in the second column can take a negative value. For example, typical auto-encoder-based anomaly score calculators use the squared reconstruction error, which takes a non-negative value, while statistical model-based methods (such as GMM) use the negative log-likelihood as the anomaly score, which can take both positive and negative values.

The decision result file (in CSV format, without header row) contains the normal/anomaly decision result for each audio file in the test data of the evaluation dataset. Result items can be in any order. All rows must be the following format:

[filename (string)],[decision result (0: normal, 1: anomaly)]

We allow up to four system output submissions per participant/team. For each system, meta information should be provided in a separate file that contains the task-specific information. All files should be packaged into a zip file for submission. Detailed information on the submission process can be found on the Submission page.

Evaluation

Metrics

This task is evaluated with the AUC and the pAUC. The pAUC is an AUC calculated from a portion of the ROC curve over the pre-specified range of interest.

Because this task focuses on domain generalization, the anomaly detector is expected to work with the same threshold regardless of the domain. Therefore, data from both domains in a section are used to calculate the AUC and pAUC. Also, in order to evaluate the detection performance for each domain, the AUC is calculated for each domain. The AUC for each machine type, section, and domain (source/target) and the pAUC for each machine type and section are defined as

where \(m\) represents the index of a machine type, \(n\) represents the index of a section, \(d = \{ {\rm source}, {\rm target} \}\) represents a domain, \(\lfloor \cdot \rfloor\) is the flooring function, and \(\mathcal{H} (x)\) returns 1 when \(x\) > 0 and 0 otherwise. Here, \(\{x_{i}^{−}\}_{i=1}^{N^{-}_{d}}\) is normal test clips in the domain \(d\) in the section \(n\) in the machine type \(m\) and \(\{x_{j}^{+}\}_{j=1}^{N^{+}_{n}}\) is anomalous test clips in the section \(n\) in the machine type \(m\), respectively, and they have been sorted so that their anomaly scores are in descending order. Here, \(N^{-}_{d}\) is the number of normal test clips in the domain \(d\) in the section \(n\) in the machine type \(m\), \(N^{-}_{n}\) are the number of normal test clips in the section \(n\) in the machine type \(m\), and \(N^{+}_{n}\) is the number of anomalous test clips in the section \(n\) in the machine type \(m\), respectively.

In our metric, the pAUC is calculated as the AUC over a low false-positive-rate (FPR) range \([0, p]\). The reason for the additional use of the pAUC is based on practical requirements. If an ASD system frequently gives false alarms, we cannot trust it. Therefore, it is especially important to increase the true-positive-rate under low FPR conditions. In this task, we will use \(p=0.1\).

The official score \(\Omega\) for each submitted system is given by the harmonic mean of the AUC and pAUC scores over all the machine types, sections, and domains as follows:

where \(h\left\{\cdot\right\}\) represents the harmonic mean (over all machine types, sections, and domains), \(\mathcal{M}\) represents the set of the machine types, and \(\mathcal{S}(m)\) represents the set of the sections for the machine type \(m\).

As the equations above show, a threshold value does not need to be determined to calculate AUC, pAUC, or the official score because the threshold value is the anomaly scores of normal test clips. However, in real applications, the threshold value must be determined, and a decision must be made as to whether it is normal or anomalous. Therefore, participants are also required to submit the normal/anomaly decision results. The organizers will publish the AUC, pAUC, and official scores as well as the precision, recall, and F1-scores calculated for the normal/anomaly decision results.

Note: The submitted normal/anomaly decision results will not be used for the final ranking because the task organizers do not want to encourage participants to use a forbidden approach (i.e., threshold tuning based on the distribution in the evaluation dataset). Do not use other test clips to determine anomalies for each test clip.

Ranking

The final ranking will be decided by sorting based on the official score \(\Omega\).

Baseline system

The task organizers provide two baseline systems that gives a reasonable performance in the dataset of Task 2. They are good starting points, especially for entry-level researchers who want to get familiar with the ASD task.

Autoencoder-based baseline is an example of the inlier modeling (IM)-based detector that models the distribution of inlier samples, and the MobileNetV2-based baseline is an example of the outlier exposure (OE)-based detector that utilizes outlier samples for modeling. In the previous year's task, ensembling of IM-based detectors and OE-based detectors or IM-based detection using features learned in a machine-identification task were common.

Autoencoder-based baseline

This is an IM-based detector that uses autoencoder (AE). The anomaly score is calculated as the reconstruction error of the observed sound. To obtain small anomaly scores for normal sounds, the AE is trained to minimize the reconstruction error of the normal training data. This method is based on the assumption that the AE cannot reconstruct sounds that are not used in training, that is unknown anomalous sounds.

In the baseline system, we first calculate the log-mel-spectrogram of the input \(X = \{X_t\}_{t = 1}^T\) where \(X_t \in \mathbb{R}^F\), and \(F\) and \(T\) are the number of mel-filters and time-frames, respectively. Then, the acoustic feature at \(t\) is obtained by concatenating consecutive frames of the log-mel-spectrogram as \(\psi_t = (X_t, \cdots, X_{t + P - 1}) \in \mathbb{R}^D\), where \(D = P \times F\), and \(P\) is the number of frames of the context window. The anomaly score is calculated as:

where \(r_{\theta}\) is the vector reconstructed by the autoencoder, and \(\| \cdot \|_2\) is \(\ell_2\) norm.

To determine the anomaly detection threshold, we assume that \(A_{\theta}\) follows a gamma distribution. The parameters of the gamma distribution are estimated from the histogram of \(A_{\theta}\), and the anomaly detection threshold is determined as the 90th percentile of the gamma distribution. If \(A_{\theta}\) for each test clip is greater than this threshold, the clip is judged to be abnormal; if it is smaller, it is judged to be normal.

Parameters

Acoustic features

• The frame size for STFT is 64 ms (50 % hop size)
• Log-mel energies for 128 (\(= F\)) bands
• 5 (\(= P\)) consecutive frames are concatenated.
• 640 (\(= D = P \times F\)) dimensions are input to the autoencoder.

Network Architecture

• Input shape: 640
• Architecture:
  • Dense layer #1
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #2
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #3
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #4
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Bottleneck layer
    • Dense layer (units: 8)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #5
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #6
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #7
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Dense layer #8
    • Dense layer (units: 128)
    • Batch Normalization
    • Activation (ReLU)
  • Output layer
    • Dense layer (units: 640)
• Learning (epochs: 100, batch size: 512, data shuffling between epochs)
  • Optimizer: Adam (learning rate: 0.001)

MobileNetV2-based baseline

This is an OE-based detector that uses MobileNetV2. This baseline identifies from which section the observed signal was generated. In other words, it outputs the softmax value that is the predicted probability for each section. The anomaly score is calculated as the averaged negative logit of the predicted probabilities for the correct section.

We first calculate the log-mel-spectrogram of the input \(X = \{X_t\}_{t = 1}^T\) where \(X_t \in \mathbb{R}^F\), and \(F\) and \(T\) are the number of mel-filters and time-frames, respectively. Then, the acoustic feature (two-dimensional image) at \(t\) is obtained by concatenating consecutive frames of the log-mel-spectrogram as \(\psi_t = (X_t, \cdots, X_{t + P - 1}) \in \mathbb{R}^{P \times F}\), where \(P\) is the number of frames of the context window. By shifting the context window by \(L\) frames, \(B (= \lfloor \frac{T - P}{L} \rfloor)\) images are extracted. The anomaly score is calculated as:

where \(t(b)\) is the beginning frame index of the \(b\)-th image, and \(p_{\theta}\) is the softmax output by MobileNetV2 for the correct section.

To determine the anomaly detection threshold, we assume that \(A_{\theta}\) follows a gamma distribution. The parameters of the gamma distribution are estimated from the histogram of \(A_{\theta}\), and the anomaly detection threshold is determined as the 90th percentile of the gamma distribution. If \(A_{\theta}\) for each test clip is greater than this threshold, the clip is judged to be abnormal; if it is smaller, it is judged to be normal.

Parameters

Acoustic features

• The frame size for STFT is 64 ms (50 % hop size)
• Log-mel energies for 128 (\(= F\)) bands
• 64 (\(= P\)) consecutive frames are concatenated.
• Images of size \(P \times F\) are input to a network using MobileNetV2.
• The context window is shifted by 8 (\(= L\)) frames (called "hop frames" or "stride").

Network Architecture

• Input shape: \(64 \times 128\) image
• Architecture:
  • Triplication layer
    • Triplication of the input image to each color channel
  • MobileNetV2
    • Input: \(64 \times 128 \times 3\) image
    • Output: Softmax for 3 sections (Section 00, 01, and 02)
• Learning (epochs: 20, batch size: 32, data shuffling between epochs)
  • Optimizer: Adam (learning rate: 0.00001)

Repository

Detailed information can be found in the GitHub repository. The directory structure is briefly described here as a reference for label information. When you unzip the files downloaded from the GitHub repository and Zenodo, you can see the following directory structure. As described in the Dataset section, the machine type information is given by directory name, and the section index, domain, and the condition information are given by file name, as:

• ./dcase2022_task2_baseline_ae
  • /00_train.py
  • /01_test.py
  • /common.py
  • /keras_model.py
  • /baseline.yaml
  • /readme.md
• ./dcase2022_task2_baseline_mobile_net_v2
  • /00_train.py
  • /01_test.py
  • /common.py
  • /keras_model.py
  • /baseline.yaml
  • /readme.md
• /dev_data
  • /fan
    • /train (only normal clips)
      • /section_00_source_train_normal_0000_.wav
      • ...
      • /section_00_source_train_normal_0989_.wav
      • /section_00_target_train_normal_0000_.wav
      • ...
      • /section_00_target_train_normal_0009_.wav
      • /section_01_source_train_normal_0000_.wav
      • ...
      • /section_02_target_train_normal_0009_.wav
    • /test
      • /section_00_source_test_normal_0000_.wav
      • ...
      • /section_00_source_test_normal_0049_.wav
      • /section_00_source_test_anomaly_0000_.wav
      • ...
      • /section_00_source_test_anomaly_0049_.wav
      • /section_00_target_test_normal_0000_.wav
      • ...
      • /section_00_target_test_normal_0049_.wav
      • /section_00_target_test_anomaly_0000_.wav
      • ...
      • /section_00_target_test_anomaly_0049_.wav
      • /section_01_source_test_normal_0000_.wav
      • ...
      • /section_02_target_test_anomaly_0049_.wav
    • attributes_00.csv (attribute csv for section 00)
    • attributes_01.csv (attribute csv for section 01)
    • attributes_02.csv (attribute csv for section 02)
  • /gearbox (The other machine types have the same directory structure as fan.)
  • /bearing
  • /slider (slider means "slide rail")
  • /ToyCar
  • /ToyTrain
  • /valve
• /eval_data
  • /fan
    • /train (after launch of the additional training dataset)
      • /section_03_source_train_normal_0000_.wav
      • ...
      • /section_03_source_train_normal_0989_.wav
      • /section_03_target_train_normal_0000_.wav
      • ...
      • /section_03_target_train_normal_0009_.wav
      • /section_04_source_train_normal_0000_.wav
      • ...
      • /section_05_target_train_normal_0009_.wav
    • /test (after launch of the evaluation dataset)
      • /section_03_test_0000.wav
      • ...
      • /section_03_test_0199.wav
      • /section_04_test_0000.wav
      • ...
      • /section_05_test_0199.wav
    • attributes_03.csv (attribute csv for train data in section 03)
    • attributes_04.csv (attribute csv for train data in section 04)
    • attributes_05.csv (attribute csv for train data in section 05)
  • /gearbox (The other machine types have the same directory structure as fan.)
  • /bearing
  • /slider (slider means "slide rail")
  • /ToyCar
  • /ToyTrain
  • /valve

After you run the training script 00_train.py and the test script 01_test.py, a csv file for each section index and domain that lists the anomaly scores for each clip will be stored in the directory result/. Also, a csv file for each section index and domain that lists the normal/anomaly decision results for each clip will be stored in the same directory. You can get more detailed information in the GitHub repository.

Results with the development dataset

We evaluated the AUC and pAUC on the development dataset using several types of GPUs (RTX 2080, etc.). Because the results produced with a GPU are generally non-deterministic, the average and standard deviations from these five independent trials (training and testing) are shown in the following table.

Detailed results for AE-based baseline

Detailed results for MobileNetV2-based baseline

Citation

If you are participating in this task or using the MIMII DG and ToyADMOS2, and baseline code, please cite the following three papers:

@inproceedings{Dohi2022,
    author = "Dohi, Kota and Nishida, Tomoya and Purohit, Harsh and Tanabe, Ryo and Endo, Takashi and Yamamoto, Masaaki and Nikaido, Yuki and Kawaguchi, Yohei",
    title = "{MIMII DG}: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection for Domain Generalization Task",
    booktitle = "Proceedings of the 7th Detection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASE2022)",
    address = "Nancy, France",
    month = "November",
    year = "2022",
    pages = "1--5",
    abstract = "We present a machine sound dataset to benchmark domain generalization techniques for anomalous sound detection (ASD). Domain shifts are differences in data distributions that can degrade the detection performance, and handling them is a major issue for the application of ASD systems. While currently available datasets for ASD tasks assume that occurrences of domain shifts are known, in practice, they can be difficult to detect. To handle such domain shifts, domain generalization techniques that perform well regardless of the domains should be investigated. In this paper, we present the first ASD dataset for the domain generalization techniques, called MIMII DG. The dataset consists of five machine types and three domain shift scenarios for each machine type. The dataset is dedicated to the domain generalization task with features such as multiple different values for parameters that cause domain shifts and introduction of domain shifts that can be difficult to detect, such as shifts in the background noise. Experimental results using two baseline systems indicate that the dataset reproduces domain shift scenarios and is useful for benchmarking domain generalization techniques."
}

@inproceedings{Harada2021,
    author = "Harada, Noboru and Niizumi, Daisuke and Takeuchi, Daiki and Ohishi, Yasunori and Yasuda, Masahiro and Saito, Shoichiro",
    title = "{ToyADMOS2}: Another Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection under Domain Shift Conditions",
    booktitle = "Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)",
    address = "Barcelona, Spain",
    month = "November",
    year = "2021",
    pages = "1--5",
    abstract = "This paper proposes a new large-scale dataset called “ToyADMOS” for anomaly detection in machine operating sounds (ADMOS). As with our previous ToyADMOS dataset, we collected a large number of operating sounds of miniature machines (toys) under normal and anomaly conditions by deliberately damaging them, but extended them in this case by providing a controlled depth of damages in the anomaly samples. Since typical application scenarios of ADMOS require robust performance under domain-shift conditions, the ToyADMOS2 dataset is designed for evaluating systems under such conditions. The released dataset consists of two sub-datasets for machine-condition inspection: fault diagnosis of machines with geometrically fixed tasks and fault diagnosis of machines with moving tasks. Domain shifts are represented by introducing several differences in operating conditions, such as the use of the same machine type but with different models and parts configurations, operating speeds, microphone arrangements, etc. Each subdataset contains over 27 k samples of normal machine-operating sounds and over 8 k samples of anomalous sounds recorded with five to eight microphones. The dataset is freely available for download at https://github.com/nttcslab/ToyADMOS2-dataset and https://doi.org/10.5281/zenodo.4580270.",
    isbn = "978-84-09-36072-7",
    doi. = "10.5281/zenodo.5770113"
}

@inproceedings{Dohi2022-2,
    author = "Dohi, Kota and Imoto, Keisuke and Harada, Noboru and Niizumi, Daisuke and Koizumi, Yuma and Nishida, Tomoya and Purohit, Harsh and Tanabe, Ryo and Endo, Takashi and Yamamoto, Masaaki and Kawaguchi, Yohei",
    title = "Description and Discussion on {DCASE} 2022 Challenge Task 2: Unsupervised Anomalous Sound Detection for Machine Condition Monitoring Applying Domain Generalization Techniques",
    booktitle = "Proceedings of the 7th Detection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASE2022)",
    address = "Nancy, France",
    month = "November",
    year = "2022",
    pages = "1--5",
    abstract = "We present the task description and discussion on the results of the DCASE 2022 Challenge Task 2: “Unsupervised anomalous sound detection (ASD) for machine condition monitoring applying domain generalization techniques”. Domain shifts are a critical problem for the application of ASD systems. Because domain shifts can change the acoustic characteristics of data, a model trained in a source domain performs poorly for a target domain. In DCASE 2021 Challenge Task 2, we organized an ASD task for handling domain shifts. In this task, it was assumed that the occurrences of domain shifts are known. However, in practice, the domain of each sample may not be given, and the domain shifts can occur implicitly. In 2022 Task 2, we focus on domain generalization techniques that detects anomalies regardless of the domain shifts. Specifically, the domain of each sample is not given in the test data and only one threshold is allowed for all domains. Analysis of 81 submissions from 31 teams revealed two remarkable types of domain generalization techniques: 1) domain-mixing-based approach that obtains generalized representations and 2) domain-classification-based approach that explicitly or implicitly classifies different domains to improve detection performance for each domain."
}