Paper

An Automated Data Mining Framework Using Autoencoders for Feature Extraction and Dimensionality Reduction

Huang, Xin, Liang, Yaxin, Yao, Yue, Zhang, Ziqi, Li, Xinshi

2024

Overview

This study proposes an automated data mining framework based on autoencoders and experimentally verifies its effectiveness in feature extraction and data dimensionality reduction. Through the encoding-decoding structure, the autoencoder can capture the data's potential characteristics and achieve noise reduction and anomaly detection, providing an efficient and stable solution for the data mining process. The experiment compared the performance of the autoencoder with traditional dimensionality reduction methods (such as PCA, FA, T-SNE, and UMAP). The results showed that the autoencoder performed best in terms of reconstruction error and root mean square error and could better retain data structure and enhance the generalization ability of the model. The autoencoder-based framework not only reduces manual intervention but also significantly improves the automation of data processing. In the future, with the advancement of deep learning and big data technology, the autoencoder method combined with a generative adversarial network (GAN) or graph neural network (GNN) is expected to be more widely used in the fields of complex data processing, real-time data analysis and intelligent decision-making.

• Publisher: Cornell University Library, arXiv.org
• Subject: Anomalies / Automation / Big Data / Data analysis / Data mining / Data processing / Data structures / Encoding-Decoding / Error reduction / Feature extraction / Generative adversarial networks / Graph neural networks / Machine learning / Real time