Journal Article

A Hyper-Parameter Optimizer Algorithm Based on Conditional Opposition Local-Based Learning Forbidden Redundant Indexes Adaptive Artificial Bee Colony Applied to Regularized Extreme Learning Machine

Gonzalez, Paulo, Laroze, David, Vasquez-Iglesias, Philip, Pizarro, Amelia E., Fuentes-Concha, Juan, Ahumada-Garcia, Roberto, Zabala-Blanco, David

2024

Overview

Finding the best configuration of a neural network’s hyper-parameters may take too long to be feasible using an exhaustive search, especially when the cardinality of the search space has a big combinatorial number of possible solutions with various hyper-parameters. This problem is aggravated when we also need to optimize the parameters of the neural network, such as the weight of the hidden neurons and biases. Extreme learning machines (ELMs) are part of the random weights neural network family, in which parameters are randomly initialized, and the solution, unlike gradient-descent-based algorithms, can be found analytically. This ability is especially useful for metaheuristic analysis due to its reduced training times allowing a faster optimization process, but the problem of finding the best hyper-parameter configuration is still remaining. In this paper, we propose a modification of the artificial bee colony (ABC) metaheuristic to act as parameterizers for a regularized ELM, incorporating three methods: an adaptive mechanism for ABC to balance exploration (global search) and exploitation (local search), an adaptation of the opposition-based learning technique called opposition local-based learning (OLBL) to strengthen exploitation, and a record of access to the search space called forbidden redundant indexes (FRI) that allow us to avoid redundant calculations and track the explored percentage of the search space. We set ten parameterizations applying different combinations of the proposed methods, limiting them to explore up to approximately 10% of the search space, with results over 98% compared to the maximum performance obtained in the exhaustive search in binary and multiclass datasets. The results demonstrate a promising use of these parameterizations to optimize the hyper-parameters of the R-ELM in datasets with different characteristics in cases where computational efficiency is required, with the possibility of extending its use to other problems with similar characteristics with minor modifications, such as the parameterization of support vector machines, digital image filters, and other neural networks, among others.

• Publisher: MDPI AG
• Subject: Adaptive algorithms / Algorithms / Combinatorial analysis / Configurations / Datasets / Digital imaging / Exploitation / Heuristic methods / Image filters / Machine learning / Neural networks / Optimization / Parameter modification / Parameterization / Redundancy / Searching / Space colonies / Support vector machines / Swarm intelligence