Explore the vast range of titles available.

The performance of global ocean biogeochemical models can be quantified as the misfit between modeled tracer distributions and observations, which is sought to be minimized during parameter optimization. These models are computationally expensive due to the long spin‐up time required to reach equilibrium, and therefore optimization is often laborious. To reduce the required computational time, we investigate whether optimization of a biogeochemical model with shorter spin‐ups provides the same optimized parameters as one with a full‐length, equilibrated spin‐up over several millennia. We use the global ocean biogeochemical model MOPS with a range of lengths of model spin‐up and calibrate the model against synthetic observations derived from previous model runs using a derivative‐free optimization algorithm (DFO‐LS). When initiating the biogeochemical model with tracer distributions that differ from the synthetic observations used for calibration, a minimum spin‐up length of 2,000 years was required for successful optimization due to certain parameters which influence the transport of matter from the surface to the deeper ocean, where timescales are longer. However, preliminary results indicate that successful optimization may occur with an even shorter spin‐up by a judicious choice of initial condition, here the synthetic observations used for calibration, suggesting a fruitful avenue for future research. Plain Language Summary Global ocean biogeochemical models allow us to simulate ocean biological and chemical variables throughout the global ocean, and are necessary for climate change projections. They are very computationally expensive due to the required spin‐up needed for the model to reach a steady state, and so any improvements to the model need to be sought in an efficient way. One way we investigate here is to first make the model less computationally expensive by using a shorter spin‐up, and then we apply an efficient optimization algorithm to tune the model to observations. By shortening the spin‐up from 3,000 to 2,000 years we show that we can still reach a successfully optimized model. Preliminary results indicate that successful optimization may occur with an even shorter spin‐up when the biogeochemical model is initialized from a more accurate state, which highlights a future avenue for research that may encourage more systematic tuning of computationally expensive ocean biogeochemical models. Key Points Global ocean biogeochemical models are computationally expensive due to the long spin‐up time required to reach equilibrium A shortened spin up of 2,000 years during parameter optimization can be successfully optimized How short a spin‐up one can successfully optimize is influenced by the parameters being calibrated and the initial conditions of the model

Simulation optimization (SO) refers to the optimization of an objective function subject to constraints, both of which can be evaluated through a stochastic simulation. To address specific features of a particular simulation—discrete or continuous decisions, expensive or cheap simulations, single or multiple outputs, homogeneous or heterogeneous noise—various algorithms have been proposed in the literature. As one can imagine, there exist several competing algorithms for each of these classes of problems. This document emphasizes the difficulties in SO as compared to algebraic model-based mathematical programming, makes reference to state-of-the-art algorithms in the field, examines and contrasts the different approaches used, reviews some of the diverse applications that have been tackled by these methods, and speculates on future directions in the field.

Sustainable management of groundwater resources under changing climatic conditions require an application of reliable and accurate predictions of groundwater levels. Mechanistic multi-scale, multi-physics simulation models are often too hard to use for this purpose, especially for groundwater managers who do not have access to the complex compute resources and data. Therefore, we analyzed the applicability and performance of four modern deep learning computational models for predictions of groundwater levels. We compare three methods for optimizing the models’ hyperparameters, including two surrogate model-based algorithms and a random sampling method. The models were tested using predictions of the groundwater level in Butte County, California, USA, taking into account the temporal variability of streamflow, precipitation, and ambient temperature. Our numerical study shows that the optimization of the hyperparameters can lead to reasonably accurate performance of all models (root mean squared errors of groundwater predictions of 2 meters or less), but the “simplest” network, namely a multilayer perceptron (MLP) performs overall better for learning and predicting groundwater data than the more advanced long short-term memory or convolutional neural networks in terms of prediction accuracy and time-to-solution, making the MLP a suitable candidate for groundwater prediction.

The main objective of this article is to describe the development of two advanced multiobjective optimization methods based on derivative-free techniques and complex computational fluid dynamics (CFD) analysis. Alternatives for the geometry and mesh manipulation techniques are also described. Emphasis is on advanced strategies for the use of computer resource-intensive CFD solvers in the optimization process: indeed, two up-to-date free surface-fitting Reynolds-averaged Navier-Stokes equation solvers are used as analysis tools for the evaluation of the objective function and functional constraints. The two optimization methods are realized and demonstrated on a real design problem: the optimization of the entire hull form of a surface combatant, the David Taylor Model Basin—Model 5415. Realistic functional and geometrical constraints for preventing unfeasible results and to get a final meaningful design are enforced and discussed. Finally, a recently proposed verification and validation methodology is applied to assess uncertainties and errors in simulation-based optimization, based on the differences between the numerically predicted improvement of the objective function and the actual improvement measured in a dedicated experimental campaign. The optimized model demonstrates improved characteristics beyond the numerical and experimental uncertainty, confirming the validity of the simulation-based design frameworks.

In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

In practical applications of optimization it is common to have several conflicting objective functions to optimize. Frequently, these functions are subject to noise or can be of black-box type, preventing the use of derivative-based techniques. We propose a novel multiobjective derivative-free methodology, calling it direct multisearch (DMS), which does not aggregate any of the objective functions. Our framework is inspired by the search/poll paradigm of direct-search methods of directional type and uses the concept of Pareto dominance to maintain a list of nondominated points (from which the new iterates or poll centers are chosen). The aim of our method is to generate as many points in the Pareto front as possible from the polling procedure itself, while keeping the whole framework general enough to accommodate other disseminating strategies, in particular, when using the (here also) optional search step. DMS generalizes to multiobjective optimization (MOO) all direct-search methods of directional type. We prove under the common assumptions used in direct search for single objective optimization that at least one limit point of the sequence of iterates generated by DMS lies in (a stationary form of) the Pareto front. However, extensive computational experience has shown that our methodology has an impressive capability of generating the whole Pareto front, even without using a search step. Two by-products of this paper are (i) the development of a collection of test problems for MOO and (ii) the extension of performance and data profiles to MOO, allowing a comparison of several solvers on a large set of test problems, in terms of their efficiency and robustness to determine Pareto fronts. [PUBLICATION ABSTRACT]

Deep neuro-fuzzy systems (DNFSs) have been successfully applied to real-world problems using the efficient learning process of deep neural networks (DNNs) and reasoning aptitude from fuzzy inference systems (FIS). This study provides a comprehensive review of DNFS dividing it into two essential parts. The first part aims to provide a thorough understanding of DNFS and its architectural representation, whereas the second part reviews DNFS optimization methods. This study aims to assist researchers in understanding the various ways DNFS models are developed by hybridizing DNN and FIS, as well as gradient (derivative)-based methods and metaheuristics (derivative-free) optimization, as discussed in the literature. This study revealed that the proposed DNFS architectures performed 11.6% better than non-fuzzy models, with an overall accuracy of 81.4%. The investigation based on optimization methods revealed that DNFS with metaheuristics optimization methods has shown an overall accuracy of 93.56%, which is 21.10% higher than the DNFS models using gradient-based methods. Additionally, this study showed that DNFS networks presented in the literature have integrated DNN with typical FIS, although more satisfactory results can be obtained using a new generation of FIS termed fractional FIS (FFIS) and Mamdani complex FIS (M-CFIS). Besides, dynamic neural networks are suggested in the replacement of static DNNs to facilitate dynamic learning. Some studies have also demonstrated the optimization of DNFS using classical gradient-based approaches that can affect network performance when solving highly nonlinear problems. This study suggests implementing optimization methods with new and improvised metaheuristics to improve the training and performance of the models.

We present an approach that uses a deep learning model, in particular, a MultiLayer Perceptron, for estimating the missing values of a variable in multivariate time series data. We focus on filling a long continuous gap (e.g., multiple months of missing daily observations) rather than on individual randomly missing observations. Our proposed gap filling algorithm uses an automated method for determining the optimal MLP model architecture, thus allowing for optimal prediction performance for the given time series. We tested our approach by filling gaps of various lengths (three months to three years) in three environmental datasets with different time series characteristics, namely daily groundwater levels, daily soil moisture, and hourly Net Ecosystem Exchange. We compared the accuracy of the gap-filled values obtained with our approach to the widely used R-based time series gap filling methods ImputeTS and mtsdi. The results indicate that using an MLP for filling a large gap leads to better results, especially when the data behave nonlinearly. Thus, our approach enables the use of datasets that have a large gap in one variable, which is common in many long-term environmental monitoring observations.

Reinforcement learning is about learning agent models that make the best sequential decisions in unknown environments. In an unknown environment, the agent needs to explore the environment while exploiting the collected information, which usually forms a sophisticated problem to solve. Derivative-free optimization, meanwhile, is capable of solving sophisticated problems. It commonly uses a sampling-andupdating framework to iteratively improve the solution, where exploration and exploitation are also needed to be well balanced. Therefore, derivative-free optimization deals with a similar core issue as reinforcement learning, and has been introduced in reinforcement learning approaches, under the names of learning classifier systems and neuroevolution/evolutionary reinforcement learning. Although such methods have been developed for decades, recently, derivative-free reinforcement learning exhibits attracting increasing attention. However, recent survey on this topic is still lacking. In this article, we summarize methods of derivative-free reinforcement learning to date, and organize the methods in aspects including parameter updating, model selection, exploration, and parallel/distributed methods. Moreover, we discuss some current limitations and possible future directions, hoping that this article could bring more attentions to this topic and serve as a catalyst for developing novel and efficient approaches.

This paper introduces a novel methodology for the global optimization of general constrained grey-box problems. A grey-box problem may contain a combination of black-box constraints and constraints with a known functional form. The novel features of this work include (i) the selection of initial samples through a subset selection optimization problem from a large number of faster low-fidelity model samples (when a low-fidelity model is available), (ii) the exploration of a diverse set of interpolating and non-interpolating functional forms for representing the objective function and each of the constraints, (iii) the global optimization of the parameter estimation of surrogate functions and the global optimization of the constrained grey-box formulation, and (iv) the updating of variable bounds based on a clustering technique. The performance of the algorithm is presented for a set of case studies representing an expensive non-linear algebraic partial differential equation simulation of a pressure swing adsorption system for CO 2 . We address three significant sources of variability and their effects on the consistency and reliability of the algorithm: (i) the initial sampling variability, (ii) the type of surrogate function, and (iii) global versus local optimization of the surrogate function parameter estimation and overall surrogate constrained grey-box problem. It is shown that globally optimizing the parameters in the parameter estimation model, and globally optimizing the constrained grey-box formulation has a significant impact on the performance. The effect of sampling variability is mitigated by a two-stage sampling approach which exploits information from reduced-order models. Finally, the proposed global optimization approach is compared to existing constrained derivative-free optimization algorithms.