Explore the vast range of titles available.

An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and a smooth function which is an average of many component convex functions. Problems having this structure often arise in machine learning and data mining applications. AS-ADMM combines the ideas of both ADMM and the stochastic gradient methods using variance reduction techniques. One of the ADMM subproblems employs a linearization technique while a similar linearization could be introduced for the other subproblem. For a specified choice of the algorithm parameters, it is shown that the objective error and the constraint violation are O(1/k) relative to the number of outer iterations k. Under a strong convexity assumption, the expected iterate error converges to zero linearly. A linearized variant of AS-ADMM and incremental sampling strategies are also discussed. Numerical experiments with both stochastic and deterministic ADMM algorithms show that AS-ADMM can be particularly effective for structured optimization arising in big data applications.

In this paper, we analyze the convergence of the alternating direction method of multipliers (ADMM) for minimizing a nonconvex and possibly nonsmooth objective function, ϕ ( x 0 , … , x p , y ) , subject to coupled linear equality constraints. Our ADMM updates each of the primal variables x 0 , … , x p , y , followed by updating the dual variable. We separate the variable y from x i ’s as it has a special role in our analysis. The developed convergence guarantee covers a variety of nonconvex functions such as piecewise linear functions, ℓ q quasi-norm, Schatten- q quasi-norm ( 0 < q < 1 ), minimax concave penalty (MCP), and smoothly clipped absolute deviation penalty. It also allows nonconvex constraints such as compact manifolds (e.g., spherical, Stiefel, and Grassman manifolds) and linear complementarity constraints. Also, the x 0 -block can be almost any lower semi-continuous function. By applying our analysis, we show, for the first time, that several ADMM algorithms applied to solve nonconvex models in statistical learning, optimization on manifold, and matrix decomposition are guaranteed to converge. Our results provide sufficient conditions for ADMM to converge on (convex or nonconvex) monotropic programs with three or more blocks, as they are special cases of our model. ADMM has been regarded as a variant to the augmented Lagrangian method (ALM). We present a simple example to illustrate how ADMM converges but ALM diverges with bounded penalty parameter β . Indicated by this example and other analysis in this paper, ADMM might be a better choice than ALM for some nonconvex nonsmooth problems, because ADMM is not only easier to implement, it is also more likely to converge for the concerned scenarios.

As an extension of the alternating direction method of multipliers (ADMM), the semi-proximal ADMM (sPADMM) has been widely used in various fields due to its flexibility and robustness. In this paper, we first show that the two-block sPADMM algorithm can achieve an O ( 1 / K ) non-ergodic convergence rate. Then we propose an accelerated sPADMM (AsPADMM) algorithm by introducing extrapolation techniques and incrementing penalty parameters. The proposed AsPADMM algorithm is proven to converge globally to an optimal solution with a non-ergodic convergence rate of O (1/ K ). Furthermore, the AsPADMM can be extended and combined with the symmetric Gauss-Seidel decomposition to achieve an accelerated ADMM for multi-block problems. Finally, we apply the proposed AsPADMM to solving the multi-block subproblems in difference-of-convex algorithms for robust low-rank tensor completion problems and mixed sparse optimization problems. The numerical results suggest that the acceleration techniques bring about a notable improvement in the convergence speed.

This paper introduces a parallel and distributed algorithm for solving the following minimization problem with linear constraints: minimize f 1 ( x 1 ) + ⋯ + f N ( x N ) subject to A 1 x 1 + ⋯ + A N x N = c , x 1 ∈ X 1 , … , x N ∈ X N , where N ≥ 2 , f i are convex functions, A i are matrices, and X i are feasible sets for variable x i . Our algorithm extends the alternating direction method of multipliers (ADMM) and decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This paper shows that the classic ADMM can be extended to the N -block Jacobi fashion and preserve convergence in the following two cases: (i) matrices A i are mutually near-orthogonal and have full column-rank, or (ii) proximal terms are added to the N subproblems (but without any assumption on matrices A i ). In the latter case, certain proximal terms can let the subproblem be solved in more flexible and efficient ways. We show that ‖ x k + 1 - x k ‖ M 2 converges at a rate of o (1 /  k ) where M is a symmetric positive semi-definte matrix. Since the parameters used in the convergence analysis are conservative, we introduce a strategy for automatically tuning the parameters to substantially accelerate our algorithm in practice. We implemented our algorithm (for the case ii above) on Amazon EC2 and tested it on basis pursuit problems with >300 GB of distributed data. This is the first time that successfully solving a compressive sensing problem of such a large scale is reported.

Methods based on the alternating direction method of multipliers (ADMM) has attracted academic attention because of its excellent convergence performance and potential application scenarios in many machine learning or optimization fields. However, classical distributed ADMM algorithm assumed ideal network communication, which do not consider the impact of network delay on computing performance. In this paper, based on the strategy of selecting bridges with lowest network latency and appropriate iterative process, we propose a latency aware distributed ADMM algorithm to alleviate the impact of network delay. The classical algorithm and proposed algorithm are tested and compared in real network scenarios. Experiments show that the proposed algorithm reduces the running time and improves the computing performance. Especially in networks with large delay, the effect is more obvious.

This paper proposes a distributed empirical likelihood (DEL) method for performing an integrative analysis of multiple data sources with the flexibility of handling either homogeneous or heterogeneous data. The proposed DEL method does not require pooling individual data sets into a centralized operational platform, so the privacy of subject-level information in individual data sources is protected. The DEL method is shown to be almost surely equal to the centralized empirical likelihood approach that would be adopted if individual data sets were combined and stored at one place. We establish the large-sample properties and algorithm convergence of the DEL method. We also illustrate the numerical performance of the DEL method using simulation studies and a real-data example, in which the DEL method is clearly advantageous over the classical meta-estimation method when analyzing unbalanced data sets.

Proximal algorithms are useful for obtaining solutions to difficult optimization problems, especially those involving nonsmooth or composite objective functions. A proximal algorithm is one whose basic iterations involve the proximal operator of some function, whose evaluation requires solving a specific optimization problem that is typically easier than the original problem. Many familiar algorithms can be cast in this form, and this \"proximal view\" turns out to provide a set of broad organizing principles for many algorithms useful in statistics and machine learning. In this paper, we show how a number of recent advances in this area can inform modern statistical practice. We focus on several main themes: (1) variable splitting strategies and the augmented Lagrangian; (2) the broad utility of envelope (or variational) representations of objective functions; (3) proximal algorithms for composite objective functions; and (4) the surprisingly large number of functions for which there are closed-form solutions of proximal operators. We illustrate our methodology with regularized Logistic and Poisson regression incorporating a nonconvex bridge penalty and a fused lasso penalty. We also discuss several related issues, including the convergence of nondescent algorithms, acceleration and optimization for nonconvex functions. Finally, we provide directions for future research in this exciting area at the intersection of statistics and optimization.

We introduce a novel matrix recurrence yielding a new spectral analysis of the local transient convergence behavior of the alternating direction method of multipliers (ADMM), for the particular case of a quadratic program or a linear program. We identify a particular combination of vector iterates whose convergence can be analyzed via a spectral analysis. The theory predicts that ADMM should go through up to four convergence regimes, such as constant step convergence or linear convergence, ending with the latter when close enough to the optimal solution if the optimal solution is unique and satisfies strict complementarity. [PUBLICATION ABSTRACT]

Electroencephalogram (EEG) source imaging (ESI) is highly underdetermined, which poses a long-standing challenge in neuroimaging. Traditional methods typically rely on predefined priors to constrain the solution space; however, the need for manual parameter adjustments often makes it difficult to achieve optimal integration of prior information. Although recent deep learning methods can automatically update parameters in a data-driven manner, their black-box characteristics lead to a lack of interpretability and the need for extensive training sets. To integrate the advantages of these two types of methods, we propose a novel neural network model based on deep unfolding, called variation sparse source imaging network (VSSI2p-Net). Specifically, we introduce variation sparsity and ℓ2,p norm (0

Journal Article

Share this book

Add to My Shelf