Catalogue Search | MBRL
Are you sure you want to remove the book from the shelf?
{{itemTitle}}
209
result(s) for
"Yin, Wotao"
Sort by:
On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers
The formulation
min
x
,
y
f
(
x
)
+
g
(
y
)
,
subject
to
A
x
+
B
y
=
b
,
where
f
and
g
are extended-value convex functions, arises in many application areas such as signal processing, imaging and image processing, statistics, and machine learning either naturally or after variable splitting. In many common problems, one of the two objective functions is strictly convex and has Lipschitz continuous gradient. On this kind of problem, a very effective approach is the alternating direction method of multipliers (ADM or ADMM), which solves a sequence of
f
/
g
-decoupled subproblems. However, its effectiveness has not been matched by a provably fast rate of convergence; only sublinear rates such as
O
(1 /
k
) and
O
(
1
/
k
2
)
were recently established in the literature, though the
O
(1 /
k
) rates do not require strong convexity. This paper shows that global linear convergence can be guaranteed under the assumptions of strong convexity and Lipschitz gradient on one of the two functions, along with certain rank assumptions on
A
and
B
. The result applies to various generalizations of ADM that allow the subproblems to be solved faster and less exactly in certain manners. The derived rate of convergence also provides some theoretical guidance for optimizing the ADM parameters. In addition, this paper makes meaningful extensions to the existing global convergence theory of ADM generalizations.
Journal Article
A Globally Convergent Algorithm for Nonconvex Optimization Based on Block Coordinate Update
Nonconvex optimization arises in many areas of computational science and engineering. However, most nonconvex optimization algorithms are only known to have local convergence or subsequence convergence properties. In this paper, we propose an algorithm for nonconvex optimization and establish its global convergence (of the whole sequence) to a critical point. In addition, we give its asymptotic convergence rate and numerically demonstrate its efficiency. In our algorithm, the variables of the underlying problem are either treated as one block or multiple disjoint blocks. It is assumed that each non-differentiable component of the objective function, or each constraint, applies only to one block of variables. The differentiable components of the objective function, however, can involve multiple blocks of variables together. Our algorithm updates one block of variables at a time by minimizing a certain prox-linear surrogate, along with an extrapolation to accelerate its convergence. The order of update can be either deterministically cyclic or randomly shuffled for each cycle. In fact, our convergence analysis only needs that each block be updated at least once in every fixed number of iterations. We show its global convergence (of the whole sequence) to a critical point under fairly loose conditions including, in particular, the Kurdyka–Łojasiewicz condition, which is satisfied by a broad class of nonconvex/nonsmooth applications. These results, of course, remain valid when the underlying problem is convex. We apply our convergence results to the coordinate descent iteration for non-convex regularized linear regression, as well as a modified rank-one residue iteration for nonnegative matrix factorization. We show that both applications have global convergence. Numerically, we tested our algorithm on nonnegative matrix and tensor factorization problems, where random shuffling clearly improves the chance to avoid low-quality local solutions.
Journal Article
Global Convergence of ADMM in Nonconvex Nonsmooth Optimization
In this paper, we analyze the convergence of the alternating direction method of multipliers (ADMM) for minimizing a nonconvex and possibly nonsmooth objective function,
ϕ
(
x
0
,
…
,
x
p
,
y
)
, subject to coupled linear equality constraints. Our ADMM updates each of the primal variables
x
0
,
…
,
x
p
,
y
, followed by updating the dual variable. We separate the variable
y
from
x
i
’s as it has a special role in our analysis. The developed convergence guarantee covers a variety of nonconvex functions such as piecewise linear functions,
ℓ
q
quasi-norm, Schatten-
q
quasi-norm (
0
<
q
<
1
), minimax concave penalty (MCP), and smoothly clipped absolute deviation penalty. It also allows nonconvex constraints such as compact manifolds (e.g., spherical, Stiefel, and Grassman manifolds) and linear complementarity constraints. Also, the
x
0
-block can be almost any lower semi-continuous function. By applying our analysis, we show, for the first time, that several ADMM algorithms applied to solve nonconvex models in statistical learning, optimization on manifold, and matrix decomposition are guaranteed to converge. Our results provide sufficient conditions for ADMM to converge on (convex or nonconvex) monotropic programs with three or more blocks, as they are special cases of our model. ADMM has been regarded as a variant to the augmented Lagrangian method (ALM). We present a simple example to illustrate how ADMM converges but ALM diverges with bounded penalty parameter
β
. Indicated by this example and other analysis in this paper, ADMM might be a better choice than ALM for some nonconvex
nonsmooth
problems, because ADMM is not only easier to implement, it is also more likely to converge for the concerned scenarios.
Journal Article
Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions
In this paper, we provide a comprehensive convergence rate analysis of the Douglas-Rachford splitting (DRS), Peaceman-Rachford splitting (PRS), and alternating direction method of multipliers (ADMM) algorithms under various regularity assumptions including strong convexity, Lipschitz differentiability, and bounded linear regularity. The main consequence of this work is that relaxed PRS and ADMM automatically adapt to the regularity of the problem and achieve convergence rates that improve upon the (tight) worst-case rates that hold in the absence of such regularity. All of the results are obtained using simple techniques.
Journal Article
Acceleration of Primal–Dual Methods by Preconditioning and Simple Subproblem Procedures
Primal–dual hybrid gradient (PDHG) and alternating direction method of multipliers (ADMM) are popular first-order optimization methods. They are easy to implement and have diverse applications. As first-order methods, however, they are sensitive to problem conditions and can struggle to reach the desired accuracy. To improve their performance, researchers have proposed techniques such as diagonal preconditioning and inexact subproblems. This paper realizes additional speedup about one order of magnitude. Specifically, we choose general (non-diagonal) preconditioners that are much more effective at reducing the total numbers of PDHG/ADMM iterations than diagonal ones. Although the subproblems may lose their closed-form solutions, we show that it suffices to solve each subproblem approximately with a few proximal-gradient iterations or a few epochs of proximal block-coordinate descent, which are simple and have closed-form steps. Global convergence of this approach is proved when the inner iterations are fixed. Our method opens the choices of preconditioners and maintains both low per-iteration cost and global convergence. Consequently, on several typical applications of primal–dual first-order methods, we obtain 4–95
×
speedup over the existing state-of-the-art.
Journal Article
Feasibility-based fixed point networks
Inverse problems consist of recovering a signal from a collection of noisy measurements. These problems can often be cast as feasibility problems; however, additional regularization is typically necessary to ensure accurate and stable recovery with respect to data perturbations. Hand-chosen analytic regularization can yield desirable theoretical guarantees, but such approaches have limited effectiveness recovering signals due to their inability to leverage large amounts of available data. To this end, this work fuses data-driven regularization and convex feasibility in a theoretically sound manner. This is accomplished using feasibility-based fixed point networks (F-FPNs). Each F-FPN defines a collection of nonexpansive operators, each of which is the composition of a projection-based operator and a data-driven regularization operator. Fixed point iteration is used to compute fixed points of these operators, and weights of the operators are tuned so that the fixed points closely represent available data. Numerical examples demonstrate performance increases by F-FPNs when compared to standard TV-based recovery methods for CT reconstruction and a comparable neural network based on algorithm unrolling. Codes are available on Github: github.com/howardheaton/feasibility_fixed_point_networks.
Journal Article
A New Alternating Minimization Algorithm for Total Variation Image Reconstruction
We propose, analyze, and test an alternating minimization algorithm for recovering images from blurry and noisy observations with total variation (TV) regularization. This algorithm arises from a new half-quadratic model applicable to not only the anisotropic but also the isotropic forms of TV discretizations. The per-iteration computational complexity of the algorithm is three fast Fourier transforms. We establish strong convergence properties for the algorithm including finite convergence for some variables and relatively fast exponential (or $q$-linear in optimization terminology) convergence for the others. Furthermore, we propose a continuation scheme to accelerate the practical convergence of the algorithm. Extensive numerical results show that our algorithm performs favorably in comparison to several state-of-the-art algorithms. In particular, it runs orders of magnitude faster than the lagged diffusivity algorithm for TV-based deblurring. Some extensions of our algorithm are also discussed.
Journal Article
An Efficient TVL1 Algorithm for Deblurring Multichannel Images Corrupted by Impulsive Noise
We extend the alternating minimization algorithm recently proposed in [Y. Wang, J. Yang, W. Yin, and Y. Zhang, SIAM J. Imag. Sci., 1 (2008), pp. 248-272]; [J. Yang, W. Yin, Y. Zhang, and Y. Wang, SIAM J. Imag. Sci., 2 (2009), pp. 569-592] to the case of recovering blurry multichannel (color) images corrupted by impulsive rather than Gaussian noise. The algorithm minimizes the sum of a multichannel extension of total variation and a data fidelity term measured in the $\\ell_1$-norm, and is applicable to both salt-and-pepper and random-valued impulsive noise. We derive the algorithm by applying the well-known quadratic penalty function technique and prove attractive convergence properties, including finite convergence for some variables and $q$-linear convergence rate. Under periodic boundary conditions, the main computational requirements of the algorithm are fast Fourier transforms and a low-complexity Gaussian elimination procedure. Numerical results on images with different blurs and impulsive noise are presented to demonstrate the efficiency of the algorithm. In addition, it is numerically compared to the least absolute deviation method [H. Y. Fu, M. K. Ng, M. Nikolova, and J. L. Barlow, SIAM J. Sci. Comput., 27 (2006), pp. 1881-1902] and the two-phase method [J. F. Cai, R. Chan, and M. Nikolova, AIMS J. Inverse Problems and Imaging, 2 (2008), pp. 187-204] for recovering grayscale images. We also present results of recovering multichannel images.
Journal Article
A Fast Algorithm for Edge-Preserving Variational Multichannel Image Restoration
Variational models with $\\ell_1$-norm based regularization, in particular total variation (TV) and its variants, have long been known to offer superior image restoration quality, but processing speed remained a bottleneck, preventing their widespread use in the practice of color image processing. In this paper, by extending the grayscale image deblurring algorithm proposed in [Y. Wang, J. Yang, W. Yin, and Y. Zhang, SIAM J. Imaging Sci., 1 (2008), pp. 248-272], we construct a simple and efficient algorithm for multichannel image deblurring and denoising, applicable to both within-channel and cross-channel blurs in the presence of additive Gaussian noise. The algorithm restores an image by minimizing an energy function consisting of an $\\ell_2$-norm fidelity term and a regularization term that can be either TV, weighted TV, or regularization functions based on higher-order derivatives. Specifically, we use a multichannel extension of the classic TV regularizer (MTV) and derive our algorithm from an extended half-quadratic transform of Geman and Yang [IEEE Trans. Image Process., 4 (1995), pp. 932-946]. For three-channel color images, the per-iteration computation of this algorithm is dominated by six fast Fourier transforms. The convergence results in [Y. Wang, J. Yang, W. Yin, and Y. Zhang, SIAM J. Imaging Sci., 1 (2008), pp. 248-272] for single-channel images, including global convergence with a strong $q$-linear rate and finite convergence for some quantities, are extended to this algorithm. We present numerical results including images recovered from various types of blurs, comparisons between our results and those obtained from the deblurring functions in MATLAB's Image Processing Toolbox, as well as images recovered by our algorithm using weighted MTV and higher-order regularization. Our numerical results indicate that the processing speed, as attained by the proposed algorithm, of variational models with TV-like regularization can be made comparable to that of less sophisticated but widely used methods for color image restoration.
Journal Article