Explore the vast range of titles available.

We introduce Devito, a new domain-specific language for implementing high-performance finite-difference partial differential equation solvers. The motivating application is exploration seismology for which methods such as full-waveform inversion and reverse-time migration are used to invert terabytes of seismic data to create images of the Earth's subsurface. Even using modern supercomputers, it can take weeks to process a single seismic survey and create a useful subsurface image. The computational cost is dominated by the numerical solution of wave equations and their corresponding adjoints. Therefore, a great deal of effort is invested in aggressively optimizing the performance of these wave-equation propagators for different computer architectures. Additionally, the actual set of partial differential equations being solved and their numerical discretization is under constant innovation as increasingly realistic representations of the physics are developed, further ratcheting up the cost of practical solvers. By embedding a domain-specific language within Python and making heavy use of SymPy, a symbolic mathematics library, we make it possible to develop finite-difference simulators quickly using a syntax that strongly resembles the mathematics. The Devito compiler reads this code and applies a wide range of analysis to generate highly optimized and parallel code. This approach can reduce the development time of a verified and optimized solver from months to days.

Seismic imaging and parameter estimation are an import class of inverse problemswith practical relevance in resource exploration, carbon control and monitoring systemsfor geohazards. The goal of seismic inverse problems is to image subsurface geologicalstructures and estimate physical rock properties such as wave speed or density. Mathe-matically, this can be achieved by solving an optimization problem in which we minimizethe mismatch between numerically modeled data and observed data from a seismic sur-vey. As wave propagation through a medium is described by wave equations, seismicinverse problems involve solving a large number of partial differential equations (PDEs)during numerical optimization using finite difference modeling, making them computation-ally expensive. Additionally, seismic inverse problems are typically ill-posed, non-convexor ill-conditioned, thus making them challenging from a mathematical standpoint as well.Similar to the field of deep learning, this calls for software that is not only optimized for per-formance, but also enables geophysical domain specialists to experiment with algorithmsin high-level programming languages and using different computing environments, such ashigh-performance computing (HPC) clusters or the cloud. Furthermore, they call for theadaption of dimensionality reduction techniques and stochastic algorithms to address com-putational cost from the algorithmic side. This thesis makes three distinct contributions toaddress computational challenges encountered in seismic inverse problems and to facili-tate algorithmic development in this field. Part one introduces a large-scale framework forseismic modeling and inversion based on the paradigm of separation of concerns, whichcombines a user interface based on domain specific abstractions with a Python packagefor automatic code generation to solve the underlying PDEs. The modular code structuremakes it possible to manage the complexity of a seismic inversion code, while matrix-freelinear operators and data containers enable the implementation of algorithms in a fashionthat closely resembles the underlying mathematical notation. The second contribution of this thesis is an algorithm for seismic imaging, that addresses its high computational costand large memory imprint through a combination of on-the-fly Fourier transforms, stochas-tic sampling techniques and sparsity-promoting optimization. The algorithm combines thebest of both time- and frequency-domain inversion, as the memory imprint is independentof the number of modeled time steps, while time-to-frequency conversions avoid the needto solve Helmholtz equations, which involve inverting ill-conditioned matrices. Part threeof this thesis introduces a novel approach for adapting the cloud for high-performancecomputing applications like seismic imaging, which does not rely on a fixed cluster ofpermanently running virtual machines. Instead, computational resources are automaticallystarted and terminated by the cloud environment during runtime and the workflow takesadvantage of cloud-native technologies such as event-driven computations and container-ized batch processing. The performance and cost analysis shows that this approach is ableto address current shortcomings of the cloud such as inferior resilience, while at the sametime reducing operating cost up to an order of magnitude. As such, the workflow providesa strategy for cost effectively running large-scale seismic imaging problems in the cloudand is a viable alternative to conventional HPC clusters.

We present the SLIM (https://github.com/slimgroup) open-source software framework for computational geophysics, and more generally, inverse problems based on the wave-equation (e.g., medical ultrasound). We developed a software environment aimed at scalable research and development by designing multiple layers of abstractions. This environment allows the researchers to easily formulate their problem in an abstract fashion, while still being able to exploit the latest developments in high-performance computing. We illustrate and demonstrate the benefits of our software design on many geophysical applications, including seismic inversion and physics-informed machine learning for geophysics (e.g., loop unrolled imaging, uncertainty quantification), all while facilitating the integration of external software.

We present the Seismic Laboratory for Imaging and Modeling/Monitoring (SLIM) open-source software framework for computational geophysics and, more generally, inverse problems involving the wave-equation (e.g., seismic and medical ultrasound), regularization with learned priors, and learned neural surrogates for multiphase flow simulations. By integrating multiple layers of abstraction, our software is designed to be both readable and scalable. This allows researchers to easily formulate their problems in an abstract fashion while exploiting the latest developments in high-performance computing. We illustrate and demonstrate our design principles and their benefits by means of building a scalable prototype for permeability inversion from time-lapse crosswell seismic data, which aside from coupling of wave physics and multiphase flow, involves machine learning.

Fourier neural operators (FNOs) are a recently introduced neural network architecture for learning solution operators of partial differential equations (PDEs), which have been shown to perform significantly better than comparable deep learning approaches. Once trained, FNOs can achieve speed-ups of multiple orders of magnitude over conventional numerical PDE solvers. However, due to the high dimensionality of their input data and network weights, FNOs have so far only been applied to two-dimensional or small three-dimensional problems. To remove this limited problem-size barrier, we propose a model-parallel version of FNOs based on domain-decomposition of both the input data and network weights. We demonstrate that our model-parallel FNO is able to predict time-varying PDE solutions of over 2.6 billion variables on Perlmutter using up to 512 A100 GPUs and show an example of training a distributed FNO on the Azure cloud for simulating multiphase CO\\(_2\\) dynamics in the Earth's subsurface.

Following AI scaling trends, frontier models continue to grow in size and continue to be trained on larger datasets. Training these models requires huge investments in exascale computational resources, which has in turn driven developtment of distributed deep learning methods. Data parallelism is an essential approach to speed up training, but it requires frequent global communication between workers, which can bottleneck training at the largest scales. In this work, we propose a method called Pseudo-Asynchronous Local SGD (PALSGD) to improve the efficiency of data-parallel training. PALSGD is an extension of Local SGD (Stich, 2018) and DiLoCo (Douillard et al., 2023), designed to further reduce communication frequency by introducing a pseudo-synchronization mechanism. PALSGD allows the use of longer synchronization intervals compared to standard Local SGD. Despite the reduced communication frequency, the pseudo-synchronization approach ensures that model consistency is maintained, leading to performance results comparable to those achieved with more frequent synchronization. Furthermore, we provide a theoretical analysis of PALSGD, establishing its convergence and deriving its convergence rate. This analysis offers insights into the algorithm's behavior and performance guarantees. We evaluated PALSGD on image classification and language modeling tasks. Our results show that PALSGD achieves better performance in less time compared to existing methods like Distributed Data Parallel (DDP), and DiLoCo. Notably, PALSGD trains 18.4% faster than DDP on ImageNet-1K with ResNet-50, 24.4% faster than DDP on TinyStories with GPT-Neo-125M, and 21.1% faster than DDP on TinyStories with GPT-Neo-8M.

Solving partial differential equations with deep learning makes it possible to reduce simulation times by multiple orders of magnitude and unlock scientific methods that typically rely on large numbers of sequential simulations, such as optimization and uncertainty quantification. Two of the largest challenges of adopting scientific AI for industrial problem settings is that training datasets must be simulated in advance and that neural networks for solving large-scale PDEs exceed the memory capabilities of current GPUs. We introduce a distributed programming API in the Julia language for simulating training data in parallel on the cloud and without requiring users to manage the underlying HPC infrastructure. In addition, we show that model-parallel deep learning based on domain decomposition allows us to scale neural networks for solving PDEs to commercial-scale problem settings and achieve above 90% parallel efficiency. Combining our cloud API for training data generation and model-parallel deep learning, we train large-scale neural networks for solving the 3D Navier-Stokes equation and simulating 3D CO2 flow in porous media. For the CO2 example, we simulate a training dataset based on a commercial carbon capture and storage (CCS) project and train a neural network for CO2 flow simulation on a 3D grid with over 2 million cells that is 5 orders of magnitudes faster than a conventional numerical simulator and 3,200 times cheaper.

We introduce Devito, a new domain-specific language for implementing high-performance finite difference partial differential equation solvers. The motivating application is exploration seismology where methods such as Full-Waveform Inversion and Reverse-Time Migration are used to invert terabytes of seismic data to create images of the earth's subsurface. Even using modern supercomputers, it can take weeks to process a single seismic survey and create a useful subsurface image. The computational cost is dominated by the numerical solution of wave equations and their corresponding adjoints. Therefore, a great deal of effort is invested in aggressively optimizing the performance of these wave-equation propagators for different computer architectures. Additionally, the actual set of partial differential equations being solved and their numerical discretization is under constant innovation as increasingly realistic representations of the physics are developed, further ratcheting up the cost of practical solvers. By embedding a domain-specific language within Python and making heavy use of SymPy, a symbolic mathematics library, we make it possible to develop finite difference simulators quickly using a syntax that strongly resembles the mathematics. The Devito compiler reads this code and applies a wide range of analysis to generate highly optimized and parallel code. This approach can reduce the development time of a verified and optimized solver from months to days.

In inverse problems, we often have access to data consisting of paired samples \\((x,y)\\sim p_{X,Y}(x,y)\\) where \\(y\\) are partial observations of a physical system, and \\(x\\) represents the unknowns of the problem. Under these circumstances, we can employ supervised training to learn a solution \\(x\\) and its uncertainty from the observations \\(y\\). We refer to this problem as the \"supervised\" case. However, the data \\(y\\sim p_{Y}(y)\\) collected at one point could be distributed differently than observations \\(y'\\sim p_{Y}'(y')\\), relevant for a current set of problems. In the context of Bayesian inference, we propose a two-step scheme, which makes use of normalizing flows and joint data to train a conditional generator \\(q_{\\theta}(x|y)\\) to approximate the target posterior density \\(p_{X|Y}(x|y)\\). Additionally, this preliminary phase provides a density function \\(q_{\\theta}(x|y)\\), which can be recast as a prior for the \"unsupervised\" problem, e.g.~when only the observations \\(y'\\sim p_{Y}'(y')\\), a likelihood model \\(y'|x\\), and a prior on \\(x'\\) are known. We then train another invertible generator with output density \\(q'_{\\phi}(x|y')\\) specifically for \\(y'\\), allowing us to sample from the posterior \\(p_{X|Y}'(x|y')\\). We present some synthetic results that demonstrate considerable training speedup when reusing the pretrained network \\(q_{\\theta}(x|y')\\) as a warm start or preconditioning for approximating \\(p_{X|Y}'(x|y')\\), instead of learning from scratch. This training modality can be interpreted as an instance of transfer learning. This result is particularly relevant for large-scale inverse problems that employ expensive numerical simulations.

Uncertainty quantification for full-waveform inversion provides a probabilistic characterization of the ill-conditioning of the problem, comprising the sensitivity of the solution with respect to the starting model and data noise. This analysis allows to assess the confidence in the candidate solution and how it is reflected in the tasks that are typically performed after imaging (e.g., stratigraphic segmentation following reservoir characterization). Classically, uncertainty comes in the form of a probability distribution formulated from Bayesian principles, from which we seek to obtain samples. A popular solution involves Monte Carlo sampling. Here, we propose instead an approach characterized by training a deep network that \"pushes forward\" Gaussian random inputs into the model space (representing, for example, density or velocity) as if they were sampled from the actual posterior distribution. Such network is designed to solve a variational optimization problem based on the Kullback-Leibler divergence between the posterior and the network output distributions. This work is fundamentally rooted in recent developments for invertible networks. Special invertible architectures, besides being computational advantageous with respect to traditional networks, do also enable analytic computation of the output density function. Therefore, after training, these networks can be readily used as a new prior for a related inversion problem. This stands in stark contrast with Monte-Carlo methods, which only produce samples. We validate these ideas with an application to angle-versus-ray parameter analysis for reservoir characterization.