Journal Article

Status, challenges and trends of data-intensive supercomputing

Lei, Yujia, Ren, Pei, Dong, Xiaoshe, Wu, Weiguo, Zhang, Xingjun, Chen, Mo, Qu, Yuqi, Wei, Jia, Jiang, Qiyu, Wang, Qiang, Wang, Longxiang, Zhang, Kaili

2022

Overview

Supercomputing technology has been supporting the solution of cutting-edge scientific and complex engineering problems since its inception—serving as a comprehensive representation of the most advanced computer hardware and software technologies over a period of time. Over the course of nearly 80 years of development, supercomputing has progressed from being oriented towards computationally intensive tasks, to being oriented towards a hybrid of computationally and data-intensive tasks. Driven by the continuous development of high performance data analytics (HPDA) applications—such as big data, deep learning, and other intelligent tasks—supercomputing storage systems are facing challenges such as a sudden increase in data volume for computational processing tasks, increased and diversified computing power of supercomputing systems, and higher reliability and availability requirements. Based on this, data-intensive supercomputing, which is deeply integrated with data centers and smart computing centers, aims to solve the problems of complex data type optimization, mixed-load optimization, multi-protocol support, and interoperability on the storage system—thereby becoming the main protagonist of research and development today and for some time in the future. This paper first introduces key concepts in HPDA and data-intensive computing, and then illustrates the extent to which existing platforms support data-intensive applications by analyzing the most representative supercomputing platforms today (Fugaku, Summit, Sunway TaihuLight, and Tianhe 2A). This is followed by an illustration of the actual demand for data-intensive applications in today’s mainstream scientific and industrial communities from the perspectives of both scientific and commercial applications. Next, we provide an outlook on future trends and potential challenges data-intensive supercomputing is facing. In a word, this paper provides researchers and practitioners with a quick overview of the key concepts and developments in supercomputing, and captures the current and future data-intensive supercomputing research hotspots and key issues that need to be addressed.

• Publisher: Springer Nature Singapore,Springer Nature B.V
• Subject: Big Data / Cloud computing / Computation / Computer Hardware / Computer peripherals / Computer Science / Computer Systems Organization and Communication Networks / Data analysis / Data processing / Deep learning / Earthquakes / High performance computing / Optimization / Power / R&D / Research & development / Review Paper / Software / Storage systems / Supercomputers / System reliability / Trends