Paper

A Roadmap for Big Model

Qin, Yujia, Liu, Zhenghao, Hu, Shengding, Zhang, Jiajun, Zhang, Jing, Wang, Guoqiang, Chang, Baobao, Ding, Ning, Pang, Tianyu, Yao, Yuan, Pan, Yingwei, Long, Zhou, Shi, Chence, Zhang, Quanshi, Li, Juanzi, Dong, Qingxiu, Liu, Yang, Lv, Xin, Hu, Anwen, Cao, Xigang, Zhai, Jidong, Li, Haoran, Wang, Liwei, Zhu, Jun, Su, Hang, Lu, Jiwen, Xu, Minghao, He, Xiaodong, Sui, Zhifang, Liu, Zhiyuan, He, Jiaao, Zhan, Weidong, Yan, Rui, Qin, Jin, Tang, Jie, Wang, Shuo, Zhao, Wenliang, Zhang, Liang, Zhang, Zhengyan, Wang, Yisen, Zhao, Weilin, Ling, Zenan, Ding, Ming, Han, Xianpei, Gao, Yizhao, Cao, Shulin, Nanyi Fei, Shen, Huawei, Sha Yuan, Yu, Jifan, Zhang, Hui, Ji-Rong, Wen, Zhu, Fangwei, Cao, Boxi, Wang, Xiaozhi, Xiao, Chaojun, Zhang, Borui, Song, Ruihua, Xue, Weicheng, Li, Peng, Dong, Yinpeng, Chen, Weize, Sun, Maosong, Bao, Junwei, Tan, Zhixing, Tang, Jian, Huang, Lingxiao, Zeng, Zheni, Lu, Zhiwu, Long, Mingsheng, Huang, Minlie, Zheng, Wenzhao, Zheng, Liang, Sun, Hao, Wang, Ziyi, Li, Mengjie, Pan, Xiang, Xu, Han, Zhou, Yu, Hou, Lei, Lin, Zhouchen, Chu, Xiaoyu, Li, Haoyang, Yao, Zijun, Fang, Cong, Lin, Yankai, Leng, Jiahong, Zhao, Shuai, Cui, Peng, Zhang, Weinan, Zhou, Shao, C

2022

Overview

With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm. Researchers have achieved various outcomes in the construction of BMs and the BM application in many fields. At present, there is a lack of research work that sorts out the overall progress of BMs and guides the follow-up research. In this paper, we cover not only the BM technologies themselves but also the prerequisites for BM training and applications with BMs, dividing the BM review into four parts: Resource, Models, Key Technologies and Application. We introduce 16 specific BM-related topics in those four parts, they are Data, Knowledge, Computing System, Parallel Training System, Language Model, Vision Model, Multi-modal Model, Theory&Interpretability, Commonsense Reasoning, Reliability&Security, Governance, Evaluation, Machine Translation, Text Generation, Dialogue and Protein Research. In each topic, we summarize clearly the current studies and propose some future research directions. At the end of this paper, we conclude the further development of BMs in a more general view.

• Publisher: Cornell University Library, arXiv.org
• Subject: Machine translation / Reliability analysis / Training