Paper

INTELLECT-1 Technical Report

Obeid, Fares, Ong, Jack Min, Hagemann, Johannes, Jaghouar, Sami, Panahi, Maziyar, Ryabinin, Max, Atkins, Lucas, Goddard, Charles, Bakouch, Elie, Basra, Manveer, Straube, Jannik, Keiblinger, Michael

2024

Overview

In this report, we introduce INTELLECT-1, the first 10 billion parameter language model collaboratively trained across the globe, demonstrating that large-scale model training is no longer confined to large corporations but can be achieved through a distributed, community-driven approach. INTELLECT-1 was trained on 1 trillion tokens using up to 14 concurrent nodes distributed across 3 continents, with contributions from 30 independent compute providers dynamically joining and leaving the training process, while maintaining 83-96% compute utilization and 36.2-41.4% model FLOPS utilization. We leverage PRIME, our scalable distributed training framework designed for fault-tolerant, high-performance training on unreliable, globally distributed nodes. Key innovations in PRIME include the ElasticDeviceMesh, which manages dynamic global process groups for fault-tolerant communication across the internet and local process groups for communication within a node, live checkpoint recovery kernels, and a hybrid DiLoCo-FSDP2 implementation. Using PRIME with DiLoCo and our custom int8 all-reduce, we achieve a 400x reduction in communication bandwidth compared to traditional data-parallel training settings while delivering comparable performance. These results demonstrate the feasibility and promise of training frontier foundation models in a decentralized network of global GPU resources.

• Publisher: Cornell University Library, arXiv.org
• Subject: Communication / Fault tolerance / Nodes / Scale models