Paper

RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

Berrada, Leonard, Culliton, Phil, Smith, Samuel L, Hadsell, Raia, Paszke, Adam, Tafti, Pouya, Cameron, Glenn, Love, Juliette, Anushan Fernando, Joulin, Armand, Peran, Ludovic, Haroun, Ruba, De, Soham, Rivière, Morgane, Dadashi, Robert, Doucet, Arnaud, Budden, David, Risdal, Meg, Desjardins, Guillaume, Sessa, Pier Giuseppe, Léonard Hussenot, Farabet, Clément, Chauhan, Nilay, Ghahramani, Zoubin, Girgin, Sertan, Borgeaud, Sebastian, Bachem, Olivier, Zucker, Arthur, Devanathan, Nesh, Bandy, Elisa, George-Cristian Muraru, Cassidy Hardin, Pathak, Shreya, Kavukcuoglu, Koray, Chen, Yutian, Pascanu, Razvan, Warkentin, Tris, Mooney, Paul, Andreev, Alek, Srinivasan, Srivatsan, Kenealy, Kathleen, Bhupatiraju, Surya, Huntsperger, David, Giang, Minh, Martins, Luiz Gustavo, Gundluru, Raj, Chen, Charlie, Hassabis, Demis, Brock, Andy, Teh, Yee Whye, Paterson, Antonia, Brennan, Jenny, Gale, Trevor, Botev, Aleksandar, Mihir Sanjay Kale, Senter, Evan, Sifre, Laurent, Fiedel, Noah, Nando de Frietas, Vikram, Sharad, Ferret, Johan, Mesnard, Thomas

2024

Overview

We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.

• Publisher: Cornell University Library, arXiv.org