Exactly! This is called "model parallelism" - each layer of the graph is spread ...

		thrtythreeforty on May 3, 2022 \| parent \| context \| favorite \| on: OPT: Open Pre-trained Transformer Language Models Exactly! This is called "model parallelism" - each layer of the graph is spread across multiple compute devices. Large clusters like the V100s or the forthcoming trn1 instances (disclosure, I work on this team) need _stupid_ amounts of inter-device bandwidth, particularly for training.