Palm 2 technical details: "PaLM 2 has an improved architecture and was trained on a variety of different tasks, all of which helps PaLM 2 learn different aspects of language."
Palm 1 technical details: "PaLM uses a standard Transformer model architecture (Vaswani et al., 2017) in a decoder-only setup (i.e.,
each timestep can only attend to itself and past timesteps), with the following modifications:
• SwiGLU Activation – We use SwiGLU activations (Swish(xW ) · xV ) for the MLP intermediate
activations because they have been shown to significantly increase quality compared to standard ReLU,
GeLU, or Swish activations (Shazeer, 2020). Note that this does require three matrix multiplications in
the MLP rather than two, but Shazeer (2020) demonstrated an improvement in quality in compute-
equivalent experiments (i.e., where the standard ReLU variant had proportionally larger dimensions).
• Parallel Layers – We use a “parallel” formulation in each Transformer block (Wang & Komatsuzaki,
2021), rather than the standard “serialized” formulation. Specifically, the standard formulation can be
written as:
y = x + MLP(LayerNorm(x + Attention(LayerNorm(x)))
Whereas the parallel formulation can be written as:
y = x + MLP(LayerNorm(x)) + Attention(LayerNorm(x))
The parallel formulation results in roughly 15% faster training speed at large scales, since the MLP
and Attention input matrix multiplications can be fused. Ablation experiments showed a small quality
degradation at 8B scale but no quality degradation at 62B scale, so we extrapolated that the effect of
parallel layers should be quality neutral at the 540B scale.
• Multi-Query Attention – The standard Transformer formulation uses k attention heads, where the
input vector for each timestep is linearly projected into “query”, “key”, and “value” tensors of shape
[k, h], where h is the attention head size. Here, the key/value projections are shared for each head, i.e.
“key” and “value” are projected to [1, h], but “query” is still projected to shape [k, h]. We have found that
this has a neutral effect on model quality and training speed (Shazeer, 2019), but results in a significant
cost savings at autoregressive decoding time. This is because standard multi-headed attention has low
efficiency on accelerator hardware during auto-regressive decoding, because the key/value tensors are
not shared between examples, and only a single token is decoded at a time.
• RoPE Embeddings – We use RoPE embeddings (Su et al., 2021) rather than absolute or relative
position embeddings, since RoPE embeddings have been shown to have better performance on long
sequence lengths.
• Shared Input-Output Embeddings – We share the input and output embedding matrices, which
is done frequently (but not universally) in past work.
• No Biases – No biases were used in any of the dense kernels or layer norms. We found this to result
in increased training stability for large models.
• Vocabulary – We use a SentencePiece (Kudo & Richardson, 2018a) vocabulary with 256k tokens, which
was chosen to support the large number of languages in the training corpus without excess tokenization.
The vocabulary was generated from the training data, which we found improves training efficiency. The
vocabulary is completely lossless and reversible, which means that whitespace is completely preserved
in the vocabulary (especially important for code) and out-of-vocabulary Unicode characters are split
into UTF-8 bytes, with a vocabulary token for each byte. Numbers are always split into individual
digit tokens (e.g., “123.5 → 1 2 3 . 5”)."
Palm 1 technical details: "PaLM uses a standard Transformer model architecture (Vaswani et al., 2017) in a decoder-only setup (i.e., each timestep can only attend to itself and past timesteps), with the following modifications: • SwiGLU Activation – We use SwiGLU activations (Swish(xW ) · xV ) for the MLP intermediate activations because they have been shown to significantly increase quality compared to standard ReLU, GeLU, or Swish activations (Shazeer, 2020). Note that this does require three matrix multiplications in the MLP rather than two, but Shazeer (2020) demonstrated an improvement in quality in compute- equivalent experiments (i.e., where the standard ReLU variant had proportionally larger dimensions). • Parallel Layers – We use a “parallel” formulation in each Transformer block (Wang & Komatsuzaki, 2021), rather than the standard “serialized” formulation. Specifically, the standard formulation can be written as: y = x + MLP(LayerNorm(x + Attention(LayerNorm(x))) Whereas the parallel formulation can be written as: y = x + MLP(LayerNorm(x)) + Attention(LayerNorm(x)) The parallel formulation results in roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused. Ablation experiments showed a small quality degradation at 8B scale but no quality degradation at 62B scale, so we extrapolated that the effect of parallel layers should be quality neutral at the 540B scale. • Multi-Query Attention – The standard Transformer formulation uses k attention heads, where the input vector for each timestep is linearly projected into “query”, “key”, and “value” tensors of shape [k, h], where h is the attention head size. Here, the key/value projections are shared for each head, i.e. “key” and “value” are projected to [1, h], but “query” is still projected to shape [k, h]. We have found that this has a neutral effect on model quality and training speed (Shazeer, 2019), but results in a significant cost savings at autoregressive decoding time. This is because standard multi-headed attention has low efficiency on accelerator hardware during auto-regressive decoding, because the key/value tensors are not shared between examples, and only a single token is decoded at a time. • RoPE Embeddings – We use RoPE embeddings (Su et al., 2021) rather than absolute or relative position embeddings, since RoPE embeddings have been shown to have better performance on long sequence lengths. • Shared Input-Output Embeddings – We share the input and output embedding matrices, which is done frequently (but not universally) in past work. • No Biases – No biases were used in any of the dense kernels or layer norms. We found this to result in increased training stability for large models. • Vocabulary – We use a SentencePiece (Kudo & Richardson, 2018a) vocabulary with 256k tokens, which was chosen to support the large number of languages in the training corpus without excess tokenization. The vocabulary was generated from the training data, which we found improves training efficiency. The vocabulary is completely lossless and reversible, which means that whitespace is completely preserved in the vocabulary (especially important for code) and out-of-vocabulary Unicode characters are split into UTF-8 bytes, with a vocabulary token for each byte. Numbers are always split into individual digit tokens (e.g., “123.5 → 1 2 3 . 5”)."