colossalai.nn.layer.colossalai_layer.normalization

class colossalai.nn.layer.colossalai_layer.normalization.LayerNorm(normalized_shape, eps=1e-05, dtype=None)

Layer Normalization for colossalai

Parameters

• normalized_shape (int) – input shape from an expected input of size. \([* \times \text{normalized_shape}[0] \times \text{normalized_shape}[1] \times \ldots \times \text{normalized_shape}[-1]]\) If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size.

• eps (float, optional) – a value added to the denominator for numerical stability, defaults to 1e-05

• dtype (torch.dtype, optional) – The dtype of parameters, defaults to None