colossalai.context.process_group_initializer.process_group_initializer

class colossalai.context.process_group_initializer.process_group_initializer.ProcessGroupInitializer(rank, world_size, config, data_parallel_size, pipeline_parallel_size, tensor_parallel_size)

An object, knowing the parallelism configuration, that initializes parallel groups.

Parameters

• rank (int) – The rank of current process.

• world_size (int) – Size of whole communication world.

• config (Config) – Running configuration.

• data_parallel_size (int) – Size of data parallel.

• pipeline_parallel_size (int) – Size of pipeline parallel.

• tensor_parallel_size (int) – Size of tensor parallel.