colossalai.context.process_group_initializer.initializer_data

class colossalai.context.process_group_initializer.initializer_data.Initializer_Data(*args, **kwargs)

A ProcessGroupInitializer for data parallelism.

Parameters

• args – Args used to initialize ProcessGroupInitializer

• kwargs – Kwargs used to initialize ProcessGroupInitializer

init_dist_group()

Initialize data parallel groups, and assign local_ranks and groups to each gpu.

Returns

Data parallelism’s information

Return type

Tuple(local_rank, group_world_size, process_group, ranks_in_group, mode)