o }oi @s6 d dedededededededefd d Zd S) global_batch_sizemicro_batch_sizedevices num_nodestp_sizepp_sizecp_sizereturnc Cstdd|||||||fDrtd||}|||}||dkr/td|d|d||} ||| } | sJtd|d || dt| S) aCalculate valid gradient accumulation steps based on the given parameters. Args: global_batch_size (int): The desired global batch size micro_batch_size (int): The micro batch size per GPU devices (int): Number of GPUs per node num_nodes (int): Number of nodes tp_size (int, optional): Tensor parallel size. Defaults to 1. pp_size (int, optional): Pipeline parallel size. Defaults to 1. cp_size (int, optional): Context parallel size. Defaults to 1. Returns: int: The calculated gradient accumulation steps Raises: ValueError: If the parameters result in invalid configuration css|]}|dkVqdS)N).0xr r M/home/ubuntu/.local/lib/python3.10/site-packages/nemo/automodel/misc_utils.py ,sz:calculate_valid_accumulate_grad_batches..zAll parameters must be positiver z World size (z,) must be divisible by model parallel size ()z*Invalid configuration: global_batch_size (z>) must be divisible by micro_batch_size * data_parallel_size ()any ValueError is_integerint) rrrrrrr world_sizemodel_parallel_sizedata_parallel_sizeaccumulate_grad_batchesr r r'calculate_valid_accumulate_grad_batchess    rN)rrr)rrr r r rs(