o }o™iQ ã@sPddlmZddlZdd„Zdd„Zdd„Zdejfd d „Zd ej fd d „Z dS)éNcCsNi}|D] \}}t|dƒr|j||<qt|dƒr$t|jdƒr$|jj||<q|S)zw Extracts dtype from the input iterator ckpt can be module.named_parameters or module.state_dict().items() ÚdtypeÚdata)Úhasattrrr)ÚckptÚdtypesÚkeyÚval©r úP/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/utils.pyÚextract_dtypess    €r cCs0t|tƒsJ‚|dvrtjS|dvrtjStjS)z< Convert a str precision to equivalent torch dtype. )Úfloat16Úfp16Ú16z16-mixed)Úbfloat16z bf16-mixed)Ú isinstanceÚstrÚtorchr rÚfloat32)rr r r Údtype_from_str"s rcCsBt|dƒs Jdƒ‚|j}t|tjƒr|St|tƒrt|ƒStdƒ‚)z/ Extracts torch dtype from a HF config Ú torch_dtypez*Expected config to have attr `torch_dtype`z*torch_dtype is not of type str/torch.dtype)rrrrrrrÚ ValueError)Úconfigrr r r Ú dtype_from_hf/s  rÚmodelcCs t|dƒS)z8 Returns true if trainer is attached to a model Útrainer)r)rr r r Úis_trainer_attached=s rrcCsDt|jddƒdkr |jSt|jdƒr t|jjddƒdkr |jjSdS)aÚ Extracts the automodel from a PyTorch Lightning trainer instance. This function checks whether the `trainer.model` is an automodel (e.g. `HFAutoModelForCausalLM`). It handles different distributed training strategies: - If no DistributedDataParallel (DDP) or Fully Sharded Data Parallel (FSDP) is used, `trainer.model` directly holds the automodel. - If DDP is used, `trainer.model.module` contains the actual automodel. - If FSDP is used, `trainer.model` still holds the automodel wrapped inside an FSDP structure. Args: trainer (lightning.pytorch.Trainer): The PyTorch Lightning trainer instance. Returns: nn.Module or None: The automodel if found, otherwise `None`. Ú is_hf_modelFTÚmoduleN)Úgetattrrrr)rr r r Úget_automodel_from_trainerDs  r) Úlightning.pytorchÚpytorchÚplrr rrÚLightningModulerÚTrainerrr r r r Ús