o }oix @sddlZddlmZddlmZddlmZmZmZddl Z de de fddZ dd ee efd e d ee ddfd d Z de de fddZddeee fde de jfddZde jjde fddZdee e jffddZdddZdS) N)Counter)Path)DictOptionalUnioncheckpoint_pathreturncCst|}|dS)z Checks if the checkpoint is in NeMo 2.0 format. Args: checkpoint_path (str): Path to a checkpoint. Returns: bool: True if the path points to a NeMo 2.0 checkpoint; otherwise false. context)ris_dir)r ckpt_pathr K/home/ubuntu/.local/lib/python3.10/site-packages/nemo/export/utils/utils.pyis_nemo2_checkpoints r model_dirdelete_existing_filessubdircCs^t|}|r|rt|nt|rtd|d|dur&||}|jddddS)a Prepares model_dir path for the TensorRTT-LLM / vLLM export. Makes sure that the model_dir directory exists and is empty. Args: model_dir (str): Path to the target directory for the export. delete_existing_files (bool): Attempt to delete existing files if they exist. subdir (Optional[str]): Subdirectory to create inside the model_dir. Returns: None zThere are files in z0 folder: try setting delete_existing_files=True.NT)parentsexist_ok)rexistsshutilrmtreeanyiterdir RuntimeErrormkdir)rrr model_pathr r r prepare_directory_for_export$s  rpathcCst|}|o |jdkS)z Checks if the path exists and points to packed NeMo 1 checkpoint. Args: path (str): Path to possible checkpoint. Returns: bool: NeMo 1 checkpoint exists and is in '.nemo' format. z.nemo)rrsuffix)rrr r r is_nemo_tarfile@s rT precisionmegatron_amp_O2cCsD|stjS|dvr tjS|dvrtjS|dvrtjStd|d)ar Mapping from PyTorch Lighthing (PTL) precision types to corresponding PyTorch parameter data type. Args: precision (Union[int, str]): The PTL precision type used. megatron_amp_O2 (bool): A flag indicating if Megatron AMP O2 is enabled. Returns: torch.dtype: The corresponding PyTorch data type based on the provided precision. )bf16z bf16-mixed)16z16-mixed) 32z32-truez"Could not parse the precision of 'z' to a valid torch.dtype)torchfloat32bfloat16float16 ValueError)r r!r r r torch_dtype_from_precisionNs r,modulecCsZdd|D}dd|D}||}t|dkr%td|d|r+|SdS)zEFind the device type the model is assigned to and ensure consistency.cSh|]}|jjqSr devicetype).0paramr r r iz(get_model_device_type..cSr.r r/)r2bufferr r r r4jr5z;Model parameters and buffers are on multiple device types: z@. Ensure all parameters and buffers are on the same device type.cpu) parametersbuffersunionlenr+pop)r-param_device_typesbuffer_device_typesall_device_typesr r r get_model_device_typefs   rAcCst|ddgddgdd}|S)zvGets example data to feed to the model during ONNX export. Returns: Dictionary of tokenizer outputs. zexample query onezexample query twozexample passage onezexample passage twopt)return_tensors)dict) tokenizerexample_inputsr r r get_example_inputswsrGcCstddl}g}|D]}|j|jj|jjhvr||q|s#d}t|tdd|D}|jj |vr8d}t|dS)zChecks the network to ensure it's compatible with fp8 precison. Raises: ValueError if netowrk doesn't container Q/DQ FP8 layers rNz#No Quantize/Dequantize layers foundcss|]}|jVqdSN)r )r2layerr r r sz'validate_fp8_network..z>Found Quantize/Dequantize layers. But none with FP8 precision.) tensorrtr1 LayerTypeQUANTIZE DEQUANTIZEappendr+rDataTypeFP8)networktrtquantize_dequantize_layersrI error_msg quantize_dequantize_layer_dtypesr r r validate_fp8_networks  rWrH)T)rN)r collectionsrpathlibrtypingrrrr'strboolrrrintdtyper,nnModulerATensorrGrWr r r r s*    "