o wi@sddlmZddlmZmZddlZddlmZddl m Z ddl m Z m Z mZddlmZddlmZdd lmZejjd d d         ddede dedededBdedBdedBdedBde ee eje fdedBdededefdd ZdS))Path)CallableOptionalN)Console) Annotated) ExportConfigQuantizationConfig Quantizer)HFAutoModelForImageTextToText)2setup_trainer_and_restore_model_with_modelopt_spec)is_global_rank_zeroptqvlm)name namespaceF model_path export_configcalibration_tpcalibration_pp"num_layers_in_first_pipeline_stage!num_layers_in_last_pipeline_stagedevices num_nodesquantization_config forward_loop legacy_ckpttrust_remote_codereturnc Cs |st}|dur |}|dur|}t||} t|s$Jd|dt|d} d}| rE|jdks9Jdt|| dd}|n |jd ksNJd t|||||||d | d d d idd id \}}| || }| |||t rt }| d|j d|j S)aO Applies Post-Training Quantization (PTQ) for a vision-language model using the specified quantization and export configs. It runs calibration for a small dataset to collect scaling factors low-precision GEMMs used by desired quantization method. By default, this function produces TensorRT-LLM checkpoint ready for deployment using nemo.export and nemo.deploy modules or directly using TensorRT-LLM library. Args: model_path (str): The path to model to be quantized. export_config (ExportConfig): Export configuration for output checkpoint. calibration_tp (int): Calibration tensor parallelism. calibration_pp (int): Calibration pipeline parallelism. num_layers_in_first_pipeline_stage (int): Number of layers in the first pipeline stage. num_layers_in_last_pipeline_stage (int): Number of layers in the last pipeline stage. devices (int): Number of devices to use for calibration. Default: calibration_tp. num_nodes (int): Number of nodes to use for calibration. Default: calibration_pp. quantization_config (QuantizationConfig): Configuration for quantization algorithm. forward_loop (Callable): Forward loop to use for calibration. If not provided, a forward loop will be created using the calibration dataset. legacy_ckpt (bool): If True, allow loading ckpt saved with older version of TE. trust_remote_code (bool): Trust remote code when loading HuggingFace models. Returns: Path: The path where the quantized checkpoint has been saved after calibration. NzPath z does not existz config.jsonnemoz1Automodel PTQ does not support export format nemoauto) model_namer device_maphfz/Automodel PTQ does not support export format hfTF)sequence_parallel lazy_initr$) rtensor_model_parallel_sizepipeline_model_parallel_sizerrrrinference_onlyrstrategy_kwargstrainer_kwargsmodel_config_overridesu;[green]✓ PTQ succeeded, quantized checkpoint exported to z[/green])rr rexists export_formatr configure_modelr quantizeexportr rprintpath)rrrrrrrrrrrr quantizer is_automodeltrainermodelconsoler8U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/vlm/api.pyr sJ)   ) rrNNNNNNFF)pathlibrtypingrrnemo_runrun rich.consolertyping_extensionsrnemo.collections.llm.modeloptrrr nemo.collections.vlmr nemo.collections.vlm.modeloptr nemo.utils.get_rankr cli entrypointstrintConfigboolr r8r8r8r9s^