o ۷iN@sddlZddlmZddlmZmZddlmZddlm Z ddl m Z m Z ddl mZmZddlmZdd lmZmZmZe d Zgd ZGd d d eZdededefddZde fddZddZdededeeeeeeeffddZ d>dededede edef dd Z!ded!e"d"ed#e#fd$d%Z$ded!e"d"efd&d'Z%  ( )d?deded"ed!e"de ed#e#defd*d+Z&d,edefd-d.Z'd,edefd/d0Z(d1efd2d3Z)e*d4krmeZ+e+,Z-ee-j./e-_.zce0d5e&e-j1e-j2e-j.e-j3e-j4e-j5e-je-j(r"eee-j1d6kre0d7e0d8e'e-j.e-_6e(e-j6e-_7e-j8rHe0d9e)e-j.e9e-d:r:e)e-j6e9e-d;rKe)e-j7WdSWdSWdSe:ylZ;ze0d= than 2Gbz --quantizez/Quantize the neural network to be run with int8output)super__init__ add_argumentstrSUPPORTED_PIPELINESint)self __class__X/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/convert_graph_to_onnx.pyr!3sJ z$OnnxConverterArgumentParser.__init__)__name__ __module__ __qualname____doc__r! __classcell__r)r)r'r*r.srfilename identifierreturncCs|j|j||jS)aE Append a string-identifier at the end (before the extension, if any) to the provided filepath Args: filename: pathlib.Path The actual path object we would like to add an identifier suffix identifier: The suffix to add Returns: String with concatenated identifier at the end of the filename )parentjoinpathstem with_suffixsuffix)r0r1r)r)r*generate_identified_filename\s r8minimum_versioncCsPzddl}t|j}|tkrtd|jd|dWdSty'tdw)z Check onnxruntime is installed and if the installed version match is recent enough Raises: ImportError: If onnxruntime is not installed or too old version is found rNz*We found an older version of onnxruntime (z&) but we require onnxruntime to be >= zp to enable all the conversions options. Please update onnxruntime by running `pip install --upgrade onnxruntime`zonnxruntime doesn't seem to be currently installed. Please install the onnxruntime by running `pip install onnxruntime` and relaunch the conversion.) onnxruntimer __version__ORT_QUANTIZE_MINIMUM_VERSION ImportError)r9r: ort_versionr)r)r*check_onnxruntime_requirementsis   r?cCsxtd|jjj}gg}}|ddD]}||vr'|||||qt|dtd||t|fS)a Ensure inputs are presented in the correct order, without any Non Args: model: The model used to forward the input data tokens: BatchEncoding holding the input data input_names: The name of the inputs Returns: Tuple z$Ensuring inputs are in correct orderNz, is not present in the generated input list.zGenerated inputs order: )printforward__code__ co_varnamesappendtuple)modeltokens input_namesmodel_args_name model_argsordered_input_namesarg_namer)r)r*ensure_valid_inputs    rNnlp frameworkc sdtdtdtffdd |jd|d}|jjd|d kr&|jdi|n||}t|tr4| }t|t t fs>|f}t | }fd d | D}g}|D]}t|t t frc||qT||qTd d tt|D}fdd t||D} t|fi| } ||| |fS)a? Attempt to infer the static vs dynamic axes for each input and output tensors for a specific model Args: nlp: The pipeline object holding the model to be exported framework: The framework identifier to dispatch to the correct inference scheme (pt/tf) Returns: - List of the inferred input variable names - List of the inferred output variable names - Dictionary with input/output variables names as key and shape tensor as value - a BatchEncoding reference which was used to infer all the above information nameis_inputseq_lencst|ttfrfdd|DSddt|jDddi}r:t|jdkr/d|d<n td t|jd fd dt|jD}|t |dt d rUd nddd||S)Ncsg|] }|qSr)r)).0t)build_shape_dictrRrQrSr)r* sz:infer_shapes..build_shape_dict..cSsg|] \}}|dkr|qS)r@r))rTaxisnumelr)r)r*rWrbatchsequencer@zUnable to infer tensor axes ()csg|] \}}|kr|qSr)r))rTdimshape)rSr)r*rWrZzFound inputr z with shape: ) isinstancerFlist enumerater`len ValueErrorupdatedictfromkeysrA)rQtensorrRrSaxesseq_axes)rV)rRrQrSr*rVs "z&infer_shapes..build_shape_dictzThis is a sample output)return_tensorsrc i|] \}}|||dqS)Tr)rTkvrVrSr)r*  z infer_shapes..cSsg|]}d|qS)output_r))rTir)r)r*rWsz infer_shapes..crp)Fr)rqrtr)r*rurvNr))r#boolr% tokenizer input_idsr`rGrcr to_tuplerdrFkeysitemsextendrErangerfzipri) rOrPrHoutputs input_varsinput_dynamic_axes outputs_flatr output_namesoutput_dynamic_axes dynamic_axesr)rtr* infer_shapess& "     r pipeline_namerGrzcKs`|dur|}|dkrtstd|dkrtstdtd|d|dt|||||d S) a Convert the set of arguments provided through the CLI to an actual pipeline reference (tokenizer + model Args: pipeline_name: The kind of pipeline to use (ner, question-answering, etc.) framework: The actual model to convert the pipeline from ("pt" or "tf") model: The model name which will be loaded by the pipeline tokenizer: The tokenizer name which will be loaded by the pipeline, default to the model's value Returns: Pipeline object NrLCannot convert because PyTorch is not installed. Please install torch first.rLCannot convert because TF is not installed. Please install tensorflow first.zLoading pipeline (model: z , tokenizer: r^)rGrzrP model_kwargs)r Exceptionr rAr )rrPrGrz models_kwargsr)r)r*load_graph_from_argssropsetruse_external_formatc Cststdddl}ddlm}td|j|*t|d\}}}} t |j | |\} } ||j | | | ||d|dWddS1sJwYdS) a Export a PyTorch backed pipeline to ONNX Intermediate Representation (IR Args: nlp: The pipeline to be exported opset: The actual version of the ONNX operator set to use output: Path where will be stored the generated ONNX model use_external_format: Split the model definition from its parameters to allow model bigger than 2GB Returns: rrN)exportzUsing framework PyTorch: rT)frIrrdo_constant_folding opset_version) rrtorch torch.onnxrrAr;no_gradrrNrGas_posix) rOrrrrrrIrrrHrLrKr)r)r*convert_pytorchs&   "rc ststdtdzGddlddl}ddlm}tdjjd|t|d\}}}}|j |j fd d | D} |j j|j | ||d \} } WdStym} ztd | jd | jd| d} ~ ww)av Export a TensorFlow backed pipeline to ONNX Intermediate Representation (IR) Args: nlp: The pipeline to be exported opset: The actual version of the ONNX operator set to use output: Path where will be stored the generated ONNX model Notes: TensorFlow cannot export model bigger than 2GB due to internal constraint from TensorFlow rzD/!\ Please note TensorFlow doesn't support exporting model > 2Gb /!\rN)r;zUsing framework TensorFlow: z , tf2onnx: rcs g|] \}}jj||dqS))rQ) TensorSpec from_tensor)rTkeyrkrr)r*rWCrvz&convert_tensorflow..)r output_pathzCannot import z6 required to convert TF model to ONNX. Please install z first. )r rrA tensorflowtf2onnxr;versionVERSIONrrGpredictdatar~convert from_kerasrr=rQ) rOrrrt2ovrIrrrHinput_signature model_proto_er)rr*convert_tensorflow&s(  rFrc Kstdttd|t||||fi|}|js-td|jt|jnt t |jdkrCt d|jd|dkrPt ||||dSt |||dS) a Convert the pipeline object to the ONNX Intermediate Representation (IR) format Args: framework: The framework the pipeline is backed by ("pt" or "tf") model: The name of the model to load for the pipeline output: The path where the ONNX graph will be stored opset: The actual version of the ONNX operator set to use tokenizer: The name of the model to load for the pipeline, default to the model's name if not provided use_external_format: Split the model definition from its parameters to allow model bigger than 2GB (PyTorch only) pipeline_name: The kind of pipeline to instantiate (ner, question-answering, etc.) model_kwargs: Keyword arguments to be forwarded to the model constructor Returns: zoThe `transformers.convert_graph_to_onnx` package is deprecated and will be removed in version 5 of TransformerszONNX opset version set to: zCreating folder rzFolder z" is not empty, aborting conversionrN)warningswarn FutureWarningrArr3existsrrrfrrrr) rPrGrrrzrrrrOr)r)r*rNs ronnx_model_pathcCsTddlm}m}t|d}|}||_|||}td|dtd|S)a> Load the model at the specified path and let onnxruntime look at transformations on the graph to enable all the optimizations possible Args: onnx_model_path: filepath where the model binary description is stored Returns: Path where the optimized model binary description has been saved rInferenceSessionSessionOptionsz -optimizedz$Optimized model has been written at : ✔zY/!\ Optimized model contains hardware specific operators which might not be portable. /!\)r:rrr8roptimized_model_filepathrA)rrropt_model_path sess_optionrr)r)r*optimizes  rc Csddl}ddl}ddlm}ddlm}ddlm}ddlm }| | }t |j t dkr4td|}||t |j t d krX||d d |jd d d dddt|d } n||d d |jd d d dddt|d } | t|d} td| d|| jj| | S)z Quantize the weights of the model from float32 to in8 to allow very efficient inference on modern CPU Args: onnx_model_path: Path to location the exported ONNX model is stored Returns: The Path generated for the quantized rN) ModelProto)QuantizationMode) ONNXQuantizer)IntegerOpsRegistryz1.5.0zpModels larger than 2GB will fail to quantize due to protobuf constraint. Please upgrade to onnxruntime >= 1.5.0.z1.13.1FT) rG per_channel reduce_rangemodestatic weight_qType input_qType tensors_rangenodes_to_quantizenodes_to_excludeop_types_to_quantize) rGrrrrractivation_qTyperrrrz -quantizedz$Quantized model has been written at r)onnxr: onnx.onnx_pbronnxruntime.quantizationr'onnxruntime.quantization.onnx_quantizerr!onnxruntime.quantization.registryrloadrrr;rACopyFrom IntegerOpsrdquantize_modelr8 save_modelrG) rrr:rrrr onnx_model copy_model quantizerquantized_model_pathr)r)r*quantizesZ       rpathc Csddlm}m}ddlm}td|dz|}|||dgd}td|d WdS|yI}ztd |d WYd}~dSd}~ww) Nrr)RuntimeExceptionz"Checking ONNX model loading from: z ...CPUExecutionProvider) providerszModel u correctly loaded: ✔zError while loading the model u: ✘)r:rr+onnxruntime.capi.onnxruntime_pybind11_staterrAr)rrrr onnx_optionsrrer)r)r*verifys r__main__z' ====== Converting model to ONNX ======raV Using TensorFlow might not provide the same optimization level compared to PyTorch. For TensorFlow users you can try optimizing the model directly through onnxruntime_tools. For more information, please refer to the onnxruntime documentation: https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers z$ ====== Optimizing ONNX model ======z+ ====== Check exported ONNX model(s) ======optimized_outputquantized_outputz"Error while converting the model: r@)N)NFr)=rargparserosrrpathlibrtypingrpackaging.versionrrtransformers.pipelinesr r transformers.tokenization_utilsr transformers.utilsr r rr<r$rr#r8r?rNrFrdrirrr%ryrrrrrrr+parser parse_argsargsrabsoluterArPrGrrzrrr check_loadinghasattrrrexitr)r)r)r*s     . *A %- 2K