o ٷiS@sddlZddlZddlZddlmZddlmZddlZddl Z ddl Z ddl m Z ddl mZddl mZmZddlmZddlmZdd lmZdd lmZmZmZmZdd lmZeeZ Gd d d e j!j"Z#dS)N)chain)Path)convert_float_to_float16)RepeatedCompositeFieldContainer) ModelProtoValueInfoProto) OnnxModel)PastKeyValuesHelper) WhisperConfig)convert_inputs_for_ortget_model_dynamic_axesget_sample_decoder_inputsgroup_past_key_values)InferenceSessioncseZdZdZd5dedejjdede ffdd Z d6d ej d ej d Bd e e ej d Bfd dZ d6d ej d ej d Bd e e ej d BfddZ d6d ej d ej d Bd e e ej d BfddZddZddZddZd5de de de fddZd7dede d e fd!d"Zd5d#ed e fd$d%Zdefd&d'Zdede fd(d)Z *   *  *d8d+ed,ed-e d.e de de d/e d0e fd1d2Zd+ed,ede de fd3d4ZZS)9WhisperDecoderz/A Whisper decoder with optional past key valuesFconfigmodel model_implno_beam_search_opcst||_|j|_||_||_|dkrdn|jj|_|dkr#dn|j|_|dkr-|nd|_|jj |_ |jj |_ |jj |j |_ dSNopenai)super__init__rdevicerrrdecoderproj_outmax_source_positionsdecoder_attention_heads num_headsd_model head_size)selfrrrr __class__k/home/ubuntu/.local/lib/python3.10/site-packages/onnxruntime/transformers/models/whisper/whisper_decoder.pyr%s   zWhisperDecoder.__init__Ndecoder_input_idsencoder_hidden_statespast_key_valuesc CsJ|j|||dd}||j}|j}|dur||fSt|\}}||fS)NT)r' input_idsr( use_cache)rrlast_hidden_stater(r group_by_self_and_cross) r!r&r'r(outputslogitspresent_key_values present_self present_crossr$r$r% hf_forward4s zWhisperDecoder.hf_forwardcs&i}|dur`t|\}}dd|D}dd|D}dd|D}dd|D}tjjjD]0\}}|d|||jj<|d|d||jj<|d|||jj<|d|d||jj<q/j \} } jj|||d} |durjjjD]0}t j ||jj| |jjgdd  | |jj<t j ||jj| |jjgdd  | |jj<qygg} } jjjD]*}| | |jj| | |jj|dur| | |jj| | |jjqfd d| D} fd d| D} | D]}|q|durt| | t| d}| |fS| | fS) NcSg|]}|ddqS transpose.0past_kvr$r$r% Zz.WhisperDecoder.oai_forward..cS*g|]}|g|jdddRqSNr6reshapeshaper9r$r$r%r<[*cSr3r4r7r9r$r$r%r<\r=cSr>r?rAr9r$r$r%r<]rDr6r5)xxakv_cache)dimc8g|]}|g|jdddjRddqSNr6r@r5rBrCr r8r: present_kvr!r$r%r<*crIrJrKrLrNr$r%r<rO)r enumeraterrblocksattnkeyvalue cross_attninstall_kv_cache_hookstorchcatdetachappendremover group_by_layerlen)r!r&r'r( past_kv_cacheself_attn_kv_cachescross_attn_kv_cachesidxblockrGhooksr.r0r1hookr/r$rNr% oai_forwardPs`        zWhisperDecoder.oai_forwardcCs&|jdkr ||||S||||Sr)rrer2)r!r&r'r(r$r$r%forwards zWhisperDecoder.forwardcCs>|jr ddg}|Sddgttddt|jjD}|S)Nr)r'cs4|]}d|d|d|d|fVqdS)past_key_self_past_value_self_past_key_cross_past_value_cross_Nr$r:ir$r$r% s " z-WhisperDecoder.input_names.. first_passlistr from_iterablerangerdecoder_layers)r! input_namesr$r$r%rus    zWhisperDecoder.input_namescCsZ|jrdgttddt|jjD}|Sdgttddt|jjD}|S)Nr.csrg)present_key_self_present_value_self_present_key_cross_present_value_cross_Nr$rlr$r$r%rns z.WhisperDecoder.output_names..css$|] }d|d|fVqdS)rvrwNr$rlr$r$r%rns ro)r! output_namesr$r$r%rzs&    zWhisperDecoder.output_namescCs*t|j||}d|vr|js|dd=|S)Nr)r5)r rr)r!rurz dynamic_axesr$r$r%r{s zWhisperDecoder.dynamic_axesuse_fp16_inputsuse_int32_inputs return_dictc Csnt|j|jd|jr dnd|jrdnd||d}|r!|jr|d=|S|jr,|d|dfS|d|d|dfS) Nr6rr5) batch_sizepast_sequence_lengthsequence_lengthuse_fp16 use_int32r(r&r')r rrrp)r!r|r}r~inputsr$r$r%rs*   zWhisperDecoder.inputsiois_cross is_outputcCs|jjjjd}d|jvr||j|_|jjjjd}d|jvr4||r-|j|_n|r1dnd|_|jjjjd}d|jvrI||j |_|S)Nr5_dim_r6total_sequence_lengthr) type tensor_typerCrH dim_paramClearr dim_valuerr )r!rrrrrr r$r$r%fix_key_value_cache_dimss    z'WhisperDecoder.fix_key_value_cache_dimsio_listcCsg}g}g}|D]D}d|jvrd|jvr||qd|jvr6|j|d|d}|jr0||q||q|j|d|d}|jrG||q||q|jsV|||7}|S)Npastpresentr!F)rrT)namerZrr)r!rr reordered_ior_r`rnew_ior$r$r%fix_ios$       zWhisperDecoder.fix_iocCs|j|jjdd}t|jjdkr|jjt|jjdks|jj||j|jjdd}t|jjdkrE|jjt|jjdks7|jj||S)NF)rrT)rgraphinputr]popextendoutput)r!rreordered_inputsreordered_outputsr$r$r%fix_inputs_and_outputs0s  z%WhisperDecoder.fix_inputs_and_outputscCs|jdkr |r t|}|Sr)rr)r!rr|r$r$r%fix_layernorm_weights>sz$WhisperDecoder.fix_layernorm_weightsTonnx_model_pathproviderverboseuse_external_data_formatuse_encoder_hidden_statesuse_kv_cache_inputsc  Cs*|o| |_| o ||_|js|jsJd|j||d} |} |} || | } t|jjdddt J} t j | d}t|jjddd|rO|n|}tjj|| |d| | | dd|d tj||d}||}|||}tj|||dd Wd n1swY|||||d S) alExport decoder to ONNX Args: onnx_model_path (str): path to save ONNX model provider (str): provider to use for verifying parity on ONNX model verbose (bool, optional): print verbose information. Defaults to True. use_external_data_format (bool, optional): use external data format or not. Defaults to False. use_fp16_inputs (bool, optional): use float16 inputs for the KV caches. Defaults to False. use_int32_inputs (bool, optional): use int32 inputs for the decoder_input_ids. Defaults to True. use_encoder_hidden_states (bool, optional): use encoder_hidden_states as model input for decoder-init/decoder-without-past models. Defaults to False. use_kv_cache_inputs (bool, optional): use KV caches as model inputs for decoder-with-past models. Defaults to True. zVOnly one of `use_encoder_hidden_states` and `use_kv_cache_inputs` can be true at once.)r|r}T)parentsexist_okz decoder.onnx) argsf export_paramsrurzr{ opset_versiondo_constant_foldingr)load_external_data)save_as_external_dataall_tensors_to_one_fileN)rp later_passrrurzr{rparentmkdirtempfileTemporaryDirectoryospathjoinrWonnxexport load_modelrrrsave verify_onnx)r!rrrrr|r}rrrrurzr{ tmp_dir_nametemp_onnx_model_pathout_pathrr$r$r% export_onnxJsJ $      zWhisperDecoder.export_onnxcCsJ|j||dd}g}|jr:|jd i|}||d|dD]}|D] } || q*q&n'|jd i|}||d|dD] } || qSt||gd} | dt || } z-t | D]#\} }t || | | }td|dtd t |qxWdSYdS) awVerify ONNX model outputs and PyTorch model outputs match Args: onnx_model_path (str): path to save ONNX model provider (str): execution provider for ONNX model use_fp16_inputs (bool, optional): use float16 inputs for the KV caches use_int32_inputs (bool, optional): use int32 inputs for the decoder_input_ids T)r|r}r~rr5) providersNz Comparing z...z Max diff: r$)rrprfrZrYcpunumpyrrunr rPrznpabsloggerwarningmax)r!rrr|r}r pt_outputsoutpresent_key_value_layerpresent_key_valuepresent_self_key_valuesess ort_outputsrm output_namediffr$r$r%rs0  zWhisperDecoder.verify_onnx)F)NN)FF)TFFTFT)__name__ __module__ __qualname____doc__r rWnnModulestrboolrTensorrqtupler2rerfrurzr{rrrrrrrrrr __classcell__r$r$r"r%r"s$  ]   Qr)$loggingrr itertoolsrpathlibrrrrrWfloat16r#google.protobuf.internal.containersrrr onnx_modelr past_helperr transformersr whisper_inputsr r r r onnxruntimer getLoggerrrrrrr$r$r$r%s$