o i3T @sdZddlZddlmZddlZddlmZddlZddl m Z ddl m Z m Z ddlZddlmZmZddlmZmZerCddlZerRdd lmZdd lmZeeZ dd d Zdee dej!de"e ej!fde deee ej!ff ddZ#ddZ$ddZ%ddZ&ddZ'dS)z!PyTorch - Flax general utilities.N)UnpicklingError) from_bytes) flatten_dictunflatten_dict)is_safetensors_availableis_torch_available)check_torch_load_is_safelogging) safe_open) load_fileFc Cs|swtj|}td||drKsz=load_pytorch_checkpoint_in_flax_state_dict..,z parameters.)ospathabspathloggerinfoendswithr keys get_tensortorch ImportErrorModuleNotFoundErrorerrorr loadsumvalues"convert_pytorch_state_dict_to_flax*convert_pytorch_sharded_state_dict_to_flax) flax_modelpytorch_checkpoint_path is_shardedallow_missing_keyspt_path pt_state_dictfkr#flax_state_dictrrr*load_pytorch_checkpoint_in_flax_state_dict0s4    &  r5 pt_tuple_key pt_tensorrandom_flax_state_dict model_prefixreturncsdttdtffdd }|ddd}|ddvr$||r$||fS|ddd }|dd kr:||s:||fS|ddd }|dd krP||sP||fS|ddd }|ddkrf||rf||fS|ddd}|ddkr|jdkr||s|dddd}||fS|ddd}|ddkr||s|j}||fS|ddd}|ddkr||fS|ddd}|ddkr||fSd}|ddddkr|dd}n|ddddkr|dd}|dur|dd|f}||fS||fS)zYRename PT weight names to corresponding Flax weight names and reshape tensor if necessarykeyr:cstt|f|h@dkS)zAChecks if `key` of `(prefix,) + key` is in random_flax_state_dictr)lenset)r;r9r8rris_key_or_prefix_key_in_dict\szCrename_key_and_reshape_tensor..is_key_or_prefix_key_in_dictN)scale)weightgamma)mean running_mean)var running_var) embeddingrB)kernelrrrBrC)biasbeta)parametrizations original0_g)rQ original1_v)tuplestrboolndim transposeT)r6r7r8r9r?renamed_pt_tuple_keynamerr>rrename_key_and_reshape_tensorTsH   r_c Cs^tottt|tj}|rtjnd}dd|D}|r;|D]\}}|j |kr2| }| ||<q%|j }d|jvrI|jd}n|j}t|} d|jvrat|jd} | | i} ||voo|dd|Dv} ||vo||dd|Dv} |D]\}}t|d }|||k}|d |k}| r|r|d d}t||| |\}}|f|| v}| r|r|f|}|| vr|j| |jkrtd |d | |jd|jd d|jvrd|dvsd|dvrt|| d|<qd|dvr| |dq|s t|ntj|tjd| d|<q|s t|ntj|tjd| |<qt| S)Nbfloat16cSi|]\}}||jqSrdtyperr3vrrr z6convert_pytorch_state_dict_to_flax..params batch_statscSh|] }|ddqS.rsplitrr3rrr z5convert_pytorch_state_dict_to_flax..cSrjrkrmrorrrrprqrlrr1PyTorch checkpoint seems to be incorrect. Weight  was expected to be of shape , but is rDr@rFrinum_batches_trackedrbrh)r isinstancenextiterr)r#Tensorr`itemsrcfloatrnumpybase_model_prefixrhrupdaterWrnr_shape ValueErrorjnpasarraypopr)r1r,from_binr` weight_dtypesr3rer9flax_model_paramsr8flax_batch_statsr4$load_model_with_head_into_base_model$load_base_model_into_model_with_headpt_keyr7r6 is_bfloat_16has_base_model_prefixflax_key flax_tensorrequire_base_model_prefixrrrr*sp            r*c sDddli}|D]}tj|dd}dd|D}fdd|D}|j}d|jvrE|jd}t|}|t|jdn|j}t|}||voX|d d |Dv} ||voe|d d |Dv} |D]\} } t| d } || j k}| d|k}| r|r| d d} t | | ||\}}|f||v}| r|r|f|}||vr|j ||j krt d| d||j d|j d d|jvr d|dvrt||d|<qjd|dvrt||d|<qjd|dvr||dqj|st|ntj|tj d|d|<qj|st|ntj|tj d||<qjqt|S)NrT)rcSrarrbrdrrrrfrgz>convert_pytorch_sharded_state_dict_to_flax..cs2i|]\}}||jjkr|n|qSr)rcr`r~r}rdr#rrrfs&rirhcSrjrkrmrorrrrprqz=convert_pytorch_sharded_state_dict_to_flax..cSrjrkrmrorrrrprqrlrrrrsrtrDr@rurFrvrbrw)r#r r'r|rrhrrrWrnr`r_rrrrrr)shard_filenamesr,r4 shard_filer1rr9rr8rrrr7r6rrrrrrrrr+sv             0r+c Cstj|}td|ttd|jj}| dr't |}t |dd}n.t |d!}z t ||}WntyEtd|dwWd n1sPwYt||S) (Load flax checkpoints in a PyTorch modelzLoading Flax weights from Flaxr rl)seprbzUnable to convert z to Flax deserializable object. N)rrrrrgetattr transformers __class____name__r safe_load_fileropenrreadrOSError"load_flax_weights_in_pytorch_model)modelflax_checkpoint_pathflax_clsr4state_frrr%load_flax_checkpoint_in_pytorch_modelLs     rc Cszddl}Wnttfytdwttjdd| }t |r5t dtjdd|}t|}| }|j |voK|j dd |Dv}|j |voZ|j d d |Dv}g}t|} |D]9\} } | d|j k} d |j f| |v} |r| r| d d} n |r| r|j f| } | d dkr| jdkrd | |vr| dd d} t| d} nE| d dkrd | |vr| dd d} | j} n,| d dvr| dd d} nd| d vr| dd d} nd| d vr| dd d} d|vrd | d d}nd | }i}|D]F}|d }d}|dddddgkr-|dd}n|dddddgkr?|dd}|durV|dd|g}d |}|||<q||vra||}||vr| j||jkrtd | d!||jd"| jd t| tjst| n| } || ||<| |qg||qg| |t!| } t"|dkrt d#|j#j$d$|d%|j#j$d&|j#j$d' n t d(|j#j$d)t"| dkrt d*|j#j$d+| d,|St d-|j#j$d.|j#j$d/|S)0rrNzLoading a Flax weights in PyTorch, requires both PyTorch and Flax to be installed. Please see https://pytorch.org/ and https://flax.readthedocs.io/en/latest/index.html#installation for installation instructions.cSs |jtjkSr)rcrr`)xrrrps z4load_flax_weights_in_pytorch_model..zFound ``bfloat16`` weights in Flax model. Casting all ``bfloat16`` weights to ``float32`` before loading those in PyTorch model.cSs|jtjkr |tjS|Sr)rcrr`astypenpfloat32rwrrrryscSrjrkrmrorrrrprqz5load_flax_weights_in_pytorch_model..cSrjrkrmrorrrrprqrlrr@rIrJrM)rLrKrr)rArHrD)rErF)rGrirPrKrQrRrSrTrUrVz.Flax checkpoint seems to be incorrect. Weight rsrtzQSome weights of the Flax model were not used when initializing the PyTorch model z: z, - This IS expected if you are initializing z from a Flax model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a FlaxBertForPreTraining model). - This IS NOT expected if you are initializing z from a Flax model that you expect to be exactly identical (e.g. initializing a BertForSequenceClassification model from a FlaxBertForSequenceClassification model).z3All Flax model weights were used when initializing z. zSome weights of zE were not initialized from the Flax model and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z were initialized from the Flax model. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.)%r#r$r%rr&rjax tree_utiltree_mapr)anywarning state_dictrr=r!r|joinrZrr[r\rnrrrxrndarrayr from_numpyremoveappendload_state_dictlistr<rr)pt_model flax_stater# is_type_bf16r4 pt_model_dictrrunexpected_keys missing_keysflax_key_tuplerrrrspecial_pt_namesr;key_componentsr^ key_to_checkrrrrbs      $                  r)F)(__doc__rpicklerr jax.numpyr~rrflax.serializationrflax.traverse_utilrrrrrutilsr r r# safetensorsr safetensors.flaxr r get_loggerrrr5rWrXrdictr_r*r+rrrrrrsD       $  C\Y