o }oî @sjUddlZddlmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)GdddeZde j*fddZ+eGdddee"j,Z-eGddde-Z.eGddde.Z/eGddde-Z0eGdd d e0Z1eGd!d"d"e-Z2eGd#d$d$e2Z3eGd%d&d&e-Z4eGd'd(d(e4Z5eGd)d*d*e2Z6eGd+d,d,e4Z7e"8ed-Gd.d/d/e"j9defZ:e"8ed0Gd1d2d2e:Z;e0e1e2e6e3e4e7e5e.e/d3 Ze e-fe?d4<gd5Z@dS)6N) dataclass)Path)CallableLiteralOptionalType)parallel_state)GPTInferenceWrapper)InferenceWrapperConfig) AttnBackend)TransformerConfig)GPTModel gpt_data_step)hyena_stack_spechyena_stack_spec_no_te) HyenaModel)hyena_no_weight_decay_cond)get_vocab_sizeioteardown)NEMO_MODELS_CACHE) TransformFns)loggingc@sveZdZdZ ddejfddZ      ddejdejdeejd eejd eejd eejdejfd d ZdS)rz This is a wrapper around the MCoreHyenaModel to allow for inference. Our model follows the same API as the GPTModel, but the megatron model class is different so we need to handle the inference wrapper slightly differently. NreturncCs|j}|rt|tur nt|dd}|s|dust|tur"tdd}|jdur.|jj}nt|jdr9|jj}ntdt |jj ||||d}t ||}|S)a Gets the inference wrapper for the Hyena model. Args: params_dtype: The data type for model parameters inference_batch_times_seqlen_threshold: Threshold for batch size * sequence length during inference inference_max_seq_length: Maximum sequence length for inference Returns: GPTInferenceWrapper: The inference wrapper for the model Raises: ValueError: If MCoreHyenaModel instance not found or vocab size cannot be determined moduleNz@Exact MCoreHyenaModel instance not found in the model structure. vocab_sizezlUnable to find vocab size. Either pass in a tokenizer with vocab size, or set vocab size in the model config) hidden_size params_dtype&inference_batch_times_seqlen_thresholdpadded_vocab_sizeinference_max_seq_length) rtypeMCoreHyenaModelgetattr ValueError tokenizerrhasattrconfigr rr )selfrrr mcore_modelrinference_wrapper_configmodel_inference_wrapperr,X/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/hyena.pyget_inference_wrapper0s2       z HyenaModel.get_inference_wrapper input_ids position_idsattention_masklabels decoder_input loss_maskc Cs8|durd|ini} |j|||f||||d| } | S)a7 Forward pass of the Hyena model. Args: input_ids: Input token IDs position_ids: Position IDs for input tokens attention_mask: Optional attention mask labels: Optional labels for loss computation decoder_input: Optional decoder input loss_mask: Optional loss mask inference_context: Optional inference parameters packed_seq_params: Optional parameters for packed sequences Returns: torch.Tensor: Output tensor from the model Npacked_seq_params)r3r2inference_contextr4)r) r(r/r0r1r2r3r4r6r5 extra_kwargs output_tensorr,r,r-forward`s zHyenaModel.forwardN)NNNNNN) __name__ __module__ __qualname____doc__torchTensorr.rr9r,r,r,r-r)s6 4 rrcCs4|d|d|d|dd}d|d<|di|S) a Performs a forward step for the Hyena model. Args: model: The Hyena model batch: Dictionary containing input batch data with keys: - tokens: Input token IDs - position_ids: Position IDs - labels: Labels for loss computation - loss_mask: Mask for loss computation Returns: torch.Tensor: Output from the model forward pass tokensr0r2r4)r/r0r2r4Nr1r,r,)modelbatch forward_argsr,r,r-hyena_forward_stepsrEcseZdZUdZdZeed<dZeed<ej Z ej ed<dZ eed<dZ eed<d Zeed <d Zeed <d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeded<dZeed<d Z eed!<dZ!e"eed"<dZ#eed#<d$Z$eed%<dZ%eed&<dZ&eed'<d(Z'eed)<dZ(eed*<dZ)eed+<dZ*eed,<d-Z+eed.<e,j-Z.e,ed/<dZ/eed0<d1Z0eed2<d3Z1eed4<d5Z2eed6<e3Z4e5ed7<e6Z7e5ed8<dZ8eed9<dZ9eed:<dZ:eed;<dZ;eed<<dZ<dZ>eed?<dZ?eed@<dZ@eedA<dZAeedB<dZBeedC<dZCeedD<dEZDeedF<dZEeedG<dZFeedH<dZGeedI<fdJdKZHdQdLe"edMdNfdOdPZIZJS)R HyenaConfigz Configuration dataclass for Hyena. For adjusting ROPE when doing context extension, set seq_len_interpolation_factor relative to 8192. For example, if your context length is 512k, then set the factor to 512k / 8k = 64. Ffp16_lm_cross_entropyTparallel_outputrfp16bf16 num_layersirnum_attention_headsNnum_groups_hyenanum_groups_hyena_mediumnum_groups_hyena_shorthybrid_attention_ratiohybrid_mlp_ratiohybrid_override_pattern post_process pre_processi seq_lengthrope)learned_absoluterYnoneposition_embedding_typeg?rotary_percenti' rotary_baseseq_len_interpolation_factorapply_rope_fusionmake_vocab_size_divisible_bygated_linear_unitfp32_residual_connectionRMSNorm normalizationadd_bias_linearhidden_dropoutattention_dropoutư>layernorm_epsilonattention_backendget_attention_mask_from_fusionfullrecompute_granularityuniformrecompute_methodrecompute_num_layersforward_step_fn data_step_fntokenizer_model_pathhyena_init_methodhyena_output_layer_init_methodhyena_filter_no_wd"remove_activation_post_first_layeradd_attn_proj_biascross_entropy_loss_fusiontp_comm_overlapbias_activation_fusionbias_dropout_add_fusionadd_bias_outputuse_tenormalized_weightedto_upperuse_short_conv_biasvortex_style_fp8use_b2b_causal_conv1dcs$t|jr t|_dSd|_dS)zP Post-initialization hook that sets up weight decay conditions. N)super __post_init__ryrhyena_no_weight_decay_cond_fnr( __class__r,r-rs zHyenaConfig.__post_init__vp_stagerr"cCs|jrdn|j|_t|dddur|dusJdt|fid|jr#tnPtdt||j|j d|j d|j d |j d |j d |jd |jd |jd|jd|jdtdtddd|jd|jd|jd|j}|Sdt||j|j d|j d|j d |j d |j d |jd |jd |jd|jd|jdtdtddd|jd|jd|jd|j}|S)a Configures and returns a Hyena model instance based on the config settings. Args: tokenizer: Tokenizer to use for the model vp_stage: Virtual pipeline stage Returns: MCoreHyenaModel: Configured Hyena model instance F$virtual_pipeline_model_parallel_sizeNzGVirtual pipeline model parallelism is temporarily unsupported in Hyena.rrmax_sequence_lengthrOrPrQrUr\r]r^r_rWrV#share_embeddings_and_output_weightsTrwrxrzr{)rzr~r#r"rrrrrrbrXrOrPrQrUr\r]r^r_ris_pipeline_first_stageis_pipeline_last_stagerwrxr{)r(r%rrBr,r,r-configure_models           zHyenaConfig.configure_modelr:)Kr;r<r=r>rGbool__annotations__rHr?bfloat16rdtyperIrJrLintrrNrOrPrQrSfloatrTrUstrrVrWrXr\rr]r^r_rr`rbrcrdrfrgrhrirkr flashrlrmrorqrsrErtrrrurvrwrxryrzr{r|r}r~rrrrrrrrr __classcell__r,r,rr-rFsr                                                    rFc@sreZdZUdZdZeed<dZeed<dZ eed<dZ eed <dZ eed <d Z eed <d Z eed <dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<ejZejed<d Zeed!<dZeed"<dZeed#<d$Z eed%<d&Z!eed'<d(Z"eed)<d*Z#eed+<d,Z$eed-<d.Z%eed/<dZ&eed0<dZ'eed1<dZ(eed2<d3S)4HyenaTestConfigz'Configuration for testing Hyena models.zSDH*rUrrrL rXrrOrPrQrMrb byte-leveltokenizer_librarybase mapping_type+ffn_hidden_sizeTrc rNFuse_cpu_initializationrRrhrirrerf add_qkv_biasrgrjrkrnrorprqrKrs small_initrw wang_initrxryrrN))r;r<r=r>rUrrrLrrXrrOrPrQrbrrrrcrrNrrhrrir?rrrrfrrgrkrorqrsrwrxryrrr,r,r,r-rs>                            rc@6eZdZUdZdZeed<dZeed<dZeed<dS)HyenaNVTestConfig  Several unintentional design choices were made to the original Arc implementation that are required to use the original Arc checkpoints, but may result in less stable model training. If you are training from scratch, these are the recommended configs. Frzr{TrN r;r<r=r>rzrrr{rr,r,r,r-r8   rc@sfeZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <d Z eed<d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed <ejZejed!<d"Zeed#<dZeed$<dZ eed%<d&Z!eed'<d(Z"eed)<d*Z#eed+<dZ eed<d,Z$eed-<d.Z%eed/<dZ&eed0<d1S)2 Hyena1bConfigz,Config matching the 1b 8k context Evo2 modelzSDH*SDHSDH*SDHSDH*SDHSDH*rUrLrsrrXirrOrarPrQrMrbrrrrirTrcrNFrrRrhrirrerfrrgrjrkrnrorprqrrwrrxryN)'r;r<r=r>rUrrrLrrsrXrrOrPrQrbrrrrcrrNrrhrrir?rrrrfrrgrkrorqrwrxryr,r,r,r-rEs<                           rc@r)HyenaNV1bConfigrFrzr{TrNrr,r,r,r-rgrrc@sZeZdZUdZdZeed<dZeed<dZ eed<dZ eed <dZ eed <d Z eed <d Z eed <dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<ejZejed<dZeed <dZeed!<dZeed"<d#Z eed$<d%Z!eed&<d'Z"eed(<d)Z#eed*<d+Z$eed,<d-Z%eed.<dZ&eed/<d0S)1 Hyena7bConfigz,Config matching the 7b 8k context Evo2 modelz SDH*SDHSDH*SDHSDH*SDHSDH*SDHSDH*rUrrLrrXrrrOrrPrQrMrbrrrrrrTrcrNFrrRrhrirrerfrrgrjrkrnrorprqrrrsrrwrrxryN'r;r<r=r>rUrrrLrrXrrOrPrQrbrrrrcrrNrrhrrir?rrrrfrrgrkrorqrsrwrxryr,r,r,r-rt:                          rc@r)HyenaNV7bConfigrFrzr{TrNrr,r,r,r-rrrc@sZeZdZUdZdZeed<dZeed<dZ eed<dZ eed<dZ eed <d Z eed <d Z eed <d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<ejZejed<dZeed <dZeed!<dZeed"<d#Z eed$<d%Z!eed&<d'Z"eed(<d)Z#eed*<d+Z$eed,<d-Z%eed.<dZ&eed/<d0S)1Hyena40bConfigz-Config matching the 40b 8k context Evo2 modelz2SDH*SDHSDH*SDHSDH*SDHSDH*SDHSDH*SDH*SDHSDH*SDHSDH*rU2rLrrXrrOirPrQrMrbrrrriUrTrc@rNFrrRrhrirrerfrrgrjrkrnrorprqrKrsrrwrrxryNrr,r,r,r-rrrc@r)HyenaNV40bConfigrFrzr{TrNrr,r,r,r-rrrc@eZdZUdZdZeed<dS)Hyena7bARCLongContextConfigpThe checkpoint from ARC requires padding to the FFN dim due to constraintes from large TP size for training.i,rNr;r<r=r>rrrr,r,r,r-r rc@r)Hyena40bARCLongContextConfigriXrNrr,r,r,r-rrrpytorchcs|eZdZdZddeffdd ZdefddZd d Zdd e d ede fddZ ddZ e ddZ e defddZZS)PyTorchHyenaImporterzL Importer class for converting PyTorch Hyena models to NeMo format. Npathcst||}||_|S)z Creates a new importer instance. Args: path: Path to the PyTorch model model_config: Optional model configuration Returns: PyTorchHyenaImporter instance )r__new__ model_config)clsrrinstancerr,r-rs zPyTorchHyenaImporter.__new__rcCst|j|jdS)zt Initializes a new HyenaModel instance. Returns: HyenaModel: Initialized model )r%)rr'r%rr,r,r-initszPyTorchHyenaImporter.initcCstjt|ddS)+ Returns the source model. cpu) map_location)r?loadrrr,r,r-get_source_modelsz%PyTorchHyenaImporter.get_source_model torch_dist output_pathcheckpoint_formatcCs|}d|vr |d}Gddd}|}|j|d|d}||jjdd|D}|||jj|}||jj|||| ||t d|t ||~~|S) a Applies the model conversion from PyTorch to NeMo format. Args: output_path: Path to save the converted model checkpoint_format: Format for saving checkpoints Returns: Path: Path to the saved NeMo model rBc@s8eZdZdZddZddZddZdd Zd d Zd S) z.PyTorchHyenaImporter.apply..ModelStatez_Wrapper around the source model state dictionary that also handles some weight transformations.cSs ||_||}||_||_dS)aWrapper around the source model state dictionary that also handles some weight transformations. Args: state_dict: original state dictionary from the source model num_layers: number of layers in the source model N)rLtransform_source_dict _state_dict fp32_suffixes)r( state_dictrLrr,r,r-__init__s  z7PyTorchHyenaImporter.apply..ModelState.__init__cS|jS)zReturn the state dictionary.)rrr,r,r-r#sz9PyTorchHyenaImporter.apply..ModelState.state_dictc Ss~|jD]7\}}d|vr<|j|kr"td|d|jd|d|dd}||jvr2tj}n|}| ||j|<qdS) z1Convert the state dictionary to the target dtype._extraz Converting z from z (source model) to z (target model).N) ritemsrrwarningsplitrr?float32to)r(rkvk_suffix_dtyper,r,r-r's   z1PyTorchHyenaImporter.apply..ModelState.tocSsNddlm}|D]\}}d|vsd|vr$|ddd|jf||<q |S)zAdjust the medium filter.r)rFzfilter.hz filter.decayN):nemo.collections.llm.gpt.model.megatron.hyena.hyena_configrFrhyena_medium_conv_len)r( updated_datarFrrr,r,r-adjust_medium_filter4s zCPyTorchHyenaImporter.apply..ModelState.adjust_medium_filterc Ssddl}ddt|jD}|j||jd<i}t|dD]J}d|vr.|d|q |d|}|rbt|d }||vrY| d |d t |||}|d|||<q |d|||<q |d|||<q | |}|S) zTransform the source state dictionary, applying some challenging layer name re-mappings and removing extra keys, as well as truncating a filter that didn't need to extend to the full sequence length dim. rNcSsi|]}|d|qS)rKr,).0ir,r,r- DszXPyTorchHyenaImporter.apply..ModelState.transform_source_dict..rrzsequential\.(\d+)z\b) rerangerLlistkeyspopsearchrgroupsubrr) r(sourcer layer_maprkeymatchoriginal_layer_numnew_keyr,r,r-r=s"  zDPyTorchHyenaImporter.apply..ModelState.transform_source_dictN) r;r<r=r>rrrrrr,r,r,r- ModelStates  rF)ckpt_async_savesave_ckpt_formatcSs*h|]\}}|jtjkr|ddqS)rr)rr?rr)rnpr,r,r- ^s*z-PyTorchHyenaImporter.apply..z.Converted Hyena model to Nemo, model saved to ) rr nemo_setuprr'rnamed_parametersrL convert_state nemo_saverinfor)r(rrrrtargettrainerrr,r,r-applys  G   zPyTorchHyenaImporter.applyc Csi}d|d<d|dt|jjd<|jj}t|jjD]7\}}|r0d|d|d|d<n d|d|d|d<d|d |d|d <|d kr|r\d|d |d|d <n d|d|d|d <d|d|d|d<d|d|d|d<d|d|d|d<d|d|d|d<|dkrd|d|d|d<q|dkrd|d|d|d<d|d|d|d<d|d|d|d<q|dkrd|d|d|d<d|d|d|d<d|d|d|d<d|d|d|d<q|d krN|rd|d|d|d <n d|d |d|d <d|d|d|d<d|d|d|d<d|d|d|d<qtd |tj|||tjd!d"t j d#gd$S)%z Converts the state dictionary from source format to target format. Args: source: Source model state target: Target model Returns: Result of applying state transforms z embedding.word_embeddings.weightz#sequential.0.word_embeddings.weightzdecoder.final_norm.weightz sequential.z .norm.weightzdecoder.layers.z!.mlp.linear_fc1.layer_norm_weightz.pre_mlp_layernorm.weightz.mlp.linear_fc2.weightz.mlp.w3.weight*z).mixer.dense_projection.layer_norm_weightz.input_layernorm.weightz.mixer.dense_projection.weightz(.mixer.hyena_proj_conv.short_conv_weightz.mixer.dense.weightz.mixer.dense.biasSz).mixer.mixer.short_conv.short_conv_weightDz.mixer.mixer.conv_biasz.mixer.mixer.filter.hz.mixer.mixer.filter.decayHz.mixer.mixer.filter.gammaz.mixer.mixer.filter.Rz.mixer.mixer.filter.pz,.self_attention.linear_qkv.layer_norm_weightz!.self_attention.linear_qkv.weightz".self_attention.linear_proj.weightz .self_attention.linear_proj.biaszUnknown symbol: )zsequential.*.mlp.w1.weightzsequential.*.mlp.w2.weightz&decoder.layers.*.mlp.linear_fc1.weight) source_key target_keyfn)mapping transforms) lenr'rUr enumerater$rapply_transformsstate_transformr merge_fc1)r(rrr te_enabledrsymbolr,r,r-rksr             z"PyTorchHyenaImporter.convert_statecCsddlm}||jjd}|S)zd Gets the tokenizer for the model. Returns: Tokenizer instance r)get_nmt_tokenizer)library)3nemo.collections.nlp.modules.common.tokenizer_utilsrrr)r(rr%r,r,r-r%s zPyTorchHyenaImporter.tokenizercCr)zn Gets the model configuration. Returns: HyenaConfig: Model configuration )rrr,r,r-r'szPyTorchHyenaImporter.configr:)r)r;r<r=r>rrrrrrr rpropertyr%rFr'rr,r,rr-rs g[ rhfc@seZdZdZddZdS)HuggingFaceSavannaHyenaImportera2 Importer class for converting HuggingFace Savanna Hyena models to NeMo format. See: https://huggingface.co/arcinstitute/savanna_evo2_7b for an example of a savanna model that this can import and convert to NeMo format. Any of the Arc models that start with "savanna_" should work. cCs@ddl}ddlm}tjt|r&tdt|t j t|dddSdt|vr6t| d\}}nt|}d}| d d }tt |}|d }z |||||d }Wnt ytd |tj||}tj|r~td|ng} d} z|||||d| d } | | | d7} Wn |jjyYnwqt|d1} | D]&} t| d} |d}|sn| |qWdn1swYqWdn1swY| D],} zt| Wnty}ztd| d|WYd}~nd}~wwtd|qYnwt j |dddS)rrN)hf_hub_downloadzLoading model from local path rF)r weights_only:/rz.pt)repo_id local_dirrevisionfilenamez?Single path download failed, try loading checkpoint shards for zFound Tz.partrwbrbizError removing z: z,Cleaned up shards, final checkpoint saved to)huggingface_hub.errorshuggingface_hubr!osrexistsrrrr?rrr ExceptionrjoinappenderrorsEntryNotFoundErroropenreadwriteremoveOSErrorprint)r(r,r!r%r' modelname download_dirweights_filename weights_pathpartspart_num part_pathoutfilepartinfilechunker,r,r-rs|                +z0HuggingFaceSavannaHyenaImporter.get_source_modelN)r;r<r=r>rr,r,r,r-r s r ) 1b1b_nv7b7b_arc_longcontext7b_nv40b40b_arc_longcontext40b_nvtesttest_nvHYENA_MODEL_OPTIONS) rFrrrrrrrrrrrP)Ar- dataclassesrpathlibrtypingrrrrr? megatron.corerJmegatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapperr Imegatron.core.inference.model_inference_wrappers.inference_wrapper_configr megatron.core.transformer.enumsr ,megatron.core.transformer.transformer_configr #nemo.collections.llm.gpt.model.baser r?nemo.collections.llm.gpt.model.megatron.hyena.hyena_layer_specsrr9nemo.collections.llm.gpt.model.megatron.hyena.hyena_modelrr"9nemo.collections.llm.gpt.model.megatron.hyena.hyena_utilsrnemo.lightningrrrnemo.lightning.basernemo.lightning.io.stater nemo.utilsrr@rEIOMixinrFrrrrrrrrrrmodel_importerModelConnectorrr rPdictrr__all__r,r,r,r-sx             `r" !       O