o }oii2@sddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZdd gZejejejejejejejejeejejejejd ZGd dde Z Gd d d e Z!dS) )ListTupleN)Callback) GPTConfig)PEFT)flops_formulaslogging)hyenaFLOPsMeasurementCallbackMM_FLOPsMeasurementCallback) gpt3llama2llama3llama4 nemotron3 nemotron4mixtralbertr deepseekv3 transformerqwen3 nemotronhc@sreZdZdZdZdedejdefddZ dd Z d e fd d Z d e eBe BdefddZdeeeffddZdS)r a Calculate and log FLOPs per second after every ``trainer.log_every_n_steps`` steps. Args: model_config (GPTConfig): Model parameters. data_config (pl.LightningDataModule): Data module being used in the experiment. model_name (str): Name of the model being run. The following models are supported: gpt3, llama2, llama3, nemotron, mixtral, bert, hyena. T model_config data_config model_namec Cs||_||_||_|jj}|jj}|jj}|jj}|jj}|jj} |jj } t |jdd} t |jdr7|jj j nd} |jj} | durC| } |||||| | | | | d }ddlm}t|j|rz|jj|d<|jj|d<|jj|d<|jj|d <|jj|d <|jj|d <|jj|d <|jj|d <|jj|d<|jjrd|d<|jj|d<|jj|d<|jj|d<|jj|d<|jj |d<t!j"di||_#|jdur|j$n|j|_d|_%dS)Nhybrid_override_pattern tokenizer) gbs enc_seq_lenhslayersffn_hsattention_headsmoe_router_topk query_groups vocab_size model_patternr)MLATransformerConfig qk_head_dimqk_pos_emb_head_dim v_head_dim q_lora_rank kv_lora_rankmoe_layer_freq#moe_shared_expert_intermediate_sizemoe_ffn_hidden_sizemtp_num_layersTis_hybrid_modelmamba_state_dimmamba_head_dimmamba_num_groupsmamba_num_heads)& model_cfgdata_cfgmodelglobal_batch_size seq_length hidden_size num_layersffn_hidden_sizenum_attention_headsr#getattrhasattrrr%num_query_groups,megatron.core.transformer.transformer_configr' isinstancer(r)r*r+r,r-r.r/r0r1rr2r3r4r5r FLOPSConfig flops_configloweravg_train_step_time)selfrrrrrrr r!r"r#r&r%r$ config_kwargsr'r6r6c/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/callbacks/flops_callback.py__init__>s^                z!FLOPsMeasurementCallback.__init__cCs"|jD] }t|trtdqdS)z PyTorch Lightning callback hook. Ensures that user is not using PEFT as FLOPS callback does not support it. z3FLOPs measurement not supported for finetuning jobsN) callbacksrDrNotImplementedError)rItrainer pl_modulecallbackr6r6rKon_train_starts  z'FLOPsMeasurementCallback.on_train_start batch_idxc Csz |j|jd7_Wn tytdYnw|jj}||jdkrU|jdkr,dS|j|j}||\}} d|_|jd|ddddd | d |} |d | dSdS) zd PyTorch Lightning callback hook to calculate TFLOPs per sec per GPU after training ztrain_step_timing in szb'train_step_timing in s' not found. Make sure to use TimingCallback with FLOPsMeasurementCallback.rNTFLOPS_per_GPUTF)on_stepon_epoch batch_sizeprog_barmBTFLOPS) rHprogress_bar_metricsKeyErrorprintstrategycurrent_epoch_steplog_every_n_stepseval_tflops_per_sec_per_gpulog) rIrOrPoutputsbatchrSntrain_step_timetflops_per_gpuflopstflopsr6r6rKon_train_batch_ends4     z+FLOPsMeasurementCallback.on_train_batch_endrgreturncCsT|\}}t|ts|g}t|}t|t|dd}|d|}||fS)ae Args: train_step_time (Any[List, float, int]): Train step time (in seconds). Step time will be less stable for initial steps (~10 steps)- less accurate measurement Use average step time over several steps for higher accuracy Returns: (float): Model TFLOPs per sec per gpu NrZ)eval_model_flopsrDlistnparraymeanlen)rIrg total_flops flops_per_gpu step_time_arrflops_per_sec_per_gpur6r6rKrbs   z4FLOPsMeasurementCallback.eval_tflops_per_sec_per_gpucsjdurfddtD}t|dkr|dnj_jtvr5tdtttdjtjj}t j rGt j nd}||}||fS)z9 Calculate model FLOPs for a given model Ncsg|] }|jvr|qSr6)r9).0r9rIr6rK sz=FLOPsMeasurementCallback.eval_model_flops..r FLOPs measurement supported for JFailed to extract valid model name from or missing FLOPs calculations for rU) r9_model_flops_maprsrinforokeysr]rFtorch distributedis_initializedget_world_size)rI model_matchesrt num_devicesrur6ryrKrns  z)FLOPsMeasurementCallback.eval_model_flopsN)__name__ __module__ __qualname____doc__higher_is_betterrplLightningDataModulestrrLrRintrkrfloatrbrrnr6r6r6rKr /s  C c@s0eZdZdZdZdedejfddZddZ d S) r a Calculate and log FLOPs per second after every ``trainer.log_every_n_steps`` steps for multi-modal models. The following models are supported: hf_clip_vit_l, neva_projection, gpt3, llama2, llama3, nemotron, mixtral, bert, hyena Args: model_name_config_dict (dict): Dictionary containing all the individual model configs that make up the multi-modal model. data_config (pl.LightningDataModule): Data module being used in the experiment. Tmodel_name_config_dictrcCs||_t|_|D]\}}t}|jj|d<|j|d<|dvrD|j|d<|j|d<|j|d<|j|d<|j |d<|j |d <d |d <nZ|d vr_|j |d <|j |d<|j |d<|dj|d<n?|dvr|j|jg|d<|j|d<|j|d<|j|d<|j|d <|j|d<n|j|d<|j|d<|j |d<|j|d<|j|d<z|j}|dur|j}||d<WnYtjdi||j|<q d|_dS)Nrr) hf_clip_vit_lr img_seq_lenimg_himg_w patch_dim in_channelsrUclass_token_len)neva_projectionprojector_typer!inp_sr)fluxmodel_channels vec_in_dimrr"r#r$rr6)r8dictflops_config_dictitemsr:r<num_hidden_layersnum_image_embeddings_per_tile image_size patch_size num_channelsrr> input_sizenum_joint_layersnum_single_layersr context_dimrrr;r=r?r#rBrrErH)rIrrrr7kwargsr$r6r6rKrLsP                        z$MM_FLOPsMeasurementCallback.__init__cCsittjtjtjd}d}}|jD]#\}}||vr1tdt | t d|||||7}qt j rDt j nd}||}||fS)zh Calculate model FLOPs for a given model recursively when model has multiple sub-models )rrrrr{r|rU)r}r clip_vit_lrrrrrr~rorr]rrrr)rImm_model_flops_maprtrur flops_cfgrr6r6rKrns$z,MM_FLOPsMeasurementCallback.eval_model_flopsN) rrrrrrrrrLrnr6r6r6rKr s  5)"typingrrlightning.pytorchpytorchrnumpyrprlightning.pytorch.callbacksr#nemo.collections.llm.gpt.model.baser nemo.lightning.pytorch.callbacksr nemo.utilsrrnemo.utils.hyena_flops_formulasr __all__r r rnemotronrrrrrrr}r r r6r6r6rKs6     #