o wi @sddlZddlmZmZddlZddlZddlmZddlm Z defddZ ddee e ej jfd ee d ee d eej jddffd d Zdej jd efddZdS)N) GeneratorIterable)LightningModule)loggingmodelcCst|ds Jdd|jvsJdt||jdg|jdgd}tjj|jj|dd }d|i}d |jvrJtj|jj |}|d d d |d <|S)a Re-usable optimizer configuration function for top-level PyTorch Lightning modules in this collection. It sets up parameter freezing, optimizer, and LR scheduler. The ``model`` object is expected to have a ``model.cfg`` attribute with OmegaConf configuration. The following fields are expected: * ``optimizer`` with hydra-style ``_target_`` pointing to optimizer class, and the remaining options passed directly to its ``__init__`` method. * (optional) ``freeze_params`` with a list of regex pattern for identifying frozen parameters. * (optional) ``prevent_freeze_params`` with a list of regex pattern for keeping specific parameters trainable (overrides ``freeze_params``). * (optional) ``lr_scheduler`` with hydra-style ``_target_`` pointing to LR scheduler class, and the remaining options passed directly to its ``__init__`` method. Returns: PyTorch Lightning Trainer-compatible dict with structure:: { "optimizer": , "lr_scheduler": {"scheduler": , "interval": "step", "frequency": 1} } cfgz(Expected `model.cfg` attribute to exist. optimizerz:Expected `model.cfg` to contain 'optimizer' configuration. freeze_paramsprevent_freeze_params)exclude_patterns keep_patternsall) _convert_ lr_schedulerstep) schedulerinterval frequency) hasattrrfreeze_and_subsetnamed_parametersgethydrautils instantiaterr)r parametersransrri/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/speechlm2/parts/optim_setup.pyconfigure_optimizerss   r rr r returnc#sdd|D|sidtfdd}ndd|Ddd|Dd tdtffd d }d d|Dd tdtffd d }d\}}|D]%\}}d} ||rZ||sZd|_d} | sf|V||7}qG||7}qG||} td|d|| dd| ddD} rdd| d} td| dddD} rdd| d} td| ddSdS)a Utility used to freeze select model parameters, and skip them for the purpose of initializing an optimizer's parameter group. Args: named_parameters: The output of `torch.nn.Module.named_parameters()` exclude_patterns: A list of regex patterns matching parameter names to be frozen and excluded from optimization. keep_patterns: A list of regex patterns matching parameter names to be trained. This list overrides all matches to `exclude_patterns`. Returns: A generator over parameters, equivalent to calling `torch.nn.Module.parameters()`, that will be passed to the optimizer and trained. Example: >>> model = MyModel() ... # freeze all LLM parameters in "model.llm" ... params = freeze_and_subset(model.named_parameters(), [r'^llm\.\..+$']) ... optimizer = torch.optim.AdamW(params, lr=1e-3) cSi|]}|dqSrr.0prrr _z%freeze_and_subset..r!cSsdS)NFr)_rrr _must_keepdsz%freeze_and_subset.._must_keepcSr"r#rr$rrrr'hr(cSg|]}t|qSrrecompiler$rrr iz%freeze_and_subset..namec4D]}||dur|jd7<dSqdSNrTFmatchpatternr1r&)compiled_keep_patterns keep_counterrrr*k cSr+rr,r$rrrr/rr0cr2r3r4r7)compiled_exclude_patternsexclude_counterrr_excludetr:z#freeze_and_subset.._exclude)rrFTzParameters | trainable=z (z.2%z ) | total=cSg|] \}}|dkr|qSr#rr%kvrrrr/z['z', 'z']z=Parameter freezing patterns UNMATCHED against any parameter: z (bad regexp?)cSr>r#rr?rrrr/rBzFParameter freeze-preventing patterns UNMATCHED against any parameter: N) boolstr requires_gradnumelrinfoitemsjoinwarning)rr r r*r= trainable nontrainabler1paramdiscardtotalunused_excluded_patternsmsgunused_keep_patternsr)r;r8r<r9rrCs: "rmodulecCstdd|DS)Ncss|]}|j VqdSN)rEr$rrr szis_frozen..)r r)rSrrr is_frozensrVrT)r-typingrrrtorch lightningr nemo.utilsrr tuplerDnn ParameterlistrModulerCrVrrrrs$   . P