o wi@@sNddlmZddlmZddlmZddlmZ ddl m Z GdddZ dS)N)nn)Self)base)loggingcseZdZdZfddZddejdedefdd Zddej d ede fd d Z ddej d ede fd dZ dddZ dddZZS)FNMixina A mixin class providing utility methods for operating on PyTorch modules. This mixin class offers methods to apply functions, check predicates, and modify the state (freeze/unfreeze) of PyTorch modules within a container. It is designed to be used with classes that are composed of multiple PyTorch modules, facilitating operations that affect all contained modules either directly or recursively. Methods ------- forall: Checks if a predicate holds for all modules. map: Applies a function to each module. walk: Traverses each module, applying a function. freeze: Freezes the parameters of all modules. unfreeze: Unfreezes the parameters of all modules. Examples -------- >>> class MyModel(nn.Module, FNMixin): ... def __init__(self): ... super().__init__() ... self.layer1 = nn.Linear(10, 10) ... self.layer2 = nn.Linear(10, 10) ... >>> model = MyModel() >>> model.freeze() # Freezes all parameters in the model >>> model.forall(lambda module: not module.parameters().requires_grad, recurse=True) True c s:t|tjrddlm}||ddtjdi|dS)Nr)hook_class_init_with_callbackson_model_init_starton_model_init_end) issubclassplLightningModulenemo.lightning.callback_grouprsuper__init_subclass__)clskwargsr __class__r Z/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/fn/mixin.pyr6s   zFNMixin.__init_subclass__FfuncrecursereturncCs$t|tjs Jdtj|||dS)a Evaluates a predicate for all modules in the container, optionally recursively. This method checks if a given predicate holds for all modules in the container. If `recurse` is True, it also checks all submodules recursively. Args: func (fn.ModulePredicate): A predicate function to apply to each module. recurse (bool, optional): Whether to apply the predicate recursively. Defaults to False. Returns ------- bool: True if the predicate holds for all modules, False otherwise. Example: >>> model = MyModel() >>> model.forall(lambda module: isinstance(module, nn.Linear), recurse=True) True self is not a nn.Module)r) isinstancerModulefnforall)selfrrr r rr?szFNMixin.forall leaf_onlycC&t|tjs Jdtj|||ddS)a Applies a function to each module in the container, optionally to leaf modules only. This method applies a given function to each module in the container. If `leaf_only` is True, the function is applied to leaf modules only. Args: func (fn.ModuleFunc): A function to apply to each module. leaf_only (bool, optional): Whether to apply the function to leaf modules only. Defaults to False. Returns ------- Self: The container itself after applying the function. Example: >>> model = MyModel() >>> model.map(lambda module: module.double() if isinstance(module, nn.Linear) else module) rTr _skip_map)rrrrmaprrrr r rr#Wz FNMixin.mapcCr )a Traverses each module in the container, applying a function, optionally to leaf modules only. This method is similar to `map`, but it is typically used for operations that do not modify the modules but instead collect information or perform checks. Args: func (fn.ModuleFunc): A function to apply to each module. leaf_only (bool, optional): Whether to traverse leaf modules only. Defaults to False. Returns ------- Self: The container itself after the traversal. Example: >>> model = MyModel() >>> model.walk(print, leaf_only=True) rTr!)rrrrwalkr$r r rr&or%z FNMixin.walkNcCPt|tjs Jdt|}|std|jjdS|D]}d|_ q dS)zu Freezes the parameters of all modules in the container by setting `requires_grad` to False. rNo parameters found in module FN rrrlist parametersrinfor__name__ requires_gradrparamsparamr r rfreeze zFNMixin.freezecCr')zv Unfreezes the parameters of all modules in the container by setting `requires_grad` to True. rr(TNr)r/r r runfreezer3zFNMixin.unfreeze)F)rN)r- __module__ __qualname____doc__rrModulePredicateboolr ModuleFuncrr#r&r2r4 __classcell__r r rrrs   r) lightning.pytorchpytorchr torchrtyping_extensionsrnemo.collections.llm.fnrr nemo.utilsrrr r r rs