o bi9@sddlZddlmZddlmZddlmZddlmZddZ edGd d d Z ed Gd d d e Z edGddde Z dS)N)tree) keras_export)serialization_lib)trackingcCsB|durdSt|trtj|}|durtd|d|S|S)a/Convert a loss string identifier to a loss function. Arguments: loss_item: Either a string identifier, a loss function instance, or `None`. Returns: A loss function instance, or `None`. Raises: ValueError: If the loss string identifier is unknown. NzUnknown loss function: 'z'.) isinstancestrkeraslossesget ValueError) loss_itemloss_fnr\/home/ubuntu/.local/lib/python3.10/site-packages/keras/src/distillation/distillation_loss.py_convert_loss_to_functions   rz#keras.distillation.DistillationLossc@s(eZdZdZddZddZddZdS) DistillationLossaBase class for distillation loss computation. Distillation losses define how to compute the distillation loss between teacher and student outputs. Each loss implements a specific approach to knowledge transfer, from simple logits matching to feature-based distillation. To create custom distillation losses, subclass this class and override the `compute_loss` method. cKstd)aCompute distillation loss between teacher and student outputs. This method should implement the specific distillation logic for transferring knowledge from teacher to student. Arguments: teacher_outputs: Outputs from the teacher model. Can be a single tensor or a list/tuple of tensors for multi-output models. student_outputs: Outputs from the student model. Can be a single tensor or a list/tuple of tensors for multi-output models. **kwargs: Additional arguments for custom distillation_loss. Returns: Distillation loss tensor. z&Subclasses must implement compute_loss)NotImplementedError)selfteacher_outputsstudent_outputskwargsrrr compute_loss-szDistillationLoss.compute_losscCstj||dS)aValidate that teacher and student outputs are compatible. Arguments: teacher_outputs: Outputs from the teacher model. student_outputs: Outputs from the student model. Raises: ValueError: If outputs are not compatible. N)rrassert_same_structure)rrrrrrvalidate_outputs>s z!DistillationLoss.validate_outputscCsdS)aValidate that teacher and student models are compatible. Arguments: teacher: The teacher model. student: The student model. Raises: ValueError: If models are not compatible with this distillation loss. Nr)rteacherstudentrrrvalidate_model_compatibilityIs z-DistillationLoss.validate_model_compatibilityN)__name__ __module__ __qualname____doc__rrrrrrrr s   rz&keras.distillation.FeatureDistillationcsVeZdZdZej dddZddZfdd Zd d Z d d Z e ddZ Z S)FeatureDistillationa6Feature distillation loss. Feature distillation transfers knowledge from intermediate layers of the teacher model to corresponding layers of the student model. This approach helps the student learn better internal representations and often leads to better performance compared to logits-only distillation. Arguments: loss: Loss function to use for feature distillation. Can be: - String identifier (e.g., 'mse', 'cosine_similarity', 'mae') - Keras loss instance - Nested structure of losses matching the layer output structure - `None` to skip distillation for that output (useful for multi-output models where you only want to distill some outputs) At least one loss must be non-`None`. Defaults to 'mse'. teacher_layer_name: Name of the teacher layer to extract features from. If `None`, uses the final output. Defaults to `None`. student_layer_name: Name of the student layer to extract features from. If `None`, uses the final output. Defaults to `None`. Examlpe(s): ```python # Basic feature distillation from final outputs distillation_loss = FeatureDistillation(loss="mse") # Distill from specific intermediate layers distillation_loss = FeatureDistillation( loss="mse", teacher_layer_name="dense_1", student_layer_name="dense_1" ) # Use cosine similarity for different feature sizes distillation_loss = FeatureDistillation( loss="cosine_similarity", teacher_layer_name="conv2d_2", student_layer_name="conv2d_1" ) # With custom loss instance distillation_loss = FeatureDistillation( loss=keras.losses.MeanAbsoluteError() ) # For multi-output models distillation_loss = FeatureDistillation( loss=["mse", "cosine_similarity"] ) # For multi-output models, only distill some outputs distillation_loss = FeatureDistillation( loss=["mse", None, "cosine_similarity"] # Skip middle output ) ``` mseNcCsD||_||_tt||_t|j}tdd|Dr tddS)Ncs|]}|duVqdSNr.0lrrr z/FeatureDistillation.__init__..zXThe `loss` argument in `FeatureDistillation` must contain at least one non-`None` value.) teacher_layer_namestudent_layer_namer map_structurerlossflattenallr )rr-r*r+ flat_lossesrrr__init__s zFeatureDistillation.__init__c Cs|jdus |jdurAt|d p|jdu}t|d p|jdu}|s$|rAg}|r-|d|r4|dd|}td|d|jdurcz |j|jdWntyb}ztd |d}~ww|jdurz |j|jdWdSty}ztd |d}~wwdS) zYValidate that teacher and student models are compatible for feature distillation.Ninputsrrz and z\FeatureDistillation with specific layer names requires Functional or Sequential models. The z model(s) appear to be subclassed (no symbolic inputs/outputs). Either use Functional/Sequential models, or use FeatureDistillation without layer names (to distill final outputs only), or use LogitsDistillation instead.)namezIn teacher model: zIn student model: )r*r+hasattrr2appendjoinr get_layer)rrrteacher_is_subclassedstudent_is_subclassedsubclassed_models models_strerrrrsB       z0FeatureDistillation.validate_model_compatibilityc sdt||z t|j|WdSty1}ztdt|jdt|d|d}~ww)z>Validate that outputs are compatible for feature distillation.z)Loss structure mismatch. Loss structure: z, Output structure: z . Error: N)superrrrr-r structure)rrrr< __class__rrrs z$FeatureDistillation.validate_outputscKs8dd}t||j||}t|}tjtj|S)aTCompute feature distillation loss using extracted features. Arguments: teacher_outputs: Extracted features from teacher layer. student_outputs: Extracted features from student layer. **kwargs: Additional arguments (ignored). Returns: Scalar distillation loss tensor. cSs"|durdStj|||}|S)N)ropsmean)r teacher_featuresstudent_featuresr-rrr apply_losssz4FeatureDistillation.compute_loss..apply_lossrr,r-r.rrBsumstack)rrrrrF loss_valuesr0rrrrs    z FeatureDistillation.compute_losscCstj|j|j|jdS)$Get configuration for serialization.)r-r*r+)rr serializer-r*r+rrrr get_configs zFeatureDistillation.get_configcC*|}tj|d|d<|di|Sz#Create instance from configuration.r-Nrcopyrr deserializeclsconfigrrr from_configzFeatureDistillation.from_config)r"NN)rrrr r no_automatic_dependency_trackingr1rrrrN classmethodrW __classcell__rrr?rr!Vs9  ,r!z%keras.distillation.LogitsDistillationc@s@eZdZdZej  d ddZddZdd Ze d d Z d S)LogitsDistillationaHDistillation loss that transfers knowledge from final model outputs. This distillation loss applies temperature scaling to the teacher's logits before computing the loss between teacher and student predictions. It's the most common approach for knowledge distillation. Arguments: temperature: Temperature for softmax scaling. Higher values produce softer probability distributions that are easier for the student to learn. Typical values range from 3-5. Defaults to 3.0. loss: Loss function to use for distillation. Can be: - String identifier (e.g., 'kl_divergence', 'categorical_crossentropy') - Keras loss instance - Nested structure of losses matching the model output structure - `None` to skip distillation for that output (useful for multi-output models where you only want to distill some outputs) At least one loss must be non-`None`. Defaults to 'kl_divergence'. Examlpe(s): ```python # Basic logits distillation with KL divergence distillation_loss = LogitsDistillation(temperature=3.0) # With categorical crossentropy loss distillation_loss = LogitsDistillation( temperature=4.0, loss="categorical_crossentropy" ) # With custom loss instance distillation_loss = LogitsDistillation( temperature=4.0, loss=keras.losses.CategoricalCrossentropy(from_logits=True) ) # For multi-output models distillation_loss = LogitsDistillation( temperature=3.0, loss=["kl_divergence", "categorical_crossentropy"] ) # For multi-output models, only distill some outputs distillation_loss = LogitsDistillation( temperature=3.0, loss=["kl_divergence", None] # Skip second output ) ``` @ kl_divergencecCst||_tt||_t|j}tdd|Drtdt|jt t fs/tdt |j|jdkr8tddS)Ncsr#r$rr%rrrr(Cr)z.LogitsDistillation.__init__..z%At least one loss must be non-`None`.z"temperature must be a number, got rAztemperature must be positive.) temperaturerr,rr-r.r/r rintfloattype)rr_r-r0rrrr19s  zLogitsDistillation.__init__c  sdtfdd|}tfdd|}fdd}t|j||}t|}tjtj|S)aCompute distillation loss using the configured loss function. Arguments: teacher_outputs: Logits from teacher model. Can be a single tensor, list/tuple of tensors, or dict of tensors. student_outputs: Logits from student model. Can be a single tensor, list/tuple of tensors, or dict of tensors. **kwargs: Additional arguments (ignored). Returns: Distillation loss tensor. ctj|jSr$rrBdivider_xrMrr[z1LogitsDistillation.compute_loss..crcr$rdrfrMrrrh^ricsl|durdSt|tjjr-tjj|dd}tjj|dd}tj|||}|jdStj|||S)NrA)axis)rrr KLDivergencerBsoftmaxrCr_)r teacher_logitsstudent_logits teacher_probs student_probsr-rMrrrFbsz3LogitsDistillation.compute_loss..apply_lossrG) rrrrteacher_scaledstudent_scaledrFrJr0rrMrrMs     zLogitsDistillation.compute_losscCs|jt|jdS)rK)r_r-)r_rserialize_keras_objectr-rMrrrrNzs zLogitsDistillation.get_configcCrOrPrQrTrrrrWrXzLogitsDistillation.from_configN)r]r^) rrrr rrYr1rrNrZrWrrrrr\s3 -r\) r keras.srcrkeras.src.api_exportrkeras.src.savingrkeras.src.utilsrrrr!r\rrrrs    5.