o biY@s`ddlZddlmZddlmZddlmZddlmZddl m Z edGdd d eZ dS) N)tree) keras_export)_convert_loss_to_function)Model)serialization_libzkeras.distillation.DistillercseZdZdZ   d-fdd ZddZd d Zd d Zd dZddZ ddZ ddZ ddZ ddZ ddZddZddZd.fd d! Zd/d"d#Z $d0d%d&Zfd'd(Zfd)d*Zed+d,ZZS)1 Distillera Distillation model for transferring knowledge from teacher to student. Knowledge distillation transfers knowledge from a large, complex model (teacher) to a smaller, simpler model (student). The student learns from both ground truth labels and the teacher's predictions, often achieving better performance than training on labels alone. Arguments: teacher: A trained `keras.Model` that serves as the knowledge source. The teacher model is frozen during distillation. student: A `keras.Model` to be trained through distillation. distillation_losses: List of distillation losses to apply. Can be a single distillation loss or a list of distillation losses like `keras.distillation.LogitsDistillation`, `keras.distillation.FeatureDistillation`, or custom distillation losses. distillation_loss_weights: List of weights for each distillation loss. Must have the same length as `distillation_losses`. If `None`, equal weights are used. student_loss_weight: Weight for the student's supervised loss component. Must be between 0 and 1. Defaults to 0.5. name: Name for the distiller model. Defaults to `"distiller"`. **kwargs: Additional keyword arguments passed to the parent `Model` class. Attributes: student: The student model being trained. Access this to get the trained student model for independent use after distillation training. teacher: The teacher model providing knowledge. This model is frozen during training. Examples: ```python # Basic distillation with KerasHub models import keras_hub as hub teacher = hub.models.CausalLM.from_preset("gemma_2b_en") student = hub.models.CausalLM.from_preset( "gemma_1.1_2b_en", load_weights=False ) # Single distillation loss distiller = Distiller( teacher=teacher, student=student, distillation_losses=LogitsDistillation(temperature=3.0), ) # Compile the distiller (like any Keras model) distiller.compile( optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'] ) # Train the distiller distiller.fit(x_train, y_train, epochs=10) # Access the trained student model trained_student = distiller.student # Multiple distillation losses distiller = Distiller( teacher=teacher, student=student, distillation_losses=[ LogitsDistillation(temperature=3.0), FeatureDistillation( teacher_layer_name="dense_1", student_layer_name="dense_1" ) ], distillation_loss_weights=[1.0, 0.5], ) # Compile with custom settings distiller.compile( optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'] ) ``` N? distillerc  sTtjdd|i||||||_||_t|ttfs'tdt ||dks/|dkr6td|||_ |durAtdt|t t fsQ|g|_ dg|_n*||_ |duradgt||_nt|t|krxtdt|dt|d ||_|j D] }||||q~|d |j_tjjd d |_tjjd d |_tjjdd |_dS)Nnamez*student_loss_weight must be a number, got ?z5student_loss_weight must be between 0.0 and 1.0, got z'distillation_losses' cannot be `None`. Provide a distillation loss (e.g., LogitsDistillation or FeatureDistillation) or a list of distillation losses.z%Number of distillation_loss_weights (z,) must match number of distillation_losses ()F student_lossr distillation_loss total_loss)super__init___validate_modelsteacherstudent isinstanceintfloat ValueErrortypestudent_loss_weightlisttupledistillation_lossesdistillation_loss_weightslen)_validate_distillation_loss_compatibility _create_multi_feature_extractors trainablekerasmetricsMeanstudent_loss_trackerdistillation_loss_trackertotal_loss_tracker) selfrrr r!rr kwargsr __class__rT/home/ubuntu/.local/lib/python3.10/site-packages/keras/src/distillation/distiller.pyr`sb    zDistiller.__init__cCsdt|tjstdt|t|tjstdt||||||||||dS)z8Validate that teacher and student models are compatible.z#Teacher must be a keras.Model, got z#Student must be a keras.Model, got N)rr&rrr_validate_input_compatibility_validate_output_compatibility_validate_dtype_compatibilityr,rrrrr0rs      zDistiller._validate_modelsc Csxt|t|krtd|d|d|dt||D]\}}|dur9|dur9||kr9td|d|d|dqdS)z&Assert that two shapes are compatible.Teacher and student z, shapes have different dimensions. Teacher: , Student: .Nz# shapes are incompatible. Teacher: z. All dimensions must match.)r"rzip)r,shape1shape2contextdim1dim2rrr0_assert_shapes_are_compatibles&z'Distiller._assert_shapes_are_compatiblecCs(||krtd|d|d|ddS)z4Assert that teacher and student dtypes are the same.r5z dtypes must match. Teacher: r6r7N)r)r, teacher_dtype student_dtyper;rrr0_assert_same_dtypeszDistiller._assert_same_dtypecZt|dr t|ds dSt|d}t|d}|dus|dur dStfdd||dS)z?Validate that teacher and student have compatible input shapes.inputsNc|j|jdSNinputr>shapetisir,rr0 z9Distiller._validate_input_compatibility..hasattrgetattrr map_structure)r,rrteacher_inputsstudent_inputsrrLr0r1   z'Distiller._validate_input_compatibilitycrB)z@Validate that teacher and student have compatible output shapes.outputsNcrDNoutputrGtosorLrr0rMrNz:Distiller._validate_output_compatibility..rO)r,rrteacher_outputsstudent_outputsrrLr0r2rUz(Distiller._validate_output_compatibilitycst|dr t|ds dS|jdus|jdurdStfdd|j|jt|dr/t|ds1dS|jdus;|jdur=dStfdd|j|jdS)z=Validate that teacher and student have compatible data types.rCNcrDrErAdtyperIrLrr0rMsz9Distiller._validate_dtype_compatibility..rVcrDrWr^rYrLrr0rM rN)rPrCrrRrVr4rrLr0r3s$  z'Distiller._validate_dtype_compatibilitycCs|||dS)zZValidate that the distillation loss is compatible with teacher and student models.N)validate_model_compatibility)r,rrrrrr0r#sz3Distiller._validate_distillation_loss_compatibilitycCsg}g}|jD](}t|dr|jr|j|vr||jt|dr/|jr/|j|vr/||jq||j||_||j||_ dS)z?Create feature extractors for efficient multi-layer extraction.teacher_layer_namestudent_layer_nameN) r rPraappendrb_create_feature_extractorr_teacher_feature_extractorr_student_feature_extractor)r,teacher_layer_namesstudent_layer_namesrrrr0r$s<  z*Distiller._create_multi_feature_extractorscCs|sdSt|dr|jdurtd|jdt|tjr$|jdj}n|j}d|i}|D] }|j |d}|j||<q-tj |j||jdd S) arCreate a feature extractor for a model. Arguments: model: The model to create an extractor for. layer_names: List of layer names to extract features from. Returns: Feature extractor model or `None` if no layer names provided. Raises: ValueError: If model has no symbolic inputs/outputs. NrCz$Cannot create feature extractor for z-. The model has no symbolic inputs attribute. final_outputr_multi_feature_extractor)rCrVr ) rPrCrr rr& SequentiallayersrX get_layerr)r,model layer_namesrjrV layer_namelayerrrr0rd=s$      z#Distiller._create_feature_extractorcCs*|jdur |j|ddSd|j|ddiS)z6Extract all teacher features in a single forward pass.NFtrainingrj)rer)r,xrrr0_extract_all_teacher_featurescs z'Distiller._extract_all_teacher_featurescCs |jdur |j|ddSd|iS)z6Extract all student features in a single forward pass.NTrsrj)rf)r,ruy_predrrr0_extract_all_student_featuresjs z'Distiller._extract_all_student_featurescCsF|r|jpd}n|jp d}||vrtd|dt|||S)z8Get the specific features needed by a distillation loss.rjzLayer 'z.' not found in extracted features. Available: )rarbrrkeys)r,r all_features is_teacherrqrrr0_get_distillation_loss_featuresqs   z)Distiller._get_distillation_loss_featuresadamc sj|durtdtt||_||_|dur&t|ttfs&tdt |t j d|d|d|dS)aCompile the distiller with proper integration. Arguments: optimizer: Optimizer for training the student model. loss: Student loss function for the student's supervised learning. Can be a string identifier or a loss function instance. metrics: Additional metrics to track during training. **kwargs: Additional arguments passed to parent compile. Nz'loss' cannot be `None`.z%metrics must be a list or tuple, got ) optimizerlossr'r) rrrRr _student_loss_student_loss_for_serializationrrrrrcompile)r,r~rr'r-r.rr0rs   zDistiller.compilecKs|j|fd|i|S)z)Forward pass returns student predictions.rt)r)r,rCrtr-rrr0callszDistiller.callTc Cs|dur |||d}d}|jdkrK|durKtdd|j||}t|}t|dkr5tjtj |n|d}t |drKt|j dkrKtj |}d} |jd kr| |} |||} t|j|jD]{\} } t | d r| jdurz|j| | d d }|j| | d d }Wn(ty}ztdt| jd| jd| jd||d}~ww| d}|}| ||| ||}t |drt|j dkrtd| jjd|j dtj| tj| |} qdtjtj|j|tjtjd |j| }|j ||j! | |j" ||S)aMCompute combined distillation loss. Arguments: x: Input data. y: Target data. y_pred: Model predictions. sample_weight: Sample weights (currently unused). training: Whether the model is in training mode. Returns: Combined loss tensor. Nrsr cSs |||SNr)loo_predrrr0rMs z(Distiller.compute_loss..rrHr raT)r{FzFailed to extract features for z targeting teacher layer 'z' and student layer 'z'. Original error: rjzDistillation loss z' returned a non-scalar loss with shape z6. The compute_loss method must return a scalar tensor.)#rrrRrflattenr"r&opssumstackrPrHmeanrvrxr8r r!rar|r RuntimeErrorr__name__rbvalidate_outputs compute_lossr/addmultiplysubtractr) update_stater*r+)r,ruyrw sample_weightrtr loss_values flat_lossesrteacher_featuresstudent_featuresdistillation_loss_fnweight distillation_loss_teacher_output distillation_loss_student_outputecurrent_distillation_lossrrrr0rs              zDistiller.compute_losscs,t|j|j|jdS)zReset all metrics.N)r reset_metricsr) reset_stater*r+rLr.rr0r)s   zDistiller.reset_metricscsDt}|t|jt|jdd|jD|j|j d|S)z$Get configuration for serialization.cSg|]}t|qSr)rserialize_keras_object.0rrrr0 ;z(Distiller.get_config..)rrr r!r) r get_configupdaterrrrr r!r)r,configr.rr0r0s zDistiller.get_configcCsP|}t|d|d<t|d|d<dd|dD|d<|di|S)z#Create instance from configuration.rrcSrr)rdeserialize_keras_objectrrrr0rQrz)Distiller.from_config..r Nr)copyrr)clsrrrr0 from_configEs zDistiller.from_config)Nrr )r}NNr)NNNNT)r __module__ __qualname____doc__rrr>rAr1r2r3r#r$rdrvrxr|rrrrr classmethodr __classcell__rrr.r0r s6ZO$&    r) r& keras.srcrkeras.src.api_exportr(keras.src.distillation.distillation_lossrkeras.src.models.modelrkeras.src.savingrrrrrr0s