o oi@sddlZddlmZmZmZmZddlmZmZeGdddZ eGdddZ eGdd d Z eGd d d Z eGd d d Z GdddZdS)N)DictListAnyOptional) dataclassfieldc@seZdZUdZeed<eed<eed<eeed<dZeee eefed<dZ eed <dZ eeed <e d efd d Z d e eeffddZdS) DatasetConfigz"Configuration for a single datasetnametext_column_nameaudio_column_namespeaker_column_nameN add_constanttrainsplitsub_namereturncCs|jddS)z/Extract dataset prefix from name (part after /)/)r rselfr?/home/ubuntu/kanitts-2-dataset-pipeline/utils/config_manager.pydataset_prefixszDatasetConfig.dataset_prefixcCs|jsiSdd|jDS)z$Get constant columns as a dictionarycSsi|] }|d|dqS)keyvaluer).0itemrrr z6DatasetConfig.get_constant_columns..)r rrrrget_constant_columnssz"DatasetConfig.get_constant_columns)__name__ __module__ __qualname____doc__str__annotations__rr rrrrpropertyrrrrrrrs   rc@sVeZdZUdZeed<eed<eed<eed<eed<eed<eed<d Zeed <d S) BaseSettingszBase pipeline settings audio_codec num_readersqsizeOUT_DIR gzip_level buffer_sizelines_per_fileload_dataset_num_procN)r r!r"r#r$r%intr0rrrrr's r'c@s*eZdZUdZeeed<eeed<dS) SaveSettingsz&Settings for saving/uploading datasetslocal hf_uploadN)r r!r"r#rr$r%rrrrr2*s  r2c@s*eZdZUdZeed<eed<eed<dS)GroupEmbeddingSettingsz5Settings for the per-speaker averaged embedding step.do_thisgroup_by_column_namegrouped_embedding_columnN)r r!r"r#boolr%r$rrrrr51s  r5c@s~eZdZUdZeed<eed<eed<eed<eed<eed<eed<eed <e ed <e ed <d Z eed <dZ e e ed<dS)SpeakerEmbeddingSettingsz1Settings for the speaker embedding pipeline step.add_speaker_emb model_nameembedding_column target_sr max_audio_sec batch_sizeuse_multiprocessing do_clusters umap_paramshdbscan_params speaker_idclustering_speaker_columnN group_emb)r r!r"r#r9r%r$r1floatdictrFrGrr5rrrrr:9s  r:c@sfeZdZdZddefddZddd Zdeefd d Z de fd d Z de fddZ defddZdS) ConfigManagerz,Manages configuration loading and validation config.yaml config_pathcCst|d}t||_Wdn1swYtd&i|jd|_td&i|jd|_dd|jdD|_|j di}| di}t | d d | d d | d dd}t | dd | di dd| di dd| di dd| di dd| di dd| di dd| d i d!d | d i d"i| d i d#i| d i d$d |d% |_ dS)'Nr base_settings save_settingscSsg|] }tdi|qS)r)r)rdsrrr Trz*ConfigManager.__init__.. hf_datasetsspeaker_embedding_settings group_sp_embr6Fr7rEr8grouped_sp_emb)r6r7r8r;modelr r=wavlm_embeddingaudiotarget_sample_ratei>max_duration_secg>@ processingr@rAT clusteringrBUMAPHDBSCANspeaker_column) r;r<r=r>r?r@rArBrCrDrFrGr) openyaml safe_loadconfigr'rNr2rOdatasetsgetr5r:rS)rrLfspgrprGrrr__init__Ns6       zConfigManager.__init__rNcCs|jstdt}g}|jD]}t|}||j|f||q|rA|D]\}}||}|r@td|d|dq+tddS)z Validate that all datasets have matching additional columns. This prevents conflicts when merging the final dataset. z&No datasets specified in configurationz Dataset 'z' is missing constant columns: zF. All datasets must have the same constant columns for proper merging.uL✅ Dataset validation passed: All datasets have matching additional columnsN) rf ValueErrorsetrkeysappendr updateprint)rall_constant_keysdataset_constant_keysrP constant_keysds_namern missing_keysrrrvalidate_datasetsls"    zConfigManager.validate_datasetscC|jS)z"Get list of dataset configurations)rfrrrr get_datasetszConfigManager.get_datasetscCrx)zGet base pipeline settings)rNrrrrget_base_settingsrzzConfigManager.get_base_settingscCrx)zGet save/upload settings)rOrrrrget_save_settingsrzzConfigManager.get_save_settingscCrx)zGet speaker embedding settings)rSrrrrget_speaker_embedding_settingsrzz,ConfigManager.get_speaker_embedding_settings)rK)rN)r r!r"r#r$rkrwrrryr'r{r2r|r:r}rrrrrJKs rJ)rctypingrrrr dataclassesrrrr'r2r5r:rJrrrrs