o wi<@s|dZddlZddlZddlZddlZddlZddlmZddl m Z Gdddej j j ZGdddej j j Zd d ZdS) zBlendable dataset.N)logging)AppStatec@s0eZdZdZddZddZddZdd Zd S) BlendableDataset c Cs||_t|}|t|ksJ||_tj|tjd}t|}|dks%J||}t}|dks3Jtj|jtj d|_ tj|jtj d|_ t }z|jdkrSttjddlm}Wn tyjtdw||j |j |||jtjdktdt|dS)Ndtyper)helperszhCould not compile megatron dataset C++ helper functions and therefore cannot import helpers python file.zC> elapsed time for building blendable dataset indices: {:.2f} (sec))datasetslensizenparrayfloat64sumtimezerosuint8 dataset_indexint64dataset_sample_indexr local_rankcompile_helpertorch distributedbarriernemo.collections.common.datar ImportErrorbuild_blending_indicesget_rankrinfoformat) selfr weightsr num_datasets sum_weights start_time app_stater r)k/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/data/blendable_dataset.py__init__sD       zBlendableDataset.__init__cC|jSNr r#r)r)r*__len__JzBlendableDataset.__len__cCsp|j|}|j|}t|j|}||kr1td|d|d||}td|d|d|j||S)NzIndex z out of bounds for dataset z. Reusing existing examples.zReusing index z for dataset .)rrr r rwarning)r#idx dataset_idx sample_idx dataset_sizer)r)r* __getitem__Ms  zBlendableDataset.__getitem__cCs|jD]}|qdS)rN)r create_data_mmap)r#datasetr)r)r*r9Ys  z!BlendableDataset.create_data_mmapN)__name__ __module__ __qualname____doc__r+r0r8r9r)r)r)r*rs + rc@s>eZdZdZdddZddZddZd d Zed d Z d S)MemoryEfficientBlendableDatasetz A BlendableDataset implementation that uses less memory than the original implementation. Indices are computed algorithmically instead of storing them in memory. To test call: MemoryEfficientBlendableDataset.test_index_blending() dc sx|_t|}|t|ksJt||}|_|_tj|tjd}|dks*Jt |}|dks5J||_ g}g}t j D]\} } t | |} | | g| | t| qC|t|} | | g| | t|d|d| tj|tjd_tjfddt|Dtjd_jdksJdjtj|tjd_tjdd|Dtjd_dS) Nrrcsg|] }j|kqSr))ds_indexr.0ir/r)r* sz.rzvSome datasets have no samples in the blendable dataset, increase weight_bins or the offending weight. ds_index_size = cSsg|]}t|qSr))r )rDdsr)r)r*rFs)r r minr weight_binsrrrallrr$ enumerateintextendrangeuint32rB ds_index_sizeds_biasds_size) r#r r$r rIr%r&rBrQrEwnr)r/r*r+gs:      $ z(MemoryEfficientBlendableDataset.__init__cCsD||j}|j|}|j|||j|j||j|}||fS)z_Returns ds index and sample index (within the ds) for the given index in the blendable dataset.)rIrBrQrPrR)r#r4binds_idxr6r)r)r*get_ds_sample_idxs   z1MemoryEfficientBlendableDataset.get_ds_sample_idxcCr,r-r.r/r)r)r*r0r1z'MemoryEfficientBlendableDataset.__len__cCs||\}}|j||Sr-)rWr )r#r4rVr6r)r)r*r8sz+MemoryEfficientBlendableDataset.__getitem__csddlm}|Gdddtjjj}dD]U}t|dd|dd|dd ggd d |d fd dtd D}t t |d}t t |d}| |j |dd|j |dd| ||d|qdS)z$Visualize indices of blended datasetrNc@s(eZdZdZddZddZddZdS) z?MemoryEfficientBlendableDataset.test_index_blending..DSrcSs||_||_dSr-)r data)r#r rXr)r)r*r+s zHMemoryEfficientBlendableDataset.test_index_blending..DS.__init__cSr,r-r.r/r)r)r*r0r1zGMemoryEfficientBlendableDataset.test_index_blending..DS.__len__cSs |j|Sr-)rX)r#r4r)r)r*r8s zKMemoryEfficientBlendableDataset.test_index_blending..DS.__getitem__N)r;r<r=r>r+r0r8r)r)r)r*DSs  rY) r@rZabc)g?g333333?g?2)rIcsg|]}|qSr))rWrCblend_dsr)r*rFszGMemoryEfficientBlendableDataset.test_index_blending..zds idx)labelsamplez weight_bins=)matplotlib.pyplotpyplotionrutilsrXDatasetr?rNlistzipfigureplotlegendgridtitle)clspltrYrIds_sample_idx_listds_list sample_listr)r_r*test_index_blendings"  $z3MemoryEfficientBlendableDataset.test_index_blendingN)r@) r;r<r=r>r+rWr0r8 classmethodrur)r)r)r*r?_s ( r?cCsRtjtjt}tdd|g}|jdkr't dddl }| ddSdS)zVCompile helper function ar runtime. Make sure this is invoked on a single process.makez-Crz2Making C++ dataset helpers module failed, exiting.Nra) ospathabspathdirname__file__ subprocessrun returncodererrorsysexit)ryretrr)r)r*rs  r)r>rxr}rnumpyrr nemo.utilsrnemo.utils.app_staterrgrXrhrr?rr)r)r)r*s  C i