o wi(@sddlZddlZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl mZddlmZddlmZddlZddlmZdZd Zd Zd d Zejd dZejddddZejddddZejddddZejddddZejddddZejddddZ ejdddd Z!d+d"d#Z"ejddd$ee#e$ffd%d&Z%ejddd$ee#e$ffd'd(Z&d)d*Z'dS),N)mkdir)dirnameexistsgetsizejoin)Path)rmtree)Tuple) Singletonztest_data.tar.gzz;https://github.com/NVIDIA/NeMo/releases/download/v1.0.0rc1/z.datacCsT|jdddd|jdddd|jdddd|jd d d d|jd dd ddS)z Additional command-line arguments passed to pytest. For now: --cpu: use CPU during testing (DEFAULT: GPU) --use_local_test_data: use local test data/skip downloading from URL/GitHub (DEFAULT: False) --cpu store_truezCpass that argument to use CPU during testing (DEFAULT: False = GPU))actionhelpz--use_local_test_dataz[pass that argument to use local test data/skip downloading from URL/GitHub (DEFAULT: False)--with_downloadszHpass this argument to active tests which download models from the cloud.z--relax_numba_compat store_falsezpnumba compatibility checks will be relaxed to just availability of cuda, without cuda compatibility matrix check --nightlyzepass this argument to activate tests which have been marked as nightly for nightly quality assurance.N) addoption)parserrK/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/tests/conftest.pypytest_addoption%s. rcCs|jdrdSdS)z?Simple fixture returning string denoting the device [CPU | GPU]r CPUGPU)config getoption)requestrrrdeviceFs rT)autousecCs>|jdr|jdjd|krtd|dSdSdS)N run_only_onrzskipped on this device: {})nodeget_closest_markerargspytestskipformatrrrrrrun_only_on_device_fixtureOs r&cC.|jdr|jdstddSdSdS)Nwith_downloadsrz_To run this test, pass --with_downloads option. It will download (and cache) models from cloud.rr rrr"r#r%rrrdownloads_weightsV  r*cCr')NnightlyrzTo run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests are mostly used for QA.r)r%rrrrun_nightly_test_for_qa_r+r-ccstdr JtdrJtdrJdVtdr(tdddtdr4tdddtdrBtddddSdS)Nz./lightning_logsz./NeMo_experimentsz./nemo_experimentsT) ignore_errors)rrrrrrrcleanup_local_folderhs     r/cCs it_dSN)r _Singleton__instancesrrrrreset_singletonsys r2ccs,ttj}dVtjtj|dSr0)dictosenvironclearupdate) original_envrrrreset_env_vars~s   r9session)scopecCstttt}|S)z Fixture returns test_data_dir. Use the highest fixture scope `session` to allow other fixtures with any other scope to use it. )rr__file____TEST_DATA_SUBDIR)test_data_dir_rrr test_data_dirsr?Fc Cst|rH|s t|n=t1}tdt||tdt|t|tdttj |tj ||Wdn1sCwYt|sPt||dur]|s]t j ||td|t|}|j|d|dS)Nz,Copying local tarfile to temporary storage..z%Deleting test dir to cleanup old dataz#Restoring local tarfile to test dirz0Extracting the `{}` test archive, please wait...)path)rrtempfileTemporaryDirectoryprintshutilcopy2rr4r@rbasenameurllibr urlretriever$tarfileopen extractallclose)test_dirtest_data_archiveurl local_datatemp_dirtarrrrextract_data_from_tars(         rSreturnc CsHz ddlm}WdSty#}ztj|ddWYd}~dSd}~ww)Nrk2)Tzk2 is appropriate.T)exc_info)Fz6k2 is not available or does not meet the requirements.)nemo.core.utils.k2_guardrV Exceptionlogging exception)rVerrrk2_is_appropriates r]cCsF|ds|Sddl}ddlm}|jr|jrdS|jr!dSdS)NrrU)Tzk2 supports CUDA.)FzFk2 does not support CUDA. Consider using a k2 build with CUDA support.)Fz'k2 needs CUDA to be available in torch.)torchrXrVcuda is_available with_cuda)r]r^rVrrrk2_cuda_is_enableds  rbc Cs|dd|dd|ddtttt}ttttt}zt|}Wnd}Y|jjrH|dkr?t d |n t d t||d}|jjsz t t}tj|}Wn|dkrkt d |n t d |||YdSY|}t|d }||krt d tt t||||jjd nt d t||n t|||jjd|jjdurddlm}t d|jj|j|jjddSdS)a Initial configuration of conftest. The function checks if test_data.tar.gz is present in tests/.data. If so, compares its size with github's test_data.tar.gz. If file absent or sizes not equal, function downloads the archive from github and unpacks it. markerszErun_only_on(device): runs the test only on a given device [CPU | GPU]z?with_downloads: runs the test using data present in tests/.dataz&nightly: runs the nightly test for QA.z+Test data `{}` is not present in the systemzAUsing the local `{}` test archive ({}B) found in the `{}` folder.NzBTest data not present in the system and cannot access the '{}' URLzOCannot access the '{}' URL, using the test data ({}B) found in the `{}` folder.zContent-Lengthz;Downloading the `{}` test archive from `{}`, please wait...)rOrPz9A valid `{}` test archive ({}B) found in the `{}` folder.)rPr) numba_utilszSetting numba compat :)strict)addinivalue_linerrr<r=__TEST_DATA_FILENAMEroptionuse_local_test_datar"exitr$rC__TEST_DATA_URLrGrurlopeninfointrSrelax_numba_compatnemo.core.utilsreset_numba_compat_strictness) rrMrNtest_data_local_sizerOumetatest_data_remote_sizererrrpytest_configuresz    rw)NF)(rZr4os.pathrDrIrAurllib.requestrGrrrrrpathlibrrtypingr r"nemo.utils.metaclassesr rhrlr=rfixturerr&r*r-r/r2r9r?rSboolstrr]rbrwrrrrsP      !