o Ni+@sdZddlmZddlmZddlmZddlZddlZddlZddlm Z ddl m Z ddl m Z d Zd Zd Zd Zd ZddZGddde jZedkrSe dSdS)zTests for c4_utils.)absolute_import)division)print_functionN)testing) lazy_imports)c4_utilsuCThis line has enough words and ends in punctuation, Dr. Roberts! Economic History | All Institutions | Open Access Articles | Digital Commons Network "Open Access. Powered by Scholars. Published by Universities." Digital Commons Network™/ Social and Behavioral Sciences... Too few words. You need to enable javascript in your browser in order to see this page. You have JavaScript disabled and that means you can't see this page. Adam Roberts has a cookie policy: always eat them. Colin has a privacy policy: don't share people's secrets. You'd better follow our terms of use! zThis line has enough words and ends in punctuation, Dr. Roberts! "Open Access. Powered by Scholars. Published by Universities."5793z text/plainz2019-04-24T09:23:58Zcs ttdfdd }|fS)Ncs||7<dSN)camtcountersr Z/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/text/c4_utils_test.pycounter_inc_fn6sz%_get_counters..counter_inc_fn)r ) collections defaultdictintrr rr _get_counters3s rc@s^eZdZdddZddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ dS) C4UtilsTestNcCsPt\}}tt|d|f|d}|t|d|sdn|dd}||fS)Nurl)url_and_featuresrr r)rlistrget_clean_page_fnassertLessEquallen)selffeaturesbadwordsrrresultsresultr r rrun_clean_page>s zC4UtilsTest.run_clean_pagec CsJ|ttttd\}}|t|d|dddddddt|dS)Ntextz content-typezcontent-length timestampr%r ) lines-validzlines-too-shortlines-no-endmarkzlines-javascript lines-policyemitted-clean-pages)r#EN_TEXTFAKE_CONTENT_TYPEFAKE_CONTENT_LENGTHFAKE_TIMESTAMP assertEqualEXPECTED_CLEAN_ENdict)rclean_enrr r rtest_clean_pageGs" zC4UtilsTest.test_clean_pagecCBd}||tttd\}}|d||dddt|dS)NziThis first line has one sentence. This line looks like it has three sentences...but it's actually just 1.r$r'r )r)zfiltered-page-toofewsentencesr#r.r/r0r1r3)rtext_with_toofewsentencesr4rr r rtest_clean_page_toofewsentencesYs z+C4UtilsTest.test_clean_page_toofewsentencescCr6)NzThis page starts out with some text. Everything looks good at first, since these are sentences. But then, all of a sudden, there's a bunch of code like the next block. fn foo(a) { bar = a + 10; }.r$r r()zfiltered-page-squigglybracketr)r7)rtext_that_is_actually_coder4rr r rtest_clean_page_squigglybrackeths z+C4UtilsTest.test_clean_page_squigglybracketcCs@d}||tttd\}}|d||ddit|dS)NaLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.r$zfiltered-page-loremipsumr r7)rlorem_ipsum_textr4rr r rtest_clean_page_loremipsumys z&C4UtilsTest.test_clean_page_loremipsumc Csd}gd}gd}ddddddddddddg}t|||D]1\}}}||}|j|tttd d gd \} } |rC|d| n||| d ||t| q"dS) NzThis page starts out with some text. Everything looks good at first, since these are sentences. But then, all of a sudden, there's a badword... or not? )z6I asked my friend for assistance polishing my cutlass.zCHe took the saddle and put it on his ass in preparation for travel.zdAss is one of several species of small, horse-like animals.Donkey is one synonym for the word "ass".)FTTTr )r)r,r()r)zfiltered-page-badwordr$ass)r r%)zipr#r.r/r0r1r3) r padding_textfinal_sentencesoutputs_should_be_noneexpected_countersfinal_sentenceoutput_should_be_noneexpected_counterr%outrr r rtest_clean_page_badwordssB  z$C4UtilsTest.test_clean_page_badwordscCsPd}d}dddd}||tttd\}}|||d||t|dS)NzThis page has some text. Some lines don't end with punctuation And some of these lines end with citations.[3] Or have requested citations[citation needed]. Or the option to edit.[edit] zxThis page has some text. And some of these lines end with citations. Or have requested citations. Or the option to edit.r(r )r)r*r,r$r%r7rr%expected_clean_textrDrHrr r rtest_clean_page_citationss z%C4UtilsTest.test_clean_page_citationscCsPd}d}dddd}||tttd\}}|||d||t|dS)NzThis page has with some text. So, that's good! But at the end it has some polciy lines. This line mentions the Terms of Use. This line should be okay. The privacy policy is mentioned in this line. Let's talk about the Cookie Policy now. zqThis page has with some text. So, that's good! But at the end it has some polciy lines. This line should be okay.r(r )r)r+r,r$r%r7rJr r rtest_clean_page_policys z"C4UtilsTest.test_clean_page_policycCsddlmm}tj}gd}ddg}|(}||dd|DB}t|}| || dd|DWddS1sCwYdS)Nr))url/1-0aFThis is a duplicated line. This is a unique line. This one comes first and so it stays. This one is duplicate within the page so the others are removed. Here is a sentence between the duplicates. This one is duplicate within the page so the others are removed. this One is Duplicate WITHIN the page so the others are removed. )zurl/2-1zThis is 2nd unique line. This one comes second so it is removed even though the capitalizaiton is different. this is a Duplicated line. )url/3-4zThis is a 3rd unique line. This is a duplicated line. This one comes third and so it is removed. But the page stays because there are still 3 sentences remaining.)zurl/4-4zThis is a 4th unique line. This is a duplicated line. This one comes third and so it is removed, and the page is too since there aren't enough sentences left.)rNzThis is a duplicated line. This is a unique line. This one comes first and so it stays. This one is duplicate within the page so the others are removed. Here is a sentence between the duplicates.)rOzThis is a 3rd unique line. This one comes third and so it is removed. But the page stays because there are still 3 sentences remaining.cSg|] \}}|d|ifqSr%r .0rr%r r r z:C4UtilsTest.test_remove_duplicate_text..cSrPrQr rRr r rrTrU) apache_beam.testing.utilrutilr apache_beamPipelineCreaterremove_duplicate_text assert_thatequal_to)rbeam_testing_utilbeaminput_urls_and_textexpected_urls_and_textpipelinepages deduped_pagesr r rtest_remove_duplicate_texts$   "z&C4UtilsTest.test_remove_duplicate_textcCsRtjrdSt\}}ttjtjt dd|d| ddddt |dS)Nc4zcc_0.warc.wet.gzrr r')zwet-filez page-emittedzpage-filtered-nourl) sixPY2rrrsplit_wet_fileospathjoinrfake_examples_dirr1r3)rrrr r rtest_split_wet_files  zC4UtilsTest.test_split_wet_filer ) __name__ __module__ __qualname__r#r5r9r;r=rIrLrMrernr r r rr<s  3 .r__main__)__doc__ __future__rrrrrjrgtensorflow_datasetsr)tensorflow_datasets.core.lazy_imports_librtensorflow_datasets.textrr-r2r/r.r0rTestCaserro test_mainr r r rs*        n