o Ni ;@spdZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mmZddlmZdZdZd Zd Zd Zd Zd ZdZdZdZdZgdZda ddZ!ddZ"ddZ#ddZ$dd!d"Z'ddeeefd#d$Z(d%d&Z)d'd(Z*d)d*Z+efd+d,Z,d-d.Z-d>d/d0Z.d1d2Z/d?d4d5Z0d6d7Z1d8d9Z2d:d;Z3dS)@z(Utilities for generating the C4 dataset.)absolute_import)division)print_functionN)loggingzWARC/1.0zWARC-Target-URI:z WARC-Date:z Content-Type:zContent-Length:)WARCzCONTENT-zContent-i).?!"z...)z terms of usezprivacy policyz cookie policyz uses cookieszuse of cookiesz use cookiescsdfdd }|S)Ncs tjjjjj||dSN)tfdscore lazy_imports apache_beammetricsMetricscounterinc)ramt namespaceU/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/text/c4_utils.pycounter_inc_fn9s z*get_counter_inc_fn..counter_inc_fn)r r)rrrrrget_counter_inc_fn8srcsfdd}|S)Ncs2|\}}tttj|dd}|S)Nutf-8)inthashlibmd5tfcompatas_textencode hexdigest)elurl_val predicate_fnrr filter_fn@s z+get_hashed_url_filter_fn..filter_fnr)r-r.rr,rget_hashed_url_filter_fn?s r/cCsLtjjj}t|d|jdWdS1swYdS)z)Returns a sentence tokenization function.punktz$nltk:tokenizers/punkt/english.pickleN) rrrnltk threadingLockdownloaddataload)r1rrr_load_sentence_tokenizerHs    $r7cCs tstatttj|Sr)_SENTENCE_TOKENIZERr7listtokenizer#r$r%)textrrr_get_sentencesSsr<Fcs2g}|D]}|fddt|Dq|S)Ncsg|] }r |n|qSrlower).0sr=rr ]sz*_get_sentences_by_line..) splitlinesappendr<)r;r> sentenceslinerr=r_get_sentences_by_lineZs   rFGz?c Cs|\}}|d}td}tjjj}d|j_z||}Wn|jj y,|dYdSw|s5|ddS|d} | j |krD|ddS| j |krV|d|d | j dS|d | j d S) zGReturns True iff text is in `language` with at least `min_probability`.r;z detected-langrzlangdetect-exceptionFzpage-filtered-nolangpredictionszpage-filtered-lowlangdetectconfzpage-filtered-ignoredlangzpage-filtered-ignoredlang-%szpage-emited-%sT) rrrr langdetectDetectorFactoryseed detect_langslang_detect_exceptionLangDetectExceptionproblang) pagelanguagemin_probability unused_urlfeaturesr;rrH predictionsbest_predictionrrr is_languagecs0   rWcCs>td}|rtdd|pg}nd}tjt||dS)zDReturns `clean_page` with pre-compiled badword and citation regexes.z$\[\d*\]|\[edit\]|\[citation needed\]z[^a-z]({})[^a-z]|N)citation_regexbadwords_regex)recompileformatjoin functoolspartial clean_page)badwordsrYrZrrrget_clean_page_fns rcc#s|\}}|d} |std}| } g} d} fdd} | D]}|}| |r.|dq|d|}|tr>|trC|dqt||krP|d q| d vr_|d d Sd vrh|dqd|vrs|dd St fddt Dr|dq|r| }|d ur|dd S| tt |7} | ||dq| |kr|dd S|dd| |d<||fVd S)aBCleans a CommonCrawl page, yielding nothing if it should be skipped. Cleaning removes lines with no end marks or with too few words. After line filtering, pages are filtered out if they have too few sentences based on a simple count of end marks. Args: url_and_features: tuple(string, dict), the url and features of the page. citation_regex: Regex to use for finding Wikipedia-like citations to filter. badwords_regex: Regex to use for finding badwords. Default None, which means don't apply badwords filtering. counter_inc_fn: function, a function taking the name of a counter to be incremented and the (optional) amount. Defaults to a beam Metric counter. min_words_per_line: int, the minimum number of words a line needs to not be removed. min_num_sentences: int, the minimum number of sentences a page needs to not be skipped. max_word_length: int, the maximum number of characters allowed in a word. Lines containing a word with too many characters are removed. Yields: The url and cleaned text for the page. r;z clean-pagercs$|D] }t|krdSqdS)NTF)splitlen)rEword)max_word_lengthrrline_has_too_long_words  z*clean_page..line_has_too_long_wordzlines-with-too-long-wordzlines-no-endmarkzlines-too-shortz lorem ipsumzfiltered-page-loremipsumN javascriptzlines-javascript{zfiltered-page-squigglybracketc3s|]}|vVqdSrr)r?p) line_lowerrr szclean_page..z lines-policyzfiltered-page-badwordz lines-validzfiltered-page-toofewsentenceszemitted-clean-pages )rrBstripsubendswith _END_MARKS _ELLIPSISrerdr>any_POLICY_SUBSTRINGSsearchr<rCr^)url_and_featuresrYrZrmin_words_per_linemin_num_sentencesrgr)rTr;lines valid_lines num_sentencesrhrEbadwords_foundr)rmrgrras`     racCs0t}|tj|d| SNr) r!r"updater#r$r%r&rpr>r')rEmrrr _hash_lines rccs4|\}}|d}|dD] }t||fVqdS)z-Emits url to all (lower-cased, hashed) lines.r;roN)rdr)rPr)rTr;rErrr_emit_url_to_liness rccsV|\}}t|}t|ddd}|D] }||kr||fVq|dt|dddS)z'Emits (hashed) line to all but one url.cSsttj|dSr)r!r"r#r$r%r&r')xrrrsz$_emit_line_to_urls..)keyzemitted-line-duplicater )rN)r9minre)r(rrEurlsskip_urlr)rrr_emit_line_to_urlss rccs|\}}|d}t|dksJdt||f|d}|d}t|d}g}t} |dD]} t| } | |vr@|dq1| | vrN|| | | q1d|} tt| |krb|d d S|} | | d<|| fVd S) a<Removes matching lines from the page. Process the result of a join containing a single value for 'features' and zero or more values for 'lines'. Each value in 'lines' is a lower-cased, hashed line. If a line has fewer sentences than `max_window_size`, the full line is compared for a match. Args: el: `(string, {'features': features_dict, 'lines': [string]})`, element containing the result of a join on key with both the page text and lower-cased, hashed lines to remove. counter_inc_fn: function, a function taking the name of a counter to be incremented and the (optional) amount. min_num_sentences: int, the minimum number of sentences a page needs to not be skipped. Yields: url: The URL of the page. features: The page features with lines removed from text. rTr zInvalid page count (%d) for %srr;r{rozfiltered-lines-duplicatezfiltered-doc-toofewsentencesN) resetrdrrCaddr^r<copy)r(rrzr) join_valuesrTr;lines_to_remove new_lines hashed_linesrE hashed_linenew_text new_featuresrrr_remove_lines_from_text s4     rcCsbtjjj}td}||tBd|?B|jt|dB}||dd| ?B|jt |dB}|S)z8Utility to remove duplicate lines across text documents.z dedupe-linesgroup_sentences)r)rTr{group_features_and_lines_by_url) rrrrrFlatMapr GroupByKeyr CoGroupByKeyr)pagesbeamrr final_docsrrrremove_duplicate_text;s*     rc #std|s tddtjj|d}tj|d}dddddfdd}tj |d d D]f}| }|sFq=|t kr\|}|rR|Vd gd d d | t rk|tt d | trz|ttd | tr|ttd | tr|ttd | trq=|q=|}|r|VWdn1swYWddSWddS1swYdS) z%Split a WET file into separate pages.zSplitting file: %szsplit-wet-filezwet-filerb)fileobjNcss dur dsdurdsdurds(dur(ds2dur2drGrGddd fSdS) z"Generate a (url, {features}) page.Nzpage-filtered-nourlzpage-filtered-nocontentzpage-nocontenttypezpage-nocontentlenzpage-notimestampz page-emittedro)r;z content-typezcontent-length timestampr))r^rcontent content_len content_typerrr)rr_maybe_get_pagebs(     z'split_wet_file.._maybe_get_pager)encodingri)rinforr#iogfileGFilegzipGzipFile TextIOWrapperrp_PAGE_DELIMITER startswith_URL_KEYre _URL_DATE _CONTENT_TYPE _CONTENT_LEN_METADATA_PREFIXESrC) wet_file_pathrfgrrErPrrrsplit_wet_fileSsX       PrcCsHtd}|\}}d}d}|D]}|d7}q|d|d|d||fS)z(Returns the first value for a given URL.z dedupe-urlsrNr zfiltered-url-duplicatez unique-url)r)r(rr)valscntvrrr dedupe_urlss r1AcCs8td}|\}}t|d|kr|ddS|ddS)z*Returns False iff page's text is too long.zis-valid-lengthr;zfiltered-page-contenttoolongFz valid-lengthTrre)r( max_lengthrr*rPrrris_valid_lengthsrcCsxtd}|\}}tjjj|}|jd|j}||vr"|ddS||}t|t r6|j |vr6|ddS|ddS)z4Returns False iff page's (sub)domain is not allowed.zis-realnews-domainr zfiltered-url-invaliddomainFzfiltered-url-invalidsubdomainzrealnews-domainT) rrrr tldextractextractdomainsuffix isinstancer9 subdomain)r(realnews_domainsrr)r*ext main_domainallowed_subdomainsrrris_realnews_domains  rccsltd}|\}}|d}|d}|s|ddS|s!|ddSt|dks)J|d||d fVdS) z3Yields only pages with a matching WebText-like URL.zfilter-by-webtextliker;webtextlike_urlszfiltered-url-notwebtextlikeNzmissing-webtextliker zfound-webtextlikerr)r(rr)rr; webtextlikerrrfilter_by_webtextlikesrcCsB|\}}tj|}tdd|}tdd|}|d}||fS)Nzhttps?:\/\/(www\.)?riz\?(utm_|ref|feed).*/)r#r$r%r[rqrstrip)r(r)r+rrr normalize_urls   r)F)rGr)r)4__doc__ __future__rrrr_rr!rr[r2abslrtensorflow.compat.v2r$v2r#tensorflow_datasets.public_api public_apirrrrrrr_MIN_WORDS_PER_LINE_MIN_NUM_SENTENCES_MAX_WORD_LENGTHrsrtrvr8rr/r7r<rFrWrcrarrrrrrrrrrrrrrrsf         \ 2 J