o TiG@sddlmZmZddlZddlmZmZmZm Z m Z ddl m Z e de ddgZ eGdddZd d Zed krPe eZedZeejejejdSdS) ) dataclassfieldN)DatasetFeaturesImageSequenceValue)HfArgumentParserstring)contentrolec@sZeZdZUdZedddidZeed<edddidZe ed <ed dd idZ e ed <d S)ScriptArgumentsa Arguments for the script. Args: test_size (`float`, *optional*, defaults to `0.1`): Fraction of the dataset to include in the test split. push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the dataset to the Hugging Face Hub. repo_id (`str`, *optional*, defaults to `"trl-internal-testing/zen"`): Hugging Face repository ID to push the dataset to. g?helpz5Fraction of the dataset to include in the test split.)defaultmetadata test_sizeFz4Whether to push the dataset to the Hugging Face Hub. push_to_hubztrl-internal-testing/zen-imagez2Hugging Face repository ID to push the dataset to.repo_idN) __name__ __module__ __qualname____doc__rrfloat__annotations__rboolrstrrrV/home/ubuntu/.local/lib/python3.10/site-packages/scripts/generate_zen_image_dataset.pyr s  r cCs|tjjdddd}gddd|Dd}tj|ttd tdd }|j|d d }|r3|j |d dtjjdddd}gddd|Dd}tj|ttd tdd }|j|d d }|rf|j |ddtjjdddd}gdgddd|Dd}tj|ttd td tdd }|j|d d }|r|j |ddtjjdddd}gdgdgddd|Dd}tj|ttd td td tdd }|j|d d }|r|j |ddtjjdddd}gdgddd|Dd}tj|ttd td tdd } td dt td ddd t td!ddd d"| j|d d } |r/| j |d#dtjjdddd}gdgd$gd%d&d|Dd'}tj|ttd td td!td'd } | j|d d } |ro| j |d(dtjjdddd}gd)d*d+ggd,d-d.gd/d0gd1d2gd3gd4gd5d6gd7d8gd9d:gd;ggd<d=d>gd?gd@gdAgdBgdCdDgdEggd dFggdGd dFgdFdFgdFd gdFgd gdFd gd d gd d gdFggdHdFdFgd gdFgd gd gdFdFgd ggdId|DdJ}tj|ttd t td t td!tdJd } | j|d d } |r| j |dKdtjjdddd}dLdMdNdOdPdNgdLdQdNdOdRdNgdLdSdNdOdTdNgdLdUdNdOdVdNgdLdWdNdOdXdNgdLdYdNdOdZdNgdLd[dNdOd\dNgdLd]dNdOd^dNgdLd_dNdOd`dNgdLdadNdOdbdNgdLdcdNdOdddNgdLdedNdOdfdNgdLdgdNdOdhdNgdLdidNdOdjdNgdLdkdNdOdldNgdLdmdNdOdndNgdLdodNdOdpdNgdLdqdNdOdrdNgdLdsdNdOdtdNggdud|Ddv}tj|tt tdvd } | j|d d } |r| j |dwdtjjdddd}dLdMdNgdLdQdNgdLdSdNgdLdUdNgdLdWdNgdLdYdNgdLd[dNgdLd]dNgdLd_dNgdLdadNgdLdcdNgdLdedNgdLdgdNgdLdidNgdLdkdNgdLdmdNgdLdodNgdLdqdNgdLdsdNggdxd|Dd}tj|tt tdd } | j|d d } |r{| j |dydtjjdddd}dLdMdNgdLdQdNgdLdSdNgdLdUdNgdLdWdNgdLdYdNgdLd[dNgdLd]dNgdLd_dNgdLdadNgdLdcdNgdLdedNgdLdgdNgdLdidNgdLdkdNgdLdmdNgdLdodNgdLdqdNgdLdsdNggdOdPdNgdOdRdNgdOdTdNgdOdVdNgdOdXdNgdOdZdNgdOd\dNgdOd^dNgdOd`dNgdOdbdNgdOdddNgdOdfdNgdOdhdNgdOdjdNgdOdldNgdOdndNgdOdpdNgdOdrdNgdOdtdNggdzd|Dd}tj|tt t tdd }|j|d d }|rk|j |d{dtjjdddd}dLdMdNgdLdQdNgdLdSdNgdLdUdNgdLdWdNgdLdYdNgdLd[dNgdLd]dNgdLd_dNgdLdadNgdLdcdNgdLdedNgdLdgdNgdLdidNgdLdkdNgdLdmdNgdLdodNgdLdqdNgdLdsdNggdOdPdNgdOdRdNgdOdTdNgdOdVdNgdOdXdNgdOdZdNgdOd\dNgdOd^dNgdOd`dNgdOdbdNgdOdddNgdOdfdNgdOdhdNgdOdjdNgdOdldNgdOdndNgdOdpdNgdOdrdNgdOdtdNggdOd|dNgdOd}dNgdOd~dNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOddNgdOdpdNgdOddNggdd|Dd}tj|tt t t tdd }|j|d d }|r|j |ddtjjdddd}dLdMdNdOdPdNgdLdQdNdOdRdNgdLdSdNdOdTdNgdLdUdNdOdVdNgdLdWdNdOdXdNgdLdYdNdOdZdNgdLd[dNdOd\dNgdLd]dNdOd^dNgdLd_dNdOd`dNgdLdadNdOdbdNgdLdcdNdOdddNgdLdedNdOdfdNgdLdgdNdOdhdNgdLdidNdOdjdNgdLdkdNdOdldNgdLdmdNdOdndNgdLdodNdOdpdNgdLdqdNdOdrdNgdLdsdNdOdtdNggdLdMdNdOd|dNgdLdQdNdOd}dNgdLdSdNdOd~dNgdLdUdNdOddNgdLdWdNdOddNgdLdYdNdOddNgdLd[dNdOddNgdLd]dNdOddNgdLd_dNdOddNgdLdadNdOddNgdLdcdNdOddNgdLdedNdOddNgdLdgdNdOddNgdLdidNdOddNgdLdkdNdOddNgdLdmdNdOddNgdLdodNdOddNgdLdqdNdOdpdNgdLdsdNdOddNggdd|Dd}tj|tt t tdd }|j|d d }|rD|j |ddtjjdddd}dLdMdNgdLdQdNgdLdSdNgdLdUdNgdLdWdNgdLdYdNgdLd[dNgdLd]dNgdLd_dNgdLdadNgdLdcdNgdLdedNgdLdgdNgdLdidNgdLdkdNgdLdmdNgdLdodNgdLdqdNgdLdsdNggdOdPdNgdOdRdNgdOdTdNgdOddNgdOdXdNgdOdZdNgdOd\dNgdOddNgdOd`dNgdOddNgdOdddNgdOddNgdOdhdNgdOddNgdOddNgdOdndNgdOdpdNgdOdrdNgdOdtdNgggddd|Dd'}tj|tt t td!td'd }|j|d d }|r<|j |dddSdS)N @))size)zBeautiful is better than ugly.z!Explicit is better than implicit.zSimple is better than complex.z#Complex is better than complicated.zFlat is better than nested.zSparse is better than dense.zReadability counts.z7Special cases aren't special enough to break the rules.z#Although practicality beats purity.z"Errors should never pass silently.zUnless explicitly silenced.z9In the face of ambiguity, refuse the temptation to guess.zEThere should be one-- and preferably only one --obvious way to do it.zBAlthough that way may not be obvious at first unless you're Dutch.Now is better than never.z0Although never is often better than *right* now.z:If the implementation is hard to explain, it's a bad idea.z@If the implementation is easy to explain, it may be a good idea.z@Namespaces are one honking great idea -- let's do more of those!cS0g|]\}}tjjdd||dfdtjqSggo@)lowhighr"nprandomuniformastypeuint8.0hwrrr N0zmain..)textimager )featuresF)rshufflestandard_language_modeling) config_name)Beautiful is better thanz Explicit iszSimple is betterComplexFlat is better thanzSparse is better ReadabilityzSpecial cases aren't specialAlthough practicality beatszErrors should neverzUnless explicitly In the face of ambiguity, refusez$There should be one-- and preferablyz;Although that way may not be obvious at first unless you'rezNow iszAlthough never is oftenz)If the implementation is hard to explain,zIf the implementation is easyz Namespaces are one honking greatcSr$r%r)r/rrrr3lr4)promptr6standard_prompt_only) ugly.z better than implicit.z than complex. is better than complicated. nested.z than dense. counts.z enough to break the rules. purity. pass silently.z silenced. the temptation to guess.! only one --obvious way to do it.z Dutch. better than never.z better than *right* now. it's a bad idea.z# to explain, it may be a good idea.z idea -- let's do more of those!cSr$r%r)r/rrrr3r4)rA completionr6standard_prompt_completion)z the moon. worse than nothing. than a long vacation.z is always the answer.z chocolate. without any context.z is optional. enough to become unicorns.z reality.z pass their driving test. forgotten.z the opportunity to laugh.z two or more confusing methods. a time traveler.z never better. not even a possibility.z it's clearly the best choice. it's probably magic. watermelon -- let's plant some!cSr$r%r)r/rrrr3r4)rAchosenrejectedr6standard_preference)z"Beautiful is better than the moon.zExplicit is worse than nothing.z&Simple is better than a long vacation.zComplex is always the answer.zFlat is better than chocolate.z%Sparse is better without any context.zReadability is optional.z7Special cases aren't special enough to become unicorns.z$Although practicality beats reality.z,Errors should never pass their driving test.zUnless explicitly forgotten.z:In the face of ambiguity, refuse the opportunity to laugh.zCThere should be one-- and preferably two or more confusing methods.zLAlthough that way may not be obvious at first unless you're a time traveler.zNow is never better.z/Although never is often not even a possibility.zGIf the implementation is hard to explain, it's clearly the best choice.z2If the implementation is easy it's probably magic.z@Namespaces are one honking great watermelon -- let's plant some!cSr$r%r)r/rrrr3r4)rXrYr6)dtype)featurelengthr)rA completionslabels#standard_implicit_prompt_preference)rCrOrPrDrErQrFrRrGrHrSrIrJrTrKrUrLrVrW)TFFTTFTFTTFTTFTFTFFcSr$r%r)r/rrrr3Or4)rArMlabelr6standard_unpaired_preference)r;zExplicit is better thanzSimple is better thanzComplex is better thanr=zSparse is better thanzReadability countsz#Special cases aren't special enoughr?zErrors should never passr@z0There should be one-- and preferably only one --zAlthough that way may not bezNow is better thanzNever is often better thanz.If the implementation is hard to explain, it'sz,If the implementation is easy to explain, itzNamespaces are onez-Although practicality sometimes beats purity,z, let me think...rC)z , of course,z implicit.z because clarity matters.z... let's keep it basic,z complex.z when needed,z complicated.z in terms of structure,rEz... especially for readability.z especially when others read it.z , unless...z they follow the rules.z some theoretical elegance,rGz silently,z unless explicitly silenced.rI)z way to do it,z but sometimes it's not obvious.z3 especially when there's more than one possibility.z clear at first,z it will eventually emerge.z later.z problematic fixes.z% likely because it's too complicated.z might be a good design.z of those great ideas,z that solve many problems.z' the code should still aim for balance.T)FTF)TTFcSr$r%r)r/rrrr3r4)rAr_r`r6standard_stepwise_supervisionuserzWhat is better than ugly?)r r assistantz Beautiful.zWhat is better than implicit?z Explicit.zWhat is better than complex?zSimple.z What is better than complicated?zComplex.zWhat is better than nested?zFlat.zWhat is better than dense?zSparse.z What counts?z Readability.z,Are special cases enough to break the rules?z;No, special cases aren't special enough to break the rules.zWhat beats purity?z Practicality.z What should never pass silently?zErrors.zWhen can errors pass silently?zWhen explicitly silenced.z,What should you do in the face of ambiguity?zRefuse the temptation to guess.z'How many ways should there be to do it?zOne, and preferably only one.z-For whom may the way not be obvious at first?zDutch.zWhat is better than never?r#z!Is never better than *right* now?z Yes, often.z;What does it mean if the implementation is hard to explain?zIt means it's a bad idea.z;What does it mean if the implementation is easy to explain?zIt means it may be a good idea.zAny great ideas?z&Namespaces are one honking great idea.cSr$r%r)r/rrrr3r4)messagesr6 conversational_language_modelingcSr$r%r)r/rrrr3r4conversational_prompt_onlycSr$r%r)r/rrrr3r4 conversational_prompt_completionz Acceptable.z Explained.z Very complex.zVery complicated.z Circular.zHeavy.zLooking complicated.z9Yes, special cases are special enough to break the rules.zNothing.z Warnings.zNever.zGive up.zAs many as possible.zFrench.z Some day.z No, never.zIt means it's a good idea.z Recursion.cSr$r%r)r/rrrr3Mr4conversational_preferencecSr$r%r)r/rrrr3r4)conversational_implicit_prompt_preference)TTTFTTTFTFTFTFFTTTTcSr$r%r)r/rrrr3r4"conversational_unpaired_preference) r*r+randintr from_dictrrrtrain_test_splitrrMessage)rrrsizesdata"standard_language_modeling_datasetstandard_prompt_only_dataset"standard_prompt_completion_datasetstandard_preference_dataset+standard_implicit_prompt_preference_dataset$standard_unpaired_preference_dataset%standard_stepwise_supervision_dataset(conversational_language_modeling_dataset"conversational_prompt_only_dataset(conversational_prompt_completion_dataset!conversational_preference_dataset1conversational_implicit_prompt_preference_dataset*conversational_unpaired_preference_datasetrrrmain5s   "- (B "-. (. 0A                                                            -                                                          B -                                        .r__main__) dataclassesrrnumpyr*datasetsrrrrr transformersr rqr rrparserparse_args_into_dataclasses script_argsrrrrrrrs$