o Niޖ@sFdZddlmZddlmZddlmZddlZddlZddlZddlm Z ddl Z ddl m mZddlmZdZdZd Zd Zd Zd Zd ZdZdZdZdZdZedZ Gdddej!j"Z#Gdddej!j$Z%ddZ&ddZ'   d4ddZ(d d!Z)d5d"d#Z*d6d%d&Z+d'd(Z,d)d*Z- + , -d7d.d/Z. + 0 1 -d8d2d3Z/dS)9zCBIS-DDSM mammography dataset.)absolute_import)division)print_functionN)loggingaThe CBIS-DDSM (Curated Breast Imaging Subset of DDSM) is an updated and standardized version of the Digital Database for Screening Mammography (DDSM). The DDSM is a database of 2,620 scanned film mammography studies. It contains normal, benign, and malignant cases with verified pathology information. The default config is made of patches extracted from the original mammograms, following the description from http://arxiv.org/abs/1708.09427, in order to frame the task to solve in a traditional image classification setting. Because special software and libraries are needed to download and read the images contained in the dataset, TFDS assumes that the user has downloaded the original DCIM files and converted them to PNG. The following commands (or equivalent) should be used to generate the PNG files, in order to guarantee reproducible results: ``` find $DATASET_DCIM_DIR -name '*.dcm' | \ xargs -n1 -P8 -I{} bash -c 'f={}; dcmj2pnm $f | convert - ${f/.dcm/.png}' ``` aH@misc{CBIS_DDSM_Citation, doi = {10.7937/k9/tcia.2016.7o02s9cy}, url = {https://wiki.cancerimagingarchive.net/x/lZNXAQ}, author = {Sawyer-Lee, Rebecca and Gimenez, Francisco and Hoogi, Assaf and Rubin, Daniel}, title = {Curated Breast Imaging Subset of DDSM}, publisher = {The Cancer Imaging Archive}, year = {2016}, } @article{TCIA_Citation, author = { K. Clark and B. Vendt and K. Smith and J. Freymann and J. Kirby and P. Koppel and S. Moore and S. Phillips and D. Maffitt and M. Pringle and L. Tarbox and F. Prior }, title = {{The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository}}, journal = {Journal of Digital Imaging}, volume = {26}, month = {December}, year = {2013}, pages = {1045-1057}, } @article{DBLP:journals/corr/abs-1708-09427, author = {Li Shen}, title = {End-to-end Training for Whole Image Breast Cancer Diagnosis using An All Convolutional Design}, journal = {CoRR}, volume = {abs/1708.09427}, year = {2017}, url = {http://arxiv.org/abs/1708.09427}, archivePrefix = {arXiv}, eprint = {1708.09427}, timestamp = {Mon, 13 Aug 2018 16:48:35 +0200}, biburl = {https://dblp.org/rec/bib/journals/corr/abs-1708-09427}, bibsource = {dblp computer science bibliography, https://dblp.org} } zhttps://wiki.cancerimagingarchive.net/download/attachments/22516629/calc_case_description_test_set.csv?version=1&modificationDate=1506796343686&api=v2zhttps://wiki.cancerimagingarchive.net/download/attachments/22516629/calc_case_description_train_set.csv?version=1&modificationDate=1506796349666&api=v2zhttps://wiki.cancerimagingarchive.net/download/attachments/22516629/mass_case_description_test_set.csv?version=1&modificationDate=1506796343175&api=v2zhttps://wiki.cancerimagingarchive.net/download/attachments/22516629/mass_case_description_train_set.csv?version=1&modificationDate=1506796355038&api=v2)CCMLO)LEFTRIGHT)BENIGNBENIGN_WITHOUT_CALLBACK MALIGNANTzr^.*/(?P1.3.6.1.4.1.9590.100.1.2.[0-9.]+)/(?P1.3.6.1.4.1.9590.100.1.2.[0-9.]+)/(?P.+).dcm$cs"eZdZdZdfdd ZZS)CuratedBreastImagingDDSMConfigz+BuilderConfig for CuratedBreastImagingDDSM.Nc s8tjdd|d<tt|jdi|||_||_dS)Nz2.0.0z6New split API (https://tensorflow.org/datasets/splits)version)tfdscoreVersionsuperr__init__ image_size patch_size)selfrrkwargs __class__rf/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/image_classification/cbis_ddsm.pyr}s  z'CuratedBreastImagingDDSMConfig.__init__)NN)__name__ __module__ __qualname____doc__r __classcell__rrrrrzsrc@seZdZdZdZedddddedd d ed d d gZd dZddZddZ ddZ ddZ ddZ ddZ ddZddZdd Z   ! !d%d"d#Zd$S)&CuratedBreastImagingDDSMz&Curated Breast Imaging Subset of DDSM.z You can download the images from https://wiki.cancerimagingarchive.net/display/Public/CBIS-DDSM Please look at the source file (cbis_ddsm.py) to see the instructions on how to convert them into png (using dcmj2pnm). patcheszPatches containing both calsification and mass cases, plus pathces with no abnormalities. Designed as a traditional 5-class classification task.iir')name descriptionrr original-calczFOriginal images of the calcification cases compressed in lossless PNG.)r(r) original-massz=Original images of the mass cases compressed in lossless PNG.cCsP|j|j|jd}|jj|vrtd|jjtjj |t ||jjdt dS)N)r*r+r$&Builder config named {} not supported!z>https://wiki.cancerimagingarchive.net/display/Public/CBIS-DDSM)builderr)featureshomepagecitation) _get_features_original_calc_get_features_original_mass_get_features_patchesbuilder_configr( ValueErrorformatrr DatasetInfo _DESCRIPTION _CITATION)rfeatures_fn_maprrr_infos  zCuratedBreastImagingDDSM._infoc Cs~tjtjjtdtjjddtjjtdtjtjjdtj dtjjddtjjt dtjjt dtjjt dddS) N)namesNN)shaper)r?dtype) num_classesidmask assessment pathologysubtlety)rCbreastimageviewpatient abnormalities) rr.Text ClassLabel_BREAST_LABELSImage_IMAGE_VIEW_LABELSTensortfint32_ASSESSMENT_NUM_CLASSES_PATHOLOGY_LABELS_SUBTELTY_NUM_CLASSESrrrr_get_features_original_bases       z4CuratedBreastImagingDDSM._get_features_original_basec C||}|dtjjtjtj dddtjjtjtj ddddtj tj |d|d<tj |S)NrLimage_classificationzcbis_ddsm_calc_types.txt names_filez cbis_ddsm_calc_distributions.txt) calc_typecalc_distribution rYupdaterr.rNr get_tfds_pathospathjoinSequence FeaturesDictrr.rrrr1s,  z4CuratedBreastImagingDDSM._get_features_original_calcc CrZ)NrLr[zcbis_ddsm_mass_shapes.txtr\zcbis_ddsm_mass_margins.txt) mass_shape mass_marginsr`rhrrrr2s*  z4CuratedBreastImagingDDSM._get_features_original_massc CsBtjtjtjjdddtjjtjtj ddddS)Nr=jpeg)r?encoding_formatr[zcbis_ddsm_patch_labels.txtr\rCrIlabel) rr.rgrMrPrNrrbrcrdrerXrrrr3sz.CuratedBreastImagingDDSM._get_features_patchescCs>|jjdvr ||S|jjdkr||Std|jj)N)r*r+r$r,)r4r(_split_generators_original_split_generators_patchesr5r6)r dl_managerrrr_split_generatorss    z*CuratedBreastImagingDDSM._split_generatorscCs|jjdkr t}t}n|jjdkrt}t}n td|jj||d}||}t |j |}t j j t jj|j|dddt j j t jj|j|dddgS) Nr*r+r,)testtrainT) generate_fn patients_datayield_from_train_csvr( gen_kwargsF)r4r(_CALC_TEST_CSV_URL_CALC_TRAIN_CSV_URL_MASS_TEST_CSV_URL_MASS_TRAIN_CSV_URLr5r6download_and_extract_load_csv_files manual_dirrrSplitGeneratorSplitTRAIN_generate_examples_originalTEST)rrqtest_url train_url resourcesresource_pathsrvrrrros6     z3CuratedBreastImagingDDSM._split_generators_originalc Csttttd}dd|D}||}t|j|}t|\}}}t ||} t ||} t ||} t j j t j j|j| |jj|jjddt j j t j j|j| |jj|jjddt j j t j j|j| |jj|jjddgS)N)z calc-testz calc-trainz mass-testz mass-traincSs(i|]\}}|tjj|tjjjdqS))urlextract_method)rdownloadResource ExtractMethod NO_EXTRACT).0keyrrrr -s  zFCuratedBreastImagingDDSM._split_generators_patches..)rurvrrrx)rzr{r|r}itemsr~rr_split_patients_select_patients_datarrrrr_generate_examples_patchesr4rrr VALIDATION) rrqresources_urlsrrrv patients_testpatients_trainpatients_validpatients_data_testpatients_data_trainpatients_data_validrrrrp&sR       z2CuratedBreastImagingDDSM._split_generators_patchescKs|di|S)zYields examples.Nrr)rrurrrr_generate_examplesjsz+CuratedBreastImagingDDSM._generate_examplesc #sfdd}t|D]4\}}t|D])\}}||r@|d|d|d|d|ddd |d Dd }|d|fVqq dS) Ncsr|ddkS|ddkS)Ncsv_keyrtrsr)examplerwrr_include_example_in_splitps  zWCuratedBreastImagingDDSM._generate_examples_original.._include_example_in_splitrCrKrIrJrHcSsg|] }dd|DqS)cSsi|] \}}|dvr||qS))typerrkvrrrrszSCuratedBreastImagingDDSM._generate_examples_original...rr abnormalityrrr s  zHCuratedBreastImagingDDSM._generate_examples_original..rL)rCrKrIrJrHrL)sortedr)rrvrwr_patient_examplesrrecordrrrrns$  z4CuratedBreastImagingDDSM._generate_examples_original c #stjdt|t|D]\}}t|D]\}}t|d|dfdd|dD} dd| D} t|d| | D]\\} } } | dd rPd }nd }| d d kr]|d }n| d dkrh|d}nt d| d t t | d| | ||D]\}}d|d| d|f}|tj |dd|d}||fVq}qAt t |d| | ||D]\}}d|d|f}|tj |dddd}||fVqqqdS)NE#rIrcsg|] }t|djdqS)rDr) _read_imager?rrIrrrszGCuratedBreastImagingDDSM._generate_examples_patches..rLcSsg|] }t|dkqS)rnpsum)rrDrrrrrFr BENINGrcalc_CALCIFICATIONmass_MASSzUnknown abnormality type: %rrDz%s/abnorm_%s/patch_%drC)axisrmz%s/background_%d BACKGROUND)rrandomseedlenrrrzip startswithr5 enumerate_sample_positive_patches expand_dims_sample_negative_patches)rrvrr$num_positive_patches_per_abnormality num_background_patches_per_imagerrrabnormalities_masksabnormalities_areasrabnormality_maskabnormality_areabenign_or_malignantrnrpatchpatch_idrid_rrrrsl              z3CuratedBreastImagingDDSM._generate_examples_patchesN)r%r&rr)rrr r!MANUAL_DOWNLOAD_INSTRUCTIONSrBUILDER_CONFIGSr;rYr1r2r3rrrorprrrrrrrr#sB  "Dr#c Cs(i}t|D]\}}tjj|d}t|}t|dD]\}}dd|D}t d|d} dD]1} ||  dd || <t j j|g|| d R|| <tjj|| shtd |||| fq7|d } |d } t|d} t| }t| }| j|jkr| | } } n| j|jkrtd||| j|j|jgq t|d| |d|d|dd}d|vrd|vrd|d<|d|d<|d|d<nd|vrd|vrd|d<|d|d<|d|d<ntd| |d |d!|d|d"|g|d#}t||q Wd$n 1s wYq|S)%aGLoad the ground-truth data from the given dictionary of CSV files. Args: manual_dir: Path of the directory containing the images. dictionary_of_csv_files: Dictionary containing the key and filepath of each CSV file to load. Returns: A dictionary containing the ground-truth loaded from the CSV files. rcSsi|] \}}||qSr)striprrrrrrz#_load_csv_files..z\g/\gimage file path)rROI mask file pathcropped image file pathz.dcmz.png/zCError processing line %d from csv file %s: Image %r does not exist!rrzError processing line %d from csv file %s: No suitable mask for the given image (expected size: %r, candidate sizes: %r). This abnormality will NOT be included in the dataset.zabnormality idrErFrGrBz calc typezcalc distributionrrr^r_z mass shapez mass marginsrrirjz%CSV file is missing required columns.zleft or right breast patient_idz image view)rCrHrKrIrJrLrN)rrrSiogfileGFilecsv DictReaderr _DCIM_REGEXsubreplacercrdresplitexistsr5rr?rerrorint_append_example_to_data)rdictionary_of_csv_filesdatarcsv_pathf csv_readerirow example_idrmask_file_pathcrop_file_path full_image mask_image crop_imagerrrrrrsv  "         ErcCs|d}|d}||vrq|||dvre||||dks J||||dks,J|d|||dks:J|d|||dksHJ|d|||dksVJ|||d|ddS|||d|d<dS|d|i||d<dS)z0Append the given example to the data dictionary.rCrKrHrIrJrLN)extend)rrrrrrrrsr333333?{Gz?(\µ?c Cs|dkr |dkr |dksJt|||ddksJtt|}tjjddtj|t|t |}|d|}|t|t |}|||}||d} t |t |t | fS)zJSplit the patients in the data dictionary into test, train and valid sets.r?g& .>r)rN) rabsrlistkeysrrshufflerrset) r test_fractiontrain_fractionvalid_fractionall_patient_ids cutoff_testr cutoff_trainrrrrrr1s    rcsfdd|DS)Ncsi|] \}}|vr||qSrrr patient_idsrrrDsz)_select_patients_data..r)rr rrrrCsrcCstjjj}tjj|d/}|jt j | t j d|j d}|r3|||d|df}|j|ks3J|WdS1s?wYdS)zERead an image and optionally resize it (size must be: height, width).rb)r@)flagsr>rN)rr lazy_importscv2rSrrrimdecoder fromstringreaduint8IMREAD_GRAYSCALEresizer?)filepathrr rrIrrrrGs $r皙?cs~tjjjt||}||dj\}}|j j \}}}fdd|D}t |} t |||djS)z6Get the binary mask of the breast region of the image.cg|]}|qSr contourArearcontr rrr[z$_get_breast_mask..)rrr r rmax threshold THRESH_BINARY findContours RETR_LISTCHAIN_APPROX_SIMPLErargmax drawContours zeros_likeFILLED)rImin_breast_color_thresholdrr image_binarycontourscontours_areasbiggest_contour_idxrrr_get_breast_maskTs    r-csFtjjj|jj\}}}fdd|D}t|}||S)Ncrrrrrrrrfrz&_get_roi_from_mask..) rrr r r!r"r#rr$)rDrr*r+r,rrr_get_roi_from_maskbs    r.c Cs||d|d}t||D].\}}t||||d|||dfdk} | |} | |} | |ks8| |kr;dSq dS)aFReturn True if the given patch overlaps significantly with any abnormality. Given a patch and a single abnormality, the overlap between the two is significant if, and only if, the relative area of the intersection of the two w.r.t. the area of the patch is above `min_overlap_threshold` OR the area of the intersection w.r.t. the total abnormality area is above `min_overlap_threshold`. Args: y: Top-most coordinate of the patch. x: Left-most coordinate of the patch. patch_size: Tuple with (height, width) of the patch. abnormalities_masks: List with the binary mask of each abnormality. abnormalities_areas: List with the total area of each abnormality. min_overlap_threshold: Returns: Returns True if the above condition is met for any of the given abnormalities, or False otherwise. rr>TF)rrr) yxrrrmin_overlap_threshold patch_area abnorm_mask abnorm_areaabnorm_in_patch_areaabnorm_in_patch_wrt_patchabnorm_in_patch_wrt_abnormrrr/_patch_overlaps_any_abnormality_above_thresholdks&r8r?dccstjjj}t|} || \} } } } d}|dkrt| |dt| |d}}t| |dt| |d}}| td||}| td||}| | t||}| | t||}t|d}t|d}tt||j d|dd|}tt||j d|dd|}|}||d||d|krt d||||||d||d}t |D]=}t j||d}t j||d}t||||g|g|r|d7}||||d|||dfV||krdSq|d}t d|||dkstd ||||f) aSample random patches from the image overlapping with the given abnormality. The abnormal area of the patch with respect to either (a) the total area of the patch, or (b) the total area of the abnormality, must be at least `min_overlap_threshold` (i.e. 90% by default). After `max_number_of_trials_per_threshold` samples, if not enough patches meeting this requirement have been generated, the `min_overlap_threshold` is reduced by 5%. This procedure is repeated until min_overlap_threshold < 0.1 (which should not happen ever, if the dataset is correct). Args: image: Image to patch from. abnormality_filepath: Only used for logging. abnormality_mask: Binary mask of the abnormality in the image. abnormality_area: Precomputed area of the abnormality. patch_size: Size of the patch to extract. number_of_patches: Number of patches to sample around the abnormality ROI. min_overlap_threshold: Minimum relative area of the patch overlapping with the abnormality. max_number_of_trials_per_threshold: Maximum number of random samples to try before reducing the `min_overlap_threshold` by 5%. Yields: The patch cropped from the input image. r皙?r>rzThe sampling region for patches of size %r with min_overlap_threshold=%f contains less possible patches than max_number_of_trials_per_threshold=%d, in abnormality %sNffffff?zrOverlapping constraints relaxed to min_overlap_threshold=%f while sampling positive patches for the abnormality %szOnly %d positive patches of size %r could be sampled satisfying the current conditions (min. relative overlapping area = %f) for the abnormality %s)rrr r r. boundingRectrminrr?rdebugrangerrrandintr8r5)rIabnormality_filepathrrrnumber_of_patchesr1"max_number_of_trials_per_thresholdr abnormality_roiabnorm_xabnorm_yabnorm_wabnorm_hnumber_of_yielded_patchesmax_hmin_hmax_wmin_wmin_ymin_xmax_ymax_xeffective_range_sizerpatch_ypatch_xrrrrsf "   ""   -r?ffffff?c #sXtjjj} t|fdd} t} | | \} } }}d}dkr!|dkr!t|dt|d}}t|dt|d}}| t d|}| t d|}| |t |}| |t |}t|d}t|d}tt||j ddd|}tt||j ddd|}|}||d||d|krt d||||d||d}t |D]A}tj||d}tj||d}| ||rt|||||s|d7}|||d||dfV||krd Sqƈd |d }t d ||dkr!|dks+td |||f)aSample background patches from the image. The relative area of breast tissue in the patch must be, at least, `min_breast_overlap_threshold` of the total patch area. This is to prevent too easy negative examples. Similarly, the relative area of the abnormal tissue in the patch must be, at most, `max_abnorm_overlap_threshold` The relative area of the patch must overlap with the breast tissue with, at least, `min_breast_overlap_threshold` (relative) pixels. In addition, it must also overlap with abnormal tissue with, at most, `max_abnorm_overlap_threshold` (relative) pixels. Args: image: Image to patch from. image_filepath: Only used for logging. abnormalities_masks: List of binary mask of each abnormality in the image. abnormalities_areas: List of precomputed area of each abnormality. patch_size: Size of the patch to extract. number_of_patches: Number of negative patches to sample from the image. min_breast_overlap_threshold: Minimum (relative) number of breast pixels in the patch. max_abnorm_overlap_threshold: Maximum (relative) number of abnormal pixels in the patch. max_number_of_trials_per_threshold: Maximum number of random samples to try before reducing the `min_breast_overlap_threshold` by 5% and increasing the `max_abnorm_overlap_threshold` by 5%. Yields: The patch cropped from the input image. csF||d||df}t|dkddkS)z7Return True if the patch contains enough breast pixels.rr>r)r/r0breast_in_patch breast_maskmin_breast_overlap_thresholdrrr$patch_overlapping_breast_is_feasible$s $ zF_sample_negative_patches..patch_overlapping_breast_is_feasiblerr;r9r>rzThe sampling region for negative patches of size %r with min_breast_overlap_threshold=%f contains less possible patches than max_number_of_trials_per_threshold=%d, in mammography %sNr<g?zOverlapping constraints relaxed to min_breast_overlap_threshold=%f and max_abnorm_overlap_threshold=%f while sampling negative patches for the mammography %szOnly %d negative patches of size %r could be sampled satisfying the current conditions (min. relative overlapping area with breast = %f, max. relative overlapping area with abnormalities = %f) for the mammography %s)rrr r r-r.r=rr>rr?rr?r@rrrAr8r5)rIimage_filepathrrrrCr[max_abnorm_overlap_thresholdrDr r\ breast_roibreast_xbreast_ybreast_wbreast_hrJrKrLrMrNrOrPrQrRrSrrTrUrrYrrsx )    ""    0r)rrr)N)r)rr9r:)rrVrWr:)0r! __future__rrrrrcreabslrnumpyrtensorflow.compat.v2compatv2rStensorflow_datasets.public_api public_apirr8r9rzr{r|r}rQrO_BREAST_DENSITY_NUM_CLASSESrVrUrWcompilerr BuilderConfigrGeneratorBasedBuilderr#rrrrrr-r.r8rrrrrrsd     ' EV    ) m