o Xi5@sdZddlmZddgZddlZddlZddlZddlZ e e Z dd d Z ddZGddde jjZGddde jjZdS)z>Pass for removing duplicated initializer tensors from a graph.) annotationsDeduplicateInitializersPass!DeduplicateHashedInitializersPassN initializerir.Value size_limitintreturnboolcCsd|s|rtd|jdS|j}|dur!td|jdS|j|kr0td|j|dSdS)z=Check if the initializer should be skipped for deduplication.zJSkipped deduplication of initializer '%s' as it is a graph input or outputTNzqSkipped deduplication of initializer '%s' as it has no constant value. The model may contain invalid initializerszDSkipped initializer '%s' as it exceeds the size limit of %d elementsF)is_graph_inputis_graph_outputloggerwarningname const_valuesizedebug)rr const_valrc/home/ubuntu/.local/lib/python3.10/site-packages/onnx_ir/passes/common/initializer_deduplication.py_should_skip_initializers* rcCs$|jrt|S|S)aStringTensor does not support tobytes. Use 'string_data' instead. However, 'string_data' yields a list of bytes which cannot be hashed, i.e., cannot be used to index into a dict. To generate keys for identifying tensors in initializer deduplication the following converts the list of bytes to an array of fixed-length strings which can be flattened into a bytes-string. This, together with the tensor shape, is sufficient for identifying tensors for deduplication, but it differs from the representation used for serializing tensors (that is string_data) by adding padding bytes so that each string occupies the same number of consecutive bytes in the flattened .tobytes representation. )dtype is_stringnparray string_datatobytes)valrrr_tobytes2s rc.eZdZdZd dfdd Zdd d ZZS)rasRemove duplicated initializer tensors from the main graph and all subgraphs. This pass detects initializers with identical shape, dtype, and content, and replaces all duplicate references with a canonical one. Initializers are deduplicated within each graph. To deduplicate initializers in the model globally (across graphs), use :class:`~onnx_ir.passes.common.LiftSubgraphInitializersToMainGraphPass` to lift the initializers to the main graph first before running pass. .. versionadded:: 0.1.3 .. versionchanged:: 0.1.7 This pass now deduplicates initializers in subgraphs as well. rrct||_dSNsuper__init__rselfr __class__rrr%Ss  z$DeduplicateInitializersPass.__init__modelir.Modelr ir.passes.PassResultc Csd}|D]T}i}t|jD]H}t||jrq|j}|dus#J|jt|jt |f}||vrUd}||}| ||j dusDJ|j |j t d|j |j q|||<qqtjj||dS)NFT8Replaced initializer '%s' with existing initializer '%s'r*modified)graphstuple initializersvaluesrrrrshaperreplace_all_uses_withrpopr infoirpasses PassResult) r'r*r/graphr2rrkeyinitializer_to_keeprrrcallWs.     z DeduplicateInitializersPass.call)r rrr*r+r r,__name__ __module__ __qualname____doc__r%r> __classcell__rrr(rrDscr)raRemove duplicated initializer tensors (using a hashed method) from the graph. This pass detects initializers with identical shape, dtype, and hashed content, and replaces all duplicate references with a canonical one. This pass should have a lower peak memory usage than :class:`DeduplicateInitializersPass` as it does not store the full tensor data in memory, but instead uses a hash of the tensor data. .. versionadded:: 0.1.7 rrcr!r"r#r&r(rrr%s  z*DeduplicateHashedInitializersPass.__init__r*r+r r,c Csd}|D]|}i}t|jD]p}t||jrq|j}|dus#Jt}| }| || } t|j } |j | | f} | |vr}t|| jt|krZtd|| |qd}|| } || |jduslJ|j|jtd|j| jq||| <qqtjj||dS)NFzVInitializer deduplication failed: hashes match but values differ with values %s and %sTr-r.)r0r1r2r3rrrhashlibsha512numpyupdate hexdigestr4rrr rr5rr6r7r8r9r:) r'r*r/r;r2rrhashed tensor_data tensor_digest tensor_dimsr<r=rrrr>sF       'z&DeduplicateHashedInitializersPass.call)rGr?r@rArrr(rrus )rrrrr r )rE __future__r__all__rHloggingrJronnx_irr8 getLoggerrBr rrr9 InPlacePassrrrrrrs   1