.ɷij.ddlmZddlZddlmZmZmZddlmZddl m Z ddl m Z m Z ddlZddlZddlmZmZddlmZmZdd lmZdd lmZdd lmZdd lmZej:eZGd dZ ddZ!GddejDZ#y)) annotationsN)CallableIterableIterator) nullcontext)partial)AnyLiteral)Tensornn)get_device_statesset_device_states)util)StaticEmbedding)SentenceTransformer)all_gather_with_gradc(eZdZdZddZddZddZy) RandContexta Random-state context manager class. Reference: https://github.com/luyug/GradCache. This class will back up the pytorch's random state during initialization. Then when the context is activated, the class will set up the random state with the backed-up one. c`tj|_t|\|_|_yN)torch get_rng_state fwd_cpu_stater fwd_gpu_devicesfwd_gpu_states)selftensorss r/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/losses/CachedMultipleNegativesRankingLoss.py__init__zRandContext.__init__s("0024Ew4O1d1ctjj|jd|_|jj tj |jt|j|jy)NT)devicesenabled) rrandomfork_rngr_fork __enter__ set_rng_staterrrrs rr'zRandContext.__enter__"s^\\**43G3GQU*V   D../$..0C0CDr cL|jj|||d|_yr)r&__exit__)rexc_typeexc_valexc_tbs rr+zRandContext.__exit__(s Hgv6 r N)returnNone)__name__ __module__ __qualname____doc__rr'r+r rrrsPE r r"CachedMultipleNegativesRankingLossc |jJ|jJtj5t ||j|jD]\}}}t |j |dd||D]Z\\}}}|j stj|j|j|z} | j\ dddy#1swYyxYw)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)sentence_feature with_gradcopy_random_state random_states) cacher;r enable_gradzipembed_minibatch_iter requires_graddotflattenbackward) grad_outputsentence_featuresloss_objr8gradr;reps_mb_grad_mb surrogates r_backward_hookrL-s >> %% %  ! ! -- -    )589JHNN\d\r\r5s ) 1 dM),--%5"&+"/ . * )% !g(( % '//*;W__=N OR] ]I&&( ) ))))sAC"A C""C+c eZdZdejdddddddf dfd Z d dd Z d dd Zdd Zddd Z dd Z ddZ e ddZ e ddZxZS)r6g4@ F) query_to_docjointNc t |t|dtr t d||_||_||_||_||_ hd} |s t dt|j| st dt|| z d| d|vr t dt||_ |d vrt d ||d k(rt|d d hzr t d||_||_hd} | | vrt d| d| | |_| dkr t d| |_| | dk(rt$j'd| dd|_d|_y)a! Boosted version of :class:`MultipleNegativesRankingLoss` (https://huggingface.co/papers/1705.00652) by GradCache (https://huggingface.co/papers/2101.06983). Constrastive learning (here our MNRL loss) with in-batch negatives is usually hard to work with large batch sizes due to (GPU) memory limitation. Even with batch-scaling methods like gradient-scaling, it cannot work either. This is because the in-batch negatives make the data points within the same batch non-independent and thus the batch cannot be broke down into mini-batches. GradCache is a smart way to solve this problem. It achieves the goal by dividing the computation into two stages of embedding and loss calculation, which both can be scaled by mini-batches. As a result, memory of constant size (e.g. that works with batch size = 32) can now process much larger batches (e.g. 65536). In detail: (1) It first does a quick embedding step without gradients/computation graphs to get all the embeddings; (2) Calculate the loss, backward up to the embeddings and cache the gradients wrt. to the embeddings; (3) A 2nd embedding step with gradients/computation graphs and connect the cached gradients into the backward chain. Notes: All steps are done with mini-batches. In the original implementation of GradCache, (2) is not done in mini-batches and requires a lot memory when the batch size is large. One drawback is about the speed. Gradient caching will sacrifice around 20% computation time according to the paper. See :class:`MultipleNegativesRankingLoss` for more details about the underlying loss itself. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value. In some literature, the scaling parameter is referred to as temperature, which is the inverse of the scale. In short: ``scale = 1 / temperature``, so ``scale=20.0`` is equivalent to ``temperature=0.05``. A higher scale (lower temperature) puts more emphasis on the positive example, and values between 10 and 100 are common. similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) mini_batch_size: Mini-batch size for the forward pass, this denotes how much memory is actually used during training and evaluation. The larger the mini-batch size, the more memory efficient the training is, but the slower the training will be. It's recommended to set it as high as your GPU memory allows. The default value is 32. gather_across_devices: If True, gather the embeddings across all devices before computing the loss. Recommended when training on multiple GPUs, as it allows for larger batch sizes, but it may slow down training due to communication overhead, and can potentially lead to out-of-memory errors. directions: Which similarity interaction terms to include in the loss. Options: - "query_to_doc": query -> all documents (always included as it covers the paired positive). - "query_to_query": query -> all other queries in the batch. - "doc_to_query": document -> all queries (symmetric term). - "doc_to_doc": document -> all other documents in the batch, excluding those belonging to the same query. The default ("query_to_doc",) matches the standard MultipleNegativesRankingLoss / InfoNCE behavior. partition_mode: How to normalize the scores (the softmax denominator): - "joint": One joint softmax over all selected directions. - "per_direction": One softmax per direction. A loss is computed for each direction and then averaged. Not compatible with ``"query_to_query"`` or ``"doc_to_doc"`` directions. show_progress_bar: If True, a progress bar for the mini-batches is shown during training. The default is False. hardness_mode: Strategy for applying hardness weighting. ``None`` (default) disables hardness weighting entirely. Options: - ``"in_batch_negatives"``: Adds ``hardness_strength * stop_grad(cos_sim)`` to every in-batch negative logit inside the softmax (`Lan et al. 2025 `_, Eq. 5). The in-batch negatives are all positives and hard negatives from other samples in the batch. - ``"hard_negatives"``: Applies ``hardness_strength * stop_grad(cos_sim)`` only to the logits of explicit hard negatives, leaving in-batch negatives unpenalized. Only active when explicit negatives are provided. As used in `Schechter Vera et al. 2025 `_ (EmbeddingGemma). - ``"all_negatives"``: Applies ``hardness_strength * stop_grad(cos_sim)`` to every negative logit, both in-batch negatives and explicit hard negatives, leaving only the positive unpenalized. Combines the effect of ``"in_batch_negatives"`` and ``"hard_negatives"``. hardness_strength: Strength of the hardness weighting. The meaning depends on ``hardness_mode``: - For ``"in_batch_negatives"``: acts as ``alpha`` in the hardness penalty, `Lan et al. 2025 `_ uses 9. - For ``"hard_negatives"``: acts as ``alpha`` in the hardness penalty, `Schechter Vera et al. 2025 `_ uses 5. Must be non-negative. Ignored when ``hardness_mode`` is ``None``. References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://huggingface.co/papers/1705.00652 - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://huggingface.co/papers/2101.06983 Requirements: 1. (anchor, positive) pairs, (anchor, positive, negative) triplets, or (anchor, positive, negative_1, ..., negative_n) n-tuples 2. Should be used with large `per_device_train_batch_size` and low `mini_batch_size` for superior performance, but slower training time than :class:`MultipleNegativesRankingLoss`. Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - Equivalent to :class:`MultipleNegativesRankingLoss`, but with caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. This loss also trains roughly 2x to 2.4x slower than :class:`MultipleNegativesRankingLoss`. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.CachedMultipleNegativesRankingLoss(model, mini_batch_size=64) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rzCachedMultipleNegativesRankingLoss is not compatible with a SentenceTransformer model based on a StaticEmbedding. Consider using MultipleNegativesRankingLoss instead.> doc_to_doc doc_to_queryrOquery_to_queryz)At least one direction must be specified.zInvalid directions: z . Valid: rOzB'query_to_doc' direction is required (contains the positive pair).)rP per_directionz7partition_mode must be 'joint' or 'per_direction', got rVrUrSapartition_mode='per_direction' requires every direction's candidate pool to include the positive pair. 'query_to_query' and 'doc_to_doc' only contain same-type similarities and never include the positive, making the per-direction loss ill-defined. Use partition_mode='joint' instead.>N all_negativeshard_negativesin_batch_negativeszhardness_mode must be one of z, got rQz'hardness_strength must be non-negative.Nzhardness_mode=z is set but hardness_strength=0.0, so hardness weighting has no effect. Set hardness_strength to a positive value to enable hardness weighting.)superr isinstancer ValueErrormodelscalesimilarity_fctmini_batch_sizegather_across_devicessetissubsettuple directionspartition_modeshow_progress_bar hardness_modehardness_strengthloggerwarningr<r;)rr]r^r_r`rarerfrgrhrivalid_directionsvalid_hardness_modes __class__s rrz+CachedMultipleNegativesRankingLoss.__init__IsL  eAh 0G    ,.%:"[HI I:''(893C OFV4V3WW`aq`rst t  +ab b + !; ;VWeVfgh h _ ,ZDTVbCc1c a  -!2^  4 4<=QRS T T\!!&  T T T\!!s),B6+C3/B<"C<C CCc # K|d}|j\}}ttjd||jd|j D];\}} | |jz} |j || | |||dn||\} } | | f=yw)z5Iterate over mini-batches of sentences for embedding. input_idsrzEmbed mini-batchesdescdisableN)r8rtrur9r:rv)shape enumeratetqdmtranger`rgr}) rr8r9r:r;r batch_sizerIirtrur|rvs rr?z7CachedMultipleNegativesRankingLoss.embed_minibatch_iters-[9 ! A! KK$$) 222    %HAu$...C!%!5!5!1#"3%2%:T a@P "6" D, $ $% %sB Bc|j|d}|jj}|Dcgc]}|Dcgc]}|jc}c}}|_|Scc}wcc}}w)zMCalculate the cross-entropy loss and cache the gradients wrt. the embeddings.T) with_backward)calculate_lossdetachrequires_grad_rGr<)rr|lossrsrs r"calculate_loss_and_cache_gradientszECachedMultipleNegativesRankingLoss.calculate_loss_and_cache_gradients8sY""4t"<{{}++-59:rr*!qvv*:  +:s A(A# A(#A(c tj|d}|ddDcgc]}tj|}}t|}d}|jrdt |}|Dcgc] }t |}}tj j r#tj j} | |z}|jd} tj|d} |d} tj|||z|j} tj| |j}t|}g}tjd||jd|j D]X}t!||jz|}| ||}tjt||j}||}| |}i}|j#|| |d<d |j$vr.|j#|||d <tj& |d ||f<d |j$vr|j#||j(|d <d |j$vrf|j#| |j(|d <||j+d|j-}|d j/|tj& i}|j0d vr|j2d kDr|j2|dj5z}tj| |jtj,|}|j+d|}|j0dk(r|}d|ddd| f<n.|j0dk(r|}n|j0dk(r |}d|dd| df<d |<||d<|D]}|||j6z||<|j9D]\}}|||z||<|d||f}|j:dk(rFtjt=|j?d} tj@| d}!n?d }!|j?D]}"|!tj@|"dz }!|!t|z}!||!z }#|#jCt|z|z }$|r |$jE|$j5}$|jG|$[tI|Scc}wcc}w)zPCalculate the all-pairs InfoNCE loss without caching gradients (for evaluation).rN)dim)devicezCalculating lossrrOrUrTrS)rYrXrWrQ)rdtyperXTrYrWFrP)%rcatlenrar distributedis_initializedget_ranksizearangereyerrr`rgminr_reinfTrepeatbool masked_fill_rhrirr^rrrflistrs logsumexpmeanrCappendsum)%rr|rqueriesrdocsroffsetdocrankworld_batch_sizedocs_alldocs_pos local_indicesidentitynum_docslossesrtru local_batch row_indices local_queries local_docs sim_matricessame_query_doc_mask penaltiespenalty own_doc_maskpenalty_exclusion_maskrypenpositive_scores all_scoreslog_z sim_matrixper_sample_loss loss_mbatchs% rrz1CachedMultipleNegativesRankingLoss.calculate_lossAs))DG$&*12h/ ! //\   % % +73G9=>#(->D>  //1((113 *"<<?99Tq)7 VVj-@X 99-gnnEt9%'[[   #...  [ 'Eed222J?C'c2K,,s;'7OK#K0M!+.JL+/+>+>}h+WL (4??2151D1D]T[1\ -.LQII: -.{K/GH0/3/B/B7J/W/Y/Y ^,t.-1-@-@:-V-X-X \*&.{&;&B&B1h&O&T&T&V#\*778KeiiZX I""&__**S000<3O3V3V3XX %yy)9'..X]XbXbcdop +221h? %%)99.:]*CG*1.?/?.?+?@''+??-9*''?:-9*CH*1.>.?+?@25./,3 .)$ C$0$5 $B S! C%OO- <S$0$5$; S! <+>:; ;STO""g-"YYtL,?,?,A'BJ  :"."5"5"7@JU__ZQ??E@\** /% 78O)..03{3CCjPK$$&)002 MM+ &w[ 'z6{o0?s S&ScLt|}t|dkrtdt|g}g|_|D]}g}g}|j |ddD]C\}}|j |j j|j |E|j ||jj |tjr4|j|} | jtt||| S|j|} | S)Nz Expected at least 2 inputs, got FT)r8r9r:)rErF)rrr\r;r?rrrris_grad_enabledr register_hookrrLr) rrElabelsr|r8reps_mbsrandom_state_mbsrHrvrs rforwardz*CachedMultipleNegativesRankingLoss.forwards2 !23  !A %?DU@V?WXY Y 1 8 H! )-)B)B!1"&*C* 6%  0 ? ? AB '' 5  6 KK !    % %&6 7 8  "::4@D   w~IZeij k  &&t,D r c |j|jj|j|j|j |j |j|jdS)N)r^r_r`rarerfrhri) r^r_r1r`rarerfrhrir)s rget_config_dictz2CachedMultipleNegativesRankingLoss.get_config_dictsVZZ"11::#33%)%?%?//"11!//!%!7!7  r c d|jz S)Ng?)r^r)s r temperaturez.CachedMultipleNegativesRankingLoss.temperaturesTZZr cy)Na @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } r5r)s rcitationz+CachedMultipleNegativesRankingLoss.citations r )r]rr^floatr_z"Callable[[Tensor, Tensor], Tensor]r`intrarrezStuple[Literal['query_to_doc', 'query_to_query', 'doc_to_query', 'doc_to_doc'], ...]rfz!Literal['joint', 'per_direction']rgrrhzGLiteral['in_batch_negatives', 'hard_negatives', 'all_negatives'] | Nonerirr/r0r)r8dict[str, Tensor]rtrrurr9rr:rrvzRandContext | Noner/z!tuple[Tensor, RandContext | None]) r8rr9rr:rr;zlist[RandContext] | Noner/z+Iterator[tuple[Tensor, RandContext | None]])r|list[list[Tensor]]r/r )F)r|rrrr/r )rEIterable[dict[str, Tensor]]rr r/r )r/zdict[str, Any])r/r)r/str)r1r2r3rcos_simrr}r?rrrrpropertyrr __classcell__)rns@rr6r6Hsx=A\\!&+ rs"88" G&8I;   8 $.))2)1) )6qqr