.ɷi2WddlmZddlZddlmZmZddlmZmZddl Z ddl m Z m Z ddl m Z ddlmZddlmZej$eZGd d e j*Zy) ) annotationsN)CallableIterable)AnyLiteral)Tensornn)util)SentenceTransformer)all_gather_with_gradceZdZdejdddddf d fd ZddZdd Zdd Ze dd Z e dd Z xZ S)MultipleNegativesRankingLossg4@F) query_to_docjointNc lt |||_||_|dkr t d||_||_hd} |s t dt|j| st dt|| z d| d|vr t dt||_ |d vrt d ||d k(rt|d d hzr t d||_ hd} || vrt d| d|||_ |dkr t d||_ | |dk(rtjd|dyyy)u' Given a dataset of (anchor, positive) pairs, (anchor, positive, negative) triplets, or (anchor, positive, negative_1, ..., negative_n) n-tuples, this loss implements a contrastive learning objective that encourages the model to produce similar embeddings for the anchor and positive samples, while producing dissimilar embeddings for the negative samples. In plain terms, the loss works as follows: 1. For each anchor (often a query) in the batch, we want the similarity to its matched positive (often a document) to be higher than the similarity to all other documents in the batch (including optional hard negatives). This is the standard forward MultipleNegativesRankingLoss / InfoNCE term, denoted with "query_to_doc". 2. Optionally, we can also require the opposite: for each document, its matched query should have higher similarity than all other queries in the batch. This is the symmetric backward term, denoted with "doc_to_query". 3. Optionally, we can further require that for each query, its similarity to all other queries in the batch is lower than to its matched document. This is the "query_to_query" term. 4. Optionally, we can also require that for each document, its similarity to all other documents in the batch is lower than to its matched query. This excludes documents that belong to the same query in the case of hard negatives (i.e. columns beyond the first two in the input). This is the "doc_to_doc" term. All of these are implemented via different choices of interaction directions and how we normalize the scores, but they all share the same core idea: the correct pair (query, positive) should have the highest similarity compared to all in-batch alternatives. All of these are expressed via the same underlying formulation by choosing different ``directions`` and ``partition_mode`` values. Optional negatives in the input are treated as additional hard-negative documents for the corresponding query. The default configuration is also known as the InfoNCE loss, SimCSE loss, cross-entropy loss with in-batch negatives, or simply in-batch negatives loss. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value. In some literature, the scaling parameter is referred to as temperature, which is the inverse of the scale. In short: ``scale = 1 / temperature``, so ``scale=20.0`` is equivalent to ``temperature=0.05``. A higher scale (lower temperature) puts more emphasis on the positive example, and values between 10 and 100 are common. similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) gather_across_devices: If True, gather the embeddings across all devices before computing the loss. Recommended when training on multiple GPUs, as it allows for larger batch sizes, but it may slow down training due to communication overhead, and can potentially lead to out-of-memory errors. directions: Which similarity interaction terms to include in the loss. Options: - "query_to_doc": query -> all documents (always included as it covers the paired positive). - "query_to_query": query -> all other queries in the batch. - "doc_to_query": document -> all queries (symmetric term). - "doc_to_doc": document -> all other documents in the batch, excluding those belonging to the same query. The default ("query_to_doc",) matches the standard MultipleNegativesRankingLoss / InfoNCE behavior. partition_mode: How to normalize the scores (the softmax denominator): - "joint": One joint softmax over all selected directions. - "per_direction": One softmax per direction. A loss is computed for each direction and then averaged. Not compatible with ``"query_to_query"`` or ``"doc_to_doc"`` directions. hardness_mode: Strategy for applying hardness weighting. ``None`` (default) disables hardness weighting entirely. Options: - ``"in_batch_negatives"``: Adds ``hardness_strength * stop_grad(cos_sim)`` to every in-batch negative logit inside the softmax (`Lan et al. 2025 `_, Eq. 5). The in-batch negatives are all positives and hard negatives from other samples in the batch. Works with all data formats including pairs-only. - ``"hard_negatives"``: Applies ``hardness_strength * stop_grad(cos_sim)`` only to the logits of explicit hard negatives, leaving in-batch negatives unpenalized. Only active when explicit negatives are provided. As used in `Schechter Vera et al. 2025 `_ (EmbeddingGemma). - ``"all_negatives"``: Applies ``hardness_strength * stop_grad(cos_sim)`` to every negative logit, both in-batch negatives and explicit hard negatives, leaving only the positive unpenalized. Combines the effect of ``"in_batch_negatives"`` and ``"hard_negatives"``. hardness_strength: Strength of the hardness weighting. The meaning depends on ``hardness_mode``: - For ``"in_batch_negatives"``: acts as ``alpha`` in the hardness penalty, `Lan et al. 2025 `_ uses 9. - For ``"hard_negatives"``: acts as ``alpha`` in the hardness penalty, `Schechter Vera et al. 2025 `_ uses 5. Must be non-negative. Ignored when ``hardness_mode`` is ``None``. Requirements: 1. (anchor, positive) pairs, (anchor, positive, negative) triplets, or (anchor, positive, negative_1, ..., negative_n) n-tuples Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - :class:`CachedMultipleNegativesRankingLoss` is equivalent to this loss, but it uses caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. However, it is slightly slower. - :class:`GISTEmbedLoss` is equivalent to this loss, but uses a guide model to guide the in-batch negative sample selection. `GISTEmbedLoss` yields a stronger training signal at the cost of some training overhead. Loss variants from the literature: - Standard InfoNCE / classic MultipleNegativesRankingLoss (query -> doc only), e.g. as in `van den Oord et al. 2018 `_:: loss = MultipleNegativesRankingLoss( model, directions=("query_to_doc",), # default partition_mode="joint", # default ) This variant is recommended if you are training with (anchor, positive, negative_1, ..., negative_n) n-tuples. - Symmetric InfoNCE (query -> doc and doc -> query), e.g. as in `Günther et al. 2024 `_:: loss = MultipleNegativesRankingLoss( model, directions=("query_to_doc", "doc_to_query"), partition_mode="per_direction", # forward/backward computed separately and averaged ) This variant may outperform the standard variant in some scenarios. - GTE improved contrastive loss (query/doc + same-type negatives), e.g. as in `Li et al. 2023 `_:: loss = MultipleNegativesRankingLoss( model, directions=("query_to_doc", "query_to_query", "doc_to_query", "doc_to_doc"), partition_mode="joint", # single softmax over all selected interaction terms ) This variant is recommended if you are training with only (anchor, positive) pairs or (anchor, positive, negative) triplets, as it may provide a stronger training signal. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.MultipleNegativesRankingLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rzScale must be a positive value.> doc_to_doc doc_to_queryrquery_to_queryz)At least one direction must be specified.zInvalid directions: z . Valid: rzB'query_to_doc' direction is required (contains the positive pair).)r per_directionz7partition_mode must be 'joint' or 'per_direction', got rrrapartition_mode='per_direction' requires every direction's candidate pool to include the positive pair. 'query_to_query' and 'doc_to_doc' only contain same-type similarities and never include the positive, making the per-direction loss ill-defined. Use partition_mode='joint' instead.>N all_negativeshard_negativesin_batch_negativeszhardness_mode must be one of z, got rz'hardness_strength must be non-negative.Nzhardness_mode=z is set but hardness_strength=0.0, so hardness weighting has no effect. Set hardness_strength to a positive value to enable hardness weighting.)super__init__modelscale ValueErrorsimilarity_fctgather_across_devicessetissubsettuple directionspartition_mode hardness_modehardness_strengthloggerwarning) selfrrrr r$r%r&r'valid_directionsvalid_hardness_modes __class__s l/var/www/html/venv/lib/python3.12/site-packages/sentence_transformers/losses/MultipleNegativesRankingLoss.pyrz%MultipleNegativesRankingLoss.__init__sN   A:>? ?,%:"[HI I:''(893C OFV4V3WW`aq`rst t  +ab b + !; ;VWeVfgh h _ ,ZDTVbCc1c a  -^  4 4<=Q#(->D>  //1((113 *"<<?99Tq)7 VVj-@X ll:gnnE  . m,  '+':':=('S ^$ t .-1-@-@PW-XL) *JO))L) *; +E F T__ ,+/+>+>w +S+U+UL ( 4?? *)-)<)L  ?!) 8HC ,S 1C 7L  8'~6{M7QR   ' )YYtL$7$7$9:BFOOF2EE*113 < ;; < S& &E 5(..00 q?sP$c|j|jj|j|j|j |j |jdS)N)rrr r$r%r&r')rr__name__r r$r%r&r'r*s r.get_config_dictz,MultipleNegativesRankingLoss.get_config_dictSsMZZ"11::%)%?%?//"11!//!%!7!7  r/c d|jz S)Ng?)rrns r. temperaturez(MultipleNegativesRankingLoss.temperature^sTZZr/ct|jhdk(r|jdk(ryt|jddhk(r|jdk(ryy) N>rrrrra @misc{li2023generaltextembeddingsmultistage, title={Towards General Text Embeddings with Multi-stage Contrastive Learning}, author={Zehan Li and Xin Zhang and Yanzhao Zhang and Dingkun Long and Pengjun Xie and Meishan Zhang}, year={2023}, eprint={2308.03281}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2308.03281}, } rrru @misc{günther2024jinaembeddings28192token, title={Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents}, author={Michael Günther and Jackmin Ong and Isabelle Mohr and Alaeddine Abdessalem and Tanguy Abel and Mohammad Kalim Akram and Susana Guzman and Georgios Mastrapas and Saba Sturua and Bo Wang and Maximilian Werk and Nan Wang and Han Xiao}, year={2024}, eprint={2310.19923}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2310.19923}, } a_ @misc{oord2019representationlearningcontrastivepredictive, title={Representation Learning with Contrastive Predictive Coding}, author={Aaron van den Oord and Yazhe Li and Oriol Vinyals}, year={2019}, eprint={1807.03748}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/1807.03748}, } )r!r$r%rns r.citationz%MultipleNegativesRankingLoss.citationbsY  $d d##w.  t NN#C CH[H[_nHn  r/)rr rfloatrz"Callable[[Tensor, Tensor], Tensor]r rJr$zStuple[Literal['query_to_doc', 'query_to_query', 'doc_to_query', 'doc_to_doc'], ...]r%z!Literal['joint', 'per_direction']r&zGLiteral['in_batch_negatives', 'hard_negatives', 'all_negatives'] | Noner'rtreturnNone)r3zIterable[dict[str, Tensor]]r4rrur)r6z list[Tensor]r4rrur)ruzdict[str, Any])rurt)rustr) rm __module__ __qualname__r cos_simrr7r2ropropertyrqrs __classcell__)r-s@r.rrs=A\\&+ rsD". &I;   8 $x299xr/