Skip to content

Conversation

@emapco
Copy link
Contributor

@emapco emapco commented Nov 4, 2025

This pull request makes a small fix to the logic that restores the original output settings in the call_grow_cache method of AdaptiveLayerLoss.py. The change ensures that the all_layer_embeddings key is deleted from the output only if the original setting for output_hidden_states was False.

  • Fixed conditional logic in call_grow_cache to correctly remove all_layer_embeddings from the output only when output_hidden_states was originally False (sentence_transformers/losses/AdaptiveLayerLoss.py).

…ecorator/AdaptiveLayerLoss

When training with AdaptiveLayerLoss, the `all_layer_embeddings` are deleted erroneously when `output_hidden_states` is True.
Copilot AI review requested due to automatic review settings November 4, 2025 07:13
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes a logic error in the AdaptiveLayerLoss class where the condition for removing all_layer_embeddings from the output was inverted.

  • Corrects the conditional logic to properly clean up all_layer_embeddings when it was not originally requested

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@emapco emapco requested a review from Copilot November 4, 2025 16:58
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +81 to +82
if self.transformer.auto_model.config.output_hidden_states and "all_layer_embeddings" in features:
output["all_layer_embeddings"] = features["all_layer_embeddings"]
Copy link

Copilot AI Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The call_use_cache method returns cached data without calling original_forward, so the input features parameter will not contain all_layer_embeddings. This check will always fail because features is the raw input to the forward method, not the output. If all_layer_embeddings needs to be included when output_hidden_states is enabled, it should be reconstructed from the cached self.embeddings data instead of reading from features.

Suggested change
if self.transformer.auto_model.config.output_hidden_states and "all_layer_embeddings" in features:
output["all_layer_embeddings"] = features["all_layer_embeddings"]
if self.transformer.auto_model.config.output_hidden_states:
# Reconstruct all_layer_embeddings from cached self.embeddings
output["all_layer_embeddings"] = self.embeddings[self.call_idx]

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.embeddings doesn't contain all of the hidden states/layer embeddings. Perhaps call_grow_cache() should also storeself.all_layer_embeddings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant