FULL STORY
Flash Attention Bug Degrades Embeddings, Patched
Enabling Flash Attention 2 caused silent degradation in models like BGE-M3, which was subsequently fixed in the 5.6.1 patch of Sentence Transformers.
2026-07-23 ~ 2026-07-23 · 2 episodes · 5 posts
Episode 1 · Flash Attention Causes Performance Drops in BGE-M3 and Other Models (2026-07-23, 2 posts)
Incompatibility between packed position IDs and RoBERTa when using flash_attention_2 caused significant performance drops, notably degrading BAAI/bge-m3's STSB test Spearman score from 0.8485 to 0.7239 before the fix.
- Sentence Transformers packed position IDs wrong for RoBERTa models under flash attention — tomaarsen · 2026-07-23
- BAAI/bge-m3 lost accuracy under flash attention until the Sentence Transformers patch — tomaarsen · 2026-07-23
Episode 2 · Sentence Transformers 5.6.1 Fixes Flash Attention Embedding Regression (2026-07-23, 3 posts)
Sentence Transformers v5.6.1 has been released as a patch to fix a silent embedding degradation bug caused by Flash Attention, primarily affecting RoBERTa and XLM-R models.
- Sentence Transformers 5.6.1 fixes silent flash-attention embedding regressions — tomaarsen · 2026-07-23
- Sentence Transformers 5.6.1 fixes a silent flash-attention regression in RoBERTa models — tomaarsen · 2026-07-23
- sentence-transformers 5.6.1 fixes Flash Attention embedding bug in XLM-R models — tomaarsen · 2026-07-23