swift-transformers 1.4.0 ships with breaking decoding changes from 15 contributors
pcuenq · x · 2026-11-02
Hugging Face released swift-transformers 1.4.0, with contributions from 15 people (10 first-timers), focused on tokenization correctness, performance, and alignment with Transformers.
Key changes: cleanuptokenizationspaces now defaults to false when absent, and BPE tokenizers skip cleanup even when set to true — so x != y stays intact instead of becoming x!= y. Token IDs are unchanged but decoded text may differ for affected models, including Llama 3.x configs with cleanup enabled; set the flag explicitly in tokenizerconfig.json to restore old behavior. Tokenizers also now throw TokenizerError.invalidRegex at load time when a Split pre-tokenizer or Replace normalizer regex fails to compile.
More from Infra
- T3 Code claims 2x OpenAI's token throughput — mocked for conflating routing with inference — scaling01 · 2026-11-02
- CI tip: GitHub Actions rounds each job up to a full minute — merge short jobs to save cost — DanielLockyer · 2026-11-02
- UBS: Musk's Terafab rethinks fabs — wafers spend only ~25% of cycle time processing — BenBajarin · 2026-11-02
- Unsloth's 1-bit DeepSeek-V4.1-Flash GGUF beats other quants by 10%+ on agentic benchmarks — solyarisoftware · 2026-11-02
- Musk: superintelligence compute will 'round up to 100%' be in space — elonmusk · 2026-11-02
- Qualcomm demos first-gen HBC silicon: 18x LPDDR5X bandwidth, 5x HBM4 per watt — ryanshrout · 2026-11-02