Major OpenCLIP Update Integrates Audio Models and Variable Resolution
wightmanr · x · 2026-08-14
The author has made extensive modifications to the OpenCLIP library, focusing on supporting variable resolution/aspect ratio NaFlexViT encoders and matching WebDataset pipelines, alongside integrating existing audio CLAP models.
Furthermore, the author experimented with combining NaFlex and CLAP by adding a mel patch embedder to the timm NaFlexViT base, achieving variable time capability for audio processing without major architectural changes. A new 'modern-text' encoder, utilizing recent ideas similar to ModernBERT, was also introduced. Preliminary training of modest-sized models has successfully validated the architecture and code changes.
Related event: OpenCLIP Overhaul Integrates Audio Models and Variable Resolutions(3 posts)→
More from Multimodal
- Flux 3 Klien first impressions: occasional body horror like SD3, otherwise solid — rionix88 · 2026-10-16
- XGRIDS builds world's largest 3DGS town: 23.5 km² and 6.2 billion Gaussian Splats — bilawalsidhu · 2026-10-16
- IMMERSION: AI film compilation links 12 shorts with one continuous camera move — AI_Andrew · 2026-10-16
- A song with vocals, lyrics and 3D head animation created entirely by Opus 5.5 — repligate · 2026-10-16
- Alibaba open-sources CommerceGen-Studio for commercial-grade e-commerce images — alibabagroup · 2026-10-16
- EVA-Cap: Fudan and OPPO align video captioning with atomic audiovisual events via GRPO — Fudan-University · 2026-10-16