GigaChat Audio: Time-aware Large Audio Language Model
Aleksandr Kutsakov, Mariia Sadovina, Georgii Gospodinov, Alexandr Maximenko, Oleg Kutuzov, Pavel Bogomolov, Fyodor Minkin
eess.AS, cs.CL
2026-07-12
Interleaving periodic hh:mm:ss time anchors into the audio token stream lets this audio LLM reach 53.8 mIoU temporal grounding on 20-40 min recordings and scale to 120 min, far above Qwen3-Omni and Gemini 3 Flash.
Audio LLMs can understand what is in a recording, but temporal grounding over long audio still fails. Ask "at what minute does this happen" and the model emits unparseable timestamps, overly coarse ranges, or unsupported claims. GigaChat Audio (from Sber's AI team, weights released on HuggingFace) targets reliable timestamped answers over up to 120 minutes of input.
The authors frame three sub-questions: how to represent time, how to generate temporal supervision at scale, and how to generalize across durations. On the third they find an asymmetry: training on short audio only fails on long recordings, while training on long audio only hurts short-audio performance. Mixing is required.
The base is a 10B-parameter, 1.8B-activated MoE text model with 256k-token context. The audio side is a standard encoder to subsampler to projector stack with FlashAttention and chunk-wise attention (8s chunks, 40ms stride), producing embeddings at a 160ms frame rate. The audio encoder is HuBERT-like, pretrained on 22M hours.
The core idea is to interleave time anchors (inter-timings) directly into the audio token stream at a fixed interval (7s to 240s, configurable), so the model keeps reading "you are now at mm:ss" while listening. Anchors use plain-text hh:mm:ss plus an end marker. This is both cheaper and easier to train than dedicated timing tokens: with hh:mm:ss the model reaches 57.5 mIoU at a 16.7% temporal-data ratio, while special tokens need a 50% ratio to match.
Supervision comes from a cascaded pipeline: 14k hours of English filtered from YODAS2, word-level timestamps from WhisperX, QA generated from roughly 10-minute slices, then a separate verifier filters inconsistencies. At evaluation, five answer variants per question are aggregated.
| Task / length | GigaChat Audio | Baseline |
| Temporal grounding mIoU (20-40 min) | 53.8 (inter=60s) / 65.2 (inter=7s) | Qwen3-Omni-30B 3.6 |
| AMI meetings (15-50 min) MAE | 3.50 s | Qwen3-Omni 290.5 s |
| Temporal grounding mIoU (0-1 min) | 53.0 | Gemini 3 Flash 41.7 |
| Without time anchors (20-40 min) | 14.2 mIoU | 53.8 with anchors |
Anchor frequency is a precision-cost tradeoff: every 7s reaches 63.0 mIoU but adds 16% tokens; every 60s only 50.9 mIoU but 1.9%. Among formats, hh:mm:ss (50.9) far beats treating time as an integer "second N" token (20.9), which trains poorly.
For meetings, lectures, podcasts, and surveillance, "at what minute did this happen" is a constant need. This work pushes the usable horizon of audio LLMs from minutes to roughly two hours with the most basic trick, interleaving text timestamps, and the anchor rate is tunable for the accuracy-cost tradeoff at deploy time. For meeting and podcast AI builders, this is an open base model you can pick up.
The authors disclose that generative AI was used for language editing, synthetic supervision, and LLM-as-judge evaluation (all human-reviewed, they say). Data is English-only; multilingual is not covered. The length-generalization asymmetry shows sensitivity to the training-duration distribution. The collapse when time is formatted as an integer second shows the capability depends heavily on representation choice.