papers.luk.media

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

arXiv:2610.05608 · Paper published 2026-10-04 · Original paper ↗

Connecting separate video and audio diffusion networks with blockwise bidirectional cross-attention allows video and audio streams to continuously exchange temporal information, enabling joint generation of 5-second videos with synchronized 44 kHz audio.

Listen · 7 minOpen audio
Problem
Previous approaches generate video and audio separately or connect them late in pipeline processing. Lacking mid-generation signal exchange, audio drift occurs, causing misaligned mouth movements and broken visual-sound timing.

How it works

  1. Encode text and initialize noise. Text descriptions enter text encoders, while random noise grids are initialized for video and audio frames. Text embeddings provide shared semantic guidance for both modalities.
  2. Exchange video and audio signals. Noisy video and audio features enter parallel diffusion streams. At each block, video queries attend to audio features and audio queries attend to video features. This mutual attention forces mouth shapes to track speech sounds and visual impacts to align with audio spikes.
  3. Denoise latents in ten steps. The network iteratively predicts and removes noise across ten sampling steps. This produces clean low-resolution video latents and synchronized 44 kHz audio latents.
  4. Upscale video to high definition. Clean video latents pass into a latent upscaler and a super-resolution diffusion network. The upscaler expands spatial grids before the diffusion network adds detail, outputting 1920x1080 video with synchronized audio.

Example from the paper

For a prompt of a girl saying '<S>When I grow up, I want to be a doctor!<E>', text embeddings guide initial noise grids. During dual-stream denoising, audio speech tokens attend directly to facial video tokens. This forces frame-by-frame lip shapes to match spoken phonemes, producing a video where visible mouth contractions synchronize with spoken words.

Evidence

Reported evidence and its scope
Reported resultTest conditionsWhat it supports
Speech word error rate dropped 47% from 0.235 (fine-tuning) to 0.124 after reinforcement learning post-training.Whisper-large-v3 transcription on 64 speech prompts across 3 seeds per prompt.Confirms multi-objective reinforcement learning improves speech intelligibility.
Achieved top scores on VABench for speech quality (1.496), lip-sync (2.072), and visual realism (4.485).Automated VABench benchmark comparing against open-source LTX 2.5.Demonstrates superior audio-visual synchronization compared to open-source baselines.
10-step distilled student achieved a 51% win rate against the 20-step teacher in human preference evaluation.Side-by-side human evaluation on image-to-video generation tasks.Establishes that two-stage distillation halves sampling evaluations without quality degradation.

Limits

  • Generation duration is capped at 5 seconds (121 frames at 23–25 fps) per generation pass.
  • Full-HD (1920x1080) output requires a separate multi-tile super-resolution pass, adding processing latency.
  • Running on 16GB VRAM GPUs requires text encoder quantization, which alters text embeddings and output clips.
  • Visual prompt adherence lags behind top proprietary systems like MiniMax H3 and Seedance 2.0 in human evaluation.
Terms used here
CrossDiT
A dual-stream transformer linking separate video and audio diffusion networks through cross-attention.
Bidirectional Cross-Attention
Layers where video tokens query audio features and audio tokens query video features.
Latent Upscaler
A neural network that expands video latent spatial dimensions directly without decoding pixels.
GRPO
Group Relative Policy Optimization, an online reinforcement learning algorithm using group-centered rewards.
Visual summaryVisual summary of Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
Source and authors

Paper authors: Kandinsky Lab

arXiv:2610.05608 ↗ · Episode released 2026-10-16 · Explanation updated 2026-10-16

  • AgentGarten: Code Worlds for Evolving Agents

    AgentGarten couples physics engines to a shared neural renderer through untextured 3D geometry maps. This separation lets AI agents interact with photorealistic virtual worlds in real time while keeping physical rules and object coordinates explicitly controllable in code.

    2026-10-08 · AI agents · Computer vision · Efficient AI

  • Scaling Properties of Same-Family On-Policy Distillation

    By measuring student learning progress through divergence from initialization, this paper fits joint power laws that predict a student language model's peak test accuracy and learning speed from model scale and teacher accuracy before distillation begins.

    2026-09-26 · Language models · Reinforcement learning · Efficient AI