How text-to-video models work

Every production text-to-video model in 2026 is a video diffusion model: it starts from random noise in a compressed latent space and iteratively denoises toward a video that matches the text prompt — the same core mechanism as image diffusion extended to the temporal dimension. Some earlier systems (VideoPoet, Emu Video) instead tokenized frames and predicted them autoregressively, similar to how GPT predicts text, but diffusion transformers now dominate the frontier because they scale more predictably with compute.

Sora is no longer part of the comparison: OpenAI's Sora, first shown publicly in February 2024 and relaunched as the Sora 2 app in September 2025, was for two years the highest-profile text-to-video product. OpenAI discontinued the consumer app on April 26, 2026 and is shutting down the Sora API on September 24, 2026 — the day after this article was last checked — reallocating compute toward coding tools and enterprise products while continuing Sora internally as a 'world models' research effort. If you are choosing a text-to-video tool today, Sora is not an option; Veo 3.1, Kling 3.0, and Runway Gen-4.5 are the live frontier.

The consistency problem: Generating a consistent clip is vastly harder than generating a single image. The model must maintain the identical appearance of every object across hundreds of frames while simulating realistic motion and physics. This is why text-to-video quality improved more slowly than text-to-image: a face that looks right in frame 1 must look identical in frame 200, despite being generated one latent step at a time.

DiT architecture — why it replaced U-Net for video

The Diffusion Transformer (DiT) architecture, introduced by Peebles & Xie (2023), replaced the U-Net backbone previously used in image diffusion models. DiT treats diffusion as a sequence-to-sequence problem: the noisy latent video is divided into patches (spatially and temporally), each patch becomes a token, and a standard transformer denoises the full sequence in parallel. This architecture scales better with compute than U-Net — larger DiTs consistently outperform smaller ones — and handles the temporal dimension naturally because attention is computed across all space-time patches simultaneously.

Current limitations every user must know

• Character consistency across shots: without explicit character-reference features, the same person in frames 1 and 150 will drift in facial appearance — a hard problem because the model has no persistent entity memory. Runway Gen-4.5 targets this directly with dedicated consistency conditioning.
• Physics edge cases: hands interacting with objects, liquids with complex dynamics, and crowd scenes with many independent agents remain challenging. Veo 3.1's physics prior helps but doesn't fully solve this.
• Prompt adherence: text-to-video models interpret prompts more loosely than image models like DALL-E 3 or Ideogram do. Expect 3–5 generation attempts to get close to a specific vision.
• Length/resolution/detail tradeoff: the frontier has not converged on one answer. Veo 3.1 renders up to 4K on a short base clip with native audio; Kling 3.0 extends a single generation to 3 minutes at native 4K; Runway Gen-4.5 natively renders 720p and relies on upscaling for higher resolution, trading raw output resolution for character consistency and camera control. Choosing a tool means choosing which constraint matters more for your shot.
• Computational cost: a high-resolution multi-second generation still takes minutes on cloud infrastructure even on current-generation GPUs — real-time video generation remains economically prohibitive in 2026.

Best prompt structure for video: The five-element structure that consistently produces better results: [Subject] + [Action] + [Environment] + [Camera movement] + [Visual style]. Example: 'A young woman in a red sari [subject] walking confidently toward camera [action] through a busy Mumbai market at golden hour [environment] in a slow push-in tracking shot [camera] with warm cinematic color grading [style].' Camera and style specifications reduce the model's degrees of freedom and produce more predictable, intentional outputs.

Practice questions

1. What are the three major architectural approaches to text-to-video generation? (Answer: (1) Diffusion-based: extend image diffusion models to video by adding temporal attention layers, now dominated by the DiT approach used in Veo 3.1 and Kling 3.0. Denoise across space AND time jointly. High quality, and — since FlashAttention-era optimization — no longer prohibitively slow. (2) Autoregressive: predict video frames token-by-token like language tokens (VideoPoet, Emu Video). Can model long-range temporal consistency but lower quality; mostly overtaken by diffusion transformers by 2026. (3) GAN-based hybrid: use adversarial training for temporal consistency with diffusion for frame quality — an older approach, superseded by pure diffusion.)
2. What is the 'temporal consistency' problem in video generation and how do modern models address it? (Answer: Temporal consistency: objects should not change identity, shape, or appearance between frames (no flickering, morphing, or disappearing objects). Early video generation treated each frame independently, so characters could change faces between frames and backgrounds could flicker. Modern solutions: (1) 3D temporal attention — attention operates across space AND time simultaneously, as in DiT. (2) Optical flow conditioning — explicitly model motion between frames. (3) Latent video diffusion — denoise in a compressed video latent space where temporal structure is preserved by the VAE. Runway Gen-4.5 adds explicit character-consistency conditioning on top of these.)
3. OpenAI's Sora 2 (discontinued September 2026) generated 60-second 1080p videos. What made that computationally demanding, and how do today's surviving models trade off differently? (Answer: A 60-second 1080p video at 24fps is 1,440 frames processed by a video diffusion transformer with 3D spatiotemporal attention — attention complexity scales roughly as O((H×W×T)²) before latent compression. Sora achieved this via heavy VAE-based compression and model parallelism across large GPU clusters; a single generation took several minutes of compute. Its successors make different tradeoffs: Veo 3.1 renders up to 4K on a short base clip with native audio, Kling 3.0 extends single-generation length to 3 minutes at native 4K, while Runway Gen-4.5 keeps native resolution lower (720p) to spend its compute budget on character and motion consistency instead. The length/resolution/compute tradeoff Sora exposed is still being actively renegotiated, not solved.)
4. What is ControlNet for video and why is it important for production workflows? (Answer: ControlNet conditions video generation on structural control signals: depth maps, pose skeletons, edge maps, or optical flow. This constrains the generated video to follow specific motion or layout — enabling consistent character motion, choreography matching, or structure-preserving style transfer. Production use: directors can sketch rough motion, and ControlNet generates photorealistic video matching that motion. Applying the ControlNet signal consistently across frames, rather than per-frame independently, is what prevents flickering.)
5. What is the key difference between Kling 3.0 and Runway Gen-4.5 in terms of use case and workflow? (Answer: Kling 3.0 (Kuaishou) prioritizes length and native resolution — native 4K, up to 3 minutes in a single generation — making it well suited to commercial product video and longer narrative shots without manual stitching, at the lowest cost per clip among the three. Runway Gen-4.5 prioritizes character and scene consistency across multiple generated clips, plus fine-grained creative controls such as motion brush and camera control, aimed at professional filmmaking workflows where a director combines several shots into a coherent sequence — at the cost of a lower native output resolution. In short: Kling favors single-generation length and resolution; Runway favors multi-shot consistency and creative control.)