How LLMs See Images, Audio, and More
Data Streaming + AI: Shaping the Future Together (Sponsored)
Join us at the Data Streaming + AI Summit on Sept 30, 2025 at the Grand Hyatt at SFO — the most open data streaming conference bringing together leaders from OpenAI, Netflix, Uber, Google, Salesforce, LinkedIn, Blueshift, Confluent, Ververica, StreamNative, and more. Hear keynote insights from Reynold Xin (Co-founder & Chief Architect, Databricks), learn how OpenAI powers AI with real-time streaming, and explore 30+ technical sessions across four tracks. On September 29, level up your skills with Data Streaming Fundamentals Training (for beginners) or dive deeper with the Hands-On Streaming Lakehouse Workshop (advanced, in partnership with AWS).
🎟️ Use code BBG50 for 50% off conference tickets.
Language models are no longer limited to text. Today's AI systems can understand images, process audio, and even handle video, but they still need to convert everything into tokens first. Just like how "Hello world!" becomes discrete tokens for text processing, a photograph gets chopped into image patches, and a song becomes a sequence of audio codes.
This is Part 2 of the ByteByteGo LLM Tokenization Series. You can view Part 1 here.
Each type of media requires completely different tokenization strategies, each with unique trade-offs between quality, efficiency, and semantic understanding.
As models extend beyond pure text, multimodal tokenization lets them ingest images, audio, and video alongside natural language. While the core idea of mapping raw input into a discrete sequence remains the same, each modality uses a different strategy.
Image Tokenization
Image tokenization transforms visual data into discrete tokens that can be processed by LLMs. Unlike text, which naturally exists as discrete symbols, images are continuous pixel arrays that require conversion into token sequences.
Main Approaches to Image Tokenization
1. Patch Embeddings (Vision Transformer Style)
Patch embeddings divide images into fixed-size, non-overlapping “patches”. Each patch is treated as a single token, similar to how words function in text sequences. This allows the image to be processed into understandable units, just like text tokens.
Technical Process:
An input image (e.g., 224×224 pixels) is divided into patches of size P×P (commonly 16×16 or 14×14)
Each patch contains P²×C values (where C is the number of channels, typically 3 for RGB)
These pixel values are flattened and linearly projected through a learned embedding matrix to create a D-dimensional vector
Positional encodings are added to preserve spatial relationships between patches
The result is a sequence of N tokens, where N = (H×W)/(P²)
What to know:
Advantages: Simple, preserves spatial structure, works well with standard transformer architectures
Limitations: Fixed patch size may not align with natural object boundaries, can be inefficient for high-resolution images
Typical Applications: Image classification, object detection, general computer vision tasks
Example: A 224×224 image with 16×16 patches yields 196 tokens (14×14 grid).
2. Discrete VAE and Vector Quantization
Vector quantization is like creating a visual dictionary. It converts values into “buckets” so that AI can pattern match image parts into buckets.
Technical Process:
An encoder network compresses the input image into a smaller, more manageable representation
The system then matches each part of this compressed image to the closest pattern in its learned "visual vocabulary"
This vocabulary typically contains 512-8,192 different visual patterns (like textures, shapes, or color combinations)
The result is a grid of pattern numbers - for example, a 32×32 grid of codes that represents a 256×256 pixel image
A decoder network can recreate the original image by looking up each code and reconstructing the corresponding visual pattern
Variants (remove?):
VQ-VAE: Basic vector quantization with straight-through gradient estimation
VQ-GAN: Adds adversarial training for better reconstruction quality
RQ-VAE: Uses multiple quantization layers for hierarchical representations
What to know:
Advantages: True discrete tokens, excellent compression ratios
Limitations: Requires pre-training the codebook, potential information loss during quantization
Typical Applications: Image generation, image compression, multimodal language models
3. CLIP-Style Contrastive Embeddings
Contrastive learning approaches like CLIP create embeddings in a shared vision-language space, which can then be discretized into pseudo-tokens for downstream tasks.
Technical Process:
Images and text are embedded into a shared high-dimensional space through contrastive learning
The continuous embeddings capture semantic relationships between visual and textual concepts
Discretization techniques (k-means clustering, product quantization, or learned codebooks) convert continuous embeddings into discrete tokens
These tokens maintain semantic meaning while being compatible with discrete sequence models
What to know:
Advantages: Semantically meaningful tokens, natural vision-language alignment, transferable across tasks
Limitations: Requires large-scale contrastive pre-training, discretization may lose fine-grained details
Typical Applications: Multimodal retrieval, vision-language understanding, cross-modal generation
Trade-offs to Know
Each tokenization approach has important design decisions to be aware of that impact performance, efficiency, and applicability:
Computational Efficiency:
Patch size directly affects sequence length and computational cost
Smaller patches (8×8) preserve more detail but create longer sequences
Larger patches (32×32) are more efficient but may lose fine-grained information
Token count impacts transformer attention complexity (quadratic scaling)
Information Preservation vs Compression:
Higher compression ratios reduce token count but may lose visual details
VQ-based methods offer excellent compression but require careful codebook design
Patch embeddings preserve all information but with minimal compression
The choice depends on whether perfect reconstruction is necessary for your task
Audio Tokenization
Audio tokenization converts continuous audio waveforms into discrete tokens that can be processed by sequence models. Unlike text or images, audio has unique challenges due to its temporal nature, high sampling rates, and the need to preserve both semantic content and acoustic properties like prosody, timbre, and speaker characteristics.
Main Approaches to Audio Tokenization
1. Codec Tokens (Neural Audio Codecs)
Neural audio codecs like EnCodec and SoundStream compress audio into token sequences while preserving quality. They map audio features to a finite set of learned codes using something called “vector quantization.”
Technical Process:
Audio is segmented into overlapping windows (typically 10-25 ms)
An encoder network converts each window into a continuous feature representation
Vector quantization maps features to the nearest codebook entry (typically 1024-4096 codes)
Multiple quantization layers create hierarchical representations
The result is a sequence of discrete tokens at ~50-75 Hz (much lower than original audio sampling rate)
Models:
EnCodec: Facebook's codec with residual vector quantization, achieving high compression ratios
SoundStream: Google's approach with efficient temporal modeling
What to know:
Advantages: Preserves acoustic properties, enables high-quality reconstruction, suitable for music and speech
Limitations: Requires pre-trained codec models, may introduce artifacts at very low bitrates
Applications: Audio generation, compression, music synthesis, speech synthesis
2. Phoneme/Character Tokens (ASR-based)
Automatic Speech Recognition converts spoken audio into text representations, creating tokens that capture semantic content but discard acoustic properties.
Technical Process:
Audio is processed through an ASR system (e.g., Whisper, Wav2Vec2)
Speech is converted to phonemes (sound units) or characters/words
Tokens represent linguistic content rather than acoustic features
Can include timing information and confidence scores
Often combined with speaker identification or language detection
Token Types:
Phonemes: Fine-grained linguistic units (typically 40-50 phonemes per language)
Characters: Direct transcription to text characters
Words/Subwords: Higher-level semantic units using BPE or similar tokenization
What to know:
Advantages: Compact representation, language-model compatible, captures semantic meaning
Limitations: Complete loss of prosody, speaker identity, emotional content, and non-speech audio
Applications: Speech-to-text, voice assistants, content indexing, cross-modal retrieval
3. Multi-Scale Token Stacks (Hierarchical Representations)
Hierarchical approaches use multiple token sequences at different temporal resolutions to capture both global structure and fine-grained details, similar to multi-scale image or video representations.
Technical Process:
Coarse tokens: Capture global audio structure at low temporal resolution (e.g., 12.5 Hz)
Fine tokens: Represent detailed acoustic information at higher resolution (e.g., 50 Hz)
Multiple quantization layers create a pyramid of representations
Tokens can be processed jointly or in stages (coarse-to-fine generation)
Cross-attention mechanisms often link different scales
Implementation Examples:
Residual Vector Quantization: Multiple codebooks applied sequentially to capture different levels of detail
Temporal Hierarchies: Different encoding rates for different time scales
Frequency Hierarchies: Separate tokenization for different frequency bands
What to know:
Advantages: Balances efficiency with quality, enables progressive generation, captures multi-scale patterns
Limitations: Increased model complexity, requires careful design of hierarchy levels
Applications: High-quality audio generation, music modeling, efficient compression
Tradeoffs to Remember
Temporal Resolution vs Efficiency:
Higher token rates preserve more temporal detail but increase sequence length
Codec tokens typically achieve 50-100x compression over raw audio
ASR tokens provide extreme compression but lose all acoustic information
Semantic vs Acoustic Information:
Codec tokens preserve acoustic properties (pitch, timbre, speaker identity)
ASR tokens capture semantic content but get rid of rhythm/intonation information
Multi-scale approaches try their best to balance both types of information
Quality vs Compression:
More codebook entries improve reconstruction quality but increase model size
Hierarchical approaches can achieve better quality-compression trade-offs
Domain Usage:
Speech-optimized codecs may not work well for music and vice versa
Universal codecs like EnCodec aim to handle diverse audio types
ASR-based approaches are limited to speech and may not handle accents or languages well
The best tokenization approach depends on your specific use case: codec tokens for high-quality audio generation, ASR tokens for semantic understanding, and multi-scale approaches when you need both efficiency and quality.
Video Tokenization
The most common way that videos are tokenized today is to turn video into video frames and send the video in as a sampling of images, with audio attached. For example, Gemini currently treats video as sequential images interweaved with text/audio (from online sources, so we can link).
The Future of Tokenization
Researchers are exploring ways to improve tokenization in LLMs:
Tokenization-Free Models: Some cutting-edge research proposes eliminating tokenization entirely, allowing models to operate directly on raw bytes or Unicode code points. This approach avoids the need for a predefined vocabulary, letting models natively handle any language, symbol, or character without relying on heuristics or subword units. However, training such models is more computationally intensive.
Adaptive Tokenization: Instead of using a fixed token vocabulary, adaptive tokenization allows the model to determine the optimal token split dynamically based on the input. This could mean breaking down complex or rare words into smaller units while preserving common phrases or numbers as whole tokens. The goal is to strike a balance between efficiency (fewer tokens) and semantic clarity (keeping meaningful units intact).
Conclusion
Multimodal tokenization extends the concept of text tokens to images, audio, and video. Images get converted through patch embeddings (splitting into grid squares), vector quantization (learning visual codebooks), or contrastive embeddings (CLIP-style). Audio uses neural codecs (preserving sound quality), ASR transcription (converting to text), or hierarchical approaches (multiple resolution levels).
Each method has tradeoffs, and these tokenization choices directly impact what AI systems can understand and generate.
ByteByteGo Technical Interview Prep Kit
Launching the All-in-one interview prep. We’re making all the books available on the ByteByteGo website.
What's included:
System Design Interview
Coding Interview Patterns
Object-Oriented Design Interview
How to Write a Good Resume
Behavioral Interview (coming soon)
Machine Learning System Design Interview
Generative AI System Design Interview
Mobile System Design Interview
And more to come
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.