architecture comparison

Andi Marafioti ✓ @andimarafioti · 1h I trained two VLM architectures side by side on the exact same data, and the difference was… almost nothing. For my upcoming O'Reilly book on Vision-Language Models, I wanted to go beyond just citing papers and actually show readers what happens when you compare cross-attention VLMs (Flamingo-style) to unified sequence VLMs (SmolVLM-style). Same vision encoder (SigLIP2), same LLM (SmolLM2-135M), same training loop, same packed batches from FineVision. The cross-attention model has a Perceiver Resampler, gated fusion layers, 15% more parameters, and a considerably more complex forward pass. The unified sequence model only has a single linear projection. After 5,000 steps with all parameters unfrozen: nearly identical loss curves. 👇 The takeaway: at this scale with trainable backbones, the architecture is not the bottleneck. Data, learning rate and batch size dominate. The LLM adapting to the task overwhelms whatever advantage one fusion strategy might have over another. That's why the unified sequence approach has become the default in Qwen2-VL, DeepSeek-VL, SmolVLM, and most recent VLMs. Same results, fewer parameters, far simpler code. The book covers full implementations of both architectures with runnable training comparisons. More details soon! Let me know if this matches your intuitions or if you've seen different results at larger scale. What do you think? [Embedded chart: line graph titled with legend "Self-attention loss" (blue) vs "Cross-attention loss" (orange), y-axis "Loss" from 1.0 to 2.5+, both lines closely overlapping and declining together over training steps]