Andi Marafioti

Andi Marafioti ✓ @andimarafioti · 1h I trained two VLM architectures side by side on the exact same data, and the difference was… almost nothing. For my upcoming O'Reilly book on Vision-Language Models, I wanted to go beyond just citing papers and actually show readers what happens when you compare cross-attention VLMs (Flamingo-style) to unified sequence VLMs (SmolVLM-style). Same vision encoder (SigLIP2), same LLM (SmolLM2-135M), same training loop, same packed batches from FineVision. The cross-attention model has a Perceiver Resampler, gated fusion layers, 15% more parameters, and a considerably more complex forward pass. The unified sequence model only has a single linear projection. After 5,000 steps with all parameters unfrozen: nearly identical loss curves. 👇 The takeaway: at this scale with trainable backbones, the architecture is not the bottleneck. Data, learning rate and batch size dominate. The LLM adapting to the task overwhelms whatever advantage one fusion strategy might have over another. That's why the unified sequence approach has become the default in Qwen2-VL, DeepSeek-VL, SmolVLM, and most recent VLMs. Same results, fewer parameters, far simpler code. The book covers full implementations of both architectures with runnable training comparisons. More details soon! Let me know if this matches your intuitions or if you've seen different results at larger scale. What do you think? [Embedded chart: line graph titled with legend "Self-attention loss" (blue) vs "Cross-attention loss" (orange), y-axis "Loss" from 1.0 to 2.5+, both lines closely overlapping and declining together over training steps]

Andi Marafioti @andimarafioti OpenAI's latest repo has an interesting 3rd top contributor. [Embedded screenshot of GitHub repo "parameter-golf" (Public), description: "Train the smallest LM you can that fits in 16MB. Best model wins!" MIT license, 3.6k stars, 30 watching, 2k forks, 24 contributors. File list shown (data, records, .gitignore, LICENSE, README.md, THIRD_PARTY_NOTICES.md, requirements.txt, train_gpt.py, train_gpt_mlx.py). A contributor popup shows "claude Claude — Committed to this repository in the past week" with a Follow button, indicating a "Claude" GitHub account/bot as an active contributor.] 8:46 AM · Mar 23, 2026 · 90.3K Views

Andi Marafioti (@andimarafioti), 10h: This is such a beautiful way to present ablations. Kind of jealous tbh [Embedded image — "Figure 2: Robust Image Pretraining" bar chart, table format] Columns: Robustness avg of 6 / ImageNet val | Training ZFLOPs 1. Baseline — 75.3 / 78.9 — 1.0 2. Prog. Res — 75.1 / 78.9 — 0.5 3. Batch Sz — 76.2 / 79.5 — 1.1 4. LAMB — 76.9 / 79.9 — 1.1 5. High Res — 78.3 / 80.4 — 1.2 6. RoPE — 79.2 / 80.7 — 1.2 7. Attn Pool — 80.1 / 81.0 — 1.2 8. Data Aug — 80.8 / 81.1 — 1.2 9. Mask Reg — 80.9 / 81.3 — 1.2 Figure caption: Figure 2 Robust Image Pretraining. We tune our pretraining recipe (§2.1) to maximize performance on a fixed set of data, starting with an OpenCLIP [49] ViT-L/14 model. We report cumulative zero-shot classification results for each modification. The inner bars show robustness evaluation, calculated as the average of 6 robustness benchmarks [4, 24, 44, 45, 109, 138], and the outer bars show ImageNet val [24] alone. Several changes significantly improve robustness, indicating that ImageNet val scales more with data, while robustness can scale with refined training techniques.