> [in 2020/2021] the dominance of autoregression was not as well-established as it is today: GPT-3 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022
I disagree with this. Decoders clearly worked. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.
But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass.
So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.
I disagree with this. Decoders clearly worked. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.
But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass.
So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.
also haven't read too deep into the deepseek agent harness but the math in there was really cool. it sounded promising, at least.