Packing Input Frame Context in Next-Frame Prediction Models for Video Generation

 (lllyasviel.github.io)

123 points | by GaggiX 4 hours ago

6 comments

  • Jaxkr 2 hours ago
    This guy is a genius; for those who don’t know he also brought us ControlNet.

    This is the first decent video generation model that runs on consumer hardware. Big deal and I expect ControlNet pose support soon too.

    [-]
    • msp26 1 hour ago
      I haven't bothered with video gen because I'm too impatient but isn't Wan pretty good too on regular hardware?
      [-]
      • dewarrn1 45 minutes ago
        LTX-Video isn't quite the same quality as Wan, but the new distilled 0.9.6 version is pretty good and screamingly fast.

        https://github.com/Lightricks/LTX-Video

      • vunderba 37 minutes ago
        Wan 2.1 is solid but you start to get pretty bad continuity / drift issues when genning more than 81 frames (approx 5 seconds of video) whereas FramePack lets you generate 1+ minute.
  • IshKebab 2 hours ago
    Funny how it really wants people to dance. Even the guy sitting down for an interview just starts dancing sitting down.
    [-]
    • jonas21 23 minutes ago
      Presumably they're dancing because it's in the prompt. You could change the prompt to have them do something else (but that would be less fun!)
    • Jaxkr 1 hour ago
      Massive open TikTok training set lots of video researchers use
  • ZeroCool2u 3 hours ago
    Wow, the examples are fairly impressive and the resources used to create them are practically trivial. Seems like inference can be run on previous generation consumer hardware. I'd like to see throughput stats for inference on a 5090 too at some point.
  • WithinReason 1 hour ago
    Could you do this spatially as well? E.g. generate the image top-down instead of all at once
    [-]
  • modeless 1 hour ago
    Could this be used for video interpolation instead of extrapolation?
    [-]
    • yorwba 1 hour ago
      Their "inverted anti-drifting" basically amounts to first extrapolating a lot and then interpolating backwards.
  • fregocap 2 hours ago
    looks like the only motion it can do...is to dance
    [-]
    • jsolson 52 minutes ago
      It can dance if it wants to...

      It can leave LLMs behind...

      'Cause LLMs don't dance, and if they don't dance, well, they're no friends of mine.