The problem with udi's pipeline is that you have little control over video output, and to have more control you have to add more detail to the scene, and to add more detail to the scene you eventually have to build the scene itself, which this other pipeline actually solves:

Going video -> scene is much better because you can control it much more granularly, and then you can also add udi's pipeline on top if you so desire, therefore this latter idea seems like the more correct move. Also just look at it, this is pretty soulful AI jank, which will get hill-climbed over the next years but enjoy it while it lasts.