It should work, I don't think you need to bother with training it on individual performers. Someone made a thing like this to improve low res anime, that worked well.

In theory you could use this to increase temporal resolution as well. Turn 24 fps movies into 60 fps, and upscale regular HD to 4k.