(Author here.) Maybe it's worth moving to a GitHub issue. Try `--model=small`. The demo server limits the number of pixels to around 320x200 or 256x256 and can do only 4 at the same time to fit in RAM.
(Author here.) Unlike most other non generative adversarial network (GAN) approaches to super-resolution, it does try to inject high-frequency detail; see the faces example on GitHub. But I tuned down that parameter in the released models a bit so it performed better generally.
(Author here.) If you have the luxury to train on domain-specific textures, the results will definitely be better. That's why I included all the training code in the repository as well—to allow for this kind of solution.
If you scroll down on GitHub to see the faces examples, those are achieved by a domain-specific network. I suspect you'll similarly get extremely high-quality if you have good input images.
(Author here.) Did you see the faces example on the GitHub page? It was a domain-specific network trained adversarially for that purpose, but I have yet to see any super-resolution of that quality with or without machine learning.
Most other approaches don't even try to inject high-frequency detail into the high-resolution images because the PSNR/SSIM benchmarks drop. Until those metrics/benchmarks are dropped, there'll be little more progress in super-resolution.
(Author here.) Yeah, I knew this would come up but decided to proceed with the pixelated comparison anyway. I couldn't get the GIFs to reflect the results because of 8-bit quantization/dithering. The images show the neural network inputs and outputs, not a comparison with other super-resolution algorithms (still fascinating :-).
I'm working on the Docker instance now, that should help anyone with interest/experience in the field compare results easily.
(Author here.) Absolutely! Using multiple super-resolution networks, not only continuity would present problems, but also blending between different regions. I agree there's a lot of value for domain-specific networks here, as you can see from the faces example on GitHub.
I'd be curious to see an ensemble-based super-resolution, where each model can output the confidence of a pixel region, then have another network learn to blend the result.
Conversely, these results are achieved using a single top-of-range GPU. Everything fits in memory for a batch-size 15 at 192x192. By distributing the training somehow, you could make the network 10x bigger and train for a whole week and likely get much better general purpose results.
(Author here.) My biggest insight from this project is that super-resolution with neural networks benefits significantly from being domain specific. If you train on broader datasets, it does pretty well but has to make compromises. Many recent papers do a comparison in terms of pixel similarity (PSNR/SSIM), and using those metrics the quality drops because high-frequency detail is punished under those criteria (even though it may look better perceptually). Reference: http://arxiv.org/abs/1609.04802
On GitHub, below each GIF there's a demo comparison, but on the site you can also submit your own to try it out (click on title or restart button). Takes about 60s currently; running on CPU as GPUs are busy training ;-)
> super-resolution with neural networks benefits significantly from being domain specific. If you train on broader datasets, it does pretty well but has to make compromises.
To what extent could the need for this trade-off be overcome with a larger network?
You can use any image as source, but to create annotations you have to do that yourself currently. Using simple segmentation libraries (or clustering) can do a good job for certain images, or look at better solutions for semantic segmentation: http://gitxiv.com/search/?q=segmentation
From that perspective, this research is two steps further than Neural Style, I wrote about it yesterday here: http://nucl.ai/blog/neural-doodles/
First, the paper I call "Neural Patches" (Li, January 2016) makes it possible to apply context-sensitive style, so you have more control how things map from one image to another. Second, we added extra annotations (which you can specify by hand or from a segmentation algorithm) that helps you control exactly how you want the styles to map. We call that "semantic style transfer" (Champandard, March 2016).
You're right about it being hard otherwise, it was for many months and that's what pushed this particular line of research! Try it and see ;-)
This reminds me of "If Edison didn't invent the light bulb, someone else would have: there were thousands of other engineers experimenting with the exact same thing, a natural next step after electricity came about" (-- paraphrased from Kevin Kelly)
One was called Swan, Edison tried to sue him for patent infringement but Edison's lawyers warned him about prior art, so instead he negotiated a joint venture.
