Researchers at Intel labs have been looking at enhancing the realism of synthetic images. Stephan R. Richter, Hassan Abu Alhaija, and Vladlen Kolten recently published a research paper and accompanying demo video showing a new ML-based technique in action – changing the great but obviously synthetic graphics in GTA V to a much more realistic and photoreal experience.
The new ML rendering method is described as enhancing the in-game graphics by leveraging a convolutional network that has been trained on the ‘Cityscapes dataset’. The results are clearly different and appealing in a real-world way. However, it is explained that the ‘Cityscapes dataset’ isn’t derived from driving around Southern California, but mostly from driving around cities in Germany with an automotive grade camera. One unintended side effect of using the German data is that the landscapes in San Andreas become greener, and the roads smoother and less pot-holed. Other effects like the shading and lighting of cars and buildings translate very well, but I think SoCal-centric data would make the whole photorealism effort a bit sunnier.
Above you can see a diagram which outlines the process each frame will go through in order to apply the new Intel Photorealism Enhancement process. A very important part of that flow is the use and analysis of game rendering buffers called ‘G-buffers’ which help the ML work by categorising. These buffers help by revealing data such as object distance to the camera, scene materials, light positioning, and object IDs – thus trees in the scene will be treated and enhanced differently to cars.
The Intel researchers learnt a lot about optimising this type of Photorealism Enhancement in the process of implementing it in GTA V, and that helped them refine this ML technique as well as proposing a new strategy for sampling image patches during training, and introducing multiple architectural improvements in the deep network modules used for photorealism enhancement. The new technique also stands up well against other photorealism techniques with “substantial gains in stability and realism” in comparison.
If you want to read more and play with some GTA V photorealism image comparison sliders then head on over to the project’s official GitHub page.