Baked Textures with Machine Learning Feature Enhancements

Baked Textures for Games

Baking of textures is a very common technique to pre-calculate rendering attributes to enhance quality and performance in real-time rendering applications.

In games, it is used to bake:

  • Lightmaps for enhancing shading by capturing complex lighting (e.g. GI, AO, …etc).
  • Texture properties (e.g. Normal maps from high-res models, …etc).

Good quality texture baking depends on many factors. The most important are surface area of the mesh and the output texture's size. These two factors are measured by what is called "texel density". For decades, the industry relied on this unit of measurement to determine the quality of baked textures. Although this is technically correct, practically it still misses several other important contributing aspects.

I would claim that texture quality evaluation is based on human perception on the final composition, not just on one abstract number. The challenge with human perception is that it is very hard to quantify, as there are many factors involved (lighting, shadows, color contrast ...etc) thus it is very hard to implement a tool that can take into cosideration all the contributing factors to generate the best results that would please human eyes.

Intelligent Baking

At INFramez, our pipeline depends on baked textures. So doing R&D on new approaches for this tech is major for our production. Before doing Hyper Void, we were planning for a first-person emotional puzzle/adventure game inspired by Silent Hill. For that project, lightmap baking was a crucial technique for achieving the targeted quality. This made for a good opportunity to invest in researching the previous idea and hopefully materializing a solution for it into a production tool. Due to the nature of the problem, we decided to investigate machine-learning approaches to achieve a computer implementation of this concept. The results from our early prototypes were promising, and with the final prototype the tool was ready for production. Unfortunately, the game project was archived due it is high production costs for our small team. However, the tool was still used in baking lightmaps in Hyper Void.

The following screenshots from Hyper Void shows the panels texture of the spaceship baked with regaluar tool versus the baking with the Machine Learning Enhancment tool.

First image captured from Softimage viewport showing the baked results using the out-of-the-box baking tool:

Second image from in-game showing the same texture baked with this tool:

The main draw-back of the implementation was the processing time. As with most machine learning algorithms, it requires multiple iterations for leaning and evolving. The result from each iteration is used to learn and evolve towards the final bake. Still this drawback was a very acceptable trade for the significant increase in quality and big memory savings on the baked texture sizes. To demonstrate the results of the new technique, I used some of the common test scenes to compare between standard baked textures and textures processed with this new tech we call "machine-learning feature enhancement".

In addition to the significt quality enhancment, this tool is beneficial for reducing production times. The tool takes care of all art-work assoiciated with the texture-baking process (e.g. uv layouting). Just provide the hi-res mesh and get yourslelf a cup of coffee (maybe two since the processing time is longer).

Please note that the exact same render settings were used when processing the textures for the standard case and the enhanced case.

Future Work

As shown in the videos the algorithm was implemented in Softimage few years ago. In that implementation, the processing time for small scenes is relativaly good (usually few minutes). However, for large complex scenes the processing time can reach several hours as with Sponza. Solving this issue can be done by moving the processing to the GPU to cut the processing time drastically. Luckly, this algorithm is very GPU-friendly and should be a straight-forward job to do.

Since our engine has native support for Softimage, the workflow for using this tool through the editor was straightforward too. But I'd love to get a working version of it to the major commercial game engines too (e.g. Unreal and Unity). If this interests you let us know.

The Algorithm in Action

Baking Lightmap with Global Illumination for Sponza

  • Baked Texture Size: 1024x1024
  • Number of Triangles: 110,324 triangles
  • Processing Time: ~4 hrs

Baking Lightmap with Final Gathering for a City Proxy

  • Baked Texture Size: 512x512
  • Number of Triangles: 11,406 triangles
  • Processing Time: ~10 mins

Baking Ambient Occlusion from Hi-Res to Low-Res Model

  • Baked Texture Size: 512x512
  • Hi-Res Number of Triangles: 77,280 triangles
  • Low-Res Number of Triangles: 10
  • Processing Time: ~20 secs

Texture Baking Workflow in Softimage

Comments (4) -

  • Waqas Zia Chaudhry

    11/29/2016 4:41:07 AM | Reply

    Hello,
    This is looking very exciting with amazing results!
    Could you by any chance show us the UV layout for the light maps in addition to showing us the end results?
    It would be very interesting to see exactly what  your algorithm does at the UV level.

    • Homam Bahnassi

      12/3/2016 6:49:31 PM | Reply

      Yes sure... for scenarios where all UVs are disconnected, they are re-sized individually based on some weights found by the algorithm. In case where UVs are connected in certain toplogy, the problem is harder where applying the weights directly will cause bad UV stretching and distortion. Currently, the algorithm doesn't support the later case.

      I'll upload few UV snapshots with heatmaps visualizing the weights.

      Regards,
      Homam

  • Waqas Zia Chaudhry

    12/5/2016 4:45:54 AM | Reply

    Thanks for taking the time to answer!
    Glad to hear you confirming my assumptions about how such a program might work. Would love to see some images. On the off chance you might have some time to answer I would like to ask a few more questions:
    What criteria does the algorithm use to determine what UV islands to prioritize? How does machine learning play into it?
    Is it required to have a higher density mesh  in-game to successfully implement this technique so more UV islands can be created?

    Regards
    Waqas

    • Homam Bahnassi

      12/7/2016 10:40:39 AM | Reply

      No problem...
      - The criteria depends on what the tool is baking. For example when baking shadows lightmap it uses very different criteria from baking a normal map from hi-res to low-res.
      - Currently the tool works more like an expert system than pure machine learning since the available data is very limited. That's why the criteria are pre-defined based on my CG experience. The goal is to replace this section.
      - No it doesn't require high density mesh to implement this technique if used the correct criteria. The building example has very low density (basically box with 5 sides) and it works perfect.

Loading