Baking Lighting in 3ds Max: A General Overview, Part 2

Baking in Max can be a daunting process. Last time, in Part 1 we talked about what you need to get started, options for renderers, a few of the different kinds of bakes you can do, plus how and why to do test renders.

Part 2

This time, in Part 2, we go into the approach for Unwrapping, the details of Render to Texture, and the actual baking.

Part 1 | Part 2

Initial Bake

Unwrap Model
Bakes work by recording the lighting intensity for each point on a model's surface into a texture. This means that, under normal circumstances, you don't want any overlaps or tiling used in the UV channel that you bake to otherwise you may have some very strange results in your baked lighting. While unwrapping by hand produces the best results, automatic unwraps are usually more than enough. Bake UVs are usually unwrapped to channel 3.

There are a few common options for creating lightmap UVs for large numbers of objects. One is to use the automatic unwrap feature of Render to Texture, though this always groups matching material IDs together resulting in a lot of wasted UV space. Another option is to use a scripted tool written in-house called SteamRoller, which does a similar function but isolated from any other dialog and without requiring the material ID option. Paid plugins also exists that can do unwrapping, some of which also have their own special baking options.

Bake Settings
Render to texture is a complex-looking dialog, but as with many other parts of max when looked at individually it isn't so intimidating. The output path tells Max where you want the baked textures to go. Render Settings works with the classic settings dialog. Objects to bake shows you your models and lets you pick what you want to do with them, and the other parts below specify properties about your output.

General Settings
Start off by picking a path. This is typically a subfolder in the path where you're working on your project files. If you archive the project and bring it back, you'll want to have all the baked maps with it. If you're doing network baking, this will need to be specified as network path the render machines have write access to. If you've already saved out a preset from a previous bake, you can load that up from the dropdown, or set the settings yourself in the Render Setup dialog by clicking the Setup... button.



Objects to Bake
This is a list of objects you have selected. You can use the dropdown list to load a preset file for the map types and sizes if you've saved that out from a previous bake. Otherwise, set the padding to at least 16 and carry on down to Mapping Coordinates. Padding sets the amount of border "safe" color around each UV island, which helps keep things looking normal when lower level mip maps are in effect. Since it has minimal impact on render time and most renderers prevent it from overlapping anything it shouldn't, there is no downside to setting it to a "high" value. You can ignore the Projection Mapping option, which isn't typically used in lightmap baking.

In Mapping Options, if you've already unwrapped your models then select "Use Existing Channel" and set it to channel 3, or whatever channel you're baking to. Channel 3, as the default, is most common. If not, you can either use Render to Texture built in option, which clusters material IDs, Steamroller, which doesn't, or the unwrapper of your choosing.



Output
The top part of output shows the list of what maps are going to be made for the current model, or for a selection, their map sizes, and their target slots. This starts off blank, as nothing has been assigned yet, until the "add" button is picked, and element is selected, and its target slot is selected below. Target slot? This is where Render to Texture does things out of order. To know where to put the map, it has to know what the final material is going to look like. While you can set what maps to bake ahead of time, the options that appear in the "target map slot" list is based on the settings in the next rollout below - Baked Material.


Baked Material
The Baked Material rollout has a few options. You can overwrite a slot in your original material (not a great option), or create a shell material to work nondestructively. Shell materials save a copy of both your original material, and create a duplicate that is easy to swap to and get rid of. Just be careful when attaching baked objects together - this can really mess up multi-materials. Within the shell option are a couple other options. Both options store a copy of your original as a "source" material, while the "output" material can either be a duplicate of the original or a new material of your choosing. Usually you'll want to do whichever lets you output to a Standard:Blinn since, at of the time of this writing (Dec 2017), this is the material that is the most taken advantage of for the exporter. See this page on the shader effects that can be achieved when exporting from a Standard:Blinn.



Automatic Mapping
Last on the list is the Automatic Mapping rollout. This is both where you'd set the parameters for an automatic unwrap, as well as some settings that let you base the texture size on the surface area of the model. The "Scale" option is related to texel density based on generic units, but the spinner range is somewhat limited.

Tweaking Render to Texture's Code
If you need larger sizes and are comfortable enough with code to play around with some parameters, you may want to find and edit the Maxscript that Render to Texture is built from, "Macro_BakeTextures.mcr", stored in plain text and modifiable using Max's built in Maxscript editor. This is found in the MacroScripts subfolder of your 3ds Max install directory, e.g. C:\Program Files\Autodesk\3ds Max 2018\MacroScripts\. Specifically, you'd want to be editing the parameters for the spinners in the ~AUTOMATIC_MAP_SIZE~ group. If none of this makes sense right now don't worry about it too much. Just know that it's an option. If you're going to make any changes, be sure to make a backup first.

If you have "use automatic unwrap" enabled way back up in Objects to Bake -> Mapping Coordinates, hitting render at this point will now both unwrap the models and start off the bake. This is usually a bad idea. If you've worked with Max for any length of time, you know it's prone to crashing. Since Render to Texture does not save the model after the unwrapping is complete, if it runs out of RAM or crashes for any other reason while baking your texture, that unwrap data will be lost and any maps that have successfully rendered will be useless. There appears to be some element of randomness in the way Max unwraps its models, so running the unwrap again will not necessarily produce the same result as the last time. We strongly recommend that you unwrap first with whatever method you prefer, then save before baking out the scene.

Test Bakes
Once you know how Render to Texture works, it's time for some test bakes. Similar to test renders, test bakes are bakes done at a low quality setting to detect problems in models early instead of after waiting for a several hour long final bake to finish. These problems are mostly of the same nature as what you're looking for with the initial test renders: things like flipped normals, explicit normals, unwelded vertices, bad/overlapping unwrap areas, overblown lighting, seam artifacts, z-fighting and internal/duplicate geometry. As you make your bakes, a quick way to check out your work in engine is to set up the OSG preview toolbar buttons. Once everything is cleaned up it's time to move on to the final bake.

Final Bake
Now that everything is cleaned up it's time to crank up the settings, set things up to render overnight, then check back in the morning. Everything look good? Good.

Now if you need to do any revisions, the rest of your model will already be good to go. Just make sure that if you need to attach any meshes together that you clear the shell materials first. Otherwise max can accidently nest a multimaterial in a shell, and that shell inside another multimaterial causing Max to become very confused about what's going on. If you undo soon enough you may be able to stop the damage. Otherwise you may need to fix them by hand. If you think your scene may have nested multi materials, you can grab a script that helps find them for you here.

Conclusion

In this guide we covered what you need to get started, how you can test your scene before you start baking, some different approaches you can take, the details of render to texture, and even some ways you can tweak it to improve on it's capabilities. While this is by no means comprehensive in the details of any one specific renderer, the techniques discussed above apply fairly universally. Experiment with the capabilities and quirks of what you have on hand, and you'll be able to take your environments to the next level.