Don Ott LCAD Game Modeling Class: Mental Ray - Ambient Occlusion walkthrough

When rendering Ambient Occlusion for 3dsMax 2011, the following steps should be taken. Read below for earlier versions of 3dsMax (2010 or before).

First and foremost, ensure that you have no inverted or overlapped UVs in your primary 0-1 UV square. The following blog post can help you with ensuring you move your overlapped UVs outside of the 0-1 square: http://donott-lcad.blogspot.com/2010/02/accurately-offsetting-overlapping-uv.html

Once your UVs are not overlapped or inverted, start by bringing up the Render Setup. (Hotkey - F10)
Switch to the Common tab.
Scroll to the very bottom, and expand Assign Renderer.
Click the "..." next to your Production renderer settings, and when it says "Choose Renderer", choose Mental Ray.
Switch to the Renderer tab, and adjust the Samples per Pixel. Higher = better, but longer to render. I generally keep mine at Minimum:1 and Maximum: 4.

The object you are baking is a single mesh - not a group, not multiple pieces. Everything is attached.
Next, with your object selected, bring up the Render to Texture dialog. (Hotkey - 0)
Make sure you turn "Use Existing Channel" on, and it is set to Channel 1.
Under Add, choose Ambient Occlusion (MR).

Next, specify where you want to save the file by click the "..." next to File Name and Type. 3dsMax will render the image and drop it here when finished. This should be a 24-bit image, no compression or alphas.
Specify how big you want the image to be. (512x512, 1024x1024, etc.)
Bump up the Samples for a higher quality render. The higher the number the more accurate the render, but longer it will take.
Ignore any error messages regarding a Target Map slot.

Ignore the rendered image on screen. Once the render is completed, browse to your computer where you told 3dsMax to save your file. You should have a black and white Ambient Occlusion map.
If you notice issues with sharp gradients or harsh shadows, check your smoothing groups, as well as the number of Samples as mentioned above.
That's it! Take this image into Photoshop and use it as a Multiply layer on top of your textures. Feel free to slightly blur it for the desired affect.