Unfortunately I don't have Maya at home, so I can't replicate the process to post here. However, I'm inclined to give Blender a spin and see what I can do with it. It is also possible to fake the ambient occlusion (the nice shadowing effect) by layering images with blending modes. One method I've been experimenting with that draws upon Eduard Imhof's principles is:
- Hypsometric tint layer, blended with multiply, overlay, or color, depending on how it looks. The color blend mode preserves the contrast of the base hillshade the best, I think.
- Focal statistics process on the DEM, then hillshaded. Use this to simulate atmospheric haze.
- Slope layer by percent, represented as a dark gray to white color ramp. Blended with multiply and 20-30% opacity. This is the layer that fakes the ambient occlusion that Blender applies. Also note that shadows are never black, so don't use a black to white color ramp. The multiply blend mode will drop out the white and all that will be left is a nice shadowing on the relief.
- Hillshade, normal greyscale hillshade.
I've only attempted this process with ArcMap and Photoshop, but it seems like it would work in any GIS and image manipulation package.