Forces of Light and the Geometry of Shadows

[teemu_om 47]

Hopefully this is the right place to ask...

I switched to Mental Ray and now I'm having a problem with lightning when I try to export my scene. The first view renders fine, but after that the renders start getting gradually darker with each new view. A picture speaks more than...:

Those are from left to right: Zoom 5 south, Zoom 5 east and Zoom 5 west. Those are all day views (but this issue concerns night views as well). It goes on and on like that until it's completely black (except for the self-illuminated night materials) somewhere around Zoom level 3. Do you have any idea what could be causing this?

I use Max 9 (Mental Ray 3.5) and I use a custom light rig. Here are my settings:

[SimFox 804]

Oh yeah!
You come to the right place, kaveri!
The issue is caused by the tinting used for night view rendering not resetting to normal day value (255,255,255) this is a result of some issues in the export script. It could, however, be fixed without getting into script at all. All you need to do is to place at least one light into the scene. It could be somewhere far outside of your scene and be set to any low value the very presence of light will force tint to reset. However, and THIS IS IMPORTANT there are some issues that are cause by the combo of light and X-REFED MR Sky as it will fail to be taken into calculations. By trial I've found that this situation is resolved by either placing SKY into the scene instead of it being a part of the rig, or by placing LAST light into the scene in the TOP view !! Don't ask me why I know it sound really bizarre but there you have it! Since I've found the "solution" rather quickly I never went to farther investigate the problem or look for "proper" solution.
One more note.
Placing Sky into the scene (if you do it in accordance with daylight system working principals will significantly alter the look of you render. This is caused by the fact that Sky proper position isn’t respected when your Daylight system is X-Refed and it resets to the top down state.

I would advise you to change the haze value of the sky (this will control the one of Sun as well at least if both are together in daylight system. Default values of 1 produce too harsh of a rendering.  But that is my personal view.

Hope this is of help, if you have further questions – ask!

[teemu_om 47]

Many thanks, mr. Fox! I'm going to try your advises when I get home. About haze... If I remember right, the default value was actually 0 and I raised it to make the shadowed areas a bit brighter (is there a better way to achieve this?). Also, what exactly do you mean by harsh, should I lower the haze value or make it higher?

Again, kiitoksia very much!

[Jasoncw 12,060]

I've moved this thread from the BAT - Workshop to here at the request of SimFox.  Remember everyone, if anyone (you don't need to be the thread author) wants to continue the discussion in a thread, just PM me about it, and I'll move it to the BAT - Open Discussion from the BAT - Workshop.

[ILL Tonkso 1,239]

Just one question, i created my nightmap, how do I apply it to certain materials?

[Jasoncw 12,060]

Well, if you made a material library with a certain material in it, in your scene, you make a material with the same name (or whatever the naming convention is, I forget), and it will automatically be swapped out during the night renders.  In the BAT Export rollout it will say if there is a night material library loaded or not, which would initially indicate if things were operational.

[SimFox 804]

Jason I think he asked where shall he put that Texture he made inside of the Material.

Ill:

Material have to be self-illuminated. there are several ways to make it, or to be precise several type of materilas that could be self illuminated - standard, Architectural, A&D. You can also build you own on the basis of either of those. But I think it would be safe to start with something ready.

So put your texture you've made into defuse slot and also into self-illumination slot.If you make it from Standard material make sure that defuse slot color swatch is BLACK! this is very important as it with prevent sun or any other light for that matter affecting bitmap. Architectural is better fitting to make light fixtures as it has proper brightness control. Also it is key to use MR with FG in order to get best resuilts.

[ILL Tonkso 1,239]

Thanks, Jason got the right one, great tutorial btw Simfox

I have working nightlights now, so im just setting up a lighting rig now to make it look like a proper NDEX night building.

Edit: Just rendered, the nightmaps show up in the render, but when i use the building ingame, only th omnis in the lobby light up, no window maps :S

[Jasoncw 12,060]

I think that in order to show up, they need to have some kind of light hitting them, so add some self-illumination maybe, or do the include/exclude nitelite method but at a low intensity.

[SimFox 804]

Per Pixel accurate "preview" windows sizes for 3ds Max. Setting your output windows in render properties to these values will give you exact replica of the size you'll see in game. That is, naturally with appropriate camera selected for render.

Zoom Level	Output window width
zoom 5	793 pix
zoom 4	797 pix
zoom 3	769 pix
zoom 2	974 pix
zoom 1	700 pix

Output window of 793 pix may prove insufficient for some large models at zoom5 level. Luckily you can do a trick with zoom4. Both cameras are identical in their location and as such in a view angle they provide. Zoom5 is roughly 2 times large then zoom4 (2,0134 times larger to be precise) so you choose zoom4 camera and compensate the zoom difference (60,2mm vs 29,9 mm) by increasing the width of you output window to 1604 pixels.

This works ONLY with zoom4 and 5 cameras. all other cameras are placed in unique locations and as such are NOT interchangeable

[SimFox 804]

TruNite tutorial

The whole concept of truNite came out of two not directly linked ”needs”.

First one was to make night view more realistic by doing away with the strong shadows present in the original Night view. To turn day into night game’s simply dropping darkish-blueish overlay over the day view not unlike seeing things through the darkened glass – method commonly used  in really old movies when they filmed night during the day with really strong filter.

Second was the nature of how such a view of a building with nitelites was assembled by the game. In fact what you see there in a game is sort of sandwich. First and always you have your day view. Then on top of that you have layer of RGB 128 128 191 (that overlay I’ve mentioned earlier) set to multiply (those familiar with Photoshop will understand those not just think of it as a blue glass). And then, as a third layer, you have fragments of night render chopped out by the mask representing the areas affected by nitelites. The process of generating this mask and particularly compressing it into the FSH causes noticeable banding to appear. It is particularly obvious on large flat surfaces.

Such a lighting mask is blended using transparency mode. So you still have that banding in the resulting game view. More than that, this nasty effect gets amplified since you have first banding in compressed texture on top of banding in the mask!

Here I have to make a little “side trip” to point the differences in export process as it’s implemented in GMAX (original BAT) and Bat4Max:

GMAX

Export, when nitelites are present in the scene, does only ONE night pass. It renders building with all the lights in the Lighting Rig off so all you see are the areas that are light by your nitelites. This image is saved to be used as night view overlay in a final sandwich. Lighting channel (special max/gmax term meaning the parts of image where light has effect) is used as a mask. Process is very straight forward and taking the full advantage of the way night view is presented in the game.

MAX / Bat4Max

For some reason very different approach has been implemented here. I don’t understand why (since in the end nothing new or improved has been done to the night pass - night library is just an implementation of the nightWindows). I couldn’t find the original thread, where creation of Bat4Max was discussed years ago, so this choice remains a mystery to me. Anyway, what is important is that night view render had ballooned from one pass of partial rendering - totally dark areas don’t add (in practical terms)to render time to 2 full passes. One of those passes does render full night view, but, in a given system, is absolutely pointless as only areas affected by the nitelites will ever be displayed. This pass is followed by separate render of night mask. This means about 3 fold increase in render time, which was, naturally offset great many times by far faster rendering speed of Max as compared to GMAX. Still, it was inefficient and error prone solution. Takes far longer than necessary, doesn’t improve quality, and, on top of it all, results in much larger files putting strain on memory... But it has a great benefit (at least for me personally) as it revealed the mechanics of the whole process.

At first I thought to limit the dreaded banding by including the full night view (no need of superimposition of double compressed images). This can be accomplished by substitution of full day mask in place of partial mask of the night mask pass. As an additional benefit it will do away with transparency blending, making it easier on CPU when playing the game and will not result in increased files sizes as all the needed information was in them in the first place.

But thinking of it more brought me to, basically, obvious conclusion – since we could have an independent night view, why don’t we do something about it, improve it. And that’s how the idea of truNite came to be – to disable “sun” during the night render to get rid of the unwanted shadows. One thing to be pointed out -this is ONLY possible with Sun+Sky type of lighting rig and not with the default one.

So, there you have it, a brief (well relatively) history of the truNite, now to the mechanics of implementing it.

There are two (broadly speaking) ways to do it.

First one involves doing the regular export and later getting rid of all the night time masks and substituting the day time ones in their place. Good thing about this way is that you do the same old export and don’t really need to learn anything new, bad, well, everything else! It still results in much longer then needed render times, goes through all the process (very error prone) of generating something you will discard. So, what’s the point in that? I personally don’t see one. Also in such an “old “way you effectively have both day and night scene in the same file. That adds to the confusion, naming problems (which often result in failed exports) and inconvenience in previewing your work.

Second way does away with all that at the expense of little extra work (really as little as merging your objects into a new file and copying and moving some files around). Basically you go about modeling your scene as usual and keep it day only! Then you copy entire (or not - that’s up to your creative vision)scene into new empty file and use it as your night scene – set all the lights, self-illuminating materials etc, as you want. In the end you’ll be doing what is in effect 2 day exports. Well there will be a third one (just a technical one to generate needed set of *.fsh.TGI files – but that could be done in a minute or two even for a VERY large model) and then you’ll do some file moving – the process that could be also automated to take only few seconds.

I will not go into details of first approach since there isn’t anything new in it and the information of file moving/renaming that will be covered in description of Second process is fully applicable to the first one. In the end, it is your personal choice which one to follow.

I must mention here additional benefits of the second approach – having 2 separate day exports instead of single day/night one. Those are:

1. Saving time. Such a 2 stage solution is about 30% shorter in total than the first one.
2. If you do have a very large model, or your computer isn’t too powerful such twofold export  instead of occupying it for say 24+ hours allows you to do each export during the night, leaving computer free for other tasks during the time you’re awake.
3. Vast majority of the errors during the export process are happening during day/night switch. Eliminating it altogether makes the whole process far more reliable.
4. This is very much WYSIWYG approach. You can easily preview your night view, and alter it as you please to farther enhance day/night difference, perhaps even on the level of some modeling - say close doors or sun umbrellas on the patio for the night views. This would be very complicated if not impossible to do (not without some scripting anyway) with the traditional single stage export.

 

So, now to the instructions. Here is what you would need to do:

1.

Create your scenes, the day one, and,  as a copy, the night one. This could be done either by merging the entire content (or just needed) from the day scene into the new file, or into the night template (with preset lights and rendering settings), or by simply using Save As command and creating copy of your day scene as a night one (you would have to alter the all necessary settings manually).

2.

Necessary changes. The only really necessary thing to do is to get rid of the sun. There is couple of ways to go about doing that. One way is to do to you Lighting Rig file and setting sun intensity to 0. Simply turning it off may not work, as the export process will turn it back on. Alternatively, you can have the rig inside of your scene. This method has additional benefits as it’s, again, reduces possibilities of errors during the export process. I’ll make another mini-tutorial on this subject later on. The point that I’m trying to make here is that you should simply get rid of sun one way or another.

3.

The EXPORT process (the part of it that takes place in MAX) is in fact a mere generation of images that will later be used to texture the simple LOD shells that exist in game (that is an extend of 3d in SimCity4). The rendered “views” are chopped in pieces no large the 256 pixels on a side, compressed With DXF compression and additional information attached to them (like TGI – Type, Group, Instance numbers)and all are put into FSH files that are used within SC4Modle file (that is in fact just a mini database). To put long(ish) story short what we should be concerned with are FSH files.

To generate them you should do normal “day” export. When process is finished do click on FSH Batch Build in Batch CMD rollout. But DO NOT click on DAT FSH Insert at this time! Remember that!!

4.

Now you have the necessary set of day view FSH files. Before you proceed with our second, “night” render you would need to save these into safe place as otherwise they shall be overwritten. When you export the model the script creates a folder titled with the name of your SC4Modle file in your OUTPUT Folder of BAT. Usually address looks something along the lines of … gamepacks\BAT\Outputfiles…  That’s where those FSH files, we’ve have just created, “reside”. Find them and copy (or move – doesn’t matter) to safe place. Remember to mark/name everything in clear and logical way so not to get confused later on.

5.

with “day” FSHs safely out of the way do “night” export. Technically it is another “day” export but with the “night look”.

Now some explanation are in order to clarify what makes a day or night view/FSH for the game. Technicality/practically this difference is marked by the Instance ID – the last in the 3 groupings of HEX numbers making up the names of FSH files. Game’s day FSHs have “0” as a fourth digit from the end and game’s “night” FSHs have “8” in that place. For instance, say Instance number 00510120 means that this is a “0051” (model number) “0”(day) “1”(Zoom2) “2”(rotation2 – east)) ”0”(first slab). Same slab for the night view will have Instance ID 00518120. This is why some renaming will be necessary.

6.

Save new “night” FSH just the way you did with day ones.

7.

Now we would need to do a third, technical render. The only need for this comes from the fact that in order to stuff our FSH into the SC4Modle file program doing it needs and set of instructions – namely those TGI values. These are kept in .fsh.TGI files. Irony is that the entire content of each file is just the same set of numbers/letters that is in the FSH file names. This makes me believe that there must be another way, one that doesn’t call for the third render at all, but, as of now, I haven’t found it (well haven’t been really looking, if it will be discovered, or created, I’ll make it a public knowledge.)

But for now 3^rd, technical render is what could be used to accomplish that goal. This render can be made very simple as we don’t need any actual images – those could be just black… So, to save maximum time (and compress 10 hours export process into 1-2 minute one), make a copy of one of the LODs in the scene and name it anything else, say “Model”, or “BAT”. Merge this object along with 3LODS (LOD3, LOD4 and LOD5) into new scene. Set all the render settings to absolute minimum. The renderer you use is irrelevant. To save extra time you can also disable (buy setting intensity to 0) ALL the lights in your lighting rig. But this time rig HAS TO BE EXTERNAL to the scene, so keep it as Xref (the way it is by default). And press the Render BAT button.

8.

When export process is finished, leave it as is for now. DON’T press “FSH Batch Build” or “DAT FSH insert”!!! You can either close Max for now, or keep it open. Now we will have to move some files! Latest 3^rd render had created the needed set of *.fsh.TGI files, now we need to get our FSH files from first and second export back in the model folder in Outputfiles. But prior to that we have to rename our “night” FSHs so that DatCMD.exe and SimCity would understand that those are really the night views.

We can do it with Easy, Quick and VERY Flexible renaming utility called Bulk Rename Utility. And the most beautiful part of it is that BRU is a freeware! You can download it from here (http://www.bulkrenameutility.co.uk/Download.php).

This tool has option of using so called regular expressions (http://en.wikipedia.org/wiki/Regular_expression) in order to specifie the range of file/folder names to be selected and renamed. If you are not familiar with the concept and are not looking to get familiar with it I’ll provide a simple template you can use to do all the renaming you’ll need for truNite export.

As I’ve already briefly described earlier, we need to change fourth digit from the end in our “night” FSH from “0” to “8”. To do so open Bulk Rename utility, browse to the location of “night” FSH (that‘s why it was necessary to save them in a place you would remember and could easily find). Select one (for testing purpose) of FSH to be renamed and in a frame “RegExp(1)” in a match filed type:

(whatever comes before fourth digit from the end of filename)(0)([0-9]*[a-z]*)

and in the Replace filed:

\18\3

check the “include Ext.” box

here is an illustration:

9.

Now you should see resulted name in green type (if everything is right) in a “New Name” column. If resulted name is in red type that means you made some mistake. Assuming the name is in green, select all the files that need to be renamed, make sure all new names turn green and click on rename button. Voilá! All files, be their number 20 or 2000, are renamed in a split second!

Now we shall move/copy them as well as “day” FSH back to the where they belong – Model folder of Outputfiles.

10.

Open Max again, or if you kept it open just activate its window. Place SC4Modle File name into SC4Model Name filed (if it’s not there already) and now go to the Batch CMD rollout and click there on DAT FSH Insert.

you are done! Your truNite model is ready to be moded.

[patriots_1228 51]

good to see this thread active again. It is a very important source of knowledge.

Now I have no excuse not to trunite my next building

[bixel 1,228]

AWESOME!!!! This tut needs to be packaged with BAT4Maxv3 (when it comes out). When will we stop making this game better?????

(prolly when Cities XL comes out...!)

[Jasoncw 12,060]

Cool!

[Odainsaker 2,137]

Thank you greatly SimFox, the whole process actually reads very straightforward. More importantly, just having a separate nighttime scene just seems to make so much more sense than the current system of a single scene switching and coordinating between day and night. And it will even be a WYSIWYG nighttime render...we all kowtow three times!

[SimFox 804]

Odainsaker; I thought so! (I mean about having day and night separately that is)

BTW Yesterday night I sort of started tinkering with scripting some of the necessary steps of truNite export... Well it seems that I've managed to get rid of 3rd tech export and any need for ANY external tools. all could be done from withing the max with just couple of mouse clicks!

I have all the base scripts ready, but they would need some polishing and some interface and by the weekend I believe there will be almost fully automated truNite export tool...

[nihonkaranws 89]

I think I get it.... 'one way or another'? How do I get rid of the sun? Everything else seems to make sense at this moment. Thank you SimFox.

Edit: Going over it again... I came across another issue. You said that Trunite has nothing to do with self-illumination, right? Okay, then (when exporting the night scene), how do I make the window textures bright / illumination?

[Jasoncw 12,060]

To remove the sun, you open the CamLightRig and select the light(s) that represent the sun, and then you delete them.

In the material editor, find self-illumination, it's one of the first things to the right. You simply make a material with self-illumination. His night library tutorial says more about the specifics and particular techniques you can use.

[nihonkaranws 89]

How do I open CamLightRig? Is it a scene?

[Jasoncw 12,060]

Yeah

[bixel 1,228]

good moscow high rises in this pic

http://www.skyscrapercity.com/showthread.php?t=518749&page=2

[SimFox 804]

Gamma control in render output

It is commonly assumed that gamma correction/color mapping is only required to bring extremely bright light sources (like photometric Suns and Skys) to “normal” levels, to, sort of, dim them down. But those controls do much more than dimming. For various reasons practically all display units in use today do not show gradation of brightness in a linear fashion. Instead they have sort of a hole/dip in the midrange. To compensate for such a dip virtually all image standards (JPG, BMP, MPEG etc) include various methods to expand this midrange (brightness wise) to restore the linear appearance.  But here is a trick – standards include assumption of such a compensation and not automatic application of it. What this means is that data in image file is assumed to be non-linear. It is assumed that such a correction was applied at the source – be it Camera, or scanner.  And it is almost always that case, almost… One notable exception is a render output in many 3d software (especially older versions of it). The output data is normally linear. As a result it will be displayed in “un-natural” way. One reason for allowing such a situation is that render output is assumed to be an intermediate rather than final “product”. Additional processing in image editing software is expected. Sometimes even this may not be necessary – particularly in case of studio style object renderings where lights are placed to achieve exact lighting solution that may have nothing to do with “natural” look, which isn’t an objective at all. In other cases some “shortcut” solutions are devised. One of such “shortcuts” is a default lighting rig of the BAT. It compensates for otherwise totally black shadows (in part due to absence of bounced light and in part for the linear nature of render) by having special lights that set not to generate shadows and as such be all penetrating.

Of course default rig is ancient (in 3d terms) technology that has tons of short comings and limitations and results it produces are very bad unless you building is a simple box. But as CPU power and software develops it was natural to move to more natural Sun+Sky solutions. However here is a catch. This move wasn’t always coupled with introduction of Gamma correction. As a result output of such rigs although vastly improved over old one, continues to have one serious weakness. Shadows tend to be very intense and unnaturally fall off into almost complete blackness in the narrower recesses, or otherwise shaded from both sun and sky, spots.

Applying gamma correction to the render output would restore natural look (as eye would see it in the real life situation) by artificially skewing limited dynamic range of final image (of course in case that this is final 8 bit per color channel one) to fool our perception.

here is our “Tester” with Sun+Sky Rig with no Gamma Correction (brightness of either light source had been reduced to 0.018 to compensate.

I think you would agree that these shadows don’t look natural by a long shot.

Now if I would take this image into Photoshop and adjust it’s gamma by 2.2 we’ll have this:

Definitely much more natural looking result. Precise value may range between 1.6 and 2.5 but the principle is illustrated… It also pushes any non-white parts brighter but the resulting shift is smaller in the brightest parts. Final result could be controlled through various settings, I’ll show some later on.

Picture above had is a result of very crude manipulation in Photoshop. Crude, because those were performed on already 24bit (low dynamic color depth) images. Be they made on proper High Dynamic range images like those that Max ALWAYS generate during the render it would look much more balanced.

Luckily for us modern 3d packages have various tools that such a manipulation on results right during the render. Here I’ll consider few options available to 3ds Max users.

1. Gamma correction in Preferences.

Customize -> Preferences -> Gamma and LUT

This is the easiest and, naturally, worst way to introduce gamma correction to render output. It doesn’t allow for extreme brightness of the Sun and Sky (although those could be set to reduced brightness), has no fine tuning options and is not compatible with BATs prior to v2.5 In all fairness I wouldn’t consider it to be an option at all, but since it is there I thought to mention it, especially because it could be a very useful tool in conjunction with other methods.

Weakness:

blunt, no precise control/tweaking, not compatible with BAT versions earlier then 2,5

Strength:

affects material editor and hence allows for quite accurate preview of colors and textures. This becomes powerful feature when used with other compatible methods that allow for fine-tuning of the results of gamma correction.

 

[SimFox 804]

 

2. Logarithmic Exposure Control.

Rendering -> Environment -> Exposure Control Parameters

This tool allows for the more precise control over the look of the render output, available in practically all versions of MAX and compatible with both Scanline and Mental Ray.

Unlike enabling Gamma in Preferences Logarithmic exposure control gives user to adjust  some very handy settings – brightness, contrast, color balance and, very importantly, to adjust for extremely bright natural light of photometric sun and sky. However those controls are somewhat limited.

Here is a result with Logarithmic Exposure control:

Strength:
extremely flexible color correction option.

Weakness:
doesn’t work – doesn’t cooperate, to be precise,  with gamma settings in Preferences and so doesn’t allow for accurate preview of either color or textures in material editor without great difficulty.  Limited controls over highlights and shadows (only general contrast and brightness controls). Numbers appear to be absolutely arbitrary. Actual Gamma correction is unknown, but appears to be somewhere around 2.2 and not changeable.

[SimFox 804]

3. Mr Photographic Exposure Control.

Rendering -> Environment -> Exposure Control Parameters (3ds Max versions 2008 +)

Most flexible and evolved gamma correction tool available in Max to date. It was a ripe time when Metal Ray would take control over such a crucial step in the process in its own hands. Result is one of the best Gamma corrector/Color Mapper out there. It is highly recommended to use this option if it is available in your Max version. It offers greatest number (as compared with previous choices) of settings that could be used to fine-tune the outcome. Cooperates well with Gamma settings in Preferences.

 

Strength:

Very tunable, has separate controls for highlights, mid range and shadows. Control for saturation. Numbers are linked to the settings of the SLR cameras and should be as such intuitively familiar to those who had used them. Fully compatible with Preferences Gamma settings and hence allow for accurate preview of color and textures in Material editor. Actual gamma correction is 2.2 but is linked to the Preference settings and could be over-written from there to any value.

Weakness:

White balance option is somewhat limited only allowing for Kelvin temperature adjustment without of hue shift (too bad as any decent dSLR has hue shift option and here it would be extremely easy to realize. In fact this is an option in MR itself, but wasn’t given UI controls in Max. When used in conjunction with Preferences Gamma requires BAT4Max v2,5 or greater to work.

[bixel 1,228]

oooh --- this will help with my HK towers, a lot of mine suffer with wells of darkness.

[Jasoncw 12,060]

Thanks!

[SimHoTToDDy 2,939]

Ahh, the explanation at last! Thanks a million man, I'll try it out and see where we go

[SimFox 804]

^^

sorry there were some problems with image hosting service I've been using.

I've re-hosted images for the latest tutorial.

[nofunk 1,935]

Thanks SimFox!

I'm sure it depends on the light rigs being used, but do you have any suggestions for default values for these different methods? (Would these defaults happen to be shown in the screen shots above?)

[bixel 1,228]

Unfortunately Max 7, 8 do not have Logarithmic control. It has a crappy one called Automatic Exposure Control and the horrifying Pseudo Color Exposure. The Gamma correction can look horrendous very quickly, even still I can't get rid of the 'wells of darkness'. These controls only seem to saturate the textures, or brighten the image resulting in a washed out effect.

The version of Jasoncw's rig was too harsh too me and I didn't want to go through the trouble of messing around with lights in it (IM LAZY ) - so instead I reverted back to my scanline rig.

I am almost satisfied with this - I might actually go with the 1.5 gamma correction version.