DISCOVERY: File Formats, The Known Formats (FSH,S3D etc.)

[Karybdis 34]

(NOTE: The DISCOVERY threads will be replacing their earlier brothers, the File Formats threads. There's only 4 of them so they'll be much easier to keep track of)

This thread is dedicated to PHYSICAL FORMAT work on the remaining formats of which we can currently edit and work with. These formats are known, and are viewable if you have a program with the capability, but simply dont have complete specifications.

The formats that this thread covers are
FSH,TKD,LEV,UI, INI Network UK pathing, INI Network Intersection Cells, Config Files, and MAD videos. A total of 8.

A file will be included at the end of this post including the current specs for those files which we are working off of.

A Minor Description of each type and what we're missing

----------------------
FSH - COMPLETE
These textures are pretty much everything visual in the game to put it simply.

--
TKD
Track Definitions. These files are controllers that link to either a Hitlist or just a single XA file and can contain a link to a Logic Object for that hitlist/file. They're comma seperated files, where only a few of the values are known. Much more testing is needed.

--
LEV - COMPLETE
Sound Levels. These files control a full group of XA files, controlling the various speaker values and other related playback variables for them. They are used for Freshness, IE randomly playing sounds from a pool of sounds.
--
UI
The User Interface. These files control the visible interface you use in the game. They're basically a modified XML file. Most of the properties of the UI's are known, however a few arent, and the main IDs which link the functions of the various things together arent fully known yet. The spec is in a format which can be viewed, added to, and redistributed as it is worked on.

--
Ini Network UK paths. - COMPLETE
These files control various things ranging from shadow mapping for networks to the paths that are reversed for the UK version of simcity. They also control other values related to the networks. Its found at Instance 8a5993b9 in the simcity_1.dat file.

--

Ini Network Intersection Cell Data - COMPLETE
This file is the big one for networks. It controls tons of data related to the highway intersections in the game and how they're layed out. The majority of this file has been layed out by the7trumpets, and the rest was tested and figured out by me. Its found at instance 10000000 in the Simcity_1.dat file.

--
CONFIG
The config file of simcity 4 is found in the simcity4.cfg file located in the mydocuments dir for the game. Opening this cfg in a dat reader will allow you to access the config file which contains bunches of data on the visual rules the game will have when its started up, such as detail levels and resolution. The basis of this file has been layed down already however tons of it is still unknown.

--
MAD
The video format of simcity 4 that is responsible for the opening movies. They are devided into many alternating video and audio blocks that accompany eachother. The specification for the audio blocks are known, however the video blocks are significantly different. Its possible they use ASF video compression, but they contain a very modified header that we only know a bit of. What is needed for this format is the rest of the data on the header of the video blocks. the other stuff is figured out for the most part, but about 16 bytes of the video blocks header remains unknown.

-----------------------

These files need to have completed specifications to be most useful, so any info people can give is most appreciated.

[DarkMatter 21]

A quick note on the S3D format. The ANIM section seems to have many uses, although none of them are known fully. For one, in animated meshes like the horse, deer, bear, etc. I beleive the section is used for vertex movement data. In other cases, the ANIM section seems to be used for texture positioning or other information. There are at least three distinct uses of the ANIM section, and each one should be decoded with the idea in mind that each use could be vastly different from each other.

[Teirusu 38]

I dunno if this is the right thread for it or not... however, I don't feel like making THREE separate bug reports.

It has recently come to my attention that all three programs: The reader, DatGen, and Fishman are not displaying alpha channels AT ALL on some textures.

This happens on many of the one-way road textures. Just to name a few:

09000224
09000234
09000304
09000334
09000714

In all three programs it shows that there is no alpha channel for these textures. (Fishman goes farther by saying these are DXT1 textures!) But, I KNOW for a fact these textures HAVE alpha channels.

Trolca, even mentions in this thread that he's run into something similar while creating new textures for trains.

So, having the same problem in all three programs brought me to only two conclusions.

A.) Both DM and ILive made the same coding mistake that results in FSHes not being decoded right. Or...

B.) Our spec for the FSHes is somehow, bugged.

[DarkMatter 21]

Teirusu, I've researched and examined and hex-hacked dozens of FSH files, and I can assure you, the files you listed are abolutely and positively DXT1, not DXT3, textures. I can assure you this as a solid FACT. The file format of a stored image is determined by a single byte, which has one of the following values (this enum is right out of my code, and is based on several official and known correct sources):

 

public enum FSHBmpType: byte
 {
  EightBit		= 0x7B,
  SixteenBitAlpha	= 0x7E,
  SixteenBit		= 0x78,
  SixteenBit4x4		= 0x6D,
  TwentyFourBit		= 0x7F,
  ThirtyTwoBit		= 0x7D,
  DXT3   		= 0x61,
  DXT1   		= 0x60
 }

Now, open up the file with IID 0x09000224. Click the Raw Data tab in DatGen, and click on the first byte of row 0x20. That byte is 0x60, which if you check the enumeration above, means DXT1. This is the raw data, unmodified, taken directly out of the simcity_3.dat file. Its exactly how Maxis saved it. All the other files you listed also have the same value at the same byte, 0x60. They are all DXT1 files. So the problem isn't a bug, but rather that Maxis saved these files improperly, or else they had a reason for saving them as DXT1 rather than DXT3. The code for FSHLib is correct, and so is the fshtool code that Ilive uses in his Reader. So its option C...Maxis.

[Teirusu 38]

Ahh! I see I see... Thank you for taking the time to explain it, DM! I know your a busy guy being the Zeus of SimCity 4 modding and all.

So, I guess, I'll just have to keep this Maxis bug, or whatever it is, in mind for now on.

Thanks again!

[DarkMatter 21]

You know...you could try and fix them. The alpha maps look really simple. I bet you could make the right ones and make a new .dat with the "proper" FSH files in it. As long as you set the TGI's to the same as the existing wrong ones, the new ones should override the old.

[krelf 0]

Going back to the 'anim' talk. Looking through one of my 3D programs it has commands for creating & removing Keyframes for animations.

This is part of the standard 3DS file and an example in the help file shows how to make a person walk down the road, moving the legs, feet and arms properly.

Will need to play with it a bit more in relation to SC4 and see how the animation pieces relate to a flat non-moving 3D file.

[DarkMatter 21]

Just out of curiosity, have you edited any of the data in the ANIM section? Just to see what it would do? Hmm...for that matter, does any program let you edit it? I don't know if I enabled editing in DatGen or not.

[Karybdis 34]

Hmm... now that's very interesting. So it doesnt actually contain data for everything, but it does have keyframe position data and so forth?

Reader lets you edit, but not add as far as I know.

[krelf 0]

The reader allows you to add/delete anims. but everytime I try delete one it crashes.

 

Just looking at the functional grand-central station plugin. It has 20 S3D files, all at different resolutions, however all of these apart from one only have one anim. entry. The one that has multiple anim entries contains 17.

 

Just playing with an S3D and I cannot see what purpose the anim. entries serve in a non-moving 3D image.

 

[Stardog13 44]

Do we have any idea on how to import or edit s3d's or will we just have to wait for the BAT?

 

Edit: Sorry about posting in the wrong thread but I searched and didn't see anything that looked useful.

[Karybdis 34]

stardog, yes, but for ages and ages. For more info you'll have to see another thread because this one is just for the decoding work though

[krelf 0]

I've done some digging around the net and found some sites which contain detailed information on 3DS files. Hopefully this may be helpful. We might be able to relate it back to an S3D this way too.

http://www.whisqu.se/per/docs/graphics56.htm - A lot of unknowns in this description.

http://www.the-labs.com/Blender/3dsspec.html - An update spec of the above site.

http://www.spacesimulator.net/tut4_3dsloader.html - Detailed description

http://www.cyberloonies.com/3DSFTK4.htm - another good detailed description. explains what everything does.

I was going to paste the information here but some of it went for 10-20 pages.

[Karybdis 34]

Hmm....looks like I'm going to need FSH/QFS textures from "Need for Speed 2" in order to complete the FSH specification. I'm not sure if edited models or textures will work though since no programs really understood the missing bit to data, but its recorded as used in this game.

If anyone can post a couple of fsh/qfs tex's from the Original game (the ea originals) it would be most appreciated, or pm me if you want to mail it to me

EDIT: found a FSH that contains the number of special units. here she is. hope it helps somebody. I'm working on it too.

[DarkMatter 21]

Just an FYI. I think the "official" extension for 3D models is .scm, or SimCity Model. I'll explain why I think this later, but its only logical.

[Karybdis 34]

Just finished with the FSH spec finally last night. Well there goes another one thats been worked on for 10 months.

[Karybdis 34]

An update to the INI remapping format. I've completed the spec for 6 out of the 9 sections with 3 remaining

 

The three that still need to be documented are

 

Power poles (dealing with power pole connection rules)

Bridge Exemplar Data (many small sections dealing with bridge types)

Tile Verifiers (well...self explanitory)

 

For INI Intersection Cell data the sections that need info are

 

StepOffsets
Autoplace
Handleoffset
ConvertQueryID
ReplacementIntersection
Onewaydir
AutoPathBase

PlaceQueryID
ConsLayout
Piece
Costs

[Karybdis 34]

A LARGE update to the UI file specification that has bunches and bunches more of the functions identified

 

There is also a lot of new ones added (hopefully all of them) including a brand new type OptGrp and all of its values (seen at the bottom)

 

As always help with figuring out the remaining properties would be appreciated

[Karybdis 34]

This is a complete copy of the INI Network Remapping file Specification. I've been spending awhile figuring out what each part does in detail, but this should cover it all.

 

Post any mistakes or revisions in this thread if you see them, otherwise this spec is finished https://community.simtropolis.com/assets/emoticons/1.gif border=0>

 

EDIT:

In addition to this, I've cut 6 properties and a section off the unknowns for the large intersection ini network file. Only the following 5 properties havent been defined yet.

 

StepOffsets
Handleoffset
ReplacementIntersection

ConsLayout
Piece

[Karybdis 34]

Whelp, this update should finally complete the Network Cell intersection file.

 

A big thanks to 7 for all his hard work on this file. The rest was a pain in the butt and took quite awhile to figure out as you no doubt guessed

 

Once the effect directory editor is released this file should hopefully allow for creation of completely new intersections that are fully functional in the game in all ways.

[Karybdis 34]

an update on MAD

With the help of GouRou, I finished the spec for the MAD header last night, which completes the majority of the MAD video Spec

Currently we're waiting on decoding the pixel storage blocks so that the frames can be displayed properly.

Will post again once we've got the storage block formats with the full specification for it. They're 15 or 5 and 18 or 6 byte blocks for MADk and MADm blocks respectively

[DarkMatter 21]

Even if you decode teh MAD header, we still need to decode the header for audio data. Its not the exact same as XA files, possibly very different. Can you post the current spec as well, so I can wrap my grimy fingers around it? https://community.simtropolis.com/assets/emoticons/9.gif>

[Karybdis 34]

No worries both Audio and Video headers are finished along with the spec for decoding the audio streams properly

Just waiting on the video pixel storage

Will post the Whole spec as soon as I get home

[Karybdis 34]

Sorry about the delay. MAD video and audio spec as promised DM.

[redlotus 16]

I've got a minor change to the Network Cell intersection file spec.  Here is the section that it will affect as it currently reads:

 

AutoTileBase = 0x0C883000

This line is what tells the building engine where to look for the sliced up tiles.  It refferences an Exemplar instanceID which then refferences an s3d file.  The way it works is basically by placing a grid over the effected interchange.  It finds the most top left tile in order to include all the (x) values and CheckType values which are not optional in the CellLayout, and defines that as this hex value.  For each tile below it, the instance is incremented one value in the seventh digit.  For each tile to the right of it, the instance is incremented one value in the sixth digit.
 
The area in red is what needs to be changed.  The engine does not look for a new Exemplar/S3D/Path combo for the CheckType values.  These values exist simply to let the game know what kind of connections the intersection will accept.

 

-red

[Karybdis 34]

Thanks man Fixed and reuploaded.

[Tropod 144]

I am not sure how well known this is, and I don't believe I've seen it mentioned anywhere [unless I've missed something], so here goes;

Shadow Mapping; for network items, [to a big degree] the Shadow Mapping is actually controlled/influencd by the GIDs of the Exemplar files that are used to reference S3D files. If the Exemplars don't contain the proper GID when referencing the IIDs of S3D files, for the particular given network, then the referenced S3D file's image/piece won't show up properly [at all in-game] for whatever it is you're trying to build. Whether it be Bridges, Overpasses, &/or Tunnels.

The following is the list of GIDs used for the Exemplars that reference [network-based] S3D files.

Power Poles: 0x088E1962

Dirtroad:  ???

Streets: ???

Roads: 0x2821ED93

Onewayroads: 0xCBE084CB

Avenue: 0xCB730FAC

Groudhighway: 0xEBE084D1

Elevated Highway: 0xA8434037

Rail: 0xE8347989

LightRail: 0x2B79DFFB

Monorail: 0xEBE084C2

 

For the ones in question; if their appropriate GID can be figured out, and/or located/obtained somehow, then there's no telling how useful this could be [there may be clues in the games files itself. Though there's no actual definitions list of these GIDs anywhere it seems]. Whether the Networks have more than one set of GIDs that's used for the Shadow Mapping, I'm not sure, there may be more than one [i don't know, though I doubt it. I just quickly put this together].

[Karybdis 34]

Hmm. Interesting, and nope it wasnt known
Is this info actually directly tied to the format of the remapping or is it an external factor which just influences it and determines validity of the piece/helps find it

Apologies for the absence man. I should be getting off of fulltime work in less than a month so I should finally have some time to get back into decoding. Havent done any serious work since the end of january.

[Tropod 144]

<DIV>I should've pointed out;</DIV>

<DIV>It's not always the case where the GID needs to be the same for the specific network. It seems to sometimes depends on the Rules. The Intersection Ordering Rules for example, mostly all rely on Highway rule designations, therefore even for an overpass like a Road over Avenue overpass, it'll still use the same GID [for the Referencing Exemplars]

[Karybdis 34]

Finally got a goodly sized UI Specification update for you guys

This spec drops the total number of unknown/undefined properties down from 46 to 26, eliminating just under half.

Help is still really needed from everyone to figure out what the remaining properties mean though. Here's a list of the ones we still havent got tested and confirmed and which control (CLSID) types they can be found in

---------12 Unknown - Updated Aug7---------------------

tipsflag (common)
winflag_sortable (common)
winflag_mousetrans (common)
allowinsert (textedits)
insertindex (textedits)
insertpos (textedits)
selrule (grid)
fnone (grid)
fdefault (grid)
shiftcaption (button)
option (optgrp)
autofilltype (optgrp)

winflag_pbuff (common) - DONE - Buffer Control?
winflag_pbufferase (common) - DONE - Erase buffer after use?
winflag_pbufftrans (common) - DONE - Buffer Transfer Mode, Write to secondary buffer for flipping into position when needed. IE moving control, control change etc (buffer flipping)
winflag_pbuffvid (common) - DONE - Activate controls video buffer
fdpastlastcol (grid) - DONE - Draw grid past last column
fdpastlastrow (grid) - DONE - Draw grid past last row
fdrpdnmenu (grid) - DONE - Grid data has drop down menus
fall (grid) - DONE - Draw all grid standard controls, IE Scrollbars
combodownarrowrect (combo) - DONE - Combobox down arrow rectangle definition
combodowncolor (combo) - DONE - Combobox down control colour
sort (listbox) - DONE - Alphabetically sort listbox items
pagesize (scrollbar) - DONE - How much a scrollbar moves when the bar itself is clicked
linesize (scrollbar) - DONE - How much a scrollbar moves when the line is clicked
linepagecount (scrollbar) - DONE - Defines lines per page for a scrollbar, and total pages
drop (listbox) - DONE - Drop edges of a listbox for 3d layered look
notify (bitmaps) - DONE - Notify rest of controls of bitmap properties
textwrapping (grid) - DONE - Wrapping for grid text other than cells?