DISCOVERY: File Formats, The Unknown Formats (EDIR,AB,AE etc.)

[Karybdis 34]

This thread is for working on specs and figuring out how the remaining unknown formats of the Simcity4 game (not counting savegame formats) work.

Currently there are 8 of these formats, though this may change with the next expansion pack (if any)

These formats are: Effect Resource Trees, TLO, DE-L, HIT, ID, MXF, AB, AE

Info on each of these formats, and what we know about them follows.

-------------------

Effect Resource Trees

These filetypes are the first mentioned in this thread because we know so much about them already. We know they store lots of the games resources, such as ways to splice together transparent models of highway intersections, and scripts on how the disasters work. redlotus has done some amazing work and has come up with an (almost) complete physical breakdown for each section of this file (there are 14). The only one without a format of some type is section4. What each section stores and how it stores it are more or less unknown right across the board. Nevertheless, this is one of the most important formats in the game.

--

MXF

The MXF format stores all of the games fonts. To be blunt we have more or less no idea how it stores its data. An INI file calls all of the fonts and implements them in specific places in the game, however how it does this is not known. Found in the SC4 font directory.

--

DE-L - COMPLETE

De-Localizers. Basically a text string followed by a huge string of characters begun and terminated by an "x". The string is unicode characters which shouldnt be attempted to be localized by the program. In this case they appear to all be chinese though I'm no language expert. Localizing them would cause program problems in the languages.

--

ID - COMPLETE

A null content file that serves as an identifier for radio stations. It currently serves no real purpose though it probably served as a marker for the game to lock onto at a point in the past.

--

TLO

Logic Objects. We know they are called by a TKD file along with an XA or Hitlist, so we currently presume they have something to do with variables. However what is unknown. They're found in the sound.dat file. We know how the physical areas of the file are broken down more or less, however the type and data stored is unknown.

--

HIT

This audio format (or so we believe) is a relic of the The Sims as far as I know. Its a sound format of some type and seems to contain various tracks, however the storage type and compression if any are completely unknown for the audio, along with the header.

--

AB

This new type of file seems to have something to do with U-drive it sounds, as those are the only things that dissappear when it is removed. Other than that, nothing much is known besides bits of a physical breakdown of it

--

AE

A single file in simcity_1.dat, AE seems to have to do with U-drive it parameters. The AE file will crash the game when removed as soon as you try to run a udrive it mission. Bits of a physical breakdown are known, but more or less nothing else.

--

INFO - COMPLETE

.INFO files store data relating to pictures taken ingame. This data includes the city name, mayor, funds, date taken, zoom, rotation, position, etc. It was probably created for a maxis picture exchange but can still be utilized by us with a custom program.

[redlotus 16]

Well, to start this thing off, here is the hex dump file of the Effects Dir file that I have been working with.  The file is broken up into something a little easier to look at.  I have also numbered the main sections so that it becomes a little easier to see the links between the sections.  One caveat to this last bit, however:  I started numbering all of them at 0, when it's possible that they should have been started at 1.  I am still working on trying to verify this.

 

Another thing I've found is another possible section (this HAS to be the last one) that I assumed was a part of section #12 (Main index section).  It is only one entry big, so it may still yet prove to be part of the main index.  Just in case, I have speparated it in both the dump file and the specs file (both are included in the zip).

 

Last thing.  If anyone finds that I've skipped an entry in my numbering, let me know so that I can fix it.  Also, there may be a couple of small bits of ASCII code that I haven't converted into characters yet.  Let me know if you find any.

 

That's all I've got for now.  I am going to start doing some editing in the next couple of days, so hopefully I'll have some more information soon.

[DarkMatter 21]

If you give me a couple days, I'll have a viewer for this file that splits it up according to section. On the left will be a hex dump of the file, on the right a custom viewer for the data, if I can figure out how to represent it. Should help you guys decode the data better.

[redlotus 16]

All right.  I suppose I can give you a couple of days before I give up on ya.  Seriously though.  I can't wait to see the viewer.  Even if it just breaks up (and numbers) the different entries for each section, it would be a serious improvement over the hex dump I've been working with.

 

I have been able to make a few advancements with this file so far.  Just for a fun test, I made a quick mod to the Elevated Highway to Avenue On/Off ramp preview.  With the mod installed, just click on the elevated highway on/off ramps tool and press Tab until you get to the avenue overpass.  You'll know when you've got the right one.

[the7trumpets 10]

Do tell, do tell!!!!

 

I am anxiously runing arround trying to close up shop here at work so I can go home and get more confused

[redlotus 16]

Well, I can't guarantee that this will confuse you, but I'll give it my best shot.  Here is the full sum of my knowledge on the relationships between the sections in this file:

 

Sections 14 and 15:  Both of these sections simply convert a Uint32 property value into a string that the rest of the file can use.  I am not completely sure why the calling functions cannot use strings directly.  See my latest post in the 'Properties' thread for the properties that I've found to use this.

 

Section 13:  This is the 'top-level' index for the entire file.  It is basically the directory tree that points all of the effects to where they need to be in section 12.  Since this section is not included in the EffDir from the Transportation Download, I assume that it is only really used to speed up the processing of the effects, at the cost of a little bit of memory.

 

Section 12:  This is the full directory for the entire file.  I am not sure of everything these entries do, but the two things that I have been able to determine are probably the most important.  The Index key points to all of the different effects throughout the file.  The Index Flag determines which of the sections the Index key belongs to.  Here are the different Index Flags that I have found along with which section they point to:

0x00000000 = Section #1--S3Ds and animations

0x00000001 = Section #2 - Cursor FSHs

0x00000107 = Section #9 - Sounds

0x00000104 = Section #6 - Attractors and repulsors.  I believe that this section may link to the LUA files.

0x00000006 = Section #8 - Chained Redirect.  I believe that that this section is used to activate a secondary effect.  Rather than defining a duplicate entry, this section just points to another effect.

0x00000002, 0x00000108, 0x00000105 = unknown sections.  I believe that these flags point to the sections that I haven't been able to determine any sort of function for.

There may be a few other keys that I've missed, so this is definitely not a complete list.

 

That's what I have so far.  Have I confused anyone yet??  Let me know if there are any questions.

 

BTW, I cannot apologize enough for this, but I've made a stupid mistake in the specs.  For section #5, I have the DWORD 0x00000040 listed as the 'end-of-section' value.  This DWORD is actually the 'Number of Entries' value for section #6.  Here is what the specs for these two sections should now look like:

 

Start Section #5 - Brush Exemplars
DWORD (Number of entries)
    Two bytes
    DWORD (Resource Key)
    33 Bytes

 

Start Section #6 - Attractors and Repulsors

DWORD (Number of entries)
  Two bytes
  DWORD (Length of string)
  String Field
  One byte

 

Again, I completely apologize for this.  I should have seen it much sooner.

[Karybdis 34]

Wow....so much info redlotus. I cant believe you've gotten it down to the point where only 5 sections (3,4,7,10,11) dont have a known purpose.

And you know the little spec offset isnt the end of the world lol. The only one who it might effect is DM and he has plenty of time to change the proggie he's writing before its released (well maybe a week but still)

Has there been any data found in the main directory that could hint at what these 5 do? or are they mostly still just unknown things.

[DarkMatter 21]

Hey red. I have a request...well, rather a need. Your specs are great so far, but still a bit too vauge for me to really write a parser for it. I need you to go through your specs and find out the purpose of as many DWORDS as possible. In particular, I need you to look for DWORDS that specify the SIZE, or LENGTH, of each section. The first section's spec is also very vaug, and some things don't specify the data size (i.e. WORD, DWORD, BYTE, etc.). I need you to specify the data size for EVERYTHING, every tiny little itty bitty weeny thing. I can't have anything be vauge, it needs to be exact. If you can do this for me, I can write a parser for the file. Right now, I don't have the required information, and the parser either comes up short and doesn't read all the file, or comes up long, and crashes because it tries to read past the end of the file.

[redlotus 16]

I was afraid of that, DM.  I will take another look (or two, or three...) and try to refine the specs some more.  Before I get too far, though, I would like to make sure that I am understanding you correctly.  If I am, you need two main things:

1.  Break up the larger byte-chunks into smaller pieces: DWORDS, WORDS, single-bytes, etc.

2.  Find the DWORDS that define the length of each section.

 

I assume that #2 would be the most important right now, as long as every byte is accounted for in the specs of each section.  If this isn't correct, let me know.  Otherwise, I will concentrate on finding the section lengths first and then work on the large byte-chunks after.

 

Kary:  I haven't been able to nail anything down yet.  The only thing that looks promising is whatever index flag 0x00000002 points to.  All of the associated entries seem to refer to either light effects or [non-cursor related] effects that are not confined to one tile.  One thing though.  Since I have not found an index flag above 8, there may be only 9 sections above the main index instead of the 11 that I have now.  Hopefully, I will be able to find all of the section length DWORDs, which should verify the number of total sections.

 

If anyone would like to help out, I need somebody (or somebodies) to proofread the file.  In order to find the '# of entries' DWORDs, I will need an accurate count of the entries that I have numbered.  I just need to make sure that I didn't skip any numbers (or entries, for that matter) or use them twice.  If anyone finds a mistake, let me know.

[redlotus 16]

Alright, DM.  I think that I have most of what you wanted.  I have found the 'Number of Entries' DWORDs for every section except for #13.  Of course, this has scrambled the specs a little bit.  Fortunately, there are no large changes to any.  I will continue to look for the final quantity DWORD and get back with you.  I have attached the updated specs file, along with a slightly corrected hex dump file.  Ignore the spec comments in the dump file, because I haven't updated it yet.  If there are any specific areas that I haven't cleared up enough for you, let me know which ones are giving you troubles.

[DarkMatter 21]

AH, sweet. This is much better. Thanks Red. Keep up the good work, and we'll have a viewer for this soon.

[redlotus 16]

Hey DM.  Just wondering how things are going with the viewer.  I know that you've been busy with your version of the LET, so I don't want you to think that I'm being pushy.  Just wanted to make sure that it wasn't forgotten about.

 

BTW, I am afraid that there is no Number of Entries for section #13.  I think that the only way to determine the end of section is that the final entry will be: 0x00000003, end, 0xffffffff.

 

Thanks

 

-red

[DarkMatter 21]

I'm actually splitting time evenly between the lot editor and the viewer for this file. I didn't expect it, but this file is really difficult to decode and display properly. I think its because we really only know parts of each section, and don't understand its structure completely. I'll try to have something workable soon. Right now its just an instability factor...causes crashes all the time.

[redlotus 16]

Thanks for the update.  I figured that it would be a serious pain in the you know what.  I am going to be working more on the Effects Directory file that came with the transportation pack for the next few days.  Maybe I'll be able to clear things up a little.  If there are any specific sections that are giving you problems, let me know and I'll focus on those.

[DarkMatter 21]

I think what we need to do is work on each section one at a time. Each section is practically a file in and of itself. I think I'll have better luck if I try to tackle each section as such. This is a large file, and it will probably use some 5-10 megs of memory when loaded.

If possible, could you add me to your instant messenger, and maybe we can chat over this when were online at the same time. Here are my infos:

ICQ: 6839311
AIM: synapticeffect
Yahoo: synapticeffect
MSN: jrista@hotmail.com

[redlotus 16]

If you are able to break the sections down with the parser/viewer, that would be a great idea.  I was afraid that the file might be a little too big to handle all at once, anyways.

 

I had another thought that I had while working on the Effects Directory for the Transportation Download.  Maybe we should start with this smaller file first and then move to the main file after we have had some time to experiment with it.  We would lose a little bit of functionality, but I believe that we could at least get started on some first-time-around experiments with the TD file.  From what I have been able to determine, sections 1 through 10, and 13 would work with the current spec.  Sections 3 through 8 are currently 0-entry in this file, so you could simplify the parser.  The only major changes would be to sections 11, 12, 14, and 15.  Sections 11, 14, and 15 don't seem to exist and section 12 has a slightly modified spec.  Let me know what you think and I can have a new spec for this file completed in an hour or two.

 

BTW, I haven't used any chat software in ages.  I happen to still have my ICQ number in a place where I could get it, so I've installed the software and tried adding you as a contact.  Just waiting for your response.

[DarkMatter 21]

Heh, keep up the good work is all I can say. Your better at decoding files than me. You and karybdis both. About all I can request is see if you can figure out what the differences are between the main and TD EDirs. Each section should work the same, they shouldn't be different, but who knows I guess.

I'll be home 6 hours, and I'll authorize you (glad you use ICQ...i hate MSN).

[redlotus 16]

6 hours!  Ack!  I'll never make it that long.  Perhaps another day.  But I do have a little something for ya.  Here is a clean-up for section 12:

 

Start section #12
DWORD (Number of Entries)
     DWORD
     DWORD
     DWORD (Number of Primary Indices)
        DWORD (String Length)
        String (Optional)
        DWORD (Index Flag? - May indicate which section the index key belongs to)
        71 Bytes (unknown, but the same for all)

        DWORD (0x00000000)

        26 Bytes (Variable Section)
        DWORD (Index Key)
     DWORD (Number of Secondary Indices) 

          DWORD
          DWORD (String Length) 
          String (Optional)
          DWORD
          DWORD (Index Key?)
     DWORD (0x00000000)
     DWORD (0x00000000)
     2 DWORDs (0xCCCCCCCC or 0x00000000 each)

 

The affected area is in red.  I got rid of the conditional stuff, so hopefully this will be a little more palatable to the parser.  Of course, I don't know much about how the parser works, so I could be wrong.

 

Anyway, I will continue working on this thing tomorrow.  I have started working on the Effects Directory from vanilla SC4 to see if that will give me any more clues.  I'll keep you updated.

[DarkMatter 21]

Ah...you know, the way you indented that makes it MUCH more understandable. Could you indent all the sections like that to show where repetitions are and such? One of the things I'm having a hard time with is what parts are repeated and what parts arnt.

[redlotus 16]

Ask and ye shall receive.  Attached are the specs in a RTF file.  I have gone through and cleaned things up as best I can.

 

I have also made a couple of updates.  The first is a small simplification of section #2.  More importantly, I have finally worked out the spec for section #4, so all sections have at least a rough spec, now.

 

Let me know if any parts are still unclear.  I moved the specs to RTF so that I could color code things if necessary.

[redlotus 16]

Alright, DM.  I've got some new information.  I finally figured out why the Effects Directory from the Transportation Download is so much different from the current file.  It looks like Maxis upgraded the spec with Rush Hour.  If my suspicions are correct, the first DWORD for the file is actually the version number.  Vanilla SC4 and the Trans Download both use version 1.03.  Rush Hour uses version 2.04.

 

Luckily, Maxis included an Effects Directory file with both the Mayflower and Rockefeller landmarks.  I have checked the file and it does conform to my latest spec, so you should be able to do alpha-testing on this file.

[DarkMatter 21]

Sweet! Thanks for thie info, Red. I'll see what I can do with the latest spec.

[Karybdis 34]

And De-localizers bite the Dust

DWORD - size of the file minus this one
16 bytes "Don't Localize. " in plain text.
BYTE - X in text - Begin String
126 Bytes - Unicode String for non-localizable characters
BYTE - X in text - End String

Basically this file is a string of characters that its telling the program not to try and localize. The characters I'm seeing are mostly chinese or all chinese by the looks of it DM and I figured this out at 2am while talking hehehe.

This is included in simcitylocale.dat (obviously) and all plugins with ltexts since an ltext can have localizable characters due to its unicode format.

[redlotus 16]

I know it's been a long time, but I finally started to get back into testing .  I've started working on the function spec for the EffDir file.  So far, I haven't been able to complete any of the sections, but I have made some real progress in a few.  The biggest thing is that all but three of the sections have functions assigned to them, so at least I'm getting a better idea of how it all fits together.

 

Anyway, just wanted to post an update and see if the updated spec might get a spark going in someone's mind.  I think my brain is going into meltdown from staring at my monitor for so long.

 

Almost forgot.  I have also discovered and corrected a bug (*cough* mistake *coufgh*) in section 13 of the last spec.

 

-red

[DarkMatter 21]

Ah....thanks for the updated spec Red. I'm working out on paper how to deal with this file in code. I don't know if you'll ever be able to meet with me online, but my chat info is here:

 

ICQ: 6839311

MSN: jrista@hotmail.com

AIM/Yahoo: synapticeffect

 

It might be easier to figure this stuff out if we can chat realtime.

 

 

Anyway, some info on LogicObjects. They seem to have a more complex structure than the fixed-length chunk format we thought before. If you check the logic object with TGI: 0x9D796DB4 0xCA4D19E3 0xEBAAEC52, you'll notice, that at byte 0x58, there is a DWORD value...0x6. After this DWORD is a string, 6 characters in length: _start. Normaly, the first DWORD of each repeating chunk is 0, but if its nonzero, it defines the length of a string that immediately follows. I also think there are some identifiers used in logicobjects, although what they refer too, I don't know. Anyway, here is an updated spec:

 

Offset	Type	Description
======  ======  ===================================================
0	DWORD	File Identifier. Always 0x00000000.
4	DWORD	Object Count.

Repeating Chunk as follows, offsets relative to position 8 in file
===================================================================
0	DWORD	Name length
-----------------------------------------------------------
4	DWORD	Name, if name length is nonzero, all other
		values in chunk are offset by name length
-----------------------------------------------------------
4	DWORD	Unknown #1, possible identifier?
8	DWORD	Unknown #2
12	DWORD	Value A (possibly 2 WORDS?)
16	DWORD	Value B (possibly 2 WORDS?)

20	DWORD	Unknown #3
24	DWORD	Unknown #4
28	DWORD	Value C (possibly 2 WORDS?)
32	DWORD	Value D (possibly 2 WORDS?)
36	DWORD	Value E (possibly 2 WORDS?)

[redlotus 16]

I'm sure I'll be staying up way past my bedtime one of these days.https://community.simtropolis.com/assets/emoticons/3.gif>  I've got you on my contact list, so I'll keep an eye out for ya.

 

Well, it's also time for my progress report.  Things are going slower than I'd like, but faster than I really expected.  I've been able to break down the largest byte-chunk of the file and I've discovered what over half of it does.  Attached is the latest spec.  The biggest thing that I've discovered (and the reason for the update) is that every value I've found a function for so far is in Float32 format.  I don't know how much of a difference that will make for the parser, but I figured it would be better to get this information out now.

 

I apologize if a good bit of the descriptions in the spec file only make sense to me.  I hope to make everything a little more readable during my verification tests.  But if there are any questions, I would be happy to answer them.

 

-red

[redlotus 16]

Well, after a little over two and a half months of working on this file, I have finally managed to write up a tutorial on how to create your own Effects Directory file from scratch (basically).  Attached to this post is the tutorial itself along with most of the support files.  I will have to post the big file in my next post as it is too big to post everything at once.

 

I wanted to post it here before posting it in the Modding - Open Discussion because I wanted to get some feedback on it first.  I know that I am not the best at explaining things (and I am not very good at making it interesting) so if anyone could take a look at it and offer some criticism, I would greatly appreciate it.

 

Thanks

 

-red

[redlotus 16]

Here is the big file that I couldn't put in my last post.  This is the complete hex dump of the Effects Directory file, broken up into sections and entries.

 

BTW, I mention in my tutorial that there is no real good way to convert Float32 numbers into hex.  Since I finished this draft of the tutorial, I have managed to find a handy little tool called Hexit that can do the converting.  You can find it at download.com.  Just do a search for Hexit.

 

If anyone has questions/comments/criticisms/hate mail (just kidding. please don't) feel free to send me a PM or email me at redlotus5(at)comcast.net.

 

-red

[DarkMatter 21]

You can also convert from float32 to hex using the converter tool in DatGen. Its in the tools menu.

[Karybdis 34]

Here's a copy of my specification for the INFO format. Its almost complete minus 3 things listed at the end of the spec.

 

------

 

INFO

The info format is basically a storage for all the data in the picture albums. The description of the picture is kept in a text file, but all the rest of the data, including city name, mayor and other data is stored inside this format to save room.

 

There appear to be two types of info files. City pictures and region pictures which have different structures. The type of data storage for each of the values is unknown, though it is 1 byte per character of data seen in game. Each info file is comprised of sections identified by a 1 byte section header.

 

---------------- City

 

11 BYTES - SanityCheck (failed apparently )
BYTE - 71 (Start Username)
X BYTES - User name from your config-log in the simcity4 apps dir
BYTE - 71 (Start City name)
X BYTES - City name one byte per letter including spaces
BYTE - 71 (Start Mayor Name)
X BYTES - Mayor Name, One byte per letter
BYTE - 71 (Start Date of picture)
X BYTES - Date of picture
BYTE - 71 (Start Population)
X BYTES - Population
BYTE - 71 (Start Funds)
X BYTES - City Funds

 

section (71 plus data) (same through cities in a region) 27 (494C)

 

section (71 plus data) (varies per city) 1,2,57,77 in test city. Maybe something with income and mayor rating?

 

BYTE - 71 (End File)

 

--------------- Region (Picture file name as data stored)

 

X BYTES - Region name
BYTE - 56 (-)
10 BYTES - Timestamp for Time picture was taken
BYTE - 55 (.)
3 BYTES - PNG

----------------------------------------

Basically what we need to know is the meaning of the last two sections of the city data, and the type of data storage format that's being used. As far as I can tell, the characters dont directly translate to ascii.