Programmatically generating a city: A savegame experiment

[smf_16 1,835]

Hello everyone,

 

In the wake of the discovery that savegames can be edited - see this thread for more general info - I decided to set myself a goal. I want to write a program that is able to plop an entire city - or at least the lots in that city. I have actually been playing with this idea for a few years, the main reason being that I always struggle with creating nice transitions between the CBD and the suburbs. The game only gives us the possibility of zoning either Low, Medium or High, but this just doesn't do the trick. You'll always be able to see very clearly what the Low, Medium & High zones are.

One of the first achievements after the savegame discovery was the ability to turn plopped residentials into functional residentials. While this certainly helps because we are now able to build whatever we want instead of waiting for it to be grown, humans are actually pretty bad in "creating randomness". I've experimented a bit with the plopped residentials and it is really hard to create something that doesn't look too repetetive - unless that's exactly what you're aiming for of course, repetition has its beauty as well.

I figured that if I was able to programatically plop any available lot, I could also write a program that generates an entire neighbourhood in a semi-random way based on a arbitrary function. For example, we could set the "maximum height" per tile and then have the program build something by selecting an eligible lot (from a predefined set of lots) that does satisfy the height restriction. As such we could create "skyline functions" and generate a city accordingly.

Another reason that I'm starting this project is that it will a good exercise to learn what is and what isn't possible when editing savegames. I will post my findings, struggles, remarks, discoveries, ... here so that this thread can serve as a reference for other people experimenting with the savegames. Moreover I obiously don't understand completely how the game works and how all file types are related, so I will definitely need some help on the road. With this thread I also hope to find the required and much appreciated help as such.

So... Let's kick off!

As I've already mentioned above, the process to programatically generate cities is actually two-fold:

1. First we have to be able to plop any lot we want, wherever we want.
2. Once that is done, we have to write an algorithm that decides what lot should be plopped where.

The second part is pretty much agnostic of the game's inner workings, so I guess the hardest part will be part 1. If I understand correctly, if a lot is plopped in the game, the game does something like this:

• Look at the lot exemplar and figure out what building, props & flora are on the lot. I found that these things are stored in the LotConfigPropertyLotObject property of a lot exemplar.
• Modify the terrain underneath the lot according to what is stored about slope conformity in the lot exemplar.
• Put the building, the props and the flora in their respective subfiles in the savegame. This also means that - if I understand correctly - when a lot is plopped (or grown) it's "exploded", meaning that the building, props & flora are no longer tied to the lot itself. Hence if you edit the building, props or flora of the lot in the LE, the existing lot in game will not change. As far as I know this is not the case though for textures, they seem to be "Hot Reloaded" (but correct me if I'm wrong).

So if we want to programmatically plop a lot, we'll have to do something similar. Of course we will need to edit the Building Subfile, the Prop Subfile and the Flora Subfile here, but these are all nicely decoded according to the wiki, so this shouldn't be too much of a problem. As a first step I think we could ignore the props & flora and just try to plop a lot with only the building and its textures.

According to this list

0x0: Building: Defines position, and IID reference of building exemplar.
0x1: Prop: Defines position, and IID reference of a prop exemplar.
0x2: Texture: Defines position and IID reference of a base or overlay texture.
0x3: Fence: Not implemented currently.
0x4: Flora: Defines position, and IID reference of a flora/growable tree exemplar.
0x5: Water Constraint Tile
0x6: Land Constraint Tile
0x7: Network Node: Defines transit connections and automata paths

the building is stored in the LotConfigPropertyLotObject where the first entry is 0x0. The structure furthermore looks like this:

Rep #    | Value      | Description
-----------------------------------------------------------
1        | 0x0-0x7    | Type descriptor. See above.
2        | 0xAB       | Level of detail modifier (See Table 3 below)
3        | 0x0-0x3    | Orientation. 0=South, 1=West, 2=North, 3=East
4        | *          | X Location Position. 0x00000000-0xFFFFFFFF
5        | *          | Z Location Position. 0x00000000-0xFFFFFFFF
6        | *          | Y Location Position. 0x00000000-0xFFFFFFFF
7        | '''        | X1 Size Position. bounding box
8        | '''        | Y1 Size Position. bounding box
9        | '''        | X2 Size Position. bounding box
10       | '''        | Y2 Size Position. bounding box
11       | 0x00000000 | Usage (optional/mandatory) Flag. Unused.
12       | 0xABBBBCCC | ObjectID (See Table 2 below)

*   These values define position. VERY large numbers.
''' These values define bounding box size. VERY large numbers.

and for buildings Rep #13 is added which looks like

---0x0 (Building) Type---
13       | IID/ID     | For Ploppable buildings, this is the IID of an exemplar/S3D pair.
		      | For Special Buildings (tollbooth, etc.) IID of exemplar for data.
		      | For Growables, the ID of the lots compatible building family.

That seems fine so far. If we have the Lot Exemplar that contains the LotConfigPropertyLotObject entries, we should be able to find the building easily based on its IID stored in Rep #13.

However, this is where I hit my head against the wall for the first time. I opened an example lot in the reader (I chose Simmer2's Easy Money here) and looked at the LotConfigPropertyLotObject entries. It looks like this:

So I though I should be looking for a building with IID 0x79536AF6 here. Just to confirm, I've checked this in SC4 by using BuildingPlop and I noticed that the building I needed indeed had this id:

What I noticed as well though is that the corresponding lot also had the same IID. I guess that was on purpose by Simmer2 because the building only appears on one lot.

However, I looked for this number 0x79536af6 in the .SC4Desc file, but I was unable to find it anywhere:

The Exemplar in the .SC4Desc file has TGI 0x6534284a 0x7cc07882 0x28cd5dcb, while I thought that its instance ID here would be 0x79536af6. Probably that's not how the TGI system works I guess, but I don't know where this building id 0x79536af6 can be found then. I tried to do some research on TGI's, but it's all very overwhelming and I have some trouble understanding it. Is there anyone here that can help me out and give a brief explanation - or point me to some resource - of how the TGI system exactly works and where I'm supposed to find the building id 0x79536af6? If I understand correctly, the Type ID is used to designate file types but the Group ID & Instance ID aren't very clear for me. However, I think that a group ID is used to link files of different types together (which you can see in the image above: the exemplar and the xml file have the same Group ID, but the same Instance ID as well).

Additionally, I think a lot of the information I need is already well known - for example the PIM X includes a lot editor, so it sure knows where it should look for buildings & props on a lot. In this regard it would be extremely useful if the source code of both PIM X & Savegame explorer are available somewhere, but I didn't manage to find them anywhere yet. Anyone who knows if these are available somewhere?

Many thanks and I hope I can bring this project to completion with the help of the community!

 

- Seba

[mattb325 28,880]

Hi Seba

 

Awesome idea, and I love what you have done so far!

 

To start you off, a good resource is here, by Ripplejet: SC4 Container Files, Game Files and File Relations

I don't have the source code for the PIM-X, and IIRC, I think it may have been lost on a corrupted laptop, but some can be gleaned from the program installation itself, I guess you have already toyed around with the 'properties.xml' file in C:\Program Files (x86)\SC4PIM which didn't give what you needed? (I'm not programming savvy enough to know if that is what you are after).

At any rate, I hope this helps

[smf_16 1,835]

Thanks a lot! I will definitely give that one a read.

I did indeed play around already with the properties.xml file, but in this case it doesn’t help me. I’m looking for how to find the building exemplar based on a lot exemplar. The wiki says that the 13’th value of the LotConfigPropertyLotObject in a lot exemplar represents the building IID, but it didn’t correspond at all to the IID of the building exemplar of that Easy Money lot. I’ve come to think that maybe it has something to do with a building family sitting in between or something, but I hope RippleJet’s turorial makes this clear.

[mattb325 28,880]

You're welcome, anything to help the cause.

I thought as much, but it never hurts.

With the lot you have chosen, I assume it was made with the pim-x and as such the building file was part of a .loosedesc/desc for a ploppable which doesn't get assigned until the lot is created and it's all "stitched" together. Ripplejet's tutorial (and all of the links from that tutorial...it will keep you busy for a while ) explain it far better and far more thoroughly than I ever could.

[smf_16 1,835]

Well that tutorial definitely clarifies some things. I think that I picked a “bad” example lot. The Exemplar file I was looking at doesn’t seem to be a building exemplar, but rather a prop exemplar. I think that the building exemplar will be somewhere in the dependecy that was listed.

I’ll have a look at a growable lot without dependencies tomorrow, at least then I know I don’t have to track down dependencies as well. Should make things easier to follow. More soon!

[CorinaMarie 29,605]

I love this idea.

Let's jump to this (which @mattb325 touched on):

3 hours ago, smf_16 said:

So I though I should be looking for a building with IID 0x79536AF6 here.

By default, PIM-X creates two separate files. One is the .SC4Lot and the other is the .SC4Desc. The former will have the Lot Configurations exemplar and the latter has the Buildings one. Many peeps go ahead and combine those into to one file. Typically, the stuff from the .SC4Desc is copied into the .SC4Lot.

Here's one of mine showing both in the same file:

 

And the correlation of Rep 13 of the LotConfigPropertyLotObject property then corresponds to the IID of the second exemplar as well as to the Lot Resource Key if it is a ploppable:

 

3 hours ago, smf_16 said:

As a first step I think we could ignore the props & flora and just try to plop a lot with only the building and its textures.

I concur. As you well know it's best to eliminate as many variables when trying to decipher something. I would grab Block ALL Maxis Buildings by DuskTrooper and then start with one single growable industrial lot with 5 jobs available and one single growable low wealth residential. I'd make them both growth stage 1 and 1 x 1 in size.

From peeking in SGE, I believe it also keeps track of the elevation of each building so I would start with standard level ground. Ideally if your program could output the data for each lot to a text file with each item on it's own line and named by the IID concatenated with its X and Z coordinates it would then be easy to compare the data.

So, a first test would be to grow one of each, output the data then use the same city tile, but raise the land via God mode, grow them in the same place, and output again.

 

[smf_16 1,835]

Good evening everyone,

Time for some updates about the progress of this project. As I mentioned above, plopping lots would require me to edit both the Building Subfile and the Lot Subfile, so I went ahead and implemented a parser for the building Subfile and also rewrote my parser of the Lot Subfile so that I'm now able to fully edit & save buildings and lots. Given that I eventually will need to implement a parser for the Prop Subfile, I implemented this right away  as well. I am planning on merging this code into the master branch of the sc4 module so that the classes can be used by other people to experiment with editing those subfiles. I am also planning to document the code a bit more and allow "plugin classes" to be made so that anyone who wants to write a parser for a subfile type should be able to. More on that will probably follow soon.

I have also implemented an index data-structure that accepts a list of directories and builds up an index of every subfile it can find within DBPF files. As such every subfile within a DBPF file should be accessible by TGI. This means that given a TGI that corresponds to a Lot Configuration Exemplar I should be able to find the corresponding building, props, textures and flora exemplars. This should in theory give me enough information to plop the lot in a Savegame.

Given that I'm now able to add lots & buildings to a savegame, I tried to add an arbitrary lot. The lot did display successfully in the SGE, but unfortunately it did not show up in the game. Neither the building nor textures or even the zones. Hence I decided to take a step back and create a city with two existing lots and then try to move the second lot a few tiles to the right. I did already experiment with this a bit in this post as well, and the same problems showed up. The code I used to move the lot and its building looks like

// Lots are moved in tiles
let lot = lotFile.lots[1];
let dx = 5;
lot.minX += dx;
lot.maxX += dx;
lot.commuteX += dx;

// Buildings are moved in meters
let building = buildingFile.buildings[1];
dx = dx * 16;
building.minX += dx;
building.maxX += dx;

And the result can be seen here (both in SC4 and in SGE):

A few things can be noted here:

• The building did successfully move 5 tiles to the right, both in SC4 and SGE.
• The textures did not move along.
• The underlying lot has successfully moved 5 tiles to the right as well in SGE.
• When exiting to region without saving, I experience a CTD (which was also experienced when trying to move lots under bridges before).

The fact that the textures don't move seems to indicate that in contrast to what I thought, textures of a lot actually are saved in their own subfile. This seems to be Type ID 0xc97f987c because it corresponds to the C++ class cSC4LotBaseTextureOccupant. Unfortunately this sub file is not decoded yet so I will need to do this myself. The CTD's when exiting to region without saving indicate that something is still wrong, but I have no idea what this could be, so for now my best shot is to try to decode the texture subfile, move the textures along and see if this fixes the CTD's. I'm kinda fearing it won't though.

Interestingly I am also unable to delete the moved lot. If I use the bulldoze tool over the textures, the offset building highlights, but actually clicking doesn't do anything, neither when I hover the building directly and try to bulldoze it. I think this has the same root cause as the immortal lots.

I decided to try two more variants and see what the effect was on both the CTD as the bulldozing:

• Keep the lot in place, but move the building 5 tiles to the right so that it is no longer on the lot.
  • Unable to bulldoze the lot, neither bulldozing the lot itself, nor the building.
  • No CTD when exiting to region
• Keep the lot in place, and move the building half a tile to the right so that it becomes overhanging.
  • Able to bulldoze the lot
  • No CTD when exiting to region

This definitely makes it clear that thare are still some hidden variables that control the placement of a lot, apart from the data in the lot subfile itself. For the moment I have no clue what this could be. I'm kind of hoping that it's the textures as this file still seems reasonable to decode. However, I fear it's something else. I should probably do some diffing to compare what subfiles are different if I put the lot on a different spot in the game itself. Let's hope that there aren't too many changed subfiles so that I'm able figure out those hidden variables quickly.

More soon!

 

- Seba

[smf_16 1,835]

So, I did some further research on the diffs. I wrote a script that compared two cities that both contained the same two lots, except that the second lot was moved - much like I did to try with the move, but then "hand plopped". First I compared the length of the subfiles. The script told me that there were 8 subfiles with a different length:

• 0xaa371c32: Reward Advice Subfile (cSC4RewardAdvice)
• 0x8a09f5f4: Advice Subfiles (cSC4Advice)
• 0xa9bc9ab6: An undecoded file, corresponding to the cSC4Occupant class
• 0x098f964d: Item Index subfile (cSC4OccupantManager)
• 0x8a2482b9 (3 times, once IID 0x00000000, 0x00000002 and 0x00000004): The PNG previews
• 0xe86b1eef: The directory file.

It's pretty logical that the png previews did modify as well as the directory file, and we can pretty sure rule out the advice subfiles as well. That leaves us with

• The Item Index Subfile. Looking at the wiki doesn't make it any clearer for me if this could file could be related. I'm guessing it isn't but I think writing a parser for this file so that I can inspect it more in depth. Shouldn't be too hard.
• The cSC4Occupant file. Unfortunately this file is not decoded yet, but when I read it as a raw buffer it only holds 46 bytes (where the first 12 are probably size, crc and mem), so that should be doable. I'm feeling that this one could be related somehow.

That's for the files that did differ in length. Next I went on to compare the contents of the files that did not differ in length. The script told me there were 113 files that had differing contents (even with the first 12 bytes, being the size, crc and mem cut off) with only 21 that remained the same. That's a bit too much to figure out so I guess I'll have to work on decoding the texture file and this cSC4Occupant first. If anyone has any suggestions about what might be causing these CTD's when moving a lot it would be greatly appreciated!

As for the texture file, I already did some experiments with plopping a 1x1 Open grass field in the upper-left corner of a map. Thus far I've tried 5 positions: [0, 0], [1,0], [2,0], [2,1] and [2,2] and 1 position for 1x1 Open paved ([2,2]). The buffers look like this:

Spoiler

       SIZE     CRC      MEM      Maj. Min.                                                                             xPos     yPos     zPos                                   count    id?      x  z

# 1x1 Open grass field

[0,0]: 5b000000 9833149a 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 cdcccc3d 00008743 cdcccc3d 66667e41 cd0c8743 66667e41 02 01000000 00302725 00 00 00 00 ffffffff ff 03
[1,0]: 5b000000 4ed5bcbc 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 cdcc8041 00008743 cdcccc3d 3333ff41 cd0c8743 66667e41 02 01000000 00302725 01 00 00 00 ffffffff ff 02
[2,0]: 5b000000 5eadad48 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 cdcccc3d 9a993f42 cd0c8743 66667e41 02 01000000 00302725 02 00 00 00 ffffffff ff 03
[2,1]: 5b000000 2af3515f 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 cdcc8041 9a993f42 cd0c8743 3333ff41 02 01000000 00302725 02 01 00 00 ffffffff ff 03
[2,2]: 5b000000 f30300f6 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 66660042 9a993f42 cd0c8743 9a993f42 02 01000000 00302725 02 02 00 00 ffffffff ff 02

# 1x1 Open paved

[2,2]: 5b000000 6b206ec2 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 66660042 9a993f42 cd0c8743 9a993f42 02 01000000 00308e25 02 02 00 00 ffffffff ff 03

 

I've tried to add some spaces in between to what I think might be the structure. It's starting to look like something, but more experiments will be required, notably with lots that are larger than 1x1.

I also noticed that 1 texture record is created per lot that actually has textures, even if the lot uses multiple different textures. If the lot has a transparent base texture though, no entry is created in the texture sub file. That makes perfect sense actually.

Allright, that's it for today. See you next time!

 

- Seba

[clarencethemayor 13]

The three columns following the xpos, ypos, and zpos floats in the hex buffer seem to be

1. end x coordinate of texture: all entries have a number that is 0.1 less than where the next tile starts at the next multiple of 16 (meters).
2. angle of orientation: all entries have 270.0 degrees, which means they're facing due south.
3. end z coordinate of texture: same as #1 but on the z axis for each entry.

We definitely need to test lots with textures that cover multiple tiles to figure out what most of the other entries mean.

Update: I created a savefile with buildings (plopped and zoned) as in the image below and nothing else, and here is my annotated 0xc97f987c file for the save:

Spoiler

Elem School Facing North
BD000000 5DE0F16C F821F203 0200 0400 00000005 9D6D7F49 40 45 40 46 0200 0200 00000000 00000000 00000000 00000000 CDCCCC3D 00207F43 CD0CB843 3333FF41 9A397F43 33F3CF43 02 08000000
*Parking Lots*
0020A325 00 19 00 00 FFFFFFFFFF 00
00202B25 01 19 00 00 FFFFFFFFFF 01
00202B25 00 19 00 00 FFFFFFFFFF 00
0020A325 01 19 00 00 FFFFFFFFFF 00
*Main Building*
0011FB25 01 18 00 00 FFFFFFFFFF 03
0011FB25 00 18 00 00 FFFFFFFFFF 03
0011FB25 00 17 00 00 FFFFFFFFFF 02
0011FB25 01 17 00 00 FFFFFFFFFF 02 

Elem School Facing East
BD000000 BCD80D85 5822F203 0200 0400 00000005 9D6D7F49 40 45 41 46 0200 0200 00000000 00000000 00000000 00000000 66664042 11018143 CD0CB843 CDCCBF42 DE0D8143 33F3C743 02 08000000
*Parking Lots*
0020A325 03 17 01 00 FFFFFFFFFF 00
00202B25 03 18 01 00 FFFFFFFFFF 03
00202B25 03 17 01 00 FFFFFFFFFF 02
0020A325 03 18 01 00 FFFFFFFFFF 00
*Main Building*
0011FB25 04 18 01 00 FFFFFFFFFF 03
0011FB25 04 17 01 00 FFFFFFFFFF 03
0011FB25 05 17 01 00 FFFFFFFFFF 03
0011FB25 05 18 01 00 FFFFFFFFFF 01 

Elem School Facing South
BD000000 DE843A55 B822F203 0200 0400 00000005 9D6D7F49 41 45 42 46 0200 0200 00000000 00000000 00000000 00000000 3333E042 DD0D8143 CD0CB843 66E60F43 AA1A8143 33F3CF43 02 08000000
*Parking Lots*
0020A325 08 17 02 00 FFFFFFFFFF 00
00202B25 07 17 02 00 FFFFFFFFFF 03
00202B25 08 17 02 00 FFFFFFFFFF 00
0020A325 07 17 02 00 FFFFFFFFFF 00
*Main Building*
0011FB25 07 18 02 00 FFFFFFFFFF 03
0011FB25 08 18 02 00 FFFFFFFFFF 02
0011FB25 08 19 02 00 FFFFFFFFFF 03
0011FB25 07 19 02 00 FFFFFFFFFF 03 

Elem School Facing West
BD000000 0095A5BE 1823F203 0200 0400 00000005 9D6D7F49 42 45 43 46 0200 0200 00000000 00000000 00000000 00000000 9A192043 00008143 CD0CB843 66E64F43 CD0C8143 33F3C743 02 08000000
*Parking Lots*
0020A325 0C 18 03 00 FFFFFFFFFF 00
00202B25 0C 17 03 00 FFFFFFFFFF 03
00202B25 0C 18 03 00 FFFFFFFFFF 00
0020A325 0C 17 03 00 FFFFFFFFFF 00
*Main Building*
0011FB25 0B 17 03 00 FFFFFFFFFF 01
0011FB25 0B 18 03 00 FFFFFFFFFF 02
0011FB25 0A 18 03 00 FFFFFFFFFF 01
0011FB25 0A 17 03 00 FFFFFFFFFF 03

House
85000000 4E8E94C0 7823F203 0200 0400 00000005 9D6D7F49 42 45 42 45 0200 0200 00000000 00000000 00000000 00000000 9A192043 00008143 CD0CA043 66E62F43 CD0C8143 33F3AF43 02 04000000
00102B25 0A 15 00 00 FFFFFFFFFF 00
00103825 0A 14 00 01 FFFFFFFFFF 00
00102B25 0A 14 00 00 FFFFFFFFFF 02
00103725 0A 15 00 01 FFFFFFFFFF 00

Shop
77000000 ACFCC539 D823F203 0200 0400 00000005 9D6D7F49 43 45 43 45 0200 0200 00000000 00000000 00000000 00000000 9A194043 00008143 CD0CA043 66E64F43 CD0C8143 33F3AF43 02 03000000
*Pavement*
00108D25 0C 14 00 00 FFFFFFFFFF 00
00108D25 0C 15 00 00 FFFFFFFFFF 01
*Parking Lot Decal*
0010B425 0C 14 00 00 FFFFFFFFFF 00

Wind Tower
69000000 31EE8BB4 3824F203 0200 0400 00000005 9D6D7F49 42 44 42 44 0200 0200 00000000 00000000 00000000 00000000 9A192043 00008143 CD0C8843 66E63F43 CD0C8143 33F38F43 02 02000000
*Base Texture*
0000D625 0A 11 00 00 FFFFFFFFFF 00
0000D625 0B 11 00 00 FFFFFFFFFF 00

 

 

That being said, here is my guess for an interpretation of this Base Texture file:

DWORD	Size
DWORD	CRC
DWORD	Memory
WORD	Major Version?  (only seen 0x0006)
WORD	Minor Version?  (only seen 0x0004)
WORD	Unknown (only seen 0x0000)
BYTE	Unknown  (only seen 0x00)
BYTE	Unknown (only seen 0x05)
DWORD	0x9D6D7F49  (probably always the same)
BYTE	Min Tract X Coordinate  (normally between 0x40 and 0x7F)
BYTE	Min Tract Z Coordinate  (normally between 0x40 and 0x7F)
BYTE	Max Tract X Coordinate  (normally between 0x40 and 0x7F)
BYTE	Max Tract Z Coordinate  (normally between 0x40 and 0x7F)
WORD	X Tract Size? (power of 2)  (only seen 0x0002)
WORD	Z Tract Size? (power of 2)  (only seen 0x0002)
16 bytes unknown (only seen 16 0x00 bytes, could be related to SaveGame Properties)
FLOAT32	Min X Coordinate
FLOAT32	Min Y Coordinate
FLOAT32	Min Z Coordinate
FLOAT32	Max X Coordinate
FLOAT32	Max Y Coordinate
FLOAT32	Max Z Coordinate
BYTE	Unknown
DWORD	Texture Entry Count 
    DWORD Unknown (probably parameters for generating texture)
    BYTE	Tile X Coordinate
    BYTE	Tile Z Coordinate
    BYTE	Lot Orientation
    BYTE	Texture Overlay Priority? (usually 0x00, seen 0x01 for Low R with pavement on grass)
    BYTE	Unknown (only seen 0xff)
    DWORD	Unknown (only seen 0xffffffff)
    BYTE	Unknown (seen 0x00, 0x01, 0x02, 0x03 but changes between saves in same lot)

[smf_16 1,835]

I've been looking a bit more at it and inspired by how the Prop Subfile is encoded, I think that I figured out some additional fields.

       SIZE     CRC      MEM      Maj. Min.                   tract x z min/max                                         xMin     yMin     zMin    xMax     yMax      zMax        count    id?      x  z

# 1x1 Open grass field

[0,0]: 5b000000 9833149a 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 cdcccc3d 00008743 cdcccc3d 66667e41 cd0c8743 66667e41 02 01000000 00302725 00 00 00 00 ffffffff ff 03
[1,0]: 5b000000 4ed5bcbc 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 cdcc8041 00008743 cdcccc3d 3333ff41 cd0c8743 66667e41 02 01000000 00302725 01 00 00 00 ffffffff ff 02
[2,0]: 5b000000 5eadad48 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 cdcccc3d 9a993f42 cd0c8743 66667e41 02 01000000 00302725 02 00 00 00 ffffffff ff 03
[2,1]: 5b000000 2af3515f 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 cdcc8041 9a993f42 cd0c8743 3333ff41 02 01000000 00302725 02 01 00 00 ffffffff ff 03
[2,2]: 5b000000 f30300f6 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 66660042 9a993f42 cd0c8743 9a993f42 02 01000000 00302725 02 02 00 00 ffffffff ff 02

# 1x1 Open paved

[2,2]: 5b000000 6b206ec2 18202b03 0200 0400 00000005 9d6d7f49 40 40 40 40 0200 0200 00000000 00000000 00000000 00000000 66660042 00008743 66660042 9a993f42 cd0c8743 9a993f42 02 01000000 00308e25 02 02 00 00 ffffffff ff 03

The 270 seems to be the y-position because when inspecting the city with SGE, I saw that all lots are at yPos 270. As far as I know the orientation of a lot seems to be stored in a byte, where 0x00 = North, 0x01 = East, 0x02 = South and 0x03 = West. I think that the orientation is one of the two bytes after the x & z tile coordinates.

[smf_16 1,835]

Allright, so I pretty much figured out the structure of the LotBaseTexture File (Type Id 0xc97f987c )

Spoiler

DWORD	Size	
DWORD	CRC	
DWORD	Memory	
WORD	Major version (0x0002)	
WORD	Minor version (0x0004)	
BYTE	Unknown, only seen 0x00	
BYTE	Unknown, only seen 0x00	
BYTE	Unknown, only seen 0x00	
BYTE	Unknown, only seen 0x00	
DWORD	 0x497f6d9d (always the same)	
BYTE	Min Tract X (normally between 0x40 and 0x7f)	
BYTE	Min Tract Z (normally between 0x40 and 0x7f)	
BYTE	Max Tract X (normally between 0x40 and 0x7f)	
BYTE	Max Tract X (normally between 0x40 and 0x7f)	
WORD	X Tract Size? (only seen 0x0002, probably indicating the size is 2²)	
WORD	Z Tract Size? (only seen 0x0002, probably indicating the size is 2²)	
DWORD	Unknown, only seen 0x00000000	
DWORD	Unknown, only seen 0x00000000	
DWORD	Unknown, only seen 0x00000000	
DWORD	Unknown, only seen 0x00000000	
FLOAT32	Min X Coordinate	
FLOAT32	Min Y Coordinate	
FLOAT32	Min Z Coordinate	
FLOAT32	Max X Coordinate	
FLOAT32	Max Y Coordinate	
FLOAT32	Max Z Coordinate	
BYTE	Unknown, seen 0x01 and 0x02	
DWORD	Count of tiles with a texture	
	DWORD	Instance ID of the texture
	BYTE	X tile
	BYTE	Z tile
	BYTE 	Orientation
	BYTE	Priority 0x00 for base texture, 0x01 for overlay texture
	4 BYTES	Unknown, mostly 0xff, seen several other values as well
	BYTE	Unknown, mostly 0xff but seen 0x03 and 0x01 as well
	BYTE	Unknown, seen 0x00 up to 0x07

 

There are still quite a few unknown bytes, but I created a parser for it and ran a few fully developed cities through it and all were parsed nicely without any byte offset errors or something. I also kept track of the values that appeared for the unknown bytes. I added them in the structure above. Quite a lot of zeroes here, so for my purposes of moving textures I don't think I will need to investigate this further for the moment.

@jdenm8 I think that this can be put on the wiki as well.

I'm am going to test soon what the effect of moving the textures is on the CTD when exiting without saving. I'll keep you guys posted!

 

EDIT: @clarencethemayor, seems like I didn't see your edit. I think we pretty much come to the same conclusion here, so that's a nice double-check! Thanks a lot! 

[smf_16 1,835]

So, I went on to see if I was able to move the textures as well by using what I've learned from the decoded texture subfile. This time I used a small 1x1 Chicago tileset lot that @CorinaMarie made for me. I moved it 5 tiles to the right and this was the result:

As was to be expected, the texture did move nicely to the right, however it did leave a hole. I was thinking that this is perhaps related to the fact that an overlay texture was still there. I found that overlay textures are also stored in the Basetexture file because the textures array had a length of 2. Overlay textures seem to have a priority flag set to 0x01:

DWORD	Count of tiles with a texture	
	DWORD	Instance ID of the texture
	BYTE	X tile
	BYTE	Z tile
	BYTE 	Orientation
	BYTE	Priority 0x00 for base texture, 0x01 for overlay texture
	4 BYTES	Unknown, mostly 0xff, seen several other values as well
	BYTE	Unknown, mostly 0xff but seen 0x03 and 0x01 as well
	BYTE	Unknown, seen 0x00 up to 0x07

I'm able to move the base and overlay texture independently (as can be seen below) but some very strange things happen when I do this. If I switch to underground view and back, the textures have suddenly both disappeared.

I also get the CTD when exiting to region if I move the textures independently, whereas I didn't get it if I moved both textures. Weird. If anyone has a suggestion for why this hole might still be left that would be greatly appreciated! For the moment I don't really have a clue here...

Now, while doing all the investigations for the textures, there's something else that caught my eye. Both the buildings, props, textures & flora entries all have properties called "minTractX" etc. So what actually is this "tract"? The wiki often states that these values range between 0x40 and 0x7f (so 64 and 127). Well, I actually discovered "by accident" what they were. I already mentioned yesterday something about the Item Index Subfile. It states that a large city has 64 tracts, and given that a large city is 256 tiles, this means that a tract is a block of 4x4 tiles.

I managed to write a parser for the Item Index file - which wasn't too hard - and as such I was able to inspect what this Item Index Subfile looks like for a city. As stated in the wiki, the Item Index always contains 192 x 192 cells. This is a rather strange number, but more interestingly in a fully occupied large city tile the only cells that were non-zero were the ones between [64,64] and [127,127]. Aha! We've got a match here! This means that the min/maxTract values map to those entries in the item index. Vice versa every cell in the item index (so basically every block of 4x4 tiles) has a list of all items that overlap this tract, the item being referenced by the "memory address" which seems to be a C++ pointer. I'm guessing that this Item Index File is actually a kind of quadtree that the game uses to quickly find which objects can be found on a certain set of 4x4 tiles.

Some remarks I have with this Item Index:

• If we move (or plop) a lot then we have to keep our item index up to date of any changes. I have tried to remove the item index from the savegame to see if the game perhaps would build up a new one from scratch, but the game simply refused to show anything. Hence we will need to add some manual bookkeeping to keep this index up to date.
• I think that the CTD's I have been experiencing might be related to the fact that I was not updating the item index correctly, but that's again just a guess.
• I am starting to get scared a bit by those memory addresses. They are obviously C++ pointers, but how do we have to manage them? It's no problem for the game to insert them into the savegame, but I have no idea how it uses them when it reloads a savegame. Does it actually write back the entries to the same physical addresses? What if that memory address is already occupied at that time? Or does it just use them in an associative array when loading the savegame, where the pointer is simply being used as the key? I will need to do some experiments with this. If it actually writes back the entries to the same physical addresses, then I think we are going to have some trouble if we want to insert new records into the savegame. I'm turning myself to @simmaster07 here, perhaps you know more about these memory addresses?

I decided to look a bit deeper into those memory addresses. I realized that I am able to find all records that use a structure "SIZE CRC MEM" by a script that looks like this:

let all = [];
let mems = new Set();
for (let entry of dbpf) {
	let buff = entry.decompress();
	let size = buff.readUInt32LE();

	// If what we're interpreting as size is larger than the buffer, 
	// it's impossible that this has the structure size crc mem!
	if (size > buff.byteLength) continue;

	// Calculate the checksum. If it matches the second value, then we 
	// have something of the structure "size crc mem". Perhaps more follow here.
	let crc = crc32(buff, 8);
	if (crc === buff.readUInt32LE(4)) {

		let records = [];
		while (buff.length > 4) {
			let size = buff.readUInt32LE(0);
			records.push(buff.slice(0, size));
			buff = buff.slice(size);
		}

		let type = entry.type;
		for (let record of records) {
			let mem = record.readUInt32LE(8);
			if (mems.has(mem)) {
				console.log('Double memory address!');
			}
			mems.add(mem);
			all.push({
				"mem": mem,
				"type": type,
				"buffer": record
			});
		}
	}

}

Basically this calculates the CRC checksum for every record in the savegame and if it matches bytes 4 to 8 (which is where the checksum would be stored), then I decide that this record has structure "SIZE CRC MEM ..." and I can index the record by memory address. I did this for the city with the moved lot and I found 92 entries that are referenced by their memory address. I sorted the list by memory address and then compared the differences. There were 20 entries for which the difference in memory address was smaller than the entry length in bytes. This makes me believe that the actual values of the memory addresses aren't important here, but they simply have to be consistent and cannot collide. I will have to test this by playing with those memory addresses a bit, but perhaps I should first try to keep the item index up to date and see if that fixes the CTD's I've been experiencing.

Whew, that was a long one! I was expecting some hard times when I took on this project, and that did indeed prove to be true. Nevertheless I think I've already obtained a lot of valuable knowledge that will be required for anyone who wants to touch the savegame in the future.

Thanks for bearing with me until the end of this update & see you guys soon!

 

- Seba

[simmaster07 1,220]

54 minutes ago, smf_16 said:

If it actually writes back the entries to the same physical addresses, then I think we are going to have some trouble if we want to insert new records into the savegame. I'm turning myself to @simmaster07 here, perhaps you know more about these memory addresses?

I'm actually not sure. I can take a look tonight or tomorrow night to see what the game does with these values, but I wouldn't think that they're used to reinitialize memory at the same address since the memory addresses used by the game can change depending on numerous factors at the operating system level that the game can't control.

I'll also be taking a look at the game savefile format to see what I can come up with, especially for things that haven't been decoded yet. My progress is pretty minimal so far since I was inspired to take this up recently thanks to your work, but I've found some cool information about the basic DBPF format itself. https://1drv.ms/w/s!AoJBhhfu0HgWwmLZWy9Q7PYWVD6A

 

[CorinaMarie 29,605]

I've used the dump function of the 0.0.14 version after having two of the Simple Lot houses grown in the game. Both show:

"IID": "0x0b361188",
"buildingIID": "0x0b361188",

but those numbers don't match up to the IID in the .dat file. I was expecting to see:

0xA706ED25

So what I'm wondering is if there's a sub file where it stores the real IID, but then has it's own internal number by which it references it.

 

[CorinaMarie 29,605]

6 hours ago, smf_16 said:

EDIT: @clarencethemayor, seems like I didn't see your edit.

I didn't see it either until I noticed you mentioned an edit.

Speaking as staff: Keep in mind it's perfectly fine to have back to back posts when adding new info. Now, if you'd want to keep something grouped with something in a prior post then an edit is fine, but also add a new post to say: Hey! I edited my post above.

[smf_16 1,835]

7 hours ago, simmaster07 said:

I'll also be taking a look at the game savefile format to see what I can come up with, especially for things that haven't been decoded yet. My progress is pretty minimal so far since I was inspired to take this up recently thanks to your work, but I've found some cool information about the basic DBPF format itself. https://1drv.ms/w/s!AoJBhhfu0HgWwmLZWy9Q7PYWVD6A

That's great! Really glad that I inspired you and I think your knowledge and experience from the DLL-stuff will come in handy when decoding the entire savegame. Is that document something you found somewhere, or something you're writing yourself as a kind of manual? I see "endianness?" mentioned somewhere in the document. For all I know everything i've encountered is little endian - which is pretty logic as it comes from a Windows game. I'm actually wondering if this is the same for the Mac version, or whether it uses BE here.

6 hours ago, CorinaMarie said:

I've used the dump function of the 0.0.14 version after having two of the Simple Lot houses grown in the game. Both show:

"IID": "0x0b361188",
"buildingIID": "0x0b361188",

but those numbers don't match up to the IID in the .dat file. I was expecting to see:

0xA706ED25

So what I'm wondering is if there's a sub file where it stores the real IID, but then has it's own internal number by which it references it.

I think this might be related to the building being part of a building family. I'm not an expert in families though, but perhaps you should have a look in the SimCity_1.dat file if you can find this somewhere?

[CorinaMarie 29,605]

6 hours ago, smf_16 said:

I think this might be related to the building being part of a building family.

Yep! That seems to be the case. I did a new test in which I used one single 1 x 1 growable residential from our:

 

and then the IID and buildingIID in your dump file does match up exactly with what I expected to see.

[simmaster07 1,220]

11 hours ago, smf_16 said:

Is that document something you found somewhere, or something you're writing yourself as a kind of manual? I see "endianness?" mentioned somewhere in the document. For all I know everything i've encountered is little endian - which is pretty logic as it comes from a Windows game. I'm actually wondering if this is the same for the Mac version, or whether it uses BE here.

I'm writing it myself as a reference manual. Right now a lot of the content comes from existing resources on SC4DWiki/SimsWiki, but there are a couple of minor new additions like the byte sequence for embedding a DBPF in another file (which I still need to verify, I just noticed that in the DBPF loading code).

For endianness, I'm fairly confident that all architectures use little endian, but I need to double check that there actually is a byte order swap being done on big endian versions of the game.

[smf_16 1,835]

Hello everyone,

 

I finished my update yesterday with the conclusion that I needed to dive deeper and understand how those memory references work. While they seemed a bit scary at first sight, they have actually proven to be extremely useful in exploring the savegame. As I am able to read the memory reference from every record in each sub file (as they all start with SIZE CRC MEM), I realized that I could also search for where they are being referenced! I implemented a very trivial search function and had it output some nice results which look like this when I run an established city through it:

Spoiler

INFO Lot references were found in:

┌─────────┬─────────────────────┐
│ (index) │       Values        │
├─────────┼─────────────────────┤
│    0    │      'cSC4Lot'      │
│    1    │ 'cSC4ZoneDeveloper' │
└─────────┴─────────────────────┘

INFO Building references were found in:

┌─────────┬──────────────────────────────┐
│ (index) │            Values            │
├─────────┼──────────────────────────────┤
│    0    │    'cSC4BuildingOccupant'    │
│    1    │    'cSC4DepartmentBudget'    │
│    2    │      'cSC4LotDeveloper'      │
│    3    │    'cSC4OccupantManager'     │
│    4    │ 'cSC4PipeConnectionOccupant' │
│    5    │   'cSC4PlumbingSimulator'    │
│    6    │   'cSC4PollutionSimulator'   │
│    7    │     'cSC4PowerSimulator'     │
└─────────┴──────────────────────────────┘

INFO Texture references were found in:

┌─────────┬──────────────────────────────┐
│ (index) │            Values            │
├─────────┼──────────────────────────────┤
│    0    │ 'cSC4LotBaseTextureOccupant' │
│    1    │    'cSC4OccupantManager'     │
└─────────┴──────────────────────────────┘

INFO Prop references were found in:

┌─────────┬───────────────────────┐
│ (index) │        Values         │
├─────────┼───────────────────────┤
│    0    │ 'cSC4OccupantManager' │
│    1    │  'cSC4PropDeveloper'  │
│    2    │  'cSC4PropOccupant'   │
└─────────┴───────────────────────┘

 

This information is extremely useful as I am now able to easily identify how the different sub files in the savegame are linked to eachother! For example, I’m now pretty sure that the textures are only referenced in the cSC4OccupantManager – which is the Item Index Subfile – and of course in the cSC4LotBaseTextureOccupant as that’s where they are defined. Hence I am going to try first if I can insert textures anywhere I want. I now know that I will only need to insert them in the Texture Subfile and the Item Index.

Because this functionality to find references within the savegame is so useful, I have added the functionality to the command line utility as well (version 0.0.15-alpha). I added a command “refs” that can be used as

sc4 refs “your city.sc4”

By default it will look for all lot, building, texture & prop references, but you can customize this behaviour by using the respective flags (-l, -b, -t and -p) or specify type ids manually using the –types flags. For example (notice the comma in between!)

sc4 refs “your city.sc4” –types 0xc9c05c6e,0xa9c05c85

will look for all references to the network records (Type ID 0xc9c05c6e) and Flora records (Type ID 0xa9c05c85). As such anyone that feels like decoding one of the yet unknown subfiles can use this command to understand a bit more what is going on within the save file.

Beware though that you only use this command on relatively small cities. Using the command on large tiles that are completely filled up can take a significant amount of time (like hours). Therefore I added the option to limit the amount of references per type that will be searched. For example, if there are 8000+ lot records then you can use the –max flag to search for a limited amount of lot references. You don’t really need to find anyway where all your lot records are referenced. If you search for like 100 of them, you’ll probably find most file types where they are referenced anyway.

If you don’t want to look for the references of records in a specific type if, you can also choose to look for references of a specific memory address. You can use the –address flag for this. For example, consider you have opened up SGE and found that a lot has memory address 0x034b012c, you can look for it by running

sc4 refs “your city.sc4” –address 0x034b012c

Multiple addresses can be passed by separating them with a comma. I will post this in my other thread as well as this is pretty general information about the savegames.

 

Allright cool, so we figured out that the textures only get referenced in the Item Index File. Let's see what would happen if we change the memory values to something else, but the same for both. I increased the mem value with 4 and opened the game. It worked perfectly. I then changed the value of the memory reference to something else, but only in the Item Index Subfile, not in the BaseTexture File and as expected the texture did not show up.

Notice here how I did move the texture 2 tiles to the right as well. I had to do this because apparently the game fills in the base texture automatically under a lot if it doesn't find a texture underneath. So even when the memory pointers were different the texture would still show up.

I think the conclusion here is that the game loads up all records in memory and then uses the Item Index to determine what has to be drawn depending on the visible portion of the map. The item index uses pointers that refer to the physical locations of the records. That's actually quite a common technique in computer graphics. But as such we do know now that if we mess with objects on the map (such as adding and moving) we will need to update the item index as well.

Just out of curiosity, can we set the memory pointers to any value we want? Let's see what happens when we set it to 0.

Spoiler

Woops! CTD! But that was quite to be expected, computers don't really like null pointers. If I set the memory pointer to 1, the game worked fine. Let's see what happens if I set the memory pointer to 1 less than another pointer that is used throughout the entire DBPF, for example the building pointer. If the values of the pointers have any actual meaning, we would interfere with the building, so this is worth testing. If interference would occur this means that if we were to insert a new item into the savegame, we will need to pick an eligible memory address that doesn't interfere with other objects. Luckily this seemed to work fine. Even when I set the pointer to the same value as the building pointer the game still showed both the building and the textures. Hence this makes me believe that we can use any pointer we like per Type ID as long as it's not 0 and is unique for every record with the same Type ID.

I have to say that this is actually quite a relief. Imagine that we had to keep track of the size of all our records and make sure that the pointers we were assigning to our records needed to take this size into account. The world would have been a different place (well, kind of  ). Of course this is only a provisionary conclusion. Some testing will be needed, but at least we have an assumption here to test.

So, with this new load of information, would we be able to add a new texture? Unfortunately not yet. I cloned the existing texture record, assigned it its own pointer and inserted it both into the Item Index and the Texture Subfile, but it didn't show up in the game. I don't know why. I was kind of hoping that inserting it into the index would do the trick, but there still seem to be other variables under the surface. Perhaps the game just refuses to draw a "ghost" texture if it finds that it doesn't belong to a lot. I think I will need to go back and try to insert a new lot itself instead of just the texture.

If we run the city containing this single lot through the "refs" command, we find

INFO Lot references were found in:
┌─────────┬─────────────────────┐
│ (index) │       Values        │
├─────────┼─────────────────────┤
│    0    │      'cSC4Lot'      │
│    1    │ 'cSC4ZoneDeveloper' │
└─────────┴─────────────────────┘
INFO Building references were found in:
┌─────────┬────────────────────────────────────┐
│ (index) │               Values               │
├─────────┼────────────────────────────────────┤
│    0    │      'cSC4AutomataAttractor'       │
│    1    │  'cSC4AutomataControllerManager'   │
│    2    │      'cSC4AutomataGenerator'       │
│    3    │ 'cSC4BuildingDevelopmentSimulator' │
│    4    │       'cSC4BuildingOccupant'       │
│    5    │         'cSC4LotDeveloper'         │
│    6    │       'cSC4OccupantManager'        │
└─────────┴────────────────────────────────────┘

Probably we can skip the cSC4AutomataAttractor, cSC4AutomataControllerManager and cSC4AutomataGenerator. These things seem to be related to the people that are showing up to look at the house, as happens when you put a Mayor's statue as well for example. Nevertheless I think I will need to dive down the cSC4ZoneDeveloper and cSC4LotDeveloper classes. Unfortunately both haven't been decoded yet, so I guess we know what to do next!

2 hours ago, simmaster07 said:

I'm writing it myself as a reference manual. Right now a lot of the content comes from existing resources on SC4DWiki/SimsWiki, but there are a couple of minor new additions like the byte sequence for embedding a DBPF in another file (which I still need to verify, I just noticed that in the DBPF loading code)

Well I've gained quite some understanding of how the DBPF files work while developing all of this, so if there's something you're not sure of, you know where to find me!

 

- Seba

[smf_16 1,835]

Allright so I started decoding the cSC4ZoneDeveloper file. I read it in from an empty small city tile and it turns out there was only 1 record in it having a length of 16406 bytes in it. Wow. That would be hard to decode, right? Ok, but let's take its square root, which is 128.0859086707043. Boom! Pretty close to 128, that's what we like to see. Now 128 squared is 16384, which is a difference of 22 bytes. Let's see what those first 22 bytes look like:

SIZE      CRC      MEM     vers x size   z size 
16400000 1ab26c48 20b55715 0100 40000000 40000000 

The two unknown values here are 0x40000000, which is 64 when read as LE UInt32. Cool, that corresponds to the amount of tiles in a small city, so we can assume that some kind of table having 64 rows and 64 columns is following. 16384 / (64 x 64) = 4, so probably every entry in the table will be a UInt32.

Inspecting the values of the table, I found that all of them were 0 though. Given that we're working with an empty city this is pretty normal though, so I guess that if we fill up the city the values will become non-zero and probably hold some additional items as well. This is actually very similar to how the Index Item Subfile is encoded. I also verified what the header looks like when opening a medium sized city and it that case the size values are 0x80000000 - (128) - which corresponds to the size of a medium city.

So, I went ahead and plopped an open paved area in the upper left corner of my small city. The length of the record was now 16410, so 4 bytes have been added. Let's look at what the contents are of the table:

14001f03 4a5dbdc9 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...

Those numbers seem a little weird, but remember that we did encounter a lot pointer in this file. Hence let's carry out a ref search for 0x031f0014 (remember that the data is given as Little Endian, so we need to carry out a byte swap!). The ref search reports the following:

sc4 refs "City - Move bitch.sc4" --address 0x031f0014

INFO Searching for 0x31f0014
INFO 0x031f0014 was found in:
┌─────────┬─────────────────────┐
│ (index) │       Values        │
├─────────┼─────────────────────┤
│    0    │      'cSC4Lot'      │
│    1    │ 'cSC4ZoneDeveloper' │
└─────────┴─────────────────────┘

That makes sense. The first entry is a pointer to the lot that is occupying this tile. Now have a close look at the second entry. It's 0xc9bd5d4a which happens to be the Type ID for the Lot Subfile. So it seems that if the pointer is non-null, it simply stores the Type ID of the Lot Subfile afterwards. We actually did encounter this kind of behavior already in the Lot Subfile for the linked industrial lots. If the pointer is non-null, then 0x4A232DA8 always follows, which is the Type ID of the industry subfile. Seems like we have the same behavior here.

So... I gues we could say that another one has bitten the dust. The structure of the cSC4ZoneDeveloper looks like this:

DWORD	Size	
DWORD	CRC	
DWORD	Memory	
WORD	Major version	
DWORD	Number of tiles in x direction (64, 128 or 256)	
DWORD	Number of tiles in y direction (64, 128 or 256)	
(Repeat 64 x 64, 128 x 128 or 256 x 256 times, loops columns first, so (0,0) then (0,1) then (0,2)...)		
DWORD	Memory address of lot occupying this tile (repeat for all tiles)	
	DWORD	If memory address is not 0x00000000, always 0xc9bd5d4a (Type ID of Lot Subfile)

I'm going to add a parser for this to the code to check if there might be some variants to this structure, though I'm pretty confident that it is like that. After that I'm going to tackle the cSC4LotDeveloper class, but I'm guessing the structure of this file is going to be similar.

I am actually really happy with all the progress I'm making here. Never expected it would go this fast, but let's not cheer to early  See you soon!

 

- Seba

[CorinaMarie 29,605]

1 hour ago, smf_16 said:

City - Move bitch.sc4

 

Btw, and I doubt this will be of any use to you, but just in case there is something useful you might want to look at something I wrote in LUA code a while back. It's posted in The C.O.R.I. Reports (beta version) thread. More specifically, you might be interested in the All Tables Raw Report it can generate. I'll attach one from a fully developed large city tile so you can just peek and see it there's anything worthwhile in it.

Corillion (I6) 1863-01-27 - All Tables Report.txt

[smf_16 1,835]

Well, bad news. I was writing some tests for the ZoneDeveloperFile parser which worked nicely for a small city tile. However, when trying to parse a large city tile there was a CRC mismatch. The crc that I calculated didn't match the crc that was stored in the entry. I though that it was perhaps related to the fact that the data block length was surpassed so I re-wrote a bit of the C++ code so that it uses a streamified crc implementation by cutting the buffer in chunks, but this didn't seem to work. The CRC I generated using this chunking strategy was still not matching the CRC that was stored in the Savegame so there must be something else going on.

I'm turning once again to you @simmaster07. When you discovered the CRC checksum algorithm used in the prop file, how exactly did you find it? I think we need to find out what the game does with large buffers when calculating the CRC checksum  in the cSC4ZoneDeveloper class. The buffer that didn't work contained 383 554 bytes. In the small city where the checksum was still calculated correctly the size was only 16 502 bytes. I tried a medium city as well, where the buffer contained 65 702 bytes, and this one passed as well.

Now perhaps it is the game that calculates the checksum erroneously. I will do a test where I modify the checksum to the one I calculated and see if the city still opens correctly, but that's something for tomorrow. Meanwhile any input on why the checksum might fail for a certain amount of bytes is more than welcome!

36 minutes ago, CorinaMarie said:

Btw, and I doubt this will be of any use to you, but just in case there is something useful you might want to look at something I wrote in LUA code a while back. It's posted in The C.O.R.I. Reports (beta version) thread. More specifically, you might be interested in the All Tables Raw Report it can generate. I'll attach one from a fully developed large city tile so you can just peek and see it there's anything worthwhile in it.

I will have a look at it. What exactly is this "All Tables Raw Report" if I may ask?

 

- Seba

[CorinaMarie 29,605]

9 minutes ago, smf_16 said:

I will have a look at it. What exactly is this "All Tables Raw Report" if I may ask?

I used LUA coding to scan the memory while the game is running and the output is simply every table the game has stored in memory that I knew the name of and then the report is the names and content of said tables. (I'd have to find my notes to refresh my memory to be able to say more.) I did do a lot of testing way back then and was even able to successfully enable debug mode too (which allowed me to run some of the other LUA code that is otherwise bypassed).

[CorinaMarie 29,605]

On 6/26/2019 at 6:59 PM, CorinaMarie said:

"IID": "0x0b361188",
"buildingIID": "0x0b361188",

CB and I've been testing and it turns out I'd made a mistake and the ploppable one could also grow. That IID is from the ploppable one and when it does grow it has all the full data when viewed via the dump option.

So, false alarm before.

[simmaster07 1,220]

1 hour ago, smf_16 said:

I think we need to find out what the game does with large buffers when calculating the CRC checksum  in the cSC4ZoneDeveloper class. The buffer that didn't work contained 383 554 bytes. In the small city where the checksum was still calculated correctly the size was only 16 502 bytes. I tried a medium city as well, where the buffer contained 65 702 bytes, and this one passed as well.

If the size is greater than 250,000 bytes, only calculate the CRC32 value for the first 250,000 bytes. I can only assume the game does this to make saving a little faster for computers built in 2002.

[simmaster07 1,220]

On 6/26/2019 at 2:18 PM, smf_16 said:

I am starting to get scared a bit by those memory addresses. They are obviously C++ pointers, but how do we have to manage them? It's no problem for the game to insert them into the savegame, but I have no idea how it uses them when it reloads a savegame. Does it actually write back the entries to the same physical addresses? What if that memory address is already occupied at that time? Or does it just use them in an associative array when loading the savegame, where the pointer is simply being used as the key?

So the memory addresses are just used as a key in an associative array, where Key : Value :: Address : C++ Class Metadata. 0 is a special value indicating that no mapping exists, even if it might be a valid key into that array, which causes the game to not initialize a C++ object for that instance, causing the crash.

[smf_16 1,835]

4 hours ago, simmaster07 said:

If the size is greater than 250,000 bytes, only calculate the CRC32 value for the first 250,000 bytes. I can only assume the game does this to make saving a little faster for computers built in 2002

Thanks man! I did realize after a night of sleep that I should probably use a large city tile and plop something in the bottom-right corner and see if that caused a difference in the checksum and then do a binary search to see what the max bytes were. Still this would require the memory address to remain consistent though, which is not guaranteed I guess. You just saved me from all that work! So stupid that I didn't come up with the fact that it might simply be present already in your code...

59 minutes ago, simmaster07 said:

So the memory addresses are just used as a key in an associative array, where Key : Value :: Address : C++ Class Metadata. 0 is a special value indicating that no mapping exists, even if it might be a valid key into that array, which causes the game to not initialize a C++ object for that instance, causing the crash.

Cool, so that confirms the conclusion I made after my experiments. This should get me rolling again.

[smf_16 1,835]

Hello everyone,

Time for an update of today's progress! Quite of a scary moment I had yesterday with the CRC mismatches, but the solution as @simmaster07 pointed out was indeed to limit the CRC checksum to 250 000 bytes. As I mentioned yesterday, after I decoded the cSC4ZoneDeveloper class I still had to decode the cSC4LotDeveloper class as well. This one turned out to be rather easy:

DWORD	Size
DWORD	CRC
DWORD	Mem
WORD	Version
DWORD 	City dimension + 1 (65, 129 or 257)
DWORD	Unknown (seen 0x44800000 and 0x45800000)
DWORD	Unknown (seen 0x44800000 and 0x45800000)
DWORD	Count (= amount of buildings in the city)
	DWORD	Building pointer
	DWORD	Building type id (0xa9bd882d)
DWORD	Unknown (seen 0x00000000)
WORD	Unknown (seen 0x0000)

So basically this is just a list of all buildings in the city. If the same building is used mutliple times, it also occurs multiple times.

I did also run another "refs" query on another city of mine (one with just 4 lots: a residential, a commercial, an industrial and agricultural, I use this one for testing the growify command) and this was the result:

Spoiler

INFO Searching for Lot, Building, Texture, Prop
INFO Searching 78 refs (100%)
INFO Started at 23:07:10 GMT+0200 (GMT+02:00)
OK Took 0.392s
INFO Lot references were found in:
┌─────────┬─────────────────────┐
│ (index) │       Values        │
├─────────┼─────────────────────┤
│    0    │      'cSC4Lot'      │
│    1    │ 'cSC4ZoneDeveloper' │
└─────────┴─────────────────────┘
INFO Building references were found in:
┌─────────┬────────────────────────┐
│ (index) │         Values         │
├─────────┼────────────────────────┤
│    0    │ 'cSC4BuildingOccupant' │
│    1    │   'cSC4LotDeveloper'   │
│    2    │ 'cSC4OccupantManager'  │
└─────────┴────────────────────────┘
INFO Texture references were found in:
┌─────────┬──────────────────────────────┐
│ (index) │            Values            │
├─────────┼──────────────────────────────┤
│    0    │ 'cSC4LotBaseTextureOccupant' │
│    1    │    'cSC4OccupantManager'     │
└─────────┴──────────────────────────────┘
INFO Prop references were found in:
┌─────────┬───────────────────────┐
│ (index) │        Values         │
├─────────┼───────────────────────┤
│    0    │ 'cSC4OccupantManager' │
│    1    │  'cSC4PropDeveloper'  │
│    2    │  'cSC4PropOccupant'   │
└─────────┴───────────────────────┘

 

If we ignore the props for now, we see that we did decode all files where lots, buildings & textures are referenced, the cSC4LotDeveloper being the last one that was decoded. Cool, so this should allow us to plop a new lot, right? We just insert an entry in the Lot SUbfile (cSC4Lot) and also insert its position in the cSC4ZoneDeveloper file (which we decoded yesterday). Then we put a building on it by inserting the building into the building subfile (cSC4BuildingOccupant), insert it in the freshly decoded cSC4LotDeveloper file and then add it to the item index at the correct position as well (cSC4OccupantManager). Same story for the textures, we create a new one in the BaseTexture Subfile (cSC4LotBaseTextureOccupant) and then add it to the item index. We are not going to add props for now to our lot, so we don't have to do anything there. If we were to do this, the only sub file we still need to decode is cSC4PropDeveloper.

Now I did very carefully craft this all together and so I opened up SC4 with high expectations but ... the lot didn't show! Aaargh. It still recognized it somehow though because the tile that the lot did occupy could not be built on ("Cannot build on top of reserved tiles"), but nothing was showing. No building, no textures. To be honest this was quite demotivational. Looking at all files that could possible reference the lot, its building or its textures, I was pretty much convinced that I had everything I needed.

Basically I'm out of options here. I verified like a thousand times that I did insert everything correctly. I hand-compared it with a city where I did plop the lot manually and except for the memory pointers and the CRC's, every property of every file (cSC4Lot, cSC4ZoneDeveloper, cSC4BuildingOccupant, cSC4LotDeveloper, cSC4OccupantManager, cSC4LotBaseTextureOccupant) corresponded to city with the hand-plopped building. But still nothing was showing. Again a question for @simmaster07: do you have access to the code what happens when one uses the LotPlop cheat? There has to be something that is still preventing the lot from showing up correctly, but I have absolutely no idea what it could be.

Being out of options, I decided to have a look at the list of all class names again. My eye fell on the cSC4SimGrid* classes. Perhaps they still had some flags set about what lots are visible? I had a look at what those files look like and I found that they store data in a square table. In order to find out what the data in the table actually represents, I wrote some code that was able to visualize this data and output it into a pdf file. I've attached it to this post. It shows the record number in the concerned cSC4SimGrid file, then the header and then the visualized data. The test city I used for this is shown in the picture below.

Spoiler

I've added an example header of a cSC4SimGrid file below. I've figured out most of the structure of the header, although not everything is completely clear to me.

Size     CRC      MEM      vers (A) Type id  Data id? (B)     (C)      x size   z size   (D)       (E)     (F)      (G)
37100000 f47eadfb 984cd113 0100 01 02e6b949 86bcd549 01000000 00000000 40000000 40000000 00008041 00008041 0000803d 0000803d

For (A) I've only seen the value 1. From the pdf it seems that (B) is the tile size used for the data, 0x00000001 meaning every data point is 1x1 tiles, 0x00000002 meaning every data point is 2x2 tiles etc.. (C) Seems to have a similar meaning where 0x00000000 means 1 data point is 1x1, 0x00000001 means 1 data point is 2x2, ... (D), (E), (F) and (G) are all unknown to me, but again they seem related to the how many tiles a single data point represents.

For an overview, I'd say the structure is

DWORD	Size	
DWORD	CRC	
DWORD	Memory	
WORD	Version	
BYTE	Unknown (only seen 0x01)	
DWORD	File Type ID	
DWORD	Data id of what is represented in this table?	
DWORD	Related to tile resolution (1 for 1x1, 2 for 2x2, 4 for 4x4)	
DWORD	Related to tile resolution (0 for 1x1, 1 for 2x2, 2 for 4x4)	
DWORD	X size	
DWORD	Z size	
DWORD	Unknown (values depend on tile resolution)	
DWORD	Unknown (values depend on tile resolution)	
DWORD	Unknown (values depend on tile resolution)	
DWORD	Unknown (values depend on tile resolution)	
	BYTES	Repeat x * z times, data type depends on file type id

I haven't been able yet to find out what all this data represents though. One that I did figure out was that for cSC4SimGridUInt8 data id 0x49d5bc86 (the first one in the pdf) corresponds to the power carrying capacity (for example a 1x1 park carries out power over a 5x5 block). Based on the data visualization you can guess for some what they're related to - for example crime or education - but a lot of it is still unknown. Perhaps that running a bigger a more well-established city through this might shed some more light on this.

Anyway, I started decoding those cSC4SimGrid files because I thought that maybe some of them control the appearance of lots, though I haven't found one that does yet. I think those things are actually simply related to the core simulation itself and not the visualization. I don't think that they're supposed to mess with, but I will need to do some tests here.

So, I hope you're all still with me here because there's a lot of technical stuff in here. Unfortunately that's just how it is with Savegames: they are far more complex than the files we're usually dealing with when creating "normal" mods.

And by the way, in the end I am still left here without any clue why my created lot was not showing up. Guess I'll have to be extra creative in the coming days!

 

- Seba

output.pdf

[Cyclone Boom 16,875]

Hi @smf_16

Just popping in here with a few words of encouragement...

Admittedly a lot of the information is way above my current level of technical understanding, so I'm not specifically sure what to suggest in terms of a way forward. I'm actually quite in awe about all what you've explored thus far. What you've done continuing on from your main saved game editing thread is to achieve what has never been achieved in the history of SC4 modding. That's a remarkable thing right there already. Who knows what could happen next as well in this new field of research.

This really is uncharted territory, and I feel there's lots more territory to chart too. While there might be some aspects which don't immediately add up as you've realised, there's a lot which does. Compared with the 16 years prior, the info which aligns and corresponds is significantly more than what the community was aware of up to this past week. There's so much potential for what could be done too, and so it's something to build upon. You're very determined to keep going with these explorations, and also have the skills and patience to succeed. Also it's great how you've shared all these details, even if for some of us it's rather very complicated.

These continued investigations into saved game editing likely encompass a long term project. It's an endeavour which might not be "solved" in the days ahead, but thinking in the longer term of the weeks, months, or even years to come. Oftentimes ideas form a chain reaction of knowledge, and that's how new developments can transpire.

So do keep it up. You're doing a mighty fine job.

[clarencethemayor 13]

Hey @smf_16, even though you did not see your inserted lot in the game, did the Query Tool or any of the Data Views recognize it as a lot?