Programmatically generating a city: A savegame experiment

[carlosmarcelo 975]

1 hour ago, smf_16 said:

It's real fun to play with. The city below is only one out of 100+ cities that I've generated

Can I test one or two this cities?  I would like to do all for city: streets, water, power, police, jobs...

Is it possible send me by PM (or here if you prefer), please? 

[smf_16 1,835]

49 minutes ago, carlosmarcelo said:

Can I test one or two this cities?  I would like to do all for city: streets, water, power, police, jobs...

You mean you want to test the program, or a specific generated city?

The thing is, there's not really much to test yet. It's basically just eye candy because I haven't figured out yet how to insert the population into the city. Even though the city looks very crowded, there are no people living in it according to the stats. Also, it would be difficult for you to build streets because the buildings are inserted completely at random so a lot of them will not even be reachable by roads. That would require a "smarter" algorithm that takes into account where the streets would be, but that's not my focus at the moment. I want to focus now on inserting the props and the textures.

About those props, I've been able to insert most of them, but for some reason I'm dealing with brown boxes even though all of it is just pure Maxis content. I will also need to figure out how to include timed props etc. etc. The Prop Subfile isn't extremely clear to me though unfortunately.

[carlosmarcelo 975]

34 minutes ago, smf_16 said:

You mean you want to test the program, or a specific generated city?

Test the generated city.  But, seeing the image, I noticed that all the buildings are "floorless", that is, even if I hit where to place the street, I don't know if it will work without the floor.

I do not advise you to worry about placing streets as well because I believe it is something of enormous effort for little return, especially at this time of learning. It is more worthwhile to think just about getting the total and complete generation of buildings right.

My guess is that the empty boxes are invalid codes: those TID, GID, TGIs that you usually say that I don't understand anything about. 

But, regardless of everything, his work is getting more and more beautiful. Congratulations! 

[smf_16 1,835]

53 minutes ago, carlosmarcelo said:

Test the generated city.  But, seeing the image, I noticed that all the buildings are "floorless", that is, even if I hit where to place the street, I don't know if it will work without the floor.

I sent you one of the generated cities in a PM, but as you mention yourself, it will probably be next to useless.

53 minutes ago, carlosmarcelo said:

I do not advise you to worry about placing streets as well because I believe it is something of enormous effort for little return, especially at this time of learning. It is more worthwhile to think just about getting the total and complete generation of buildings right.

Yeah, generating streets will probably be extremely difficult, especially with everything the NAM has done. I've been thinking of this: if I want to be able to generate networks, I will probably need to be able to interpret the RUL codes in the NAM so that I can look up the correct textures and paths - at least that's how I think it works. That would be an enormous effort though, so I'm not pursuing this at all at the moment. As you say, it's an enormous effort for very little return.

A valuable alternative which I think is doable is to put placeholders or so - for example the "single road tiles" which you place for flattening terrain - where the network should come. Subsequently it's up to you to determine how to fill in the network: streets, roads, NWM, you name it. But even then, that's not the focus at the moment.

53 minutes ago, carlosmarcelo said:

My guess is that the empty boxes are invalid codes: those TID, GID, TGIs that you usually say that I don't understand anything about. 

As far as I know, a brown box appears whenever a model file is missing. Looking at the Prop Subfile, you see that you have to fill in a TGI here of the prop exemplar, which is what I do in my code. However, it seems that somehow this prop exemplar is not referencing the correct model file, which is weird as this doesn't happen in the game of course. I will need to test single lots where this happens I guess so that I can compare what happens in the Prop subfile when the game plops the lot and when I do it programmatically.

[smf_16 1,835]

While trying to include the textures, I noticed something strange. It seems that the game sometimes just picks textures at random. Below is a lot that @CorinaMarie made for me to test. Even without generating anything, the texture randomly changes between those two. I don't even need to save the city. Simply reopening is enough to make it switch to the other one.

Is this known behavior of the game? I'm trying to understand what's going on here because it's pretty annoying to debug if the textures are just changing whenever they feel like it.

[CorinaMarie 29,605]

I believe they are part of a family. So when they grow they can pick one of the set at random. Then in the normal game they stay whatever they were when grown. There might be another flag that game buries in some sub-file that tells it which it picked.

[smf_16 1,835]

I thought about families as well, but the difference with building and prop families is that for those a variant is picked "at plop time" and is then baked into the Savegame which one was actually picked. So as far as I know nothing changes anymore when reopening the city. That's what's weird with the textures: the game seems to pick one at random when the city is opened. I don't think it's baked into the Savegame because they change even if I reopen the city without saving.

[CorinaMarie 29,605]

And if you plop the lots normally in a game do they also change upon reopening the city?

[Cyclone Boom 16,876]

20 minutes ago, smf_16 said:

Is this known behavior of the game?

As @CorinaMarie said, this is due to being a texture family. These are like prop families and are a group of textures which are part of a set.

An example I know of these are with the Maxis 1x1 Plaza lot, and this alternates every once in a while to display a different texture instead. On the lot itself though, it's one individual texture and so it'll be a case of looking up what this is to know the group of those inside.

For instance, on Cori's test lot, this is an overlay texture which has a TGI of:

0x7AB50E44, 0x0986135E, 0x25291000

 

[smf_16 1,835]

23 minutes ago, CorinaMarie said:

And if you plop the lots normally in a game do they also change upon reopening the city?

Yes, the picture that I've shown was not generated by my program, but by hand plopping.

 

12 minutes ago, Cyclone Boom said:

As @CorinaMarie said, this is due to being a texture family. These are like prop families and are a group of textures which are part of a set.

An example I know of these are with the Maxis 1x1 Plaza lot, and this alternates every once in a while to display a different texture instead. On the lot itself though, it's one individual texture and so it'll be a case of looking up what this is to know the group of those inside.

For instance, on Cori's test lot, this is an overlay texture which has a TGI of:

0x7AB50E44, 0x0986135E, 0x25291000

So if I understand correctly, the lot itself points to a single TGI, but inside the file with that TGI reside multiple textures which the game is then choosing from at random? That would make sense, but it's still confusing that this behavior is so different from how building and prop families work.

[Cyclone Boom 16,876]

42 minutes ago, smf_16 said:

So if I understand correctly, the lot itself points to a single TGI, but inside the file with that TGI reside multiple textures which the game is then choosing from at random? That would make sense, but it's still confusing that this behavior is so different from how building and prop families work.

Yeah, it does seem that way.

How I've seen prop families get defined once plopped or grown on a lot, but for the textures like the Open Paved Area they keep alternating automatically every so often. Like when seeing the texture appearing in a certain way to begin with, it changes between the 4 styles in the family. Seemingly the game doesn't store which texture is shown once the lot is built, and so it sees the first one as the reference and then keeps switching through the others.

This appears to happen once reloading a city tile, from where it then randomly picks what each texture should be shown from the family.

Here's an animated preview I made to show this:

The frames aren't in real time, of course. That would be super silly.

[mattb325 28,880]

Looking good (I see others beat me to the various questions over the last few days). Like Cyclone Boom said, this texture family uses a different methodology (which incidentally, can also be used to create prop families).

Outside of Maxis, it is seldom used, but it is worth being mindful of. You can read a bit more about it here: https://www.sc4devotion.com/forums/index.php?topic=5350.msg169424#msg169424 and here: https://www.sc4devotion.com/forums/index.php?topic=15673.0

[smf_16 1,835]

Thanks @Cyclone Boom and @mattb325, it's clear now to me. It's a bit annoying while debugging but ok, will have to live with that.

Now, let's go back to making the textures appear. Quick recap of what the structure of the Texture Subfile is assumed to look like:

Spoiler

DWORD	Size	
DWORD	CRC	
DWORD	Memory	
WORD	Major version (0x0002)	
WORD	Minor version (0x0004)	
BYTE	Unknown, only seen 0x00	
BYTE	Unknown, only seen 0x00	
BYTE	Unknown, only seen 0x00	
BYTE	Unknown, mostly 0x05 (A)
DWORD	Unknown (B)
BYTE	Min Tract X (normally between 0x40 and 0x7f)	
BYTE	Min Tract Z (normally between 0x40 and 0x7f)	
BYTE	Max Tract X (normally between 0x40 and 0x7f)	
BYTE	Max Tract X (normally between 0x40 and 0x7f)	
WORD	X Tract Size? (only seen 0x0002, probably indicating the size is 2²)	
WORD	Z Tract Size? (only seen 0x0002, probably indicating the size is 2²)	
DWORD	Unknown, only seen 0x00000000	
DWORD	Unknown, only seen 0x00000000	
DWORD	Unknown, only seen 0x00000000	
DWORD	Unknown, only seen 0x00000000	
FLOAT32	Min X Coordinate	
FLOAT32	Min Y Coordinate	
FLOAT32	Min Z Coordinate	
FLOAT32	Max X Coordinate	
FLOAT32	Max Y Coordinate	
FLOAT32	Max Z Coordinate	
BYTE	Unknown, seen 0x01 and 0x02	
DWORD	Count of tiles with a texture	
	DWORD	Instance ID of the texture
	BYTE	X tile
	BYTE	Z tile
	BYTE 	Orientation
	BYTE	Priority 0x00 for base texture, 0x01 for overlay texture
	4 BYTES	Unknown, mostly 0xff, seen several other values as well
	BYTE	Unknown, mostly 0xff but seen 0x03 and 0x01 as well
	BYTE	Unknown, seen 0x00 up to 0x07 (C)

 

I've noticed that byte (A) and dword (B) do play an important role here. I need to set (A) to 0x05 and (B) to 0x497f6d9d, otherwise the base texture does not show up and a greenish grass texture is used as base texture. Interestingly though, this doesn't work for textures - or lots - that are on every 4'th tile of the grid, as can be seen in the picture below ¯\_(ツ)_/¯.

The lot I circled in red is one I plopped by hand so that I could compare, all other lots are created with my program (notice how I'm actually creating them, not just cloning the lots and then moving them). There's still something wrong with the building alignment as well by the way. It has to do with the orientation not being taken into account correctly, I'm still looking into that issue.

Anyway, I decided to take a look what the texture looks like when plopped on a 4th tile. Intrestingly (B) was still 0x497f6d9d. What changes was byte (C) for the texture with priority 0, being the base texture. Below is a table of how this changes in function of the different tile coordinates (x, z) for a 5x5 grid that I plopped:

Spoiler

┌─────────┬───┬───┬─────┐
│ (index) │ x │ z │ (C) │
├─────────┼───┼───┼─────┤
│    0    │ 0 │ 0 │  3  │
│    1    │ 0 │ 1 │  3  │
│    2    │ 0 │ 2 │  1  │
│    3    │ 0 │ 3 │  1  │
│    4    │ 0 │ 4 │  2  │
│    5    │ 1 │ 0 │  3  │
│    6    │ 1 │ 1 │  0  │
│    7    │ 1 │ 2 │  1  │
│    8    │ 1 │ 3 │  0  │
│    9    │ 1 │ 4 │  0  │
│   10    │ 2 │ 0 │  2  │
│   11    │ 2 │ 1 │  2  │
│   12    │ 2 │ 2 │  0  │
│   13    │ 2 │ 3 │  2  │
│   14    │ 2 │ 4 │  2  │
│   15    │ 3 │ 0 │  3  │
│   16    │ 3 │ 1 │  1  │
│   17    │ 3 │ 2 │  0  │
│   18    │ 3 │ 3 │  0  │
│   19    │ 3 │ 4 │  3  │
│   20    │ 4 │ 0 │  1  │
│   21    │ 4 │ 1 │  3  │
│   22    │ 4 │ 2 │  0  │
│   23    │ 4 │ 3 │  3  │
│   24    │ 4 │ 4 │  3  │
└─────────┴───┴───┴─────┘

 

If anyone sees the pattern here, please do tell because I really don't have a clue.  It's also possible that I've been staring too long at it. Let's hope tomorrow clears it up a bit!

[rivit 3,391]

5 hours ago, smf_16 said:

If anyone sees the pattern here

Well I cant see an immediate pattern, but I wonder if these are controlled in a similar way to the Network Props T21, which operate, in general, on a 4x4 grid. The x05 could be a bitfield mask not a size field.  indicating 0101 or alternating. If you draw a 4x4 grid on your image it does repeat/tile

Actually the more I think about this A=5 is a constant not a bitfield. It is a 4x4 pattern though, and the C values look randomly selected in range 0-3

~~~~~~

On family FSH - the game features a number of multi-textures (usually 4) in the base and overlay textures. They are not addressable, but display randomly to combat tiling and give variation. You can only specify the FSH TGI - the game does the rest. 

[smf_16 1,835]

9 hours ago, rivit said:

Well I cant see an immediate pattern, but I wonder if these are controlled in a similar way to the Network Props T21, which operate, in general, on a 4x4 grid. The x05 could be a bitfield mask not a size field.  indicating 0101 or alternating. If you draw a 4x4 grid on your image it does repeat/tile

Actually the more I think about this A=5 is a constant not a bitfield. It is a 4x4 pattern though, and the C values look randomly selected in range 0-3

I think it is in fact a bit field. I've compared it to what happens in the Prop Subfile and there a similar thing happens: you have the appearance flag, always followed by a fixed dword (0xA823821E in this case). Most of the time this appearance flag is set to 0b00001001 as wel, so that's probably also the case here.

About those "greenish grass textures" that are appearing on the 4x4 grid, I've modified the lot a bit to remove the overlay texture so that all I have is a base texture, makes it a bit less chaotic when you look at it. Now what I've noticed is that actually the green texture that the game inserts is present everywhere, it simply isn't always showing up because within the "cells" the base texture that actually should be drawn is showing up. However, if you zoom in and out a bit it's clear that there's some z-fighting going on here, as I show below. It baffles my mind a bit. Also note how the z fighting is not happening with the "hand plopped" lot, so that indicates to me that no green grass texture is inserted underneath.

Anyway, there are basically two questions that need to be solved here:

• Why is the game inserting this grass texture on its own?
• Why doesn't the base texture show up on the grid lines?

EDIT: This is what happens when I fill up an entire city tile, which was to be expected:

Spoiler

 

[carlosmarcelo 975]

14 hours ago, smf_16 said:

I sent you one of the generated cities in a PM, but as you mention yourself, it will probably be next to useless

NO CTD at all time. This validates your work very well. 

Before starting anything, I made some observations:

1. I noticed that in the zone view (remembering that I use the NAM plugin), all the brown boxes are visible (in addition to other small props). This can give you a clue as to what they represent.

2. I don't know if it was intentional, but it is possible to pass the streets from end to end through the city without buildings along the way, forming a fairly regular grid. (ignore the lower left corner because there were other tests)

3. Almost every box has the graffiti demonination.

After placing the complete infrastructure, nothing happened by itself, I needed to create a mini RCI neighborhood and connect with neighbors (simnation). Now follow new observations after starting the city:

4. Logically, there are "no street" zots, but when you pass the street, the commercial zots disappear and they become functional, but the residential ones are abandoned.

5. When passing the streets over the buildings, they are not demolished, they are over the street:

5.1. when trying to demolish a building that had a street passed under it, the street is demolished, the animation of the building falling is done, but it is still there;

5.2. when zoning in this situation, nothing grows;

Unfortunately that is all I have been able to do so far. I hope these tests can help you to complete something. 

I still want to compare this city with the PlopAllBuildings and PlopAllLots functions. In the past, I have done tests similar to these cheats in order to have an unknown city of random generation, but it was not feasible because the mixture is so intense that you cannot work. Your generator seemed much more efficient for that purpose. 

Note: I'm sorry if the English is bad: I used the translator and I didn't have much time to review... 

 

[smf_16 1,835]

I have to say that the textures are rather intriguing. Have a look at the picture below. There are two things I tried here:

1. In my code I forced the texture 0x25cd0000 to be inserted (the football pitch), rather than the one that is stored in the LotConfigurations. I also set this as the base texture for the hand plopped lot. When I did that, the texture showed up correctly in the grid cells. On the grid lines, I still had the grass everywhere. Also, the base texture of the lot that was already present in the city did not change. Therefore I'm inclined to say that somewhere in the Savegame the game stores what textures it has already inserted. If it doesn't find a texture here for a specific lot, it looks it up in the Texture Subfile and then inserts it. This means that the texture subfile cannot be used to modify existing textures. Interesting.
2. After that I modified the lots orientation. I accidentally set it to (x + z) instead of (x + z) % 4 and that resulted in the image below. Tiles where (x+z) >= 4 did not show the grass texture underneath. I corrected my mistake to (x + z) % 4 and then the grass came back everywhere. Just to be clear, this orientation (x + z) got set on the entry in the Lot Subfile, it has nothing to do with the texture. So I can only imagine that the game is looking for something that it can't find (because the lot's orientation is set to an invalid value) and that results it just giving up and not inserting the grass texture - which is actually what we want.

Definitely some stuff to think about... Gotta say it's a lot harder than I hoped it would be 

[smf_16 1,835]

I just had a tiny breakthrough and was able to insert the textures on the grid lines as well.

So, what happened here? Well, the Texture Subfile needs to have the bounding box of the texture:

FLOAT32	Min X Coordinate	
FLOAT32	Min Y Coordinate	
FLOAT32	Min Z Coordinate	
FLOAT32	Max X Coordinate	
FLOAT32	Max Y Coordinate	
FLOAT32	Max Z Coordinate

However, when inspecting the hand plopped lot I noticed that the minX coordinate was not 16 as one would expect, but 16.1. Similarly the maxX coordinate was not 32 but 31.9. Therefore I somehow assumed that the game required some kind of "inset" on the textures and so I did this as well. Removing the inset made the textures show up, however I had to set maxX to 16 as well, otherwise nothing was showing up!

My guess here is that the game uses those insets for the textures once they're "baked in" - which is what happend with the plopped lot. For textures that are not "baked in" yet, the game probably requires the coordinates of the upper-left corner of the tile to be set exactly. However, the quest to finding where this "baking in" happens is still open!

Cool, now what happens if we apply this to the skyline?

The textures seem to be properly inserted, however it looks like they're put on the wrong tiles. That's just because I still messed up the building alignment though, so I guess that's something that I will need to fix first. As far as I could see no z-fighting of green grass base textures appeared in the city by the way, which still happened though with the test case of the simple 1x1 R§ lot.

[rivit 3,391]

The grass texture between the 3x3s with the football pitch looks like the default 300m terrain texture - I think 7AB50E44-891B0E1A-00000434. The other looks like high wealth Grass 7ab50e44-0986135e-252b3004

If you edited a city at a higher elevation or loaded an alternate terrain texture pack it would look different if that were the case.  I take it you have deliberately varied the lot orientations in a regular way to get the pattern you have. Is that C from your discussion above or some other value?

Also If the save game stores the textures at tile level then presumably there's a chunk of say (citysize x citysize in tiles)*4 + 12 somewhere. in fact there may be two, one for the terrain, and one for all lots/network tiles placed. From other properties in exemplars it seems there are layers of tiles used for different purposes, so it may then be a layer in a 3d array, which would make it harder to find but that would be a bigger contiguous chunk.

~~~~~~

Meanwhile your latest find at least clears up why row/col 0 never worked properly. It also suggests a lookup somewhere based on coordinate mod 16 - perhaps the committed textures block.

Also a thought on Prop Boxes - are these family props? they start with a even high order hex digit and require an exemplar of their family list to be substituted at instantiation.  

[smf_16 1,835]

1 hour ago, rivit said:

The grass texture between the 3x3s with the football pitch looks like the default 300m terrain texture - I think 7AB50E44-891B0E1A-00000434. The other looks like high wealth Grass 7ab50e44-0986135e-252b3004

The default 300m terrain texture I guess is simply the terrain texture showing from underneath, not anything the game has entered. That was because I messed up the lot orientation and set it to something higher than 3 which apparently caused the game not to insert that high wealth grass texture.

 

1 hour ago, rivit said:

If you edited a city at a higher elevation or loaded an alternate terrain texture pack it would look different if that were the case.  I take it you have deliberately varied the lot orientations in a regular way to get the pattern you have. Is that C from your discussion above or some other value?

No, that C byte is not of any importance apparently. I though it was because it was the only difference I could find between a texture on (0,0) and (1,1). However, this was caused to the "texture insetting" that I did. As such byte (C) seems to be completely random. I tried setting various values for it: all didn't seem to have any effect at all. Perhaps that it's related to orientation or something because I've seen values up to 0x07 in fully established cities (taking into account orientation has 8 values if you count mirroring as well).

1 hour ago, rivit said:

Also If the save game stores the textures at tile level then presumably there's a chunk of say (citysize x citysize in tiles)*4 + 12 somewhere. in fact there may be two, one for the terrain, and one for all lots/network tiles placed. From other properties in exemplars it seems there are layers of tiles used for different purposes, so it may then be a layer in a 3d array, which would make it harder to find but that would be a bigger contiguous chunk.

Well in my command line utility I have a command called refs which can look up what subfiles contain a specific byte sequence. I searched for the football pitch texture 0x25cd0000 and it only appeared within the Texture Subfile - which I created a Wiki page for by the way. So if the game contains such a chunk, it is in any case not referencing the texture by that id.

That's something that completely baffled my mind by the way: in a city with only the hand plopped lot where I changed the base texture 0x252b1000 to 0x25cd0000 the lot still showed the original texture, but 0x252b1000 was nowhere to be found anymore inside the savegame! If the game was showing the original texture, you'd expect it to at least have a reference to it somewhere.

1 hour ago, rivit said:

Also a thought on Prop Boxes - are these family props? they start with a even high order hex digit and require an exemplar of their family list to be substituted at instantiation. 

No idea, I haven't looked into them for now. Maybe just to clarify how the program works:

• It accepts a LotConfigurations exemplar (Exemplar Type 0x10) and finds all props defined as LotConfigPropertyLotObject.
• It then looks up the IID, which is rep 13 inside the LotConfigPropertyLotObject.
• Subsequently I check the index of all families if I can find this as a prop family. This index is built up by reading in all prop exemplars and looking up the Prop Family property (0x27812870). Then I pick a random prop of the family. If I don't find a family with that IID, I look up my index of all prop exemplars and just use that one.
• Once I have suitable Prop exemplar, I insert the TGI of the exemplar into the Prop Subfile.

The thing is that my program does not encounter any errors, meaning that it can find all prop exemplars specified on the lots. The case that I get a brown box looks to me as if the 3D model that is referenced from the prop exemplar is missing. It's going to need some research, but I went on to the textures first in the hope that those would be easier. Boy, was I wrong. 

[rivit 3,391]

Working backwards in your post

  There are 3 props with missing models in the core SC4 data - on the Seaport Dock. Three power poles have missing or partly missing models. There are a couple with missing animations or effects.  But other than that the buildings and props are clean in terms of models and textures.

9 hours ago, smf_16 said:

It then looks up the IID, which is rep 13 inside the LotConfigPropertyLotObject

In general your methods looks sound, but may be in reverse order of lookup. Method should be Direct Lookup on IID in the Prop List and if that fails, then PropFamily List lookup and choose one at random, if that fails we have a missing dependency.  Be aware that rep 13 in the Lot config type 1 can be a list of prop exemplars and/or families . In this case nreps>13 and an entry is chosen randomly (from the nreps-12 values) and then subjected to the same lookup sequence  - this is like a family but local to the lot config itself. Its a shorthand way of ensuring fixed chance use of a prop. 

9 hours ago, smf_16 said:

If the game was showing the original texture, you'd expect it to at least have a reference to it somewhere.

Yes I'd agree, but we probably need to draw the conclusion that the texture is therefore always referenced through the lot config - in which case if the lot changes it is updated internally on city loading when all of the required resources are gathered together. This seems reasonable as missing base textures are left that way (a fact exploited to make transparent lots), and missing prop textures show a 4 colour flag texture.  

9 hours ago, smf_16 said:

As such byte (C) seems to be completely random

It may be which sub texture to use from a multifsh and stored - but I can't see why the game couldn't choose those randomly at any time - they do seem to change in game at regular intervals perhaps on a modulo sequence. I've made a small test 0x252b1000 multitexture with 8 sub textures - you could use this to try things - each sub texture is numbered in the top-left corner.

Test Base Multitexture 252b1000.dat

 

 I have to say you're making great advances here - first class digital archeology.

[smf_16 1,835]

10 hours ago, rivit said:

Be aware that rep 13 in the Lot config type 1 can be a list of prop exemplars and/or families . In this case nreps>13 and an entry is chosen randomly (from the nreps-12 values) and then subjected to the same lookup sequence  - this is like a family but local to the lot config itself. Its a shorthand way of ensuring fixed chance use of a prop. 

Thanks, I didn't know this. I'll take that into account in my prop insertion algorithm and enforce that a prop that is picked actually exists.

10 hours ago, rivit said:

Yes I'd agree, but we probably need to draw the conclusion that the texture is therefore always referenced through the lot config - in which case if the lot changes it is updated internally on city loading when all of the required resources are gathered together. This seems reasonable as missing base textures are left that way (a fact exploited to make transparent lots), and missing prop textures show a 4 colour flag texture.

I think you're right. I ran a quick test which did the following:

• Hand plopped a lot with base texture 0x252b1000
• Verified that it shows up like that in game
• Modified the Texture Subfile in the savegame to have it reference 0x25cd0000 (the football pitch) instead
• Opened up the city, which was still showing 0x252b1000 as was to be expected
• Saved the city and inspected the Texture Subfile, which did reset the base texture to 0x252b1000

So your conclusion is in correspondence with this, adding to it that the game actually will overwrite the base texture to what it finds in the Lot exemplar.

Thinking of it, it also corresponds to a test I did where the game inserted the textures after moving a lot, even though I didn't touch the Texture Subfile - because I didn't know it existed back then.

Still it's interesting to see that with the lots I plopped with my program where I set the football pitch as texture - as opposed to 0x252b1000 that is set in the Lot Exemplar - it was actually the football pitch that was showing up, and not the one in the Lot Exemplar (although with a grass texture underneath). This indicates to me that somewhere in the Savegame there's a hidden variable that tells the game "disregard the textures stored in the Texture Subfile and use the ones you find in the Lot Exemplar". My guess is that this would be stored either in the Lot Subfile, or in the Texture Subfile, but then not at the "tile level", but at the "lot level". Perhaps that this unknown appearance flag (A) has something to do with it?

10 hours ago, rivit said:

It may be which sub texture to use from a multifsh and stored - but I can't see why the game couldn't choose those randomly at any time - they do seem to change in game at regular intervals perhaps on a modulo sequence. I've made a small test 0x252b1000 multitexture with 8 sub textures - you could use this to try things - each sub texture is numbered in the top-left corner.

Could very well be the case. Still, my guess is that upon saving, the game just serializes the C++ data structure which includes whathever texture variant it was showing at time, but completely disregards it when deserializing upon city loading. Anyway, I'll carry out some tests with your lot to see if it influences it.

[smf_16 1,835]

Allright, found two interesting things.

First of all, I found the reason why the R§§§ grass texture was appearing. I realized that when I plop lots, I currently set the zoneWealth to 0x03 in the Lot Subfile (which corresponds to High Wealth). If I leave this to 0x00, I experience CTDs, as I reported earlier in this thread. I was aware that I still had to look up the zone wealth in the building exemplar and then set the appropriate value in the Lot Subfile, I just didn't implement it yet. Anyway, If I set this to 0x01, which is Low Wealth, the correct texture is showing up:

Now, I went looking in the PIM-X for that 0x252b1000 texture and found this:

So apparently the game uses the zoneWealth as well to pick the appropriate texture from a set of multitextures. I wasn't aware of this behavior. However, it seems that the behavior is know because the PIM-X shows an grid of textures, where I can only imagine that each row is a certain zoneWealth (no preview for 0x00, which is logical). The game then probably picks one of the available textures at random in that row to avoid tiling and then puts this in the (C) byte. Do you know if this is reflected that way in the FSH file format, @rivit?

Concerning that appearance flag, I've figured out that it does in fact play a role in whether the game looks up the texture in the lot exemplar or not. If I forced the football pitch texture again in the Texture Subfile and looped all possible values of the flag from 0x00 to 0xff, I got this pattern:

 

Notice here how I'm skipping over the lot at (1, 1) because that was the one I hand plopped, which causes the "shift". Note that every lot still has the "base texture" that was set it the Lot Exemplar as well, and z-fighting happens when I pan & zoom. Note that it doesn't seem to be a single bit that controls it, but rather a combination, though I don't really know which one it is. Anyway, we can see that the value that was observed most of the times (0x05) results in the game not using the texture of the Texture Subfile.

[rivit 3,391]

17 minutes ago, smf_16 said:

The game then probably picks one of the available textures at random in that row to avoid tiling and then puts this in the (C) byte. Do you know if this is reflected that way in the FSH file format

yes that's what I would expect - the wealth is reflected in the IID (digit 5) down the rows, the random choice in the FSH itself (across the row). My test texture should show that by demonstrating different colours corresponding to the C value. Only need to put it in plugins to see it when you plop your usual wealth x01 lots.  

[smf_16 1,835]

This is the result with your test texture:

It does look like it that the variant it was showing upon save is reflected in that (C) byte, so at least we got that one figured out.

Thinking of it, if we look at the Texture Subfile, the byte coming before that (C) byte is mostly set to 0xff, but I've also seen 0x01 and 0x03 in fully established city. My guess is that this is perhaps also related to the texture it has picked, but that when set to 0xff it just picks the exture according to the lot's ZoneWealth.

Now, what was very interesting though is that upon saving, it left the Lot Subfile entries that I created completely untouched, and just inserted them again in the savegame! This means that for my 64x64 city, I now did not have 4096 entries in the Lot Subfile, but 8191! The fact that this equals 2 x 4096 - 1 is probably related to the "hand plopped" lot. Even though there's light at the end of the tunnel, it keeps getting more and more interesting. 

[rivit 3,391]

That looks pretty cool actually, and I think your thoughts on the 0xFF preceding byte is probably true too.

Now if you manually edit that city again just 1 tile do you get an additional 1 new entry in the LotSubfile or is the table blown away and started again. Given a city has many ways of being edited and even varied by the game itself perhaps its semi-redundant to allow fixes, and then revised from time to time - otherwise the file would just get bigger and bigger over time.

[smf_16 1,835]

2 hours ago, rivit said:

Now if you manually edit that city again just 1 tile do you get an additional 1 new entry in the LotSubfile or is the table blown away and started again. Given a city has many ways of being edited and even varied by the game itself perhaps its semi-redundant to allow fixes, and then revised from time to time - otherwise the file would just get bigger and bigger over time.

Yes, if I edited 1 of the textures inserted by the game, opened up the city and saved again an additional entry got added. The change that I made was the fixing the "insets". Remember that I noticed that the texture entries the game creates did have an inset of 0.1: if the lot was at (1,1), then the minX of the texture was 16.1, maxX 31.9 etc. However, back then I was unaware of the fact that the zone wealth plays an important role, so I'm inclided to say that the game actually does require this inset. Otherwise it just neglects everything in the Texture Subfile, while keeping it there when saving it.

That's something that is really annoying when trying to figure out how the textures work: you never know if it's the game that regenerated the textures from the lot config, or whether it reuses the entries in the Texture Subfile. I guess that I'll have to keep a close look on whether texture entries get added or not: if they get added, I know the game is refusing to use my texture entries.

Now about those insets. If we consider that they are required, let's see what happens if we generate them ourselves and then save the city again. I'm no longer going to fill up the entire city, let's scale down a bit and just try plopping 1 lot. The city looked like this:

So, no texture showing up. Pity. Now let's see what happens if we save the city. First of all, after saving, the game properly showed the texture for the plopped lot. That's already something.

More important though, inspecting the Savegame afterwards shows that the game did not add an additional entry in the Texture Subfile, so I guess the conclusion is that the inset is in fact required. I think that when I set minX, minZ etc. to the integer coordinates of the lots (16, 32, ...) the game simply identified them as invalid and decided to redraw them, but in that case it saw that the lots ZoneWealth was set to 0x03 and so it decided to use the green grass instead of the low wealth grass.

After that I closed the city and reopened it again: the texture of the second lot disappeared again! However, upon saving it came back, which was more or less to be expected. This behavior reminds me very much of how the beach textures work: if you save your city, the transparency is lost. You need to enter the zone view to get the transparency back. However, entering the zone view did not remove the texture here, so it's not completely the same.

I'd say that we're almost there. The only thing left - hopefully - is figuring out why the game does not show the texture initially.

Now, keep in mind that all this research did only concern the base texture. I will still need to be able to show the overlay textures as well. One thing that I've already noticed is that the LotConfigPropertyLotObject does not make a distinction between base textures and overlay textures, but I will need it to properly set the priority byte in the Texture Subfile. I've also noticed in the PIM-X that the order they appear isn't relevant either. Do you have an idea how I can distinguish between base and overlay textures, @rivit? By the way, thanks for all your input. It's really valuable to my research and it's nice to have someone with this amount of technical experience thinking along with me!

[smf_16 1,835]

On 11-3-2020 at 11:06 PM, smf_16 said:

The "zoneWealth" in the Lot Subfile must be set to a valid value. If you set it to 0x00, prepare for a CTD.

Coming back on this, I no longer think that setting the ZoneWealth to 0x00 actually causes the CTD. The reason is rather that if there's no texture for the ZoneWealth set in the lot, the game can't find the texture and therefore decides to crash. I've successfully been able to extract the ZoneWealth of the building exemplars and use that one on my skyline and it opened up nicely without any CTD, so I'm confident that it's actually working that way.

Now, good news as well. I was still wondering why the texture did not show up initially, but that was simply because my code was still accidentally setting all values from 0x00 to 0xff for the appearance flag. Setting the appearance flag back to 0x05 fixed the problem and the textures are now all showing up nicely. No new entries in the Texture Subfile are created either upon saving. So I'm quite confident that I'm done with the base textures now 🎉.

And in the skyline:

[smf_16 1,835]

Quick update: I've managed to align the buildings and also position the textures for lots that are larger than 1x1 correctly. The caveat here was that a lot with orientation 0 in the city is actually oriented to the North, while in the Lot Editor - and as such all coordinates of the building, prop and textures - are given with the lot facing South. This means that when positioning the building, props & textures in the city I had to rotate them 180° around the center of the lot first. Took me a while to figure it out, but it seems to be correct now. When compared to the image above, the textures just feel a lot more in line with the buildings. Up next is hunting down those brown boxes of the props.

 

[rivit 3,391]

8 hours ago, smf_16 said:

Do you have an idea how I can distinguish between base and overlay textures

There's only one way I know of to distinguish base from overlay and that requires reading the texture to see if it has transparency - base have none, overlays do.  Even that is potentially unreliable as with Reader its possible to replace a base texture with an overlay so that there is no normal base texture. It might do (for testing purposes) to assume base comes before overlay for the same tile in the Lot Config - but for custom content that's bound to be a faulty premise since it's easily scrambled in LE or Reader.  There is a Timestamp at rep 12 but even that I think is really only for uniqueness - no guarantee overlay will be later than base.

~~~~

And yes I've never figured out why Lot Editor is flipped/rotated by default - it plays havoc with placing non symmetrical props. It orients relative to the road so if the road is at the bottom in LE's display that's N in-game. Everything is oriented relative to N (Top) in the Lot Config unless otherwise stated in the LotConfig line, but its possible to have it look right in LE and then turn out reversed in-game or correct in game with a screwy generated PNG icon file.