discovery Modifying .sc4 savegames: It IS possible!

[smf_16 1,835]

24 minutes ago, CorinaMarie said:

(I personally prefer passing the parameters since I create a separate .cmd file for what I want to do.)

I thought about that too, but I just didn't think people would already be making .cmd files in this early stage!  I'll re-add this functionality, perhaps other people are using cmd's already as well.

[CorinaMarie 29,605]

15 minutes ago, smf_16 said:

I thought about that too, but I just didn't think people would already be making .cmd files in this early stage!  I'll re-add this functionality, perhaps other people are using cmd's already as well.

Woot! Thanks.

(Later I have in mind to create a front end for your program so peeps can just double click an icon they'd place on the desktop (or somewhere) and save them from that scary dos prompt window and as such I'll need all parameters to be passable.)

[Lucario Boricua 7,028]

9 hours ago, CorinaMarie said:

I did some testing forcing different hex values for the zone change and here's the list you prolly already know:

0x01 = R - Low
0x02 = R - Medium
0x03 = R - High
0x04 = C - Low
0x05 = C - Medium
0x06 = C- High
0x07 = I - Low
0x08 = I - Medium
0x09 = I - High
0x0A = Military
0x0B = Airport
0x0C = Seaport
0x0D = Spaceport
0x0E = Plopped Building
0x0F = Plopped Building
 

Ofc, the ones at 10 and up aren't useful in anyway for this project.

Has anyone ever looked into repurposing the military, seaport and spaceport zones? One way I can think of those is actually making mixed residential/commercial zones, albeit with some limitations due to being unrelated zones. It would require creating a whole new class of buildings that can hold both residents and jobs, which, I believe, Cori did demonstrate as being feasible by editing residential buildings.

[CorinaMarie 29,605]

13 minutes ago, Lucario Boricua said:

Has anyone ever looked into repurposing the military, seaport and spaceport zones?

@simmaster07 did look into the unused zones here and created a .dll linked in that post.

 

15 minutes ago, Lucario Boricua said:

It would require creating a whole new class of buildings that can hold both residents and jobs, which, I believe, Cori did demonstrate as being feasible by editing residential buildings.

I did? Cool.

Tho I'm fairly certain that was someone else's mod called something like Work at Home Jobs or such.

[Lucario Boricua 7,028]

( @CorinaMarie Thanks for the clarification on that part, might actually look into that mod.)

Even then I think it's worth trying. There's tons and tons of custom content that's aesthetic, functional mods are harder to come by and require breakthroughs on the game's files. If mixed use zones are made to work, this opens up a whole new world for transit-oriented development and historic districts, where buildings most commonly house residents with either commercial services, commercial offices or light industry. It could be the case that newly gained knowledge on how to externally edit the game files could help with testing such mod.

[smf_16 1,835]

@CorinaMarie I've published a new version on npm (0.0.12-alpha). It gives you the option to prepend the commands with a --no-interactive flag which will ignore interactive mode in that case. Have a look at the help for growify and historical to get an idea of how to use the flags. I guess you now how to update by know

Some examples:

# Makes all residentials and industrials (not agricultural) historical
sc4 --no-interactive historical "My city.sc4" --residential --industrial

# Makes all buildings historical & write to "My historical city.sc4"
sc4 --no-interactive historical "My city.sc4" --all --output "My historical city.sc4"

# Makes all buildings historical & force overrides.
sc4 --no-interactive historical "My city.sc4" -rcig

# Growifies all residentials and sets zone to Medium Density
sc4 --no-interactive growify "My city.sc4" -r Medium

# Growifies all agriculturals & force overrides
sc4 --no-interactive growify "My city.sc4" -g --force

# Growifies all types, setting residential to low, industrial to high and saves to "My growified city.sc4"
sc4 --no-interactive growify "My city.sc4" -r Low -i High -g -o "My growified city.sc4"

Now, I did not test all the possibilities here, so there might be some bugs here and there, but just let me know and I'll fix them. I desperately need to start writing more unit tests.

[Cyclone Boom 16,875]

33 minutes ago, CorinaMarie said:

Tho I'm fairly certain that was someone else's mod called something like Work at Home Jobs or such.

I remember this topic posted by @Terring which mentions a teleworking technique of having Sims work from home:

Linked there is a mod named "RCImulti" which contains functionality to suppress residential caps and does away with the need for commuting jobs. Perhaps if there might be a way to edit a saved city to add this ability into all existing residential. Also thinking about a shared usage of zoning, and that might be quite fascinating if possible to implement. How it'd allow any given developer type to grow given suitable conditions, and the desirability being a key for what can grow where.

[CorinaMarie 29,605]

19 minutes ago, smf_16 said:

I guess you know how to update by now

Well, I sure thought I did, but it cursed at me.

[CorinaMarie 29,605]

Disregard! I forgot the -g in there. Got it updated now.

[smf_16 1,835]

Seems like you forgot to add the -g flag. This is actually important because as such you update the globally installed module. Now you've installed the module locally, but the globally installed module (which provides the sc4 command here) wasn't updated. Go have a look, a folder will have been created C:\node_modules. You can delete it. If you run it again with the -g  (or --global, that's the same) it will work.

[mattb325 28,880]

Always such amazing progress in this thread. It is very cool!

1 hour ago, Cyclone Boom said:

Also thinking about a shared usage of zoning, and that might be quite fascinating if possible to implement.

That would be an incredible achievement!

Years ago in the BSC we did testing with the multizones, and we were able to get both R&C exemplar types in the lot and it wouldn't crash when added in game, but it didn't function correctly at all (the lot wasn't able to query correctly, nor would it alter the jobs/population stats of the city and of course, it would abandon), but together with the unimplemented zones and this, it may well be possible now to achieve something unimaginable just a few weeks ago.

[Cyclone Boom 16,875]

Super job with the continued updates, and I also really like the idea to add back the command line options for finer tuning control.

It's really awesome to see so much interest in these discoveries already from everyone who's got involved thus far. It bodes very well for the future.

Looks like I also made a mistake with the command, and instinctively I typed in @update instead of the required @latest to get the new global module package. It was just how the thought with needing to update something, and that would be the instruction to use, whereas a program only understands specific commands it expects.

Here was my own completely self-inflicted "oops" moment:

@smf_16

If you don't mind, I've added the command to update latest in an info banner atop the thread, just to spread even greater awareness. I switched ownership of it to yourself, though I don't believe you'll have the ability to edit it since it's a moderator tool. If you would like it updating to amend or add extra info though, just let us know.

[Handyman 765]

4 hours ago, CorinaMarie said:

Tho I'm fairly certain that was someone else's mod called something like Work at Home Jobs or such.

The mod is  RCImulti - MODpack for SimCity 4 Deluxe / RushHour by Nilo
It contains several useful features when you build cities without roads. The one you want to look at is - Homejobs*

EDIt : Just noticed this was already mentioned.

[Wiimeiser 444]

How does the game react to having zone type 16+ in a city? Does it simply crash on startup?

Spaceport could be arcology, perhaps. I've been wanting remakes of the SC2K arcologies, if only as eyecandy landmarks for like a CJ or something.

[CorinaMarie 29,605]

1 minute ago, Wiimeiser said:

How does the game react to having zone type 16+ in a city? Does it simply crash on startup?

The game didn't really care about it at all.

After bulldozing my plopped 4 x 4 lot it wouldn't allow another to be placed saying it was occupied tho no zoning color showed. Dezoning and it charged me $16, but still would not allow a plop on it. The only way to get rid of it was to zone over it and then dezone that.

[carlosmarcelo 975]

On 23/06/2019 at 1:15 PM, smf_16 said:

I've added support for plopped industrials & plopped agriculturals in the growify command.

Unfortunatelly, didn't work for me...  The industrial growified receipts workers, but doesn't have trucks going out theese lots.

A question: is it normal the new file is smaller of original file?  

I did multiples tests, all unsuccessful. In this image, I show you 2 times and the version I used.

No trucks around industrials growified. (here is the only exit of city)

I thought strange industrials growified have same color of the residencials growified. 

After bulldoze and re-zone with industrial high density, the trucks flow started.

Finally, I did a simple case test: a empty small city with only 2 plop industrials (sc4 native builds) and only exit of city. I had the same problem.

  Edited June 25, 2019 by carlosmarcelo  
Removed attach zip with sc4 files to case test because the bug already was fixed in 0.0.13-alpha

[smf_16 1,835]

8 hours ago, Cyclone Boom said:

If you don't mind, I've added the command to update latest in an info banner atop the thread, just to spread even greater awareness. I switched ownership of it to yourself, though I don't believe you'll have the ability to edit it since it's a moderator tool. If you would like it updating to amend or add extra info though, just let us know.

Great, thanks! As you suspected I am not able to edit it, but I'll know where to find you guys. The content of the banner is fine for me though.

6 hours ago, CorinaMarie said:

The game didn't really care about it at all.

After bulldozing my plopped 4 x 4 lot it wouldn't allow another to be placed saying it was occupied tho no zoning color showed. Dezoning and it charged me $16, but still would not allow a plop on it. The only way to get rid of it was to zone over it and then dezone that.

That's something that I'm really happy about: the game is a lot more tolerant than I thought it would be. Definitely helps with experimenting instead of having CTD's all the time!

I think this "unable to plop over"-thing is related to the reserved tiles. Still haven't found out how this works though. I did experiment a tiny bit with it yesterday and it doesn't seem that there's still a building in the way (the building subfile showed the correct amount of buildings in the city after bulldozing). I noticed a difference in the SGProps between a naturally grown lot and a growified plopped lot. The growified had only 6 SGProps, while the naturally grown had 8. The code doesn't allow me to play with SGProps yet though, so I couldn't investigate this further. But I still think this issue has to be resolved before we can actually consider the growify'ing as stable.

2 hours ago, carlosmarcelo said:

A question: is it normal the new file is smaller of original file?  

Yes this is normal. This is related to the fact that a DBPF file may contain holes in it. These holes are just junk data and hence I simply ignore them when I re-save the dbpf file. I noticed that Reader has the same behavior as well.

2 hours ago, carlosmarcelo said:

I thought strange industrials growified have same color of the residencials growified. 

Sounds like I messed up the zone types somewhere then.  I'll have a look at it tonight and will try to fix it. It's pretty sure related to an error in the cli layer around the core functionality because when I did my initial tests I modified the files programmatically (see this line of code) and this did actually work. Like I said, it's about time I start writing some unit tests for the cli layer as well because the commands are becoming more and more complex and this can quickly get out of hand. Yesterday I simply was eager to actually play the game again. Haven't done this for like in a week. 

[vortext 573]

This is very exciting, opens a whole new avenue of modding to explore!  

With regard to mixed use, perhaps the following property is also worth looking into. 

Allow Joint Occupancy 0x2781286 Bool if true different types can occupy the building simultanously

note it seems to apply to buildings, not lots themselves, so I guess a building examplar should define multiple occupant types and have this property be true. Whether or not the simulation actually does anything with this.. anybody's guess atm but given the success of converting ploppable res into growables worth a shot. 

[smf_16 1,835]

Hello everyone,

Turns out the 0.0.12-alpha I published yesterday was badly broken (as @carlosmarcelo already pointed out). I've published a new version on npm (0.0.13-alpha). It has undergone some major refactoring were I extracted the CLI part from the actual API and I added unit tests for both parts. This should significantly reduce the probability of breaking existing functionality when updates are released.

As you all probably have guessed, to update run

npm install -g sc4@latest

 

- Seba

[jdenm8 362]

Some very interesting developments in here!

I went ahead and added Speeder's C++ class names to the wiki, they can all be found in the Savegame article. It's cleared up quite a bit of mystery, though, interestingly, it isn't comprehensive.

I'd be interested to see if anything could be done about Landfills. They're pretty broken in the base game and removing one that's been expanded after the fact can be impossible.

Also, TID c97f987c controls lot base textures, a9c05c85 controls Flora (trees, bushes, etc), a9bd882d controls buildings and 2977aa47 controls props. I think the positions of each are stored in those.

Just deleting 2977aa47 or a9c05c85 from the savefile forces the game to regenerate all of the Props or Flora from scratch (was an early Prop Pox fix). That should make the props appear in the correct location after moving a lot. The building would liikely have to be found and moved manually, there should be a reference to it in the Lot Subfile.

[Wiimeiser 444]

So wait, I've heard that landfills are broken, but how exactly do they work? I know they can't be bulldozed (that would be too easy) and they probably prevent terrain manipulation too close. How exactly does the spreading break, I wonder?

[carlosmarcelo 975]

8 hours ago, smf_16 said:

I've published a new version on npm (0.0.13-alpha)

Thanks, @smf_16!!! Now it works fine, perfect. It is wonderful! My dream was to assemble a city, building to building, now it is possible!

8 hours ago, smf_16 said:

npm install -g sc4@latest

A curiosity question: this command do a automatic download of latest version? how it's knows where get new version?

 

[jdenm8 362]

Landfills are just weird. They also have some special properties that mean they completely lock the terrain under them.

My understanding of how they work: All the tiles of a Landfill zone are treated as one building. The visual area indicates the usage of the storage capacity and how much the zone costs. This area expands (starting from the North-West corner IIRC) every (IIRC) 100t. A 3x3 Landfill zone uses 4 tiles at 0t, 6 at 400t, 8 at 600t and 9 from 800t, which allows the Max capacity for a 3x3 (900t). The area shrinks (starting from the most recently occupied tile) at the same intervals. There's some rules that change how they appear based on the configuration of the buildings in the Landfill Zone. SC4 is *supposed* to be able to make 1x1 landfills (IIRC the renders and everything are there), but can't for some reason and starts with at 2x2 instead.

The issue starts when you expand an existing landfill zone, or if it isn't rectangular. This causes the game to mess it up and make some of the landfill tiles never despawn. Usually it's an entire line that won't disappear. As far as I'm aware, there's no way to fix it. I'm curious to see if messing with either of the Landfill subfiles can shed some light on the problem.

[smf_16 1,835]

6 hours ago, jdenm8 said:

I went ahead and added Speeder's C++ class names to the wiki, they can all be found in the Savegame article. It's cleared up quite a bit of mystery, though, interestingly, it isn't comprehensive.

Thanks! I think that the checksum algorithm should probably be added here as well. The code is found here (credits go to @simmaster07). It should probably be added that if there's an entry that looks like

0  SIZE
4  CRC
8+ [REST]

(which for all I know is the general structure of all checksum protected files) that it's the [REST] part that has to be ran through the checksum function.

1 hour ago, jdenm8 said:

As far as I'm aware, there's no way to fix it. I'm curious to see if messing with either of the Landfill subfiles can shed some light on the problem.

I think we could try to simply delete them and see what happens. I think this should even be possible in the reader as deleting files wouldn't involve recalculating checksums.

5 hours ago, carlosmarcelo said:

A curiosity question: this command do a automatic download of latest version? how it's knows where get new version?

It knows that because the module is published on the npm package registry. When you npm install a package, it looks for the given package in the registry and then downloads it from there. It also takes into account the version, so if you're feeling nostalgic you could npm install sc4@0.0.1-alpha as well. The @latest here is simply a way to tell it to get the latest version.

Npm is actually one of the main reasons why node.js has become so popular. It's super easy to install libraries you need for your program and there are so much packages on it (1,016,010 when I last checked) that there's a good chance that there's already a package for what you need. This greatly improves developer efficiency as you no longer need to re-invent the wheel all the time. For example, in the sc4 program I used the commander module for the cli-stuff and inquirer for the interactive stuff so I did not have to write these things myself.

[simmaster07 1,220]

5 hours ago, jdenm8 said:

I went ahead and added Speeder's C++ class names to the wiki, they can all be found in the Savegame article. It's cleared up quite a bit of mystery, though, interestingly, it isn't comprehensive.

Nice! I went ahead and added most of the missing class names. There are a couple I can't figure out yet since they go through a few levels of indirection, but the weirdest case is TID 0x2b689130 ("Deals") which isn't directly referenced anywhere in the code.

[jdenm8 362]

I think the Deals one might be related to Business Deals. I can't chase up where exactly it came from, the trail runs cold at the transfer from WakkaWikki to MediaWiki back in 2005. It does mention something about a Savegame XML file though. I might see if I can find it, though it was probably lost on the old MTS2 Forum.

Also, if you do make any changes to the wiki, let me know. I'm working on cloning the Wiki into a newer version of MediaWiki (Making a WikiMarkup-format backup in the process) for when the new hosting is sorted out instead of transferring that outdated version from 2008.

[CorinaMarie 29,605]

6 hours ago, jdenm8 said:

The issue starts when you expand an existing landfill zone, or if it isn't rectangular. This causes the game to mess it up and make some of the landfill tiles never despawn. Usually it's an entire line that won't disappear. As far as I'm aware, there's no way to fix it.

In some testing I did, I found they can be irregular shape providing that there is always 2x2 twists and turns so it has a place for the 4 rounded corner piles of garbage. But even then they are tricky to work with and it's easy for the landfill to bork itself. As far as recovering from a messed up landfill, the Landfill Eraser Mod comes in handy. As stated there by @cogeo: " The mod is very simple in its concept, it just sets the garbage decay rate to 100%. " and I've used it to empty out the goofiest of landfills and after that the landfill dezone tool will remove the now empty zones.

 

4 hours ago, smf_16 said:

Npm is actually one of the main reasons why node.js has become so popular.

Seems like it's a similar concept to the aptget tool of Linux.

 

3 hours ago, jdenm8 said:

Also, if you do make any changes to the wiki, let me know.

Maybe we should have a dedicated topic for this somewhere? I've edited several minor things on the Wiki, but nothing in the past few months.

[CorinaMarie 29,605]

This could also go in your other topic: Programmatically generating a city: A savegame experiment, but I feel it fits here just a wee bit better.

I exported the Cute House from the Chicago tileset and I've made a separate Growable and Ploppable version of it with new IIDs so they won't conflict with Maxis content. (I removed all the props to hopefully reduce the amount of data per lot.) The growable needs at least the Chicago tileset enabled or all tilesets. The ploppable is found in the Rewards menu since it's one that usually isn't crowded. (The menu icon was created by @Cyclone Boom.) I've made them 1 x 1 in size at growth stage 1 so they can sprout as soon as a city tile is created. With no other plugin content that house is the only one which can grow on a 1 x 1 low density residential in a newly started city.

Also included with the two lots in the attached zip file is a small city tile which can be imported into your region. (Or just make you own with some jobs available.) Here's what my test city looks like with one grown and one plopped:

 

What I'm hoping you can write in your code is whatever it takes to output the data from each lot in a separate text file with the info for each variable associated with the lot on its own line so that they can be compared using WinMerge.

Here's a sample I mocked up cropping and pasting from SGE:

 

I'm guessing that when you convert the underlying zoning the reason they will not un-abandon if the city's economy went sour and was then fixed is that it doesn't have all the rest of that data. So, if we can figure out what all that other stuff means then it might be that in addition to changing the zoning you could have it write in all the other missing stuff and it might make converted ploppables even more viable for use. As it is with the zoning switch only, I believe it's the same as someone re-discovers every year or so that plopping a residential on top of an existing grown one will work, but once abandoned (as happens from time to time in an ever changing city) and then it won't repopulate itself.

Attachment: Simple Lots - Two 1 x 1 Residentials.zip

Edit: The attachment above is still there, but I have fixed the Ploppable lot so it cannot grow on its own.

So, download this new version instead: Two Simple 1 x 1 Residential Lots v2.zip

[smf_16 1,835]

Thanks Cori! I will definitely be able to use that one in my "programmatically generating a city" project.

As for the textual output, this one should be rather easy to implement, especially since I've rewritten my Lot Subfile parser. I'll give it a shot tonight.

I want to come back on something else as well. While re-writing my Lot Subfile parser, I did discover that the total capacities

FLOAT32	Total § Capacity  (for C, I and Civics, 0x00000000 for R) - see also SGPROP 0x8A0B4100
FLOAT32	Total §§ Capacity  (for C, I and Civics, 0x00000000 for R) - see also SGPROP 0x8A0B4100
FLOAT32	Total §§§ Capacity  (for C, I and Civics, 0x00000000 for R) - see also SGPROP 0x8A0B4100

are indeed UInt32's instead of floats (see this line of code). I think the wiki should be updated, and perhaps we can also add some indentation to make things a bit clearer.

EDIT: Woops, disregard the above. What I actually meant is that I was able to confirm that these values are actually 4-byte values (because I used to think they were UInt16s, which are two bytes). Inspecting the values though it does make more sense that they are floats, while that is still a bit weird though. Nevertheless if you'd interpret the numbers as UInt32's, the values are insanely high, while they make more sense when being read as Float32's.

I propose something like this:

Spoiler

DWORD	Size
DWORD	CRC
DWORD	Memory
WORD	Major Version?  (only seen 0x0008)
DWORD	Lot IID
BYTE	Flag byte 1  (See Appendix 1 below)
BYTE	Min X Tile
BYTE	Min Z Tile
BYTE	Max X Tile
BYTE	Max Z Tile
BYTE	Commute X Tile
BYTE	Commute Z Tile
FLOAT32	Y Position
FLOAT32	Y Coordinate if slope conforming, height above Y Position (0x00000000 if Lot Foundation)
FLOAT32	Y Coordinate if slope conforming, height above Y Position (0x00000000 if Lot Foundation)
BYTE	Lot Width
BYTE	Lot Depth
BYTE	Lot Orientation
BYTE	Flag byte 2  (See Appendix 2 below)
BYTE	Flag byte 3  (See Appendix 3 below)
BYTE	Zone Type
BYTE	Zone Wealth
DWORD	Date Lot Appeared (was plopped or grew)
DWORD	Building IID
BYTE	Unknown, several values seen, but the same value often appears in several lots
DWORD	Memory Address for linked industrial lot
    DWORD	0x4A232DA8, Industry Subfile Type ID. Appears only if the previous DWORD is not 0x00000000
DWORD	Memory Address of the Anchor or Farm Lot
    DWORD	0xC9BD5D4A, Lot Subfile Type ID. Appears only if the previous DWORD is not 0x00000000
BYTE	Count  (0x00 or 0x01)
    BYTE	Count of RCI Types  (0x01, 0x02 or 0x03)
        DWORD	Demand Source Index  (See Appendix 4 below)
        WORD	Capacity
BYTE	Count of RCI Types  (0x00, 0x01, 0x02 or 0x03)
    DWORD	Demand Source Index  (See Appendix 4 below)
    WORD	Total Capacity  (including filler lots, e.g. farm fields, mechanics and outs)
FLOAT32	Total § Capacity  (for C, I and Civics, 0x00000000 for R) - see also SGPROP 0x8A0B4100
FLOAT32	Total §§ Capacity  (for C, I and Civics, 0x00000000 for R) - see also SGPROP 0x8A0B4100
FLOAT32	Total §§§ Capacity  (for C, I and Civics, 0x00000000 for R) - see also SGPROP 0x8A0B4100
WORD	0x0002  (Unknown)
DWORD	Count of Properties  (I've seen up to 0x00000007)
    SGPROP	SaveGame Properties, repeated Count times  (See Appendix 5 below)
DWORD	Count of Commute Blocks  (number of destinations commuted to from here)
    DWORD	Count of Commute Paths  (usually 0x00000002 = morning and evening)
        DWORD	Count of Bytes in the Path
        BYTES   Commute Path  (See Appendix 6 below)
    DWORD	Number of Commuters
    BYTE	Unknown, 0x00 or 0x03
    BYTE	Unknown, 0x00, 0x01 or 0x02
    BYTE	Unknown, 0x08, 0x09, 0x0A or 0x0B  (0x00 if the commute path has disappeared)
    BYTE	Unknown, 0x00 or 0x03  (0x04 if the commute path has disappeared)
    SINT16	Commute Destination, X Tile  (See Appendix 7 below)
    SINT16	Commute Destination, Z Tile  (See Appendix 7 below)
    FLOAT32	Trip Length
    DWORD	Unknown  (See Appendix 8 below)
BYTE	Unknown  (takes many different values)

 

Allthough I'm not entirely sure yet of the structure of the commute blocks, as I haven't been able to parse this one correctly yet, but that's more out of laziness and not really needing the commute paths for now.

EDIT: I'm unable to reply to this topic anymore. Is this normal?

  Edited June 26, 2019 by smf_16  
Error, seems that the values are floats after all

[CorinaMarie 29,605]

3 hours ago, smf_16 said:

I think the wiki should be updated, and perhaps we can also add some indentation to make things a bit clearer. I propose something like this:

Those three which are listed as Floats are inconsistent with all other references to them and your testing confirms they should be Uint32. Since @jdenm8 is currently working on Wiki stuff, I'll leave the editing to them.

This is additional, useful information you are discovering.