discovery Modifying .sc4 savegames: It IS possible!

[rsc204 21,808]

6 hours ago, smf_16 said:

EDIT: I'm unable to reply to this topic anymore. Is this normal?

If by this you mean the textbox to do so fails to appear, that's a forum software bug that never seems to go away. Usually resolved by pressing F5 (refresh page).

BTW: clearly what's being done here in respect of opening up the savefiles of SC4 is exciting new ground. I'm trying to follow everything, but I have very limited time right now to get into such a complex project. 

However, one thing that comes to mind, you may want to read up on this thread at SC4D:

Ploppables with jobs and residents

Whilst we can plop commercial and industrial buildings already, there are certain caveats that should be observed. It seems to me, the same pitfalls would apply to either Ploppable Residential or generated buildings.

The key problem here is that unless sufficient demand exists prior to said buildings existence, these seemingly working lots may actually prove to be very broken. At the least, it might be advisable to follow the same testing procedure RippleJet used, to see if such lots are really working as it first seems.

[jdenm8 362]

The jobTotalCapacities property, and the lack of it on the Plopped version, sticks out to me. I wonder if maybe Maxis had used it interchangeably with jobCapacities in some contexts.

The fact they're different makes me wonder if things on the lot other than buildings (like maybe a Prop) were supposed to be able to add Job Capacity at one point. Or maybe that's how Machinery, Outbuilding and Field lots add routes to their parent.

I'll look at adding that to the Wiki tomorrow night. Anyone else is welcome to do it before then though, I'll check the edit log before I do it.

[smf_16 1,835]

I've published a new version on npm (0.0.14-alpha) that now includes a "dump" command. Usage is as easy as

sc4 dump "your city.sc4"

It will log the contents of each lot (except for the commute paths) to the console. You can write this away to a file by using

sc4 dump "your city.sc4" > output.txt

That should get you started. I assume that it would be useful as well to be able to modify values and see what effect they have in the game. I am still planning on writing a bit of documentation about how this could be achieved - haven't got the time for the moment as I'm still doing a lot of research - but if you're feeling adventurous you can have a look at my test files in the test/ folder. The tests are pretty much self-documenting on how to use the different classes.

18 hours ago, CorinaMarie said:

As it is with the zoning switch only, I believe it's the same as someone re-discovers every year or so that plopping a residential on top of an existing grown one will work

I have actually never understood this. I had never been able to get the plopped residentials to work, even when plopping them over an existing building. Sure, the building didn't always abandon right away, but the people were never actually able to find a way out of it. I wonder if those reports that plopped residentials worked when plopping them on top of existing buildings actually included that people were able to find a way out. If this is the case, perhaps this indicates that the game would sometimes randomly decide to not remove the old lot and create a new one, but actually just "swap" the buildings on the lot.

 

3 hours ago, rsc204 said:

However, one thing that comes to mind, you may want to read up on this thread at SC4D:

Ploppables with jobs and residents

Whilst we can plop commercial and industrial buildings already, there are certain caveats that should be observed. It seems to me, the same pitfalls would apply to either Ploppable Residential or generated buildings.

The key problem here is that unless sufficient demand exists prior to said buildings existence, these seemingly working lots may actually prove to be very broken. At the least, it might be advisable to follow the same testing procedure RippleJet used, to see if such lots are really working as it first seems.

I do agree that plopped residentials probably suffer from the same problems as commercials if there isn't sufficient demand for them. However, the main problem with plopped residentials has always been that the pathfinder was unable to find a way out. The experiments here have shown that by switching the zoneType of the lot entry this can be resolved. Whether or not this has any other side effects is still an open question - although I did have quite some success continuing to play with already developed cities - but I do agree that this definitely requires some more testing.

[jaredh 195]

wow.  Just wow.

I wondered when the Pox problem got fixed it might lead to savegame editing.  Sure enough, here it is.  

This SCREAMS for a savegame editor where you can click on a building and run whatever update you want (historical, convert to growable, etc etc) for fine-tuning down to a specific lot.

But big kudos @smf_16.  The whole plopped res abandonment issue is really historically the biggest issue with SC4 (Pox included).  Its glorious this is finally possibly fixed.

bravo.  

[smf_16 1,835]

Fyi, I've published a new verision on npm (0.0.15-alpha) which includes a new command called refs. It basically finds out which files are related in a savegame. I explain it a bit more in depth in my other thread about programmatically generating a city. I think that the command may be useful when decoding some yet undecoded files of the savegame. Specifically it indicates what file could be causing something work not to work if we changed something in another file and hence it gives a good suggestion of what file should be next to be decoded.

23 hours ago, jaredh said:

This SCREAMS for a savegame editor where you can click on a building and run whatever update you want (historical, convert to growable, etc etc) for fine-tuning down to a specific lot.

Well that is certainly possible and I think that we will eventually get there. However, there is still a lot of work to do, notably we will need to decode pretty much all of the subfiles in the savegame if we want to understand how it works internally. Hence I think the keyword will be: patience and hard work, and I mean like, a lot of it. 

[smf_16 1,835]

Hello everyone,

Perhaps that some of you did already find out, but I thought it was useful to mention that it's possible to run the growify and historical commands even while SC4 is running. You only have to exit to region. This saves you some time because you don't have to exit the game, run the commands you want and then start it up again. Simply exit to region, run the commands you want and then open up the city again. Saves some time, especially when you're experimenting a lot!

[smf_16 1,835]

Hello everyone,

I assume that most of you are following my other topic where I try to programmatically generate a city as well. While I did have some success with it already, it is more meant as a personal learning experience about how savegames work, and the experiment will probably not be very useful for real cities.

However, while working on it I had the idea to add something to the command line tool that might be rather useful for "real life" situations: allowing cities to switch between seasonal or evergreen flora and being able to synchronize all flora in the city - meaning that flora that wasn't all planted on September 1st will no longer be out-of-sync.

Personally I don't really like to play with seasonal flora for several reasons, mainly that I don't want to wait for September 1st everytime to start planting flora and I just like the green look of summer trees - I'm kind of a summer person. However, I have to admit that autumn colors are simply gorgeous. If I would ever start a CJ, it just has to include autumn shots. I personally really like the approach taken by the Central European Tree Controller: it gives you the possibility to use a static, non-seasonal tree controller where you can simply swap files if you want a different appearance of your trees. That's perfect for me: I get to plant evergreen trees and whenever I want a nice autumn shot, I just swap the "Evergreen" file by the "Fall" file et voila.

While I can imagine that this approach could be used for MMP Flora as well (by mapping the Instance ID's of the evergreen trees to the fall versions) I decided to give it a shot with modifying the Savegame. It wasn't too hard: I created a script that scanned my plugins folder for all of Girafe's trees (and the excellent Fagus created by Orange, my personal favorite) and mapped all Instance IDs of the evergreen flora exemplars to their corresponding seasonal Instance IDs. Then I scanned my city's Flora Subfile and replaced the IID's with their respective counterparts. The results were as expected:

As you can see, I changed all seasonal flora to their evergreen counterparts. Note that that's the easy part though. Apart from changing the IIDs, going from Evergreen to Seasonal also requires me to synchronize the flora's cycle correctly so that it appears as if all of them were planted on September 1st and they match with the date. It's here that I'm still struggling to get things right.

I created a new city with some out-of-sync seasonal flora and inspected it with my tool. It reported that all "date of appearance" were equal to the "cycle date" fields, so I figured that it should work if I just set both to September 1st 2000. SC4 uses Julian dates for this apparently, so September 1st 2000 would be 2451788 according to this site.

I tried it out on the city you see above, but when I re-opened the city everything just looked the same and when I unpaused the simulation, the summer and winter trees simply disappear after some time. Clearly there's something I'm still missing here, but I'm not really familiar with the inner workings of seasonal flora in the game. Is there somebody with some suggestions here? Thanks!

 

- Seba

[catty-cb 4,392]

We use julian dating at work and the 1st of sep 2000 would be either be

00244 (or 245 if it's a leap year); or

2000244

[smf_16 1,835]

Are you of this? When I look at the initial values of the appearance dates, they are all close to 2451809 which would be September 21st 2000 accodring to the same site, which makes sense because I've planted some trees around this date in the game.

[catty-cb 4,392]

Pretty certain, it depends on whether Maxis was using Julian Dates

Quote

A Julian date is sometimes used to refer to a date format that is a combination of the current year and the number of days since the beginning of the year. For example, January 1, 2007 is represented as 2007001 and December 31, 2007 is represented as 2007365

or the Julian Calendar in which case your website would be correct

[smf_16 1,835]

Then I think it’s the Julian Calendar. I’ve read somewhere that the Julian Calendar should not be confused with Julian Dates, but looks like I just did here 

Anyway, the thing that surprises me is that the “appearance date” always seems to be equal to “cycle date”. I wonder then how the game determines what state the flora is in.  

[catty-cb 4,392]

Only two pages in the Prima Guide cover trees am attaching them to this post

[carlosmarcelo 975]

1 hour ago, catty-cb said:

Prima Guide

How I can get this guide? 

[CorinaMarie 29,605]

4 hours ago, carlosmarcelo said:

How I can get this guide? 

Lookey here or here.

[Cyclone Boom 16,875]

5 hours ago, carlosmarcelo said:

How I can get this guide? 

The SC4 Prima Strategy Guide was originally published as a hard copy book (I remember getting that from a retail store shortly after the game released). From some places it might be available to pick this up nowadays 2nd hand, such as from Amazon. Additionally now it's available in digital form on the Internet Archive site. The same also for SC3K too, and it's possible to view them from there directly, or download a PDF local copy.

As follows:

• SimCity 4 Rush Hour Prima Official eGuide
• SimCity 3000 Prima Official eGuide

(See the "Download Options" on the right side for a range of formats.)

Edit: 2 answers for the price of one!
See the links @CorinaMarie posted above for the SC4 guide digitally.

  Edited July 7, 2019 by Cyclone Boom  

[vortext 573]

Seasonal versus evergreen flora examplar files differ in which Resource Key Type (RKT) it uses. Simply put evergreen flora reference a single model by means of RKT1, while seasonal flora examplars references 3 models by means of RKT4. The timing itself is controlled by the Flora Parameter properties in the examplar file. Note this property is always present but obviously isn't used when the flora examplar file contains RKT1. 

Before continuing it's important to realize the notion of September 1st, or any other date for that matter, is by convention. Or to put that another way; the timing mechanism relies on counting the number of days passed since planting date (in a roundabout way), and progressing through the models referenced by the RKT4 untill the entire cycle resets.

Now for the technicalities, this is going from memory but iirc it works as follows. First of all the order in which the RKT4 models are shown is determined by the first rep for their entry. (in bold below, example is Grfe_norway_maple_seasonal_A)

0x00000000,0x00000000,0x00000000,0x00000000,0x27812821,0x5AD0E817,0x9C03BDEF,0x00030000 - 1st

0x00000001,0x00000000,0x00000000,0x00000000,0x27812821,0x5AD0E817,0xBC03B8A4,0x00030000 - 2nd

0x00000002,0x00000000,0x00000000,0x00000000,0x27812821,0x5AD0E817,0xBC05058A,0x00030000 - 3rd

This is important because the first rep in the Flora Parameters property determines the duration of appearance (in days) for the last RKT4 entry. The overall duration is determined by the 6th rep in the Flora Parameters. The duration for the other 2 RKT4  entries are determined by subtracting said value from the overall cycle duration and dividing by 2. 

So for seasonal flora the 1st rep in the Flora Parameters typically is 180,  the 6th rep is 363. So this means the last RKT4 model will show for 180 days, leaving (363-180) / 2 = 91.5 days for the other 2 entries. Now obviously the game doesn't do halve days but the flora update interval isn't that exact to begin with so that's okay.

Now a slightly different example; the vanilla noble fir mmp. While technically not 'seasonal' it uses the exact same timing mechism to simulate growth over time. The 1st Flora Parameter rep has a value of 1200, the 6th a value of 6570. So this tells us the overal growth cycle will last for 6570/365 = 18 years. And because this uses RKT5 the first 2 models will actually last 1200/365 = ~3.25 years each, meaning the last full-grown model will be on display for a little under 12 years. 

So all this to say; the dates involved don't really matter and are a semi-arbitrary convention, which was arrived at by lots of experimenting when seasonal flora was developed. The game instead seems to keep track of 'days passed since planting' for each flora item, and uses that day count, in combination with the Flora Parameter property, to determine which model from the RKT4 it should display. 

[smf_16 1,835]

@vortext Thank you, that is a very comprehensive explanation and clarifies a lot for me. I already suspected that the game would use the "days since planted" somehow. It was a bit weird however how my experiment with changing the appearance & cycle date caused some flora to disappear. Perhaps I should try to change the appearance date only. Do you have any idea what could be the difference between this appearance & cycle date? In my test city - which was only very young - I found that both had the same values, but with another more established city I found that 78% were different. Perhaps that the cycle date resets every year? I'm definitely going to experiment with this a bit more.

Also thank you for clarifying that RKT 4 format. I currently have the idea to make a mod that would replace all of Girafe's Evergreen flora by "Evergreen Fall" or "Evergreen Winter" flora, but in a static way such as the CET does. As such you could drop a file in your plugins and have it load after Girafe's flora to change the appearance of the evergreens that have been planted. I'm planning on writing a script that automates this task, but I have to extract the Spring, Summer, Fall & Winter IIDs for this from the RKT 4, which I wasn't able to because I didn't fully understand the format. Now it should be easy with your explanation on the format.

[smf_16 1,835]

Allright, I had a small success with synchronizing the flora. I played a bit with the appearance and cycle date. I set the appearance date to September 1st the year before the appearance date that was already set and then set it as cycle date as well. This caused the flora to be synced as you can see below - however only after the start of the next month - but interestingly it seemed to be completely agnostic of actual date I set. The change between Summer and Autumn happened at November 1st of each year, regardless of what date I set the appearance and cycle date to.

Pretty interesting stuff. I guess that my disappearing flora from my earlier experiment hence was related to the fact that I accidentally set the appearance date to a date in the future. Still it completely baffles me that the game doesn't care about what the exact values of the appearance and cycle date were. Guess I'll have to do some more experiments here!

 

- Seba

 

[Eggman121 1,035]

Hello @smf_16

I have been following this thread with quite a keen interest. I am curious of how your work could help users that install the latest NAM when they eventuate to update the standards to new content?

On main player here is the Real Railway component that uses find that if they have quite an established city that quite a bit of work needs to be done to update the rails. My question is is that is it possible to replace network tiles base on its IID allocation? One example is the Maxis Rail IID is 0x03031500 to 0x03031504 (Orthogonal Piece) and the RRW IID is 0x5d540000 to 0x5d540004 for the same tile. Any information about if this would be possible would be greatly appreciated.

You have certainly made a great discovery here and have opened the door to many new possibilities.

-eggman121 (NAM Team Member)

[smf_16 1,835]

Hello @Eggman121,

I think that this is relatively easy to do. I've found that changing single values within a Savegame is easy. It's only when you start adding or removing stuff that things get a bit more challenging because in that case you have to figure out how the different subfiles in a Savegame are related. I don't know if you follow my Programmatically Generating a City thread, but it nicely illustrates the difficulties encountered when adding a lot somewhere.

I've carried out a search for 0x03031500 in an old city of mine of which I know it still has Maxis Rail in it. It found that this value only occurred in the Network Subfile 1. I haven't written a parser for this yet, but the file seems to be fully decoded so I should be able to do that. I'll do some experiments tonight if you want. It would help me if you could give me a city that contains some Maxis rails in the range 0x03031500 - 0x03031504. I don't have a lot of these cities left myself. 

- Seba

[Eggman121 1,035]

Thanks for the reply @smf_16

I have made a basic city using just maxis pieces to see how well a stock city goes with programmatically upgrading it to NAM 36 (Particularity the Real Railway). Which is the best way to send the city too you?

Either way I will probably send it tomorrow by a method preferred by you.

-eggman121

[smf_16 1,835]

Well if you don't mind others being able to use it as well, adding it to your post is fine for me.

[CorinaMarie 29,605]

1 hour ago, Eggman121 said:

Which is the best way to send the city too you?

1 hour ago, smf_16 said:

adding it to your post is fine for me.

If it turns out to be too large for an attachment, I recommend WeTransfer.com. (Use the Take me to Free option.) This latter method has the advantage of not using up one's attachment quota here on ST and it allows files up to 2 GB. The drawback to WeT is the linky is temporary and the file is removed from their server after 7 days.

[vortext 573]

On 10/07/2019 at 12:47 PM, smf_16 said:

Do you have any idea what could be the difference between this appearance & cycle date? In my test city - which was only very young - I found that both had the same values, but with another more established city I found that 78% were different.

Neither term rings a bell unfortuantly. However I noticed it seems like you're using one of CP's terrain mods, which are not suitable for seasonal flora out-of-the-box, and require installation of the Seasonal Flora Patch. Do you have this installed in this test city, or otherwise adjusted the Flora Tuning Parameters?

Thing is this timing mechanism initially was intented to simulate forest growth over time, and consequently Maxis introduced some randomness into the cycling over stages. Otherwise a sappling forest planted at the same time would continue to grow in lockstep which arguably looks kinda silly. However for seasonal flora to work this randomized update pattern is undesirable, and figuring out how to enforce 100% updating of the flora during each cycle actually was the breakthrough which made it possible in the first place.

As you might have guessed this is done in the Flora Tuning Parameters. The discrepency in appearance vs cycle date seems to echo the randomized updating that occurs with vanilla settings, versus patched setting necesarry for seasonal flora to properly work. 

[jeffryfisher 2,176]

When writing a program to synchronize all seasonal flora, it will be important to recognize non-seasonal flora and NOT synchronize tree maturity... or are the two types never mixed due to incompatibility?

And if they haven't been compatible in the past, do we now have enough data to mix them yet age them separately? For that matter, can we circumvent the difference between pre-Mayor trees and post-Mayor trees mentioned in the Prima guide (i.e. enable God-in-Mayor mode trees to mature and propagate)?

[smf_16 1,835]

@Eggman121 I went ahead and already experimented a bit. I renamed my plugin folder so that I was playing vanilla and drew some rail. I then restored my plugins folder with RRW enabled and opened the city - that's the picture on the left.

I then changed all IIDs within the ranges you gave to their counterparts and reopened the city - that's the picture in the middle. I guess that this is the result that you expected: the orthogonal pieces changed correctly. I noticed though that when dragging over the parts that still have maxis textures the game needs some time to process it, but it changes the textures correctly after all. I don't know if that's normal or not.

Finally I tried to offset all IIDs with the difference 0x5d540000 - 0x03031500 which you can see on the right. Based on the picture I assume that the RRW does not simply use a fixed offset for the textures with respect to the Maxis textures. Dragging over the parts - even the hidden ones - did restore everything correctly though, albeit requiring even more time to process it.

If you give me a complete list of Maxis-to-RRW IIDs I can test if it works for non-orthogonal pieces as well if you want.

6 hours ago, vortext said:

Neither term rings a bell unfortuantly. However I noticed it seems like you're using one of CP's terrain mods, which are not suitable for seasonal flora out-of-the-box, and require installation of the Seasonal Flora Patch. Do you have this installed in this test city, or otherwise adjusted the Flora Tuning Parameters?

Yes, I am using CPT's Italia Terrain Mod and I did install the Seasonal Flora Patch. Where exactly is it that I would have to modify Flora Tuning Parameters? I don't really know what they are, so I don't think I changed them in any way.

6 hours ago, vortext said:

The discrepency in appearance vs cycle date seems to echo the randomized updating that occurs with vanilla settings, versus patched setting necesarry for seasonal flora to properly work.

Based on this, my guess is that the appearance date is the date the source tree was planted in the game, while the cycle date is the date the "child" tree appeared. If the tree is still the source tree, then both are equal. This also matches with the fact that the cycle date still comes before the actual city date. If this would be the case, then in theory I have to offset the cycle date with the same amount as I offset the appearance date. Still this probably won't solve the "November 1st" problem because I already tested a file where the appearance date and cycle date were both the same and as I mentioned above it seemed to be agnostic of the appearance date I did set.

1 hour ago, jeffryfisher said:

When writing a program to synchronize all seasonal flora, it will be important to recognize non-seasonal flora and NOT synchronize tree maturity... or are the two types never mixed due to incompatibility?

I'm planning to do this based on a fixed list, meaning that I will only synchronize flora I know are seasonal. I'm using Girafe's flora for now, but if I have the IIDs of other seasonal flora as well, I should be able to synchronize those as too - even if they have another start date than September 1st.

[Ramona Brie 121]

Is it possible to bulldoze individual buildings in a city the game refuses to load with this?

I have a case of a city crashing on loading and I think I know the lot that might be causing the issue.

[Wiimeiser 444]

On 30/07/2019 at 3:09 PM, Ramona Brie said:

Is it possible to bulldoze individual buildings in a city the game refuses to load with this?

I have a case of a city crashing on loading and I think I know the lot that might be causing the issue.

Does it load in that Save Game Explorer thing? If so, please tell/show us which building you think is the culprit, that'll make it easier for us to narrow down.

[smf_16 1,835]

On 30-7-2019 at 7:09 AM, Ramona Brie said:

Is it possible to bulldoze individual buildings in a city the game refuses to load with this?

I have a case of a city crashing on loading and I think I know the lot that might be causing the issue.

I'm not sure. Based on my research I've been doing in my other topic it should be possible to remove a lot from a city by removing it from the Lot Subfile, subsequently delete its building, props, textures and flora from their respective subfiles and finally updating the ItemIndex, LotDeveloper and ZoneDeveloper files. However, this would only physically remove the lot. I am not sure how SimCity handles some of the lot's properties such as jobs, EQ effect, ...

To make this more concrete, I'll give an example of a case I encountered a few years ago. Basically I changed the EQ-vs-age curve using LEProp of a school that I made myself after I already plopped it in my city. I now know that this is a very bad idea, but at that time I didn't know any better. So, based on the CTD's I've been experiencing when trying to demolish that lot, I'm guessing that there's more to it than simply physically removing the lot. I guess one also has to update the city data based on the stats of the lot. Therefore, as @Wiimeiser pointed out, it would be valuable if you could provide some more info about what type of lot you think might be causing the issue

It's funny that you posted this, as I was actually thinking of this a few days ago. I was wondering if I could revive those corrupted cities that I abandonded due to this problem using the knowledge of savegame editing. The first step here will be to test what physically removing the corrupted lots in the city does. If we're lucky, the game is somehow able to handle this, but I wouldn't count on it.

 

- Seba

[Ramona Brie 121]

7 hours ago, smf_16 said:

It's funny that you posted this, as I was actually thinking of this a few days ago. I was wondering if I could revive those corrupted cities that I abandonded due to this problem using the knowledge of savegame editing. The first step here will be to test what physically removing the corrupted lots in the city does. If we're lucky, the game is somehow able to handle this, but I wouldn't count on it.

9 hours ago, Wiimeiser said:

Does it load in that Save Game Explorer thing? If so, please tell/show us which building you think is the culprit, that'll make it easier for us to narrow down.

I had already figured out what the lot was and was able to open the city (I had two culprits, luckily), but I still wanted to raise the idea because it's a potentially practical application of this research.