Do you want a save editor?

[OrSpeeder 72]

I've been trying to understand how the simulators work (so I can mod the rules and balance the game better, and fix the commute time bug).

 

Part of that ended with me studying how save files work.

 

I think I can code a savegame editor, but I will only do it if there is some serious interest, do you people want it?

[rsc204 21,867]

If you could fix so called "Prop Poxed" files, I think this would be seriously useful. I was under the impression though that the save files had a checksum in them. Have you found a way to edit them without messing with this checksum, because otherwise modded saves would never work again?

I can see some use in such an editor, but aside from fixing broken saves I'm not sure what other benefits this might have.

[cmdp123789 710]

What could we do with such thing?

[A Nonny Moose 8,262]

Necessity is the mother of invention.  How many of us have had corrupted saves and wound up having to abandon a city or even a whole region.

On the checksum issue, if you change anything in the file, you'd have to rebuild the checksum.  Dicey at best, but if you have an idea that might work, it would be a coup at this late date.  I am starting to feel that EA's dog in the manger attitude on releasing the source is getting to the point where someone with a good decompiler for C++ Version 6 might dare to tackle it.

And don't kid yourselves, I've worked in a university computing environment and students of computer science create decompilers for fun.

[OrSpeeder 72]

The CRC is seemly only a CRC, in the sense it is used only to check for file integrity, not any serious protection.

Also, all files have the CRC in the header, but the save file is just a normal DBPF file otherwise, I think if you only need to recalculate the CRC for the file you edited, not all of them, and the algorithm is probably the same for all of them anyway.

If you people wanna see what each file in the save file do right now, just open the file in Reader, copy the "type" part of the Resource Key (TGI), and look for it in the list I pasted on SC4Devotion

 

Example: in the wiki, it is stated that the meaning of the type id 6990c1aa is that it is a "Sims to Jobs Match" but if you look for it on my list, you will see this is the class if for the class cSC4TrafficSimulator

The class is serializable, meaning that when saving the game, the engine converts the whole class into data, and paste into your save file. (also, it DOES have "sims to jobs" information in it, but this class controls the whole traffic system, not just that, it is where is stored the commute times, traffic volume, paths, etc...)

[A Nonny Moose 8,262]

Interesting, but not much to do with constructing an editor capable of fixing errors.  I do agree that there is probably a single checksum for the entire file. 

It is difficult to consider data within such a file as being divided into files or sub-files but I suppose that is as good a view as any.  The problem will be determining the boundaries of these tables which are probably variable arrays.  One needs to decode the system used, which may or may not be uniform depending on how many cooks had their fingers in the save broth.

I am still of the opinion that there are some off-by-one errors occasionally cropping up in Maxis work in general.  It is easy to forget that C++ arrays are zero based.  Only programmers think of the zeroth instance of anything.

[OrSpeeder 72]

From what I've seen so far, the save files serialize things based on the IFF format specification from 1985, with some modifications.

Basically all files seem to follow this structure:

 

File size, CRC stuff, more CRC stuff, data.

 

The data part usually having a format that mixes C structs fields, with arrays, arrays have immediately before them the number of elements, arrays can contain objects with variable lengh, when this happen the object lists its own length on a header.

 

I made a script for 010 Editor that understand the history warehouse subfile (I made it aiming to export the data and make pretty graphs... pretty much useless, since most of the graphs are in-game too anyway).

Here is the code.

struct FILE
{
    struct HEADER
    {
        int fileLength;
        int unknown1;
        int unknown2;
        short version;
        int amountOfSets;
    } header;

    struct HISTORY_DATA
    {
        DATASOURCE_ID datasourceId <format=hex>;
        unsigned int graphType <format=hex>;
        unsigned int dataPairAmount;
        struct DATA_PAIR
        {
            unsigned int timestamp;
            unsigned int data;
        } dataPairs[dataPairAmount];
    } historyData[header.amountOfSets] <optimize=false, open=true>;
} file <open=true>;

 

If you are wondering, <open=true> just tells the 010 editor to automatically open the tree (so I don't have to manually open every time I run the script), and <optimize=false> warns 010 editor that the array contains variable-length elements (it by default assume all members are equal, this make it run faster, but it breaks badly on SimCity 4 save files).

[A Nonny Moose 8,262]

Interesting.  What's the definition of DATASOURCE_ID?  Is this a private struct or just a macro?  If the data is "hex" is it in nybbles or justified (left or right) in single byes?  Rather hard to manipulate in C/C++.  Lots of <<, >> II, or && operators?

[OrSpeeder 72]

Oh, DATASOURCE_ID is a enum that I made

I mostly copied it from here: http://www.wiki.sc4devotion.com/index.php?title=DataSourceID (mostly, because some stuff I figured by myself, I plan in updating the wiki later).

 

As for how it reads, I don't figured much yet, but seemly it uses a stream system, that MAYBE (I am not sure) is based on STLPORT streams.

The stream system that reads for example has these functions according to the Aspyr port debug symbols:

 

Skip
GetSint8
GetUint8
GetSint16
GetUint16
GetSint32
GetUint32
GetSint64
GetUint64
GetFloat32
GetFloat64
GetRZCharStr
GetGZStr
GetGZSerializable
GetVoid
GetError
SetUserData
GetUserData

 

 

You also find references to this in Ingred.ini

 

In fact, I am almost sure that the save files are written and saved in a way identical to any stuff that Ilive's Reader can read.

 

By the way: I am not sure yet what is "UserData" Error or Void, Serializable I think reads a entire class, GZStr reads a GZString (it is a string that has some custom data + inherits from stlport basic_string on Windows and from std::string on Mac) RZCharStr is just the C classic null terminated string, but seemly supporting UTF-8

[A Nonny Moose 8,262]

Have you compared the saves from the Aspyr port to the saves from the Windows version?  There is no guarantee they will match as Aspyr's port is not really all there with respect to the EP1 fixes and the BAT nightlights patch.

[OrSpeeder 72]

I am using only the debug symbols of Aspyr port, and only "roughly", there is a bunch of stuff that don't match, mostly because windows version use STLPORT and early EASTL while Aspyr use regular STL.

 

I suspect this is why Aspyr port is also quite buggy.

[A Nonny Moose 8,262]

Therein lies the rub.  Because of EA's attitude to releasing source information, even the debug schema, this becomes a daunting task.  Good luck with it.  Take a break now and then.  It feels very good when you stop beating your head against a wall.