Embedding DBPFs in other files

[simmaster07 1,220]

Nothing major, just throwing a fun fact out here. I've been looking at savegame-related stuff lately and found a weird quirk in how SC4 loads data. Almost every modder already knows that most game data (plugins, .dat files, savegames) is packaged as DBPF files. Here's what the SC4Devotion wiki has to say about them:

Quote

DBPF files can be identified by opening the file in notepad, or preferably a hex editor, as the first 4 characters/bytes will be the letters DBPF. If a file in the plugins folder does not start with these 4 letters the game will not load them [...]

This actually isn't true! For some absolutely insane reason, if a file doesn't start with "DBPF," the game will still scan through the entire file and try to load it by looking for this byte sequence:

80 9D 88 EC 8F 24 03 6C C9 A6 31 56 5B CF 77 22

If the game finds these magic bytes it'll try to load any data from that point on as a DBPF. This lets you do weird things like combine .dat and .jpg files. If you want an example, download the demo.jpg file I attached to this post and put it in your plugins folder. If it worked, you'll see this hastily made icon in your parks menu:

Now as a novelty I think this is pretty funny, but I can't think of a practical reason for why anyone would do this. Here's what I think this could mean for modding though: it's a pretty solid confirmation that the game will waste time on non-game files like JPEGs that get stuck in plugin folders. This is inefficient since the game is forcing itself to fully scan every file in the Plugins folder for this kind of data. But it's possible to patch out this check so that the game only looks for the "DBPF" sequence at the start, and bails out early if it's not there.

This could meaningfully improve load times for people with large plugin folders that still have JPGs, ZIP files and other non-game data in them. But since spinning hard drives are going out of style, this might not offer much improvement either.

Anyway, here's the demo file I mentioned earlier:

demo.jpg

[Cyclone Boom 16,851]

Hi @simmaster07

I initially tried testing with the file you'd attached, but didn't notice it appear for the Small Flower Garden icon as per your pic. Knowing that today wasn't April 1st and I knew something must be awry for it not to work. I tried with an empty Plugins folder, but still no go.

Then I had a thought spring to mind...

Perhaps the server-side compression for images has affected the data internally, and hence disrupted the special byte sequence you'd crafted. Sure enough after downloading the original JPG image directly from FTP via the ST server (admin access), I noticed the file was notably larger in size (52,426 versus 6,250 bytes).

Adding that bigger demo.jpg file into Plugins, and lo and behold it does work!

Here's a preview matching your pic:

 

I've attached the correct file in place using some admin trickery, so this way it downloads like a BMP file would.

This is super cool as a proof of concept, and shows how the game scans through files to determine whether it is of DBPF form or not. This way for having clutter in one's Plugins folder, and why it's recommended to clear this up to save it doing so. It would all add up when having thousands upon thousands of individual files, and likewise why DatPacking reduces loading time given there are fewer files to process.

Maybe there could be some way of making this a revolutionary new method of loading certain data for the game to interpret, who knows...

[rsc204 21,806]

15 hours ago, simmaster07 said:

Nothing major, just throwing a fun fact out here. I've been looking at savegame-related stuff lately and found a weird quirk in how SC4 loads data. Almost every modder already knows that most game data (plugins, .dat files, savegames) is packaged as DBPF files. Here's what the SC4Devotion wiki has to say about them:

I guess that's a simplified version of things, since I'm pretty sure this has been known, even if perhaps not the specific reasons for it.

A CleanItol file exists here, which anyone can run periodically to remove any non-game files automatically, for example.

I have also personally experienced problems as a result of non-game files being present in Plugins. I can attest that too many such files, can lead to instability of SC4, and not simply whilst loading either. In my case, I returned my game to being rock solid stable once more, after clearing all the crap from Plugins, I note no SC4 file formats were removed as part of this either.

15 hours ago, simmaster07 said:

But it's possible to patch out this check so that the game only looks for the "DBPF" sequence at the start, and bails out early if it's not there.

I guess that would be a boon considering how many users are likely unaware of the issue.

15 hours ago, simmaster07 said:

This could meaningfully improve load times for people with large plugin folders that still have JPGs, ZIP files and other non-game data in them. But since spinning hard drives are going out of style, this might not offer much improvement either.

Even when using an SSD, there is probably going to be a noticeable difference, especially if during the indexing process, data needs to be shuffled into RAM. But the better solution would be if those files didn't exist in the first place, so they simply were no longer a part of the loading process full stop, which is achieved real easy using Cleanitol.

[CorinaMarie 29,571]

15 hours ago, simmaster07 said:

This lets you do weird things like combine .dat and .jpg files.

I've peeked inside and see how the bytes up thru 0x00001969 are the JPG image, then 0x0000196A to 0x00001979 are the magic numbers, and 0x0000197A to the end of the file are the regular .dat file that Reader and the game can understand.

 

15 hours ago, simmaster07 said:

I can't think of a practical reason for why anyone would do this.

Chatting with CB and he had one idea, tho we both don't see it as practical, is one could have their plugins identifiable with JPG images for casually looking at them in File Explorer. It would also have the disadvantage of the files not being editable directly in Reader.

Still, a quite interesting discovery.

 

15 hours ago, simmaster07 said:

But it's possible to patch out this check so that the game only looks for the "DBPF" sequence at the start, and bails out early if it's not there.

That would be an excellent addition to your SC4Fix.dll for peeps who do have extraneous files in their plugins. I, myself, have some zip files in there because I'd been led to believe the game just ignored them. (I do said zipping when modding to have a quick, handy backup before tweaking something.)

[Propfam 2,286]

Savegames are DBPFs. So, this opens up possibilities to have a journal that can be opened thru the Explorer and also an actual region file that can be played. This can be developed into a new mode of SimCity 4 gaming. Or, it can be used in multiplayer setting where you can literally upload the savegame into YouTube and only the video section is played and you can download the same video and play the region, at least on the paper. Practically, YouTube compresses and even recodec (if any) the data inside the video so the region is probably lost. Alternatively, we can host a website similar to CuriousityStream but with SimCity 4 content. Here you can upload a video containing the region and in the website, you can see the journal. But if you want to play, just literally download the video and place it to the Regions folder.

Unfortunately, you have to repeat the same process each tile. So, no continuous episode from the first city to the last city. It must be parted. If you know how to combine savegames into one region file (not folder, though the user might have to create a separate folder and region.ini), that will be appreciated.

[CorinaMarie 29,571]

22 minutes ago, chfzdn said:

So, this opens up possibilities to have a journal that can be opened thru the Explorer and also an actual region file that can be played.

I tested by renaming a city save file to have .jpg as the extension instead of the normal .sc4 then inserting the magic bytes via a hex editor. This prevents the game from loading the city tile. It seems likely @simmaster07's discovery pertains to the Plugins folder only.

[Propfam 2,286]

On 7/12/2020 at 12:25 PM, CorinaMarie said:

I tested by renaming a city save file to have .jpg as the extension instead of the normal .sc4 then inserting the magic bytes via a hex editor. This prevents the game from loading the city tile. It seems likely @simmaster07's discovery pertains to the Plugins folder only.

Or, the user just rename the file to .sc4 if they wish to play the savegame inside the video.

[CorinaMarie 29,571]

6 hours ago, chfzdn said:

Or, the user just rename the file to .sc4 if they wish to play the savegame inside the video.

Interesting. Can I presume you've tested this?

[rsc204 21,806]

Even if this works, it kinda begs the question why would you want a save file to also be a video? Seems to me, especially where Save data is concerned, it’s better off left well alone. 

[Propfam 2,286]

I also forgot that sky background pngs are read as is, not like described above.