[SC4DBPFLoading] A DLL that reduces SC4's startup time

[Null 45 3,048]

A DLL Plugin for SimCity 4 that optimizes the DBPF plugin loading that the game performs on startup before the SC4 logo appears.

DBPF is the name of the format that .DAT, .SC4Desc, .SC4Lot, etc. plugins use.

The effect that the plugin has on the game's startup time time will depend on your system and plugins. It may have little to no effect for some configurations.

Download: https://community.simtropolis.com/files/file/36244-startup-performance-optimization-dll-for-SimCity-4/

Source Code: https://github.com/0xC0000054/sc4-dbpf-loading

Features

• Reduces the time for the game to show the SC4 logo by up to 90%.
• Adds an optimization to the game code for large data reads.
• Allows the game to handle plugin paths that are longer than the standard Microsoft Windows limit of 260 characters.
  • Some of the file paths used by sc4pac packages may exceed the above limit.
• Reduce lag and city load times with large numbers of .SC4* plugins.
• Disables the game code that searches the entire file if the DBPF signature was not found.
• Changes the game code that loads the .SC4* files to ignore files with non-DBPF file extensions, .txt, .png, etc.
  • With this change only .SC4Desc, .SC4Lot, .SC4Model, and files without any file extension will be loaded.
  • Files without any file extension are included due to some older plugins having that issue.
• Changes the 'missing plugin' dialog to display the plugin pack ID in hexadecimal.

System Requirements

• SimCity 4 version 641
• Windows 10 or later

The plugin may work on Windows 7 or later with the Microsoft Visual C++ 2022 x86 Redistribute installed, but I do not have the ability to test that.

Installation

1. Close SimCity 4.
2. Copy SC4DBPFLoading.dll into the top-level of the Plugins folder in the SimCity 4 installation directory or Documents/SimCity 4 directory.
3. Start SimCity 4.

Command Line Argument

The plugin adds a -StartupDBPFLoadTrace: command line argument that can be used for advanced tracing/debugging of the plugin loading process.

It supports to following options (only 1 can be used at a time):

• ShowLoadTime - shows a message box with the resource loading time in milliseconds.
• WinAPI - shows message boxes before and after the resource loading code runs, this allows the user to start and stop a Process Monitor trace when the message box is shown.
• ListLoadedFiles - writes the loaded DBPF files to the plugin's log file in the order SC4 reads them.

Troubleshooting

The plugin should write a SC4DBPFLoading.log file in the same folder as the plugin.
The log contains status information for the most recent run of the plugin.

[Membrillo 56]

1 hour ago, Null 45 said:

up to 90%.

[Propfam 2,291]

Look, @Null 45 posted another BANGER!!! This is the mod that we've been anticipating since forever. Waiting 45 sec for a tile to be loaded is a masochistic action we've to do every day. Now, it's mostly gone, even for slower system. Null is officially the GOAT .

[hugues aroux 3,812]

Hello,
First of all... the idea of being able to shorten the loading of the game is great, especially when you are a Modder, Lotter,... Thank you for looking into this subject.
  .. tested! but failed -> CTD, 1 second after displaying the Region UI elements.
I can send you, in PM, Logs and other config information..!?

However, I note the time difference (\plugins=5,8Go): 46s without DLL -> 4s with ; before the start-screen - and the CTD 3s after ;-)...

[Haljackey 17,331]

Thank you for letting us beta test. I think I have experienced slightly faster times loading/launching the game the game itself, but loading cities and whatnot seems the same.

Note that I do have 10GB of plugins and city tiles with populations in the millions.

If you need more detailed investigations from me, just let me know. Thanks for all the effort you have put into this and others (and all those who have supported this effort too!)

---

9 hours ago, Jidan said:

My request: properly multithreaded and multicored SC4.

I mean this has likely been requested for literal decades now. Not saying it's impossible, but it's been looked at from various angles without much success. Let's keep this topic specific to the startup dll.

[Null 45 3,048]

7 hours ago, hugues aroux said:

I can send you, in PM, Logs and other config information..!?

The exception reports would be helpful (.mdmp, .txt). If the DLL is responsible for a crash, SC4DBPFLoading would be listed in the Exception module: line of the .txt file.

11 hours ago, Jidan said:

My request: properly multithreaded and multicored SC4.

Send your request to EA, they are the only ones who can make it happen.

[Haljackey 17,331]

So first bug to report- game freezes when I take a in-game screenshot

 

[Null 45 3,048]

51 minutes ago, Haljackey said:

So first bug to report- game freezes when I take a in-game screenshot

Confirmed. It looks like one of the memory patches is sending the game into a loop of some kind. I will have to disable them individually to try and find the culprit.

[Null 45 3,048]

Release beta version 0.8.1, which fixes a hang when taking a screenshot.

[Ke|is 3,336]

First of all, many thanks for your efforts, we really appreciate this.

I have tested this, I can see that loading times while starting the game are way faster, even for me with a long list of plugins I can notice the improvement instantly.

My problem comes as soon as I try to open a tyle, it does not matter whether this is empty or not, as soon as I click to access the tile the game CTD.

Do we have any clue on this?

If I can help with something just let me know.

[Null 45 3,048]

6 hours ago, Ke|is said:

If I can help with something just let me know.

Could you zip and upload the exception reports (.txt and .mdmp)?

If the crash is in SC4 itself, there is probably nothing I can do about it. But if the crash is in the DLL, that would be a bug that I can hopefully fix.

There have been previous cases where the the file read memory patch was breaking unrelated parts of the game code, so I may add a command line argument to disable that for testing or remove it entirely. All of the other memory patches are confined to the code that SC4 uses to load plugins on startup, so they shouldn't be affecting other parts of the game.

[Ke|is 3,336]

1 hour ago, Null 45 said:

Could you zip and upload the exception reports (.txt and .mdmp)?

If the crash is in SC4 itself, there is probably nothing I can do about it. But if the crash is in the DLL, that would be a bug that I can hopefully fix.

There have been previous cases where the the file read memory patch was breaking unrelated parts of the game code, so I may add a command line argument to disable that for testing or remove it entirely. All of the other memory patches are confined to the code that SC4 uses to load plugins on startup, so they shouldn't be affecting other parts of the game.

Thank you mate. I already sent you a PM with the Exception report, both .txt and .mdmp.

Looking forward to hear from you.

[Peter_One 1]

@Null 45

So just tested this. It does seem to decrease time till loading into region view. 

My game was set to load into an empty region with zero cities..

But when I tried to load a region with actual cities on it (Fairview), I got a CTD.

The log says:

Quote

SC4DBPFLoading v0.8.1
Disabled the built-in DBPF loading debug code.
Patched the DBPF Open header check.
Changed the missing plugin error message to use hexadecimal.
Installed the cRZFile hooks.
Installed the .SC4* plugin scan patch.
Failed to load: C:\Users\Peter\Documents\SimCity 4\Plugins\Network Addon Mod\1 Core\locale\NetworkAddonMod_Locale_sv.dat

So it's my guess that as soon as anything calls upon plugins needed for a city, it crashes..

EDIT: Did some further testing.

I removed NAM. Now it will CTD as soon as I see clouds load in, does not get to load even the region land. (The region probably is still set to one with cities on it)

The log file shows no more "Fail to load".  It stops at "Installed the .SC4* plugin scan patch."

[Null 45 3,048]

45 minutes ago, Peter_One said:

So it's my guess that as soon as anything calls upon plugins needed for a city, it crashes..

I can't explain that behavior. I suspect the most likely culprit is the Installed the cRZFile hooks. line. That change would affect the code that SC4 uses to read any file, and I am wondering if I broke it for some people due to something that doesn't occur on my system.

Could you post the log after starting the game with -StartupDBPFLoadTrace:ListLoadedFiles? If you want to edit the paths before posting it to remove your user name and the full install folder, that is fine.That command will list every plugin the game has loaded at the time the SC4 logo is shown.

The NetworkAddonMod_Locale_sv.dat error is harmless, it is an empty file that SC4's DBPF reader reports that it failed to load.

[Peter_One 1]

37 minutes ago, Null 45 said:

I can't explain that behavior. I suspect the most likely culprit is the Installed the cRZFile hooks. line. That change would affect the code that SC4 uses to read any file, and I am wondering if I broke it for some people due to something that doesn't occur on my system.

Could you post the log after starting the game with -StartupDBPFLoadTrace:ListLoadedFiles? If you want to edit the paths before posting it to remove your user name and the full install folder, that is fine.That command will list every plugin the game has loaded at the time the SC4 logo is shown.

The NetworkAddonMod_Locale_sv.dat error is harmless, it is an empty file that SC4's DBPF reader reports that it failed to load.

ok no prob, I sent it to you via private message.

[IBSTRIKER777 0]

Awesome - Works great! Cut my time to initially load the game and the time to load my cities from the region dramatically - close to 75% faster! And that's with a 11GB plugins folder that had brought my loading times to a crawl until I tried this out - thank you

[Peter_One 1]

2 hours ago, IBSTRIKER777 said:

Awesome - Works great! Cut my time to initially load the game and the time to load my cities from the region dramatically - close to 75% faster! And that's with a 11GB plugins folder that had brought my loading times to a crawl until I tried this out - thank you

It doesn't crash the game for you ? What version of windows? It's crashing as soon as the region view shows up for many of us..

[Ke|is 3,336]

There must be a specific trigger for this... For me, everything works fine, region view is perfectly displayed, the crash comes when I try to open a tile.

[Ulisse Wolf 4,416]

I did a test and found no CTD. Everything was working corectly.

[Shleaky 6]

This is very exciting! I just moved houses, but I will use this as soon as my computer is set up!

[IBSTRIKER777 0]

On 4/23/2024 at 11:11 AM, Peter_One said:

It doesn't crash the game for you ? What version of windows? It's crashing as soon as the region view shows up for many of us..

I am running Windows 11 pro - very latest version on insiders preview in the canary channel.. Nope not one crash yet it works without any problems for me.. I have used it on empty cities I am just starting and completely developed cities I have had for months all in the same region

[Wiimeiser 444]

Didn't crash for me, though I loaded a fairly simple city that's light on plugins. Windows 10 64 bit, if it helps.

[hugues aroux 3,812]

.. Well there has to be someone to blame.. and I don't want to be that one.. (aaaarrrrrgh)... So, it's YOUR DLL that doesn't like big packages of uncompressed PNGs!! Booooo he does his job too well oooooohh !

Well, enough of the joke, I'm continuing my investigations, but... I assumed, obviously rightly, that a plugin, and one of mine first, could "block". To make it short, after many tests and retro-pedaling,  ..  .. I noticed that: having wanted to do well, like Maxis, I did not compress the PNGs of the User interfaces (and no Dir-index). So I reinjected the PNGs with compression (and DIR file!!) ==> no crash!! Um.. there's another problem, probably because of a re-edited INI file... I'll see that tonight or tomorrow.

[rsc204 21,817]

On 20/04/2024 at 6:27 AM, Jidan said:

My request: properly multithreaded and multicored SC4.

Please for the love of god can you stop asking. It isn’t possible and asking over and over isn’t going to change the answer. If you ask again I will delete the post in question. 

[Peter_One 1]

Lol ... So I guess the crash is the fault of the ui mod by @hugues aroux 😆

[hugues aroux 3,812]

10 hours ago, Peter_One said:

So I guess the crash is the fault of the ui mod

Unfortunately not so simple, because my other UI (blue-skin) does not cause a CTD at the "Region" level, nor when loading a city! (And uncompressed PNG files like original UI!).

[hugues aroux 3,812]

Well, this is beyond my knowledge and understanding.
Although in use for years without any crash reports for my UI Mod, there seems to be a conflict. (the Raymond7cn UIs present the same problem!).
Your DLL seems less permissive than SC4 is.
The "start/splash-screen" png should not be in the same DAT as the UI PNGs. A stopgap, by redeclaring all UI PNGs in compressed mode. And we cannot redeclare an INI -> CTD file after start-screen! Which could pose a problem for future HD screen interface developers. I didn't see "ini" in the extensions you filter. But since I know nothing about programming... it has nothing to do perhaps! ;-)
That's all I can say about it. Available for further testing of course.

[Ke|is 3,336]

From my side I was able to go a step forward, but I am stuck again... I just removed all my plugins and I have been adding one by one my folders until find the culprit... I've started with terrain textures and NAM, everything good so far... then I added "MODS" folder and .... CTD... I have some mods that I've packed back in 2020, as soon as I removed this packed .DAT file, CTD no longer ocurrs... but wait, the game is not stable yet... so even after being able to open  the tile, after some seconds / actions my good friend "CTD" is back...

Thing is that It is not clear to me what is "inside" of this mod package... Progress, no progress

Below is my MODs folder, after removing " 4-Saitama Mods.dat" CTD is gone, and at least tile is accessible:

[Null 45 3,048]

5 hours ago, hugues aroux said:

Well, this is beyond my knowledge and understanding.

Same.

6 hours ago, hugues aroux said:

Your DLL seems less permissive than SC4 is.

I am using SC4's DBPF class to open the DAT files, same as the SC4 class that I am replacing. As long as all plugins are still being loaded, I can't explain how that would cause a crash. Unless I am somehow triggering a bug in the game's DBPF loading code that the original code doesn't hit. The game's DBPF loading code is a black box, I have no idea what it does behind the scenes.

At one point I had considered writing my own DBPF loading code for the DAT files instead of using the game's code. But I shelved that idea due to it being a massive amount of work, and having a high probability of breaking things even further.

[SneakyDeaky 1,128]

I want to test this to help out, I have a 6.5G sized plugins folder. I want to provide the correct information; I don't know this game very well, even after 15 years lmao, so IDK which files/error logs, ect. should I share that would be of help? I guess I should back up everything first. 

Not that this is anything to do with this really, but one way I have got "around" the tedious loading times is to load the lot editor first, strangely enough, then once that loads all plugins properly, I close that then start the game. That really does shorten the loading times for me, and at least I'm not stuck with a blank screen for two minutes; I can do what I want while the lot editor loads. Just an old routine I wanted to mention.