SC4Fix: third-party patches for SC4

[simmaster07 1,221]

This is a highly experimental fix I developed to attempt to fix the crash that occurs when hovering a puzzle piece over a transit-enabled lot. I'm not sure what practical purpose this serves for regular users but I hope this is a valuable resource for modders and other community members nonetheless.

How it works

This mod can now be downloaded from the STEX.

The files for this fix can be located on GitHub and is just a DLL which is unzipped to the Plugins folder. The DLL first overwrites code in SC4 that would normally unload the DLL for being nonstandard. The DLL then overwrites other parts of SC4's code to take control of potentially broken portions and catch and resolve issues that would normally cause crashes.

Source code can be found on the release page, and the GitHub repository as well. Visual Studio 2013 is needed for compiling.

Limitations

Originally, this mod only worked with DRM-removed copies of retail SC4. However, as of recent updates, this mod will now work on any fully-patched copy of retail SimCity 4 (i.e. update 640) and on the Steam version of the game (update 641). Presumably, GOG and Origin will work as well as they also use 641. It should be safe to use this mod on unsupported versions as it'll give you a nag message and just not do anything.

For retail SC4, testing was done on SKU 1, but I have no reason to believe that SKU 2 (European) EXEs are actually different.

Demonstration

This video shows the normal CTD behavior of hovering a puzzle piece over a transit-enabled lot:

This video demonstrates how hovering behaves with the fix in place:

More technical details

The puzzle piece CTD is caused by a null dereference, though I haven't reverse-engineered the game enough to figure out what exactly it's trying to access. The fix acts like the Detours library in Windows and relies heavily on modifying the x86 assembly instructions of the game on-the-fly. The DLL overwrites the crashy code with a jump to one of the DLL's own functions. These functions first execute whatever instructions were overwritten by the new jump instruction. For the puzzle piece issue, the fix checks to see if certain pointers (stored in x86 registers) are null, and if they are, executes the proper instructions to resolve the issue. To reference the three functions in the source code:

• Hook_Sub96D8E9 — if the pointer in question is null, it skips an instruction that would try to increase whatever would normally be there.
• Hook_Sub65EBA0 — if the pointer in question is null, it jumps directly to a piece of code which cleans up the function being called and returns out.
• Hook_Sub65EBA0_Pt2 — if the pointer in question is null, it sets another pointer (which would normally dereference the bad pointer) to null as well, which is handled properly by the game.

I wasn't aware of other CTD issues that would be as simple to replicate, though I have heard of issues with plugin conflicts and excessive plugins causing CTDs. However, this requires significantly more time to test, catch, and resolve. A couple of minidumps posted in technical issue threads didn't indicate much, and other posts in technical issue threads were capped to 20-ish lines since it had been thought that without debugging info, these crashes couldn't be solved. It'd be useful to have more of these.

With crashdumps, the EIP address can be used for finding the address of the crash, and provided it occurs within the SimCity 4.exe module (you'll have to refer to the crashdump to see if this address is between the base address and upper limit address for the game), the registers, stack frames and instruction dump can be used to try to deduce what went wrong.

What now?

The code for disabling SC4's DLL unloading allows modders to create DLLs which can remain in memory for as long as SC4 is open, and while it's not as versatile as a DLL with a proper framework that SC4 recognizes, it's the next best thing. This fix also provides a good basis for further excursions into the game's internals, and could be more widely distributed and used to fix more issues in the game once the problems regarding digital copies are resolved.

  Edited December 27, 2015 by simmaster07  
Updated for STEX upload

[rsc204 21,859]

I'll certainly be testing this out in the coming days. I know other CTDs exist, but this is the only problem I still experience. Every now and again I'm forgetful with the save function and miss a TE lot and it crashes, so it would be great to know that may be solved.

Better still is the prospect of somehow injecting code into the game, the possibilities that brings would be immense. Great work here.

[simmaster07 1,221]

I've managed to get a DLL working that prevents SC4 from unloading it. The DLL now overwrites a small piece of code in SC4 with instructions that do nothing; SC4 won't recognize the DLL as being loaded, but Windows no longer unloads the DLL from memory. This vastly simplifies the fix, since it can now just be dropped in the Plugins folder.

This also keeps the DLL from being loaded until after SafeDisc, Steam, Origin, or other DRM solutions are done decrypting the game code. Retail copies should work without no-CD fixes, as long as they're patched to version 640. Steam and Origin still won't function properly.

I've tagged this as Prerelease 2 and the source code can be found in the repo as usual.

[simmaster07 1,221]

Well this is moving a lot faster than I expected, hopefully making this post just a few hours later isn't frowned upon.

Apparently Steam itself sets a flag on any game it launches (excluding non-Steam games) to hide programs' threads from debuggers. Thankfully, someone discovered this long ago and developed a fix that prevents Steam from setting this flag, so I'm now able to get a debugger into version 641 and reproduce crashes there. Which means that I've been able to port this fix to the Steam (and presumably Origin/GOG) version of the game .

I've also added code for checking what version of SimCity 4 is running to prevent this from trying to inject code into versions like 610, 613, or 638. If an unsupported version is detected, SC4Fix pops up a message box giving update instructions and does nothing further.

Since the Mac App Store version of the game is just a Wine wrapper, I wonder if it's a wrapper around a regular copy of version 641, and if dropping DLLs in the Plugins folder still works. Would be interesting if this were true.

Anyway, here's prerelease 3, which, if all went well, should be the last version before uploading this to the STEX and moving on to possibly fixing other CTDs.

[APSMS 1,893]

I think there are things that modders have dreamed of doing with DLLs that are impossible beyond mere bugfixes (not to say that the TE-lot/Puzzle-Piece CTD is in any way an insignificant bug to fix).

This seems like it opens up whole new possibilities to modders, if the code can be properly written. Very incredible find! Thanks for the hard work.

[mrsmartman 200]

Good news. This magical mod should probably be bundled with future NAM releases.

[Tarkus 6,933]

I've just had a chance to test this briefly on my 1.1.640 disc copy, and my mind is completely blown.  Getting the DLLs to stay in memory has been the holy grail--to finally see it happen and for it to fix one of the most notorious bugs is one of the most amazing things I've been able to witness in my 10 years in this community.

simmaster07, you are officially awesome.

-Tarkus

[Cyclone Boom 17,031]

Wow, this is a revolutionary achievement for SimCity 4, and it doesn't even modify the game's internal files. After all these years of modding, to achieve what you've done is a remarkable breakthrough. Genius!

Can't wait to see what further possibilities this may open.

I smell a Trixie already...

[BLANKBLANK 4,414]

There is something more amazing- placing transit enabled lots right together facing the same direction. This means incredible new possibilities for TE lots. A limitation has always been that one tile gap.

[metarvo 711]

You are truly remarkable, Simmaster!    CTDs have always been my top problem with SC4 as far as gameplay goes,  and although I can't speak for everyone, I would hazard a guess that this particular issue contributes to a big part of CTDs across the SC4 community.  Seeing the puzzle piece/TE lot CTD put to bed would be sweet!

[Mister Giggles 310]

Congratulations, you've buried the last great barrier to modding SC4. Enjoy being a legend.

This fix works for the GOG version, as expected.

[Compdude787 916]

This is amazing! I ought to try this out!! I didn't think it would be possible to fix the "puzzle piece over transit-enabled lot" bug.

 

EDIT: Just tried it out, and it is awesome!!!!! Yaaaay, no more crashes when I hover a puzzle piece over a transit-enabled lot. I have had this happen with hovering a puzzle piece over a railyard located right next to an RHW and it sucked.

[simmaster07 1,221]

Thanks for the support, everyone! I've uploaded prerelease 3 to the STEX since I think this is ready for a wider release in its current state judging from my own testing and the positive feedback so far. Until I get the chance to analyze some more CTDs or find any lingering issues with this DLL, I'm mostly going to be using this project to do some experimenting with SC4's internals and trying to identify useful functions in the game.

If you have any suggestions for bugs to fix and know how to replicate them somewhat consistently, do let me know so I can investigate.

[rsc204 21,859]

I did some brief testing on the GoG digital version.

A number of situations where CTDs would previously be caused are no longer happening, so I'm pretty sure it's working.

I did find a replicatable CTD when hovering the Rail Viaduct tool over the HSRP Transit Hub included in the NAM.

As for other CTDs, most of these are solvable by setting SC4 up correctly and installing all the patches. Thought the obvious one for me would be the CTD when multiple cores are active. I had a brief but noticeable improvement with multi-core active, but it's not worth it whilst the regular as clockwork CTDs occur.

[simmaster07 1,221]

Hey @rsc204, I tried to replicate the rail viaduct crash but didn't have any luck. This is what I tried doing, maybe you're doing something differently?

Pardon the random voice in the audio, I was streaming the Chaos Communication Conference.

[rsc204 21,859]

Sorry, I tested the first release and somehow I didn't pick up on the instructions to move the DLLs location from Apps to the Plugins folder for the newer releases.

I had thought it was working, since I tested a spot where I had previously had a CTD a few weeks back. In the end it turns out I was using the wrong piece. Having now got it in the right place I'm no longer having any troubles.

I've been trying out every TE lot I have with an array of Puzzle Pieces since I got it setup correctly and so far no issues whatsoever.

[OrSpeeder 72]

I came to check the forums, and stumble on this...

 

WHOA!!!

 

Maybe I will even get back making my mod (I had sort of given up because I had some issues with it that were unsolvable without this kind of modding)

[simmaster07 1,221]

Since at least a couple of people brought up the idea of packaging this with the NAM, I'd say it's worth noting that this mod is MIT licensed, so anyone is free to redistribute it anywhere and do as they please with the code.

I'd wager that the only real issue is what happens when multiple copies of this DLL are installed since that could cause conflicts. That's also something I'm working toward fixing, though. New versions going forward can report conflicts to players. The one thing that needs to be added is detecting versions without this feature, which shouldn't be difficult.

[Sabretooth78 808]

This is awesome news!

I would personally be most appreciative of something that could fix the multi-core issues - as a Mac user running the Amazon/GOG version under WINE (i.e. I can load DLLs), the CPUCount tag doesn't work and I have to manually throttle the whole PC down to 1 CPU which is something of a PITA requiring reboots before and after playing.  (I can run under a virtual machine as well but it doesn't run as well and in fact crashes with my very large region.)  Not sure if that's in the cards though and I'm certainly not going to expect it.

Do you suppose there could ever be modding potential with this; say for instance the hard-coded nuisances with tunnels or even additional networks?  Obviously speculative at this point, but maybe something worth shooting for?

[simmaster07 1,221]

It'd likely take exponentially more effort to deal with hardcoded game issues since it'd require reverse engineering large blocks of game code. That's not to say it's impossible, it's just that it'd probably be another few years at the least before we could reach a point where we could perform that kind of modding. Then again, I've yet to look too far into it anyway.

As far as the multi-CPU issue goes, that might be an issue with Wine. What SC4 does internally with the CPUCount flag is literally the same as if you'd set the processor affinity in the task manager, which is probably unimplemented in Wine. I'm curious as to what exactly is going on with the multi-CPU crash, though. It could be a race condition but I don't know why or for what purpose SC4 would spawn threads dealing with the same bits of memory.

[Tropod 144]

On 26/12/2015 at 9:24 PM, Tarkus said:

I've just had a chance to test this briefly on my 1.1.640 disc copy, and my mind is completely blown.  Getting the DLLs to stay in memory has been the holy grail--to finally see it happen and for it to fix one of the most notorious bugs is one of the most amazing things I've been able to witness in my 10 years in this community.

simmaster07, you are officially awesome.

-Tarkus

ditto.

I've had a very quick/brief look at the source code & yeah, I'm gobsmacked. Very impressed.  x^ for your efforts.

[JP Schriefer 5,591]

A huuuuuge thanks for this awesome mod!

[phalanxs 1]

This is awesome ! Thank you !

[Mister Giggles 310]

A suggestion for a fix...

Perhaps this? Less difficult to repeat than random crashes, at least.

[simmaster07 1,221]

2 hours ago, Mister Giggles said:

A suggestion for a fix...

Perhaps this? Less difficult to repeat than random crashes, at least.

Interestingly, I can't reproduce this. (Win10 x64, Intel graphics)

[Mister Giggles 310]

Which portion? The inability to launch in 32-bit color depth in exclusive mode, or the color banding? The latter can be explained by the Intel chipset doing dithering...

[simmaster07 1,221]

58 minutes ago, Mister Giggles said:

Which portion? The inability to launch in 32-bit color depth in exclusive mode, or the color banding? The latter can be explained by the Intel chipset doing dithering...

Neither; the game launches in 32-bit color depth in exclusive mode, and if I force 16-bit it seems like my integrated graphics are doing dithering.

[Mister Giggles 310]

Oh well.

At least we know Intel works properly, eh?

[rsc204 21,859]

I bet that's only on Windows 10.

It's quite clear that something about Windows 10 improves compatibility of Intel's integrated graphics with SC4. On Windows 7/8 you'll be lucky if you can even use DirectX or OpenGL rendering.

@simmaster07 If you are really itching for something else to fix, I'm not sure if CTD's are really worthwhile.

There are many reports from those with problems running SC4, but honestly every one can be solved with a little effort. The problem is many users lack the technical ability or will to do anything about it and just complain SC4 is a buggy mess (it's not). The PP over TE lot was the only one that couldn't be fixed, until you came along ;).

Two situations where consistent CTDs will always occur:

• Switching tree controllers when flora from the old one remains in a city. (Widely known, every one I've seen has a warning, people fail to read it).
• Loading the game with textures larger than 256x256 installed using OpenGL or Software rendering.

If you fixed those CTDs, it wouldn't change the fact you couldn't do those things. The CTD itself is harmless enough.

Multiple cores being used will consistently cause CTDs (some report otherwise, but honestly I'm not sure). If fixed from what I've seen and heard, maybe we'd get 10% improvement in CPU performance. Sure I'd take that 10% and I'm sure others would too, provided it was proven to be stable. The code still isn't multi-core optimised though, so perhaps it's a moot point?

All other CTDs I can think of right now are going to be much harder to track because they can't easily be triggered.

[mrsmartman 200]