simmaster07

This is a highly experimental fix I developed to attempt to fix the crash that occurs when hovering a puzzle piece over a transit-enabled lot. I'm not sure what practical purpose this serves for regular users but I hope this is a valuable resource for modders and other community members nonetheless. How it works This mod can now be downloaded from the STEX. The files for this fix can be located on GitHub and is just a DLL which is unzipped to the Plugins folder. The DLL first overwrites code in SC4 that would normally unload the DLL for being nonstandard. The DLL then overwrites other parts of SC4's code to take control of potentially broken portions and catch and resolve issues that would normally cause crashes. Source code can be found on the release page, and the GitHub repository as well. Visual Studio 2013 is needed for compiling. Limitations Originally, this mod only worked with DRM-removed copies of retail SC4. However, as of recent updates, this mod will now work on any fully-patched copy of retail SimCity 4 (i.e. update 640) and on the Steam version of the game (update 641). Presumably, GOG and Origin will work as well as they also use 641. It should be safe to use this mod on unsupported versions as it'll give you a nag message and just not do anything. For retail SC4, testing was done on SKU 1, but I have no reason to believe that SKU 2 (European) EXEs are actually different. Demonstration This video shows the normal CTD behavior of hovering a puzzle piece over a transit-enabled lot: This video demonstrates how hovering behaves with the fix in place: More technical details The puzzle piece CTD is caused by a null dereference, though I haven't reverse-engineered the game enough to figure out what exactly it's trying to access. The fix acts like the Detours library in Windows and relies heavily on modifying the x86 assembly instructions of the game on-the-fly. The DLL overwrites the crashy code with a jump to one of the DLL's own functions. These functions first execute whatever instructions were overwritten by the new jump instruction. For the puzzle piece issue, the fix checks to see if certain pointers (stored in x86 registers) are null, and if they are, executes the proper instructions to resolve the issue. To reference the three functions in the source code: Hook_Sub96D8E9 — if the pointer in question is null, it skips an instruction that would try to increase whatever would normally be there. Hook_Sub65EBA0 — if the pointer in question is null, it jumps directly to a piece of code which cleans up the function being called and returns out. Hook_Sub65EBA0_Pt2 — if the pointer in question is null, it sets another pointer (which would normally dereference the bad pointer) to null as well, which is handled properly by the game. I wasn't aware of other CTD issues that would be as simple to replicate, though I have heard of issues with plugin conflicts and excessive plugins causing CTDs. However, this requires significantly more time to test, catch, and resolve. A couple of minidumps posted in technical issue threads didn't indicate much, and other posts in technical issue threads were capped to 20-ish lines since it had been thought that without debugging info, these crashes couldn't be solved. It'd be useful to have more of these. With crashdumps, the EIP address can be used for finding the address of the crash, and provided it occurs within the SimCity 4.exe module (you'll have to refer to the crashdump to see if this address is between the base address and upper limit address for the game), the registers, stack frames and instruction dump can be used to try to deduce what went wrong. What now? The code for disabling SC4's DLL unloading allows modders to create DLLs which can remain in memory for as long as SC4 is open, and while it's not as versatile as a DLL with a proper framework that SC4 recognizes, it's the next best thing. This fix also provides a good basis for further excursions into the game's internals, and could be more widely distributed and used to fix more issues in the game once the problems regarding digital copies are resolved.

What is this? Scion is a (very work-in-progress) reference implementation of the Gonzo-Rizzo game engine used by Maxis in the late '90s and 2000s. You can find the code here: https://github.com/nsgomez/scion Who is this for? This project is mainly for programmers either developing DLL mods, or digging into the internals of 3D-era Maxis games. While this framework version is specifically based on SimCity 4 Deluxe, it might still be helpful for looking at different versions of the framework that were used in other Maxis projects, like: SimCity 3000 The Sims The Sims 2 Spore Since this is an SDK and not a standalone mod, it won't immediately benefit players. How is this different from gzcom-dll? gzcom-dll is a basic SDK for writing a DLL that the game engine will load, and for plugging into different parts of the game. It's mostly meant for making as many SC4 game interfaces accessible as possible. Scion is the game engine that would be loading DLLs and tying them all together. How does it work? (This section summarizes a lot of stuff that's been written about the game engine by myself, speeder, and Null 45, so if you're already familiar with DLL modding, this probably isn't news to you.) Gonzo-Rizzo is named for Muppets characters Gonzo the Great and Rizzo the Rat. As best I can tell, Gonzo ("GZ" in code), like his name suggests, is responsible for a lot of the main functionality: the core framework, graphics, sound, resource management, and so on. Rizzo ("RZ") mostly handles utilities like string parsing, filesystem access, thread synchronization, and reusable helpers. The game engine is modeled after Microsoft's Component Object Model (COM). At the core of the engine is the Framework, which provides: A GZCOM class registry, for registering component classes and creating new instances of registered classes; A registry of system services, which are singleton classes that can be used by components and do work on game ticks; Coordination for game ticks and framework hooks. Above the COM are directors, which act as framework hooks and can (and usually do) have their own registered classes, as well as child directors. The engine comes with prefabricated directors for doing graphics, sound, UI, and resource management, among other things. If you ever checked at the Apps folder in SimCity 3000 and saw a bunch of GZ*D files, that's what those are: GZSoundD.DLL, for example is the GZ Sound Director. Games build on top of the engine by doing the basic framework initialization and registering their own directors. For example, SimCity 4 has its own cSC4COMDirector to register its 3D renderer, simulator, lot developers, etc. SC3K has a bunch of SIM* DLLs that serve the same purpose, like SimTransit.dll, a GZCOM DLL director that implements transit networks. The game also takes responsibility for loading plugin DLLs, which is where expansion functionality comes in. (Aside: for any enterprising SC3K modders, those built-in DLLs might be a useful entry point for modding that game.) How far along is Scion? As of posting, the Scion implementation has a core GZCOM implementation that's mostly done with some scattered TODO items, and a mostly-done implementation of DBPF for resource management. There's still some more work to be done on the resource manager, and no work has started on the other core directors. The README on GitHub has a table showing the latest status.

Worth noting that STLport is a fork of SGI STL that worked on making it portable to different platforms, and doing general maintenance when SGI stopped maintaining their library. MSVC was also unreliable at the time since the string class in VC6 had bad bugs like corrupting memory when multi-threading and being slow at string assignment (Paul Pedriana called this out in his GDC talk). My best guess is that SC4 Deluxe uses a pre-release version of STLport 4.6, but I also don't think the changes since 4.5.1 or 4.5.3 were major enough to really break function ID. I opened a PR to add FID databases for STLport 4.6 and Lua 5.0, built with Visual C++ 2003, and I get pretty solid coverage on SC4.exe v641 from that. Unfortunately a lot of the STL template types are both inlined and only generated at compile time with those types, which was necessary for performance but makes the FIDBs fairly limited. I'm also looking at getting FIDBs set up for libpng and zlib, but I suspect those were built with a different compiler than the game was using, because the function prologues are a lot less optimized than VC6 or VC7, and I can't find a combination of compiler flags that reproduces that. (For example, libpng functions always copy registers edi, esi and ebx to the stack and preserve them, even if those registers aren't used by that function.)

That's my bad, I was aiming for LGPL-2.1 to be more permissive (e.g. allowing GPLv2 projects to use it), but I didn't mean to restrict the use of LGPL-3.0. I just pushed a couple of commits to SCGL and Scion to relicense as LGPL-2.1-or-later (and add the notice headers to SCGL), and if I were going to relicense gzcom-dll, it would also be under LGPL-2.1-or-later.

re: gzcom-dll, I think you and I are the two main contributors on that project, and one contributor was making an automated fix to a solution file, so as long as @memo is okay with relicensing to LGPL to enforce source code sharing then we should be fine.

Fair point - from a licensing perspective it'd be fine to vendor those classes separately and link them with the MIT code, but for simplicity's sake I'll probably just relicense Scion to MIT when I get home tonight. I'll also have to check out the BaseMultiPackedFile implementation, that should be a big help for wrapping up the GZResourceD implementation. Thankfully Aspyr at least included retail symbols for SC4 and TS2 in their Mac ports, and a lot of the stuff that isn't platform specific is reused between those two games and SC3K, so the work is still doable, just takes additional time to correlate those symbols with the Windows ship builds. Might be worth setting up a Ghidra server at some point for that.

What's the deal with hardware rendering? If it's so hard, why don't they make easyware instead? 20 years after its release, SimCity 4 has aged remarkably well; the graphics engine has not. (The graphics themselves are pretty good though.) The Graphics Rules file has no understanding of newer GPUs and has a bad habit of picking sub-optimal graphics settings. The DirectX driver was effectively obsolete by the time the game came out; errors like "Could not initialize DirectDraw" have randomly come up since release. And if you get that error, your framerate falls off a cliff as the game puts itself into software rendering mode. (We don't talk about the software renderer.) There is a hidden option to use OpenGL instead of DirectX by passing "-d:OpenGL" in the command line options. This isn't viable either since everything falls apart after loading a city: Sad to say this is not a snow mod. What can be done? We could use a wrapper like DDrawCompat or dgVoodoo, and that works fairly well. But adding translation layers also means taking a performance hit, as documented in that thread. There is one other saving grace here, though, which you can probably guess from the thread title. If you know the magic words, the game can load a DLL that upgrades an existing component. So join me on an adventure as we try to write our own graphics driver! (Spoiler: we're not there yet.) SimCity 4 has a couple of components to handle putting pixels on the screen. There's a GraphicsSystem that translates the game state to finer graphical commands, and a GDriver (graphics driver) that sends those commands to a graphics API, like DirectX or OpenGL. The GDriver interface is a target that the game calls SimGL. As the name implies, it's an abstract OpenGL 1.2 target, so the DirectX and Software drivers have to translate OpenGL-like commands to something that makes sense for those APIs (*). Also like OpenGL, GDrivers support "extensions," which are optional but can be leveraged by the GraphicsSystem to improve performance or add features if they're available. For example, lighting support is an extension supported by OpenGL and DirectX, but not by Software. If the abstract machine is based so heavily on OpenGL, then why is SC4's OpenGL support poor? I'm not totally sure, but I think there's a few issues that play a part: A lot of SC4 textures (FSHs) are compressed using DXT/S3TC, and the game was written to target OpenGL 1.2.1 (as far as I can tell). DirectX standardized support for these textures, but texture compression wasn't a standard feature until OpenGL 1.3, and this specific format couldn't be built into OpenGL because of patent issues. Vertex buffers, which would improve performance, were also not standard in OpenGL yet. The DirectX GDriver implements them, but OpenGL does not. There's also an extension for "buffer regions" to boost performance which, again, is not standard in OpenGL. The game does try to provide them as an extension, but it seems this does not work for Intel graphics. A lot of these features became standard or ubiquitous long after SimCity 4's development cycle, so I've made a project called SCGL (pronounced like "sigil") to rewrite the OpenGL GDriver and add these features. I picked OpenGL 3.0 since the game depends on a lot of features that are now deprecated. 3.0 was the last OpenGL version to offer these features for compatibility, and recent GPU drivers still have compatibility modes for it. It's still very early days, but the good news is that I'm at the point where the game loads and we have something showing up on the screen. If you can believe it, this is region view, and those green boxes are clouds. Again, super early days. But it's a starting point for debugging, so the next immediate goal is to fix the texture corruption. Getting region view to be recognizable would be nice, but honestly I would settle for having the buildings in a city show up as untextured white boxes and iterating from there. (*) This has some interesting implications for the Mac port. Since SC4's DirectX driver was the only stable hardware renderer, Aspyr had to write their own DirectX wrapper, even though Macs only supported OpenGL. So it has an OpenGL GPU pretending to be a DirectX card (Aspyr's wrapper) pretending to be an abstract OpenGL API (SimGL).

SCGL is actually almost feature complete, there's just one rendering bug involving alpha transparency that I've been racking my brain on for... ever. You're still welcome to try it out if you're cool with cloud shadows sometimes showing up as black splotches, or cars not fading in and out right. @grebe48 I think the assert is because you built the DLL in Debug mode. It's trying to write some OpenGL debug messages to a log file at C:\temp\cGDriver.opengl.log, so if the C:\temp folder doesn't exist, it'll fail out like that. Creating that folder should make the asserts go away; or, you could change the path in cGDriver_Viewport.cpp under the LogGLMessage function; or you could switch to a Release build.

The degree of improvement seems like it'd depend mostly on the type of drive the Plugins folder is on, and the amount of space that's taken up by non-DBPF files. A 10GB plugins folder on an NVMe drive (~7500 MB/s read speed) that had DATPacker and Cleanitol run on it might not see much benefit. The same size plugins folder on a 7200rpm HDD (~100MB/s read speed) with a bunch of images and readme files would load a lot faster.

Hey everyone, After SC4Fix, I went on a hiatus for college and decided to turn my attention to loading DLLs into the game using native methods. SC4Fix works but uses a really hackish method where it overwrites bits of code that would unload DLLs the game doesn't recognize. This is something that I've wanted to do for six years since Paul Pedriana (very graciously) sent me some incomplete but critical code for making proper SC4 DLLs. I recently threw out all of the code I wrote and started from scratch with Paul's code as a starting point. Using the Aspyr port of the game for cross-checking (since it holds a lot of debug information that the Windows version doesn't have), I finally managed to create a DLL that the game can load, recognize, and communicate with. This also essentially gives us the foundation of the SimCity 4 SDK that would be needed for accomplishing more advanced tasks. Source Code Test DLL What I have right now is pretty basic: SC4 can load the DLL, recognize it as a real framework DLL, and our code can subscribe to game events like PostAppInit, PostFrameWorkInit, PreAppShutdown, and others. I feel that the next step would be to expand the code to replicate the functionality of the Extra Cheats DLL. Buggi has partial source code that includes the most crucial components aside from the rest of the SDK, and I've managed to assemble a basic SDK to work with, so this should be done pretty soon. The best part about having this kind of DLL is that it works with every Windows version of SC4 at least, and probably works with the Mac version (but I haven't tested this) — v610, v638, v640, and v641 should all be able to use this with no problem because the API never changes. --- As an aside, while investigating how DLLs are loaded by the game, I also learned that you can create INI files that tell the game whether or not to load certain DLLs. You can create a file in the Plugins folder called DLLName.ini (where DLLName corresponds to the filename of the DLL minus the file extension) with the following content: Changing "true" to "false" stops the game from loading the DLL, and changing it back to true allows it to load again. Pretty neat in my opinion, and I hadn't seen this documented anywhere else, so I figured I'd share it here. --- Final aside, I'm planning on making this a general development thread for myself so I don't have to make a new thread for everything I find. Once I have more to report on I'll probably change the title of this thread.

There's interest in it and some efforts have popped up occasionally, but there hasn't been a coordinated push for it, and the work that has been done is coming from a lot of different angles: @Null 45 has been contributing back to the gzcom-dll project and writing their own DLLs, which work from the "top down," modifying game memory and inspecting SC4's interfaces to make changes to the game logic. I, personally, have mostly pivoted to "bottom up" work by reverse engineering the game engine and framework components (Scion) before getting at the SC4 bits built on top of that. (And I don't exactly have the free time I did in college, so the time commitment's been pretty shaky for years now.) Others still have been taking a complete clean room approach (OpenSC4) and rewriting as much as they can in game engines like Godot. I've also gotten emails from maintainers on "Open Sims 2"-like projects -- even though it's a different game, SC4 and TS2 share a lot of internals, so there's mutual interest in whatever's happening between the two. There's still a lot of quality-of-life changes that can be made without making a whole OpenSC4 project, so it's probably not something worth prioritizing right now. But having an active DLL modding scene helps with building up the kind of knowledge and expertise that can push a project like that forward, and I predict it'll gain more traction when other avenues for improving the game are exhausted. It's also a project that would require a lot of manpower, since SimCity 4 is a bigger game with a more complicated engine than projects like TTD and RCT2. There would also have to be ground rules established on what a project like that should look like. For example, even if no assets like DAT files are distributed, it might not be okay to create a straight-up EXE replacement, since that could be used to circumvent copy protection, which is what got projects like Bnetd in trouble. It would also be a bad idea to try porting the game to mobile, as FreeSO tried with The Sims Online before getting a cease-and-desist. FreeSO's port would've competed with The Sims Mobile; doing that for SC4 would compete with SimCity BuildIt. EA backed off of FreeSO when that port was cancelled.

Drag the DLL to the Plugins folder, add -d:OpenGL to your game launch options. Nah, that's further up the game logic, can't do anything with that at the level we're working at.

Biffa's also got a 30 minute let's play:

I would have to dig deeper into how the game uses the rdtsc instruction to know what impact it has. It's not like the instruction just counts time on modern processors though. My observation is that it increments at a constant rate, but the rate it increments at is still roughly based on your processor speed. The problem is that modern CPUs change clock frequency all the time to optimize power usage, so using raw clock data to measure how fast a processor is also causes timing issues, since time could speed up or slow down essentially at random. Now it's fixed at your processor's base clock, which is... probably fine for SimCity 4? As far as I know there are rules in the game that use CPU speed to judge your hardware capabilities, but beyond that everything is based on timers and increments of time, not increments of clock cycles. As for multithreading, you wouldn't want a completely separate process since the cost of inter-process communication is unreasonably high. I guess in theory you could make a separate thread that handles rendering commands, and SCGL just dumps commands into a queue for that thread and hands control back to the game. But DirectX and OpenGL already queue commands to the GPU asynchronously, so the benefit of that is also questionable. You would have to tear out a lot of code further up the stack to get a reasonable multithreading model going.

Got some updates, but first a quick word on RTX Remix since people have been asking and I don't think I ever responded. I haven't tried it yet, and it requires DirectX 9 instead of OpenGL, so it's not likely I'll write another renderer to test it. I'm also really skeptical that it'll work for raising the fidelity of SimCity 4. Path-traced lighting depends on there being some kind of 3D geometry to simulate light against. The problem is that buildings are pre-lit and pre-rendered by the BAT, and essentially flattened into a picture to be copied and pasted onto the screen. I suspect the lighting simulation will suffer a lot as a result, and not produce the results people are looking for. --------------------------------------------------------------------- As for the main topic, let's start with the last showstopper bug. I found what was causing textures to get drawn in the wrong places, as I showed back in January: This is now mostly fixed: We also have working night mode now: I have a rough roadmap for making this a stable mod, which looks like this: Polish up the rendering engine. Retaining walls and terrain edges are still broken, which you can see above. Some transparency effects are also broken. Fix other bugs with how SC4 deals with SCGL. Screenshots and CustomResolution still crash the game. Make it fast. The focus so far has been entirely on correctness and not speed, so SCGL runs slower than DirectX. I have some ideas on how to close that gap. Also, thanks to @TorchedSammy, you can get unstable builds of SCGL from GitHub, which get compiled as soon as the source code is updated. I don't recommend doing that quite yet, but it is there if you're interested in the novelty of it. You will need to use one of the game's "known good" resolutions (800x600; 1024x768; 1280x1024; 1600x1200). You also need the latest Visual C++ 2022 redistributable. Finally, I rigged my own custom Graphics Rules.sgr file that skips parsing Video Cards.sgr and just assumes the fastest possible hardware. That probably has some value even if you're just using the DirectX hardware renderer, so check that if you're interested.

Long time no chat! Life got pretty busy after I posted my last update, so I didn't have much time to work on side projects like this. No, DirectX is a different way of writing graphics code. I'm also trying to bring this part of the game up to modern standards, which necessarily means dropping older GPUs with no OpenGL 3 support. (If it works, it works, but if it doesn't work, I have no plans to fix it.) If you need support for older GPUs, stick with the DirectX hardware renderer that the game came with. I don't think we'll be able to upscale the region view from this project, but also, there's not much fun in just packaging dgVoodoo. I'm doing this out of my own personal curiosity to drill into the game internals more than anything else. Anyway, I think I'm settled in again now, so let's revisit this, because this is the second-most important graphics optimization in the game: Consider your typical scene: How much effort does it take to render this? It depends on your zoom level and some graphics settings, but the high-level steps would (and I'm really oversimplifying here) look like this: Clear the screen. Draw the background (i.e. the gridlines on the edges of the map). Draw the terrain. Paint lots, roads, train tracks, and everything else "on the ground." Draw sims and vehicles. Draw the buildings and props. Draw the visual effects. Draw the UI. This isn't a real-time shooter or anything, so you can get by with a "cinematic" 24fps, but drawing all these lots is still going to be a struggle. Every "thing" on the screen -- bits of terrain, lots, buildings, props -- requires the CPU to send a separate message to the GPU saying "hello, please draw these triangles on the screen with these textures", and when you're zoomed out or working with a dense city, the overhead of sending these messages can get overwhelming. You know what would be nice? If the CPU could batch all these small messages together into one big message, so the GPU can just go through a whole list of draw calls at a time with way less overhead. Unfortunately, you are a graphics developer in the year 2002 and there won't be any useful way of doing that for another 10+ years, and you're stuck targeting a GeForce 256 from 1999. Even then, you need a software rendering mode for kids and parents who don't know how to read the system requirements on the box. You're already doing 2.5D rendering like in SimCity 3000, and that takes a huge load off since almost everything is now a pre-rendered box, which is the #1 most important optimization you could make. But it's still a lot of effort to render every lot in a loop. This is where the dirty rectangle system comes into play. Conceptually, dirty rectangles are a very fancy screenshot system. After the scene is rendered once, it copies the rendered result to an invisible region of graphics memory (a.k.a. a buffer region). Then whenever the game needs to render the same scene, it just copies what it computed from that off-screen buffer region. If the game needs to change something on screen -- for example, because a lot is being redeveloped -- then that part of the screen is marked with a rectangle to indicate to the graphics engine that it's dirty (hence the name): And then only what's in those rectangles is re-drawn, and the result gets merged with that saved image: The same thing happens with scrolling: This isn't as effective if the player is using the minimap to jump around parts of the city, if they're zooming and unzooming, or if they're scrolling very fast, since those actions cause almost everything to be redrawn anyway, which is why you might see severe frame drops when scrolling. But this helps a lot with the typical case of "just looking at a city and planning your next move," and that's all it really needs to do. As far as SCGL is concerned, this doesn't require a lot of implementation work. The graphics engine takes care of calculating, drawing, and merging dirty rectangles further up the chain of command. All our little driver has to tell it is that we have support for OpenGL buffer regions and expose the graphics API for it. This has sort of been supported in SCGL using an obsolete OpenGL extension that only NVIDIA supports now. Just to dip my toes back into this, I replaced that implementation with a better-supported one that should work on Intel and AMD too. Next step: the core rendering loop is still kinda busted, but I did narrow it down to when certain props are on screen, so I have some hope that it's debuggable.

Status report Core renderer Textures are now broken in a new and exciting way! We've managed to leapfrog Maxis by getting terrain to sometimes render correctly: This is a totally cherry-picked screenshot though, and even then you can notice the geological textures on the edges aren't drawn right. If I scroll further to the left, a lot of stuff ends up rendered in the wrong place: So the goal for the next update is fixing that. Vertex buffers? You probably noticed those screenshots say we're chugging at 15fps, which is not great. There's a potential optimization here by implementing the game's vertex buffer extension. Here's the basic concept: it's not optimal for a game to send a large amount of data from the CPU to the GPU every frame. What you can do instead is carve out memory on the GPU, send the geometry data once to fill that buffer, and then keep referencing that buffer whenever you need to draw that mesh. So, I experimented with implementing the game's vertex buffer extension, and it went very poorly. SC4 doesn't take advantage the "write-once, use-many" philosophy; instead, it will still send the same terrain data to the GPU on every frame, but now it does it with a vertex buffer instead of using system memory. Like @rsc204 said, there's an upper limit to how much you can improve the graphics engine from the level we're working at. It's kind of curious since the DirectX renderer handles this fine, but in OpenGL on my machine, the framerate drops to a handful of frames per second and the memory usage skyrockets to >1.5GB. There's probably a bug that I'm not picking up on yet, so I'm setting that aside for now. Screenshots don't work, but maybe they will soon The in-game screenshot tool doesn't work and if you try to use it, the game crashes. But I saw that someone -- probably one of you -- forked my code to start working on the snapshot functionality, which is pretty cool! I'm looking forward to getting a pull request soon, or at least merging those changes myself back upstream.

Textures are working now - sort of. Everything is tinted with a blue-green gradient and city view is still a disaster, especially when you zoom out, but I'll take the wins where I can find them. Latest code is on GitHub.

The Origin copy of SimCity 4 should only be unpatchable if you buy a new copy. If you contact EA Support to redeem your CD key for a digital copy, you should get a patched version, so it may be worth a shot. And if the digital version you get turns out to still be unpatched, you'll have lost a bit of time talking to customer support, but otherwise no harm done.