Let's write our own SC4 hardware renderer

[simmaster07 1,220]

What's the deal with hardware rendering? If it's so hard, why don't they make easyware instead?

20 years after its release, SimCity 4 has aged remarkably well; the graphics engine has not. (The graphics themselves are pretty good though.) The Graphics Rules file has no understanding of newer GPUs and has a bad habit of picking sub-optimal graphics settings. The DirectX driver was effectively obsolete by the time the game came out; errors like "Could not initialize DirectDraw" have randomly come up since release. And if you get that error, your framerate falls off a cliff as the game puts itself into software rendering mode. (We don't talk about the software renderer.)

There is a hidden option to use OpenGL instead of DirectX by passing "-d:OpenGL" in the command line options. This isn't viable either since everything falls apart after loading a city:

Sad to say this is not a snow mod.

What can be done? We could use a wrapper like DDrawCompat or dgVoodoo, and that works fairly well. But adding translation layers also means taking a performance hit, as documented in that thread. There is one other saving grace here, though, which you can probably guess from the thread title. If you know the magic words, the game can load a DLL that upgrades an existing component. So join me on an adventure as we try to write our own graphics driver! (Spoiler: we're not there yet.)

SimCity 4 has a couple of components to handle putting pixels on the screen. There's a GraphicsSystem that translates the game state to finer graphical commands, and a GDriver (graphics driver) that sends those commands to a graphics API, like DirectX or OpenGL. The GDriver interface is a target that the game calls SimGL. As the name implies, it's an abstract OpenGL 1.2 target, so the DirectX and Software drivers have to translate OpenGL-like commands to something that makes sense for those APIs (*).

Also like OpenGL, GDrivers support "extensions," which are optional but can be leveraged by the GraphicsSystem to improve performance or add features if they're available. For example, lighting support is an extension supported by OpenGL and DirectX, but not by Software.

If the abstract machine is based so heavily on OpenGL, then why is SC4's OpenGL support poor? I'm not totally sure, but I think there's a few issues that play a part:

1. A lot of SC4 textures (FSHs) are compressed using DXT/S3TC, and the game was written to target OpenGL 1.2.1 (as far as I can tell). DirectX standardized support for these textures, but texture compression wasn't a standard feature until OpenGL 1.3, and this specific format couldn't be built into OpenGL because of patent issues.
2. Vertex buffers, which would improve performance, were also not standard in OpenGL yet. The DirectX GDriver implements them, but OpenGL does not.
3. There's also an extension for "buffer regions" to boost performance which, again, is not standard in OpenGL. The game does try to provide them as an extension, but it seems this does not work for Intel graphics.

A lot of these features became standard or ubiquitous long after SimCity 4's development cycle, so I've made a project called SCGL (pronounced like "sigil") to rewrite the OpenGL GDriver and add these features. I picked OpenGL 3.0 since the game depends on a lot of features that are now deprecated. 3.0 was the last OpenGL version to offer these features for compatibility, and recent GPU drivers still have compatibility modes for it.

It's still very early days, but the good news is that I'm at the point where the game loads and we have something showing up on the screen.

If you can believe it, this is region view, and those green boxes are clouds.

Again, super early days. But it's a starting point for debugging, so the next immediate goal is to fix the texture corruption. Getting region view to be recognizable would be nice, but honestly I would settle for having the buildings in a city show up as untextured white boxes and iterating from there.

(*) This has some interesting implications for the Mac port. Since SC4's DirectX driver was the only stable hardware renderer, Aspyr had to write their own DirectX wrapper, even though Macs only supported OpenGL. So it has an OpenGL GPU pretending to be a DirectX card (Aspyr's wrapper) pretending to be an abstract OpenGL API (SimGL).

[Ulisse Wolf 4,398]

It's a very interesting project. The fact that this mod, although in the prototype stage, managed to show a rudimentary menu makes me understand that we are close to modernizing the graphic behavior of SC4 and solving all those graphical bugs that SC4 suffers if you do not use DirectX

[simmaster07 1,220]

Textures are working now - sort of. Everything is tinted with a blue-green gradient and city view is still a disaster, especially when you zoom out, but I'll take the wins where I can find them. Latest code is on GitHub.

[Turnstyled 485]

This looks great! What are the implications for SCGL and playing at higher resolutions, such as 4K? Will it be usable with a wrapper such as dgVoodoo2 or dxwrapper? Will resolutions such as 4K be playable with it without the use of wrappers? Thank you for your work!

Edit: My apologies. You state in the first post that a point of SCGL is to use OpenGL without the use of a wrapper. I'm still curious about my second question. Thank you again!

  Edited January 21, 2023 by Turnstyled  

[rsc204 21,804]

I would assume the intention would be to implement the entire game in OpenGL, i.e. not DirectX, hence no need for wrappers. I’ll note the resolution limitations for SC4 are a direct result of using DirectX 7, but of course similarly i assume there is a limit for OpenGL 3, although most likely higher in practise than the archaic DX7.

[Turnstyled 485]

8 minutes ago, rsc204 said:

I would assume the intention would be to implement the entire game in OpenGL, i.e. not DirectX, hence no need for wrappers. I’ll note the resolution limitations for SC4 are a direct result of using DirectX 7, but of course similarly i assume there is a limit for OpenGL 3, although most likely higher in practise than the archaic DX7.

I see. Your explanation makes sense. Thank you.

[TorchedSammy 7]

i imagine another use would be for linux players (like me) since that's again 1 less wrapper to use (dxvk) and that could possibly mean better performance.

either way, this is a very cool project I'd like to see complete! nice work

[Turnstyled 485]

4 minutes ago, TorchedSammy said:

i imagine another use would be for linux players (like me) since that's again 1 less wrapper to use (dxvk) and that could possibly mean better performance.

either way, this is a very cool project I'd like to see complete! nice work

That also makes complete sense! Gaming on Linux has come such a long way. I'd like to see this project completed as well!

[junspud 1,066]

I don't really have anything to contribute other than to say this looks like an awesome project and I'm excited to see what comes of it. I really wish an 'OpenTTD' style project of SC4 was possible but obviously that's a pipe-dream for a variety of reasons, not the least of which being the amount of work involved even if it was actually technically feasible.

But that fact this works at all so far is cool to see, the awful performance and compatibility of SC4 I imagine is a non-starter for a lot of people, I'm not even sure how it will run on my newer laptop once I get around to trying it, I'm not optimistic. Personally on my laptop, the graphics/performance is pretty bad, shadows in particular are a total disaster, the rendering of them is broken somehow. Trying to find a fix for whatever is happening with shadows is actually what led me to Simtropolis and into modding the game in the first place.

[Turnstyled 485]

2 hours ago, junspud said:

Personally on my laptop, the graphics/performance is pretty bad, shadows in particular are a total disaster, the rendering of them is broken somehow. Trying to find a fix for whatever is happening with shadows is actually what led me to Simtropolis and into modding the game in the first place.

I noticed that with the shadow rendering! I didn't know that it might've been because of a broken DirectX implementation.

Well, @TorchedSammy who posted in this thread is doing work on OpenSC4 that the previous two developers seemingly abandoned. I don't know much about coding, but it seems to be a different implementation than OpenTTD/OpenRCT2. The original idea was to create a clone of SimCity 4 in the Godot game engine. It's exciting to me!

Whenever something new is to release within an already-established game community, there's a lot of competition as to whether or not what will be new will replace what has already existed. I find that to be an exhausting waste of energy. I saw it plenty when I was following flight sims with the impending 2020 release of Microsoft Flight Simulator and I saw it plenty before the release of Cities: Skylines. Should OpenSC4 become a reality through the Godot implementation, then I'll be happy to see the SimCity 4 that we've known for twenty years continue to exist!

[TheMurderousCricket 5,042]

Despite the fact that the current stage of your project reminds me of the lost SimCity DLC named "Neil Fairbanks takes LSD", I am truly happy that someone has taken on the subject and made a move regarding the OpenGL (OpenGL-ish) rendering. This is one of the options I always wanted to try out and play my game with but for reasons obvious enough, never did.

Keeping my fingers crossed.  Thank you for your input. Do keep us updated and don't loose heart in case some problems appear! 

[rsc204 21,804]

Impressive little project, I'll be keeping an eye on this, could be a great improvement.

I have always considered that there is a limit performance-wise that we can get from SC4, regardless of hardware, which has always been especially true of the graphics. Whilst ridding folks of the need to mess around setting up their hardware, to just popping a simple .dll file into a folder would be a huge step in and of itself, I do wonder how far it's possible to improve performance, because I've generally had it good here. A patch that would allow say the ability to better handle shadows, so cities chock full of trees can show full shadows without killing performance/redraw times. Also removing lag from things like scrolling/zooming/rotating, or say the speed of refreshing from Subway/RCI views too, clearly that would benefit everyone because I don't know of any hardware that can keep up, especially as the city and region population grows and your Plugins folder expands.

I'm quite certain 15 year old GPUs already have more power than SC4 can handle, having experienced this first hand. Then, I suspect this isn't totally within the realm of the GPU, but that data through-put is limited by things other than the installed system's limitations (RAM/HDD/Bus Speeds). Likewise the killer is always the CPU usage with SC4, so it will be interesting to see what a working OpenGL implementation will be able to improve.

Obviously a tiny fish in the grand sea of things, but I couldn't help think how nice it would be if users of high resolutions (outside of the DX7 limits), could once again be able to show the correct screenshot in Region View. In case you were not aware, when saving with higher resolutions, the portions outside of the 2048x2048 imposed limit get cut off when the game save takes a 'screenshot'. Which rather messes the region view up, which can be a bit annoying.

On 23/01/2023 at 2:41 AM, Turnstyled said:

I noticed that with the shadow rendering! I didn't know that it might've been because of a broken DirectX implementation.

The implementation probably works, what it doesn't always play nice with is more modern versions of DirectX and therefore cards without the relevant legacy support. Essentially if your GPU has full DirectX 9.0c compatibility, that should also cover the legacy DirectX 7 requirements. However, if your card, driver or version of Windows which is very much tied to the version of DirectX you can use, don't fully support the DirectX 7 features needed, you may run into problems.

When I bought my current system in 2013, I originally intended to only use the included Intel HD 4600 GPU, it was after all powerful enough to run SC4 as well as any other game from that period. I quickly discovered that the drivers lacked support for certain areas, at that time the game was unplayable with it. I used an old Radeon 3870 I had around for a bit until settling on an NVidia GTX 750ti. In 2015 Intel updated the drivers and for the first time SC4 properly worked with DirectX rendering. At the time I did find performance wanting, but since I had already bought the NVidia card, I never worried about it. I ran some tests more recently to see if a Wrapper helps with Intel GPUs and the answer is a resounding yes. I'd imagine the same would be true for most Intel cards, but of course I can't test every configuration and I'm also using Win7 which isn't so common these days.

The situation is much the same with NVidia cards, it's very common these days that without a Wrapper they can't handle the legacy DX calls SC4 needs. I'm sure the situation with ATI cards isn't much different, this is just a sort of natural evolution problem for computers. Although most ATI cards get wrongly detected, it's only because around release but a handful of ATI cards worked with the game well.

Spoiler

# The Radeon series has a very slow path for partial
   # depth buffer copies. This has apparently been
   # fixed for 9500 and later models in the latest drivers,
   # but cannot be fixed on previous Radeons because of
   # hardware limitations.
   # NOTE: disabling this rule unnecessarily may lead to 
   # very poor graphics performance.

The problem with the solution to this problem, is when an ATI card is detected, that is not explicitly detailed in the Video Cards.sgr file, the game assumes it suffers this problem. As a result, it actually acts counter-productively effectively limiting the GPUs ability to work with the game. Of course your GPU won't be listed either, because the list hasn't been updated since the Radeon 9800 card! I believe there may have been another change to ATI cards that requires similar meddling to fix, something to do with the DirtyRectMergeFrames setting not being optimal, for anyone who knows what that does.

[simmaster07 1,220]

Status report

Core renderer
Textures are now broken in a new and exciting way! We've managed to leapfrog Maxis by getting terrain to sometimes render correctly:

This is a totally cherry-picked screenshot though, and even then you can notice the geological textures on the edges aren't drawn right. If I scroll further to the left, a lot of stuff ends up rendered in the wrong place:

So the goal for the next update is fixing that.

Vertex buffers?
You probably noticed those screenshots say we're chugging at 15fps, which is not great. There's a potential optimization here by implementing the game's vertex buffer extension. Here's the basic concept: it's not optimal for a game to send a large amount of data from the CPU to the GPU every frame. What you can do instead is carve out memory on the GPU, send the geometry data once to fill that buffer, and then keep referencing that buffer whenever you need to draw that mesh.

So, I experimented with implementing the game's vertex buffer extension, and it went very poorly. SC4 doesn't take advantage the "write-once, use-many" philosophy; instead, it will still send the same terrain data to the GPU on every frame, but now it does it with a vertex buffer instead of using system memory. Like @rsc204 said, there's an upper limit to how much you can improve the graphics engine from the level we're working at.

It's kind of curious since the DirectX renderer handles this fine, but in OpenGL on my machine, the framerate drops to a handful of frames per second and the memory usage skyrockets to >1.5GB. There's probably a bug that I'm not picking up on yet, so I'm setting that aside for now.

Screenshots don't work, but maybe they will soon
The in-game screenshot tool doesn't work and if you try to use it, the game crashes. But I saw that someone -- probably one of you -- forked my code to start working on the snapshot functionality, which is pretty cool! I'm looking forward to getting a pull request soon, or at least merging those changes myself back upstream.

[Turnstyled 485]

13 hours ago, rsc204 said:

The implementation probably works, what it doesn't always play nice with is more modern versions of DirectX and therefore cards without the relevant legacy support. Essentially if your GPU has full DirectX 9.0c compatibility, that should also cover the legacy DirectX 7 requirements. However, if your card, driver or version of Windows which is very much tied to the version of DirectX you can use, don't fully support the DirectX 7 features needed, you may run into problems.

That's pretty much what I had meant. I had meant that it probably had to do with the hardware and software being used. Thanks for the additional information!

This is seriously awesome work!

[Jake1702 5]

Just happened to stumble across this very interesting project. Everyone here is interested in possible performance improvements which will be great, but is it bad that the first major place my brain went to was how we could abuse utilize this to greatly improve visuals? (Far more than just resolution of course)

[TheMurderousCricket 5,042]

5 hours ago, Jake1702 said:

Just happened to stumble across this very interesting project. Everyone here is interested in possible performance improvements which will be great, but is it bad that the first major place my brain went to was how we could abuse utilize this to greatly improve visuals? (Far more than just resolution of course)

I think that you may want to focus your attention on SimCity Open (SCO) in this case.  It is a project which aims to improve certain visuals, graphical and ambient aspects of the game.

I'm saying this because from what I see @simmaster07's focus is to allow OpenGL rendering to make sense in SC and not redesign the game's content as such. I agree though that "SimmGL" might be an element of the SCO...

[Ulisse Wolf 4,398]

I'm curious about how this DLL Mod is progressing. But I would like to ask a question

Does this version of OpenGL support Reshade? Is it possible to separate UI graphics with game graphics to prevent Reshade from applying effects to the UI as well?

[Propfam 2,286]

On 26/1/2023 at 3:59 AM, rsc204 said:

certain 15 year old GPUs

I wonder those Kyro GPUs listed in the SGR. Like, it's made by PowerVR, the same manufacturer of iPhone/iPad GPU before iPhone 8. We could theoretically port SC4 to iOS, much like Maxis did with reskinned SC4, but with full-blown features. At least, before we stumble on Metal issues. 

[rsc204 21,804]

You can theoretically port any code to any system and make it work, provided you throw sufficient resources and expertise at the problem. But in respect of making SC4 work on an mobile phone or tablet my question would be, why would you want to?

Over the years I've had the original SimCity for the Psion Series 5 Organiser, actually surprisingly playable with a touchscreen (stylus) and keyboard. SimCity 2k on Windows Mobile devices, terrible even with a super hi-res (for such) 800x600 display and device with full keyboard. Literally every attempt I've seen to play a game like this on the go is either hideously compromised and simplistic or simply is unplayable for all but the most determined. If I really needed to play SC4 on the go, I'd just buy a laptop and be done with it, because such games never work on small devices.

[Turnstyled 485]

@rsc204 I get it. SimCity 4 on SteamOS on my Steam Deck installed through Lutris took forever to do anything. Thus, I thought I'd try it on my Windows partition on my Steam Deck. I encountered several problems.

1.) Setting SimCity 4 to a 16:10 resolution made things...weird. The cursor was warped, for one thing.

2.) Trying to bring up the Steam on-screen keyboard resulted in the game window losing focus. Bringing SimCity 4 back in focus resulted in text looking corrupted.

3.) I couldn't use Steam Input while running SimCity 4 as an administrator.

All of these problems would be fixed using a GPD Win 4, which I have been very excited for. The GPD Win 4 has a 1080p 16:9 display. It also has a tactile keyboard under the screen that you slide up when you want to use the keyboard. Additionally, while I don't know the technical details, the controls on my GPD Win 2 clamshell handheld don't require Steam Input and thus, they work just fine with programs running as an administrator. I assume that it's the same on the GPD Win 4. I take it that it has to do with controls being hardware-level on GPD devices. My GPD Win 2 has a physical switch to switch to XInput. I don't know if the GPD Win 4 has that switch.

Still, it wouldn't be a very pleasant experience playing SimCity 4 on a GPD Win 4. I like to play around with what is technically possible, even it isn't a pleasant, practical, or very playable experience.

 

[ASV7900 89]

Just to share my personal experience trying to Install SimCity 4 on two of my laptops after reformatting my old one, with the same installation methods for both:

1) Selenite X - Running on AMD processor + GTX 3080, Windows 11
= Cannot run on Hardware (DX) or OpenGL.

= Runs on software but extremely laggy & buggy.

= Gave-up and uninstalled the game.

2) Selenite Pro - Runnin on Intel processor + GTX2060, Windows 10

= Able to run on Hardware (DX).

= Smooth most of the time, unless running large cities that have populations more than 1 million.

= Getting back into playing the game again with CAM + NAM + plug-ins that I've accumulated over the past year.

 

As much as I love playing the game and the support community, I really hope there's a better City Builder that's up & coming to replace the best of the best, SC4.

[Propfam 2,286]

Can you make a mod that enables touchpad support in SimCity 4? I mean, this will be a huge W for laptop users since they won't need mouse anymore. You will able to plop buildings using left-click, scroll the view using 2 finger gestures, and zoom using pinch. Windows 10 and 11 has native support for this, so does Mac (tho the implementation is very different). Linux however will be left behind by this mod. Hopefully, there will be someone who will implement touchpad support for SimCity 4 on Linux.

And also, we need a mod that enables NVMe loading system. This will end the 1 minute mark start time for NVMe users. SimCity 4 loading is unacceptable these days. Even apps such as AutoCAD, Solidworks, and even triple-A games can be loaded under 20 sec, at worst, even on worst of NVMe drives.

[Ulisse Wolf 4,398]

12 hours ago, elfrjz said:

Can you make a mod that enables touchpad support in SimCity 4? I mean, this will be a huge W for laptop users since they won't need mouse anymore. You will able to plop buildings using left-click, scroll the view using 2 finger gestures, and zoom using pinch. Windows 10 and 11 has native support for this, so does Mac (tho the implementation is very different). Linux however will be left behind by this mod. Hopefully, there will be someone who will implement touchpad support for SimCity 4 on Linux.

And also, we need a mod that enables NVMe loading system. This will end the 1 minute mark start time for NVMe users. SimCity 4 loading is unacceptable these days. Even apps such as AutoCAD, Solidworks, and even triple-A games can be loaded under 20 sec, at worst, even on worst of NVMe drives.

Although these things are great it would make more sense to get all the source code of SC4. At least we know how SC4 works and consequently rewrite SC4 to make it work on modern computers.

[Propfam 2,286]

55 minutes ago, Ulisse Wolf said:

it would make more sense to get all the source code of SC4.

Good luck with that without reverse, you know, of the logic of SC4. Maxis won't release their source anytime soon.

[rsc204 21,804]

What you are talking about is more of a driver issue than something wrong with SC4. If your trackpad was sending the correct commands (via whatever is used to convert them to Windows standards), then SC4 would play nice with it. So either your drivers need updating or the software that allows Windows code to run under Linux needs to support this feature.

But ultimately the game’s code (Windows) does support these functions and it is the system that needs to parse what you do to the correct DirectX commands to make it happen. Could you imagine if every game engine needed to be coded to support every single mouse out there? That’s exactly why DirectX exists, manufacturers make DX compatible drivers, the DirectX API is implemented in the game code and then it all just works. Of course backwards compatibility may well be an issue, but that all comes from whatever is passing these commands through to DirectX under Linux.

[Propfam 2,286]

7 minutes ago, rsc204 said:

What you are talking about is more of a driver issue than something wrong with SC4. If your trackpad was sending the correct commands (via whatever is used to convert them to Windows standards), then SC4 would play nice with it. So either your drivers need updating or the software that allows Windows code to run under Linux needs to support this feature.

But ultimately the game’s code (Windows) does support these functions and it is the system that needs to parse what you do to the correct DirectX commands to make it happen. Could you imagine if every game engine needed to be coded to support every single mouse out there? That’s exactly why DirectX exists, manufacturers make DX compatible drivers, the DirectX API is implemented in the game code and then it all just works. Of course backwards compatibility may well be an issue, but that all comes from whatever is passing these commands through to DirectX under Linux.

No, I mean something like the Photos app in macOS. You can scroll the pictures using touchpad gesture. Same with panning camera in SimCity 4.

[rsc204 21,804]

Right, but that's also pretty specific requiring a code rewrite to add, I mean is holding the right mouse button whilst using the trackpad so hard? I mean ultimately it seems like a huge effort to solve something that's at worst a minor annoyance? Bear in mind too, even most laptop users use a proper mouse whenever it's convenient to do so.

[ASV7900 89]

4 hours ago, rsc204 said:

Right, but that's also pretty specific requiring a code rewrite to add, I mean is holding the right mouse button whilst using the trackpad so hard? I mean ultimately it seems like a huge effort to solve something that's at worst a minor annoyance? Bear in mind too, even most laptop users use a proper mouse whenever it's convenient to do so.

Yup! I love using me mouse instead of the damn trackpad...

[Ulisse Wolf 4,398]

Nvidia has just released RTX Remix that allows you to create mods that introduce new RTX technologies such as DLSS 3 and Ray Tracking.

https://github.com/NVIDIAGameWorks/rtx-remix

Also I'm noticing irregularities with the rendering software and Direct X (Using the Wrapper) with the night lights where DirectX tends to apply an explosive light effect. I wonder what night vision will be like with OpenGL?

[Apothocary 21]

This is a fantastic project. I'll be watching closely!

After my re-entry to SimCity 4 recently, it was a little turbulent getting it to play nice with my overly powerful RTX4090.