[SOLVED] Extreme slowdown when loading cities

[carlosmarcelo 974]

Hey guys!

I finally got some time to talk about what I started in this topic as a quick question.
As I didn't want to pollute that topic by talking about the same subject, I decided to open a new one, believing that my experience can be useful for those who face the same obstacle.

 

1) THE PROBLEM

For everyday life, I use an old notebook, from 2011, with Windows 10 Pro updated, with SSD and memory to spare. I have SC4 from GOG.

Due to various factors, the last time I played with my plugins folder (which is approximately 1.5GB) was in November 2020; although it took a few minutes, it was tolerable time. There were a few times that I opened the game to test something specific, but at this point I removed my original folder to have no interference and the plugin under test to be isolated.

Finally, last month I had a break to go back to playing with my plugins folder and the game simply wouldn't load any city: it took about 2 minutes to go to the region and then it was indefinitely trying to open a city, even if she was small and empty; I even left about 10 minutes loading without success. I couldn't understand the cause if I hadn't made any changes to the game, much less to the plugins.

 

2) THE INITIAL ACTIONS

I created a zero region with half a dozen spaces, but only in one I created and saved an empty city with the plugins folder zeroed.

a. I removed the plugins folder and opened the game: it opened in less than 10 seconds.

Here comes the challenge: figuring out which plugin was the problem.
But I still had a question: if the folder is exactly the same, why did any item become a problem? Something changed and I didn't notice. Is it a corrupt file?

I confess that I was too lazy to look among almost 6 thousand files who was the villain,  so I decided to try other things.

b. I rescued a backup I had from the plugins folder.
Same result: extreme slowness.

c. I reinstalled the game.
Same result: extreme slowness.

d. I ran an SSD checkup tool (Kingston SSD Manager and HD Tune).
Everything is OK.

e. Defragmented the SSD with Defraggler.
Same result: extreme slowness.

f. I used DatPacker to reduce the plugins to just a dozen files.
Same result: extreme slowness.

Well, after that, there was no other way but to start plucking my plugins folder.

g. I started by removing the dlls, then the tree controller, the larger plugins and folders with more files.
After removing almost everything, it became possible at least for the game to load the city, but even so for an absurd time compared to the amount of plugins in question.

Conclusion so far: indeed some plugin is causing the problem of slowness, but I still didn't believe it was a plugin that, out of nowhere, became a problem. I was suspicious of this being a consequence rather than a cause.

That's when I came to the forum, to the quick questions thread, and some people helped me by explaining how SC4 loads. The most relevant information was that the NAM RULs, which are in "NetworkAddonMod_Controller.dat" are really slow to load.

h. I left only this file in the plugins folder, which in my case was very small, with less than 20MB in size, from version 36.
It took approximately 100 seconds to load an empty city. That is, a still high time for a very small file.

i. So I did something not recommended, but I already had no options: I removed the antivirus from the computer (Kaspersky).
Charging time dropped to 50 seconds. I was already getting happy with the result, but surely when I put all the plugins back in the time would probably jump up again.

Then I installed another antivirus (bitdefender) and the loading time did not change. But this test step made me convinced that something external was directly interfering with the loading of the plugins.

j. I installed SC4 on another computer and take the NetworkAddonMod_Controller.dat there.
The city loaded in a few seconds, if I'm not mistaken in less than 20 seconds.
Well, it was a computer with processing power almost twice as much as mine, but it seemed like a good sign. However, to proceed I needed to take the performance factor out by putting it on a machine with less than or equal performance to mine.

k. I installed a virtual machine from microsoft (Hyper-V) on my machine and downloaded an image of windows 10 also provided by microsoft. Evidently within the virtual machine the CPU, memory and disk resources are fractions of my machine. I installed SC4 on it and put NetworkAddonMod_Controller.dat there.
Surprising: even with fewer resources, the city loaded in less than 20 seconds.

Bottom line: something on my computer, but specifically on my OS setup was directly interfering with the loading of RULs.

In the end, using Process Explorer, I managed to generate these 2 graphics that illustrated the 2 scenarios.

In this image, we have the load of the city as it was on my computer. Note that the in the blue graph is as if there is a ceiling (marked with a red line) limiting the reading (I/O) for no reason and slowing down loading. 

In this image another image, we have the loading of the city as it was in the virtual machine. Note that the blue graphic is short and sharp, that is, the CPU reads as much as it can from the SSD and quickly solves what it needs, speeding up loading.

If I was too lazy to look for the problematic plugin, now I have a more tedious task: looking among more than 180 windows processes (services and apps), which one is impacting the loading of cities. 

l. I started the OS in safe mode, opened SC4 and loaded a city, with only NetworkAddonMod_Controller.dat of less than 20MB.
To my amazement, the city loaded in less than 20 seconds.

 

3) THE SOLUTION

Process Explorer is able to export the process listing so I generated a list of the virtual OS and one of the real OS (host; my machine). Comparing the 2, I came to a difference of some processes. After several eliminations and tests, in the end there were 2 difference processes that they had in the real OS and didn't have in the virtual OS and that I had never seen in my life: "LsaIso.exe" (Credential Guard & Key Guard) and "Secure System"

As far as I've researched, these items are special security components of Windows 10 Enterprise version and Windows 2016 Server. According to the link: "Windows Defender Credential Guard uses virtualization-based security to isolate secrets so that only privileged system software can access them. Unauthorized access to these secrets can lead to credential theft attacks such as Pass-the-Hash or Pass-The-Ticket attacks. Windows Defender Credential Guard prevents these attacks by protecting NTLM password hashes."

Ok, I understand that it is a security component of an OS that is not mine and that I have no idea how it was installed here... 

Well, the step now is to uninstall the device. Searching on Google I found the following link (it's in Portuguese, but it has an English version on it):

In it, it is taught how to enable. I used this information to find what was enabled to disable, involving registry changes.

I apologize for not specifying here exactly which windows registry lines I have changed as this may change over time as we have OS updates, ideally follow the official Microsoft link. Also, I didn't have to see all of that: for some unknown reason, my "Windows Defender Credential Guard" was half installed, meaning not all the requirements indicated in that doc were enabled.

Anyway, after I made the changes and booted, the processes "LsaIso.exe" and "Secure System" no longer appeared. So I loaded the SC4 and... HAPPINESS!!! It finally loaded as it should. 

After I put all my plugins back in the load time was like 2 and a half minutes, more than reasonable time for more than 1.5GB of plugins. 

 

4) FINAL CONSIDERATIONS

It was a long story but I made sure to tell because I know it's a recurrent problem that we have "slow loading" of the game, even with more modern computers.

I found it interesting to report a sequence of basic tests and observations that users in general can make to try to map the problem themselves (item 2 - initial actions).

Another point of the text is to make us realize that for the level of public knowledge that the community has about the game today, after almost 20 years of exploring all its minutiae, we will hardly have a NEW problem directly caused by the game, which could be a problem of machine or OS, as was my case.

Hope the text can be helpful. If anyone managed to read this far, thank you for your patience.   If anyone wants to exchange any ideas why this OS component caused this delay with the RULs or any other issue, the topic is open. 

[CorinaMarie 29,578]

Wow, Carlos!

Thanks so much for posting the intricate details of your investigation.

[rsc204 21,807]

Overall I think you can sum things up here as the problem being caused by other software slowing all operations on your PC. You probably didn't notice it opening your web browser or other applications, but when you load a game like SC4 with a lot of mods, the effects are suddenly much more obvious.

14 hours ago, carlosmarcelo said:

Another point of the text is to make us realize that for the level of public knowledge that the community has about the game today, after almost 20 years of exploring all its minutiae, we will hardly have a NEW problem directly caused by the game, which could be a problem of machine or OS, as was my case.

Absolutely, we can say with some certainty what problems might arise, because the code of the game hasn't changed in any tangible way provided you've got the official patches. Likewise, even as newer OSs come along, it quickly becomes apparent from community feedback/reports, if there is some inherent incompatibility or problem that stems from newer editions of Windows.

So as you've found out, issues like this tend to be the result of something that is unique to your computer or setup and probably nothing directly related to SC4 or related mods.

I wouldn't rule out your Anti-Virus software as having been a part of your problem, if it was scanning those files as SC4 tried to access them, that would certainly slow things down. Ideally you want to whitelist or otherwise get the AV to ignore both the game's installed files and the Plugins/Regions folders.

14 hours ago, carlosmarcelo said:

Anyway, after I made the changes and booted, the processes "LsaIso.exe" and "Secure System" no longer appeared.

So again, here we see the effects of security software, slowing things down because security checks take time to process. It's a trade-off you'd probably accept for higher security if you were running a business, but it's really not necessary for normal consumer use IMO. The real question becomes, why was this enabled?, the answer is either a user of the PC or some software you installed has turned this feature on, when by default it is off. It wouldn't surprise me to find that your security software did this as part of securing your system.

So whilst security software is often a good first place to look in these cases, booting into Safe Mode generally helps to quickly see if a problem is the result of some other installed/running app. Using a VM to compare a working/non working setup is a very good tip here too.

14 hours ago, carlosmarcelo said:

e. Defragmented the SSD with Defraggler.

Whilst I feel the need to point out that Defragmenting any type of drive in 2021 is a pointless endeavour, since Windows XP the OS automatically takes care of this, unless you've overridden the default behaviour. However it is specifically recommended to never defragment an SSD or other NAND based drive. For starters, there is nothing to optimise anyway, the whole point of defragmenting a drive is to move all the data that's used together into the same physical area of the disc. By organising the data in this way, much like the track ordering of a music CD, the computer can 'play' it without having to constantly seek (spin the disc so another location can be accessed). Solid State drives simply don't function like this, they never need to seek data, which is just one reason why they are so much faster. Instead everything is indexed by the controller, so any byte of data, regardless of where it is, can be accessed just as quickly as any other byte. Frankly I'm surprised Windows would even allow you to defrag an SSD, it probably didn't actually do it. But re-organising all the data on an SSD, especially if done regularly, would take a large toll on the drive's life expectancy and have zero benefits.

The only thing you really need worry about with SSDs is running TRIM commands and occasionally clearing the drive's cache, the former should be handled automatically by your OS, SSD manufacturer's utilities typically include such features though.

14 hours ago, carlosmarcelo said:

The most relevant information was that the NAM RULs, which are in "NetworkAddonMod_Controller.dat" are really slow to load.

It would be more accurate to say that the NAM Controller can be very large and that this takes longer to load as a result. It's not inherently slower to load than anything else, but if you have the full controller which I think is around 5 million new lines of RUL code now, loading that from the drive into RAM is going to take a while. The full NAM controller probably adds 1.5m to 2m to the load time of the game, but it only needs to load once for each time you start it.

[carlosmarcelo 974]

1 hour ago, rsc204 said:

I wouldn't rule out your Anti-Virus software as having been a part of your problem,

I didn't rule it out, he was certainly making things worse. 
Usually when I have problems like this, I disable the antivirus to help with investigations. I didn't report it up there, but when I did, the extreme slowness problem remained. Only when I UNINSTALLED it did I get the 50% gain I reported. So I just really wanted an excuse to switch to bitdefender which luckily worked. 
Until last year I used Norton (Symantec), but I decided to switch to a quality free one and I was recommended these 2: bitdefender and kaspersky. The last one proved to be a little more robust than I wanted, but I kept it on my computer, while in the others here at home I adopted bitdefender. 

Another test I did that I didn't think it was relevant to mention because it was something very particular was returning with the processor's hyperthreading (HT). I had turned it off believing that this would give more performance for older games, but after some measurements I didn't feel relevant gains (they were less than 5 seconds).
When I re-enabled HT, the computer as a whole had an extraordinary gain. Added to the antivirus replacement, without a doubt, my computer as a whole is much nicer to use, even before I remove Windows Defender Credential Guard.

 

1 hour ago, rsc204 said:

The real question becomes, why was this enabled?

After the problem was fixed, I did a retrospective of activities on the computer and noticed that I had installed several HD recovery SWs. I believe that some of them (for some reason I don't know) incompletely and NUTLY enabled the Windows Defender Credential Guard because there were several missing registry keys that I needed to create to assign the disabled value.
The antivirus certainly weren't because on the other computer where I installed the 2 antiviruses (one at a time), this additional security feature was not enabled.
I don't remember any other relevant computer changes during this period.

 

1 hour ago, rsc204 said:

Whilst I feel the need to point out that Defragmenting any type of drive in 2021 is a pointless endeavour

Here I will partially disagree with you. Undoubtedly, doing many and excessive defragments on the SSD reduces its useful life, so it should only be used in VERY EXTREME situations and with specialized apps, preferably from the manufacturer. I didn't use the Windows one (which in fact is useless), the one I was recommended is Defraggler.

My daughter works with video editing on an excellent performance computer. Recently she was having difficulties editing a video using a 2GB video as a source, with slowness and crashes... I installed this app and had it analyzed: the file was divided into more than 2000 fragments and the free space was also very fragmented. I ran defrag and the 2GB file was intact and solid, and the free space was now a giant airstrip. After that, the slow issues when editing the video ended and it was no longer necessary to run defragmentation.

The conclusion I came to from your comment and my experiences is that it is a possible solution for extreme and large-scale situations, which was not the case with my problem of slowness in SC4, but I wanted to try anyway as the damage would be too much low.

 

1 hour ago, rsc204 said:

It would be more accurate to say that the NAM Controller can be very large and that this takes longer to load as a result.

A curious moment that I didn't mention, at the beginning of the tests, is that just by removing the NAM controller, the loading time already improved a lot, even the other files having much more than 12MB, which was the size of my NAM controller .

After everything was resolved, I installed NAM 42 full, whose controller is over 300MB and it took about 3 and a half minutes to load only the controller. Luckily I don't like RHW and other stuff 😅 so my controller today is around 50MB. Making it clear: all NAM has is SPECTACULAR, it's just a matter of personal taste. 

Thanks for the help there in the previous topic and for the comments here.

Now I'm still intrigued by this: what exactly are RULs "threatening and dangerous" that would require a detailed check of a security component that other files (plugins) don't have? 

[CorinaMarie 29,578]

3 hours ago, carlosmarcelo said:

Now I'm still intrigued by this: what exactly are RULs "threatening and dangerous" that would require a detailed check of a security component that other files (plugins) don't have?

I would suspect nothing inherently threatening or dangerous, but the scanner doesn't know that and so feels the need to check each and every one as they load. So the simple act of it doing that 5 million times is the cause of the slowdown rather than any need to avert disaster.

I believe it'd be similar to downloading a huge compressed file and scanning it. While we both already know it's totally safe, the AV doesn't and so will take minutes (or hours) to check it out.

[rsc204 21,807]

21 hours ago, carlosmarcelo said:

what exactly are RULs "threatening and dangerous" that would require a detailed check of a security component that other files (plugins) don't have? 

To answer that you'd need to know why your security software deemed it a potential threat. I wouldn't assume either that the other plugins files were not checked. If it takes your computer say 40 seconds to load 1 million lines of RUL code into memory, which is simply data being loaded, it's logical to suggest that when anther application analysed that same data, it would take about twice as long. In reality the game has to wait for the scan to authorise it before it would be given access to the file, so essentially it's similar to loading the same data twice over.

There is nothing special about RUL code, it's just a bunch of data, but it is a lot of very dense data compressed into a file. The real question is why does YOUR computer feel such a file needs checking, because this is not the default behaviour of a Windows 10 system. Normally I'd suggest the fact that such a file/operation is not hugely common in the general world of computing, means your security software won't assume it's safe, except it wants to scan it every time it's accessed. That tells me that your security software and perhaps in particular the settings used, have made such files something that require validation at all times It's unlikely that the same wouldn't have also applied to the other Plugins, but being much less dense files, you just don't notice the speed difference like you do with a single large file like the NAM controller. Either way though, once those security restrictions are removed, everything works as you would normally expect.

21 hours ago, carlosmarcelo said:

I ran defrag and the 2GB file was intact and solid, and the free space was now a giant airstrip.

Are you certain you were actually defragging the drive, because it's more likely the software was just initiating the TRIM command or some other drive-based performance optimisation. This can be necessary if NAND drives become slow, but are best handled by the manufactures utilities, for example Intel's SSD Toolbox or Samsung's Magician etc. An actual defrag does not fix anything on an SSD and simply reduces it's lifespan for no gain. Now, which method the maker of a software utility use to visually represent what's going on, doesn't necessarily mean that's what has actually happened at the level of the individual NAND cells. I suspect the term defrag is familiar and so they used it, but it's not technically correct, as I mentioned before there is no seek time on an SSD, it doesn't matter where that data is:

Quote

However, this isn't an issue with SSDs because the seek time are about 0.1ms. You won’t really notice the benefit of defragged files — which means there is no performance advantage to defragging an SSD.

This quote is from Crucial, an SSD manufacturer, however you'll find plenty more examples, all of which state the same thing from those who actually make the hardware. Likewise, this article does a very good job of explaining why defragging is not a good idea, although it get's into the nuance of what people might refer to as defragging, when for SSDs it's actually something different, even if the end results might appear to be similar.

[twalsh102 1,601]

I think the best advice, rather than trying to overthink what might be causing the issues, (if this hasn't already been mentioned) would be to just exempt the entire NAM folder from all future scans. There does come a point of diminishing returns when the expenditure of time and effort trying to figure out the "why's" of an issue, end up being all out of proportion to the possible returns, especially when a simple, safe work-around fixes the problem.