Speeding up Game loading times

[DavidDHetzel 1,816]

I'm getting rather annoyed by the game taking 5+ minutes to load and I sware I can remember there was a way to fix this, I think was some sort of program BSC made called "datpacker". But I am not 100% sure.

 

Does anyone know how I can improve game loading speeds? I don't want to wait over 10 minutes to load my city anymore.

[rsc204 21,808]

There are a number of worthwhile steps you can take to help improve loading times.

• Remove any non-SC4 files from your plugins folder. I.e. readmes, images, .zip/.exe files etc.
• DAT pack your plugins folder.
• Clean up your plugins folder.

The first you should do if you have not already, such non game files can cause problems. The second is optional, DAT Packer is available here, but remember to keep backups of your non dat-packed files.

The last option is only useful if you've got duplicate files hanging around. It's all too easy to end up with duplicates, so if it's not organised, give it a sprucing up.

[A Nonny Moose 8,261]

Remember that DATpacker should not be used on volatile plugins such as the NAM.  Such should be held out of the packing.  It is probably not very good to pack all your plugins into one huge *.dat file because it makes maintenance rather tedious.

There is also the Files2DAT program by ilive which can handily pack a few files up, and this helps considerably if you don't want to go whole hog with DATpacker.  Not much space saving, but does speed up loading.

[nos.17 5,120]

I definitely agree with what rsc mentioned above make sure your plugins folder is clear of non-needed files. Removing those alone could save a minute or two (assuming you have them in there, at least). 

Dat-packing will do wonders also. It works best if you have some sort of organized structure in your plugins folder, Furthermore, make extra sure you have EVERYTHING backed up before you pack. It is a one-way process and cannot reasonably be undone if you wish to add files. To add items, delete whichever old files are no longer required, copy from your backups to plugins and repack.

 To add to that, I have been dat-packing the NAM for awhile now and have rarely had issues; you should however, be careful and test things before you go forth...it may or may not have catastrophic consequences. If you get it to work, the risk is definitely worth the reward: even more time saved (on the magnitude of 30 sec to 1 min depending).

[DavidDHetzel 1,816]

1 hour ago, nos.17 said:

Dat-packing will do wonders also. It works best if you have some sort of organized structure in your plugins folder,

There in is the problem, I don't have an organized plugin folder -_-

[A Nonny Moose 8,261]

1 hour ago, DavidDHetzel said:

There in is the problem, I don't have an organized plugin folder -_-

Well, if you are going to clean up non-plugin files, this is also a good time to spend a day getting things organized.  Subfolders are not hard to make.  Pick a long weekend or something, and just do it, or do it over time.  As you complete each section, you can Datpack it (keeping a backup) and start to realize the loading advantages.

If it helps you at all, I keep mine by either author or working group.  Sometimes author within working group.  Here is an example of the head of my current plugins:

SimCity 4/Plugins
├── Aaron Graham
│   └── Aronic Place
│       └── AronicPlace.dat
├── Airman15
│   └── steamshipsPack
│       ├── SS Hornet V2.dat
│       ├── SS Karma V2.dat
│       ├── SS Union V2.dat
│       └── SS Warrior II V2.dat
├── Andisart
│   └── AndisArt - SC13-style_homes MAXISNITE
│       ├── AA.dat
│       ├── Colorfuls
│       │   └── Colorfuls.dat
│       └── Neutrals
│           └── Neutrals.dat
├── Andreas Roth
│   └── CityHall_fix.dat
├── A Nonny Moose
│   ├── Anti-Matter Power Plant
│   │   └── Anti-matterPowerPlant.dat
│   ├── Artesian Water Collector
│   │   └── ArtesianWater.dat

My loading time is somewhere less than a minute and I have 851 directories, 3859 files, lean and mean plugins only.  If I could DATpack this it would be smaller.  Unfortunately there is a problem with Linux/wine.

[rsc204 21,808]

I'll be honest, my plugins folders (I have many) are all a shambolic mess for the most part. I don't DAT pack much (all done manually via the reader when I do). I've managed to keep 4gb or so running smoothly without tedious loading times. It helps I'm running the bulk of this off an SSD though, moving that to my HDD slows things considerably. One of the issues with modding a lot, things are always changing :/.

There is no problem theoretically with DAT Packing the NAM, two things to note about this though. The first is because the NAM has a specific loading order, chucking everything into a couple DATs can easily mess with this. Secondly the NAM is one of those mods you can reinstall again and again. A lot of work went into one of the best installers the game has, it manages a massively complex install for you. If you DAT pack it, then the NAM installer may not be able to function as intended. Generally speaking we advise against it, because for many users it would cause more harm than good. Whatever you do, always return to a non dat-packed NAM before running the installer. And overall, proceed with caution if you are set on doing this.

From the sounds of things you need to have a cleanup of your plugins folder. That should help get things working smoother, I would wait and see how this improves things before thinking of DAT packing anything.

 

[MushyMushy 3,617]

Organizing and weeding out unnecessary files certainly helps a lot. I noticed a significant increase in the game startup speed after I removed all of the images and readme files. I can't remember though, do the specs any of the computer's hardware (other than the difference between HDD/SSD) play a part in this? Or does that only affect the maximum amount of mods it can load without crashing?

When I installed the game on my new desktop and copied over my plugins folder, the game booted up noticeably faster. It's still running off a HDD, but the new computer has far more RAM and a much more powerful processor than my old laptop did.

[DavidDHetzel 1,816]

3 hours ago, rsc204 said:

I'll be honest, my plugins folders (I have many) are all a shambolic mess for the most part. I don't DAT pack much (all done manually via the reader when I do). I've managed to keep 4gb or so running smoothly without tedious loading times. It helps I'm running the bulk of this off an SSD though, moving that to my HDD slows things considerably. One of the issues with modding a lot, things are always changing :/.

My problem is I have 12gbs of plugins and that's AFTER removing the uneeded files.

[rsc204 21,808]

Then you have ask yourself a question, it is worth spending money on an SSD to improve things? Because one way or another, every time you start the game it needs to index 12GB data. Every wonder why it's slower after a reboot to start up SC4? That's the game indexing your plugins. Even when you close SC4, it retains some of this data until the computer is shut down.

Personally, I'd expect a significant improvement with a $100 investment here. Not to mention you'll never go back to running Windows on regular hard drives once you've installed one. Hard drives are in the dark ages compared to the CPU, Memory and GPU speeds we have come to expect. On most systems data access is the bottleneck, SSDs really help here.

[DavidDHetzel 1,816]

I have one and that's where I ended up putting the junk stuff.

[rsc204 21,808]

Sorry, are you saying you've put the archived files on your fast storage? Whilst leaving the files you use everyday on the slow disc-based drives?

That would be like taking your family station wagon to the race circuit and using the Ferrari to go shopping at the supermarket.

You can actually tell the game where you want to put your plugins folder. So if having more than one drive means by default it's not on the SSD, it's a simple fix. But really the point of an SSD is to use it to run the operating system and your most-used applications. SSDs are really poor when it comes to data protection, but really good with fast frequent access. Your hard drive is the opposite and makes a much better place for storing the important stuff like your personal files.

[DavidDHetzel 1,816]

I'm not really tracking with what you said unfortunately. Are you saying my entire plugins should be in the SSD rather than on my computers hardrive?  Or that if I use Datpacker I should have the plugins backedup on the harddrive?

[rsc204 21,808]

I would put the plugins onto the SSD, yes. Is Windows running from the SSD, if it's not, why is that?

You should also backup any plugins folders if you DAT pack them. For now see how much quicker the SSD is, if you decide to dat pack things you may get a further improvement here, but the SSD will likely be the biggest gain here.

[A Nonny Moose 8,261]

Maybe it would be clearer if you got a handle on what happens on an HDD vs. an SSD.  Each time the system goes for a file on an HDD there are two commands:  Seek, then Read.  The seek moves the access arm with the read/write heads over the requested track, then the read fires to read in the necessary data.  There is a chance that the read will be held up, on the average, for one half of the rotation time of the disk.  This is quick, but in terms of computer time it is just ages and ages.

An SSD doesn't have the rotation time (latency) to deal with and when the read goes in, transfer starts immediately.  The saving can be only a few milliseconds, but over a game session this can add up to minutes and hours.

It is not a bad idea to look at an SSD as a read-mostly device.  Files like plugins work very well there as do system residency files such as the executable files in the system as these are usually never changed.  While the technology is very good now with SSDs, they do have a shorter write count limit (mean time between failures) than rotating storage because the disks are simply more robust.

From a production point of view, keep backups and files you write to often (regions) on disk, and the files that are read a lot on the SSD.  You don't need to go overboard on this, but just a little attention to these principles will make a real difference.

Now, about that huge set of plugins:  How many of them are hit regularly?  If you have some you never use, and you can identify them, I'd send them to an archive just to get them out of my maintenance system.  Maintenance is a headache at best, and keeping unused stuff around is a muggs game.

[rsc204 21,808]

11 hours ago, A Nonny Moose said:

Now, about that huge set of plugins:  How many of them are hit regularly?  If you have some you never use, and you can identify them, I'd send them to an archive just to get them out of my maintenance system.  Maintenance is a headache at best, and keeping unused stuff around is a muggs game.

Agreed this is sound in principle. But don't forget that saving your game whilst an object contained in the city does not exist in your plugins folder can wreak havoc, that's an absolute no-no.

[DavidDHetzel 1,816]

so what you guys are saying is, that I should move my plugins folder entirely from the SC4 Folder in my hardrive to my SSD? I'm sorry I'm not tracking with your guys computer lingo (Who'd have thought given I went to college for Web Design). 

[A Nonny Moose 8,261]

In a word, yes.

and BTW, Web Design is a limited subset of computer programming.  How's your XML?

[DavidDHetzel 1,816]

So 6 months later and I've finally gotten around to moving my entire 13gb's of plugin from my documents library to my SSD. I took me about 15 minutes which happens to also be the average time it took to load the game up

[CorinaMarie 29,605]

On 1/27/2016 at 6:02 PM, rsc204 said:

There is no problem theoretically with DAT Packing the NAM

If I do DAT Pack NAM can I still tweak settings with the TSCT or do I need to do that only without it packed? (I'm cool with reverting to unpacked for the other parts you mentioned.) And, can I DAT Pack NAM by itself? I can move all other plugins out first if that's the way to do it.

[rsc204 21,808]

To avoid things getting more complicated to manage than necessary, I'd have two copies of the NAM. One DAT packed, the other (archived somewhere) that isn't. Whenever you make changes, switch to the non-DAT packed set of files beforehand. Then create a new DAT packed version using these updated files. Possibly a good idea to test the modifications before doing this.

Technically you can get away with it if you simply don't DAT pack the file(s) the TSCT amends. Looking at it , I think that's just NetworkAddonMod_Traffic_Plugin_Z_XXX.dat, but you'll need to see what works.

[DavidDHetzel 1,816]

Ran into a minor issue, Loading times are a bit shorter (still around 6-7 minutes I think),The Plugins work fine, but I am having issues with the maps showing up.

[Fantozzi 4,067]

DAT-packing became quite a holy grail to me an I experimented a lot and I had different results, depending how I dat-packed my plugins. Without real knowledge (as I can't realy see physically the data on the disc) I can only report what I noticed and what I concluded.

I think, It has much to do how windows writes data on disc and what is called "fragmentation". Windows searches the disk from the inner to the outer for free space and drops the data wherever the space is sufficent. Especially when you have small data it may happen - even when the folder pretends they are all together at one place - they a spread out throughout the whole disc, as small data fits in small spaces on the disc. So by starting the game and reading the plugin-folder it may happen that the magnetic head has to run up and down the magnetic disk to drag the data together. 

Therefore - I discovered that you have to do both, dat-packing and defragmentation - and the best is, to datpack, then copy the dat-packed data to a different disc, then defrag, then put the packed plugins back.

Second I discovered that dat-packer does it's magic mostly on those small files, lot-files f.e. are often 2k and smaller. It's not like zipping files, it collects the information from those files and writes it dow in a single file of the same type. Different with model files - with model files dat-packer is more like a zipper and simply puts them together without rewriting the information.

In my testing it came up, that if you dat-pack bigger model files, let's say bigger than 5MB, almost nothing is gained. If you instead collect only the small files, let's say below one MB, it can speed up load time tremendously.

Then I discovered there seems to be an optimal dat-pack size, can't exactly amount it, seems to be between 15 and 25 MB. Maybe also dependent from the size of the hard drive = the size of  the sectors (?). But at least for me it turned out, that it loads faster to have 5 files of 20MB than having one file of 100MB. This may be a windows-specific behaviour. Don't know. These are my experiences so far, on my pc, not scientific results.

 

EDIT

I don't own a SSD, so I can't do experiments with it. But my guessing would be that dat-packing on a SSD does almost nothing as dat-packing resolves issues you may have with magnetic hard drives, but on SSD the reading and writing process, to my short knowledge - is completely different and those tiny SC4-files shouldn't cause the same problems on a SSD as on a "classical" HD.

EDIT2

Would even go further - regarding the age of sc4 - a well packed and well organized pluginfolder on a HD could be faster (leaving the program itself on the SSD), as you can't achive any "preferenced" load order on a SSD, as - to my knowledge - there is no hierarchy on a SSD - all data is equal = no mather of organisation, speed is always the same? That's why they recomment to put only programs on the SSD and not the content (apart from the limited lifecycle). Parallelisation is more effective. Isn't it that way? So you had to look on the busses first, on the type of disk on the second - if program and data is loaded through different busses, speeds up time more than having both on the fastest bus (means even USB+SATA is faster than a single SATA). If you use different busses parallel the speed of a single controller becomes less important. Also this, in my opinion, should be considered. Me f.e. I use a SATA Raid 0 what means I achieve a higher thruput by using two disc parallel. Don't think a single SSD beats this easily if the interface of the SSD is shared with the HD. When data flows, most of the work is done by the controller - by the pipe, not by the bucket. If it passes through one controller both ways the same time it's allways less efficient as if there are two different controllers, one serving data, one recieving. Don't know. Would need some testing, if - and under what circumstances - a SSD really can speed up plugin-loading. I would guess, it's faster to keep programm and readed data on separate busses, this means, if sc4 runs from the SSD, leave the plugins on the HD. But this had to be verified.

 

EDIT3

Even internal data-mangement seems to bee a heavy point, I have another pc with two similar disc, purchased at the same day in the same shop. If one of them reaches less then about 9% free disk space it rapidly gets extremely slow compared to the other disk. Seems someone can't handle the amount of data anymore. Who? Windows? Don't know.

Then: to my experience the relation between plugin-folder-size and loading time isn't linear. For me loading time with 2.9 GB was much faster as with 3.5 GB. Seems to me, there must be a point between 3 and 4 GB where loading time goes down somewhat exponentially. Would need more testing - I discovered this, when I was searching for a malicious plugin. Divided the folder first in a half and so on - you may know the principle.

What I try to explain: there are many factors influencing loading time of plug-ins - it can become a passion to play witch all the knobs and see what happens. And I'm even not shure if you can transfer the observations from one computer to another - these are quite complex systems. 

 

EDIT4

Sorry for the wall of text, but I remembered I had a discussion with A Nonny Moose about that, and this is quite important: there are two relevant types of fragmentation (interfile and intrafile fragmentation). If you have a big file and only smaller free spaces on a hard disk, Windows will write the file physically in pieces on the disk, drop a part here, drop a part there (intrafile fragmentation). If you start the defragmentation utility, windows will try to sort this pieces and to unite them physically. If you have many small files, spread throughout the disk (interfile fragmentation), defragmentation-utility won't do anything good to them, as they are separated files and defragmentation-algorithm sees no reason to bring them physically closer together. That's where the dat-packer becomes really handy, to make the small files into a big one practically makes them recognizable that they belong also physically together. And if you now - after datpacking - do a defragmentation, you can gain appreciable loading speed. This would mean: defragmenting a disk without dat-packing could even worsen the loading speed of SC4, as the small lot- and dat-files could be spread out even more across the disc. To my poor knowledge - newer windows version (starting with NTFS) have a writing algorith to avoid intrafile defragmentation, that tries to spread out small files the way, that large free areas are kept for bigger files.

 

 

[markussaage 762]

On 28.1.2016 at 4:20 AM, rsc204 said:

Then you have ask yourself a question, it is worth spending money on an SSD to improve things? Because one way or another, every time you start the game it needs to index 12GB data. Every wonder why it's slower after a reboot to start up SC4? That's the game indexing your plugins. Even when you close SC4, it retains some of this data until the computer is shut down.

Personally, I'd expect a significant improvement with a $100 investment here. Not to mention you'll never go back to running Windows on regular hard drives once you've installed one. Hard drives are in the dark ages compared to the CPU, Memory and GPU speeds we have come to expect. On most systems data access is the bottleneck, SSDs really help here.

I have almost 8GB of data in my well organized Plugins folder. I suggest you use tools like Heatsoft Clone Cleaner to find and eliminate double files and the mentioned unnecessary files like JPEGs,TXT, HTML files etc., the DataNode Tool to find files that are not necessary anymore, because they overwrite other files, broken files and dependencies. Organizing the plugins is a MUST with that amount of data. And after the cleanout/organizing/backup the files use the DATPacker tool. My game on a huge map is loaded in less than 2 minutes. And as Nonny Moose and rsc204 mentioned, a SSD help here, too.

Kind regards!

[QuantumChicken 82]

My SC4 is on a RAID-0 10k RPM SAS3 Hard Drive array. It takes only 15 seconds to totally load 8GBs of data on a huge map.

[rsc204 21,808]

That's nice, but back in the world of reality, what does such an array cost compared to a SSD. I am sure using PCI-E Flash, you could get that down further, if money is no object.

[QuantumChicken 82]

HDDs for $30 each (4 of them @ 300GB each) and $25 for the SAS controller. Got them off of ebay used about 4 years ago and never have let me down.

[LightSkinnedPisces 71]

I used the extra cheats plugin but then removed from plugins before giving a couple cities the $9999999, could that be a reason it's so slow to load? 

[CorinaMarie 29,605]

6 minutes ago, LightSkinnedPisces said:

I used the extra cheats plugin but then removed from plugins before giving a couple cities the $9999999, could that be a reason it's so slow to load? 

Nope. That only alters a variable in the city tiles where you adjusted the funds and would not slow down load times. Removing the DLL itself would speed up the load time by a few micro seconds, but otherwise have no noticeable effect.

[rsc204 21,808]

The loading time for SC4 is wholly dependent on a couple of factors. How much data and how many files you have in your Plugins folder being key. Upon loading the game, it will index every item that exists in Plugins. This means every file must be read and where necessary, some key information stored so it can be retrieved quickly in-game. The mechanics of this are a bit technical, but it was necessary for a game made at the time, due to the limited amount of hardware most users had. In real terms, the following things will speed this process up to some extent:

• Removing all non-SC4 files from Plugins folders
  The game doesn't index non-SC4 data, but it does check the files beforehand. Of course if we're talking the odd readme of image file, that wouldn't be a big deal. But if you've got a lot of such files, it can really impact this area.
• DAT-Packing your Plugins
  Reading less files is quicker than the same amount of data, spread over more files. This is simply related to how Hard Drives work, older spinning-disk technology, needs to manually seek out every file to read/write to it. Whilst in practise this is done reasonably quickly, if you've thousands of files, the extra time can be quite large. DAT-Packing is a method of combining many files into as few as possible, in order to improve both load times and responsiveness in-game.
• Use an SSD
  SSD technology doesn't have to seek files, because it can almost instantaneously access any data, regardless of where it is located in the drive. This means both that DAT-Packing will be much less beneficial if using one, but it will allow the game to load data much quicker than without one. SSDs are getting very reasonably priced, if you don't have one, almost for certain your HDD is holding your machine back from it's potential speed.

You must also bear in mind the limitations of a game designed for very different hardware than we have today. If you choose to use a lot of plugins, there will always be a noticeable loading time for SC4. The solutions above just help to optimise that as much as possible.