Restoring lost building exemplar files

[11241036 5,095]

I've recently run into really big trouble due to an oversight of mine that will probably mess up my entire region, I hope someone can help me with this. I can't exactly estimate how bad this will be, but if there's no help, I may have to wipe out a region I've been working on the last 4 years.

I'm usually keeping the building exemplar files and lot exemplar files of growable lots in separate folders, to avoid losing important files when modifying a growable lot in LE, since it only saves the lot exemplar file with growables. Accidentally, I deleted an important file with building exemplars, and I had no backup. More than 100 lots are affected, about 50-60 building exemplars are lost. This renders me impossible to grow the affected lots.

While it's no problem for me to just copy any building exemplar file multiple times and alter the IID's to match the required IID, the issue is the region: It saves all data, especially how many inhabitants, and I can't recall in every case how many inhabitants these lots had. This means: If I choose the wrong number, the game will subtract more or less inhabitants than the original building had when I'm trying to bulldoze an affected lot.

Worse than that: I don't know if buildings are affected by this, but it seems that with props, the game saves the Group ID along with the Instance ID. I restored two props that I lost this way, with the proper graphics and IID, but they still do not appear in-game.

I already tried out several softwares that are supposed to restore deleted files, but without success, they can't even find the lost file at all, as if it never existed.

Does someone have an idea of how I can restore the lost files in a meaningful way? Is it possible to read a savegame and find out how many inhabitants a building has? Is it at least possible to find out the GID and IID of the lost building exemplars? Like already mentioned: I can replace the lost files, but if this isn't done in a meaningful way, I can wipe out my entire region.

[jeffryfisher 2,176]

Firstly, tell us exactly what you did to "delete" the file. Then tell us what OS your machine is running. Then look in your trash bin or whatever to see what's deleted but recoverable. If that doesn't work...

Tell us where you keep your backup of your plugins. Then tell us where you keep your backups of all the original downloads for the plugins you have installed. Then tell us where you keep the dedicated copy of your plugins for editing purposes (because nobody edits their production plugins folder EVER*). Then tell us where you keep the version history of all the files you edit. Then fetch all of the optical disks onto which you've burned all of those backups several times per year for the last 13+ years.

While you're going over all those copies of copies, see if you can find a recent version of the file you lost. If that doesn't work, then yes it may be possible to read the data from a saved city (hunt down the editing saved cities is possible thread).

* For those following along at home: When one does software development, which is what modding is, have a DEV instance for raw creation, a TEST instance for trial runs, and a PRODUCTION instance for proven good & integrated code (mods). That means at least three whole plugins folders. Because of the way SC4 works, one either renames/copies plugins folders to bring a version under its gaze (while preserving the others), or one uses the custom plugins attribute to point at the plugins to use on launch. Each instance deserves backup to separate physical media (possibly the "Cloud") plus periodic burns to optical disks.

PS: I apologize if I sound harsh; I'm trying to emphasize best practices to the wider community.

[11241036 5,095]

1 hour ago, jeffryfisher said:

Firstly, tell us exactly what you did to "delete" the file. Then tell us what OS your machine is running. Then look in your trash bin or whatever to see what's deleted but recoverable. If that doesn't work...

I removed it in Windows Explorer with Rightclick and Delete and then also cleared the trash bin.

My OS is Windows.

1 hour ago, jeffryfisher said:

Tell us where you keep your backup of your plugins. Then tell us where you keep your backups of all the original downloads for the plugins you have installed. Then tell us where you keep the dedicated copy of your plugins for editing purposes (because nobody edits their production plugins folder EVER*). Then tell us where you keep the version history of all the files you edit. Then fetch all of the optical disks onto which you've burned all of those backups several times per year for the last 13+ years.

I usually keep backups at a separate location. I don't have that many optical disks in my home, so that was the easy part.

1 hour ago, jeffryfisher said:

While you're going over all those copies of copies, see if you can find a recent version of the file you lost. If that doesn't work, then yes it may be possible to read the data from a saved city (hunt down the editing saved cities is possible thread).

* For those following along at home: When one does software development, which is what modding is, have a DEV instance for raw creation, a TEST instance for trial runs, and a PRODUCTION instance for proven good & integrated code (mods). That means at least three whole plugins folders. Because of the way SC4 works, one either renames/copies plugins folders to bring a version under its gaze (while preserving the others), or one uses the custom plugins attribute to point at the plugins to use on launch. Each instance deserves backup to separate physical media (possibly the "Cloud") plus periodic burns to optical disks.

PS: I apologize if I sound harsh; I'm trying to emphasize best practices to the wider community.

I forgive you. But to make this sure: I have years of modding experience. I know what to do. I frequently make backups, and I know about the renaming trick, and how to organize my folders and so on. I've already tried everything you wrote. I just either forgot to make a backup of this specific file, or deleted it as well. And searching for the original on the STEX or anywhere else is not an option because I deleted something I created on my own. I should have probably told this.

[jeffryfisher 2,176]

15 hours ago, 11241036 said:

I removed it in Windows Explorer with Rightclick and Delete and then also cleared the trash bin.

  That's unfortunate. In my own plugins maintenance, instead of deleting mods, I move them out to a deprecated mods folder (or a delete me folder), adding an extra layer rather than accelerating the loss. I've read too many horror stories about cities getting into trouble because they were built using mods that were subsequently removed (so I am also loath to even remove mods in the first place.

I now think your path to victory is to recreate the missing file. You created it in the first place, so you can create it again. If you need to dig up numbers from lots existing in saved cities, then make contact with the folks who've gotten into those saved-city files. Learn how to use what you know to find what you're missing. When you're done, you can write to us about it and any new techniques you develop along the way.

Good luck!

[11241036 5,095]

Well, it seems that this would be the only way to somewhat mitigate the damage. I've done some researching of my region with Save Game Explorer and it seems that there's at least partly a way to cope with this issues. But first, some results of my researchs:

It seems that if you save a city when some content, such as prop and building exemplar files, has been removed beforehand, the game partly removes the TGI. I should note that the deleted file also contained two props, and I know one city where these props are missing know (that's why I've been able to find out about the lost file, anyway). In the Prop subfile, these props are now saved this way:

TID: 0x00000000

GID: 0x00000000

IID: 0x00000000

IID2: (the actual prop IID)

I already made a replacement for the lost props, but I couldn't make them visually appear, again - at least not in-game, in LE, everything seems to be OK. I'll try to find a city which has these props saved, and then I'll try to recover the TGI this way; I haven't tried that so far. This will also take a while, I have a busy weekend ahead of me.

It also seems that the same happens to buildings, because there is a city where I have a lot of buildings where the TGI has been reduced to only zeroes.

Fortunately, there also seems to be some good news:

• I can find out the lost building ID's, anyway. I know which lots are affected, and they have at least the IID's saved in the Lot Exemplar file. Fortunately, most of these lots can be grouped together by their lot theme, so they are easy to track down in my cities.
• Even if some of my props in my cities are lost for good: These are only 2 props of minor importance, and props in general rarely affect gameplay, thus, I can just replace the affected lots.
• The main issues are the buildings saved this way: If I fail to recover their complete TGI, or if recovering it doesn't help, I already know that there are buildings affected by this issue. Fortunately, it seems that only one city has been hit really hard, and I wanted to redesign it, anyway, so I could just wipe it out. I can make screenshots of the city and then decide which parts to keep and which not.
• If one of the affected lots simply hasn't grown anywhere in my region, I don't need to do any more than creating a replacement building exemplar file.

I should note that I haven't found a way to find out important properties such as pollution, bulldoze cost and so on. This doesn't surprise me much, actually, because of the well-known fact that if you plop/grow a lot with 10 air pollution, then reduce the pollution to 5 and then remove the lot, the game only removes 5 pollution, not 10. The game apparently doesn't save the pollution with the lot/building, anyway.

However, the game at least saves how many people live/work at a certain lot. I already checked this, and there's a city where there are at least 10 instances of the same lot. Each of them had a different number of residents saved on it. While I can't recover the lost information this way, at least I know the maximum number of people that could live on them. I should point out, however, that if you bulldoze a building from your city that doesn't have a building exemplar file, the residents will just remain there. I already heard of this "ghost residents" phenomenon somewhere but can't recall it exactly.

This leads me to think of why I should actually bother that much to create an exact replica of the lost files. As for the residents, at least I have a partial solution. And I don't have that many buildings with significantly high amounts of pollution that I risk messing up a city if I type in a wrong number of pollution values.

The next step will thus be: Checking every city in Save Game Explorer and finding out the TGI and maximum number of residents of as many buildings as possible, then creating a replacement file. Until then, I can't play the game aside from laying out infrastructure and terraforming in empty cities.

I'm now tagging @rsc204 and @smf_16 for all of this; the former one may be able to point out mistakes in some of my conclusions above, the latter may probably know a way I didn't think of to recover lost information. I'm also tagging @CorinaMarie and @Cyclone Boom, I've seen them responding to the Editing Savegames thread with substantial content, maybe they have some insight that can help me.

[CorinaMarie 29,571]

Oh, my!

It seems you've already explored this much more than I have with regard to building exemplar data in the save game. About the most I've done is helping a member who'd altered the cost of a school or fire station or something and we did come up with a way to fix the ghost sliders by creating new, wrong amounts to offset the hidden totals, plopping those, then altering again, and bulldozing.

[rsc204 21,806]

Being frank, once I read this:

On 07/10/2020 at 2:52 PM, 11241036 said:

Accidentally, I deleted an important file with building exemplars, and I had no backup.

I figured in some ways you are doomed, because to properly fix this, you need those files or replacements that won't lead to anything getting upset with the game's calculations/simulators. I could imagine a scenario where everything mostly looked fine, but after some time you find there are huge problems in your region, dating back to this issue. Of course it would be heartbreaking to have to start over, but it would be pretty terrible if you put even more time/effort into the region, only to later come to the same conclusion.

The closest I've come to this is removing changes to utilities or forgetting to remove modified versions, when using existing regions. The crux of the problem is the same, you simply must have the right values for certain properties in order to restore balance. Comparatively those were simpler, since remembering a few Properties for a handful of lots isn't half the battle you face.

On 07/10/2020 at 2:52 PM, 11241036 said:

I already tried out several softwares that are supposed to restore deleted files, but without success, they can't even find the lost file at all, as if it never existed.

Is the drive in question a Spinning Disk or Flash-based (SSD / NVMe)?

For traditional spinning disks, data recovery in this sense, can be made possible due to the way drives store their data. Unless your drive is very nearly full, it's unlikely that the area which previously contained the files has been overwritten. In such cases, tools can be used to get at any residing fragments of data that may exist and it probably does. However, I've not personally come across any free tools that can do this well. Of course in this scenario, if you are paying, you rightfully expect results, which can seem a bit of a minefield. That is why most such apps offer restricted trial versions, but if you got good results with one, you can have a bit more confidence that it might actually be worth the cost. In this scenario, you should at the very least be able to get a reliable sense of which files on your drive it is able to recover, even if actually doing so requires payment for the app.

If it's a flash based drive, things are much, much murkier. What happens when you delete a file on Flash Storage?, it just removes the index to the file, i.e. you can neither see nor use it as data. However, the files remain until the drive decides to perform routine operations which actually free up such unused space. Whether or not Apps exist that can help in this scenario, I can't really say, but if that memory freeing routine has occurred already, it's probably already too late. Avoid running any "drive optimisation" manually for the moment.

In either scenario, the best advice for now is, if you want the highest chance of recovering data, you must stop using the drive that contained it. The more you keep using the drive, the higher the chance that data that was recoverable, will no longer be. I'm guessing this is a system drive by the sounds of it? If so, it should be safe to clone that to an equivalently sized drive and then use the copied drive for the time-being. Of course, I do understand this may not be practical for you.

Are you using Windows 10, is it the latest version (May 2020)?, if so this may be very benifical.

Just be aware that Data Recovery is all good and well in principle, but practically speaking it's very much a 'hail mary' and as such a last resort. It's one of the things I like most about Mac's, having Time Machine basically means you can plug in a suitable external HDD and forget about it. Windows can do similar things via the "file versions" feature, yesterday I got a 5TB Seagate external drive for 90€ (offer in Saturn), so if nothing else, make sure you learn from the experience and invest to prevent it reoccurring.

It seems to me, unless you can either recover the necessary data from the save files to recreate working Buildings Exemplars, or somehow just remove all affected lots from your save files without leaving "ghost" sims behind, there is always going to be some lingering doubts as to the ongoing viability of your region. Out of the two, I'd prefer the later approach, because rather than trying to second guess stats/properties, you can just clean the affected areas and rebuild them. In the meantime, since these files are missing, try to avoid saving any city in the affected region. Doing this updates the save file with the 'missing' data that has the potential to trigger problems, so until you can make a solid plan, it should be avoided. If you have been saving cities since this occured, restore the last archive of your region that was made before this issue started, just to be safe.

Really, my knowledge of the save file structure is simply that which has been provided by others. If the game is expecting specific values, it stands to reason it must keep a record of them somewhere? Of course the modded figures for Capacity Satisfied, reflects the maximum potential occupancy, rather than the actual number of occupants which can fluctuate. Given that, I think it's really important to get the actual Capacity Satisfied values if at all possible. Not sure if the data works the same for other properties or not? Off hand I really don't know how many must be right for your new Exemplars to not cause problems. A bit of testing here would definitely be worth your while if data recovery is not an option.

[Flann 1,709]

I'm sorry I don't have too much to offer in terms of what to do to recover the data, but I do want to recommend paying for a Dropbox account or similar cloud backup utility.  I have my entire

Documents\SimCity 4 folder backed up automagically.  Not only does this allow me to play the game on two different computers, it also offers some protection from data loss, allowing you to recover files deleted in the last 30 days.  It also allows you to undo changes to files, like plugins, or even save games.  For $12 a month or so, I'd say its worth it.

At any rate, best of luck with restoring your exemplars!

[11241036 5,095]

Thank you all for your answers so far.

It might be interesting to share what I've found so far. Before messing further with my region, I created a testing situation and recreated the situation this way. My questions in particular were:

• If you save a city with the resource files lost, what will happen after restoring them?
• If you modify the resource file by altering the number of residents - which, apparently, are saved on the lot, rather than the building, in-game - what will happen?

The preparations:

I created an empty region and renamed my Plugins folder to something different, then I created an entirely empty Plugins folder, with only a few files:

• Some money cheating lots, in case the infrastructure costs too much,
• A blocker file, blocking all Maxis residentials,
• One file with a lot exemplar file, containing the only residential that will be able to grow. It was on stage 1, so it could immediately grow.
• One resource file, assigned to the residential lot. It had 15 R-$$.

I created two testing cities with some basic infrastructure (power, water etc.), jobs, and a few residentials. After the first houses grew, I dezoned all residential areas to make sure nothing can grow anymore, then I bulldozed all houses with 15 inhabitants (I wanted to see what would happen if I alter the number of residents to a lower value). The maximum number of inhabitants on one particular lot was 13.

I should note by now that I only saved these cities when they were paused. I always do it that way, anyway, and it made sense to me to do it here as well, to better see what would happen.

Step 1: I removed the resource file (I made a backup first) while leaving the lot file where it was. Next was visiting one of the cities (let's call it City 1) and immediately saving it.

Result: When looking into City 1 with Save Game Explorer, all building exemplar files got their TGI reduced to 0x00000000, only the IID2 showed their real IID.

Step 2: I visited City 1 in-game and wanted to know if it makes a difference if you bulldoze a lot or remove it by dezoning. I dezoned/bulldozed literally all residential areas.

Result: Nothing happened (except removing/dezoning). The number of inhabitants remained the same, although City 1 should technically have 0 inhabitants. BTW, I also couldn't query the lots anymore, and bulldozing them didn't charge any money.

Step 3: I put the resource file back in and wanted to see the difference in both cities, in particular:

• if I can query the lots,
• if they will charge money when bulldozing,
• what happens when I let the game run,
• what happens to the number of inhabitants when I bulldoze the lots,
• what happens when I dezone them.

Result:

City 2 was behaving completely normal; I could still query the lots, it costed me money when bulldozing etc.

City 1: I still couldn't query the lots, they didn't charge me money for bulldozing, and the number of residents remained the same after years of running. Interesting, however: After bulldozing/dezoning the lots, the number of inhabitants dropped to 0. This seems to prove that the game now can't load the building exemplar file anymore, but the number of residents are still saved there, and the game can still remove them.

(The next two steps have only been tried in City 2.)

Step 4: I wanted to see what will happen if I alter the number of residents to a lower value than it were originally - particularly, I reduced it to 14 (remember: The highest number of residents on one of the lots was 13).

Result: Quite literally, all lots jumped up to 14/14 R-$$ residents. After removing all of them, a very low number of residents mysteriously remained in my city - probably still our dreaded "ghost residents".

Step 5: I raised the number of residents to 16 instead of 15.

Result: Nothing special. The lots only showed the number of residents they had before I modified the resource file - the query just showed me for instance 13/16 instead of 13/15.

 

My conclusion:

There still seems to be no solution for finding out how high the values of pollution and power/water consumption are.

At least for the part with the number of residents: In order to safely remove "broken" residential lots, it seems to be sufficient to restore the lost building files - but it's better to "overshoot" than to be modest. I could probably solve this issue by giving all affected building exemplar files a value of 5,000 residents. If it weren't for such a high number of affected lots and my huge Plugins folder, I could even go this far to just try it out by raising the number of residents one by one, checking out my cities every time, and see from when they won't start maxing out on residents - then this must be the right number of inhabitants.

Please feel free to point out mistakes in my conclusion. But, as already mentioned above, I don't see any point in bothering too much with finding out the real values of residents/pollution/whatever, as long as I can more or less keep playing. After all, SimCity 4 is a game with many quirks, and believe me, that's not the first time I messed up with something. I've suffered from the Phantom Slider Bug, mysteriously removed some props because I accidentally altered their GID, altered the monthly cost of a lot before removing it, changed the RKT to something that resulted in giving me a brown box, plopped a lot with CS-$$ jobs and a wealth set to No Wealth and saving the city (causes a CTD whenever you try to open a city)... In all of these instances, after fixing it, I could still play the game and never had any really bad repercussions, even after playing for (real-life) months. Call me careless if you want, but while I appreciate the many helpful tips on this site, it's sometimes hard to find them before messing up, or I'm just the first to find out, anyway.

 

Edit: @rsc204 Thank you for the link. I tried it, but to no avail. Signature mode, BTW, doesn't support .dat files. I don't know if segment mode does it, but I'm not willing to spend hours in front of my PC only to wait for the result. And all other recovery softwares couldn't even find the slightest trace of the file in question, let alone restore it.

@Flann Thank you too. I'll keep that in mind.

  Edited October 19, 2020 by 11241036  
Grammar fix, answers