Extracting certain props from a prop pack

[simnewbie 7]

Good day people. I was just wondering is it possible to extract only certain props from a prop pack, so you only load the ones that you use? I remember a while a go there was a tutorial by pegasus on how to do it? can anyone shed any light on this

[A Nonny Moose 8,261]

If the prop pack is a *.dat file, the only way I can think of is with ilive's reader.  I seriously doubt the game is worth the candle as you won't save very much.  Big learning curve and lots of handsprings.  If you are not a computer programmer I'd suggest you will have a lot of trouble.

[SimCoug 5,972]

Paeng had written a good tutorial using ilives reader, but it is now lost due to the simpeg disaster.  It is possible, but it takes some experience using the reader program.  If you are interested I can try to walk you through it.

[Mister Giggles 310]

It's a giant pain, currently, to decompile mega packs and such for individual props or to cull unused ones. It's (Somewhat) easy to isolate a solo prop and its constituent parts, but there's many steps and rules involved. A tldr version is this:

Always make a backup. Just in case.

1: Run Datanode. Take note of how many missing dependencies there are.

2: Open up all your buildings and lots in PIM-X, along with the prop pack you wish to remove things from. Depending on the size of your plugins and how you have them organized, this may be a somewhat messy process.

3: Check each and every prop you want to remove to see if they're used on a lot or building.

4: Prop families are problematic. They are better left alone unless you care to cross reference everything in the family to make sure you're removing everything and none of the family is used.

5: Never remove LUA, UI, or sound files. Datanode doesn't/can't check to see if these are used.

6: Once you've confirmed a prop isn't used by the game, you can use iLive's Reader to remove it and its related files (BMP, FSH, 3DS, etc). Most of the time all related files are clustered together when you organize the list by "Number."

7: Run Datanode again. If you have the exact number of missing dependencies you had before, you're good to go. Sometimes graphical files are used for multiple things, and this is when you'll find out. You'll need to re-add them or remove the other prop/s using them.

An alternative version involves removing a prop pack entirely, noting which lots/buildings are missing things (Via Datanode), and adding the missing things back in one at a time (Copy from the original prop pack into a new, empty one). Depending on how used the prop pack is, this may be a bigger bang for your buck.

I've been able to get rid of about a gig worth of unused props/graphics using these methods.

  Edited September 5, 2015 by Mister Giggles  

[rsc204 21,808]

There is a far simpler way, it's really not that drastic when you remember that all you are doing is copying one prop from a large pack, if it goes wrong, just go back to the original file.

Think of a DAT as a zip file, inside it are all kinds of files, that make up the components of things used in-game.

• Every model in SC4 (prop) will have 20 S3D exemplars, those are the 3D models in 5 different zooms and 4 different rotations.
• There will be at least 20 FSH files, the textures that apply to the models, although there could be more for larger props as it splits textures for these into multiple pieces.
• Lastly there will be a prop exemplar, listed as an exemplar, but when selected this will show the exemplar type "prop".
• Optionally you may also find LText files.

Here's an example file, note how the Type Group and Instance IDs group the content together. You can find things easily using the Navigator function, assuming the creator named them logically, otherwise the only hard part is finding the props to copy.

So the navigator shows the prop exemplar, here you have the Resource Key Type X property, the Group ID should then match the S3D and FSH files (Red) so you know which ones to copy. In this example, The Exemplar for the prop and the LTEXT files are handily grouped together - they usually are. LTexts (green) are not always needed, they are files used to display the name, but in essence the parts in green in the prop exemplar will have the IDs for you, if these properties don't exist, then there are no LTexts to worry about.

In essence you simply need to find the correct group of files, highlight them all in the left pane of the reader, right mouse click and select copy. Then open a blank reader file and paste them (it's handy to have two reader files open side by side for this). Repeat the process if you want multiple props and save your new file.

[Mister Giggles 310]

That is a more descriptive step 6, yes, but only part of accomplishing the topic creator's stated goal: To only have the props that are attached to something that appears in game.

I'd prefer it not be a mess of programs and obscure functions, either. Alas. Datanode can say what is missing, but not what is used. PIM-X can say what isn't used, but can't remove it. Reader can remove it, but lacks the information needed to understand if it should be removed.

  Edited September 5, 2015 by Mister Giggles  

[rsc204 21,808]

Yes and no, it might fit your step 6, but my instructions are basically a slightly cut-down version of Paeng's previous tutorial, the other steps you mention are one way of doing this, but it's not how I would approach this task. For a starters I'm suggesting doing things the opposite way round to you, the two instructions could not really be used together.

In many cases a dependency is used for not more than a small handful of props in a given set of lots. In this sense it is surely simpler to look at the props being used and make a copy with those files than going through and removing all the ones you don't individually. Especially as some prop packs are huge and contain hundreds of props. It does require a certain familiarity of your plugins folder and required props though, but if you do it this way round and find you are missing a prop (again DataNode could be used to help you), you could always go back and copy the missing ones later.

I wouldn't be doing this unless I were setting up a new plugins folder, going through each newly installed lot as I craft the custom dependency packs along the way. Trying to apply this sort of change to a pre-built plugins suite is just asking for trouble IMO.

[A Nonny Moose 8,261]

I am glad to see such keen people still exist.  You guys remind me of myself when I was in my twenties and thirties.  I was full of curiosity and had the capacity to work  on such things.  Now that I am retired, I just don't care very much about it, and I have absolute tons of disk space, so don't mind a few extra bytes even though I am very parsimonious with my system residency.  At the moment, my whole system occupies about 25% of my main partition which is half of my built-in HDD (0.5TB).  This includes the entire operating system (including a number of development languages and suites) and all my private file space, but not the backup space which is on a separate 1TB drive.

What benefit do you expect to accrue besides deep knowledge of *.dat files and the guts of a BAT package?

[Mister Giggles 310]

RSC: Based on my experience, it's like pulling teeth if you do the process with each and every new thing that enters your plugins folder. You'll always be going back to, say, BSC MEGA Props - SG Vol 01, to add a few props. On a sidenote, that's the most used prop pack, with thousands of lots relying on its plant props. Simgoober should be proud.

When you have most of the plugins you'll ever have, it's quite easy to see what prop packs are heavily used, which ones aren't, and to cull/move props in bulk. In my opinion, it's best to consolidate all lot/building exemplars so you don't miss anything when you're checking for what's being used or not. For example, I have 3 plugins I have containing all my lots and buildings, with no graphical files included.

In my mind, the hardest part of the whole process is getting to where it's the reasonable next step. Consolidating plugins, removing useless files, merging files - those are all steps that should take place beforehand. Though, be careful with merging prop packs. Those can contain thousands of individual files, and you do >not< want to be trying to slim down a mega prop pack with 20,000 files in it (Not to mention Reader starts throwing a fit with gigantic files). I'd say that was my biggest mistake, merging several massive prop packs together before I decided to remove their unused props.

Nonny: The primary benefit to slimming down your prop packs is quicker loading times if many megs worth of props are removed, and stability. You'll be removing prop overwrites, which removes many problems - prop pox is the best known of these, but it's quite obvious SimCity 4 pukes all over itself when overwritten plugins are substantially different from what's overwriting them.

You're also eliminating the absolute number of things SimCity 4 has to load and keep in memory. SImCity 4, all by itself, has about 105,000 TGI, total. With my 4 gigs of plugins, that jumps to 900,000. I used to have 1.1 million before I started culling unused props and graphical data. I noticed during this process that SimCity 4 grew progressively more responsive.

Just removing one or two unused props isn't going to do much - but removing hundreds, thousands, tens of thousands, that adds up...

It's also something of a pass time, considering the time sink involved.

[A Nonny Moose 8,261]

Well, let's see ... I don't have very many plugins because I vet them pretty hard, but here is the Disk Usage Analyzer graph of that area:

You can see from this that my plugins take slightly over 50% of the space used by the game.  This is only 1.1 GB on a 500 GB drive.  The biggest chunk is taken by BSC and is mostly libraries of dependencies with a few lots that use them.  My game loads in 40 seconds and the current city I am working on in 65 seconds.  The city includes a large Pegasus port with several supplementary lots, an industrial section including rail connections, etc.

Is it worth monkeying around weeding out unused dependencies?  I doubt it in my case.

For those who have Gigabyte upon Gigabyte of plugins it might be worth the effort.  But first, all plugins should be cleaned of junk files such as documentation and images that are not part of the plugins.  After that, consider using a compression program such as Files2DAT to consolidate the lots in each plugin.  I had one where I had to do this or my system ran out of file handles because there was a myriad of files in the set of plugins (many lots).  As things stand right now, all of my plugins are *.dat files, but each plugin is an individual file.  When I get a new plugin, if it is not already a *.dat, I clean it up and condense it.  Not much space saving, but significant speed improvement since we don't have to skip over all those null files when reading the plugins at load time, scroll time, or zoom time.

The above extract is just a screen snap, and is incomplete, but it is in order of size of directory.  Everything after LBT is less than about 2% and often is considered 0% in the scheme of things.

I am afraid for many that the temptation to download has led to some sort of downfall.  Just because the candy is in the window doesn't mean you have to buy it.

[Mister Giggles 310]

You'd be surprised. Perhaps more than most, given that you don't have nearly the stuff a lot of other people do. Let's see...

You have LBT files. That means you probably have the mega prop pack. That means you have a 10 meg file with only a few dozen of the thousand props used. I was able to cut mine down to 2.9 megs.

BSC props vary greatly in how heavily utilized they are, but I wouldn't be surprised if a few of the prop packs you have are lightly used.

Edit: Don't use Files2Dat for consolidation. Use DatPacker. The former doesn't have deduplication functions.

You'll see a massive decrease in load times if you compress your plugins from 4,500 files to a few hundred. It proved to be my biggest speed boost, going from 4 minute load times to 2 minute load times.

  Edited September 5, 2015 by Mister Giggles  

[A Nonny Moose 8,261]

Unfortunately Datpacker doesn't run on my machine.  There is some hiccup in wine that stops it from running, so I am stuck with Files2DAT.  There are some drawbacks to not running Windoze, but overall the experience is superior. 

I am currently working with a new version of Data Node that I got from Rivit, but haven't really had the time to work on it yet.  The program is really not a release candidate as yet.

[Mister Giggles 310]

You may have luck with JDatPacker.

Rivit sends you betas of his program? Lucky moose.

[simnewbie 7]

Thank you for all the input guys. I really do appreciate it. 

[simnewbie 7]

Got it figured out. A lot easier than I thought. Thanks guys for all the input. Also I would like to thank all the creators of the prop packs. Thank you for grouping all the FSH files and S3D  files.

[A Nonny Moose 8,261]

Good for you.  Now how about offering it as a tutorial for the Omnibus?

[rsc204 21,808]

There is no great mystery to the process... I could write something up sometime.

As for the grouping, that's pretty much done by default, but with a little carelessness it's easy to undo it when packing files together. Some are worse than others, but by and large you'll find everything works out. One thing to watch out for, typically 3SD/FSH files are grouped by the Group ID, but in some rare cases they will be grouped by the Instance ID and the Groups will all have unique IDs so bear that in mind too.

[simnewbie 7]

Good for you.  Now how about offering it as a tutorial for the Omnibus?

really? I would love to. I have taken so much from this site. I would love to be given the opportunity to share and give back.

  Edited September 13, 2015 by simnewbie  

[A Nonny Moose 8,261]

Good for you.  Now how about offering it as a tutorial for the Omnibus?

really? I would love to. I have taken so much from this site. I would love to be given the opportunity to share and give back.

Of course.  You've just done something that many would, perhaps, like to do but haven't the time to slug through it.  I look forward to your new article.