Organizing plugins - where to start?

[tagehring 321]

I seem to have hit the point where my plugins folder (all 23,473 files and 8GB of it) is causing the game to lag excessively, and my in-game menus are an absolute mess. So I've decided I need to impose some order on the chaos. I'm just not sure where to start. I've read a lot of posts here about various methods for organizing the folder, and for re-ordering in-game menus. What I want at the end of this process is a tight, datpacked plugins folder with in-game menus that are in some sort of coherent order: buildings in the right categories, groups of tiles for things like fences, paths, seaports, etc all grouped together, and preferably with the ones I use the most at the top. I don't want to mess around with DAMN; I'm fine with mile-long in-game menus so long as they're cleaned up. 

I'm just not sure where to start. I have an excel spreadsheet of what's in my plugins folder and plan to keep it updated going forward. My idea for a workflow:

1. get everything into a spreadsheet in the existing folder structure
2. categorize it there by function and author
3. add links to the original sources
4. download source .zip files into an archive
5. reorganize files into new folder structure
6. add a "menu sort" column to play around with different menu organizations 
7. ...at which point I can finally go about the seemingly arduous task of changing the "Item Order" exemplars to clean up the menus
8. *debug new plugins folder using DataNode* (Thanks, @jeffryfisher!)
9. datpack everything that needs datpacking

I know this last step is going to be the real challenge. I've been trying to learn more about how these files work, having just been an end user until now. As best I can tell, I'm going to have to open each file individually in iLive's Reader and change the Item Order exemplar manually one file at a time? I'm dreading that process, but I know once all of this is done it will greatly improve game performance and my own OCD. I assume there isn't a tool that would make it possible to load all of the files I need to change that one exemplar for, sort them into their new menu order, and then apply new numbers based on the way they're sorted? That'd be asking for the moon, right? 

This post really isn't for anything other than throwing my own thoughts out there to see if anyone's got suggestions for how to improve this process or to find out how other players handle this issue. I may have thought myself into a corner and am missing something blindingly obvious. Wouldn't be the first time.

[tagehring 321]

This is the folder structure I'm toying with for the reorganized plugins folder. I'm not sure how well it's going to work or if it's overly complicated.

~\a_Dependencies\
~\a_Dependencies\a_Prop Packs\
~\a_Dependencies\b_Texture Packs\
~\b_Packed Plugins\
~\c_Buildings\
~\c_Buildings\a_Residential\
~\c_Buildings\a_Residential\a_Low Density\
~\c_Buildings\a_Residential\b_Medium Density\
~\c_Buildings\a_Residential\c_High Density\
~\c_Buildings\b_Commercial\
~\c_Buildings\b_Commercial\a_Low Density\
~\c_Buildings\b_Commercial\b_Medium Density\
~\c_Buildings\b_Commercial\c_High Density\
~\c_Buildings\c_Industrial\
~\c_Buildings\c_Industrial\a_Agricultural\
~\c_Buildings\c_Industrial\b_Dirty\
~\c_Buildings\c_Industrial\c_Manufacturing\
~\c_Buildings\c_Industrial\d_High Tech\
~\c_Buildings\d_Civics\
~\c_Buildings\d_Civics\a_Police\
~\c_Buildings\d_Civics\b_Fire\
~\c_Buildings\d_Civics\c_Health\
~\c_Buildings\d_Civics\d_Education\
~\c_Buildings\e_Utilities\
~\c_Buildings\e_Utilities\a_Water\
~\c_Buildings\e_Utilities\b_Power\
~\c_Buildings\e_Utilities\c_Garbage\
~\c_Buildings\f_Landmarks\
~\c_Buildings\g_Miscellaneous\
~\d_Parks\
~\d_Parks\a_Fillers\
~\d_Parks\b_Pathways\
~\d_Parks\c_Plazas\
~\d_Parks\d_Prefab\
~\d_Parks\e_Misc\
~\e_Miscellaneous Lots\
~\f_Mods\
~\f_Mods\a_Automata\
~\f_Mods\b_Game Mechanics\
~\f_Mods\c_Terrain\
~\f_Mods\d_Texture\
~\f_Mods\e_Transit\
~\f_Mods\e_Transit\Network Addon Mod\
~\f_Mods\e_Transit\T21 Exemplars\
~\g_Texture Overrides\
~\h_Other Overrides\
~\i_Unknown Files\

[jeffryfisher 2,176]

1 hour ago, tagehring said:

My idea for a workflow

Download and install DataNode, then add steps where you find duplicates and conflicts and out-of-order loading (overriding).

If your machine does not already have an SSD, consider upgrading. Keep your working copy of SC4 regions and plugins there, but keep a backup on separate media.

[tagehring 321]

36 minutes ago, jeffryfisher said:

Download and install DataNode, then add steps where you find duplicates and conflicts and out-of-order loading (overriding).

If your machine does not already have an SSD, consider upgrading. Keep your working copy of SC4 regions and plugins there, but keep a backup on separate media.

I've got DataNode and am in the process of trying to figure it out. I know that's what I want to use to eliminate unnecessary files and see where there are conflicts. Definitely need to add that as a "debug existing files" step. 
 

Fortunately, my laptop has a SSD drive and I do keep my backups on several different external HDs. 

[rivit 3,391]

If you wish to make dependencies a separate group (that's generally props, models, and textures)  - I would recommend you make them a group late in the structure (just before Mods probably) because buildings and lots depend on them and so they need to load after the lots. This is especially true of the megapacks which are used almost everywhere so will be dependencies for many lots.

In general look to have a organisation like - game fixes; NAM; lots and buildings, scenery, automata etc. etc.; dependencies; overrides (Mods). 

This will lead to the least loading order problems out of the box, and be the easiest to debug.

[jeffryfisher 2,176]

18 hours ago, rivit said:

If you wish to make dependencies a separate group (that's generally props, models, and textures)  - I would recommend you make them a group late in the structure (just before Mods probably) because buildings and lots depend on them and so they need to load after the lots.

I don't understand why dependencies should load after the things that need them. I've always assumed that a dependency should load early so that it's ready and waiting in memory when something else comes along asking for it. What am I not understanding?

[rivit 3,391]

Your understanding is fine. You're right that In general it shouldn't make any difference at all - i.e. the dependencies and the lots are usually distinct items in terms of TGI. Furthermore dependencies are frequently .dats which are loaded in a second pass while lots are loaded in the first. Bets are off with Lots packaged in .dats.

The game doesn't care one way or another - all uses of a particular TGI will resolve to the last one loaded and that may not be the one in the original dependency files if it loads up front (that might be the plan of course).  Linkages are established/overwritten/updated while loading the plugins, so if multiple versions of a TGI exist only the last one encountered will finally count.

My experience, however,  is that over time updates of old content, mixes of one set with another and any number of accidents means that the megapacks in particular are best loaded late in the sequence - this minimises tangles and makes pinning down weirdness a bit easier (for me - ymmv).

[rsc204 21,815]

Generally speaking, when I look at your plan of action, you seem to be after much the same setup as I have been working towards. It's arduous work sometimes, but you'll reach a point where the benefits will begin to help keep you motivated.

Since you've an SSD, DAT-Packing won't improve load times much. But I DAT-Packed a very full MMP menu and it made opening it much more responsive and instantaneous. So ideally grouping for DAT-Packing would resolve around which menu items appear in. Remember that only applies to the Lots themselves, in respect of any models, best keep those with other dependencies.

Editing the Item Order can be made easier, depending on how much control you want/need over each individual set of items. Essentially each Item Order value is simply a number, albeit in Hex, though you can enter them as Decimal in Reader. I started a Spreadsheet when I began making changes to the order of my content. Leave plenty of space between IDs, you've literally millions to choose from. That way later changes/additions can be worked into your current system, this isn't a job to take on twice. Assuming each item requires a unique number, you are going to have to do each entry one-by-one. But it's easier to do this with files that have been packed together, since you can edit many in one large DAT file, rather than open many files and edit them one by one. However, when DAT-Packed, the order of the files inside the DAT will not likely be based on the order they will display in. Meaning lots of cross-referencing to edit them all. If I've a set of say 10 items that are a group of SC4Lot files, I prefer to manually copy/paste the contents of each into a new .DAT file. This way I can ensure the order of the file, is the order I want them to appear in-game. Then I just add the set to the spreadsheet, find which number group to use and editing the Item Order is pretty quick.

Planning will make life easier, first I'd just decide on a menu, note every item you eventually want in said menu and the order you want to see them in. After that start with the items you'd have from top to bottom in said menu.

I mentioned it could be easier to edit the Item Order, that's because SC4 uses three methods to order content:

• The Item Order property takes priority over the others.
• The Item Name property is next, in Alphabetical order.
• Lastly, the Plop Cost in order from smallest to largest is used.

So if you can make it so a set is already grouped by either of the 2 later conditions, the same Item Order can be used for them all. You can group-change this property for all or a select number of Exemplars in a given file. Open the Exemplar Analyser, Search (to populate the list of Exemplars) and select "Find a Property". Now you can edit the same Property for multiple Exemplars in one operation. Of course that won't always work out, but it's a useful trick to be aware of.

[CorinaMarie 29,617]

On 10/17/2019 at 6:14 PM, rsc204 said:

The Item Order property takes priority over the others.

For some testing, I'm working with vanilla plugins to which I've added 3 custom lots I made. I've set my Item Order for them as follows:

• 0x00004910
• 0x00005010
• 0x00009910

The 4910 and 5010 are in the right order at the bottom, but the game puts the 9910 at the top of the menu. Any idea why the game is ignoring my setting for that one?

Here's my files if'n that'd help figure out what's wrong: Parks with Numbers - First 3 Lots.zip

[Cyclone Boom 16,934]

1 hour ago, CorinaMarie said:

The 4910 and 5010 are in the right order at the bottom, but the game puts the 9910 at the top of the menu. Any idea why the game is ignoring my setting for that one?

Update:

@CorinaMarie and I have looked into this...

Converting these from hex to decimal values gives:

0x4910 = 18,704
0x5010 = 20,496
0x9910 = 39,184

My initial suspicion was how the range might be too high and then it loops around for the 99 file to appear atop the menu instead. I entered the hex values into the dialog as they are, and then upon hitting Apply it allows Reader to convert them to their true hex values. This made all 3 lots appear properly correctly consecutive at the bottom of the Parks menu. Then Cori had the thought how it might be with having an upper limit of 32,767 at the top, and thus 39,184 is too high beyond the maximum supported.

So upon setting the Item Order property to be 0x7FFF and this was one shy of the supposed limit. Then making this be 0x8000 and it indeed pushed the 99 lot to appear at the top as it's one greater than the true limit. This means the range is likely -32,768 to +32,767 with 0 rounding it out to 65k choices.

It seems the Omnibus article on the subject might be a little misleading when it says that -10,000,000 to +10,000,000 is supported. It seems how with it stating the 10 million range in Reader for the integer format is untrue. Very possibly there are other exemplar properties similar to this also.

Hopefully this new info helps for anyone else wondering as we did.

[xxdita 1,097]

First, I try to keep my downloads organized in folders named by where it came from. LEX, STEX, SOMY, Yuhisa, ToutSimCities, BriPizza, etc. This helps keep track of any dependencies should I eventually release something. 

My actual Plugins folder keeps the same strategy in play, mostly. If I have a noticeable amount of content from a specific person or team, then they get their own folder outside of the website's folder. This applies only to models & lots. All mods are put into a z_ folder out of habit. 

Since I have DatPacked everything up, anything I install now goes into a new folder by month/site until I decide whether it's staying in Plugins or not, until I'm ready to DatPack it all again. 

[twalsh102 1,601]

Keep in mind that the primary purpose / benefit of organizing your Plugins folder is to make it easier to determine whether or not you already have something before downloading it again.  Different people approach this in different ways.  There is no one correct or incorrect method - it's a matter of what works for you.

Having a plan to start with is always a good thing.  But just like in battle or a football game, "No plan ever survives first contact!"  So you will likely find yourself revising your folder structure as you go along.  As far as the game is concerned, pretty much any structure will "work."  Whether it's overly complicated, only you can decide.

For 99% of the stuff you download, load order really doesn't make much difference.  Most things that are required to be placed to affect a certain load order (i.e. loaded first, or loaded last) will either install with specific folder titles, or unzip with specific folder titles (for example:  a__CAM, z___NAM, zzz_terrain_sudden_valley, zzzz_Simfox_Day_n_Nite_Modd) and the Readme file will spell this out.  Note that because overrides are involved, these would need to be outside your planned folder structure.

Excessive path length can be a thing that can keep the game from finding files that exist in your plugins, possibly resulting in brown boxes, or missing dependencies.  So some things to consider to help head-off that possible problem:
a)  Abbreviate, Abbreviate, Abbreviate!
b)  Use the word Lots instead of Buildings (that's really what you're collecting)
c)  After the first folder level, do you really need to number/letter subfolders, i.e. a_, b_, c_, etc.
d)  Use Res, Com, Ind, Civ, Util, LM, Misc,
e)  You already know that Low, Medium, and High refer to density, so why not drop the word "density" and use Low, Med, and Hi
f)  Abbreviate the Ind developer types using the in-game abbreviations IA (or IR), ID, IM, and IHT.
g)  Parks often don't neatly break down to the categories you have.  Many packages have a mixture of those categories, so you will probably want to rethink those categories once you start organizing.
h)  Transportation/Transit (or Trans) Lots??
i)  Reward / Business Deal Lots??

Item Order can certainly be a sticky wicket.  But the good thing is that most top-level menus are already subdivided.  So you'll only need to change Item Order if you're concerned about the order in which (for example) Police Stations appear or want to congregate Stations separately from Jails and Prisons in the Police menu.  The biggest challenge you will have will be the Landmark and Parks menus.  Not only do you have to worry about all the Landmarks with Jobs (vs. straight Landmarks) (as well as the possibility of having both versions for the same building), but many creators have put things in those two menus, that don't strictly fit those categories.

[SIM-ple Jack 17,444]

1 hour ago, twalsh102 said:

i)  Reward / Business Deal Lots??

 

I never purposely put anything in Rewards. Usually, I change it to Parks or Landmarks. Rewards menu will not hold it's position, so you have to go in and scroll search every time you want to plop something more than once. I hardly ever use the game rewards anyway, as many of them are only allowed once. I usually get them from buildingplop so I can have as many as I want.

[markussaage 762]

On 11.1.2020 at 9:08 PM, CorinaMarie said:

For some testing, I'm working with vanilla plugins to which I've added 3 custom lots I made. I've set my Item Order for them as follows:

• 0x00004910
• 0x00005010
• 0x00009910

The 4910 and 5010 are in the right order at the bottom, but the game puts the 9910 at the top of the menu. Any idea why the game is ignoring my setting for that one?

Besides that I've known this for years and never found out, why Item Order values greater than 0x7fff are ending up at the top of the menus, I use this behavior to have lots listed at the "top top" of a menu even above the lowest Item Order value 0x0000. And those are usually the build in Maxis lots.

Kind regards!