tool SC4DataNode Support

[rivit 3,404]

Announcing v1.14 of SC4Datanode - the latest official release.

This includes all of the changes discussed above since v1.06 and adds a new column of data to the Missing Dependencies CSV Report. This Missing Item column will detail one line for each item missing in an exemplar. Sorting by this column will bring all like missing items together. This should allow much more oversight of missing dependency files as it will become more obvious when groups of items are missing.

It pays to suppress the NAM Ghost Stubs before running this report - or you will end up with a large report. In my case I had 430 missing dependencies by TGI view and 1490 items by report - many were duplicates. I discovered I had at least three missing Mega Prop or Texture files - wrongly assuming I had them, and was able to throw out a couple sets of lots that seemingly were not fixable - too many things missing or scrambled.

I highly recommend this to custodians of lots, props and packs as it shows up immediately when mega packs are incomplete or have further dependencies.     

[Turnstyled 539]

Thank you very much for this tool, @rivit. Analyzing my plugins directory through it has been enlightening. There were some instances in which I simply didn't read the documentation. Here are the instances that I researched, but that I couldn't figure out. If there is any general advice to provide instead of specific advice, then I would also be glad to read it. It might be preferred because additions to my plugins directory don't stop here.

Firstly, is BLaM Blank a dependency for anything? As you can see, I have it in my dependencies sub-directory; however, I just read the readme for it and it does not seem that it would be a dependency. Perhaps, I had mistaken it for something that it hadn't been.

The next few are all from the PLEX on the STEX. I cannot find any information on these in the provided documentation. Can I safely ignore all of it or would it be proper for me to do something else? They are all being overridden by other PLEX files. 

With this next one, I presume that LE is Lot Editor and MM is Mayor Mode. The lot editor poplars are being overridden by the mayor mode poplars. Should I delete the lot editor poplar .sc4model files? I don't plan on using the lot editor.

With this next one, should I use Reader to rebuild all the .dat files that in orange text?

Here are some examples of overridden files in my NAM directory. I'm using NAM 41. I feel that I safely assume that the NAM team knows what it's doing, so I'll leave this all alone.

Finally, I've seen that the American flag is included in the Maxis Day of Peace prop. Is the Maxis American flag prop redundant if I have dayopeace.dat in my plugins directory? The overriding file is dayopeace.dat.

Thank you so much.

[rivit 3,404]

Well you've managed to pack a few questions in there..  where to start

To start almost all of those marked files are overridden by later loading items. If you go in and look at those files you will see where they are overridden. Most times its because you've got two copies of something (which you're not aware of) in different parts of your plugins and they load over the top. The Blam one could well be a dummy building that is deliberately overridden by a real one later. Same goes for the flags. Once you identify them remove them and rescan to make sure you now dont have something missing dependencies. All clear after that you know you can remove them. 

SPAM does have some structural issues - there are duplicated iDs in there - In my setup I left them alone as it doesn't seem to stop anything appearing.

NAM does have dummies/defaults which are only overridden if a particular option is installed - these are checked regularly so should be OK. 

With Girafe props - looks like there are two versions - as props and Mayor plopables - they probably have the same IDs so you can only have one or the other. (Dont know this for sure)

Now all of the ones in orange have been combined/joined one way or another without reindexing the dats - this means they have multiple DIR entries. The fastest fix for that is (as you suggested) load the dat into Reader, delete the multiple DIRS, rebuild the index and save.   

Otherwise. I have to say its looking pretty spick and span.

  Edited May 24, 2021 by rivit  
forgot SPAM

[BartonThinks 3,416]

I think I can offer some help regarding the BLaM Blank -- That's a file you'll want to keep in your plugins folder to avoid brown boxes. It's an invisible model that is used on a lot of lots in place of a building. If you keep the file in your plugins folder, those lots will look fine. But if you remove the file, you will end up with a brown box on any lot that's using the model for an invisible building.

[Turnstyled 539]

@rivit

Thank you so much! I'll leave the PLEX and NAM directories alone. I'm aware that all PLEX files are now unsupported. With BLaM Blank, the overriding file is the .SC4Model of the SPAM abcvs 1x1 and with the SPAM abcvs 1x1, the overriding file is the .SC4Model of the SPAM abcvs 2x1. @BartonThinks Thank you as well for the information about BLaM Blank!

I'm not certain that I understand what you had said regarding the flags. The overriding file is dayopeace.dat. I deleted the American flag .dat file from my plugins directory and I rescanned. There were then no issues in my Maxis Files subdirectory. 

Regarding what you said about the Girafe poplars, if I pick one or the other, will that mean I'll have to choose between using them as ploppables or and keeping them as a dependency? Or would it not matter because they have the same IDs?

Thank you again!

Edit: I also noticed something else. The Area 5.1 Air Force Base No Exclusion mod was being overridden by the Rush Hour Rewards Unlock mod. Thus, I had appended "z_" to the directory containing the Area 5.1 Air Force Base No Exclusion mod to have it load after the Rush Hour Rewards Unlock mod. After a rescan, the override was no longer present, but Area 5.1 was not present in the Rewards menu.

[BartonThinks 3,416]

29 minutes ago, Turnstyled said:

@rivit

Thank you so much! I'll leave the PLEX and NAM directories alone. I'm aware that all PLEX files are now unsupported. With BLaM Blank, the overriding file is the .SC4Model of the SPAM abcvs 1x1 and with the SPAM abcvs 1x1, the overriding file is the .SC4Model of the SPAM abcvs 2x1. @BartonThinks Thank you as well for the information about BLaM Blank!

I'm not certain that I understand what you had said regarding the flags. The overriding file is dayopeace.dat. I deleted the American flag .dat file from my plugins directory and I rescanned. There were then no issues in my Maxis Files subdirectory. 

Regarding what you said about the Girafe poplars, if I pick one or the other, will that mean I'll have to choose between using them as ploppables or and keeping them as a dependency? Or would it not matter because they have the same IDs?

Thank you again!

Edit: I also noticed something else. The Area 5.1 Air Force Base No Exclusion mod was being overridden by the Rush Hour Rewards Unlock mod. Thus, I had appended "z_" to the directory containing the Area 5.1 Air Force Base No Exclusion mod to have it load after the Rush Hour Rewards Unlock mod. After a rescan, the override was no longer present, but Area 5.1 was not present in the Rewards menu.

For Girafe's Poplars, I've double-checked the files and confirmed what you said -- that is, the mayor mode models and lot editor models are identical and share the same ID codes. This means you should be able to safely delete one set of the model files.

The important thing is that you don't want to delete any of the mayor mode or lot editor exemplars. The mayor mode exemplars are "Grfe_poplars_evergreen.dat" and "Grfe_poplars_seasonal.dat". The lot editor exemplars are included in the same folder as the lot editor models. They use the following file name convention: "Grfe_poplar_A_autumn[TGI VALUES]"

You will need to keep the mayor mode exemplars if you want to keep using the ploppable flora in the game. You will also need to keep the lot editor exemplars if you want to use the poplar props as a dependency. That is, if you have any lots that use Girafe's poplars as props, you will need the lot editor exemplars to avoid brown boxes. However, so long as you keep these exemplars and one set of the model files, you should be a-okay.

One final and probably unnecessary note -- Just in case you're planning to try this with some of Girafe's other files, it's worth keeping in mind that a lot of his flora use different models for the mayor mode flora and lot editor props. The poplars are one case where the models are identical and use the same TGIs, but I believe they're an exception to how Girafe usually handles his files. (As I mentioned, this is probably an unnecessary note, since you only seem to be using the SC4DataNode to identify redundancies, and it wouldn't tell you to get rid of two different models.)

[Turnstyled 539]

@BartonThinks Thank you so much! I know that we have the Trixie emoticons, but I'm into typing out "thank you" to people.

Edit: It just popped in my mind that the reason that the Rush Hour Rewards Unlock mod overrides the Area 5.1 Air Force Base No Exclusion mod is probably because both mods unlock Area 5.1. The Rush Hour Rewards Unlock mod probably doesn't unlock the Air Force Base because the Air Force Base is "DLC." I don't know why it doesn't unlock the Space Port, but I can live with it.

Edit 2: I was typing up a response to a good friend of mine because she had just woken up, so I couldn't really focus on my first edit. The Area 5.1 Air Force Base No Exclusion Mod does NOT make Area 5.1 readily available if it is not already unlocked. Nonetheless, all vanilla rewards are unlocked except for the Air Force Base and the Space Port if I have Rush Hour Rewards Unlock loaded AFTER Area 5.1 Air Force Base No Exclusion. If I have Rush Hour Rewards Unlock load BEFORE Area 5.1 Air Force Base No Exclusion, then Area 5.1 is not available if it has not already been unlocked. I think that I can work out the logic to that in my head right now.

  Edited May 24, 2021 by Turnstyled  
Edited my first edit due to wording and added a second edit

[Turnstyled 539]

I wanted to provide a quick update regarding my plugins folder journey. I got rid of my SPAM_abcvs_growable_addon folder and my SPAM_Irrigation_Canals folder because they seemed to be problematic. I wasn't using them, anyway. It was just that at the start of it all, I had read the amazing second post in the thread titled "Indiana Joe and the Quest for the Ultimate Plugins Folder" and I had downloaded all SPAM-conforming content from the PLEX on the STEX.

@BartonThinks Each time that I get rid of something that overrides BLaM Blank, a new item overrides BLaM Blank. Based on your explanation on why I might need BLaM Blank, is that expected? Thank you so much.

Also, while the crowd that wants Area 5.1 in its cities is a very small minority, Area 5.1 Air Force Base No Exclusion only works when Rush Hour Rewards Unlock overrides it. I'm okay with that.

Now, the only other overriding going on is within my NAM directory. I'd call this is a success! @rivit and @BartonThinks, thank you again for your help!

  Edited May 29, 2021 by Turnstyled  
Correct "exclamation" to "explanation"

[BartonThinks 3,416]

10 minutes ago, Turnstyled said:

I wanted to provide a quick update regarding my plugins folder journey. I got rid of my SPAM_abcvs_growable_addon folder and my SPAM_Irrigation_Canals folder because they seemed to be problematic. I wasn't using them, anyway. It was just that at the start of it all, I had read the amazing second post in the thread titled "Indiana Joe and the Quest for the Ultimate Plugins Folder" and I had downloaded all SPAM-conforming content from the PLEX on the STEX.

@BartonThinks Each time that I get rid of something that overrides BLaM Blank, a new item overrides BLaM Blank. Based on your exclamation on why I might need BLaM Blank, is that expected? Thank you so much.

Also, while the crowd that wants Area 5.1 in its cities is a very small minority, Area 5.1 Air Force Base No Exclusion only works when Rush Hour Rewards Unlock overrides it. I'm okay with that.

Now, the only other overriding going on is within my NAM directory. I'd call this is a success! @rivit and @BartonThinks, thank you again for your help!

@Turnstyled

So, I'm not 100% sure, but I'm guessing there might be a number of redundant BLaM Blank files within your plugins. The original description for the file said that other users could use the blank for their own projects, and I think it was allowed to be repackaged and freely shared by users who wanted to use it to create their own content. So the BLaM Blank might be included in a number of downloads within your plugins folder.

Honestly, I wouldn't worry too much about weeding the redundant BLaM Blanks out -- the size of the files would be so small as to have a negligible effect on the performance of the game, and if the Blank has been packed into larger DATs, it might be impossible to completely get rid of the duplicates without opening up files in Reader and eliminating specific entries.

(Note: I came to SC4 and Simtropolis late, so I wasn't around when this file was being created/used, so without looking at the files in question, this is just a best guess at what's happening. There might be another reason why you're finding multiple redundancies.)

[Turnstyled 539]

9 minutes ago, BartonThinks said:

So, I'm not 100% sure, but I'm guessing there might be a number of redundant BLaM Blank files within your plugins. The original description for the file said that other users could use the blank for their own projects, and I think it was allowed to be repackaged and freely shared by users who wanted to use it to create their own content. So the BLaM Blank might be included in a number of downloads within your plugins folder.

Honestly, I wouldn't worry too much about weeding the redundant BLaM Blanks out -- the size of the files would be so small as to have a negligible effect on the performance of the game, and if the Blank has been packed into larger DATs, it might be impossible to completely get rid of the duplicates without opening up files in Reader and eliminating specific entries.

(Note: I came to SC4 and Simtropolis late, so I wasn't around when this file was being created/used, so without looking at the files in question, this is just a best guess at what's happening. There might be another reason why you're finding multiple redundancies.)

@BartonThinks Thank you so much. That makes perfect sense. If I understand correctly, it still makes sense to have the BLaM Blank in my plugins directory in case an asset needs it. Is that correct?

[BartonThinks 3,416]

Just now, Turnstyled said:

@BartonThinks Thank you so much. That makes perfect sense. If I understand correctly, it still makes sense to have the BLaM Blank in my plugins directory in case an asset needs it. Is that correct?

I think the safe move is to leave it there. You might not need the original BLaM Blank if it's been repackaged into another download somewhere in your plugins folder, which sounds likely based on what you're describing. But without comparing individual files to confirm whether or not that's the case (which, definitely not worth the time for this many bytes of plugin space), the safest thing to do is just leave it there.

[Turnstyled 539]

1 minute ago, BartonThinks said:

I think the safe move is to leave it there. You might not need the original BLaM Blank if it's been repackaged into another download somewhere in your plugins folder, which sounds likely based on what you're describing. But without comparing individual files to confirm whether or not that's the case (which, definitely not worth the time for this many bytes of plugin space), the safest thing to do is just leave it there.

I see. Thank you so much.

[Turnstyled 539]

Thanks to the responses to me in this thread from @BartonThinks, I'm starting to understand this all much better. I readded the SimPeg Irrigation Canals into my plugins directory and I saw that there were two .SC4Model files and an .SC4Desc file that were overridden by SIP_Props_v-b101.dat. The files being overridden are a SPIP-Transition and a SPIP-Valve. If I understand correctly, SIP_Props_v-b101.dat includes data of some sort that is in those three overridden files. Therefore, those three files loading has created a redundancy and it's nothing to worry about.

I have another question. In the SPAM base pack, there are an I-r5_4x6_Elevator_593083cb.SC4Lot and a I-r5_4x6_LargeElevator_593083cb.SC4Lot. I see that those files have the same ID and of course, one overrides the other. If I keep both in my plugins directory, is there a chance that I'll see both in my city or will I only ever see one of them?

Edit: I think that there's a chance that I'd see both because I also readded the SPAM abcvs growable addon directory into my plugins folder and the 2x1 .SC4Model file and the 1x1 .SC4Model file both have the same ID. Also, these files were overriding BLaM Blank and right in the readme is this:

I'm referring to the highlighted line.

I've been working my brain so hard to try to figure this all out and I now feel very comfortable with it.

[rivit 3,404]

I would expect you to only see the one that loads last - would be shown as not overridden according to DataNode. You'd have to compare the lots in detail to know how they were different - it probably just comes down to one or more props or maybe one building taller than the other.

It's clear you're now quite on top of how it all works so its down to how particular  you want to be from here..

[BartonThinks 3,416]

14 minutes ago, Turnstyled said:

Thanks to the responses to me in this thread from @BartonThinks, I'm starting to understand this all much better. I readded the SimPeg Irrigation Canals into my plugins directory and I saw that there were two .SC4Model files and an .SC4Desc file that were overridden by SIP_Props_v-b101.dat. The files being overridden are a SPIP-Transition and a SPIP-Valve. If I understand correctly, SIP_Props_v-b101.dat includes data of some sort that is in those three overridden files. Therefore, those three files loading has created a redundancy and it's nothing to worry about.

I have another question. In the SPAM base pack, there are an I-r5_4x6_Elevator_593083cb.SC4Lot and a I-r5_4x6_LargeElevator_593083cb.SC4Lot. I see that those files have the same ID and of course, one overrides the other. If I keep both in my plugins directory, is there a chance that I'll see both in my city or will I only ever see one of them?

Whichever one loads last is the only one that will appear in your city. (However, based on the PNGs in the DAT files, it looks like both lots are identical, so no big deal.)

[Turnstyled 539]

@rivit @BartonThinks Thank you so much. I was working on edit to my previous post that I've now completed. If you refer back to it, you'll see that you've both proven me wrong.

7 minutes ago, rivit said:

It's clear you're now quite on top of how it all works so its down to how particular  you want to be from here..

Very true! 

[Flann 1,709]

Rivit,

I have a request for SC4DataNode -

Currently the Q column in the csv reports identifies whether an item is overridden or not.  Would it be possible to further distinguish between files which are not overridden and do not override another file? Or can you perhaps think of another way to identify which files are overriding another file?  I am trying to track down duplicate textures in the NAM install.

 

Thanks,

Flann

 

Edit:

I think my idea would still be useful but I just wanted to share that I was able to get a start on this by using a pivot table:

 

[rivit 3,404]

@Flann,   I recognise that problem well

   I usually start with the Network Textures CSV view, then all textures are shown in override order and all the wealths represented in each file are summarised. This makes it easy to see which wealthings are actually used / or are necessary.  I usually color all NAM files in one color say -green,

example

So the Q column contains an = if it is the last loading, and a space if its overridden. here we see 1404 shows up 4 times, in SimCity, NAM RHD, MGBs Grassmod and finally my own Road versions.  the cosmetic mod added wealths 456 which were not in SimCity. notice MGbs doesnt, and I had to in order to suppress the NAM ones. The one above it 120E only exists in NAM.

To find all NAM entries that are not overridden I would filter filename for NAM, and Q for =. Similarly NAM ones that are overridden NAM + Q=space.

To find all those that do not override earlier ones depends heavily on load order so you would have to make sure NAM loads directly after SimCity (as I do) and then look for those that are in SimCity and/or NAM sets, with Q = "=" on the NAM ones - not quite as elegantly or easily put. 

By fiddling load order a bit it should be possible to also double check US/EU overrides in a similar way.