tool SC4DataNode Support

[CorinaMarie 29,621]

59 minutes ago, rivit said:

FYI Cori, this entry from the SFX.lua actually doesn't exist - however 0x0a55b832 does and it is a crowd noise - seems like a typo but could have been a fast fix.

Nice find. I've made a note of that.

CB and I have a separate Lua project (where we've extracted them all and can load them in the same order, but calling them externally) so if we ever have something to release, we'll add that in as a fix. Ofc, a quick patch of just that one file could be created and tossed on the STEX, but since we've all lived without it all these years, I'll just save it for someday. (If anyone else wants to do the fix and post it, that's fine too.)

[carlosmarcelo 979]

On 17/02/2020 at 1:54 AM, rivit said:

In recent work to upgrade SC4Datanode, version 1.0.10 arriving soon, to see more dependencies

I apologize for interrupting the great chat about sounds and simlish, but when I read about you working on version 1.0.10, I remembered that I saw something strange that seems to be a bug. 
When running DataNode in my plugins folder (3,083 Files, 430 Folders, 1.04 GB), after Rescan finishes and shows the list, the beginning seems ok, following an alphabetical order, but continuing on the list, everything seems mixed up. 

So far so good. 

Then it lists directories that, theoretically, he has gone through before and the list is all mixed up. 

In these example images, in line seq # 16 the folder "entretenimento/parques/piscinas" is resumed in line 2037.
Line 2034, folder "zonas/residencial" is resumed on line 2289.
Line 2035, folder "~dependencia" is resumed on line 2290.

Is this a bug or is there something that can explain this visualization that I am not understanding? Does SimCity really read plugins in this seemingly mixed way? 

 

[jeffryfisher 2,176]

All .DAT files load after all atomic files. Directory structure also affects load order.

[carlosmarcelo 979]

12 hours ago, jeffryfisher said:

All .DAT files load after all atomic files. Directory structure also affects load order.

Thank you! 

I decided to make a lean plugins folder for testing (above image). So I was able to confirm your statement.

Taking advantage of the moment, I tested what it is like loading files when symbols are used at the beginning of the name.  

     

It very curious to have some symbols being loaded before and others after (left image), different from the display of windows folders (right black image). 

That is, instead of having several z, zz, zzzzzzzz folders, we can use symbols like "~", "_" among others. I have used it for my final "~" folders and it has worked very well. 

[CorinaMarie 29,621]

2 hours ago, carlosmarcelo said:

That is, instead of having several z, zz, zzzzzzzz folders, we can use symbols like "~", "_" among others. I have used it for my final "~" folders and it has worked very well.

That's very useful to know.

In another thread a year or three ago I remember someone (but not which particular person) uses numbers as folder prefixes so that both the game and Windoze show the same order. Like this for folders:

00010 - Early things to load
00200 - Later things to load
09000 - Even later things to load
80000 - Currently last things to load

^ Note the preceding padded zeroes to the left in those names. And, ofc, they had real names which actually described the content of said folders. That, too, means there's no need for the z's and the counted number of underscores.

[rivit 3,391]

Well here we go, a new version of SC4Datanode v1.10. This turns out to be a fairly major expansion of the capabilities (closer to what I had originally intended) of the program and will cough up a few more missing items than before.
 
 Feature Changes:
 1) The detection of dependencies is now very extensive - all Texts, PNGs (icons), sound files, UIs, Panels and various miscellaneous TGIs, GIs and dependent exemplars all now show. This will lead to finding more things missing but, equally, more insight into how things link together.
 
 For instance: all SC4 Sailing craft have a missing AB sound file, the Flying automata mostly have missing Occupant sounds, all Civic buildings and their derivatives have missing Activation sounds. All of these were probably the result of decisions late in the SC4 development process and they just pulled the sound files. In my plugins I have also discovered a series of buildings by one author missing a query UI, and several agricultural lots missing Farm Fields. So I need to go find those somewhere. None of these things stopped the game working - SC4 is extremely tolerant of missing references - they were never obvious and what you can't see or hear isn't easily noticed or missed.
 
 2) there is a new view and report for all LTEXT exemplars - the report will not come into excel clean but only one LTEXT seems to screw this up - the game credits. This is a fun browse (there are some extremely clever ones about) and it also shows how game data can be embedded into a LTEXT. Note that the game has a particular way of dealing with localization of language outside the Locale File - it does so by incrementing the IID of certain LTEXT entries. So you may see several LTEXTs in different languages, even though in-game you mostly see only one. This is most obvious in MAXIS plugins.
 
 3) Some of the individual properties have more extensive value descriptions - most notably Occupant Group.

 4) It is possible to suppress the display of Ghost Highway stubs in the Missing view by way of a preference setting. These exist as the NAM code base migrates from puzzle pieces to flex models (they are exemplars without models) and will eventually disappear, but for now this will suppress about 4100 entries in the Missing view allowing you to concentrate on what's left. They still do go to the log. Keep in mind too that a vanilla SimCity itself will display about 70 missing items.

 5) The program will keep the screen real estate layout as you first set it rather than resetting it every time you change views - I finally got round to fixing this.

 6) I found a few minor code tweaks to improve speed.
 
 The beta can be found here on my archive and I would like a few people to give it a fly to see I haven't broken it in some way. It should behave as it did previously apart from what I mentioned above. I look forward to some feedback.
 
 TODO: I still need to update the documentation to match this blurb.
 

[xxdita 1,097]

Would there be any way to flag a file that has a Float32 property with a Rep Count of 1? 

I know this is extremely specific, but this is what has proven to be the culprit of many MAC compatibility issues. From transit station capacity bugs to now CTD's with the 64bit release. 

[rivit 3,391]

continuing on from the MAC thread, I have I found 2 HSR NAM stations with those NReps=1 but note that the exemplars for those have nearly all properties that should be singular values encoded as array types so FLOAT32A instead of FLOAT32, UINT32A instead of UINT32 etc. These have almost certainly come from a very old original that was a text exemplar.  Reader doesn't allow these sorts of errors any more when adding new properties but it doesn't check/flag existing ones as wrong.  So while the floats give problems on the MAC the others could too.

In fact its a bit ambiguous for us to see which properties can/must only have one value at a time. So I see now how these have slipped by SC4Datanode - because they are encoded as Array properties a count is expected, and SC4Datanode only enforces a correct type, not whether it should only be singular.

What I will try however is the following fix - now that I know how it happens - putting an extra flag on the properties that should always be singular, and then marking these as in error if they're not.  I'll start with the floats first since they seem to be the most problematic, potentially giving values that are not valid floating point numbers. (NANs)

I think we can get these...

[xxdita 1,097]

The UNIT32A's present no problems for MAC, as these are found in the reported spa downloads in the files that worked. 

But as you say, Reader will save in Float32 now, not Float32A, so I really can't be sure whether that's an issue or not. It's too late tonight for me to go through my entire plugins folder looking for lots that may have Float32A's, but I would be surprised if I'd managed to only download and install 4 such lots. 

A way to toggle between models, buildings, and lots would be appreciated. 

[rivit 3,391]

Ok, I've done some work on the new scanning check and in respect of exemplars that exhibit wrong use of Array types when singular values are more appropriate I have found about 1600 of them. This is many more than I had expected, but even accounting for false positives its a lot. It appears that way back in the early modding period this was the standard form. e.g. MBEAR productions almost all are array types. Checking SC4dat_1 I find none of them. There are also others in other exemplar types but I still need to check for false positives.

When it comes down to FLOAT values I have found the following three frequently

 in Buildings:
  27812811 Filling Degree                                this property is unknown in Reader and may be from the original SC4 before deluxe hence defunct
 E90E25A2 Transit Switch Entry Cost            |
 E90E25A3 Transit Switch Traffic Capacity  |  both of these are made FLOAT32A with 1 rep by SC4Tool when Transit Enabling.
 
 in addition to the Slope properties you identified in LotConfigs.
 
  I imagine these could also create havoc so this could be explicitly tested. If they always work we may have a different problem.

I should be able to release a version for others to use tomorrow.

[CorinaMarie 29,621]

4 hours ago, rivit said:

27812811 Filling Degree 

Isn't this what PIM-X adds as a base from which to calculate the number of occupants like residential capacity or number of jobs?

[twalsh102 1,601]

@carlosmarcelo.

you might want to take a look at the discussion on file load order from 2010 (and later) over at SC4D .  It was discovered back then that file load order does not follow Windows list order.   Further, the *nix file system uses a different list order than Windows, that doesn't follow the ASCII order either.  File load order actually follows the order used by the extended ASCII table (there is a copy of this table included in the discussion). There is also an extended explanation of what @jeffryfisher posted. 

I think as a result of this, the common practice became stick to letters, numbers, and underscore in folder names.  This i makes it easy to remember load order without having to look at a table to determine what's loading when.  Part of the problem with changing this in your Plugins directory is that there have been so many files released that are required to be loaded in a specific order (either before all other custom content (for example:  a__CAM), to loading after everything else ( for example:  z___NAM)) in order to work correctly.

While, you could certainly change all the installed folder names such that they follow the naming conventions you use you use in your Plugins folder, if you ever run into problems, how is anyone going to be able to help you if you use unique naming conventions?  Are you going to be able to remember all the folders that were installed using z's and underscores, that you changed to fit your naming conventions, in order to help determine if problems are caused by load order, or some other reason?

[twalsh102 1,601]

Forgot to include the link to the discussion.  It's https://www.sc4devotion.com/forums/index.php?topic=10542.msg414566#msg414566.

[twalsh102 1,601]

If you really want to confuse the issue, you can use characters from other ASCII character sets that include foreign language and other special characters.  Then you have a situation where characters above Alt 126 (all the way up to Alt 0247) aren't loaded in any particular error.  Please peruse the included spreadsheet to see what I mean.  Any Alt numbers that are skipped are either lower case versions of some character , or a graphical in nature.  The game uses the order of uppercase versions of a character, regardless of what is actually used.

Load order.xlsx

[twalsh102 1,601]

1 hour ago, CorinaMarie said:

Isn't this what PIM-X adds as a base from which to calculate the number of occupants like residential capacity or number of jobs?

You are correct.  But don't forget that PIM-X was created some time after Reader was created and Filling degree is not a standard Maxis Property.   Since (as far as I know) only PIM-X uses this property, my guess is that the creator of Reader hasn't felt a need to further define it in Reader in any updates.

Just to be 100% correct, Filling degree not only affects capacity, it also affects values for Pollution at center, Power used, Water used, and Building Value.

One of the problems with the Maxis LE has always been that although Maxis uses some factor to compensate for the fact that a model almost never 100% fills its LOD, it uses some fixed value to calculate this compensation.  But since the value is fixed, it's almost never correct.  I don't think anyone has been able to determine exactly what factor the Maxis LE uses.  This was the genesis of Filling Degree in Pim-X.

[xxdita 1,097]

8 hours ago, rivit said:

Ok, I've done some work on the new scanning check and in respect of exemplars that exhibit wrong use of Array types when singular values are more appropriate I have found about 1600 of them. This is many more than I had expected, but even accounting for false positives its a lot. It appears that way back in the early modding period this was the standard form. e.g. MBEAR productions almost all are array types. Checking SC4dat_1 I find none of them. There are also others in other exemplar types but I still need to check for false positives.

When it comes down to FLOAT values I have found the following three frequently

 in Buildings:
  27812811 Filling Degree                                this property is unknown in Reader and may be from the original SC4 before deluxe hence defunct
 E90E25A2 Transit Switch Entry Cost            |
 E90E25A3 Transit Switch Traffic Capacity  |  both of these are made FLOAT32A with 1 rep by SC4Tool when Transit Enabling.
 
 in addition to the Slope properties you identified in LotConfigs.
 
  I imagine these could also create havoc so this could be explicitly tested. If they always work we may have a different problem.

I should be able to release a version for others to use tomorrow.

Filling degree is used by PIMX to set all stats in a building exemplar. It has no defined purpose to the game, so is ignored. For growables, lack of a filling degree indicates possible incompatibility with CAM, or skewed stats like garbage production, depending on the size of the building. 

It is these 2 transit switch properties with a Rep of 1 that cause capacity bugs for MAC users. Just as the Rep 1 for the Slope properties causes the CTD for MAC 64. These are the ones I would like to see flagged in some way by DataNode, so that we can get ahead of the situation if at all possible, especially on the CTD front. 

Whether it's the Float32A or not, I can't say. But since the transit bug was first detected on MACs, the solution has been to change the Rep count. I may need to do some testing with the version of Reader available on the LEX and STEX to see if it changes the Float32A's to Float32's as the newer version does. 

I'm not aware of any other program that allows for transit enabling than SC4Tool, so it's no surprise that most stations have this error. But I have no idea how the Slope issue is happening. It isn't standard for PIMX, and I just don't believe that anyone is setting these values intentionally. So it has to be one of the tools being used. The spa lots have no transit enabling, but SC4Tool may have been used to create the ReadMe and for the dependency scanner. I don't know that this would alter the file though?

[carlosmarcelo 979]

On 22/02/2020 at 10:02 AM, CorinaMarie said:

I remember someone (but not which particular person) uses numbers as folder prefixes so that both the game

5 hours ago, twalsh102 said:

Then you have a situation where characters above Alt 126 (all the way up to Alt 0247) aren't loaded in any particular error.

I did these tests by curiosity. This question of order, for me, is not a problem, fortunately.  First because I try to keep the plugins folder small, then if any plugin requires a lot of dependencies and a special order to load, I either don't use it or I look for another one. 
I use "~" only to flag folders that I consider special, such as dependencies and game mod like removing the minimize button, parks aura etc. By chance I found out that they load the last one, I was happy. 
The idea of numbering folders makes more sense to me than putting z, zz, zzz, but that's just a matter of taste, preference. 

 

[xxdita 1,097]

20 minutes ago, carlosmarcelo said:

I did these tests by curiosity. This question of order, for me, is not a problem, fortunately.  First because I try to keep the plugins folder small, then if any plugin requires a lot of dependencies and a special order to load, I either don't use it or I look for another one. 
I use "~" only to flag folders that I consider special, such as dependencies and game mod like removing the minimize button, parks aura etc. By chance I found out that they load the last one, I was happy. 
The idea of numbering folders makes more sense to me than putting z, zz, zzz, but that's just a matter of taste, preference. 

Loading order is generally only important when conflicts are possible or likely, and where one file is meant to override another. For overrides of Maxis exemplars, as long as you only have one installed for any given exemplar, then loading order shouldn't be an issue. 

But loading order is very important for overriding existing custom content while using the same TGI's, or when you know that there are other mods using the same Maxis exemplar(s) and you want to prevent conflicts. 

As a creator, I can tell you that there is enough going on during the development and testing of a single mod. There is just no way to know all of the possible conflicts beforehand. But the more complex your mod is, and the more Maxis exemplars it contains, the more likely conflicts become. To avoid these conflicts, the easiest solution for a creator is to recommend installation to a z_ folder. 

[rivit 3,391]

A new (beta) version of SC4DataNode v1.11 is on my archive.

This incorporates the discussion above and will flag (comment only) ArrayType uses when singular values were expected for all types.

In particular FLOAT32A uses when FLOAT32 was expected will be designated as Faulty and will show Blue in the Faulty View. I have suppressed the Filling Degree Property based on the comments above*

To easily find the problem children, run the CSV report for Faulty, slurp into Excel or the like and filter for 'ArrayType Errors' in the ErrorType column.
TGI, Exemplar Type, Name and File Path are given so it should be easy to find them. Use the Faulty view to look at the details.

There are more transit enabled buildings around that you might expect! In my collection there were 87 exemplars (29 of my own making) with these errors in Transit Switch Cost and Capacity. Mine were all done with SC4Tool.

Of production stuff affected are my extra Diggers/Raisers, TSR Street Lots and the STR Tunnel - I'll fix these asap.
NAM is clean of these error as far as I can see except for 2 HSR stations and the Elevated Bus Station in Z___NAM\Transit Lots.

Once again, feedback for new weirdness/false positives would be appreciated.

~~~~~~~
* Does PIMX use templates or hard coding to maintain values in exemplars? If templates then it could be fixed easily.  

[xxdita 1,097]

I believe that is all handled in the new_properties.xml. 

None of the lots I've made with PIMX have any of these Slope issues. Nor do transit lots created with PIMX have the transit switch issues. 

We know that SC4Tool creates the transit switch problems, but I don't know if that can be fixed at this time (source code availability, original creator's SC4 activity, etc). There may very well be something like an xml that could be updated. 

I would like to get to the route of the Slope issues though. 

[CorinaMarie 29,621]

1 hour ago, xxdita said:

There may very well be something like an xml that could be updated.

In the System folder of the SC4Tool folder, there are some XML files and in particular: new_properties.xml. Looking thru that and it appears when there is a Count= entry that might correspond to the number of repetitions. Many entries do not have that Count= part and so it could very well be the default of 0 or 1 for certain things is being handled internally by the code.

[xxdita 1,097]

2 hours ago, CorinaMarie said:

In the System folder of the SC4Tool folder, there are some XML files and in particular: new_properties.xml. Looking thru that and it appears when there is a Count= entry that might correspond to the number of repetitions. Many entries do not have that Count= part and so it could very well be the default of 0 or 1 for certain things is being handled internally by the code.

Or the internal code has defaults that are overridden by the xml, and apparently xsd files. 

I've PMed Andreas at SC4D. Perhaps he'll have some insight. 

[markussaage 762]

@rivit Thanks for releasing a new BETA of your awesome tool.

I checked the files that I use to have in my Plugins, and as you already mentioned, there are a lot files out there with that FLOAT32A problem, especially in the "Transit Switch Entry Cost" and
"Transit Switch Traffic Capacity" value. Honestly, I did not know that this can lead to problems at all. But the majority of files with this problem is in the NAM. And yes, I use your STR Tunnel,
so I fixed that already.
Just to be sure, to fix it we just need to replace the Rep=1 with a 0 (zero), using iLive's Reader, right? I've seen the Rep=1 a couple of times, and the Reader says, the default is 0. Wondering how it changed to the State = 1?!

Anyway, thanks for all the work you put into this tool. It's a lifesaver!

Kind regards!

[xxdita 1,097]

Just now, markussaage said:

@rivit Thanks for releasing a new BETA of your awesome tool.

I checked the files that I use to have in my Plugins, and as you already mentioned, there are a lot files out there with that FLOAT32A problem, especially in the "Transit Switch Entry Cost" and
"Transit Switch Traffic Capacity" value. Honestly, I did not know that this can lead to problems at all. But the majority of files with this problem is in the NAM. And yes, I use your STR Tunnel,
so I fixed that already.
Just to be sure, to fix it we just need to replace the Rep=1 with a 0 (zero), using iLive's Reader, right? I've seen the Rep=1 a couple of times, and the Reader says, the default is 0. Wondering how it changed to the State = 1?!

Anyway, thanks for all the work you put into this tool. It's a lifesaver!

Kind regards!

As far as we can tell, the transit switch properties have been set by SC4Tool, which may very well be the only real option for transit enabling lots. I haven't been able to reproduce the changes to the Slope properties yet, with either SC4Tool or PIMX. But that still leaves a number of possibilities remaining. 

It is important to note that the issues we're looking into are only a problem for MAC users, especially the new 64bit version. It is my goal to get ahead of this issue as much as possible, so that we as a community are ready to embrace new & returning players that the new 64bit Aspyr version may bring. 

[markussaage 762]

1 hour ago, xxdita said:

As far as we can tell, the transit switch properties have been set by SC4Tool, which may very well be the only real option for transit enabling lots. I haven't been able to reproduce the changes to the Slope properties yet, with either SC4Tool or PIMX. But that still leaves a number of possibilities remaining. 

It is important to note that the issues we're looking into are only a problem for MAC users, especially the new 64bit version. It is my goal to get ahead of this issue as much as possible, so that we as a community are ready to embrace new & returning players that the new 64bit Aspyr version may bring. 

Hi,

just to be clear, the Rep count = 1 ain't a problem for the Windows version of SimCity4? Just asking, before I get into all the work fixing things that don't need to be fixed...

Kind regards!

[xxdita 1,097]

14 minutes ago, markussaage said:

Hi,

just to be clear, the Rep count = 1 ain't a problem for the Windows version of SimCity4? Just asking, before I get into all the work fixing things that don't need to be fixed...

Kind regards!

That's correct. Windows users can blissfully ignore this for now. No promises when we get 64bit bugs of our own though. 

[mawendir 53]

Hello there! I installed the tool to my computer. When I try to open it, I recieve this error;

SC4PTypeFromSTR() Strange PType = uint32

There's only "Okay" button to press, but when I do so, nothing happens and the error keeps popping up. I can only terminate it via task manager.

I have .NET Framework installed. My computer is running on Windows 10 with the latest update. Don't know what's wrong, I've tried couple of things but none of them worked. So, I'd appreciate a little help over here. Also, I could gladly share more info about my computer if needed in order to solve the problem, just let me know!

[rivit 3,391]

@mawendir - which version of SC4Datanode have you installed?  It shows in the top window margin.  Otherwise look at the fileinfo of the exe.

Are you using a locale other than English?

This error occurs while reading the Properties csv files, which are store in the C:\Program Files (x86)\Brainscrambler\SC4Datanode\Resources folder. These assume US English parsing rules so that may be the cause

This is the first time I've seen this error so thanks for mentioning it - we should be able to track this down together reasonably quickly.

[mawendir 53]

It's the version 1.06 but also tried 1.11 you shared here to see if anything changes, but nothing happened. I was using Turkish in my computer but after you mentioned about it, I downloaded and changed computers language to English if it fixes the error. So, the good news is, yes it actually did! Hope you can fix the problem so I don't have to change between languages Thanks for the quick reply!

[rivit 3,391]

Great, that tells me enough to be able to apply a code change to the program. Should have a new version to test soon.

 

OK a new version 1.12 which enforces US-en locale while reading property files can be found here on my archive.

I'm working on the premise this is enough to fix the problem but I may need to make the entire program US-en. Please report any weirdness - it should still show the date and time at the bottom right in your own locale form.

  Edited March 16, 2020 by rivit  
Update result