CoriBoom Cori's Lua Coding Explorations for Reader 1.5.4

[CorinaMarie 29,605]

Some fun with Lua scripting inside Reader 1.5.4

As some of you know @BartonThinks and @Raymond7cn did some poking into using Lua coding to automate some tasks in Reader 1.5.4. Raymond beeped me into the topic where I wrote a couple of lines of code to tweak what he created. The result is something which saved BT from several hours of manually modifying a few hundred exemplar names for his project.

Once I understood the possibilities, I was off and running with some code of my own by creating a Lua script which outputs all of the Properties lines and values of every Cohort and Exemplar in any given SC4 data file (that includes .dat, .sc4lot, .sc4desc, and any other file Reader can display). The output is a text file and is formatted into columns just like you see in Reader (with the exception that I re-arranged the columns so the two most important (IMO) are together on the right side.

I've already had a viable use for this inasmuch as @Sunrader needed a Raiser Lot set to a non-standard height for modifying SC4 terrain for later importing into The Sims 2 game. While chatting with @Cyclone Boom, he was sure it was set somewhere in the LotConfigurations Exemplar. So, I loaded up the 7.5m, the 8m, and the 10m Raiser Lots included in NAM and created text reports of each. Then with the help of WinMerge, I quickly found which data rep of which property was the likely controller for the height amount.

While I did simply view the output on my screen, CB made some images comparing the differences. Here's what those look like:

The 7.5 meter Raiser Lot:

 

The 8 meter Raiser Lot:

 

The 10 meter Raiser Lot:

 

So, I was able to deduce that it was indeed the 5th rep of property 0x88edc900 (LotConfigPropertyLotObject) which needed tweaked. I changed that to 0x0007C000 and it worked as expected. What I'd noticed was the hex thousands position has value 8 for the 7.5 meter raiser. Said 8 being exactly half of 16 made it likely that was the fraction of a meter while the more significant digit to the left of it was the number of whole meters. 75% of 16 is 12 and that's how I decided to try C in that position. I was pleasantly surprised it did actually work. (Tested in the game using the TerrainQuery option.)

Now, this topic isn't about that specific tweak at all. I wrote the above to showcase one of the potentially many possibilities for having the SC4 information in a pure text form.

 

Cori's new Lua code as a pretty picture:

Here's a pretty picture CB made of my new code so you can see the colors Notepad++ uses to aid programming:

^ I've attached this code in text form at the bottom of this post in case any of y'all want to take it for a test drive.

 

Applying the 4 GB Patch to Reader.exe

Before we get started running Lua code in Reader 1.5.4, it's a good idea to apply the 4 GB Patch to it. (While this might not be needed, it won't hurt anything and has the possibility to help.) Rather than run the 4gb_patch.exe with administrator rights, I copied the file to be patched to its own folder, applied the patch there, and then copied the modified file back into its home in Program Files (x86). Here's the steps I followed:

Locate the Reader.exe file:

^ Note the last modified date is 2 October, 2015. (Not important to know, but I'll reference the date again in a wee little while.)
 

Copy Reader.exe to a new folder that does not require admin rights to edit files and also copy the 4gb_patch.exe file in as well:

^ You could make the new folder in your Documents folder if you feel more comfy doing it there.
 

Double-click the 4gb_patch.exe file and it'll pop up a file selection dialog window. Select the Reader.exe that's in the new folder and click Open:

 

After it instantly does its magic, you'll see the confirmation:

 

Next, copy the updated Reader.exe and paste it back into its original location. Windoze will notice there is already a file there with that name and give you some choices for what you want to do. Click the Copy and Replace option:

 

Since Windoze is really fussy about the Program Files folders, it's going to require a signed affidavit from God and his mother to allow this confirmation from you.

Click continue:

 

And here you can see it has an updated modified date:

While we are chatting about the 4 GB Patch, this is the very same little program that is used to fix SimCity 4.exe to work with the new NAM 37 mod. There's a real nice explanation for that in the context of the game. That particular post is for the Steam version, but just as above and in that linky it's the same general steps.

 

Personalizing the Code before running it the First time.

There is one thing you MUST do before using my Lua code for the first time and that is to personalize it for your comp by telling it where you want the output text file to be created. Here's that part of the code, zoomed up:

^ That needs to be a valid path to a folder on your comp some place. The part to change is between the opening [[ and the closing ]]. You'll find this on line 59 of the code. Save the file after editing this.

Note: There is no need for a trailing backslash for the path. My code will slip that in between the path and the file name. Also, Lua actually uses forward slashes, but I've written the code so you can use the standard Windoze format and I flip them before creating the output file.

 

You can also, but don't need to, change the next line for the text output file name:

^ If editing this, it is the same as above inasmuch as you edit the part between the opening [[ and the closing ]]. While not imperative, it's good to give the file name a .txt extension as shown.

 

The next line can be left alone, but it's there if you have a techy reason to need all data returned as pure Hex instead of the more human readable XML lookup text Reader does by default (for properties and values it has names for). That's this line:

^ My notes in the attached source code file fully explain this. The options are 0 or 1. Basically the 1 option is for later code modification where I'll need to know the exact hex value for pattern matching and alteration. (More specifically, that's what I'll be using to check for certain OccupantGroups for use with our upcoming Expanded Tilesets mod.)

Ofc, save the file after you've finished editing it.

 

Running the Lua Script in Reader 1.5.4

To give my new code a workout, let's have it analyze the SimCity_1.dat file. ( No need to think small when putting it thru its paces. )

Run Reader 1.5.4 and open a file which has either cohorts or exemplars (or both) in it.  In Windoze 10, Barton said Reader needs set to Win 8 compatibility mode. (I'm running in Win 7 and didn't have to jump thru any extra hoops.)

Within Reader, I have to click something in the left-hand pane before it will enable the script option in the ribbon bar:

 

After clicking the script button to get the code area, I expand that white box where the code will go:

^ This step is not required.
 

Then copy the code from your text editor and paste it in the code section. Next, click the itty-bitty black triangle play button thinger to run the script:

 

And then as it runs, you can watch the gray display box under the code. For all normal files, it'll run so quick you won't see them as separate messages, but here's the progression when doing a huge file:

And now go open the output file in your fav ASCII Text Editor program. Remember word (or line) wrap will make it look goofy as hell, so be sure to turn that option off.

 

Conclusion

This is some exciting new territory here. While the above is simply to display the data in a useful manner, the possibilities for repetitive tasks being automated could be a significant time saver for many peeps. Let me know if the above code works for you. Without feedback, I'd likely assume I'm off in left field barking up an empty tree.

Special thanks to @Cyclone Boom for his continued help testing, proofreading, making images, and presentation advice. Without his dedicated help, this topic would be rather ho-hum looking. Also, thanks to @Raymond7cn for prodding me by creating the first script in Barton's thread.

 

Attachments

Even if you never plan to use the Lua Code, you might find the text output is a handy reference. Here are both versions:

SC_1.dat - Cohorts and Exemplar Properties - Cori's Lua Output - XML.txt.zip  (2.67 MB)

SC_1.dat - Cohorts and Exemplar Properties - Cori's Lua Output - Hex.txt.zip  (2.65 MB)

Additionally, I've tidied up the ilive_script.txt file (found in the installation directory of Reader 1.5.4):

ilive_script-vCori.txt  (19.4 kB)

And, ofc, my Lua Code:

Cori's Cohort and Exemplar Properties Script v1.00.lua  (24 kB) 
 

[BartonThinks 3,416]

Just adding a couple of small points to this...

1) When I first tried applying a script that @Raymond7cn & @CorinaMarie wrote in Reader 1.54, the script failed to run and the program repeatedly crashed. I was able to stabilize the program and run the script successfully by running the program in Windows 8 compatibility mode. Might be worth noting for anyone who's running the program in Windows 10.

2) In addition to the script that Cori included above, I think it's worth including the script that she and Raymond collaborated on to help me change exemplar names. The script performs a "find and replace" function on the "Exemplar Name" property, offering an easy way to batch modify specific parts of exemplar names, rather than the entire name. I've already found this script to be invaluable for modding prop packs for use in PIM-X.

To give a couple of quick examples of how this script might be useful to other people

• Distinguishing modded exemplars from originals. Let's say you download a prop pack, but want to modify the props in a particular way. For example, you might want to change the occupant size of all props to 1x1x1 so that they're easier to use in the lot editor. Rather than modding the original files, you copy the prop exemplars into a new file, generate new TGIs, and modify all of the occupant sizes. At this point, if you load the original and modded prop exemplars into PIM-X, both sets will use the exact same names, making it hard to tell them apart in the lot editor. This script offers an easy way to modify all of the exemplar names in one of the packs (e.g., changing all instances of "BSC" to "xBSC") so that the props are easily distinguished.
• Translating exemplar names from foreign languages. If you download files that contain exemplar names written in a language that you do not speak, this script can provide you with a way to translate sets of similarly named props in large batches. For example, BSC MEGA Props SG Vol 01 contains 32 different props that have exemplar names starting with the phrase "SG_Flora_Herbs." If you wanted to create a Polish-language version of this set, all of these exemplar names could be quickly translated so that they start with "SG_Flora_Ziola" instead. (Note: I'm relying on Google for this translation, so my apologies if I got it wrong).

Here's the script for anyone who wishes to try it for themselves:

function reader.main (this)   
    pos = reader:entries_GetHeadPosition()     
    while pos~=0 do
        entry,pos = reader:entries_GetNext(pos)
        flag = reader:entry_GetFlag(entry)  
        if (flag == 7) then
            exemplar = reader:ex_Decode(entry) 
            proppos = reader:exprop_GetHeadPosition(exemplar)  
            while proppos~=0 do
                exprop, proppos = reader:exprop_GetNext(exemplar,proppos)
                desc, descname = reader:exprop_GetDesc(exprop)  
                valuestr = reader:exprop_GetValueStr(exprop)
                if (descname == "Exemplar Name") then
                    cmTmpData = string.gsub(valuestr, "PHRASE TO REPLACE", "NEW PHRASE")
                    reader:exprop_SetValue(exprop, cmTmpData)
                    reader:ex_Encode(entry,exemplar)		
                end
            end
        end
    end
    reader:refresh()
end

[Cyclone Boom 16,876]

There is tremendous usefulness here already in what can be gleaned from exemplar data in pure text form.

As evidenced from the hole raiser lots to identify that specific hex value, the big advantage is to have data which is searchable locally inside individual and comparable text files. Using the ability for Notepad++ to view text files by searching and manipulating data there, it's possible for this to be used as a base foundation for any SC4 modding with exemplars (and potentially other data formats as well). Previously one thing lacking for me has been an ability to simply search through exemplar properties to locate certain values. This can now be achieved for the SimCity_1.dat file and any specific .dat which contains at least one Buildings (descriptor) or LotConfig exemplar.

It also has boundless potential for what can be accomplished from the automation side of things indeed, as BT eluded to from his topic. As Cori hinted above in the main post, one particular area to be explored next is the ability to parse the OccupantGroups data for the ability to edit them in batch. The idea here is for allowing it to happen for new tilesets with adding them in one fell swoop for any given .dat file, and really there are so many derived uses here for many (if not all) properties to be batch edited.

The brilliant thing is by saving time from a modding perspective, it then allows more productivity to be spent on actual development of new SC4 content, instead of the same old mundane repetitively boring tasks. From the Lua side of things, it's a framework which can be expanded and adapted to specific use cases too.

If anyone is interested to take the code for a test drive in Reader 1.5.4 and we'd really love to hear any feedback and comments.

Note: It does need to be 1.5.4 since trusty old 0.9.3 doesn't come with a local scripting console ability.

[Flann 1,709]

Wow this is really interesting stuff!  I for one am really interested in the idea of batch changing tilesets because I just did so for all of Jmyers homes, like 50 or so!  It took forever.  Not only did I change tileset, I also changed wealth which requires changing one property in the lot exemplar and 4 in the building exemplar.  Its a huge pain.  I would definitely help test something like that out.