Exemplar Names - Find & Replace?

[BartonThinks 3,416]

Hey, everyone. I've been trying to figure out a solution to this issue on my own but keep hitting a wall.

In a nutshell, I'm looking for a way where I can perform a batch "find and replace" on all of the exemplar names in a given DAT file.

The longer version is, I'm creating a large set of custom prop packs, and for each prop pack, I need eight different copies. The only two things differentiating these packs are: (A) each pack has a distinct set of group and instance IDs, generated randomly via iLive Reader; and (B) each pack uses a different naming convention for the exemplar names, so that I can distinguish between props in the PIM-X lot editor.

So far, I've created a number of these packs, and every time I do, I have to manually rename all of the props. The naming convention I use for the original pack is always "123456 [prop name]." The other seven prop packs follow the same convention, but with a different number at the beginning to identify the prop pack. That is, all of the props in one pack will be named "12 [prop name]," while another pack will use "34 [prop name]," etc.

Many of these packs contain a significant number of props, which means I'm left editing hundreds of exemplar names manually. I'm generally at peace with inane data entry tasks, but the amount of time this takes for larger packs is... substantial. It would save me a tremendous amount of time and effort if there was some way to quickly run a "find and replace" function on all of the exemplar names in a given DAT file and replace the numbers.

Does anyone have a solution to this problem? I've never scripted anything in my life, but is there a way to automate this with LUA or a third-party program? Alternatively, is there a function in Reader that I've missed or another tool that's capable of performing this task?

Thanks in advance. Any help would be greatly appreciated.

Note: Just in case there's any confusion, I can't use the batch edit function in the Reader's property analyzer, since that would overwrite the [prop name] part of each exemplar name.

[BartonThinks 3,416]

Alright, so based on this post from SC4D, I'm 99% sure that what I want to is possible through a LUA script. At least, it should be with Reader 1.4 -- but I feel like I might have 0.93? (Edit: Downloaded version 1.5.4, but from what I'm reading, it's maybe not the most stable program?)

https://www.sc4devotion.com/forums/index.php?topic=10488.0

Not going to pretend like I can make heads or tails of most of the script, but there's a find and replace example for S3D models in the post, and I'm hoping that will give me a template for what I need to do. I'll need need to figure out how to find and replace part of an ID entry and not the entire thing, but I'd assume that's possible with LUA.

I'm guessing I'll hit more than a few trouble spots with this, so if anyone has some quick tips for a LUA first-timer, I'd be happy for the help. Going through that SC4D thread is already making me dizzy.

[rsc204 21,867]

There is a bulk Property edit feature built into reader, but unless you want every Prop in the file to have an identical name then this is not going to help you.

As you’ve discovered, Reader also supports LUA scripting, which in theory could do some very wondrous things. However, I can’t tell you of a single example where someone made such a script. I’m no LUA coder myself, but if you’re a little handy in this area, it doesn’t seem all that complex code-wise. As usual, I’ll ping our resident expert on LUA, @CorinaMarie, since if anyone can provide more than my useless rambling, it’ll be her 😜

[BartonThinks 3,416]

12 minutes ago, rsc204 said:

There is a bulk Property edit feature built into reader, but unless you want every Prop in the file to have an identical name then this is not going to help you.

As you’ve discovered, Reader also supports LUA scripting, which in theory could do some very wondrous things. However, I can’t tell you of a single example where someone made such a script. I’m no LUA coder myself, but if you’re a little handy in this area, it doesn’t seem all that complex code-wise. As usual, I’ll ping our resident expert on LUA, @CorinaMarie, since if anyone can provide more than my useless rambling, it’ll be her 😜

Unfortunately, I'm not handy in this area at all. If I can get a foothold on something like this, I can usually learn the basics, at least enough to get by, but at the moment, I feel pretty overwhelmed trying to make sense of the example scripts. Same with the LUA manual. (That said, I couldn't make heads or tails of 90% of the exemplar properties I was looking at in the Reader a few months ago, so maybe this stuff's less confusing than it seems.)

It might be a case where the learning curve is a little bit too steep for me and it's actually faster to keep renaming all these files manually. But I couldn't tell you how much time I've already spent doing that, and I'm going to need to go through the process with a ton of additional files. I've also got a ton of seasonal flora props where I need to do some really repetitive modding, and I feel like there might also be a faster way to do that through scripting. 

Appreciate you pinging Cori. Hopefully she can let me know if this is worth my time or not. 

[CorinaMarie 29,899]

@BartonThinks

The little I know about it starts in @Raymond7cn's post here and then continues for a while. He figured out more about it than I did. As I noted in one of my later replies in that thread, I found this thread in which the OP is having difficulties and a following reply from Alex confirming the issues with scripting. So, based on @Tarkus's experience, I've not looked into scripting for Reader beyond what Raymond discovered.

[Raymond7cn 6,776]

7 hours ago, CorinaMarie said:

He figured out more about it than I did.

Cori can't be serious here,  Dear Barton @BartonThinks, Sorry that i may couldn't understand your project very well, In-short, i understand you have a dat file and many exemplars within. You want to Find and Replace all exemplar's names one time, Right? if so, I made a simple testing:

@CorinaMarie Hiya Cori, This is just a newbie's test, I thought we need to parse the value of 'exemplar name', after that there is also a function:

exprop_SetValue(exprop, value, [idx])

Set the value of a property.

Idx has to be filled it you want to change a value in an array

by the help of [if] and [while], we could achieve Barton's goal, Am i correct here?

Sincerely,

-- Raymond

[CorinaMarie 29,899]

10 minutes ago, Raymond7cn said:

Am i correct here?

The concept certainly looks good.

I believe another thing it'd need is for the script to open up more than one .dat file and to cycle thru those as well. However, considering how prone the versions of Reader past 0.9.3 (which are the ones where Lua can be used) are to corrupting the data, I find it difficult to find the motivation to look into this in detail.

[BartonThinks 3,416]

29 minutes ago, Raymond7cn said:

Dear Barton @BartonThinks, Sorry that i may couldn't understand your project very well, In-short, i understand you have a dat file and many exemplars within. You want to Find and Replace all exemplar's names one time, Right?

That is correct, except that I only need to replace one part of the exemplar name.

The original file would contain a set of exemplars. Each exemplar name would start with the numbers "123456." So the exemplar names would look something like this:

• 123455 Kspd Prop Barrel A
• 123455 Kspd Prop Barrel B
• 123455 Kspd Prop Barrel C
• 123456 Kspd Prop Shed A
• 123456 Kspd Prop Shed B
• etc., etc., etc.

I would then create a copy of this DAT file and assign new group and instance IDs to the props. At this point, I would need to change all of the "Exemplar Name" values. However, I only want to change the number at the start to a new value, such as "12." So the new exemplar name would all look something like this:

• 12 Kspd Prop Barrel A
• 12 Kspd Prop Barrel B
• 12 Kspd Prop Barrel C
• 12 Kspd Prop Shed A
• 12 Kspd Prop Shed B
• etc., etc., etc.

Hope that makes sense. (And sorry if I'm making myself redundant here.)

[CorinaMarie 29,899]

2 minutes ago, BartonThinks said:

I would then create a copy of this DAT file ...

Cool. That would eliminate the need to read more than one .dat file with the Lua Coding.

[BartonThinks 3,416]

13 minutes ago, CorinaMarie said:

The concept certainly looks good.

I believe another thing it'd need is for the script to open up more than one .dat file and to cycle thru those as well. However, considering how prone the versions of Reader past 0.9.3 (which are the ones where Lua can be used) are to corrupting the data, I find it difficult to find the motivation to look into this in detail.

I don't think that would be a big issue. I'm fine applying the script to individual DAT files. I just don't want to have to go through seven copies of a file containing 200+ props and have to change the numbers in the exemplar name manually each time. If a script can do that work, I'm more than happy to run the script seven times.

[Raymond7cn 6,776]

Ah, I see it, Barton, then it's more simple than i thought before, @CorinaMarie Hiya Cori, I searched from google and got this function "string.gsub(s, pat, repl [, n])", (Frankly i thought Barton's goal may needn't this now), May i ask that for running the string's function, Do i need some Lua's class database? I mean like JDK for java.

[CorinaMarie 29,899]

3 minutes ago, Raymond7cn said:

Hiya Cori, I searched from google and got this function "string.gsub(s, pat, repl [, n])", (Frankly i thought Barton's goal may needn't this now)

Once we have the data (the text in the Exemplar Name) in a string, the actual partial alteration that Barton needs is trivial. Key questions are:

1. Do we know if your code reads every exemplar in a single .dat file?
2. And do we know how to write the changed data back into each exemplar?

I honestly haven't ever looked into this enough to know.

 

1 minute ago, Raymond7cn said:

May i ask that for running the string's function, Do i need some Lua's class database? I mean like JDK for java.

All of the Lua commands should already be included within Reader's .exe in the same way the full 5.0 version is included in SimCity 4's (with the exception of a few things Maxis blocked from working). That is one of the beauties of Lua inasmuch as they've made it to be embedded in other programs and then accessible.

[BartonThinks 3,416]

Not sure if this helps, but this is the example script that performs a similar function for S3D files. (Except that it replaces the entire value.)

example 2
   function reader.main (this)   
      pos = reader:entries_GetHeadPosition()     
      while pos~=0 do
      entry,pos = reader:entries_GetNext(pos)
      flag = reader:entry_GetFlag(entry)     
      if (flag == 2) then      
         s3d = reader:s3d_Decode(entry)   
         count = reader:s3d_Count(s3d, 4)
         io.write("count :",count )
         block = reader:s3d_GetBlock(s3d, 4, 0)
         pos = reader:s3dtext_GetHeadPosition(block)
         while pos ~=0 do
            s3dtext, pos = reader:s3dtext_GetNext(block,pos)
            id, wrapModeS, wrapModeT,magFilter, minFilter,name = reader:s3dtext_GetInfo(s3dtext)
            if (id==0x00030403) then
               io.write("id:",name)
               id = 0x00000001
               name="test_lua"
               reader:s3dtext_SetInfo(s3dtext, id, wrapModeS, wrapModeT,magFilter, minFilter,name)
            end
         end   
         reader:s3d_SetBlock(s3d, 4, 0, block)
         reader:s3d_Encode(entry,s3d)
      end
      end
      reader:refresh()
   end
   
   This example parse all entries of a Dat file.
   If an entry is a S3D, it reads material block (4) then parses all textures.
   If id of one of them is 0x00030403, it changes to 0x00000001 and changes its name.   
   Once done, it encodes the data to save modifications.

(Edit: The link in my initial post should also include a full list of LUA commands for Reader. Also: thanks so much to both of you.)

[Raymond7cn 6,776]

Hiya Cori and Barton, I made a simple test:

52 minutes ago, CorinaMarie said:

• Do we know if your code reads every exemplar in a single .dat file?
• And do we know how to write the changed data back into each exemplar?

[Raymond7cn 6,776]

Sorry that my Opera crashed down due to comp was busy with rendering.

Dear Cori, @CorinaMarie I guess that the " flag == 7" which means every exemplars of entry. 

I made a test, you can tell from the first pic above that there were many files in this dat file (LD,PNG,EXEMPLAR,etc), Because of flag == 7 and descname == "Exemplar Name", i could read only the exemplar file's name and write them to a txt.

Then i used the SetValue function to set all name property's value to "Barton test".

But i don't know how to Replace the 123456 to 12, Do i need a new function to do this or a Regular expression? I almost forget all about Regular expression which i learnt from school 20 years ago.

[CorinaMarie 29,899]

4 minutes ago, Raymond7cn said:

But i don't know how to Replace the 123456 to 12, Do i need a new function to do this or a Regular expression?

This is the part I do understand.

I'll want to write it and test it before replying with the actual code. Would you post your Lua Script as text here? And attach a copy of your test .dat file?

Then, Barton, I notice you have some initial samples as 123455 and some as 123456. Are those the only two variations? And do they both change to 12? (And what they change to will be something you can edit for each successive file you run the script for.)

[Raymond7cn 6,776]

22 minutes ago, CorinaMarie said:

Would you post your Lua Script as text here? And attach a copy of your test .dat file?

Will do it soon Madame! Just a few minutes, since my comp is screaming now, I can't wait to learn Real Lua from you.

@CorinaMarie Hiya Cori, Please check the attach file, and This is the text:

Spoiler

function reader.main (this)   
       local file = io.open("E:/LUA/writetest.txt", "w")
       pos = reader:entries_GetHeadPosition()     
       while pos~=0 do
        entry,pos = reader:entries_GetNext(pos)
        flag = reader:entry_GetFlag(entry)  
        if (flag == 7) then
            exemplar = reader:ex_Decode(entry) 
            proppos = reader:exprop_GetHeadPosition(exemplar)  
            while proppos~=0 do
                exprop, proppos = reader:exprop_GetNext(exemplar,proppos)
                desc,descname = reader:exprop_GetDesc(exprop)  
                valuestr = reader:exprop_GetValueStr(exprop)
                    if (descname == "Exemplar Name") then
                          file:write("descname :",descname," desc :",desc,"value :",valuestr)  
                      file:write("\n")
                    reader:exprop_SetValue(exprop, "Barton Test")
                    reader:ex_Encode(entry,exemplar)        
                    end
                                             
            end   
        end
          end
   reader:refresh()
    file:close()
end

 

Barton.lua

SC4PATHTest1_6c2ed226-.SC4Lot

[BartonThinks 3,416]

28 minutes ago, CorinaMarie said:

This is the part I do understand.

I'll want to write it and test it before replying with the actual code. Would you post your Lua Script as text here? And attach a copy of your test .dat file?

Then, Barton, I notice you have some initial samples as 123455 and some as 123456. Are those the only two variations? And do they both change to 12? (And what they change to will be something you can edit for each successive file you run the script for.)

Ah! The "123455" instances are typos. My bad! All of the exemplar names in the initial DATs will start with "123456."

In each variant DAT, that number will be changed consistently across the exemplar name properties. That is, the exemplar names for all exemplar files in a given variant DAT should start with the same number, such as "12" or "34" or "6."

[CorinaMarie 29,899]

@Raymond7cn

I've tried your Lua code and it starts, but never finishes. I do get a zero byte writetest.txt file in the location specified, but nothing else.

[Raymond7cn 6,776]

45 minutes ago, CorinaMarie said:

@Raymond7cn

I've tried your Lua code and it starts, but never finishes. I do get a zero byte writetest.txt file in the location specified, but nothing else.

Hiya Cori, 

I'm not sure about that, But I'm basing my conjecture about it, Is it possible the problem of file name? The file i uploaded is 

SC4PATHTest1_6c2ed226-.SC4Lot, it should be "SC4PATHTest1_6c2ed226.SC4Lot", (Because of the chinese environment error). Then i see you have changed the file name to "SC4PATHTest1_6c2ed226- - Original.SC4Lot, Is it possible that we couldn't have 'blank' in name during executing script? 

I maybe wrong here, Sorry that i couldn't help you with testing now due to half sleep now, But i'm sure about the code i sent to you, After i sent it, i have downloaded it and tested it again, It's no problem in my comp, so the only difference is just the file name i observe.

[CorinaMarie 29,899]

The one I renamed with "Original" has the 123455 numbers and then I have a copy where I did change those all to 1234546. I've tried running the script on both. I then renamed it as you said, and the same occurs. The script will start and the name of the first exemplar does get updated to "Barton Test", but the script never finishes and even after closing reader there is still a process running that I have to kill manually.

I don't see anything which looks wrong in the code and since it does run for you that also shows it should be correct.

[Raymond7cn 6,776]

@CorinaMarie It's strange then, Please give me some time, I will go to have a look at it now.

[CorinaMarie 29,899]

Anyhow, even tho I can't run it for some reason, you were certainly correct that the string.gsub is the command to use.

Here's the code that should work. For actual use, you'd just change the "CM" part to "12" in the -- Cori added line. (Do note I changed the text to replace to "123456" and the same in the test file.)

-- Code Removed --

See my new post.

[BartonThinks 3,416]

Thank you both so much. Not sure when l'll have the time to test this out, but I really appreciate the help. Even if I run into issues, I'm a thousand times closer than I would have been on my own.

[Raymond7cn 6,776]

@CorinaMarie Hiya Cori, I made a test just now,

I modified the file name as yours and 123455 to 123456 with version 0.93

Run the script and it normally ends without dead procedure.

Checked out the name:

All 5 name's values have been changed to 'Barton Test'. Seems no problem in my comp, Really odd.

20 minutes ago, CorinaMarie said:

Anyhow, even tho I can't run it for some reason, you were certainly correct that the string.gsub is the command to use.

Here's the code that should work. For actual use, you'd just change the "CM" part to "12" in the -- Cori added line. (Do note I changed the text to replace to "123456" and the same in the test file.)

function reader.main (this)   
    local file = io.open("E:/LUA/writetest.txt", "w")
    pos = reader:entries_GetHeadPosition()     
    while pos~=0 do
        entry,pos = reader:entries_GetNext(pos)
        flag = reader:entry_GetFlag(entry)  
        if (flag == 7) then
            exemplar = reader:ex_Decode(entry) 
            proppos = reader:exprop_GetHeadPosition(exemplar)  
            while proppos~=0 do
                exprop, proppos = reader:exprop_GetNext(exemplar,proppos)
                desc,descname = reader:exprop_GetDesc(exprop)  
                valuestr = reader:exprop_GetValueStr(exprop)
                if (descname == "Exemplar Name") then
                    file:write("descname :",descname," desc :",desc,"value :",valuestr)  
                    file:write("\n")
                    cmTmpData = string.gsub(descname, "123456", "CM") -- Cori added
                    reader:exprop_SetValue(exprop, cmTmpData)         -- Cori changed
                    reader:ex_Encode(entry,exemplar)		
                end
            end
        end
    end
    reader:refresh()
    file:close()
end

 

Thank you Cori, I will learn and test it tomorrow, Sorry!

Sincerely,

-- Raymond

[CorinaMarie 29,899]

Ok, this is weird as hell, but if I simply add a space then backspace it in the script in the Lua window in Reader and suddenly it'll run without hanging up. That then showed the code I'd posted above worked, but was incorrect.

Now that I can actually test it, I'll figure out what it needs. In the meantime, Barton, do you have your base file with all the 123456 exemplar name entries? If so, I can test on that too.

[CorinaMarie 29,899]

Now tested and working perfectly.

Here's your code with my additions:

function reader.main (this)   
    local file = io.open("E:/LUA/writetest.txt", "w")
    pos = reader:entries_GetHeadPosition()     
    while pos~=0 do
        entry,pos = reader:entries_GetNext(pos)
        flag = reader:entry_GetFlag(entry)  
        if (flag == 7) then
            exemplar = reader:ex_Decode(entry) 
            proppos = reader:exprop_GetHeadPosition(exemplar)  
            while proppos~=0 do
                exprop, proppos = reader:exprop_GetNext(exemplar,proppos)
                desc, descname = reader:exprop_GetDesc(exprop)  
                valuestr = reader:exprop_GetValueStr(exprop)
                if (descname == "Exemplar Name") then
                    file:write("descname :",descname," desc :",desc,"value :",valuestr)  
                    file:write("\n")
                    cmTmpData = string.gsub(valuestr, "123456", "CM") -- Cori added
                    reader:exprop_SetValue(exprop, cmTmpData)         -- Cori changed
                    reader:ex_Encode(entry,exemplar)		
                end
            end
        end
    end
    reader:refresh()
    file:close()
end

 

And, unless you just want that output file, this also works and skips creation of writetest.txt. It updates the Exemplar Names only:

function reader.main (this)   
    pos = reader:entries_GetHeadPosition()     
    while pos~=0 do
        entry,pos = reader:entries_GetNext(pos)
        flag = reader:entry_GetFlag(entry)  
        if (flag == 7) then
            exemplar = reader:ex_Decode(entry) 
            proppos = reader:exprop_GetHeadPosition(exemplar)  
            while proppos~=0 do
                exprop, proppos = reader:exprop_GetNext(exemplar,proppos)
                desc, descname = reader:exprop_GetDesc(exprop)  
                valuestr = reader:exprop_GetValueStr(exprop)
                if (descname == "Exemplar Name") then
                    cmTmpData = string.gsub(valuestr, "123456", "CM") -- Cori added
                    reader:exprop_SetValue(exprop, cmTmpData)         -- Cori changed
                    reader:ex_Encode(entry,exemplar)		
                end
            end
        end
    end
    reader:refresh()
end

 

So, for either one, now you just change the part where I have "CM" to "12" or whatever you want. And be sure to save the data file in Reader after running the Lua Script.

[Cyclone Boom 17,032]

Wow! Amazing job @CorinaMarie and @Raymond7cn for figuring this out.

This again is a classic proof of what we can achieve as a community to solve problems. Also combatting quirks in the program too along the way, as it proved with it needing to edit in the Lua code windows. Brilliant job both of you with investigating this, and the the wonderful thing as well is that along with Barton finding this to help their projects, it can also help anyone else searching to locate this very thread. I can also imagine the code could be adapted for other purposes too. Bravo!

[Raymond7cn 6,776]

Thank you CB @Cyclone Boom, You know my hobby, Your words reminds me of this:

Actually, everytime when i could be helpful, i thought i was that one who get the most benefit, You and Cori always teach me more than i do, Thank you! Hallelujah!

Yours,

-- Raymond (03:00)

[rsc204 21,867]

8 hours ago, CorinaMarie said:

I believe another thing it'd need is for the script to open up more than one .dat file and to cycle thru those as well. However, considering how prone the versions of Reader past 0.9.3 (which are the ones where Lua can be used) are to corrupting the data, I find it difficult to find the motivation to look into this in detail.

I find this to be greatly overstated, but like most modders, I just don't want to loose any work I spend ages on, so you just go back to good 'ol 0.93.

But, outside of a few specific scenarios, actually you probably won't have problems, I know because I did for a time make an effort to use 1.54 as much as I dared. So if you test the process on a copied file, just start with something basic, if that works once, just only do this process in 1.54 and Save, don't do anything else, you can be fairly sure if it will work or not.

We had the chance to get these issues fixed when Tropod was working on the reader code, but a lack of feedback on the problem stifled efforts to track down all such bugs and it never happened. In other words, everyone's fear of it not working, meant we never got a working version. It's just like a Coronvirus vaccine, if at risk people aren't willing to take part in trials, despite the risks, we will never see an effective vaccine. Them's the breaks I'm afraid.

Once more though, @CorinaMarie and @Raymond7cn, you've found a solution to a problem that I think can genuinely be called a breakthrough. You are both such great assets to the community at large and I am absolutely certain when I've had a chance to properly digest your findings, they are going to save me so much time too. I never looked into this much, because even my largest such works never hit three digits, so I never invested the time to get into LUA, since painful as these tasks are, I'm no stranger to them. Automation has been key to how I approached so much of what I do, what makes it so incredibly useful, if you free up an hour of a modders time, that's an hour extra we can spend creating, which is let's face it what we'd all rather be doing. Such efforts can be a little thankless and your selflessness in coming to the aid of others will have a legacy that may not be immediately apparent. But just like every other tool we have to improve processes, their utility should not be underestimated.

Both of you are very humble about your abilities, but as your posts/efforts clearly show, in certain areas you really are top of the class.