Removing XML and LD files from lots?

[Zasco 105]

Another question for you all. It would be best suited in the SC4 Modding - Open Discussion forum, but there is no "Quick question" topic and I didn't want to start a whole new one simply for this. Lately I'm working on tweaking SC4 files with the ILive Reader. I found out there is XML in the SC4Desc and SC4Model files. However, it doesn't seem to be used by the game as changing it or completely removing it doesn't seem to affect the functionality of the mod in any way. I guessed it may be used by the lot editor. Could someone tell me what is it used for?

  Edited September 25, 2022 by Cyclone Boom  
Agreed, and I've moved the discussion to a new topic. ;-)

[Ulisse Wolf 4,414]

23 minutes ago, Zasco said:

Another question for you all. It would be best suited in the SC4 Modding - Open Discussion forum, but there is no "Quick question" topic and I didn't want to start a whole new one simply for this. Lately I'm working on tweaking SC4 files with the ILive Reader. I found out there is XML in the SC4Desc and SC4Lot files. However, it doesn't seem to be used by the game as changing it or completely removing it doesn't seem to affect the functionality of the mod in any way. I guessed it may be used by the lot editor. Could someone tell what is it used for?

XML files have no utility and are not needed for LotEditor

[Zasco 105]

8 hours ago, ulisse said:

XML files have no utility and are not needed for LotEditor

Alright, thx! Why is it there then? What brought it there? Can I safely remove it altogether for "cleanliness", easier file management/uniformity and lighter weight purposes?

[TheMurderousCricket 5,057]

1 hour ago, Zasco said:

Alright, thx! Why is it there then? What brought it there? Can I safely remove it altogether for "cleanliness", easier file management/uniformity and lighter weight purposes?

I am sure though that you can remove the LD file too. Just thought I'll mention this too if cleanliness is your current "quest". 

[Zasco 105]

On 9/23/2022 at 6:56 AM, TheMurderousCricket said:

I am sure though that you can remove the LD file too. Just thought I'll mention this too if cleanliness is your current "quest". 

By "LD file", do you mean the "LotData" file inside the SC4Lot? If so, what about the PNG in there too then?

[TheMurderousCricket 5,057]

38 minutes ago, Zasco said:

By "LD file", do you mean the "LotData" file inside the SC4Lot? If so, what about the PNG in there too then?

Emm... I guess so...  

I mean the file which shows growth stage, plugins used and other info upon clicking on it. Some of the data has drop down lists. I think we are talking about the same thing, yes. 

I wouldn't touch PNGs, as they may be necessary for icons. If you want to delete them, first make sure what is their original purpose.

[rsc204 21,815]

XML, LD and PNG files (not the icon, but lot previews) are all left-over functionality which was used by the official Maxis custom content exchange which has been dead about 10 years now. Indeed none of them are of any use whatsoever and they simply are wasting space and making things less efficient. They can all be safely removed. Frankly if you are using the LE I can not understand why, it’s just creating more work for you and is so clunky compared to the superior PIM-X. Do yourself a favour and check it out, you’ll be glad you did. 

[Zasco 105]

29 minutes ago, rsc204 said:

XML, LD and PNG files (not the icon, but lot previews) are all left-over functionality which was used by the official Maxis custom content exchange which has been dead about 10 years now. Indeed none of them are of any use whatsoever and they simply are wasting space and making things less efficient. They can all be safely removed. Frankly if you are using the LE I can not understand why, it’s just creating more work for you and is so clunky compared to the superior PIM-X. Do yourself a favour and check it out, you’ll be glad you did. 

Alright, thx!  I doubt the PNG in the SC4Lot file is an icon as the mod I'm currently working on is a growable only.

Thanks for the hint about PIM-X. I didn't know it had the same functionalities as the LE. Moreover, I couldn't get the latest to work on my computer...

[Tyberius06 12,190]

1 hour ago, rsc204 said:

XML, LD and PNG files (not the icon, but lot previews) are all left-over functionality which was used by the official Maxis custom content exchange which has been dead about 10 years now. Indeed none of them are of any use whatsoever and they simply are wasting space and making things less efficient. They can all be safely removed. Frankly if you are using the LE I can not understand why, it’s just creating more work for you and is so clunky compared to the superior PIM-X. Do yourself a favour and check it out, you’ll be glad you did. 

37 minutes ago, Zasco said:

Alright, thx!  I doubt the PNG in the SC4Lot file is an icon as the mod I'm currently working on is a growable only.

Thanks for the hint about PIM-X. I didn't know it had the same functionalities as the LE. Moreover, I couldn't get the latest to work on my computer...

One thing in addition. The XMLs in the LOT files and in the *sc4desc files (TYPE ID: 0x88777602) are unnecessary indeed and can be removed safely, BUT the XMLs in the prop packs and *.sc4model files (with the TYPE ID: 0x88777601) are a good thing to keep if you want to work with either Maxis PIM (in this case it's mandatory otherwise the models won't show up in the program, though using Maxis PIM is really NOT recommended) or SC4 PIM-X, because it contains the models names which would show up in the resource model list in PIM-X. Without this type of XML in PIM-X you would only see the Group-Instance ID references of the models and these models without the name/xml would not appear in Maxis PIM at all.

And yeah LD, PNG (lot preview) can be removed safely. In the past few weeks I actually spent quite some time to clean up a sh... loads of older contents (from 2004-2006) which had confusing contents and properties (hence Project ZIP is taking sooooo long for me recently). Growables with item icon properties and icon PNGs etc... which seems to be totally unnecessary too.

- Tyberius

[rsc204 21,815]

I’m fully aware that XML is necessary for the traditional PIM -> LE workflow, specifically for models to be recognised in PIM, but if you use PIM-X this is no longer necessary.

PIM-X can pick up and use models without them, these internal names are rarely important, especially as they don’t necessarily relate to the name of the files anyhow. For example, because I create everything in 3DS Max, I only use SC4BAT to create the LODs for my models. This name comes from the filename used in SC4BAT when saving a project, but I rarely bother to save anything I do in SC4BAT, since everything I might need is actually handled in 3DS Max instead. As a result, my models rarely get 'proper' names to begin with, most are just blank or show XML not found. But this isn't such a big deal, since you can simply search by the file name instead, which assuming individual models is pretty quick. Likewise, if you follow lotting best practise and only keep those files you actually need around when lotting, it shouldn't be difficult to find the exact thing you need.

Perhaps if you had a really big MegaPack sans the XML data, it could then become a bit of a problem, but usually the Prop Exemplars already exist within such packs to begin with. For me unless you are using the antiquated PIM, it's just not a big deal to keep these XML files, especially when you consider the number of models you may have, they can add up to a lot of redundant files. In the very worst case scenario, open the model in Reader, find the ID and cross-reference it that way, it's not terribly hard and I personally can't recall needing to resort to such measures myself.

I'll note too, when DATPacking SC4 files, all XML files (along with the other discussed files), will be removed from the packed files. Just one more reason why DATPacked files are more efficient.

3 hours ago, Tyberius06 said:

And yeah LD, PNG (lot preview) can be removed safely. In the past few weeks I actually spent quite some time to clean up a sh... loads of older contents (from 2004-2006) which had confusing contents and properties (hence Project ZIP is taking sooooo long for me recently). Growables with item icon properties and icon PNGs etc... which seems to be totally unnecessary too.

It surprised me, but many MegaPacks benefit from being DATPacked, not with other files, just by themselves, the file that get's spat out is free of duplications and other unnecessary junk. You'd be shocked how much smaller you can make some of these DAT files just by doing this. You sometimes need to be careful though, since the wrong duplicates might remain, then again just as with files themselves, there is a specific load order of the contents of files, so if something goes wrong, it should have in theory been wrong in the original source too.

[Tyberius06 12,190]

19 minutes ago, rsc204 said:

I’m fully aware that XML is necessary for the traditional PIM -> LE workflow, specifically for models to be recognised in PIM, but if you use PIM-X this is no longer necessary.

PIM-X can pick up and use models without them, these internal names are rarely important, especially as they don’t necessarily relate to the name of the files anyhow. For example, because I create everything in 3DS Max, I only use SC4BAT to create the LODs for my models. This name comes from the filename used in SC4BAT when saving a project, but I rarely bother to save anything I do in SC4BAT, since everything I might need is actually handled in 3DS Max instead. As a result, my models rarely get 'proper' names to begin with, most are just blank or show XML not found. But this isn't such a big deal, since you can simply search by the file name instead, which assuming individual models is pretty quick. Likewise, if you follow lotting best practise and only keep those files you actually need around when lotting, it shouldn't be difficult to find the exact thing you need.

Perhaps if you had a really big MegaPack sans the XML data, it could then become a bit of a problem, but usually the Prop Exemplars already exist within such packs to begin with. For me unless you are using the antiquated PIM, it's just not a big deal to keep these XML files, especially when you consider the number of models you may have, they can add up to a lot of redundant files. In the very worst case scenario, open the model in Reader, find the ID and cross-reference it that way, it's not terribly hard and I personally can't recall needing to resort to such measures myself.

I'll note too, when DATPacking SC4 files, all XML files (along with the other discussed files), will be removed from the packed files. Just one more reason why DATPacked files are more efficient.

Well I don't agree with everything here, but that's because many people use different practices than others. I agree that PIM-X do not need the XMLs, though it can be really helpfull when you have the models with names. I stopped using DATpacker (2008 version) exactly because it's removing the XML properties, and I started using J-DATpacker, because that keeps them, and I actually started recovering the XML properties for many-many mega prop packs over at LEX. But as I said it's a matter of different practices; while that's true that PIM-X can read the models without the XML, when you create new exemplars out of model files (which is the recommended practice instead of copy/paste/re-ID existing prop exemplars), without the proper names (and according to my experiences after I went through a couple of hundreds of megapacks the names mostly accurate, though the naming formats are pretty inconsistent for sure) you have two options, either give random names according to your taste or you try to hunt down an existing exemplar for the same model and catch the name from it. I don't like to come up with random names, I like to use mostly the original names what the creator used and adjust it just a bit to show that mine is a bit different version. Of course sometimes I need to translate the names to english or come up with matching but random names, but many times the original names actually help to identify the model. For example when I created hundreds of car props out of Girafe car packs it was more tedious than it should have been because I couldn't recognize the cars based on the small prop preview images in PIM-X and I needed to hunt down the original props to get the car types etc... I couldn't name them just CAR1 green etc... because that felt just lazy to me and really not accurate.

Anyhow as I learned a year ago, mostly the sole purpose of removing the XML properties back in the time (from 2006-ish) was to prevent people to create their own exemplars out of models (at that time only the Maxis PIM could do that easily and that required the mentioned XML property). So people were limited to use the given exemplars (either prop or building). That changed a bit with PIM-X which allows you to create your own exemplars anyway, so removing the XML properties were not entirely necessary anymore.

32 minutes ago, rsc204 said:

It surprised me, but many MegaPacks benefit from being DATPacked, not with other files, just by themselves, the file that get's spat out is free of duplications and other unnecessary junk. You'd be shocked how much smaller you can make some of these DAT files just by doing this. You sometimes need to be careful though, since the wrong duplicates might remain, then again just as with files themselves, there is a specific load order of the contents of files, so if something goes wrong, it should have in theory been wrong in the original source too.

Would you be surprised that both version of the NCD Railyard Texture vol01 (v1 and the v1.02) are double sized because the textures in the DAT file are duplicates. It's something I learnt recently and will be fixed in next few weeks when I'm there.  

So I agree that DATpacking is necessary and useful, though I disagree of the removal of the XML (Type 0x8877761) properties, because that can be useful specially for custom content creators. 

[Zasco 105]

Alright, so in summary, what understand from all your technical lingo is that one can remove them from a game functionality point of view, but should keep them from a development one. I such case and as of how I work, I should keep them in the individual separated files (SC4Desc, SC4Lot and SC4Model), but could remove them from the published dat.

Thank you for sharing your expertise!

[rsc204 21,815]

On 25/09/2022 at 5:20 PM, Tyberius06 said:

Well I don't agree with everything here

Yes, I think this is one of those where we have to agree to disagree, but I will add one last comment in that regard.

It makes no sense to me to create loads of new exemplars manually from the models (especially if Props), when you can simply copy/paste the existing ones and re-ID them. Coupled with the reality that so few people are doing the kind of modifications you so often do, in the larger scheme of things wouldn't it make more sense if the bulk of users had more efficient files? I get that wouldn't be ideal based on what you are doing, but I really don't think it's something many would ever need or want to do.

I suspect part of the reason you prefer to remake things, is because often the original Exemplars are full of Properties that don't need to be there or even potentially errors. Personally, I'm comfortable just modifying them at this point manually, rather than re-making them, since in reality you don't need a whole lot of Properties to make things function. For example, many use settings that would work identically if the Property simply wasn't there, owing to the fact that they mirror the default setting that is used when the Property isn't there. Of course, knowing all this takes a bit of experience and knowledge and certainly I'm sure most of my earlier work would be similarly affected too.

On 26/09/2022 at 1:09 PM, Zasco said:

Alright, so in summary, what understand from all your technical lingo is that one can remove them from a game functionality point of view, but should keep them from a development one. I such case and as of how I work, I should keep them in the individual separated files (SC4Desc, SC4Lot and SC4Model), but could remove them from the published dat.

So what Tibi is saying is that you should keep them for the release versions, in case another developer wanted to make new Exemplars for your models. Personally I would remove them before releasing anything because they are in my opinion simply not useful for 99% plus of users.

Whether you prefer to release your work as separate Lots, Models Etc or DAT files is very much a matter of personal choice. I tend to go for DATs personally because it's cleaner and simpler and anyone who knows what they are doing, would have no problem re-using any parts they might wish. However, if you've a lot of separate lots, where users might not want all of them (similar with models and other files), releasing separate files is more flexible. That said, given the penchant for larger DAT files/Mega Packs, i.e. simplicity over flexibility and efficiency that exists already, frankly I suspect most users would prefer less files every time.

[Tyberius06 12,190]

22 minutes ago, rsc204 said:

in the larger scheme of things wouldn't it make more sense if the bulk of users had more efficient files? I get that wouldn't be ideal based on what you are doing, but I really don't think it's something many would ever need or want to do.

It seems to be a bit of overoptimizing and I don't see significant benefits of it. I don't think the removal of the XMLs would make significantly more efficient the files, when someone release a model or model pack. The reason is because when someone is playing with that many plugins that would cause performance issues, they are recommended to DATpack their contents and for that the Datpacker 2008 is still way better than the J-Datpacker (this is good for packing dependency packs). And since DATpacker will remove all the non-necessary files (xmls too), that won't cause efficiency issues for players. And it is also recommended not to use full plugin folders or mass-datpacked plugin folder for custom content creation. So at the end of the day the few kbyte extra because of the XMLs won't matter anyway. 

[Zasco 105]

7 minutes ago, Tyberius06 said:

Datpacker 2008

Is that SC4 DatPacker?

[Tyberius06 12,190]

1 hour ago, Zasco said:

Is that SC4 DatPacker?

Yes, that's the one. Using this will remove most of the unnecessary things. It is good for DATpacking contents of playing plugins. 

I don't like to use it Datpacking or assembling model/prop packs because of the above detailed reason.

For DATpacking dependency packs or sets which could be used as dependency, I prefer if J-DATpacker is being used because that doesn't remove otherwise usefull properties for potential custom content creators.

I personally don't like when lot and model/prop files are packed together because it would generally force people to use the original lots even if they don't need that, but only the models/props. But again this is mostly matter of different tastes in used practices. Most of them has equivalent pro-contra values.

[rsc204 21,815]

2 hours ago, Tyberius06 said:

I don't think the removal of the XMLs would make significantly more efficient the files, when someone release a model or model pack. The reason is because when someone is playing with that many plugins that would cause performance issues, they are recommended to DATpack their contents and for that the Datpacker 2008 is still way better than the J-Datpacker (this is good for packing dependency packs). And since DATpacker will remove all the non-necessary files (xmls too), that won't cause efficiency issues for players.

This is the thing about optimisations like this, taken as a few odd files it's probably not a big deal, but once you expand that to more and more files/mods, it soon makes a bigger difference than you might think, although in reality they should only affect loading times.

I guess this is what comes of the mindset of those who were educated like I was, that all code/computer-related things should be ruthlessly optimised. Back in the late 90's computers still had yet to get to the point where we've so much power we're not using, that doing things this way simply just isn't done anymore because lazy unoptimised coding is simply cheaper to realise. That however, in no way means that making everything terribly inefficient doesn't come with downsides, especially when you consider the bigger picture. Just think for a moment how much electricity computers around the world use, then imagine if we could just make things 1% more efficient, how that would have a dramatic effect on the amount of electricity the world used, with the potential for significant CO2 savings globally. When I think how just by taking a little longer to design and implement better coding, could effectively make a huge difference to one of the biggest threats facing our world today, I genuinely believe there is merit in considering the old ways of doing things.

[Tyberius06 12,190]

18 minutes ago, rsc204 said:

This is the thing about optimisations like this, taken as a few odd files it's probably not a big deal, but once you expand that to more and more files/mods, it soon makes a bigger difference than you might think, although in reality they should only affect loading times.

I guess this is what comes of the mindset of those who were educated like I was, that all code/computer-related things should be ruthlessly optimised. Back in the late 90's computers still had yet to get to the point where we've so much power we're not using, that doing things this way simply just isn't done anymore because lazy unoptimised coding is simply cheaper to realise. That however, in no way means that making everything terribly inefficient doesn't come with downsides, especially when you consider the bigger picture. Just think for a moment how much electricity computers around the world use, then imagine if we could just make things 1% more efficient, how that would have a dramatic effect on the amount of electricity the world used, with the potential for significant CO2 savings globally. When I think how just by taking a little longer to design and implement better coding, could effectively make a huge difference to one of the biggest threats facing our world today, I genuinely believe there is merit in considering the old ways of doing things.

Honestly, my argument here is related to one component, the model based XML property, which one can be useful. 

On the other hand I was wrestling with lot files recently which contained confusing and unnecessary components all over the place, and I kind of got burnt out at the end due to the amount of sh... what needed to be cleaned. 

We could find so many parts which could be improved by cleaning up the mess, though the amount of time and effort might not worth it. 

In the big picture when one is playing, the DATpacking of plugins and removing the unnecessary parts are good, and we can say it might be essential too, but keeping some elements upon custom content creation to make them easier to be reused with the tools what we have is also beneficial. 

I prefer reading the model names like that's a bench and alphabetically it's coming before C and after A, instead of checking one by one every GI ID in the list while I found an image which looks like a bench... 

But otherwise when I play, I Datpack the whole stuff.