Night Lights on Mac

[c4bl3fl4m3 405]

3 minutes ago, CorinaMarie said:

The modding side is one which most definitely needs Reader. Making BATs is something else entirely and requires other tools. I've never BATted.

Ah, okay. Good to know. Still, I think it's a slippery slope...

[c4bl3fl4m3 405]

3 hours ago, CorinaMarie said:

PEG_OMG_Iconic_Cathedral_Mac_FFDC.dat <-- Still as an override so needs the original files too. (Must load after them.)

It worked. (I put a Z_ in front of the file name to make sure it loaded correctly.)

 

BTW, how do I get it to show my screenshots smaller on here? Because they're definitely not pixelated on my computer.

[CorinaMarie 29,605]

2 minutes ago, c4bl3fl4m3 said:

BTW, how do I get it to show my screenshots smaller on here? Because they're definitely not pixelated on my computer.

They look perfectly fine to me. But, you can resize them if you like. After you place the pic in your post, just double-click it for some options.

[Cyclone Boom 16,875]

25 minutes ago, c4bl3fl4m3 said:

So what you're saying is I can change file names w/o it affecting performance? It won't break my BATs? Because I've always wondered if it would and I assumed it would because files are dependent on other files so it probably needs to know the names of them. (Or do I have to do this with folders only?)

The key thing with the file names is how they determine the processing order. It gets a little complicated with the techy details. Basically inside items of SC4 content, each individual entry is defined by an IID value. This sets the instance as part of the TGI which is the sub-grouping arrangement, and what is important for the night light fixing. As Cori explained, the file names are only important to control the order in which they're loaded into memory. This can mean how certain files override others when the same exact instance. However that's not a detriment to performance nor functionality, and just a consideration for how duplicate files are handled.

 

6 minutes ago, c4bl3fl4m3 said:

BTW, how do I get it to show my screenshots smaller on here? Because they're definitely not pixelated on my computer.

Glad to hear it worked and with the loading order too.

Try double clicking the image once added to your post, and it'll open up a dialog.

As follows:

 

And @CorinaMarie beat me to it again, but here's another two for the price of one answer.

(There's a combo deal running today you see.)

[c4bl3fl4m3 405]

6 minutes ago, Cyclone Boom said:

Glad to hear it worked and with the loading order too.

But here's the big question. Once it gets jDatPacked, will it still load in the same order and work? (I'm about to find out.)

Because I have a feeling those of us who play it on the Mac version and are serious about adding BATs, we all jDatPack our files out of necessity. And while simply leaving out some of the files out of the Dats could work at first, if this gets big, it'll get unwieldy.

EDIT: Yep, still works fine. Thank you all so much.

  Edited November 16, 2019 by c4bl3fl4m3  
answered my own question

[CorinaMarie 29,605]

1 minute ago, c4bl3fl4m3 said:

Once it gets jDatPacked, will it still load in the same order and work? (I'm about to find out.)

This I don't know the definitive answer to.   But, I believe the various packing programs read the files in the same order as the game does and so that preserves the order internally. We'll prolly need to wait for the experts to explain this part.

[Cyclone Boom 16,875]

8 minutes ago, c4bl3fl4m3 said:

But here's the big question. Once it gets jDatPacked, will it still load in the same order and work? (I'm about to find out.)

The way DatPacking works, it's effectively consolidating all individual instances into a central container.

This means how the same principle applies for loading order, and just it's determined by the compressed DatPacked files instead. Inside viewing using Reader and everything will be all bundled up that way, and the IID values are still preserved identically as they are if unpacked. As I'm sure you'll know, it's a wise move to keep a parallel copy of non-packed files to make switching over new files easier. Then they could be re-packed as needed for updating everything.

[c4bl3fl4m3 405]

Just now, Cyclone Boom said:

just it's determined by the compressed DatPacked files instead.

Okay, so you're saying...

1) that it opens up the Dat file, sees the old file names inside, and loads them according to those old file names

OR

2) once it's in a Dat file, it bases it off the the alphanumeric order of the dat files themselves, so it's best to put anything you want loaded at the end in a separate Dat file and name it with a Z?

2 minutes ago, Cyclone Boom said:

a parallel copy of non-packed files

Oh, I've been doing that for years. I called it the "shadow folder". (It's also where I keep all those readme's, etc. just in case I need them at some point. And honestly? That's what helped me find what set the OMG Iconic Cathedral was located in... searching the STEX and the SC4D site didn't render anything.)

[c4bl3fl4m3 405]

So could I ask a small favor? Because I have a bit of a mystery.

Could someone who runs Windows DL the following file, install it, put it in a city/get it to grow in a city, and post a picture of what it looks like at night? Because my grown version of it has street light night lighting but no internal lighting, which is odd... I'd expect it to have both or neither, and I'm wondering if this is a Mac bug or if that's just what it's like in Windows.

 

 

[c4bl3fl4m3 405]

Also, something else I just thought of. Is there any way of informing the people over at SC4D that we've (mostly) solved this problem & the fix?

[CorinaMarie 29,605]

7 minutes ago, c4bl3fl4m3 said:

Could someone who runs Windows DL the following file, install it, put it in a city/get it to grow in a city, and post a picture of what it looks like at night?

 

8 minutes ago, c4bl3fl4m3 said:

Because my grown version of it has street light night lighting but no internal lighting, which is odd... I'd expect it to have both or neither ...

It seems logical that the street lights are conforming to the old GID and therefore the Macs understand it. The building lights would be the expansion method.

[Cyclone Boom 16,875]

35 minutes ago, c4bl3fl4m3 said:

2) once it's in a Dat file, it bases it off the the alphanumeric order of the dat files themselves, so it's best to put anything you want loaded at the end in a separate Dat file and name it with a Z?

Yes, this is correct. I just ran a test to double check, and can confirm DatPacked files correspond to how they're alphabetically ordered. This means when seeing the files in one's Plugins folder, that order is how the entries are all combined when compressed inside packed .dat files. Therefore a file named with a "z" would load afterwards because the order inside the file matches up with this. So really the only difference is how this all happens within the .dat instead of as individual files.

 

24 minutes ago, c4bl3fl4m3 said:

Also, something else I just thought of. Is there any way of informing the people over at SC4D that we've (mostly) solved this problem & the fix?

Maybe it's worth sending @Tarkus a PM about this topic. He's the Webmaster of SC4D and would be ideally suited for spreading the word over there.

(The @ mention here in my post should serve to notify him also.)

[c4bl3fl4m3 405]

1 minute ago, Cyclone Boom said:

Yes, this is correct. I just ran a test to double check, and can confirm DatPacked files correspond to how they're alphabetically ordered. This means when seeing the files in one's Plugins folder, that order is how the entries are all combined when compressed inside packed .dat files. Therefore a file named with a "z" would load afterwards because the order inside the file matches up with this. So really the only difference is how this all happens within the .dat instead of as individual files.

 

Maybe it's worth sending @Tarkus a PM about this topic. He's the Webmaster of SC4D and would be ideally suited for spreading the word over there.

(The @ mention here in my post should serve to notify him also.)

I'm sorry, I'm even more confused. Because what you're describing corresponds with the first thing I said, not the 2nd.

I guess what I'm asking is if people put out fixes as separate files to be loaded after, can I rename them Z and put them in the DAT file & it'll load them later, or do they have to be a separate file OUTSIDE of the dat file that has the other files?

[Cyclone Boom 16,875]

48 minutes ago, c4bl3fl4m3 said:

I guess what I'm asking is if people put out fixes as separate files to be loaded after, can I rename them Z and put them in the DAT file & it'll load them later, or do they have to be a separate file OUTSIDE of the dat file that has the other files?

Yes, that's right. What happens is the files as named are in the same order when packed. By adding a Z to determine the loading order, this controls the sequence of how the individual file entries are added inside a given packed .dat file.

 

48 minutes ago, c4bl3fl4m3 said:

I'm sorry, I'm even more confused.

No worries. I'll try again to explain:

Let's look at it this way: With no dat packing, the files load alphabetically by file name. When dat packing they lose their file name, but are compressed into the dat packed file in the order their original names dictated. If you add new files alongside the dat packed one(s), the loading order is based on the file name of the dat packed one and the new files. So, if the first is A_Dat_Packed_File.dat and then you need something to load after, you just make sure the first letter of the other file name is B or later in the alphabet.

It's also possible to have files loading which aren't dat packed (where the loading order is still applicable).

For these, use file naming as normal to determine which loads before or after others. As such they can be outside of the dat packed files loading afterwards, OR copied to your Shadow Folder, named with the Z, and re-packed. And if you copy the new file into your Shadow Folder, you just do the same as you did before.

I hope this clarifies, but do ask if still unsure. It's understandably quite technical all of this.

[c4bl3fl4m3 405]

8 minutes ago, Cyclone Boom said:

I hope this clarifies, but do ask if still unsure.

Okay, I think I have it, but let me give a fictional example just to make sure.

Generic House is a BAT whose nightlighting doesn't work on the Mac.

Someone fixes it, but instead of fixing the files of Generic House, they put out another file Fixed Generic House that has to be loaded with, but after, Generic House's files.

So if I make a folder called Generic House, put the original files in it, and rename Fixed Generic House to Z_Fixed Generic House, it will load correctly.

If I jDatPack these files together in the same DAT pack, it will read the DAT package, load Z_Fixed Generic House AFTER Generic House (because it keeps the load structure the same), and load correctly.

Additionally, if I choose to DAT pack Generic House, but leave Z_Fixed Generic House separate outside of the DAT pack, it will load the DAT Packed Generic House first then Z_Fixed Generic House 2nd because it begins with a Z.

Either way works.

Do I understand?

[Cyclone Boom 16,875]

14 minutes ago, c4bl3fl4m3 said:

Do I understand?

Yep, you got it. That's completely correct for how this works.

The great thing now too is how your analogy will help other people who read this topic later. (Forums sure are a wonderful thing.)

[c4bl3fl4m3 405]

Bigger question (simply out of curiosity): what's the advantage of putting out a separate patched Fixed Generic House file to be loaded after the original Generic House file as opposed to fixing the Generic House file itself?

[CorinaMarie 29,605]

2 hours ago, c4bl3fl4m3 said:

Bigger question (simply out of curiosity): what's the advantage of putting out a separate patched Fixed Generic House file to be loaded after the original Generic House file as opposed to fixing the Generic House file itself?

Basically it's political inasmuch as we aren't altering the original files. From your standpoint of playing the game it would be better to alter the original and be done with it. Once you get Reader and Model Tweaker working, we can show you how to do that.

[Cyclone Boom 16,875]

3 hours ago, c4bl3fl4m3 said:

Of course, y'all are our illustrious leaders here so obviously what you say goes.

We do want this to be the best outcome for everyone, so it's striking a careful balance with these things.

As a compromise solution, how about adding a blue formatted message banner atop the file which stands out even more than a description note? Then linking to the dedicated thread where the fix (and other fixes) are indexed. It might also be worth making the attachment available right there on such file entry messages. This can be done using your STEX editing access rights, and we'll show you how to do that if you'd like it clarifying. There's an option for picking the colour on this file for instance. That way instead of being listed as normal under the Download button, it'd be prominently distinct and crystal clear how it's for Mac users. Maybe even with a little Mac logo too for a visual cue.

Might that be a possible way forward?

[rsc204 21,808]

On 16/11/2019 at 9:04 PM, c4bl3fl4m3 said:

If I jDatPack these files together in the same DAT pack, it will read the DAT package, load Z_Fixed Generic House AFTER Generic House (because it keeps the load structure the same), and load correctly.

Yes, but this is based on the assumption that the files being Packed end up in the same actual DAT file. You can of course control where files are stored, i.e. in the same folder you are packing, to prevent this.

On 16/11/2019 at 9:04 PM, c4bl3fl4m3 said:

Additionally, if I choose to DAT pack Generic House, but leave Z_Fixed Generic House separate outside of the DAT pack, it will load the DAT Packed Generic House first then Z_Fixed Generic House 2nd because it begins with a Z.

Now this is where things get tricky. Maybe, maybe not. The actual load order of SC4 is far more complex than simply alphabetical. If you want to see the full story, take a look at the information in this thread where Jondor worked it all out. SC4 will load in the following order:

Quote

\plugins\*.*
\plugins\asubfolder\*.*
\plugins\asubfolder\asubsub\*.*
\plugins\asubfolder\bsubsub\*.*
\plugins\bsubfolder\*.*

\plugins\*.dat
\plugins\asubfolder\*.dat
\plugins\asubfolder\asubsub\*.dat
\plugins\asubfolder\bsubsub\*.dat
\plugins\bsubfolder\*.dat

The key takeaway here is that all .DAT files will override non-DAT files. So if the override was anything other than a .DAT file, it would actually load before any DAT-Packed files, regardless of location or name.

Adding to the fun, there is always the possibility that the Mac version does not behave identically too - it really doesn't, you are best sticking to Letters and Numbers because use of many characters seems to do all sorts of unexplainable. I'm guessing this is simply the underlying file system of Linux (MacOS) is very different to Windows). Can't help you there sorry.

[c4bl3fl4m3 405]

1 minute ago, rsc204 said:

Yes, but this is based on the assumption that the files being Packed end up in the same actual DAT file. You can of course control where files are stored, i.e. in the same folder you are packing, to prevent this.

Now this is where things get tricky. Maybe, maybe not. The actual load order of SC4 is far more complex than simply alphabetical. If you want to see the full story, take a look at the information in this thread where Jondor worked it all out. SC4 will load in the following order:

The key takeaway here is that all .DAT files will override non-DAT files. So if the override was anything other than a .DAT file, it would actually load before any DAT-Packed files, regardless of location or name.

Adding to the fun, there is always the possibility that the Mac version does not behave identically too. Can't help you there sorry.

This is different from what @Cyclone Boom replied with. Not to start anything, but which of you is correct? And whichever one is correct, can the other person edit their response so that misinformation isn't being spread? (I think this is what I was referring to when I said I found conflicting information.)

And, yeah, I know there's always a chance the Mac doesn't work the same way. Out of curiosity, how did we find this out about the Windows version? Maybe the same thing or a similar thing could be done on the Mac to confirm?

[CorinaMarie 29,605]

3 minutes ago, c4bl3fl4m3 said:

This is different from what @Cyclone Boom replied with.

Actually it's an expansion of what CB said. With CB's reply it's covering the basics of .dat files lose and packed. @rsc204 is adding additional, more technical information now you are ready to delve even deeper.

[rsc204 21,808]

Just now, c4bl3fl4m3 said:

And, yeah, I know there's always a chance the Mac doesn't work the same way. Out of curiosity, how did we find this out about the Windows version? Maybe the same thing or a similar thing could be done on the Mac to confirm?

Check the thread I linked to. The reasoning and testing behind what I posted is very well documented. Better still, the tools and steps to test it yourself are attached to the post. You could use those tools to repeat his testing.

I had a couple issues tracking down load-order problems, where for what seemed like bizarre reasons, files were not overriding correctly. Once I came across this information, my hours of misery were over and I fixed them in seconds. So I can absolutely confirm the validity of this information.

2 minutes ago, c4bl3fl4m3 said:

This is different from what @Cyclone Boom replied with.

Not necessarily different, expanding upon the information given might be the way I'd describe it. It's no more than a couple caveats, which if you are aware of, will prevent running into trouble.

[c4bl3fl4m3 405]

15 minutes ago, CorinaMarie said:

Actually it's an expansion of what CB said. With CB's reply it's covering the basics of .dat files lose and packed. @rsc204 is adding additional, more technical information now you are ready to delve even deeper.

Not exactly the same. With this additional info, it's clear that our Generic House example doesn't work quite the same.

If I understand correctly, with this new info, it now changes to...

Assuming NEITHER or BOTH are DAT files, if I make a folder called Generic House, put the original files in it, and rename Fixed Generic House to Z_Fixed Generic House, it will load correctly.

If Generic House IS a DAT file but Fixed Generic House is NOT, then regardless of what I name Fixed Generic House, it will load BEFORE Generic House, and not work.

If Generic House is a NOT a DAT file but Fixed Generic House IS, then regardless of what I name Fixed Generic House, it will load AFTER Generic House, and WILL work.

If I jDatPack these files together in the same DAT pack, it's the same as the 2 examples above.

Additionally, if I choose to DAT pack Generic House, but leave Z_Fixed Generic House separate outside of the DAT pack, it follows the above rules.

...Do I understand correctly now?

[rsc204 21,808]

On 16/11/2019 at 10:36 PM, c4bl3fl4m3 said:

If I jDatPack these files together in the same DAT pack, it's the same as the 2 examples above.

You'd need to check, since I don't know how the developer of the tool you are using coded JDatPacker to work. That may not actually be the same as the one the game uses.

On 16/11/2019 at 10:36 PM, c4bl3fl4m3 said:

Assuming NEITHER or BOTH are DAT files, if I make a folder called Generic House, put the original files in it, and rename Fixed Generic House to Z_Fixed Generic House, it will load correctly.

If Generic House is a DAT file but Fixed Generic House is NOT, then regardless of what I name Fixed Generic House, it will load BEFORE Generic House, and not work.

If Generic House is a NOT a DAT file but Fixed Generic House IS, then regardless of what I name Fixed Generic House, it will load AFTER Generic House, and WILL work.

This is all correct.

On 16/11/2019 at 10:36 PM, c4bl3fl4m3 said:

Additionally, if I choose to DAT pack Generic House, but leave Z_Fixed Generic House separate outside of the DAT pack, it follows the above rules.

Yes, meaning if Z_Fixed Generic House were not a DAT file, it would be overriden. But if both were DATs it would work as intended.

Note: I've since discovered using underscores seem to do funny things to load order on a Mac. So better to stick with say zFixed Generic House and drop the underscore character.

--

One more, but rather important update. So we have a theoretical maximum of 57k IDs, well between SimCity_2, 3, 4 & 5 DAT alone, there are over 34k textures already using the GID 1ABE878D. Some of those will use duplicates of the same first four digits, but many of them do not. That's before adding anything else found too. My initial guesstimate of around 20k IDs is probably much closer to reality. As such, whilst I realise everyone's eager to get started with updating files, we should ideally put that on hold right now. It's very important to get a proper working ID allocation system in place, before getting too into this. Likely the test models used in this thread will need to be re-IDd once more as things become clearer. But with so few potential IDs, we really need to get this side of things sorted before actually editing large numbers of models.